1 Introduction
2 Background
2.1 Finite state machine based testing
2.2 Mutation analysis in FSM-based testing
2.3 FSM-based testing methods
2.3.1 W method
2.3.2 HSI method
2.3.3 SPY method
2.4 Role-based access control
-
U and R are the finite sets of users and roles;
-
Pr is the finite set of permissions;
-
UR⊆U×R is the set of user-role assignments;
-
PR⊆Pr×R is the set of permission-role assignments;
-
≤ A ⊆R×R and ≤ I ⊆R×R are the role activation and inheritance hierarchies relationships;
-
I={AS,DS,AC,DC,AP,DP} is the finite set of types of RBAC requests which respectively stand for user-role assignments (AS), deassignments (DS), activations (AC) and deactivations (DC); and permission-role activations (AC) and deactivations (DC);
-
\(S_{u},D_{u}: U \rightarrow \mathbb {Z}^{+}\) are static and dynamic cardinality constraints on users;
-
\(S_{r},D_{r}: R \rightarrow \mathbb {Z}^{+}\) are static and dynamic cardinality constraints on roles;
-
SSoD,DSoD⊆2 R are the Static and Dynamic Separation of Duty (SoD) sets, respectively;
-
\(S_{s}: SSoD \rightarrow \mathbb {Z}^{+}\) specifies the cardinality of SSoD sets;
-
\(D_{s}: DSoD \rightarrow \mathbb {Z}^{+}\) specifies the cardinality of DSoD sets.
2.5 FSM-based testing of RBAC systems
-
S P is the set of states that P reach given its mutable elements;
-
s0∈S is the initial state where P currently stands given UR and PR;
-
I P is the input domain where I P ={(rq,up,r)} for all rq∈I, u∈{U∪Pr} and r∈R};
-
O is the output domain formed by granted and denied;
-
D=S P ×I P is the specification domain;
-
δ P :D→S P is the state transition function; and
-
λ P :D→O is the output function.
2.5.1 Test generation from FSM(P)
2.6 Test case prioritization
Fault revealed by test case | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
Test case | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
A | ∙ | ∙ | ||||||||
B | ∙ | ∙ | ∙ | ∙ | ||||||
C | ∙ | ∙ | ∙ | ∙ | ∙ | ∙ | ∙ | |||
D | ∙ | |||||||||
E | ∙ | ∙ | ∙ |
Test suite id | Test cases ordering | APFD |
---|---|---|
T1 | A,B,C,D,E | 0.5 |
T2 | E,D,C,B,A | 0.64 |
T3 | C,E,B,A,D | 0.84 |
2.7 Similarity testing
3 Similarity testing for RBAC systems
3.1 RBAC similarity
3.1.1 Simple dissimilarity:
Pairs (t
i
,t
j
) |
ndt
|
avg
| d
s
d
(t
i
,t
j
) |
---|---|---|---|
(t0,t1) | 4.0 | 2.0 | 2.0 |
(t0,t2) | 3.0 | 1.5 | 2.0 |
(t0,t3) | 4.0 | 2.0 | 2.0 |
(t1,t2) | 5.0 | 2.5 | 2.0 |
(t1,t3) | 6.0 | 3.0 | 2.0 |
(t2,t3) | 5.0 | 2.5 | 2.0 |
3.1.2 RBAC applicability:
-
Policy Applicability Degree (padP(t)), which shows the ratio of test inputs applicable to any RBAC constraint over the test case length;
-
Assignment Applicability Degree (asadP(t)), which shows the number of RBAC constraints related to assignment faults reacting to t;
-
Activation Applicability Degree (acadP(t)), which shows the number of RBAC constraints related to activation faults reacting to t; and
-
Permission Applicability Degree (pradP(t)), which shows the number of RBAC constraints related to permission faults reacting to t.
Test case t
i
|
\(pad_{P(t_{i})}\)
|
\(asad_{P(t_{i})}\)
|
\(acad_{P(t_{i})}\)
|
\(prad_{P(t_{i})}\)
| RA(t
i
) |
---|---|---|---|---|---|
t
0
| 0.77 | 0.0 | 0.0 | 0.0 | 0.77 |
t
1
| 0.77 | 0.0 | 0.0 | 0.0 | 0.77 |
t
2
| 0.77 | 1.0 | 0.0 | 0.0 | 1.77 |
t
3
| 1.0 | 1.0 | 1.0 | 0.0 | 3.0 |
Pairs (t
i
,t
j
) | d
s
d
(t
i
,t
j
) | AppValue(P,t
i
,t
j
) | PriorityValue(P,t
i
,t
j
) | d
r
s
(P,t
i
,t
j
) |
---|---|---|---|---|
(t0,t1) | 2.0 | 1.55 | 1.0 | 4.55 |
(t0,t2) | 2.0 | 2.55 | 1.0 | 5.55 |
(t0,t3) | 2.0 | 3.77 | 2.0 | 7.77 |
(t1,t2) | 2.0 | 2.55 | 1.0 | 5.55 |
(t1,t3) | 2.0 | 3.77 | 2.0 | 7.77 |
(t2,t3) | 2.0 | 4.77 | 2.0 | 8.77 |
3.2 Test prioritization algorithm
4 Experimental evaluation
ID | RBAC policy name |
U
|
R
|
S
u
|
D
u
|
S
r
|
D
r
|
SSoD
|
DSoD
| Mutants |
---|---|---|---|---|---|---|---|---|---|---|
P01 | 01_Masood2010Example1 | 2 | 1 | ∙ | ∙ | ∙ | ∙ | 9 | ||
P02 | 02_SeniorTraineeDoctor | 2 | 2 | ∙ | ∙ | ∙ | ∙ | ∙ | 17 | |
P03 | 03_ExperiencePointsv2 | 2 | 4 | ∙ | ∙ | ∙ | 11 | |||
P04 | 04_users11roles2v2 | 11 | 2 | ∙ | ∙ | ∙ | 28 | |||
P05 | 05_Masood2009P2v2 | 2 | 5 | ∙ | ∙ | ∙ | ∙ | ∙ | 48 |
FSM characteristics | Test suite length | Number of resets | Avg. test case length | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
ID | States | Transitions | W | HSI | SPY | W | HSI | SPY | W | HSI | SPY |
P01 | 8 | 64 | 1240 | 753 | 542 | 285 | 176 | 93 | 3.350 | 3.278 | 4.827 |
P02 | 21 | 336 n | 14704 | 8238 | 5841 | 2528 | 1408 | 751 | 4.816 | 4.850 | 6.777 |
P03 | 203 | 6496 | 776074 | 333550 | 213799 | 119586 | 51451 | 24001 | 5.489 | 5.482 | 7.907 |
P04 | 485 | 42680 | 13125662 | 6085633 | 2392981 | 2236388 | 993492 | 138766 | 4.869 | 5.125 | 16.24 |
P05 | 857 | 34280 | 7086325 | 2970528 | 1735818 | 835600 | 353836 | 159463 | 7.480 | 7.395 | 9.885 |
effsize
package (Torchiano 2017).4.1 Analysis of the complete test suites
Percent | 1 | 5 | 10 | 15 | 20 | 25 | 30 | 35 | 40 | 45 | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90 | 95 | 100 | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
P01 + W | Simple | 0.00 | 0.50 | 0.50 | 0.75 | 0.75 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
RBAC | 0.75 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |
Random | 0.57 | 0.91 | 0.98 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |
P01 + HSI | Simple | 0.00 | 0.00 | 0.50 | 0.50 | 0.50 | 0.50 | 0.50 | 0.50 | 0.75 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
RBAC | 0.5 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | |
Random | 0.26 | 0.78 | 0.87 | 0.93 | 0.97 | 0.98 | 0.98 | 0.99 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |
P01 + SPY | Simple | 0.0 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
RBAC | 0.00 | 0.75 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |
Random | 0.033 | 0.750 | 0.917 | 0.942 | 0.958 | 0.975 | 0.992 | 0.992 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
Percent | 1 | 5 | 10 | 15 | 20 | 25 | 30 | 35 | 40 | 45 | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90 | 95 | 100 | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
P02 + W | Simple | 0.29 | 0.43 | 0.71 | 0.71 | 0.71 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
RBAC | 0.86 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |
Random | 0.79 | 0.99 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |
P02 + HSI | Simple | 0.14 | 0.57 | 0.57 | 0.71 | 0.71 | 0.71 | 0.71 | 0.86 | 0.86 | 0.86 | 0.86 | 0.86 | 0.86 | 0.86 | 0.86 | 0.86 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
RBAC | 0.43 | 0.71 | 0.71 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |
Random | 0.73 | 0.95 | 0.99 | 0.99 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |
P02 + SPY | Simple | 0.57 | 0.57 | 0.71 | 0.71 | 0.71 | 0.86 | 0.86 | 0.86 | 0.86 | 0.86 | 0.86 | 0.86 | 0.86 | 0.86 | 0.86 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
RBAC | 0.86 | 0.86 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |
Random | 0.71 | 0.94 | 0.97 | 0.99 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
Scenario | RBAC | Simple | Random |
---|---|---|---|
P01 + W | 0.964 | 0.857 | 0.951 ±0.00518 |
P01 + HSI |
0.952
| 0.726 | 0.917 ±0.00933 |
P01 + SPY | 0.917 | 0.786 | 0.908 ±0.00814 |
P02 + W | 0.969 | 0.874 | 0.965 ±0.00254 |
P02 + HSI | 0.922 | 0.779 | 0.960 ±0.00369 |
P02 + SPY | 0.963 | 0.827 | 0.958 ±0.00382 |
P01 | P02 | |||||
---|---|---|---|---|---|---|
Hypothesis | W | HSI | SPY | W | HSI | SPY |
RBAC > Simple |
0.0284
|
0.0030
|
0.0131
|
0.0267
|
0.0001
|
0.0002
|
RBAC > Random | 0.0907 |
0.0071
| 0.0538 | 0.1855 | 0.9498 | 0.2919 |
Random > Simple |
0.0290
|
0.0045
|
0.0104
|
0.0272
|
0.0002
|
0.0002
|
4.2 Analysis of the subtest suites
Percent | 1 | 5 | 10 | 15 | 20 | 25 | 30 | 35 | 40 | 45 | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90 | 95 | 100 | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
P03 + W | Simple | 0.87 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
RBAC | 0.95 | 0.99 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |
Random | 0.88 | 0.99 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |
P03 + HSI | Simple | 0.82 | 0.99 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
RBAC | 0.82 | 0.85 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |
Random | 0.83 | 0.97 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |
P03 + SPY | Simple | 0.86 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
RBAC | 0.97 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |
Random | 0.89 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
Percent | 1 | 5 | 10 | 15 | 20 | 25 | 30 | 35 | 40 | 45 | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90 | 95 | 100 | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
P04 + W | Simple | 0.58 | 0.58 | 0.60 | 0.61 | 0.61 | 0.62 | 0.63 | 0.64 | 0.64 | 0.65 | 0.66 | 0.66 | 0.67 | 0.68 | 0.68 | 0.68 | 0.69 | 0.69 | 0.70 | 0.70 | 0.71 |
RBAC | 0.58 | 0.59 | 0.59 | 0.59 | 0.63 | 0.63 | 0.63 | 0.63 | 0.63 | 0.63 | 0.63 | 0.64 | 0.71 | 0.71 | 0.71 | 0.71 | 0.71 | 0.71 | 0.71 | 0.71 | 0.71 | |
Random | 0.57 | 0.59 | 0.59 | 0.60 | 0.61 | 0.62 | 0.62 | 0.63 | 0.64 | 0.64 | 0.65 | 0.66 | 0.67 | 0.67 | 0.68 | 0.68 | 0.69 | 0.69 | 0.70 | 0.70 | 0.71 | |
P04 + HSI | Simple | 0.58 | 0.59 | 0.61 | 0.62 | 0.63 | 0.65 | 0.65 | 0.66 | 0.67 | 0.68 | 0.70 | 0.71 | 0.71 | 0.72 | 0.73 | 0.74 | 0.75 | 0.76 | 0.77 | 0.77 | 0.78 |
RBAC | 0.58 | 0.58 | 0.61 | 0.61 | 0.61 | 0.61 | 0.62 | 0.76 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | 0.78 | |
Random | 0.57 | 0.59 | 0.60 | 0.62 | 0.63 | 0.64 | 0.65 | 0.66 | 0.68 | 0.69 | 0.70 | 0.71 | 0.72 | 0.73 | 0.73 | 0.74 | 0.75 | 0.76 | 0.76 | 0.77 | 0.78 | |
P04 + SPY | Simple | 0.58 | 0.62 | 0.66 | 0.69 | 0.72 | 0.74 | 0.76 | 0.79 | 0.80 | 0.82 | 0.84 | 0.85 | 0.87 | 0.88 | 0.90 | 0.90 | 0.91 | 0.93 | 0.95 | 0.98 | 0.99 |
RBAC | 0.63 | 0.72 | 0.87 | 0.93 | 0.97 | 0.97 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | |
Random | 0.59 | 0.65 | 0.71 | 0.75 | 0.79 | 0.82 | 0.85 | 0.87 | 0.90 | 0.91 | 0.92 | 0.94 | 0.95 | 0.96 | 0.96 | 0.97 | 0.98 | 0.98 | 0.98 | 0.99 | 0.99 |
Percent | 1 | 5 | 10 | 15 | 20 | 25 | 30 | 35 | 40 | 45 | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90 | 95 | 100 | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
P05 + W | Simple | 0.62 | 0.72 | 0.75 | 0.76 | 0.77 | 0.79 | 0.81 | 0.82 | 0.82 | 0.82 | 0.83 | 0.83 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.85 | 0.85 |
RBAC | 0.69 | 0.77 | 0.80 | 0.81 | 0.83 | 0.83 | 0.83 | 0.83 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.85 | 0.85 | 0.85 | 0.85 | 0.85 | 0.85 | 0.85 | 0.85 | |
Random | 0.62 | 0.73 | 0.76 | 0.79 | 0.80 | 0.81 | 0.82 | 0.83 | 0.83 | 0.83 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.85 | 0.85 | 0.85 | |
P05 + HSI | Simple | 0.60 | 0.71 | 0.73 | 0.74 | 0.75 | 0.76 | 0.77 | 0.78 | 0.79 | 0.80 | 0.81 | 0.82 | 0.83 | 0.83 | 0.83 | 0.83 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 |
RBAC | 0.66 | 0.75 | 0.77 | 0.79 | 0.80 | 0.81 | 0.81 | 0.82 | 0.83 | 0.83 | 0.83 | 0.83 | 0.83 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | |
Random | 0.61 | 0.71 | 0.74 | 0.75 | 0.77 | 0.78 | 0.79 | 0.80 | 0.80 | 0.81 | 0.81 | 0.82 | 0.82 | 0.83 | 0.83 | 0.83 | 0.83 | 0.84 | 0.84 | 0.84 | 0.84 | |
P05 + SPY | Simple | 0.65 | 0.73 | 0.75 | 0.76 | 0.78 | 0.80 | 0.82 | 0.82 | 0.83 | 0.83 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.85 | 0.85 |
RBAC | 0.76 | 0.80 | 0.83 | 0.83 | 0.83 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.85 | 0.85 | 0.85 | 0.85 | 0.85 | |
Random | 0.64 | 0.74 | 0.77 | 0.79 | 0.81 | 0.82 | 0.82 | 0.83 | 0.83 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.84 | 0.85 | 0.85 | 0.85 | 0.85 | 0.85 |
Scenario | APFD
R
B
A
C
| APFD
S
i
m
p
l
e
| APFD
R
a
n
d
o
m
|
---|---|---|---|
P03 + W | 0.9732 ±0.0016 | 0.9700 ±0.0019 | 0.9704 ±0.0005 |
P03 + HSI | 0.9607 ±0.0027 | 0.9675 ±0.0016 | 0.9665 ±0.0007 |
P03 + SPY | 0.9746 ±0.0010 | 0.9694 ±0.0019 | 0.9710 ±0.0005 |
P04 + W | 0.6466 ±0.0110 | 0.6417 ±0.0129 | 0.6385 ±0.0036 |
P04 + HSI |
0.7062
±0.0164
| 0.6770 ±0.0166 | 0.6757 ±0.0047 |
P04 + SPY |
0.9222
±0.0073
| 0.7943 ±0.0110 | 0.8561 ±0.0033 |
P05 + W |
0.8113
±0.0025
| 0.7888 ±0.0032 | 0.7977 ±0.0011 |
P05 + HSI |
0.7973
±0.0055
| 0.7729 ±0.0052 | 0.7779 ±0.0019 |
P05 + SPY |
0.8192
±0.0027
| 0.7948 ±0.0039 | 0.8012 ±0.0012 |
P03 | P04 | P05 | |||||||
---|---|---|---|---|---|---|---|---|---|
Hypothesis | W | HSI | SPY | W | HSI | SPY | W | HSI | SPY |
RBAC > Simple | 0.500000 | 0.977250 | 0.50 | 0.147876 |
0.007448
|
0.000048
|
0.000071
|
0.000071
|
0.000048
|
RBAC > Random | 0.500000 | 0.708059 | 0.185547 |
0.035099
|
0.004014
|
0.000048
|
0.000048
|
0.000048
|
0.000048
|
Random > Simple | 0.500000 | 0.899379 | 0.500000 | 0.999147 | 0.948464 |
0.000048
|
0.000182
|
0.038077
|
0.000477
|
4.2.1 Effect size to subtest suites
P03 | P04 | P05 | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
Alt. Hypothesis | Info | W | HSI | SPY | W | HSI | SPY | W | HSI | SPY |
RBAC > Random | Mean diff | 0,0029 | -0,0058 | 0,0036 | 0,0081 | 0,0305 | 0,0662 | 0,0137 | 0,0195 | 0,018 |
Median diff | 0,006 | -0,006 | 0,006 | 0,011 | 0,0174 | 0,0696 | 0,0119 | 0,0181 | 0,0192 | |
Cohen’s d | 0,6508 | -0,8075 | 1,2479 | 0,2722 | 0,6991 | 3,2109 | 1,942 | 1,3114 | 2,3887 | |
Hedges’ g | 0,6493 | -0,8056 | 1,245 | 0,2716 | 0,6975 | 3,2036 | 1,9376 | 1,3084 | 2,3832 | |
VD’s Â12 | 0,3311 | 0,733 | 0,2809 | 0,4164 | 0,3073 | 0,0283 | 0,1202 | 0,1794 | 0,0739 | |
RBAC > Simple | Mean diff | 0,0032 | -0,0067 | 0,0052 | 0,0049 | 0,0292 | 0,1279 | 0,0226 | 0,0244 | 0,0244 |
Median diff | 0,006 | 0 | 0,006 | 0,0055 | 0,0211 | 0,1346 | 0,0238 | 0,0197 | 0,0264 | |
Cohen’s d | 0,6726 | -1,1269 | 1,2625 | 0,1521 | 0,6615 | 5,0993 | 2,9639 | 1,7059 | 2,7021 | |
Hedges’ g | 0,6639 | -1,1122 | 1,2461 | 0,1501 | 0,6529 | 5,033 | 2,9254 | 1,6837 | 2,667 | |
VD’s Â12 | 0,6772 | 0,2267 | 0,79 | 0,5689 | 0,6933 | 1 | 0,9778 | 0,9067 | 0,9783 | |
Random > Simple | Mean diff | 0,0003 | -0,001 | 0,0015 | -0,0032 | -0,0013 | 0,0617 | 0,0089 | 0,0049 | 0,0064 |
Median diff | 0 | 0,006 | 0 | -0,0055 | 0,0037 | 0,065 | 0,0119 | 0,0016 | 0,0072 | |
Cohen’s d | 0,0662 | -0,1698 | 0,3239 | -0,0997 | -0,0314 | 2,1523 | 0,9026 | 0,3052 | 0,6204 | |
Hedges’ g | 0,0661 | -0,1694 | 0,3232 | -0,0995 | -0,0313 | 2,1474 | 0,9005 | 0,3045 | 0,619 | |
VD’s Â12 | 0,5181 | 0,4761 | 0,5825 | 0,4752 | 0,5146 | 0,9277 | 0,7554 | 0,5868 | 0,677 |
5 Discussion
5.1 RBAC similarity compared to the other criteria
5.2 Random prioritization vs. simple dissimilarity
5.3 Practical feasibility
6 Threats to validity
effsize
package (Torchiano 2017). The R scripts, input and output statistical data are included in the RBAC-BT repository.