1 Introduction
2 Theoretical Background and an Illustrative Example
3 Related Work
3.1 Combinatorial t-wise test suite generators
3.2 Case studies on combinatorial t-wise interaction test generation
4 Overview of the proposed strategy
4.1 Q-learning Monte Carlo hyper-heuristic strategy
4.2 Cuckoo’s Levy Flight Perturbation Operator
4.3 Flower’s Local Pollination Operator
4.4 Flower’s global pollination operator
4.5 Jaya search operator
5 Empirical study design
-
RQ1: In what ways does the use of Q-EMCQ improve upon EMCQ?
-
RQ2: How good is the efficiency of Q-EMCQ in terms of test suite minimization when compared to the existing strategies?
-
RQ3: How good are combinatorial tests created using Q-EMCQ and 2-wise, 3-wise, and 4-wise at covering the code?
-
RQ4: How effective are the combinatorial tests created using Q-EMCQ for 2-wise, 3-wise, and 4-wise at detecting injected faults?
-
RQ5: How does Q-EMCQ with 2-wise, 3-wise, and 4-wise compare with manual testing in terms of cost?
-
RQ6: Apart from minimization problem (i.e., t-wise test generation), is Q-EMCQ sufficiently general to solve (maximization) optimization problem (i.e., module clustering)?
5.1 Experimental Benchmark setup
5.2 Experimental Benchmark Procedures
5.3 Case study object
5.3.1 Train control management system
-
Logic Block Replacement Operator (LRO) Replacing a logical block with another block from the same category (e.g., replacing an AND block with an XOR block in Fig. 5).
-
Comparison Block Replacement Operator (CRO) Replacing a comparison block with another block from the same category (e.g., replacing a greater-than (GT) block with a greater-or-equal (GE) block in Fig. 5).
-
Arithmetic Block Replacement Operator (ARO) Replacing an arithmetic block with another block from the same functional category (e.g., replacing a maximum (MAX) block with an addition (ADD) block).
-
Negation Insertion Operator (NIO) Negating an input or output connection between blocks (e.g., a variable var becomes NOT(var)).
-
Value Replacement Operator (VRO) Replacing a value of a constant variable connected to a block (e.g., replacing a constant value (\(\hbox {var}=5\)) with its boundary values (e.g., \(\hbox {var}=6\), \(\hbox {var}=4\))).
-
Timer Block Replacement Operator (TRO). Replacing a timer block with another block from the same timer category (e.g., replacing a timer-off (TOF) block with a timer-On (TON) block in Fig. 5).
5.3.2 Module clustering of class diagrams
6 Case study results
6.1 Answering RQ1–RQ5
6.2 RQ1: In what ways does the use of Q-EMCQ improve upon EMCQ?
MCA | Q-EMCQ | EMCQ | ||||||
---|---|---|---|---|---|---|---|---|
Size | Time (sec) | Size | Time (sec) | |||||
Best | Ave | Best | Ave | Best | Ave | Best | Ave | |
MCA \((N;\,2,\,5^{1}3^{3}2^{2})\) | 15 | 17.00 | 11.53 | 12.55 | 17 | 17.56 | 9.29 | 11.35 |
MCA \((N;\,3\, ,\, 5^{1}4^{2}3^{2})\) | 83 | 86.10 | 53.93 | 8.14 | 84 | 86.50 | 42.92 | 46.49 |
MCA \((N;\,4,\,5^{1}3^{2}2^{3})\) | 99 | 111.50 | 107.15 | 134.10 | 03 | 112.80 | 91.05 | 10.36 |
6.3 RQ2: How good is the efficiency of Q-EMCQ in terms of test suite minimization when compared to the existing strategies?
Meta-heuristic-based strategies | Other Strategies | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Q-EMCQ | HSS | PSTG | CS | DPSO | Jenny | TConfig | ITCH | PICT | TVG | IPOG | |||
T | v | Best | Ave | ||||||||||
2 | 2 | 7 | 7.00 | 7 | 6 | 6 | 7 | 8 | 7 | 6 | 7 | 7 | 8 |
3 | 14 | 15.35 | 14 | 15 | 15 | 14 | 16 | 15 | 15 | 16 | 15 | 17 | |
4 | 23 | 24.6 | 25 | 26 | 25 | 24 | 28 | 28 | 28 | 27 | 27 | 28 | |
5 | 35 | 35.9 | 35 | 37 | 37 | 34 | 37 | 40 | 45 | 40 | 42 | 42 | |
3 | 2 | 15 | 15.0 | 12 | 13 | 12 | 15 | 14 | 16 | 13 | 15 | 15 | 19 |
3 | 49 | 50.1 | 50 | 50 | 49 | 49 | 51 | 55 | 45 | 51 | 55 | 57 | |
4 | 112 | 115.4 | 121 | 116 | 117 | 112 | 124 | 112 | 112 | 124 | 134 | 208 | |
5 | 216 | 220.1 | 223 | 225 | 223 | 216 | 236 | 239 | 225 | 241 | 260 | 275 | |
4 | 2 | 27 | 32.2 | 29 | 29 | 27 | 34 | 31 | 36 | 40 | 32 | 31 | 48 |
3 | 148 | 153.55 | 155 | 155 | 155 | 150 | 169 | 166 | 216 | 168 | 167 | 185 | |
4 | 482 | 485.05 | 500 | 487 | 487 | 472 | 517 | 568 | 704 | 529 | 559 | 509 | |
5 | 1148 | 1162.40 | 1174 | 1176 | 1171 | 1148 | 1248 | 1320 | 1750 | 1279 | 1385 | 1349 |
Meta-heuristic-based strategies | Other Strategies | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Q-EMCQ | HSS | PSTG | CS | DPSO | Jenny | TConfig | ITCH | PICT | TVG | IPOG | |||
T
|
k
| Best | Ave | ||||||||||
2 | 3 |
9
| 9.80 |
9
|
9
|
9
|
9
|
9
| 10 |
9
| 10 | 10 | 11 |
4 |
9
| 9.00 |
9
|
9
|
9
|
9
| 13 | 10 |
9
| 13 | 12 | 12 | |
5 |
11
| 11.35 | 12 | 12 |
11
|
11
| 14 | 14 | 15 | 13 | 13 | 14 | |
6 |
13
| 14.20 |
13
|
13
|
13
| 14 | 15 | 15 | 15 | 14 | 15 | 15 | |
7 |
14
| 15.00 | 15 | 15 |
14
| 15 | 16 | 15 | 15 | 16 | 15 | 17 | |
8 |
15
| 15.60 |
15
|
15
|
15
|
15
| 17 | 17 |
15
| 16 |
15
| 17 | |
9 |
15
| 16.30 | 17 | 17 | 16 |
15
| 18 | 17 |
15
| 17 |
15
| 17 | |
10 | 16 | 16.90 | 17 | 17 | 17 | 16 | 19 | 17 |
15
| 18 | 16 | 20 | |
11 | 17 | 17.75 | 17 | 17 | 18 | 17 | 17 | 20 |
15
| 18 | 16 | 20 | |
12 | 16 | 17.95 | 18 | 18 | 18 | 16 | 19 | 20 |
15
| 19 | 16 | 20 | |
3 | 4 |
27
| 29.45 | 30 | 30 | 28 |
27
| 34 | 32 |
27
| 34 | 34 | 39 |
5 |
38
| 41.25 | 39 | 39 |
38
| 41 | 40 | 40 | 45 | 43 | 41 | 43 | |
6 |
33
| 39.00 | 45 | 45 | 43 |
33
| 51 | 48 | 45 | 48 | 49 | 53 | |
7 |
48
| 50.80 | 50 | 50 | 48 |
48
| 51 | 55 |
45
| 51 | 55 | 57 | |
8 |
51
| 53.65 | 54 | 54 | 53 | 52 | 58 | 58 |
45
| 59 | 60 | 63 | |
9 |
56
| 57.85 | 59 | 58 | 58 |
56
| 62 | 64 | 75 | 63 | 64 | 65 | |
10 |
59
| 61.25 | 62 | 62 | 62 |
59
| 65 | 68 | 75 | 65 | 68 | 68 | |
11 |
63
| 64.45 | 66 | 64 | 66 |
63
| 65 | 72 | 75 | 70 | 69 | 76 | |
12 | 66 | 67.45 | 67 | 67 | 70 |
65
| 68 | 77 | 75 | 72 | 70 | 76 | |
4 | 5 |
81
| 86.5 | 94 | 96 | 94 |
81
| 109 | 97 | 153 | 100 | 105 | 115 |
6 |
131
| 133.5 | 132 | 133 | 132 |
131
| 140 | 141 | 153 | 142 | 139 | 181 | |
7 |
150
| 153.3 | 154 | 155 | 154 |
150
| 169 | 166 | 216 | 168 | 172 | 185 | |
8 | 173 | 175.15 | 174 | 175 | 173 |
171
| 187 | 190 | 216 | 189 | 192 | 203 | |
9 |
167
| 188.65 | 195 | 195 | 195 | 187 | 206 | 213 | 306 | 211 | 215 | 238 | |
10 | 207 | 209.45 | 212 | 210 | 211 |
206
| 221 | 235 | 336 | 231 | 233 | 241 | |
11 |
221
| 225.05 | 223 | 222 | 229 |
221
| 236 | 258 | 348 | 249 | 250 | 272 | |
12 | 238 | 240.35 | 244 | 244 | 253 |
237
| 252 | 272 | 372 | 269 | 268 | 275 |
6.4 RQ3: How good are combinatorial tests created using Q-EMCQ for 2-wise, 3-wise and 4-wise at covering the code?
Measure | Test technique | Minimum | Maximum | Median | Mean | SD |
---|---|---|---|---|---|---|
Mutation score (%) | 2-wise | 0, 0 | 100, 0 | 48, 4 | 52, 3 | 34, 7 |
3-wise | 0, 0 | 100, 0 | 55, 5 | 57, 2 | 34, 3 | |
4-wise | 0, 0 | 100, 0 | 63, 4 | 60, 9 | 34, 2 | |
Branch coverage (%) | 2-wise | 50, 0 | 100, 0 | 85, 0 | 84, 1 | 14, 3 |
3-wise | 50, 0 | 100, 0 | 87, 5 | 86, 6 | 13, 0 | |
4-wise | 50, 0 | 100, 0 | 90, 6 | 88, 3 | 13, 4 | |
Cost (# test cases) | 2-wise | 6, 0 | 231, 0 | 8, 0 | 19, 6 | 41, 3 |
3-wise | 8, 0 | 732, 0 | 17, 0 | 50, 5 | 137, 3 | |
4-wise | 16, 0 | 1462, 0 | 43, 5 | 105, 8 | 273, 0 | |
Manual | 2, 0 | 62, 0 | 11, 0 | 17, 5 | 15, 3 |
6.5 RQ4: How effective are tests generated using Q-EMCQ for 2-wise, 3-wise, and 4-wise at detecting injected faults?
Measure | Method | Effect Size | p value |
---|---|---|---|
Mutation Score | 2-wise | 0.444 | 0.411 |
3-wise | |||
3-wise | 0.464 | 0.595 | |
4-wise | |||
2-wise | 0.412 | 0.193 | |
4-wise | |||
Coverage | 2-wise | 0.454 | 0.494 |
3-wise | |||
3-wise | 0.447 | 0.432 | |
4-wise | |||
2-wise | 0.412 | 0.186 | |
4-wise | |||
Cost (#test cases) | Manual | 0.536 | \(< 0.565\) |
2-wise | |||
Manual | 0.376 | \(< 0.054\) | |
3-wise | |||
Manual | 0.157 | \(< 0.001\) | |
4-wise |
6.6 RQ5: How does Q-EMCQ for 2-wise, 3-wise, and 4-wise compare with manual testing in terms of cost?
6.7 Answering RQ6
6.8 RQ6: Apart from the minimization problem (i.e., t-wise test generation), is Q-EMCQ sufficiently general to solve (maximization) optimization problem (i.e., module clustering)?
Case Studies | Hyper-heuristics | Meta-heuristics | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Q-EMCQ | EMCQ | Mod. Choi. Func. | Tabu HHH | TLBO | SCA | SOS | ||||||||
Ave MQ/ Ave Time (sec) | Best MQ | Ave MQ/ Ave Time (sec) | Best MQ | Ave MQ/ Ave Time (sec) | Best MQ | Ave MQ/ Ave Time (sec) | Best MQ | Ave MQ/ Ave Time (sec) | Best MQ | Ave MQ/ Ave Time (sec) | Best MQ | Ave MQ/ Ave Time (sec) | Best MQ | |
Card Payment Sys. | 2.031/ 40.876 | 2.226 | 1.976/ 39.016 | 2.171 | 2.002/ 43.232 | 2.226 | 2.110/ 54.876 | 2.226 | 1.999/ 33.531 | 2.226 | 1.957/ 36.953 | 2.078 | 1.983/ 45.080 | 2.117 |
Unified Inventory University | 2.315/ 41.673 | 2.899 | 2.288/ 39.312 | 2.721 | 2.543/ 47.673 | 2.899 | 2.392/ 54.673 | 2.899 | 2.135/ 40.080 | 2.623 | 2.140/ 37.782 | 2.588 | 2.118/ 47.282 | 2.581 |
Food Book | 3.43/ 63.5300 | 4.465 | 3.018/ 58.850 | 4.076 | 3.32/ 65.232 | 4.465 | 3.001/ 70.5300 | 4.377 | 3.012/ 68.130 | 3.741 | 2.991/ 56.798 | 3.551 | 2.881/ 68.250 | 3.305 |