Introduction
Related work
Microservice testing
Regression testing selection
-
Given: The program, P, the modified version of P, P′ and a test suite, T.
-
Problem: Find a subset of T, Ts, with which to test P′.
Frequent pattern mining
Belief propagation
Methodology
Service dependency mining
Data preprocessing
Service dependency matrix generation
Service | s1 | s2 | s3 | s4 | s5 | s6 | s7 | s8 | s9 |
---|---|---|---|---|---|---|---|---|---|
s1 | 0 | 0 | 0 | 0 | 0.25 | 0 | 0 | 0.20 | 0 |
s2 | 0.50 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
s3 | 0 | 0.22 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
s4 | 0 | 0 | 0.29 | 0 | 0 | 0 | 0.45 | 0 | 0 |
s5 | 0 | 0 | 0 | 0.30 | 0 | 0.32 | 0 | 0 | 0 |
s6 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
s7 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
s8 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
s9 | 0 | 0 | 0.26 | 0 | 0 | 0 | 0 | 0 | 0 |
Change impact analysis
Directed graph generation
Impact propagation calculation
-
➀ Computing all input messages. At this phase, because message propagations in the loop are not considered, it is transformed into a directed acyclic graph, and computing process converges in this phase;
-
➁ Computing message propagations in the loop. After external input messages enter the loop at the same time, impact degrees of nodes are assigned with Formula 4. Through numerical comparison, node sm with maximal impact degree can be obtained, which is denoted as pt(sm). Since messages propagated in the loop satisfy mij<pt(si), maximal value of messages in the loop is less than pt(sm). Therefore, no matter how many rounds messages propagates in the loop, pt(sm) does not changes. That is, the loop can be disconnected from the directed edge with sm as the end node, and the loop can be removed, which means calculation process converges.
-
➂ Computing all output messages. According to the calculation results of phase 2, output message of each node in the loop is calculated directly with Formula 5.
Test case selection
Empirical study
-
RQ1 Whether MRTS-BP is safe or not, and how the value of the frequent threshold c affects its safety.An RTS technique is safe if it will contain all test cases revealing faults in regression testing [5]. Safety determines the availability of RTS techniques. In MRTS-BP, the frequent threshold c is directly related to the number of mined frequent patterns, and then affects network structure of directed graph, which has a great influence on the results of change impact propagation. Therefore, the number of test cases selected by MRTS-BP is related to the value of c. It is necessary to analyze the relationship between c and the safety of MRTS-BP, and find out the range of c that can ensure the safety.
-
RQ2 On the premise of guaranteeing safety, whether MRTS-BP can save testing cost of microservice regression testing.Compared with retest-all strategy, RTS techniques save testing cost by reducing the number of test cases, but extra time consumption is caused for selecting. When time cost saved are greater than extra time consumption, RTS techniques can save time cost of regression testing overall. Through the experiments, the reduction of the number of test cases and testing time cost are both counted to determine whether MRTS-BP can save testing cost of microservice regression testing.
-
RQ3 Compare MRTS-BP with a typical artifacts based RTS approach.Theoretically, MRTS-BP does not rely on artifacts such as specifications, design models and code files. It is completely decoupled from techniques for constructing of the systems under testing, that is, the scalability of MRTS-BP is obviously better than artifacts based RTS approaches. In order to make a more comprehensive comparison, an RTS approach based on control flow analysis (RTS-CFG) [16] is chosen to compare with MRTS-BP to reveal the practicability of the two in microservice regression testing.
-
RQ4 How to choose selection strategies of MRTS-BP to optimize time cost.Existent satisfaction strategy, complete satisfaction strategy and k-existent satisfaction strategy are proposed in our approach to meet different testing requirements. To analyze the influence of selection strategies on the efficiency of MRTS-BP, experiments are needed to clarify how the selection strategies affect the number of test cases selected, the safety and the precision of MRTS-BP, which will be helpful to select appropriate strategies in practice.
Case introduction
Subject | No. of services | No. of logs | No. of versions | No. of test cases | No. of faults |
---|---|---|---|---|---|
m-Ticket | 61 | 40,000 | 5 | 1182 | 539 |
z-Shop | 43 | 20,000 | 4 | 913 | 427 |
Need | 169 | 2,000,000 | 4 | 847 | 781 |
JOA | 605 | 50,000,000 | 9 | 13,356 | 4783 |
Evaluation metrics
Experiments setup
-
Experiment 1: determine the safety of MRTS-BP and its relationship with frequent threshold cFor each case, from version v (v ≥ 2), following steps are carried out.(1)Taking logs of version v-1 as input, global item set and transaction set are generated respectively. After frequencies of all items obtained, the minimum value and maximum value are taken as the lower bound and upper bound of frequent threshold c respectively. Then divide the range of c into ten equal parts, and take the lower bound of each equal part as a value of c, which are denoted as ci (i = 1,2…10). The ten values of c are used respectively to generate SDMs through transaction set mining.(2)Based on the ten SDMs generated in step 1, ten CITs are generated respectively. For each CIT, the minimum non-zero element in CIT is taken as selection threshold p, and test cases are selected with existent satisfaction strategy.(3)NOF is counted from testing report of corresponding version. For each test suite selected, NSF and recall R are also counted.
-
Experiment 2: compare the ability to save testing cost of MRTS-BP and RTS-CFGFor each case, from version v (v ≥ 2), following steps are carried out.(1)Find out selected test suite in Experiment 1 with smallest number of test cases and R = 100% (set NA when there is no R = 100%). Collect time cost to select such test suite, including the whole process of MRTS-BP, which is TS of MRTS-BP. Then, TO and TS are computed from testing report of corresponding version, and then ET is computed.(2)Apply RTS-CFG on each version with available artifacts and tools [16] to select a test suite with R = 100% (set NA when there is no available artifacts or no R = 100%). Collect time cost to select such test suite, including the whole process of RTS-CFG, which is TS of RTS-CFG. Similarly, count TO, TS and compute ET.(3)For MRTS-BP and RTS-CFG, count the number of test cases selected and compute EN respectively.(4)Compare ET, EN of MRTS-BP and RTS-CFG.
-
Experiment 3: compare the effectiveness of MRTS-BP and RTS-CFGFor each case, from version v (v ≥ 2), following experimental steps are carried out.(1)With the results of step 1 in Experiment 2, compute recall R, precision P and F-measure F of MRTS-BP.(2)With the results of step 2 in Experiment 2, compute recall R, precision P and F-measure F of RTS-CFG (set NA when there is no available artifacts).(3)Compare R, P, F of MRTS-BP and RTS-CFG.
-
Experiment 4: analyze changing trends of EN, R, P and F with different selection strategies of MRTS-BP.For the last version of each case, following steps are carried out.(1)Find out the CIT of step 2 in Experiment 1 with smallest number of test cases and R = 100% (set NA when there is no R = 100%). Test cases are selected respectively with complete satisfaction strategy and k-existent satisfaction strategy (k = 2,3,4), if the maximal number of services contained in test paths is smaller than four, then the upper of k is set as such maximal number.(2)For each test case selected, compute EN, R, P and F.(3)Compare values of EN, R, P and F of different strategies by bar charts.
Results and discussion
Data and analysis
Subject & versions | ci/ Ri | |||||||||
---|---|---|---|---|---|---|---|---|---|---|
i = 1 | i = 2 | i = 3 | i = 4 | i = 5 | i = 6 | i = 7 | i = 8 | i = 9 | i = 10 | |
m-Ticket | ||||||||||
v2 | 0.005 | 0.026 | 0.048 | 0.069 | 0.091 | 0.113 | 0.134 | 0.156 | 0.177 | 0.199 |
100% | 100% | 100% | 92.3% | 91.2% | 75.6% | 61.3% | 50.6% | 32.4% | 9.35% | |
v3 | 0.008 | 0.030 | 0.053 | 0.076 | 0.099 | 0.122 | 0.145 | 0.168 | 0.191 | 0.214 |
100% | 100% | 100% | 100% | 100% | 78.2% | 74.6% | 49.1% | 27.2% | 5.62% | |
v4 | 0.003 | 0.040 | 0.077 | 0.114 | 0.151 | 0.189 | 0.226 | 0.263 | 0.300 | 0.337 |
100% | 100% | 100% | 100% | 93.6% | 75.2% | 69.3% | 55.1% | 38.7% | 7.20% | |
v5 | 0.003 | 0.038 | 0.074 | 0.110 | 0.145 | 0.181 | 0.217 | 0.252 | 0.288 | 0.324 |
100% | 100% | 100% | 100% | 95.5% | 73.2% | 71.6% | 43.3% | 29.6% | 8.13% | |
z-Shop | ||||||||||
v2 | 0.019 | 0.051 | 0.084 | 0.117 | 0.150 | 0.183 | 0.216 | 0.249 | 0.283 | 0.315 |
100% | 100% | 100% | 100% | 97.2% | 84.2% | 55.9% | 52.1% | 31.6% | 14.5% | |
v3 | 0.014 | 0.057 | 0.101 | 0.145 | 0.189 | 0.233 | 0.277 | 0.321 | 0.365 | 0.409 |
100% | 100% | 100% | 100% | 100% | 92.1% | 65.3% | 57.9% | 33.6% | 11.2% | |
v4 | 0.017 | 0.060 | 0.103 | 0.146 | 0.189 | 0.232 | 0.275 | 0.318 | 0.361 | 0.404 |
100% | 100% | 100% | 100% | 93.2% | 86.7% | 59.2% | 49.2% | 28.7% | 7.52% | |
Need | ||||||||||
v2 | 0.001 | 0.020 | 0.039 | 0.059 | 0.078 | 0.098 | 0.117 | 0.136 | 0.156 | 0.175 |
100% | 100% | 100% | 100% | 95.6% | 91.5% | 67.4% | 59.2% | 29.4% | 4.30% | |
v3 | 0.001 | 0.021 | 0.042 | 0.062 | 0.083 | 0.104 | 0.124 | 0.145 | 0.165 | 0.186 |
100% | 100% | 100% | 100% | 89.2% | 85.6% | 59.7% | 55.3% | 26.7% | 3.55% | |
v4 | 0.002 | 0.023 | 0.044 | 0.065 | 0.086 | 0.108 | 0.129 | 0.150 | 0.171 | 0.192 |
100% | 100% | 100% | 100% | 100% | 100% | 71.3% | 64.8% | 21.5% | 11.7% | |
JOA | ||||||||||
v2 | 0.001 | 0.028 | 0.056 | 0.083 | 0.111 | 0.139 | 0.166 | 0.194 | 0.221 | 0.249 |
100% | 100% | 100% | 100% | 100% | 85.3% | 80.6% | 56.2% | 53.4% | 21.4% | |
v3 | 0.001 | 0.030 | 0.059 | 0.088 | 0.117 | 0.147 | 0.176 | 0.205 | 0.234 | 0.263 |
100% | 100% | 100% | 97.2% | 92.5% | 86.2% | 79.4% | 63.1% | 57.1% | 19.2% | |
v4 | 0.002 | 0.037 | 0.072 | 0.107 | 0.142 | 0.177 | 0.212 | 0.247 | 0.282 | 0.317 |
100% | 100% | 100% | 100% | 90.8% | 83.6% | 77.5% | 58.2% | 50.6% | 24.7% | |
v5 | 0.003 | 0.038 | 0.074 | 0.109 | 0.145 | 0.181 | 0.216 | 0.252 | 0.287 | 0.323 |
100% | 100% | 100% | 100% | 100% | 86.2% | 81.5% | 49.6% | 43.2% | 19.8% | |
v6 | 0.003 | 0.042 | 0.081 | 0.120 | 0.159 | 0.198 | 0.237 | 0.276 | 0.315 | 0.354 |
100% | 100% | 100% | 100% | 93.7% | 87.9% | 79.6% | 53.7% | 47.7% | 18.9% | |
v7 | 0.003 | 0.040 | 0.078 | 0.116 | 0.154 | 0.192 | 0.229 | 0.267 | 0.305 | 0.343 |
100% | 100% | 100% | 100% | 100% | 95.1% | 83.5% | 69.2% | 46.6% | 22.5% | |
v8 | 0.019 | 0.059 | 0.100 | 0.141 | 0.182 | 0.222 | 0.263 | 0.304 | 0.345 | 0.386 |
100% | 100% | 100% | 100% | 100% | 95.6% | 89.7% | 70.4% | 47.5% | 20.6% | |
v9 | 0.008 | 0.044 | 0.081 | 0.118 | 0.155 | 0.192 | 0.229 | 0.266 | 0.303 | 0.340 |
100% | 100% | 100% | 100% | 100% | 100% | 91.6% | 68.3% | 56.6% | 14.1% |
Subject & versions | Original | MRTS-BP | RTS-CFG | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
NO | TO | NR | EN | TR | TS | ET | NR | EN | TR | TS | ET | |
m-Ticket | ||||||||||||
v2 | 932 | 521.24 | 410 | 56% | 223.55 | 0.13 | 57% | 437 | 53% | 240.12 | 190.94 | 17% |
v3 | 1057 | 654.43 | 507 | 52% | 295.59 | 0.17 | 55% | 540 | 49% | 337.31 | 82.83 | 36% |
v4 | 1094 | 725.98 | 612 | 44% | 366.99 | 0.19 | 49% | 625 | 43% | 409.50 | 34.07 | 39% |
v5 | 1182 | 743.11 | 591 | 50% | 334.72 | 0.21 | 55% | 631 | 47% | 401.72 | 13.40 | 44% |
z-Shop | ||||||||||||
v2 | 772 | 388.40 | 355 | 54% | 204.37 | 0.09 | 47% | 370 | 52% | 223.13 | 124.48 | 11% |
v3 | 869 | 533.93 | 521 | 40% | 258.63 | 0.10 | 52% | 568 | 34% | 338.81 | 68.02 | 24% |
v4 | 913 | 598.42 | 456 | 50% | 221.50 | 0.12 | 63% | 483 | 47% | 320.35 | 15.71 | 44% |
Need | ||||||||||||
v2 | 737 | 534.66 | 316 | 57% | 241.40 | 4.41 | 54% | 322 | 56% | 243.65 | 273.64 | 3% |
v3 | 805 | 685.31 | 362 | 55% | 367.82 | 5.12 | 46% | 394 | 51% | 395.14 | 102.18 | 27% |
v4 | 847 | 734.50 | 406 | 52% | 389.96 | 5.53 | 46% | 421 | 50% | 395.08 | 59.34 | 38% |
JOA | ||||||||||||
v2 | 10,742 | 7305.56 | 4726 | 56% | 2996.86 | 42.34 | 58% | NA | NA | NA | NA | NA |
v3 | 11,593 | 7752.70 | 6608 | 43% | 3791.62 | 57.15 | 50% | NA | NA | NA | NA | NA |
v4 | 11,989 | 7917.07 | 6833 | 43% | 3935.13 | 63.33 | 49% | NA | NA | NA | NA | NA |
v5 | 12,374 | 8253.80 | 6434 | 48% | 3494.98 | 75.22 | 57% | NA | NA | NA | NA | NA |
v6 | 12,507 | 8327.07 | 7003 | 44% | 4109.38 | 90.32 | 49% | NA | NA | NA | NA | NA |
v7 | 12,887 | 8569.12 | 5928 | 54% | 3185.81 | 99.54 | 62% | NA | NA | NA | NA | NA |
v8 | 13,252 | 9112.34 | 5963 | 55% | 3204.49 | 118.21 | 64% | NA | NA | NA | NA | NA |
v9 | 13,356 | 9257.31 | 6678 | 50% | 3886.37 | 137.36 | 57% | NA | NA | NA | NA | NA |
Subject & versions | MRTS-BP | RTS-CFG | ||||
---|---|---|---|---|---|---|
R | P | F | R | P | F | |
m-Ticket | ||||||
v2 | 100% | 38.2% | 0.55 | 100% | 35.8% | 0.53 |
v3 | 100% | 13.8% | 0.24 | 100% | 12.9% | 0.23 |
v4 | 100% | 11.9% | 0.21 | 100% | 11.6% | 0.21 |
v5 | 100% | 6.59% | 0.12 | 100% | 6.20% | 0.12 |
z-Shop | ||||||
v2 | 100% | 37.4% | 0.54 | 100% | 35.9% | 0.53 |
v3 | 100% | 12.2% | 0.21 | 100% | 11.2% | 0.20 |
v4 | 100% | 6.57% | 0.12 | 100% | 6.20% | 0.12 |
Need | ||||||
v2 | 100% | 86.7% | 0.93 | 100% | 85.1% | 0.92 |
v3 | 100% | 42.2% | 0.59 | 100% | 38.8% | 0.56 |
v4 | 100% | 37.9% | 0.55 | 100% | 36.5% | 0.53 |
JOA | ||||||
v2 | 100% | 40.1% | 0.57 | NA | NA | NA |
v3 | 100% | 5.00% | 0.09 | NA | NA | NA |
v4 | 100% | 1.30% | 0.03 | NA | NA | NA |
v5 | 100% | 0.90% | 0.02 | NA | NA | NA |
v6 | 100% | 3.50% | 0.07 | NA | NA | NA |
v7 | 100% | 1.00% | 0.02 | NA | NA | NA |
v8 | 100% | 0.60% | 0.01 | NA | NA | NA |
v9 | 100% | 0.90% | 0.02 | NA | NA | NA |
Subject & metrics | Selection strategy | ||||
---|---|---|---|---|---|
Existent satisfaction | Complete satisfaction | 2-existent satisfaction | 3-existent satisfaction | 4-existent satisfaction | |
m-Ticket | |||||
EN | 50% | 82% | 63% | 95% | NA |
R | 100% | 52.9% | 87.2% | 37.5% | NA |
P | 6.59% | 9.91% | 7.78% | 24.7% | NA |
F | 0.12 | 0.16 | 0.14 | 0.30 | NA |
z-Shop | |||||
EN | 50% | 89% | 68% | 94% | NA |
R | 100% | 66.7% | 100% | 43.6% | NA |
P | 6.57% | 20% | 10.3% | 23.9% | NA |
F | 0.12 | 0.31 | 0.19 | 0.31 | NA |
Need | |||||
EN | 52% | 92% | 67% | 89% | 96% |
R | 100% | 43.5% | 100% | 55.4% | 17.3% |
P | 37.9% | 98.1% | 55.1% | 91.6% | 78.6% |
F | 0.55 | 0.60 | 0.71 | 0.69 | 0.28 |
JOA | |||||
EN | 50% | 96% | 58% | 66% | 98% |
R | 100% | 40.0% | 100% | 82.3% | 36.0% |
P | 0.9% | 4.50% | 1.07% | 1.09% | 8.09% |
F | 0.02 | 0.08 | 0.02 | 0.02 | 0.13 |