1 Introduction
-
The root cause category includes memory and semantic related bugs like improper handling of memory objects or semantic bugs, which are inconsistent with the original design requirements or the programmers’ intention.
-
The impact category includes performance and functionality related bugs, like the program keeps running but does not respond, halts abnormally, mistakenly changes user data, or functions correctly but runs/responds slowly.
-
The software components category includes bugs related to the components implementing core functionality, graphical user interfaces, runtime environment and communication, as well as data base handling.
1.1 Vocabulary of bug reports in source code files
1.2 Our aim and contributions
2 Previous approaches
Project | Source files | Bug reports | Period | Used also by |
---|---|---|---|---|
AspectJ | 6485 | 286 | 2002/07–2006/10 | |
Eclipse | 12863 | 3075 | 2004/10–2011/03 | |
SWT | 484 | 98 | 2004/10–2010/04 | |
ZXing | 391 | 20 | 2010/03–2010/09 | |
Tomcat | 2038 | 1056 | 2002/07–2014/01 |
Ye et al. (2014) |
ArgoUML | 1685 | 91 | 2002/01–2006/07 |
Moreno et al. (2014) |
Pillar1 | 4355 | 27 | 2012/03–2013/01 | – |
Pillar2 | 337 | 12 | 2010/05–2011/01 |
Dilshener and Wermelinger (2011) |
2.1 Using stack trace and structure
2.2 Version history and other data sources
2.3 Combining multiple information sources
2.4 User studies
3 Our Approach
Approach | Underlying IR logic | Version history | Similar report | Structure | File name | Stack trace |
---|---|---|---|---|---|---|
BugLocator | rVSM | No | Yes | No | No | No |
BRTracer | rVSM \(+\) segmentation | No | Yes | Yes | No | Yes |
BLUiR | Indri | No | Yes | Yes | No | No |
AmaLgam | Mixed | Yes | Yes | Yes | No | No |
BLIA | rVSM \(+\) segmentation | Yes | Yes | Yes | No | Yes |
Rahman | rVSM | Yes | Yes | No | Yes | No |
LtR | VSM | Yes | Yes | Yes | No | No |
LOBSTER | VSM | No | No | Yes | No | Yes |
ConCodeSe | lexicalScore \(+\) VSM | No | No | Yes | Yes | Yes |
3.1 Data processing
Project | BRs with only 1 file | BRs with 2–4 files | BRs with \(\ge \) 5 files |
---|---|---|---|
AspectJ | 71 | 149 | 66 |
Eclipse | 1525 | 1066 | 484 |
SWT | 59 | 32 | 7 |
ZXing | 14 | 5 | 1 |
Tomcat | 690 | 281 | 85 |
ArgoUML | 42 | 32 | 17 |
Pillar1 | 10 | 7 | 10 |
Pillar2 | 0 | 1 | 11 |
3.2 Ranking files
BR # | Affected Java file | BugLocator | BRTracer | ConCodeSe | 1: Full/code | 2: Full/all | 3: Stem/code | 4: Stem/all |
---|---|---|---|---|---|---|---|---|
100040 | Menu | 20 | 65 |
2
| 484 | 3 | 29 |
2
|
79107 | Combo | 6 | 8 |
3
| 26 | 29 |
3
| 8 |
84911 | FileDialog | 6 | 6 |
5
|
5
| 39 | 6 | 56 |
92757 | StyledTextListener | 87 | 74 |
3
| 4 |
3
| 75 | 72 |
3.2.1 Scoring with key positions (KP score)
BR# | Summary | Position |
---|---|---|
79268 | Program API does not work with GNOME 2.8 (libgnomevfs-WARNING) | First |
78559 | [consistency] Slider fires two selection events before mouse down | Second |
92341 | DBR - Add SWT.VIRTUAL style to Tree widget | Penultimate |
100040 | Slow down between 3.1 RC1 and N20050602 due to change to ImageList | Last |
3.2.2 Scoring with stack traces (ST score)
Project | # of BRs | # of BRs with Stack Traces | % of BRs with Stack Traces |
---|---|---|---|
AspectJ | 286 | 67 | 23 |
Eclipse | 3075 | 435 | 14 |
SWT | 98 | 4 | 4 |
ZXing | 20 | 0 | 0 |
Tomcat | 1056 | 83 | 8 |
ArgoUml | 91 | 5 | 5 |
Pillar1 | 27 | 1 | 4 |
Pillar2 | 12 | 0 | 0 |
3.2.3 Scoring with text terms (TT score)
3.2.4 Rationale behind the scoring values
4 Evaluation of the results
4.1 RQ1: scoring with file names in BRs
4.1.1 Scoring with words in key positions (KP score)
SWT BR# | Affected file | BugLocator | BRTracer | BLUiR | ConCodeSe | |
---|---|---|---|---|---|---|
Without | With | |||||
79268 | Program.java | 20 | 10 | – | 19 | 2 |
78559 | Slider.java | 2 | 4 | 1 | 5 | 1 |
92341 | Tree.java | 1 | 1 | 5 | 4 | 2 |
100040 | ImageList.java | 7 | 1 | 9 | 2 | 1 |
Project | Scoring | Top-1 (%) | Top-5 (%) | Top-10 (%) | MAP | MRR |
---|---|---|---|---|---|---|
AspectJ | KP+TT only | 12.9 | 43.0 | 59.1 | 0.17 | 0.33 |
ST+TT only
|
21.7
|
45.1
|
59.4
|
0.20
|
0.40
| |
TT only | 13.6 | 43.7 | 59.1 | 0.17 | 0.34 | |
Eclipse | KP+TT only | 19.5 | 35.2 | 48.0 | 0.21 | 0.32 |
ST+TT only
|
19.9
|
35.9
|
48.0
|
0.21
|
0.33
| |
TT only | 18.3 | 34.7 | 48.0 | 0.20 | 0.32 | |
SWT |
KP+TT only
|
62.2
|
79.6
|
89.8
|
0.60
|
0.81
|
ST+TT only | 43.9 | 76.5 | 88.8 | 0.50 | 0.70 | |
TT only | 42.9 | 76.5 | 88.8 | 0.50 | 0.69 | |
ZXing |
KP+TT only
|
40.0
|
65.0
|
80.0
|
0.46
|
0.53
|
ST+TT only | N/A | N/A | N/A | N/A | N/A | |
TT only | 25.0 | 60.0 | 75.0 | 0.38 | 0.42 | |
Tomcat |
KP+TT only
|
36.2
|
56.3
|
64.1
|
0.39
|
0.49
|
ST+TT only | 34.8 | 56.3 | 64.3 | 0.39 | 0.49 | |
TT only | 34.0 | 55.7 | 64.0 | 0.38 | 0.48 | |
ArgoUML | KP+TT only | 12.1 | 48.4 | 56.0 | 0.19 | 0.32 |
ST+TT only
|
13.2
|
48.4
|
56.0
|
0.20
|
0.33
| |
TT only | 12.1 | 48.4 | 56.0 | 0.19 | 0.32 | |
Pillar1 | KP+TT only | 7.4 | 33.3 | 40.7 | 0.10 | 0.36 |
ST+TT only | 7.4 | 33.3 | 40.7 | 0.10 | 0.36 | |
TT only | 7.4 | 33.3 | 40.7 | 0.10 | 0.36 | |
Pillar2 | KP+TT only | 25.0 | 66.7 | 75.0 | 0.20 | 0.71 |
ST+TT only | 25.0 | 66.7 | 75.0 | 0.20 | 0.71 | |
TT only | 25.0 | 66.7 | 75.0 | 0.20 | 0.71 |
4.1.2 Scoring with stack trace information (ST score)
AspectJ BR# | Exception description | Affected class | Stack pos. | BugLocator | BRTracer | BLUiR | ConCodeSe | |
---|---|---|---|---|---|---|---|---|
Without | With | |||||||
138143 | NullPointerException | EclipseSourceType | 1st | 1 | 2 | 5 | 5 | 1 |
158624 | UnsupportedOperation | ResolvedMemberImpl | 2nd | 16 | 6 | 56 | 7 | 3 |
153490 | IllegalStateException | ReferenceType | 4th | 122 | 3 | 74 | 11 | 1 |
Project | KP+ST | Top-1 (%) | Top-5 (%) | Top-10 (%) | MAP | MRR |
---|---|---|---|---|---|---|
AspectJ |
On
|
42.3
|
68.2
|
78.3
|
0.32
|
0.67
|
Off | 35.0 | 67.1 | 78.3 | 0.30 | 0.62 | |
Eclipse |
On
|
37.6
|
61.2
|
69.9
|
0.37
|
0.57
|
Off | 34.6 | 59.6 | 69.7 | 0.35 | 0.55 | |
SWT |
On
|
72.4
|
89.8
|
92.9
|
0.68
|
0.94
|
Off | 59.2 | 88.8 | 92.9 | 0.62 | 0.85 | |
ZXing |
On
|
55.0
|
75.0
|
80.0
|
0.55
|
0.68
|
Off | 35.0 | 70.0 | 80.0 | 0.45 | 0.52 | |
Tomcat |
On
|
51.5
|
69.2
|
75.4
|
0.52
|
0.66
|
Off | 49.1 | 68.6 | 75.3 | 0.51 | 0.65 | |
ArgoUML |
On
|
31.9
|
61.5
|
65.9
|
0.30
|
0.55
|
Off | 31.9 | 61.5 | 65.9 | 0.30 | 0.55 | |
Pillar1 |
On
|
29.6
|
59.3
|
63.0
|
0.22
|
0.69
|
Off | 29.6 | 59.3 | 63.0 | 0.22 | 0.69 | |
Pillar2 |
On
|
33.3
|
66.7
|
83.3
|
0.26
|
0.92
|
Off | 33.3 | 66.7 | 83.3 | 0.26 | 0.92 |
Project | Approach | Top-1 (%) | Top-5 (%) | Top-10 (%) | MAP | MRR |
---|---|---|---|---|---|---|
Aspectj |
halved
|
42.0
|
68.5
|
78.3
|
0.33
|
0.67
|
reversed | 34.6 | 65.0 | 77.6 | 0.30 | 0.61 | |
uniform | 37.4 | 68.2 | 78.3 | 0.31 | 0.64 | |
close2base | 41.6 | 68.2 | 78.3 | 0.33 | 0.67 | |
Eclipse | halved | 37.2 | 61.1 | 69.8 | 0.37 | 0.57 |
reversed | 35.7 | 61.2 | 69.9 | 0.36 | 0.56 | |
uniform | 36.6 | 61.2 | 69.9 | 0.36 | 0.56 | |
close2base
|
37.8
|
67.8
|
78.3
|
0.36
|
0.57
| |
SWT | halved | 71.4 | 89.8 | 92.9 | 0.68 | 0.93 |
reversed | 69.4 | 88.8 | 92.9 | 0.66 | 0.92 | |
uniform | 71.4 | 89.8 | 92.9 | 0.68 | 0.94 | |
close2base
|
72.4
|
89.8
|
92.9
|
0.68
|
0.94
| |
Tomcat | halved | 50.9 | 68.8 | 75.5 | 0.52 | 0.65 |
reversed | 50.5 | 68.8 | 75.5 | 0.52 | 0.65 | |
uniform | 51.0 | 69.2 | 75.4 | 0.52 | 0.66 | |
close2base
|
51.7
|
69.1
|
75.4
|
0.52
|
0.66
| |
ArgoUML |
any
| 31.9 | 61.5 | 65.9 | 0.30 | 0.55 |
ZXing |
any
| 55.0 | 75.0 | 80.0 | 0.51 | 0.63 |
Pillar1 |
any
| 29.6 | 59.3 | 63.0 | 0.22 | 0.69 |
Pillar2 |
any
| 33.3 | 66.7 | 83.3 | 0.26 | 0.92 |
4.1.3 Variations of score values
4.1.4 Overall results
Project | #BRs | Tools | Top-1 (%) | Top-5 (%) | Top-10 (%) | MAP | MRR |
---|---|---|---|---|---|---|---|
AspectJ | 282 |
ConCodeSe
|
42
|
69
|
78
|
0.34
|
0.70
|
BRTracer | 33 | 59 | 72 | 0.24 | 0.54 | ||
BugLocator | 18 | 42 | 52 | 0.17 | 0.34 | ||
Eclipse | 3043 |
ConCodeSe
|
38
|
61
|
70
|
0.37
|
0.57
|
BRTracer | 32 | 57 | 66 | 0.33 | 0.51 | ||
BugLocator | 32 | 55 | 65 | 0.32 | 0.51 | ||
SWT | 93 |
ConCodeSe
|
73
|
89
|
94
|
0.71
|
0.96
|
BRTracer | 50 | 79 | 88 | 0.54 | 0.71 | ||
BugLocator | 44 | 71 | 84 | 0.50 | 0.67 | ||
Zxing | 19 |
ConCodeSe
|
53
|
74
|
79
|
0.54
|
0.65
|
BRTracer | 53 | 68 | 74 | 0.51 | 0.60 | ||
BugLocator | 42 | 68 | 74 | 0.44 | 0.52 | ||
Tomcat | 942 |
ConCodeSe
|
54
|
72
|
79
|
0.56
|
0.71
|
BRTracer | 37 | 59 | 65 | 0.40 | 0.47 | ||
BugLocator | 0 | 64 | 74 | 0.28 | 0.35 | ||
ArgoUML | 61 |
ConCodeSe
|
29
|
57
|
60
|
0.30
|
0.57
|
BRTracer | 23 | 50 | 65 | 0.20 | 0.40 | ||
BugLocator | 21 | 45 | 57 | 0.20 | 0.42 | ||
Pillar1 | 19 |
ConCodeSe
|
47
|
58
|
58
|
0.35
|
0.96
|
BRTracer | 16 | 37 | 37 | 0.16 | 0.36 | ||
BugLocator | 21 | 37 | 42 | 0.19 | 0.43 | ||
Pillar2 | 0 |
all tools
|
N/A
|
N/A
|
N/A
|
N/A
|
N/A
|
Project | #BRs | Tools | Top-1 (%) | Top-5 (%) | Top-10 (%) | MAP | MRR |
---|---|---|---|---|---|---|---|
AspectJ | 4 |
ConCodeSe
|
25
|
75
|
75
|
0.27
|
0.62
|
BRTracer | 25 | 50 | 50 | 0.20 | 0.40 | ||
BugLocator | 0 | 0 | 25 | 0.05 | 0.10 | ||
Eclipse | 32 |
ConCodeSe
|
9
|
34
|
50
|
0.14
|
0.30
|
BRTracer | 16 | 34 | 50 | 0.15 | 0.29 | ||
BugLocator | 13 | 31 | 47 | 0.14 | 0.27 | ||
SWT | 5 |
ConCodeSe
|
40
|
80
|
80
|
0.27
|
0.98
|
BRTracer | 0 | 80 | 80 | 0.19 | 0.62 | ||
BugLocator | 40 | 100 | 100 | 0.32 | 0.90 | ||
Zxing | 1 |
ConCodeSe
|
100
|
100
|
100
|
0.70
|
1.00
|
BRTracer | 0 | 0 | 100 | 0.15 | 0.23 | ||
BugLocator | 0 | 100 | 100 | 0.21 | 0.31 | ||
Tomcat | 114 |
ConCodeSe
|
24
|
39
|
45
|
0.22
|
0.41
|
BRTracer | 10 | 21 | 32 | 0.11 | 0.16 | ||
BugLocator | 0 | 26 | 40 | 0.11 | 0.21 | ||
ArgoUML | 30 |
ConCodeSe
|
38
|
72
|
79
|
0.34
|
0.62
|
BRTracer | 10 | 45 | 52 | 0.15 | 0.33 | ||
BugLocator | 14 | 38 | 55 | 0.17 | 0.36 | ||
Pillar1 | 8 |
ConCodeSe
|
13
|
63
|
75
|
0.17
|
0.49
|
BRTracer | 13 | 13 | 13 | 0.09 | 0.18 | ||
BugLocator | 13 | 13 | 13 | 0.14 | 0.23 | ||
Pillar2 | 12 |
ConCodeSe
|
33
|
67
|
83
|
0.26
|
0.69
|
BRTracer | 17 | 67 | 83 | 0.17 | 0.61 | ||
BugLocator | 17 | 58 | 67 | 0.17 | 0.61 |
Project | #BRs | Tools | Top-1 (%) | Top-5 (%) | Top-10 (%) | MAP | MRR |
---|---|---|---|---|---|---|---|
AspectJ | 124 |
ConCodeSe
|
58
|
92
|
96
|
0.48
|
0.91
|
BRTracer | 44 | 78 | 90 | 0.34 | 0.68 | ||
BugLocator | 30 | 53 | 64 | 0.26 | 0.46 | ||
Eclipse | 1083 |
ConCodeSe
|
71
|
95
|
98
|
0.62
|
0.93
|
BRTracer | 56 | 84 | 91 | 0.53 | 0.78 | ||
BugLocator | 52 | 77 | 86 | 0.49 | 0.74 | ||
SWT | 68 |
ConCodeSe
|
88
|
99
|
100
|
0.83
|
1.00
|
BRTracer | 59 | 91 | 96 | 0.63 | 0.80 | ||
BugLocator | 57 | 79 | 91 | 0.59 | 0.78 | ||
Zxing | 9 |
ConCodeSe
|
89
|
100
|
100
|
0.86
|
0.97
|
BRTracer | 78 | 89 | 89 | 0.75 | 0.84 | ||
BugLocator | 67 | 89 | 89 | 0.66 | 0.75 | ||
Tomcat | 524 |
ConCodeSe
|
81
|
92
|
93
|
0.78
|
0.95
|
BRTracer | 56 | 77 | 81 | 0.57 | 0.65 | ||
BugLocator | 0 | 86 | 91 | 0.38 | 0.46 | ||
ArgoUML | 12 |
ConCodeSe
|
58
|
100
|
100
|
0.56
|
0.86
|
BRTracer | 50 | 100 | 100 | 0.46 | 0.68 | ||
BugLocator | 33 | 75 | 83 | 0.40 | 0.54 | ||
Pillar1 | 4 |
ConCodeSe
|
100
|
100
|
100
|
0.59
|
1.00
|
BRTracer | 50 | 100 | 100 | 0.56 | 0.82 | ||
BugLocator | 75 | 100 | 100 | 0.59 | 1.00 | ||
Pillar2 | 0 |
all tools
|
N/A
|
N/A
|
N/A
|
N/A
|
N/A
|
Project | #BRs | Tools | Top-1 (%) | Top-5 (%) | Top-10 (%) | MAP | MRR |
---|---|---|---|---|---|---|---|
AspectJ | 162 |
ConCodeSe
|
30
|
52
|
65
|
0.24
|
0.54
|
BRTracer | 25 | 43 | 57 | 0.17 | 0.42 | ||
BugLocator | 9 | 33 | 42 | 0.11 | 0.25 | ||
Eclipse | 1992 |
ConCodeSe
|
19
|
43
|
54
|
0.22
|
0.37
|
BRTracer | 20 | 41 | 52 | 0.22 | 0.36 | ||
BugLocator | 21 | 43 | 53 | 0.23 | 0.38 | ||
SWT | 30 |
ConCodeSe
|
33
|
67
|
77
|
0.36
|
0.66
|
BRTracer | 20 | 50 | 70 | 0.30 | 0.49 | ||
BugLocator | 13 | 57 | 70 | 0.26 | 0.46 | ||
Zxing | 11 |
ConCodeSe
|
27
|
55
|
64
|
0.29
|
0.43
|
BRTracer | 27 | 46 | 64 | 0.29 | 0.37 | ||
BugLocator | 18 | 55 | 64 | 0.23 | 0.32 | ||
Tomcat | 532 |
ConCodeSe
|
22
|
46
|
58
|
0.27
|
0.40
|
BRTracer | 13 | 33 | 42 | 0.18 | 0.22 | ||
BugLocator | 0 | 33 | 49 | 0.14 | 0.20 | ||
ArgoUML | 79 |
ConCodeSe
|
28
|
56
|
61
|
0.27
|
0.55
|
BRTracer | 14 | 41 | 54 | 0.15 | 0.34 | ||
BugLocator | 17 | 38 | 52 | 0.16 | 0.38 | ||
Pillar1 | 23 |
ConCodeSe
|
26
|
52
|
57
|
0.22
|
0.69
|
BRTracer | 9 | 17 | 17 | 0.06 | 0.22 | ||
BugLocator | 9 | 17 | 17 | 0.10 | 0.26 | ||
Pillar2 | 12 |
ConCodeSe
|
33
|
67
|
83
|
0.26
|
0.69
|
BRTracer | 17 | 67 | 83 | 0.17 | 0.61 | ||
BugLocator | 17 | 58 | 67 | 0.17 | 0.61 |
Project | BugLocator | BRTracer | ||||
---|---|---|---|---|---|---|
Better (%) | Same (%) | Worse (%) | Better (%) | Same (%) | Worse (%) | |
AspectJ | 45 | 48 | 7 | 30 | 55 | 15 |
Eclipse | 23 | 64 | 13 | 22 | 65 | 13 |
SWT | 24 | 69 | 6 | 21 | 74 | 4 |
ZXing | 10 | 85 | 5 | 20 | 75 | 5 |
Tomcat | 22 | 66 | 12 | 56 | 37 | 7 |
ArgoUML | 25 | 62 | 13 | 29 | 59 | 12 |
Pillar1 | 48 | 48 | 4 | 48 | 48 | 4 |
Pillar2 | 58 | 33 | 8 | 50 | 33 | 17 |
4.2 RQ2: scoring without similar bugs
BR# | Affected Java Files | BugLocator | BRTracer | BLUiR | ConCodeSe | |||
---|---|---|---|---|---|---|---|---|
rVSM | SimiScore | no SimiScore | SimiScore | structure | SimiScore | |||
78856 | OS.java | 37 | 36 | 89 | 88 | 3 | 6 | 1 |
79419 | Link.java | 18 | 18 | 4 | 4 | 31 | 6 | 1 |
OS.java | 58 | 58 | 48 | 47 | 11 | 21 | 1 | |
83262 | RTFTransfer | 224 | 224 | 214 | 214 | 43 | 79 | 35 |
TextTransfer | 202 | 202 | 198 | 197 | – | – | 36 | |
87676 | Tree.java | 49 | 21 | 10 | 6 | 4 | 3 | 2 |
Project | Approach | Top-1 (%) | Top-5 (%) | Top-10 (%) | MAP | MRR |
---|---|---|---|---|---|---|
AspectJ | BugLocator | 22.7 | 40.9 | 55.6 | 0.19 | 0.18 |
BRTracer | 38.8 | 58.7 | 66.8 | 0.27 | 0.47 | |
ConCodeSe
|
42.3
|
68.2
|
78.3
|
0.33
|
0.67
| |
Eclipse | BugLocator | 24.4 | 46.1 | 55.9 | 0.26 | 0.35 |
BRTracer | 29.6 | 51.9 | 61.8 | 0.30 | 0.40 | |
ConCodeSe
|
37.6
|
61.2
|
69.9
|
0.37
|
0.57
| |
SWT | BugLocator | 31.6 | 65.3 | 77.6 | 0.40 | 0.47 |
BRTracer | 46.9 | 79.6 | 88.8 | 0.53 | 0.59 | |
ConCodeSe
|
72.4
|
89.8
|
92.9
|
0.68
|
0.94
| |
ZXing | BugLocator | 40.0 | 55.0 | 70.0 | 0.41 | 0.48 |
BRTracer | 45.0 | 70.0 | 75.0 | 0.46 | 0.55 | |
ConCodeSe
|
55.0
|
75.0
|
80.0
|
0.55
|
0.68
| |
Tomcat | BugLocator | 42.1 | 62.4 | 71.0 | 0.26 | 0.33 |
BRTracer | 36.6 | 57.3 | 65.6 | 0.45 | 0.56 | |
ConCodeSe
|
49.9
|
69.2
|
75.4
|
0.52
|
0.66
| |
ArgoUML | BugLocator | 18.7 | 42.9 | 54.9 | 0.11 | 0.48 |
BRTracer | 18.7 | 46.2 | 54.9 | 0.20 | 0.38 | |
ConCodeSe
|
31.9
|
61.5
|
65.9
|
0.30
|
0.55
| |
Pillar1 | BugLocator | 18.5 | 29.6 | 33.3 | 0.17 | 0.37 |
BRTracer | 14.8 | 29.6 | 29.6 | 0.14 | 0.31 | |
ConCodeSe
|
29.6
|
59.3
|
63.0
|
0.22
|
0.69
| |
Pillar2 | BugLocator | 16.7 | 58.3 | 66.7 | 0.17 | 0.61 |
BRTracer | 16.7 | 66.7 | 83.3 | 0.17 | 0.61 | |
ConCodeSe
|
33.3
|
66.7
|
83.3
|
0.26
|
0.69
| |
Average | BugLocator | 27 | 50 | 61 | 0.25 | 0.40 |
BRTracer | 31 | 57 | 65 | 0.33 | 0.50 | |
ConCodeSe
|
44
|
69
|
76
|
0.40
|
0.57
|
4.3 RQ3: VSM’s contribution
Project | Approach | Top-1 (%) | Top-5 (%) | Top-10 (%) | MAP | MRR |
---|---|---|---|---|---|---|
AspectJ |
VSM
|
28.0
|
47.2
|
60.1
|
0.22
|
0.49
|
Lexical similarity | 20.6 | 44.4 | 59.4 | 0.20 | 0.39 | |
Eclipse |
VSM
|
23.3
|
45.7
|
56.6
|
0.25
|
0.41
|
Lexical similarity | 18.6 | 36.3 | 48.2 | 0.21 | 0.32 | |
SWT | VSM | 39.8 | 71.4 | 82.7 | 0.46 | 0.66 |
Lexical similarity
|
63.3
|
79.6
|
89.8
|
0.60
|
0.82
| |
ZXing | VSM | 35.0 | 55.0 | 65.0 | 0.37 | 0.45 |
Lexical similarity
|
40.0
|
70.0
|
80.0
|
0.47
|
0.54
| |
Tomcat | VSM | 28.1 | 47.7 | 57.6 | 0.32 | 0.43 |
Lexical similarity
|
34.0
|
56.6
|
64.0
|
0.38
|
0.48
| |
ArgoUML |
VSM
|
24.2
|
51.6
|
58.2
|
0.24
|
0.45
|
Lexical similarity | 13.2 | 48.4 | 56.0 | 0.20 | 0.33 | |
Pillar1 |
VSM
|
25.9
|
48.1
|
55.6
|
0.18
|
0.59
|
Lexical similarity | 7.4 | 33.3 | 40.7 | 0.10 | 0.36 | |
Pillar2 | VSM | 8.3 | 66.7 | 75.0 | 0.18 | 0.62 |
Lexical similarity
|
25.0
|
66.7
|
75.0
|
0.20
|
0.71
|
5 RQ4: user study
5.1 Study design
Company | Business nature | Study dates (2015) | Participant Id | Years of experience | Years of IDE Experience | Time spent on maintenance (%) |
---|---|---|---|---|---|---|
U | Finance | 29.06–03.07 | 1 | 20 | 10 | 30 |
2 | 15 | 10 | 70 | |||
S | Logistic | 06.07–04.08 | 3 | 9 | 9 | 20 |
A | Software Services | 05.06–16.10 | 4 | 11 | 11 | 40 |
Company | Application in production since | # of source code class files | # of bug reports used |
---|---|---|---|
U | 2014 | 2840 | 10 |
S | 2009 | 2240 | 10 |
A | 2013 | 4560 | 10 |
5.2 Results
5.2.1 Pre-session interview findings
5.2.2 Post-session findings
5.3 Evaluation of the results
6 Discussion
6.1 Threats to validity
Statistics | BugLocator | BRTracer | ||
---|---|---|---|---|
Top-1 | Top-10 | Top-1 | Top-10 | |
Z-value | −4.2686 | −3.2351 | −2.73934 | −3.9656 |
W-value | 254 | 576 | 487 | 543 |
p-value | 0.047504 | 0.0326 | 0.043432 | 0.0067 |