TABLE V gcov llvm-cov N Source Code CONFIRMED BUGS 1 void func(int i){ #IDPrio. Status Type Source Affected Ver. 0 0 switch (i) gcov 83434 P4 New MissingCsmith v7.trunk 1 10 3 case 1:break; 10 case 2: gco R3465 Won't fix MissingCsmith v4~7.trunk 83486 default:break: Won't tx yss1ne●smith v4~-7.trunk 9 9 83505 P5 New v4~7.trunk 6 P5 MissingCsmith v7 trunk 10 0 gcov 83616 MissingCsmith 8 Assigned v/trunk 83617 3 Works Wrong Freq.Csmith v47 9 int main() v4 7.trunk 1 10 for oint Spurious i=0:i<10:++i) 83678 Won't fix Ic smith 83813 P3 trunk 10 Fixed : 0 func(i): gcov 85163 New return 0: Wrong Freq. trunk 85178 P3 Won't fx MissingCsmith v4~7.trunk 13 gcov 85179 P4 Assigned MissingCsmith trunk Fig 6. gcov 85188 SpuriousCsmith v4~7.trunk One program triggering multiple bugs CO 85197 P3 Won't fix Wrong Freq. Csmith v4~-7.trunk 85199 P4 Assigned Missing Csmith v4~7,trunk affected various versions of GCC and Clang for each bug. 6 gcov 85201 Assigned Wrong Freq. ICsmith trunk Note that as for GCC.we select GCC-4.8.0.GCC-5.4.1.GCC gco 85202 Won't fix Spuriou Csmith v4~7.trunk 6.4.0.GCC-7.2.0.and the latest trunk version (GCC-8.0).and 85717 Fixed SDuo1s●smith v4~7.trunk gcov 85218 P3 Won't fix Spurious Csmith v4~7.trunk for Clang.we select Clang-3.8,Clang-4.0.Clang-5.0,and the 02 gco 85219 Won't fix SpuriousCsmith v4~7,trunk latest trunk version (Clang-7.0).As can be seen.42 out of all 2CO 85225 Assigned Wrong frea. ●smith V4~/trunk the bugs can affect stable releases,and a considerable number 23 85243 P3 Won't fix Spurious Csmith v4~7.trunk of bugs have been latent in the old stable releases (such as gcov 85245 Won't fix SpunousICsmith V4~7 trunk 4 gcov 85272 P3 Won't fix Spurious v4~7.trunk GCC-4.8.0 and Clang-3.8)for many years. 5 85273 Won't fix Spurious Csmith v4~7.trunk 85274 Won't fix Spunous ICsmith v4~7,trunk D.Interesting Bugs Found 27 gcov 85276 New Wrong Freq. Csmith trunk 85294 Won't fix Spurious v4~7.trunk One program triggering multiple bugs.In Figure 6,Col- 85297 P3 Won't fix Spunous●gmth v4~7.trunk umn 4 lists a code snippet and Column 3 shows the line 85790 P3 Won't fix SpuriousCsmith v4~7,trunk 31 numbers.Column 1 and Column 2 display the coverage results 85332 Fixed Wrong Freq. GO v7.trunk 23 gcov 85333 P3 Won't fix Missing v4~7,trunk by gcov and llvm-cov respectively.This code snippet is a 85336 P4 New Wrong Freq. v4~7,trunk simple switch-statement inside a for-loop from the caller.The 4 gcov 85337 P5 Neu Wrong Freq Clang v7 trunk code at line 3 and line 4 actually executes only one time. 5 gco R533 Fixed Wrong Freq Clang v4~7,trunk 85349 P5 Spunous G●e v6~7.trunk however,they are wrongly marked as executed 10 times by 37 gcov 85350 P3 Fixed Spurious GCC v4 7.trunk llvm-cov and line 4 is wrongly marked as executed 9 times 89 85351 Assigned Missing v4~7,trunk gCO 85367 Works Wrong Freq. GCC by gcov.We have reported this case as Bug #85337 of gcov VS~/trunk 40 85370 P3 Fixed Spurious GCC trunk and Bug #37124 of llvm-cov.This case is quite simple and 85372 Fixed Missing G●● trunk common in real world.Unfortunately,this single case triggers 4 gcov 85377 P4 New Missing GCC v4~7.trunk 3llvm-co coverage bugs for both gcov and llvm-cov,indicating that 3346 P3 Fixed Missing Csmith v3~-5.trunk 4 vm-cov 3540 Contirmed SpuriousCsmith trunk coverage bugs are subtle and prevalent. llvm-cov 35426 P3 Fixed Spurious Csmith trunk Coverage bugs or compiler bugs?Figure 7 shows the 46llvm-cov 35495 Fixed Missing Csmith trunk coverage report for Bug #37082 of Clang.A definition of llvm-cov 35556 P3 Fixed MissingCsmith trunk 48 llvm-co 37001 SpuriousCsmith trunk strcmp function is first given at line 2,while strcmp is 49llvm-cov 37003 P3Confirmed SpuriousCsmith trunk redefined as the libc function.As a result,the calling of 50llvm-cov 37012 P3 Confirmed SpuriousCsmith trunk strcmp at line 6 will invoke the libc function with internal 51llvm-cov 37017 trunk Ilvm-cov 3704 P3iContrmed SpuriousCsmith trunk linkage,instead of the self-defined function (line 2).However. 53 llvm- 37046 SpuriousCsmith trunk in our test,.GCC outputs“-1”whereas Clang outputs“20”for 54llvm-cov 37070 P3 Missing Clang v3~5.trunk this case.That means,GCC invokes the correct libc function 55 llvm-cov 37071 P30 Spurious Clang trunk 56 llvm-co 37072 Confirmed ong Freq GCC trunk but Clang still invokes the self-defined function at line 2.That 57 vm-cov 3708 Clang v3~-5.trunk is why line 2 is reported as executed one time by llvm-cov but 58 llvm-cov 37083 Wrong Freq. trunk as not executed by gcov.As for this case,the root cause of 59llvm-cov 37084 Wrong Freq G●● trunk 60 llvm-cov P3Confirmed GCC trunk producing the incorrect coverage report is not the bugs inside 6 Spurious llvm-cov 37090 P3Confirmed Spurious trunk llvm-cov,but the bugs inside Clang instead. Ivm-co 37092 Spurious G●a trunk Code formatting problem.Figure 8 shows the coverage 63 llvm-cov 37099 P3 Spurious GCC trunk llvm-cov 3710 P3 Confirmed Spurious ●● trunk report for Bug #37102 of Clang.Line 8 is marked as executed vm-cov 37103 Wrong Freq. GCC v35.tunk twice by gcov,but wrongly marked as executed only once in llvm-co 37105 Confirmed GCC trunk llvm-cov.Along the execution,we can see that each of the four lvm-co 37107 P3 ong Freg G●● v3~5,trunk llvm-co 37124 P3 Confirme Wrong Freq- trunk statements at line 8 (i.e.foo ()goto L2;L1:bar()) 69 llvm-co 37125 P3Confirme Wrong Freg Clan v3~5.trunk executes actually only one time before the main function ovm-co Spurious V~5.trunk finishes.However,from the perspective of line coverage,the 495TABLE V CONFIRMED BUGS #ID Prio. Status Type Source Affected Ver. 1 gcov 83434 P4 New Missing Csmith v7,trunk 2 gcov 83465 P3 Won’t fix Missing Csmith v4∼7,trunk 3 gcov 83486 P3 Won’t fix Missing Csmith v4∼7,trunk 4 gcov 83505 P5 New Spurious Csmith v4∼7,trunk 5 gcov 83587 P5 New Missing Csmith v7,trunk 6 gcov 83616 P4 Assigned Missing Csmith v7,trunk 7 gcov 83617 P3 Works Wrong Freq. Csmith v4∼7 8 gcov 83678 P3 Won’t fix Spurious Csmith v4∼7,trunk 9 gcov 83813 P3 Fixed Missing Cmsith trunk 10 gcov 85163 P5 New Wrong Freq. Csmith trunk 11 gcov 85178 P3 Won’t fix Missing Csmith v4∼7,trunk 12 gcov 85179 P4 Assigned Missing Csmith trunk 13 gcov 85188 P3 Assigned Spurious Csmith v4∼7,trunk 14 gcov 85197 P3 Won’t fix Wrong Freq. Csmith v4∼7,trunk 15 gcov 85199 P4 Assigned Missing Csmith v4∼7,trunk 16 gcov 85201 P5 Assigned Wrong Freq. Csmith trunk 17 gcov 85202 P3 Won’t fix Spurious Csmith v4∼7,trunk 18 gcov 85217 P3 Fixed Spurious Csmith v4∼7,trunk 19 gcov 85218 P3 Won’t fix Spurious Csmith v4∼7,trunk 20 gcov 85219 P3 Won’t fix Spurious Csmith v4∼7,trunk 21 gcov 85225 P5 Assigned Wrong Freq. Csmith v4∼7,trunk 22 gcov 85243 P3 Won’t fix Spurious Csmith v4∼7,trunk 23 gcov 85245 P3 Won’t fix Spurious Csmith v4∼7,trunk 24 gcov 85272 P3 Won’t fix Spurious Csmith v4∼7,trunk 25 gcov 85273 P3 Won’t fix Spurious Csmith v4∼7,trunk 26 gcov 85274 P3 Won’t fix Spurious Csmith v4∼7,trunk 27 gcov 85276 P5 New Wrong Freq. Csmith trunk 28 gcov 85294 P3 Won’t fix Spurious Csmith v4∼7,trunk 29 gcov 85297 P3 Won’t fix Spurious Csmith v4∼7,trunk 30 gcov 85299 P3 Won’t fix Spurious Csmith v4∼7,trunk 31 gcov 85332 P3 Fixed Wrong Freq. GCC v7,trunk 32 gcov 85333 P3 Won’t fix Missing GCC v4∼7,trunk 33 gcov 85336 P4 New Wrong Freq. GCC v4∼7,trunk 34 gcov 85337 P5 New Wrong Freq. Clang v7,trunk 35 gcov 85338 P3 Fixed Wrong Freq. Clang v4∼7,trunk 36 gcov 85349 P5 New Spurious GCC v6∼7,trunk 37 gcov 85350 P3 Fixed Spurious GCC v4∼7,trunk 38 gcov 85351 P5 Assigned Missing GCC v4∼7,trunk 39 gcov 85367 P3 Works Wrong Freq. GCC v5∼7,trunk 40 gcov 85370 P3 Fixed Spurious GCC trunk 41 gcov 85372 P3 Fixed Missing GCC trunk 42 gcov 85377 P4 New Missing GCC v4∼7,trunk 43 llvm-cov 33465 P3 Fixed Missing Csmith v3∼5,trunk 44 llvm-cov 35404 P3 Confirmed Spurious Csmith trunk 45 llvm-cov 35426 P3 Fixed Spurious Csmith trunk 46 llvm-cov 35495 P3 Fixed Missing Csmith trunk 47 llvm-cov 35556 P3 Fixed Missing Csmith trunk 48 llvm-cov 37001 P3 Confirmed Spurious Csmith trunk 49 llvm-cov 37003 P3 Confirmed Spurious Csmith trunk 50 llvm-cov 37012 P3 Confirmed Spurious Csmith trunk 51 llvm-cov 37017 P3 Confirmed Spurious Csmith trunk 52 llvm-cov 37043 P3 Confirmed Spurious Csmith trunk 53 llvm-cov 37046 P3 Confirmed Spurious Csmith trunk 54 llvm-cov 37070 P3 Confirmed Missing Clang v3∼5,trunk 55 llvm-cov 37071 P3 Confirmed Spurious Clang trunk 56 llvm-cov 37072 P3 Confirmed Wrong Freq. GCC trunk 57 llvm-cov 37081 P3 Confirmed Missing Clang v3∼5,trunk 58 llvm-cov 37083 P3 Confirmed Wrong Freq. GCC trunk 59 llvm-cov 37084 P3 Confirmed Wrong Freq. GCC trunk 60 llvm-cov 37085 P3 Confirmed Spurious GCC trunk 61 llvm-cov 37090 P3 Confirmed Spurious GCC trunk 62 llvm-cov 37092 P3 Confirmed Spurious GCC trunk 63 llvm-cov 37099 P3 Confirmed Spurious GCC trunk 64 llvm-cov 37102 P3 Confirmed Spurious GCC trunk 65 llvm-cov 37103 P3 Confirmed Wrong Freq. GCC v3∼5,trunk 66 llvm-cov 37105 P3 Confirmed Spurious GCC trunk 67 llvm-cov 37107 P3 Confirmed Wrong Freq. GCC v3∼5,trunk 68 llvm-cov 37124 P3 Confirmed Wrong Freq. GCC trunk 69 llvm-cov 37125 P3 Confirmed Wrong Freq. Clang v3∼5,trunk 70 llvm-cov 37126 P3 Confirmed Spurious GCC v3∼5,trunk gcov llvm-cov #N Source Code 10 : 10 : 1 : 9 : 9 : − : 10 : − : 1 : 11 : 10 : 1 : − : 10 | 10 | 10 | 10 | 9 | 10 | 10 | | 1 | 11 | 10 | 1 | 1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 void func ( int i ) { switch (i) { case 1 : break ; case 2: ; default : break ; } } int main ( ) { for ( int i =0;i <10;++ i ) func ( i ); return 0 ; } Fig. 6. One program triggering multiple bugs affected various versions of GCC and Clang for each bug. Note that as for GCC, we select GCC-4.8.0, GCC-5.4.1, GCC- 6.4.0, GCC-7.2.0, and the latest trunk version (GCC-8.0), and for Clang, we select Clang-3.8, Clang-4.0, Clang-5.0, and the latest trunk version (Clang-7.0). As can be seen, 42 out of all the bugs can affect stable releases, and a considerable number of bugs have been latent in the old stable releases (such as GCC-4.8.0 and Clang-3.8) for many years. D. Interesting Bugs Found One program triggering multiple bugs. In Figure 6, Col￾umn 4 lists a code snippet and Column 3 shows the line numbers. Column 1 and Column 2 display the coverage results by gcov and llvm-cov respectively. This code snippet is a simple switch-statement inside a for-loop from the caller. The code at line 3 and line 4 actually executes only one time, however, they are wrongly marked as executed 10 times by llvm-cov and line 4 is wrongly marked as executed 9 times by gcov. We have reported this case as Bug #85337 of gcov and Bug #37124 of llvm-cov. This case is quite simple and common in real world. Unfortunately, this single case triggers coverage bugs for both gcov and llvm-cov, indicating that coverage bugs are subtle and prevalent. Coverage bugs or compiler bugs? Figure 7 shows the coverage report for Bug #37082 of Clang. A definition of strcmp function is first given at line 2, while strcmp is redefined as the libc function. As a result, the calling of strcmp at line 6 will invoke the libc function with internal linkage, instead of the self-defined function (line 2). However, in our test, GCC outputs “−1” whereas Clang outputs “20” for this case. That means, GCC invokes the correct libc function but Clang still invokes the self-defined function at line 2. That is why line 2 is reported as executed one time by llvm-cov but as not executed by gcov. As for this case, the root cause of producing the incorrect coverage report is not the bugs inside llvm-cov, but the bugs inside Clang instead. Code formatting problem. Figure 8 shows the coverage report for Bug #37102 of Clang. Line 8 is marked as executed twice by gcov, but wrongly marked as executed only once in llvm-cov. Along the execution, we can see that each of the four statements at line 8 (i.e. foo(); goto L2; L1: bar()) executes actually only one time before the main function finishes. However, from the perspective of line coverage, the 495