IPS:Inconsistency-triggering Program Set Figure 5(b)shows the code coverage report r2 produced by R-IPS:Reduced IPS llvm-cov for the same program.r1 and r2 are two annotated versions of the source file.However,there are three differences Generator between ri and r2.First,they have different instrumentation program set P sites.On the one hand.lines 4.6~8.12.19,and 22 are = treated as the instrumentation sites by llvm-cov but not by gcov.On the other hand,lines 3 and 18 are treated as an =P1 instrumentation site by gcov but not by llvm-cov.Hence,only Yes the common instrumentation sites used by gcov and llvm-cov need to be compared later.Second,gcov and llvm-cov have p=Pil Reducer different annotation formats for execution frequency.A non- instrumentation site is noted as hyphen""in rI but is noted coverage tool ty coverage tool f2 R-IPS as a null string in r2(e.g.line 3).Besides,an unexecuted line is noted as hashes "#####in r (e.g.line 15),while it is noted as"0"in r2(e.g.line 9).Third,coverage statistics are r印ortr, T印otr2 Inspector included in r but are not available in r2.Figure 5(c)lists the △ unified coverage reports produced by our parser for gcov and Parser BugR印os llvm-cov respectively.As can be seen,there are 9 common instrumentation sites between them:lines 5,7,9,10,11,13, 14.15.20.and21. 、unified report unified report ur C.Comparing Unified Coverage Reports Comparer After obtaining the unified coverage reports ur and ur2,we consistentNo Filter use a tool Comparer to determine whether they are consistent. If not,the corresponding p and associated coverage reports Yes will be added to a set named IPS(Inconsistency-triggering test Program Set).When comparing the unified coverage reports, i=i+l Ye- Comparer only takes into account the execution frequencies of the common instrumentation sites among the n coverage s=Ps+{(p,ic,<m.n>)}> No tools.During the comparison of uri and ur2,the following types of cases might occur: Fig.4.The framework for code coverage validation Type A:one line code is marked as executed by ti but as unexecuted by t2: B.Parsing Code Coverage Reports Type B:one line code is marked as unexecuted by ti but For each test program p in the generated program set P, as executed by t2; we obtain the raw code coverage reports r1 and r2 emitted Type C:one line code is marked as executed k times by from the coverage tools t and t2 respectively.However,the t1 and as executed I times by t2 while kl. raw coverage reports cannot be directly compared since they Consequently,the inconsistent unified reports can be di- are presented in different formats.We thus develop a parser vided into seven categories (C001 Type C:C010 to transform ri into uri in a unified format (1 i 2).Type B;C100 Type A;C0ll Type B+C;C101 Specifically,uri is a sequence of two-tuple,including the Type A+C;C110:Type A+B:C111:Type A+B+C).If monitored program chunk and the corresponding execution the unified coverage reports corresponding to the test program frequency.Given a program p with N lines of source code.p are found to have an inconsistency category ic,it will be uri is a sequence of N two-tuples (nj,fi)in ascending order handled by the filter. according to the value of nj,1<j<N.Here,nj is the line Considering the two unified coverage reports shown in number and f;is the corresponding execution frequency.If Fig.5.Comparer will compare the execution frequencies of line n;is not an instrumentation site for a particular coverage the common 9 instrumentation sites to determine whether uri tool,fj is assigned-1. and ur2 are consistent.As can be seen,there are four incon- In our study,gcov and llvm-cov are selected as the subject sistent execution frequencies in the unified coverage reports code coverage tools,which are integrated with the mature between gcov and llvm-cov:Type C at line 5,Type C at line production compilers GCC and Clang respectively.In the 13,Type A at line 9,and Type B at line 15.Consequently,the following,we give an illustrating example to demonstrate inconsistency category of ur and ur2 is found to be C111.In how the parser works in real world.Figure 5(a)shows the other words,the inconsistency introduced by the test program code coverage report r produced by gcov for a program,and p belongs to the C111 category. 491IPS: Inconsistency-triggering Program Set R-IPS: Reduced IPS Bug Reports IPS = IPS + { ( p, ic, <ur1, ur2> ) } i <= |P| program set P Generator Parser Reducer i = i+1 IPS Comparer Inspector p = P[i] R-IPS Yes Yes duplicate? No No i = 1 coverage tool t1 coverage tool t2 report r1 report r2 unified report ur1 unified report ur2 consistent? No Filter Yes Fig. 4. The framework for code coverage validation B. Parsing Code Coverage Reports For each test program p in the generated program set P, we obtain the raw code coverage reports r1 and r2 emitted from the coverage tools t1 and t2 respectively. However, the raw coverage reports cannot be directly compared since they are presented in different formats. We thus develop a parser to transform ri into uri in a unified format (1 ≤ i ≤ 2). Specifically, uri is a sequence of two-tuple, including the monitored program chunk and the corresponding execution frequency. Given a program p with N lines of source code, uri is a sequence of N two-tuples (nj , fj ) in ascending order according to the value of nj , 1 ≤ j ≤ N. Here, nj is the line number and fj is the corresponding execution frequency. If line nj is not an instrumentation site for a particular coverage tool, fj is assigned −1. In our study, gcov and llvm-cov are selected as the subject code coverage tools, which are integrated with the mature production compilers GCC and Clang respectively. In the following, we give an illustrating example to demonstrate how the parser works in real world. Figure 5(a) shows the code coverage report r1 produced by gcov for a program, and Figure 5(b) shows the code coverage report r2 produced by llvm-cov for the same program. r1 and r2 are two annotated versions of the source file. However, there are three differences between r1 and r2. First, they have different instrumentation sites. On the one hand, lines 4, 6∼8, 12, 19, and 22 are treated as the instrumentation sites by llvm-cov but not by gcov. On the other hand, lines 3 and 18 are treated as an instrumentation site by gcov but not by llvm-cov. Hence, only the common instrumentation sites used by gcov and llvm-cov need to be compared later. Second, gcov and llvm-cov have different annotation formats for execution frequency. A non￾instrumentation site is noted as hyphen “-” in r1 but is noted as a null string in r2 (e.g. line 3). Besides, an unexecuted line is noted as hashes “#####” in r1 (e.g. line 15), while it is noted as “0” in r2 (e.g. line 9). Third, coverage statistics are included in r1 but are not available in r2. Figure 5(c) lists the unified coverage reports produced by our parser for gcov and llvm-cov respectively. As can be seen, there are 9 common instrumentation sites between them: lines 5, 7, 9, 10, 11, 13, 14, 15, 20, and 21. C. Comparing Unified Coverage Reports After obtaining the unified coverage reports ur1 and ur2, we use a tool Comparer to determine whether they are consistent. If not, the corresponding p and associated coverage reports will be added to a set named IPS (Inconsistency-triggering test Program Set). When comparing the unified coverage reports, Comparer only takes into account the execution frequencies of the common instrumentation sites among the n coverage tools. During the comparison of ur1 and ur2, the following types of cases might occur: • T ype A: one line code is marked as executed by t1 but as unexecuted by t2; • T ype B: one line code is marked as unexecuted by t1 but as executed by t2; • T ype C: one line code is marked as executed k times by t1 and as executed l times by t2 while k = l. Consequently, the inconsistent unified reports can be di￾vided into seven categories (C001 : T ype C; C010 : T ype B; C100 : T ype A; C011 : T ype B + C; C101 : T ype A+C; C110 : T ype A+B; C111 : T ype A+B+C). If the unified coverage reports corresponding to the test program p are found to have an inconsistency category ic, it will be handled by the filter. Considering the two unified coverage reports shown in Fig. 5, Comparer will compare the execution frequencies of the common 9 instrumentation sites to determine whether ur1 and ur2 are consistent. As can be seen, there are four incon￾sistent execution frequencies in the unified coverage reports between gcov and llvm-cov: T ype C at line 5, T ype C at line 13, T ype A at line 9, and T ype B at line 15. Consequently, the inconsistency category of ur1 and ur2 is found to be C111. In other words, the inconsistency introduced by the test program p belongs to the C111 category. 491