×0: 1:int func (int ×0: 1:int func (int *p)( 11"▣ 2:1ntx=0; 2:int x =0; ×0: 3:for (int i =0;i<10;i++) ×0: 3: for(int1=0:i<10;i++) 10： 4 x +p[i]; x0: 4： ×+=p[i]; 1: 5:return x; 1: 5： return x; -1: 6:1 -1: 6:1 I 7:int main() 1g 7:int main() 1:8: int a(10]; 1:8: int a[10]; 11. 9 for(inti=0<10；i++) 1 9:for(int1=0;i<10;i++) 10:10: a[1】=1; -1：10: a[i]=1; 11:11: 1f(func(a)=10) 1:11: if(func(a） 1=10) ×0:12: return 1; -1:12: /return 1; -1:13: return 0; 1:13: return 0; -1:14:) -1:14:} (a)P(gcov) (b)P=Ps12Us12(gcov) Fig.7.The bug case of GCC #90420.gcov incorrectly reported that the if (func(a)!=10)in Line 11 was executed 11 times.Deleting Line #12 revealed this bug. with the same outputs as the original program.Initially,Le et One of the most attractive compiler testing techniques is al.[2]proposed to generate equivalent versions of the program based on the code coverage of a program's execution to by profiling program's execution and pruning unexecuted generate equivalence modulo inputs by stochastically pruning code inside.Once a program and its equivalent variant are its unexecuted code [2].[48].With the equivalence modulo constructed,both are fed to the compiler under test,and the in- inputs,we can differentially test compilers.It is obvious that consistencies of the outputs are checked.Following this work, the correctness of "equivalance"relies on the reliability of Athena [34]and Hermes [35]are developed subsequently. code coverage.Debugging is a common activity in software Athena [34]generates EMI by randomly inserting code into development which aims to locating the root cause of a fault. and removing statements from dead code regions.Hermes [35] Spectrum-Based Fault Localization(SBFL)is one of the most complements mutation strategies by operating on live code extensively studied debugging techniques which is heavily regions,which overcomes the limitations of mutating dead based on code coverage [4].[49]-[51].Under a specific test code regions. suite,SBFL leverages the code coverage and the corresponding In Cod.we followed the similar way to generate program failed/passed information to statistically infer which code is variants as EMI did,but focused on validating the correctness the root cause of a fault. of coverage profilers instead of optimization bugs in compilers As we can see,the correct code coverage information is As such,during the results verification,Cod not only checked one of the prerequisites for the techniques above,indicating the inconsistencies in terms of the outputs,but more impor- the importance of our work. tantly the coverage reports.Through our evaluations,it is also VI.CONCLUSION shown that only few bugs(1 among 23 confirmed bugs)can be discovered by looking at only the outputs.Moreover,different This paper presents Cod,an automated self-validator for from EMI performing a random modification,Cod mutates code coverage profilers based on metamorphic testing.Cod ad- the original program by aggressive statement pruning,thus dressed the limitation of the state-of-the-art differential testing triggering different coverage behaviors as much as possible. approach,and encouragingly found many previously unknown bugs which cannot be revealed by existing approaches. D.Techniques relied on code coverage ACKNOWLEDGMENT Code coverage is widely adopted in practice and extensively We thank the anonymous reviewers for their construc- used to facilitate many software engineering tasks,such as tive comments.We also thank the GCC and LLVM de- coverage-based regression testing,coverage-based compiler velopers especially Martin Liska for analyzing and fixing testing,and coverage-based debugging.In the context of our reported bugs.This work is supported by the Na- regression testing,test case prioritization and test suite aug- tional Key R&D Program of China(2018YFB1003901),the mentation are the two widely used techniques [36]-1431.The National Natural Science Foundation of China (61832009, former aims to improve the ability of test cases in finding 61432001,61932021,61690204,61772259,61772263, faults by scheduling test cases in a specific order [41].[43]. 61802165.61802168.61702256).the Natural Science Foun- [44].To achieve a high code coverage as fast as possible dation of Jiangsu Province BK20191247,BK20170652). is a common practice [45].The latter is to generate new the China Postdoctoral Science Foundation (2018T110481), test cases to strengthen the ability of a test suite in finding the Fundamental Research Funds for the Central Universities faults [42].[46],[47].In practice,it is often to generate new (020214380032.02021430047).We would also like to thank test cases to cover the source code affected by code changes. the support from the Collaborative Innovation Center of Novel Recent years have seen an increasing interest in compiler Software Technology and Industrialization,Jiangsu,China. testing which aims to validate the correctness of compilers. Yuming Zhou and Baowen Xu are the corresponding authors. 88×0: 1:int func (int *p) { 11 : 2: int x = 0; ×0: 3: for (int i = 0; i < 10; i++) 10 : 4: x += p[i]; 1 : 5: return x; -1: 6:} 1: 7:int main() { 1: 8: int a[10]; 11: 9: for (int i = 0; i < 10; i++) 10: 10: a[i] = 1; 11: 11: if (func(a) != 10) ×0: 12: return 1; -1: 13: return 0; -1: 14:} ×0: 1:int func (int *p) { 1 : 2: int x = 0; ×0: 3: for (int i = 0; i < 10; i++) ×0: 4: x += p[i]; 1 : 5: return x; -1: 6:} 1: 7:int main() { 1: 8: int a[10]; 1: 9: for (int i = 0; i < 10; i++) -1: 10: a[i] = 1; 1: 11: if (func(a) != 10) -1: 12: ; // return 1; 1: 13: return 0; -1: 14:} (a) P (gcov) (b) P = P\{s12}∪{s 12} (gcov) Fig. 7. The bug case of GCC #90420. gcov incorrectly reported that the if(func(a) != 10) in Line 11 was executed 11 times. Deleting Line #12 revealed this bug. with the same outputs as the original program. Initially, Le et al. [2] proposed to generate equivalent versions of the program by profiling program’s execution and pruning unexecuted code inside. Once a program and its equivalent variant are constructed, both are fed to the compiler under test, and the in￾consistencies of the outputs are checked. Following this work, Athena [34] and Hermes [35] are developed subsequently. Athena [34] generates EMI by randomly inserting code into and removing statements from dead code regions. Hermes [35] complements mutation strategies by operating on live code regions, which overcomes the limitations of mutating dead code regions. In Cod, we followed the similar way to generate program variants as EMI did, but focused on validating the correctness of coverage profilers instead of optimization bugs in compilers. As such, during the results verification, Cod not only checked the inconsistencies in terms of the outputs, but more impor￾tantly the coverage reports. Through our evaluations, it is also shown that only few bugs (1 among 23 confirmed bugs) can be discovered by looking at only the outputs. Moreover, different from EMI performing a random modification, Cod mutates the original program by aggressive statement pruning, thus triggering different coverage behaviors as much as possible. D. Techniques relied on code coverage Code coverage is widely adopted in practice and extensively used to facilitate many software engineering tasks, such as coverage-based regression testing, coverage-based compiler testing, and coverage-based debugging. In the context of regression testing, test case prioritization and test suite aug￾mentation are the two widely used techniques [36]–[43]. The former aims to improve the ability of test cases in finding faults by scheduling test cases in a specific order [41], [43], [44]. To achieve a high code coverage as fast as possible is a common practice [45]. The latter is to generate new test cases to strengthen the ability of a test suite in finding faults [42], [46], [47]. In practice, it is often to generate new test cases to cover the source code affected by code changes. Recent years have seen an increasing interest in compiler testing which aims to validate the correctness of compilers. One of the most attractive compiler testing techniques is based on the code coverage of a program’s execution to generate equivalence modulo inputs by stochastically pruning its unexecuted code [2], [48]. With the equivalence modulo inputs, we can differentially test compilers. It is obvious that the correctness of “equivalance” relies on the reliability of code coverage. Debugging is a common activity in software development which aims to locating the root cause of a fault. Spectrum-Based Fault Localization (SBFL) is one of the most extensively studied debugging techniques which is heavily based on code coverage [4], [49]–[51]. Under a specific test suite, SBFL leverages the code coverage and the corresponding failed/passed information to statistically infer which code is the root cause of a fault. As we can see, the correct code coverage information is one of the prerequisites for the techniques above, indicating the importance of our work. VI. CONCLUSION This paper presents Cod, an automated self-validator for code coverage profilers based on metamorphic testing. Cod ad￾dressed the limitation of the state-of-the-art differential testing approach, and encouragingly found many previously unknown bugs which cannot be revealed by existing approaches. ACKNOWLEDGMENT We thank the anonymous reviewers for their construc￾tive comments. We also thank the GCC and LLVM de￾velopers especially Martin Liska for analyzing and fixing ˇ our reported bugs. This work is supported by the Na￾tional Key R&D Program of China (2018YFB1003901), the National Natural Science Foundation of China (61832009, 61432001, 61932021, 61690204, 61772259, 61772263, 61802165, 61802168, 61702256), the Natural Science Foun￾dation of Jiangsu Province ( BK20191247, BK20170652), the China Postdoctoral Science Foundation (2018T110481), the Fundamental Research Funds for the Central Universities (020214380032, 02021430047). We would also like to thank the support from the Collaborative Innovation Center of Novel Software Technology and Industrialization, Jiangsu, China. Yuming Zhou and Baowen Xu are the corresponding authors. 88