ReFerences [24]W.Chan,S.C.Cheung,and K.R.Leung."Towards a metamorphic testing methodology for service-oriented software applications,"in Pro- [1]J.C.Miller and C.J.Maloney,"Systematic mistake analysis of digital ceedings of the 5th International Conference on Ouality Soffware.set. computer programs,"Commun.ACM.vol.6,no.2,pp.58-63,Feb. QSIC05.IEEE,2005.pp.470476 1963 [25]W.K.Chan,S.C.Cheung,and K.R.Leung."A metamorphic testing [2]V.Le.M.Afshari.and Z.Su."Compiler validation via equivalence approach for online testing of service-oriented software applications, modulo inputs,"in Proceedings of the 35th ACM SIGPLAN Conference International Journal of Web Services Research (IJWSR),vol.4.no.2 on Programming Language Design and Implementation,set.PLDI'14 pp.61-81.2007. New York.NY.USA:ACM,2014,pp.216-226. [26]M.Jiang.T.Y.Chen,F.-C.Kuo,and Z.Ding."Testing central processing [3]M.Bohme,V.-T.Pham,and A.Roychoudhury,"Coverage-based grey unit scheduling algorithms using metamorphic testing,"in Proceedings box fuzzing as markov chain,"in Proceedings of the ACM SIGSAC of the 4th IEEE International Conference on Software Engineering and Conference on Computer and Communications Security,ser.CCS'16. Service Science EEE,2013,pp.530-536. New York,NY,USA:ACM,2016,pp.1032-1043 [27]S.Beydeda. "Self-metamorphic-testing components,"in 30th Annual [4]J.A.Jones and M.J.Harrold,"Empirical evaluation of the tarantula International Computer Soffware and Applications Conference,ser automatic fault-localization technique,"in Proceedings of the 20th COMPSAC '06. IEEE/ACM International Conference on Automated Software Engineer- [28]M.Lindvall,D.Ganesan,R.Ardal,and R.E.Wiegand."Metamorphic ing,ser.ASE'05.New York,NY.USA:ACM,2005.pp.273-282. model-based testing applied on NASA DAT:An experience report." [5]Z.Zuo,S.-C.Khoo,and C.Sun,"Efficient predicated bug signature in Proceedings of the 37th International Conference on Soffware mining via hierarchical instrumentation,"in Proceedings of the 2014 Engineering-Volume 2.IEEE Press,2015,pp.129-138. International Symposium on Soffware Testing and Analysis,set.ISSTA [29]X.Xie.J.W.Ho,C.Murphy,G.Kaiser,B.Xu,and T.Y.Chen,"Testing ”14. New York.NY.USA:ACM,2014.pp.215-224. and validating machine learning classifiers by metamorphic testing, [6]Z.Zuo,L.Fang.S.-C.Khoo,G.Xu,and S.Lu,"Low-overhead and Journal of Systems and Software,vol.84,no.4,pp.544-558.2011. fully automated statistical debugging with abstraction refinement,"in [30]Z.Q.Zhou,S.Zhang,M.Hagenbuchner,T.Tse,F.-C.Kuo,and T.Y. Proceedings of the 2016 ACM SIGPLAN International Conference on Chen,"Automated functional testing of online search services,"Soffware Object-Oriented Programming.Systems,Languages,and Applications ser.OOPSLA '16. Testing.Verification and Reliability,vol.22,no.4,pp.221-243,2012. New York.NY.USA:ACM.2016,pp.881-896. [7]D.Lo,S.-C.Khoo,J.Han,and C.Liu,Mining Software Specifications: [31]Z.Q.Zhou.S.Xiang.and T.Y.Chen,"Metamorphic testing for software quality assessment:A study of search engines,"IEEE Transactions on Methodologies and Applications,Ist ed.Boca Raton,FL.USA:CRC Press,Inc.,2011. Software Engineering.vol.42.no.3.pp.264-284.2016. [32]T.Y.Chen,F.-C.Kuo,W.Ma,W.Susilo,D.Towey,J.Voas,and Z.Q. [8]Z.Zuo and S.-C.Khoo,"Mining dataflow sensitive specifications,"in Zhou,"Metamorphic testing for cybersecurity."Computer.vol.49.no.6. Formal Methods and Software Engineering,L.Groves and J.Sun,Eds pp.48-55.2016 Berlin,Heidelberg:Springer Berlin Heidelberg.2013,pp.36-52. [33]Z.Q.Zhou and L.Sun,"Metamorphic testing of driverless cars," [9]S.Park,R.W.Vuduc,and M.J.Harrold,"Falcon:Fault localization in Commun.ACM,vol.62,no.3,pp.61-67,Feb.2019. concurrent programs,"in Proceedings of the 32Nd ACM/IEEE Interna- [34]V.Le.C.Sun.and Z.Su,"Finding deep compiler bugs via guided tional Conference on Software Engineering Volume 1.set.ICSE '10. stochastic program mutation,"in Proceedings of the 2015 ACM SIG New York,NY,USA:ACM,2010,pp.245-254. PLAN International Conference on Object-Oriented Programming.Sys- [10]"Lighthouse -a code coverage explorer for reverse engineers,"https: tems,Languages,and Applications,ser.OOPSLA'15.New York.NY. //github.com/gaasedelen/lighthouse. [11]Y.Yang.Y.Zhou,H.Sun,Z.Su.Z.Zuo.L.Xu,and B.Xu,"Hunting for USA:ACM,20I5,Pp.386-399. [35]C.Sun.V.Le,and Z.Su."Finding compiler bugs via live code mutation." bugs in code coverage tools via randomized differential testing."in Pro- in Proceedings of the 2016 ACM SIGPLAN International Conference on ceedings of the 41st International Conference on Software Engineering. Obiect-Ortented programmmne svstems.Laneuages.and Apolicattous. ser.ICSE '19.Piscataway.NJ,USA:IEEE Press,2019,pp.488-499. [12]T.Y.Chen,S.C.Cheung,and S.M.Yiu."Metamorphic testing:A ser.OOPSLA'16.New York,NY,USA:ACM,2016,pp.849-863. [36]M.Gligoric,A.Groce,C.Zhang.R.Sharma.M.A.Alipour,and new approach for generating next test cases,"Technical Report HKUST- D.Marinov,"Guidelines for coverage-based comparisons of non- CS98-01.Department of Computer Science.Hong Kong.Tech.Rep.. adequate test suites,"ACM Trans.Sofiw.Eng.Methodol.,vol.24,no.4. 1998 pp.22:1-22:33.Sep.2015. [13]M.Liska,"Explanations on the coverage results under optimizations." [37]G.Fraser,M.Staats,P.McMinn,A.Arcuri,and F.Padberg."Does https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90420. automated white-box test generation really help software testers?"in [14]E.J.Weyuker,"On testing non-testable programs."The Computer Proceedings of the 2013 International Symposium on Software Testing Journal,vol.25.no.4.pp.465-470.1982 and Analysis.ser.ISSTA'13. .New York.NY.USA:ACM,2013.pp [15]D.Hamlet,"Predicting dependability by testing."in ACM S/GSOFT 291-301 Software Engineering Notes,vol.21,no.3.ACM.1996,pp.84-91. [16]L.Manolache and D.G.Kourie,"Software testing using model pro- [38]S.Yoo and M.Harman,"Regression testing minimization,selection and prioritization:A survey,"Softw.Test.Verif Reliab..vol.22,no.2.pp grams,"Software:Practice and Experience,vol.31,no.13,pp.1211- 67-120,Mar.2012. 1236.2001. [39]J.A.Jones and M.J.Harrold,"Test-suite reduction and prioritization for I17]“Gcov,”https:∥gcc.gnu.org/onlinedocs/gcc/Gcow.html modified condition/decision coverage,"IEEE Trans.Softw.Eng.,vol.29 [18] "Gcc."https://gcc.gnu.org/. no.3.pp.195-209.Mar.2003 [19]X.Yang.Y.Chen,E.Eide,and J.Regehr."Finding and understanding bugs in C compilers,"in Proceedings of the 32nd ACM SIGPLAN [401 L.Zhang.D.Marinov.L.Zhang,and S.Khurshid,"Regression muta- tion testing,"in Proceedings of the 2012 International Symposium on Conference on Programm ing Language nentation,ser PLDI'I.New York.NY.USA:ACM.2011 pp.283-294. Sofware Testing and Analysis,ser.ISSTA'12.New York,NY.USA: ACM,2012,pp.331-341. [20]C.Sun,V.Le.and Z.Su,"Finding and analyzing compiler warning de- [41]D.Hao,L.Zhang,L.Zhang,G.Rothermel,and H.Mei."A unified fects,"in Proceedings of the 38th International Conference on Software test case prioritization approach,"ACM Trans.Softw.Eng.Methodol.. Engineering.set.ICSE '16. New York,NY.USA:ACM,2016,pp. vol.24,no.2,pp.10:1-10:31,Dec.2014. 203-213. [21]V.Le.C.Sun,and Z.Su,"Randomized stress-testing of link-time [42]S.Artzi,J.Dolby,F.Tip,and M.Pistoia,"Directed test generation for effective fault localization,"in Proceedings of the 19th internationa optimizers."in Proceedings of the 2015 International Symposium on symposium on Software testing and analysis.ACM,2010.pp.49-60. Software Testing and Analysis,ser.ISSTA '15.New York,NY,USA: [43]Z.Li,M.Harman,and R.M.Hierons,"Search algorithms for regression ACM2015,pp.327-337. test case prioritization,"IEEE Trans.Soffw:Eng.,vol.33,no.4,pp.225- [22]T.Y.Chen,J.W.Ho.H.Liu,and X.Xie,"An innovative approach 237,Apr.2007. for testing bioinformatics programs using metamorphic testing,"BMC [44]G.Rothermel,R.J.Untch,and C.Chu,"Prioritizing test cases for Bioinfor atics.vol 10.no. .242009 23]LL Pullum and O.Ozmen. arly results from metamorphic testing of regression testing."IEEE Trans.Softw.Eng..vol.27,no.10,pp.929- 948.0ct2001 epidemiological models,"in Proceedings of the ASE/IEEE International Conference on BioMedical Computing,ser.BioMedCom'12. 89REFERENCES [1] J. C. Miller and C. J. Maloney, “Systematic mistake analysis of digital computer programs,” Commun. ACM, vol. 6, no. 2, pp. 58–63, Feb. 1963. [2] V. Le, M. Afshari, and Z. Su, “Compiler validation via equivalence modulo inputs,” in Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, ser. PLDI ’14. New York, NY, USA: ACM, 2014, pp. 216–226. [3] M. Bohme, V.-T. Pham, and A. Roychoudhury, “Coverage-based grey- ¨ box fuzzing as markov chain,” in Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’16. New York, NY, USA: ACM, 2016, pp. 1032–1043. [4] J. A. Jones and M. J. Harrold, “Empirical evaluation of the tarantula automatic fault-localization technique,” in Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineer￾ing, ser. ASE ’05. New York, NY, USA: ACM, 2005, pp. 273–282. [5] Z. Zuo, S.-C. Khoo, and C. Sun, “Efficient predicated bug signature mining via hierarchical instrumentation,” in Proceedings of the 2014 International Symposium on Software Testing and Analysis, ser. ISSTA ’14. New York, NY, USA: ACM, 2014, pp. 215–224. [6] Z. Zuo, L. Fang, S.-C. Khoo, G. Xu, and S. Lu, “Low-overhead and fully automated statistical debugging with abstraction refinement,” in Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, ser. OOPSLA ’16. New York, NY, USA: ACM, 2016, pp. 881–896. [7] D. Lo, S.-C. Khoo, J. Han, and C. Liu, Mining Software Specifications: Methodologies and Applications, 1st ed. Boca Raton, FL, USA: CRC Press, Inc., 2011. [8] Z. Zuo and S.-C. Khoo, “Mining dataflow sensitive specifications,” in Formal Methods and Software Engineering, L. Groves and J. Sun, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 36–52. [9] S. Park, R. W. Vuduc, and M. J. Harrold, “Falcon: Fault localization in concurrent programs,” in Proceedings of the 32Nd ACM/IEEE Interna￾tional Conference on Software Engineering - Volume 1, ser. ICSE ’10. New York, NY, USA: ACM, 2010, pp. 245–254. [10] “Lighthouse - a code coverage explorer for reverse engineers,” https: //github.com/gaasedelen/lighthouse. [11] Y. Yang, Y. Zhou, H. Sun, Z. Su, Z. Zuo, L. Xu, and B. Xu, “Hunting for bugs in code coverage tools via randomized differential testing,” in Pro￾ceedings of the 41st International Conference on Software Engineering, ser. ICSE ’19. Piscataway, NJ, USA: IEEE Press, 2019, pp. 488–499. [12] T. Y. Chen, S. C. Cheung, and S. M. Yiu, “Metamorphic testing: A new approach for generating next test cases,” Technical Report HKUST￾CS98-01, Department of Computer Science, Hong Kong, Tech. Rep., 1998. [13] M. Liska, “Explanations on the coverage results under optimizations.” ˇ https://gcc.gnu.org/bugzilla/show bug.cgi?id=90420. [14] E. J. Weyuker, “On testing non-testable programs,” The Computer Journal, vol. 25, no. 4, pp. 465–470, 1982. [15] D. Hamlet, “Predicting dependability by testing,” in ACM SIGSOFT Software Engineering Notes, vol. 21, no. 3. ACM, 1996, pp. 84–91. [16] L. Manolache and D. G. Kourie, “Software testing using model pro￾grams,” Software: Practice and Experience, vol. 31, no. 13, pp. 1211– 1236, 2001. [17] “Gcov,” https://gcc.gnu.org/onlinedocs/gcc/Gcov.html. [18] “Gcc,” https://gcc.gnu.org/. [19] X. Yang, Y. Chen, E. Eide, and J. Regehr, “Finding and understanding bugs in C compilers,” in Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, ser. PLDI ’11. New York, NY, USA: ACM, 2011, pp. 283–294. [20] C. Sun, V. Le, and Z. Su, “Finding and analyzing compiler warning de￾fects,” in Proceedings of the 38th International Conference on Software Engineering, ser. ICSE ’16. New York, NY, USA: ACM, 2016, pp. 203–213. [21] V. Le, C. Sun, and Z. Su, “Randomized stress-testing of link-time optimizers,” in Proceedings of the 2015 International Symposium on Software Testing and Analysis, ser. ISSTA ’15. New York, NY, USA: ACM, 2015, pp. 327–337. [22] T. Y. Chen, J. W. Ho, H. Liu, and X. Xie, “An innovative approach for testing bioinformatics programs using metamorphic testing,” BMC Bioinformatics, vol. 10, no. 1, p. 24, 2009. [23] L. L. Pullum and O. Ozmen, “Early results from metamorphic testing of epidemiological models,” in Proceedings of the ASE/IEEE International Conference on BioMedical Computing, ser. BioMedCom ’12. [24] W. Chan, S. C. Cheung, and K. R. Leung, “Towards a metamorphic testing methodology for service-oriented software applications,” in Pro￾ceedings of the 5th International Conference on Quality Software, ser. QSIC ’05. IEEE, 2005, pp. 470–476. [25] W. K. Chan, S. C. Cheung, and K. R. Leung, “A metamorphic testing approach for online testing of service-oriented software applications,” International Journal of Web Services Research (IJWSR), vol. 4, no. 2, pp. 61–81, 2007. [26] M. Jiang, T. Y. Chen, F.-C. Kuo, and Z. Ding, “Testing central processing unit scheduling algorithms using metamorphic testing,” in Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science. IEEE, 2013, pp. 530–536. [27] S. Beydeda, “Self-metamorphic-testing components,” in 30th Annual International Computer Software and Applications Conference, ser. COMPSAC ’06. [28] M. Lindvall, D. Ganesan, R. Ardal, and R. E. Wiegand, “Metamorphic ´ model-based testing applied on NASA DAT: An experience report,” in Proceedings of the 37th International Conference on Software Engineering-Volume 2. IEEE Press, 2015, pp. 129–138. [29] X. Xie, J. W. Ho, C. Murphy, G. Kaiser, B. Xu, and T. Y. Chen, “Testing and validating machine learning classifiers by metamorphic testing,” Journal of Systems and Software, vol. 84, no. 4, pp. 544–558, 2011. [30] Z. Q. Zhou, S. Zhang, M. Hagenbuchner, T. Tse, F.-C. Kuo, and T. Y. Chen, “Automated functional testing of online search services,” Software Testing, Verification and Reliability, vol. 22, no. 4, pp. 221–243, 2012. [31] Z. Q. Zhou, S. Xiang, and T. Y. Chen, “Metamorphic testing for software quality assessment: A study of search engines,” IEEE Transactions on Software Engineering, vol. 42, no. 3, pp. 264–284, 2016. [32] T. Y. Chen, F.-C. Kuo, W. Ma, W. Susilo, D. Towey, J. Voas, and Z. Q. Zhou, “Metamorphic testing for cybersecurity,” Computer, vol. 49, no. 6, pp. 48–55, 2016. [33] Z. Q. Zhou and L. Sun, “Metamorphic testing of driverless cars,” Commun. ACM, vol. 62, no. 3, pp. 61–67, Feb. 2019. [34] V. Le, C. Sun, and Z. Su, “Finding deep compiler bugs via guided stochastic program mutation,” in Proceedings of the 2015 ACM SIG￾PLAN International Conference on Object-Oriented Programming, Sys￾tems, Languages, and Applications, ser. OOPSLA ’15. New York, NY, USA: ACM, 2015, pp. 386–399. [35] C. Sun, V. Le, and Z. Su, “Finding compiler bugs via live code mutation,” in Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, ser. OOPSLA ’16. New York, NY, USA: ACM, 2016, pp. 849–863. [36] M. Gligoric, A. Groce, C. Zhang, R. Sharma, M. A. Alipour, and D. Marinov, “Guidelines for coverage-based comparisons of non￾adequate test suites,” ACM Trans. Softw. Eng. Methodol., vol. 24, no. 4, pp. 22:1–22:33, Sep. 2015. [37] G. Fraser, M. Staats, P. McMinn, A. Arcuri, and F. Padberg, “Does automated white-box test generation really help software testers?” in Proceedings of the 2013 International Symposium on Software Testing and Analysis, ser. ISSTA ’13. New York, NY, USA: ACM, 2013, pp. 291–301. [38] S. Yoo and M. Harman, “Regression testing minimization, selection and prioritization: A survey,” Softw. Test. Verif. Reliab., vol. 22, no. 2, pp. 67–120, Mar. 2012. [39] J. A. Jones and M. J. Harrold, “Test-suite reduction and prioritization for modified condition/decision coverage,” IEEE Trans. Softw. Eng., vol. 29, no. 3, pp. 195–209, Mar. 2003. [40] L. Zhang, D. Marinov, L. Zhang, and S. Khurshid, “Regression muta￾tion testing,” in Proceedings of the 2012 International Symposium on Software Testing and Analysis, ser. ISSTA ’12. New York, NY, USA: ACM, 2012, pp. 331–341. [41] D. Hao, L. Zhang, L. Zhang, G. Rothermel, and H. Mei, “A unified test case prioritization approach,” ACM Trans. Softw. Eng. Methodol., vol. 24, no. 2, pp. 10:1–10:31, Dec. 2014. [42] S. Artzi, J. Dolby, F. Tip, and M. Pistoia, “Directed test generation for effective fault localization,” in Proceedings of the 19th international symposium on Software testing and analysis. ACM, 2010, pp. 49–60. [43] Z. Li, M. Harman, and R. M. Hierons, “Search algorithms for regression test case prioritization,” IEEE Trans. Softw. Eng., vol. 33, no. 4, pp. 225– 237, Apr. 2007. [44] G. Rothermel, R. J. Untch, and C. Chu, “Prioritizing test cases for regression testing,” IEEE Trans. Softw. Eng., vol. 27, no. 10, pp. 929– 948, Oct. 2001. 89