How Far We Have Progressed in the Journey?An Examination of CPDP 1:9 Literature reading (conducted by us) (condueted by us) Foward snowballing Hosseini et al. from BMW 2002 TSE paper (60 13 (46) eross pruject"defect prediction" 46+13-59 Total number of selected supervised CPDP studies (59+3=62) 636+3-72) 'cross projeet"+"fault prediction" (62+1-63) oss company"+"fault prediction" (63+0=63) Fig.2.The process to select supervised CPDP studies. 2.4 State of Progress To understand the progress in supervised cross-project defect prediction,we conducted a search of the literature published between 2002 and 2017.The starting year of the search was set to 2002 as it is the year that the first CPDP article was published [9](abbreviated as"BMW 2002 TSE paper" in the following).Before the search,we set up the following inclusion criteria:(1)the study was a supervised CPDP study;(2)the study was written in English;(3)the full text was available;(4) only the journal version was included if the study had both the conference and journal versions; and(5)the prediction scenario was classification or ranking. As shown in Figure 2,we searched the articles from three sources:Google Scholar,the existing systematic literature reviews,and literature reading.We used Google Scholar as the main search source,as it "provides a simple way to broadly search for scholarly literature."2 First,we did a forward snowballing search [114]by recursively examining the citations of the "BMW 2002 TSE paper".Consequently,46 relevant articles were identified [1,3,5,7,9,10,12,13,15,17,26-30, 37,39,41,42,44,46,47,52,59,66,79,80,82,85-87,89,90,92-96,107,109,116,118,120,122,123 l26].Then,we used“cross project'”+“defect prediction”as the search terms to do a search.Asa result,13 additional relevant articles [14,31,32,43,45,61,71,84,88,99,101,108,129]were found with respect to the identified 46 articles.Next,we used "cross company"+"defect prediction"as the search terms,identifying 3 additional relevant articles [104,106,110]with respect to the 59 (=46 +13)articles.After that,we used "cross project"+"fault prediction"as the search terms, identifying 1 additional relevant article [100]with respect to the 62(=59+3)articles.Finally, we used the terms"cross company"+"fault prediction"and did not find any additional relevant article.By the above steps,we totally identified 63(=46+13+3+1+0)supervised CPDP articles from Google Scholar.In addition to Google Scholar,we identified 6 additional relevant articles from a systematic literature review by Hosseini et al.[40].Hosseini et al.identified 46 primary CPDP studies from three electronic databases(the ACM digital library,the IEEE Explore,and the ISI Web of Science)and two search engines (Google Scholar and Scopus).After applying our inclusion criteria to the 46 CPDP studies,we found that 6 relevant articles [60,73,74,103,105, 119]were not present in the 63 identified CPDP articles from Google Scholar.The last source was literature reading,through which we found 3 additional relevant articles [48,72,121]with respect IThe literature search was conducted on April 21,2017. 2https://scholar.google.com/intl/en/scholar/about.html. ACM Transactions on Software Engineering and Methodology,Vol.27.No.1,Article 1.Pub.date:April 2018.How Far We Have Progressed in the Journey? An Examination of CPDP 1:9 Fig. 2. The process to select supervised CPDP studies. 2.4 State of Progress To understand the progress in supervised cross-project defect prediction, we conducted a search of the literature published between 2002 and 2017.1 The starting year of the search was set to 2002 as it is the year that the first CPDP article was published [9] (abbreviated as “BMW 2002 TSE paper” in the following). Before the search, we set up the following inclusion criteria: (1) the study was a supervised CPDP study; (2) the study was written in English; (3) the full text was available; (4) only the journal version was included if the study had both the conference and journal versions; and (5) the prediction scenario was classification or ranking. As shown in Figure 2, we searched the articles from three sources: Google Scholar, the existing systematic literature reviews, and literature reading. We used Google Scholar as the main search source, as it “provides a simple way to broadly search for scholarly literature.”2 First, we did a forward snowballing search [114] by recursively examining the citations of the “BMW 2002 TSE paper”. Consequently, 46 relevant articles were identified [1, 3, 5, 7, 9, 10, 12, 13, 15, 17, 26–30, 37, 39, 41, 42, 44, 46, 47, 52, 59, 66, 79, 80, 82, 85–87, 89, 90, 92–96, 107, 109, 116, 118, 120, 122, 123, 126]. Then, we used “cross project” + “defect prediction” as the search terms to do a search. As a result, 13 additional relevant articles [14, 31, 32, 43, 45, 61, 71, 84, 88, 99, 101, 108, 129] were found with respect to the identified 46 articles. Next, we used “cross company” + “defect prediction” as the search terms, identifying 3 additional relevant articles [104, 106, 110] with respect to the 59 (=46 + 13) articles. After that, we used “cross project” + “fault prediction” as the search terms, identifying 1 additional relevant article [100] with respect to the 62 (=59 + 3) articles. Finally, we used the terms “cross company” + “fault prediction” and did not find any additional relevant article. By the above steps, we totally identified 63 (=46 + 13 + 3 + 1 + 0) supervised CPDP articles from Google Scholar. In addition to Google Scholar, we identified 6 additional relevant articles from a systematic literature review by Hosseini et al. [40]. Hosseini et al. identified 46 primary CPDP studies from three electronic databases (the ACM digital library, the IEEE Explore, and the ISI Web of Science) and two search engines (Google Scholar and Scopus). After applying our inclusion criteria to the 46 CPDP studies, we found that 6 relevant articles [60, 73, 74, 103, 105, 119] were not present in the 63 identified CPDP articles from Google Scholar. The last source was literature reading, through which we found 3 additional relevant articles [48, 72, 121] with respect 1The literature search was conducted on April 21, 2017. 2https://scholar.google.com/intl/en/scholar/about.html. ACM Transactions on Software Engineering and Methodology, Vol. 27, No. 1, Article 1. Pub. date: April 2018