J. Mending et aL./Information Systems 35(2010) 467-482 In order to measure knowledge about the application archiving system and incident agenda for the rest group. We complaints handling), we asked consider our selection strategy sufficiently randomized participants whether they had previous experience with based on the observation that neither our research complaints handling processes (yes/no). Since we did not objectives nor our hypotheses address the choice of word much domain knowledge in a student population items or the specificity of the word items used within of a more extensive scale(like the one described in these labels. Hence there was no motivation for us to dent was not considered. In order to measure respon- prefer any particular label over another. familiarity with the EPC notation, we adapted a three-item scale for notation familiarity developed by 3.2. Results Recker [42. which is based on Gemino and wands three pre-test questions about the familiarity, competence, and Demographics: The questionnaire of our survey w confidence of respondents with respect to an analysis filled out by 29 students who were at that time following nethod(see Appendix A and [26]). Accordingly, the three- a post-graduate course on process modeling at Eindhoven item familiarity scale assesses familiarity with the(EPC) University of Technology in the Netherlands. Participation process modeling notation in a sense of generally felt was voluntary, and as a reward we offered the students a familiarity(Fam1), self-perceived competence with the copy of the study results. Twenty-five participants were the notation(Fam3). Appendix A lists all items used in a notation(Fam2)and self-perceived confidence in usin male, while 4 were female. While some of the participants ly had followed university courses for one year, most of them had done so for three years or more, with 3. 8 years The second part of the questionnaire shows the process of study being the mean value. Half of the population had model as depicted in Fig. 1. In order to gather data to preliminary experience with business process modeling. examine hypothesis H1, the participants were asked either professionally or through previous courses. Four to identify the top three activity labels that they consider persons had not yet worked with EPCs, but the average to be the most ambiguous. In the third part, we sought to participant had known them for three months and created ather data to examine hypotheses H2a-H2c. In order to 10 models so far. Altogether, 25 out of the 29 participants evaluate usefulness perceptions, we developed a two-item self-assessed their familiarity with EPCs as better than measurement scale that stresses the act of understanding. 3(average total factor score), with the median being 4.5. Specifically, we used the Perceived Usefulness scales We included a brief description of the epc notation similar eveloped by Maes and poels [43 as a basis for our o[45, p. 36] such that the participants would in any case measurement development. The motivation is that their have the necessary background to understand the process PU measures were developed specifically for the con model. Finally, there were six persons who had some ceptual modeling context. Our scales were worded overal preliminary knowledge of complaint handling processes. I found [label] useful for understanding the process modeled Overall, the study population contained individuals and Overall, I think [label improves my performance when with some application domain knowledge and familiarity understanding the process modeled. We asked the partici- of the EPC notation, but without high levels of either. for their perception in these terms of six activit Studies using students have been often criticized for lack labels from the process model, using a 7-point Likert scale of external validity. Despite this criticism, we agree with with the anchor points ""Disagree strongly"and"Agree Gemino and Wand [26, 46, Recker and Dreiling [11]as well as Batra et al. (47 that the selection of students over practition this ty research can in fact be of the 12 distinct labels shown in Fig. 1 but instead to advisable. Results from both domain understanding and record these measures for six labels only. We have done problem solving tasks could have been confounded by so for the pragmatic reason of not making our data participants that are able to bring to bear prior application nd the conduct of the experiment domain knowledge in one of the areas [48. Also, post unnecessarily long. Considering six labels allowed us to graduate students (like the one participating in our study) obtain 6 (labels)x 29(number of responses)= 174 data have been found to be adequate proxies for analysts with points for hypothesis testing, which we deemed sufficient low to medium expertise levels (46, 49 for our analysis. We arbitrarily selected two labels for each Perceived amb iguity: The second part of the question- of the three styles we identified in the previous section, naire focused on the relationship between label types and these being register receipt date of complaint letter and inform complainant as verb-object labels, registration and the participants to identify those three activity labels that follow up that follow the action-noun style, as well as they consider to be the most ambiguous. Since there are 12 distinct labels in the model and 29 participants, we received 348 assessments whether a particular label (belonging to a certain label type) was considered to be among the three most ambiguous ones. The labels incident to a textual description of the agenda, complaint analysis, and archiving system were business process,which essentially is a tautology. Also note that we focus mentioned most frequently(14, 13, and 12 times). Note actual usage [44 The research by Maes and Poels [431 that the first and third labels belong to the rest group. much broader in its goal to reveal the contribution of different while complaint analysis follows the action-noun style dimensions to the quality of conceptual model In contrast, the most ambiguous label following theIn order to measure knowledge about the application domain (in our case: complaints handling), we asked participants whether they had previous experience with complaints handling processes (yes/no). Since we did not expect much domain knowledge in a student population, the use of a more extensive scale (like the one described in [26]) was not considered. In order to measure respon￾dents’ familiarity with the EPC notation, we adapted a three-item scale for notation familiarity developed by Recker [42], which is based on Gemino and Wand’s three pre-test questions about the familiarity, competence, and confidence of respondents with respect to an analysis method (see Appendix A and [26]). Accordingly, the three￾item familiarity scale assesses familiarity with the (EPC) process modeling notation in a sense of generally felt familiarity (Fam1), self-perceived competence with the notation (Fam2) and self-perceived confidence in using the notation (Fam3). Appendix A lists all items used in the questionnaire. The second part of the questionnaire shows the process model as depicted in Fig. 1. In order to gather data to examine hypothesis H1, the participants were asked to identify the top three activity labels that they consider to be the most ambiguous. In the third part, we sought to gather data to examine hypotheses H2a–H2c. In order to evaluate usefulness perceptions, we developed a two-item measurement scale that stresses the act of understanding. Specifically, we used the Perceived Usefulness scales developed by Maes and Poels [43] as a basis for our measurement development. The motivation is that their PU measures were developed specifically for the con￾ceptual modeling context. Our scales were worded Overall, I found [label] useful for understanding the process modeled and Overall, I think [label] improves my performance when understanding the process modeled. 3 We asked the partici￾pants for their perception in these terms of six activity labels from the process model, using a 7-point Likert scale with the anchor points ‘‘Disagree strongly’’ and ‘‘Agree strongly’’. We chose not to measure perceived usefulness for each of the 12 distinct labels shown in Fig. 1 but instead to record these measures for six labels only. We have done so for the pragmatic reason of not making our data collection instrument—and the conduct of the experiment— unnecessarily long. Considering six labels allowed us to obtain 6 (labels) 29 (number of responses) ¼ 174 data points for hypothesis testing, which we deemed sufficient for our analysis. We arbitrarily selected two labels for each of the three styles we identified in the previous section, these being register receipt date of complaint letter and inform complainant as verb–object labels, registration and follow up that follow the action-noun style, as well as archiving system and incident agenda for the rest group. We consider our selection strategy sufficiently randomized based on the observation that neither our research objectives nor our hypotheses address the choice of word items or the specificity of the word items used within these labels. Hence, there was no motivation for us to prefer any particular label over another. 3.2. Results Demographics: The questionnaire of our survey was filled out by 29 students who were at that time following a post-graduate course on process modeling at Eindhoven University of Technology in the Netherlands. Participation was voluntary, and as a reward we offered the students a copy of the study results. Twenty-five participants were male, while 4 were female. While some of the participants only had followed university courses for one year, most of them had done so for three years or more, with 3.8 years of study being the mean value. Half of the population had preliminary experience with business process modeling, either professionally or through previous courses. Four persons had not yet worked with EPCs, but the average participant had known them for three months and created 10 models so far. Altogether, 25 out of the 29 participants self-assessed their familiarity with EPCs as better than 3 (average total factor score), with the median being 4.5. We included a brief description of the EPC notation similar to [45, p. 36] such that the participants would in any case have the necessary background to understand the process model. Finally, there were six persons who had some preliminary knowledge of complaint handling processes. Overall, the study population contained individuals with some application domain knowledge and familiarity of the EPC notation, but without high levels of either. Studies using students have been often criticized for lack of external validity. Despite this criticism, we agree with Gemino and Wand [26,46], Recker and Dreiling [11] as well as Batra et al. [47] that the selection of students over practitioners in this type of research can in fact be advisable. Results from both domain understanding and problem solving tasks could have been confounded by participants that are able to bring to bear prior application domain knowledge in one of the areas [48]. Also, post￾graduate students (like the one participating in our study) have been found to be adequate proxies for analysts with low to medium expertise levels [46,49]. Perceived ambiguity: The second part of the question￾naire focused on the relationship between label types and perceived ambiguity, as stated in hypothesis H1. We asked the participants to identify those three activity labels that they consider to be the most ambiguous. Since there are 12 distinct labels in the model and 29 participants, we received 348 assessments whether a particular label (belonging to a certain label type) was considered to be among the three most ambiguous ones. The labels incident agenda, complaint analysis, and archiving system were mentioned most frequently (14, 13, and 12 times). Note that the first and third labels belong to the rest group, while complaint analysis follows the action-noun style. In contrast, the most ambiguous label following the ARTICLE IN PRESS 3 We chose not to adapt the PU1 item from [43]. This item cannot be reasonably applied to text labels. The item would have read Overall, I think the [label] would be an improvement to a textual description of the business process, which essentially is a tautology. Also note that we focus on perceived usefulness in our experiment for its importance as a key antecedent to actual usage [44]. The research by Maes and Poels [43] is much broader in its goal to reveal the contribution of different dimensions to the quality of conceptual models. J. Mendling et al. / Information Systems 35 (2010) 467–482 473