American Journal of Pharmaceutical Education 2007;71(5)Article 85. n adult teach rved in each lecture and the revisions that o of the le ers participated in e sing common statistical testing such as the mon understanding ofthe definition of active learning and av erage measure intrac ss correlation Therefore to recognize the cer elements that would lead to su cent greement was calculated between each ohserver ful implementation ofan active-learning activity we felt for the 3 outcomes and between the observers as a whole that it was important to include faculty members with and and the instructor.Data analysis was performed using without prio extensive education in active learning to Excel and SPSS 6.11 (SAS.Cary.NC). ensure usability by our target audience of doctoral trained faculty members who may not have training in the con- RESULTS e Learning Inventory Tool was construc h r2005 fal ted to dD ord type of th spring 20 at t thre ourt mp ty of any uld a d g be no to 。3 pa d ask sed fo nt on the ality in the N=1) nd fall s 200 during the activity the rall class atm h and the in=4).and spring scmester 2006 (n =4)in the Thera perceived ease and skill of the instruc Through the pilot assessments 12 additional mod. ophysiology (n =2)courses Students were in the third. ifications were made to the active-Learning Inventory professional year (P3)of a doctor of pharmacy degre Tool.Most modifications pertained to"Code A:Question program with an approximate class size of 100.Immedi Answer, with other adjustments made for clarity ately following each lec ture.the reviewers met to com Changes to each draft of the tool are described in Table pare initial results and propose modi and included clarification of descriptions and a summary nvento including improve its The m the ver th Code A:Que 1n8 arning Inventory Tool d ir P nd the of 13( ntory Tool wa to the nex took ar m evaluation rage of 2.2 minutes (rang :0.6-16e ach to Seven of the instructors were interviewed following Three (range:2-5)different types of active leamning were their lecture using a scripted interview guide to elicit per observed per lecture ceptions of their lesson that included:their definition of Average percent apreement among faculty observers active learning.the perceived merits of active learning in was excellent for each outcome and is presented in figure the classroom,the types of active-learning activities used The percent agreement for the total number of active in the lecture, the rational for the use of the specifi ing episodes in all lectures was 88%(61%-100%) ch the estimated amoun the number of ss nme (67 act uired 0% gh not g the us agreement am e-vers ed is. All dat tory Tool ed (Table 2) from the instructor intervie coded and analyzed of active learning varied widely.but all using anal lyst triangulation with 3 inder ndent coders included elements of"doing essing "The pri The following outcomes were used to measure agree perceived merits of using active learning that were ment among the observers using the Active-Learning cited by instructors included better retention of material Inventory Tool and between the observers and each in (57%)and improved application and critical thinking structor:number of active-learning episodes tim (29%).The most frequently reported types of active learn pe r active-learning episode,and the number of different ng used were cases (100%),think/pair/share activities active-learning episodes included in each lecture.Given (66%),and the use ofa computer-based personal response the small number of episodes of active learning that were system(33%).Past exposure or familiarity with a particularthe investigation. Of the 4 observers, 2 had received prior formal training in adult teaching and learning. All observ￾ers participated in extensive discussions to develop a com￾mon understanding of the definition of active learning and to recognize the CER elements that would lead to success￾ful implementation of an active-learning activity. We felt that it was important to include faculty members with and without prior extensive education in active learning to ensure usability by our target audience of doctoral trained faculty members who may not have training in the con￾cepts of adult learning and active learning. Lectures were selected during the summer 2005, fall 2005, or spring 2006 semesters when at least three fourths of the observers could attend and when the instructor was willing to participate. Nine lectures (3 videotaped and 6 live) were used for reliability evaluation. Lectures were given in the summer (N 5 1) and fall semesters of 2005 (n 5 4), and spring semester 2006 (n 5 4) in the Thera￾peutics(n 5 6), Self-Care Therapeutics(n 5 1) and Path￾ophysiology (n 5 2) courses. Students were in the third￾professional year (P3) of a doctor of pharmacy degree program with an approximate class size of 100. Immedi￾ately following each lecture, the reviewers met to com￾pare initial results and propose modifications to the Active-Learning Inventory Tool, including changes to improve its ease of use and clarity. The most difficult item to capture on the Active-Learning Inventory Tool was ‘‘Code A: Question & Answer.’’ Differences around this item were resolved by consensus and the Active-Learning Inventory Tool was revised accordingly prior to the next classroom evaluation. Seven of the instructors were interviewed following their lecture using a scripted interview guide to elicit per￾ceptions of their lesson that included: their definition of active learning, the perceived merits of active learning in the classroom, the types of active-learning activities used in the lecture, the rationale for the use of the specific active-learning activities chosen, the estimated amount of class time that was devoted to active-learning activi￾ties, the estimated time required to prepare the lesson and active-learning activities, any perceived barriers to the use of active learning, and the impact of using active-learning techniques on the amount of content covered.16,17 All data from the instructor interviews were coded and analyzed using analyst triangulation with 3 independent coders. The following outcomes were used to measure agree￾ment among the observers using the Active-Learning Inventory Tool and between the observers and each in￾structor: number of active-learning episodes used, time per active-learning episode, and the number of different active-learning episodes included in each lecture. Given the small number of episodes of active learning that were observed in each lecture and the revisions that occurred after some of the lectures, interrater reliability could not be estimated using common statistical testing such as the average measure intraclass correlation. Therefore, per￾cent agreement was calculated between each observer for the 3 outcomes and between the observers as a whole and the instructor. Data analysis was performed using Excel and SPSS 6.11 (SAS, Cary, NC). RESULTS The Active Learning Inventory Tool was construc￾ted to allow a trained peer observer to record the type, amount, length, and complexity of any observed active￾learning teaching behaviors. Each active-learning activity is recorded as a separate ‘‘episode’’ and asks the observer to comment on the quality of the classroom environment during the activity, the overall class atmosphere, and the perceived ease and skill of the instructor. Through the 8 pilot assessments, 12 additional mod￾ifications were made to the Active-Learning Inventory Tool. Most modifications pertained to ‘‘Code A: Question & Answer,’’ with other adjustments made for clarity. Changes to each draft of the tool are described in Table 1 and included clarification of descriptions and a summary page for the reviewer’s comment. The frequency of mod￾ifications decreased over the development process. The final version of the Active-Learning Inventory Tool is presented in Appendix 1. Over these 9 lectures, an average of 13 (range: 4-34) episodes of active learning were observed that took an average of 2.2 minutes (range: 0.6-16) each to complete. Three (range: 2-5) different types of active learning were observed per lecture. Average percent agreement among faculty observers was excellent for each outcome and is presented in Figure 1. The percent agreement for the total number of active￾learning episodes in all lectures was 88% (61%-100%), the number of different types of active learning observed was 90% (67%-100%) and the time per active-learning episode was 87% (64%-100%). Although not statistically significant, agreement among the observers improved over time as experience with the Active-Learning Inven￾tory Tool increased (Table 2). Definitions of active learning varied widely, but all included elements of ‘‘doing’’ and ‘‘processing.’’ The pri￾mary perceived merits of using active learning that were cited by instructors included better retention of material (57%) and improved application and critical thinking (29%). The most frequently reported types of active learn￾ing used were cases (100%), think/pair/share activities (66%), and the use of a computer-based personal response system(33%).Pastexposureorfamiliaritywitha particular American Journal of Pharmaceutical Education 2007; 71 (5) Article 85. 3