American Journal of Pharmaceutical Education 2007;71 (5)Article 85. RESEARCH ARTICLES A Tool for Measuring Active Learning in the Classroom Jenny A.Van Amburgh,PharmD,John W.Devlin,PharmD,Jennifer L.Kirwin,PharmD, and Donna M.Qualters,PhD ter University Submitted February 2,2007;accepted May 13,2007;published October 15,2007 Objectives.To develop a valid and reliable active-learnin entory tool for use in large clas srooms y perceptions of ac ory oy na a y by tr re the ed to en per Result 3(range ng en 06.1 ode.Both range:2- Q70/ 68% The valid and reliabl c tool ton d to Tool ach ng and its impact in other dis Keywords: active peer observation,perceptions,faculty developmen The concept ofstudent enga agement is becoming more tiple pathwa ays to engage in leamning that must begin with than iust educa tional rhetoric.Active-learning techniques ng active n行 ged in the clas om have emerged as strategies for instructors to promote en The advantages of active learning are numerous.Inte- grating active-learning strategies into the classroom ent bengPeom2natoaofmaienalhiesal results in a strong model ofteaching because active lear ing promotes the Active-learning techniques engage While students more deeply in the process of learning course instructors'engagement with students may occasionally occur outside the classroom (eg.experiential learning activities.casual conversation).it most often occurs in earning techniques not only benefits students by allowing the classroom.Student engagement in the classroom them the opportunity to practice skills and ask questions involves the student participating in a Didaktik triangle but also benefits instructors by affording them the oppor interaction between th e instructor,fellow students.and tunity to assess the students'understanding and remediate the discipline material.The most challenging aspect of important points ona nearly"real time"basis.Classroom this relationship for instructors to establish is the student environments that include active learning engage stud ents interaction and engagement with material beyond the in their learning,encourage"doing"with understanding pasic level of knowledge and comprehension.We hav provide the students with opportunities to revise anc begun to think of this learning process as"the continuum improve their own thinking(formative ass sment),and of engagement,"where students are presented with mul mation from the om to lls ou de wor nding Auther le A Van Amb gh,BPharm on University,Bouve College of Health Sciences-School of and inte of he alth cil for (ACPE h 206 Mugar Life Science Building,Boston an MA021l5.Tel:617 373-744 .Fax:617-373-7655 (AA both enc -mail:J.vanamburgh(@neu.c directly involves stu
RESEARCH ARTICLES A Tool for Measuring Active Learning in the Classroom Jenny A. Van Amburgh, PharmD,a John W. Devlin, PharmD,a Jennifer L. Kirwin, PharmD,a and Donna M. Qualters, PhDb a Bouve´ College of Health Sciences, Northeastern University b Center for Teaching Excellence, Suffolk University Submitted February 2, 2007; accepted May 13, 2007; published October 15, 2007. Objectives. To develop a valid and reliable active-learning inventory tool for use in large classrooms and compare faculty perceptions of active-learning using the Active-Learning Inventory Tool. Methods. The Active-Learning Inventory Tool was developed using published literature and validated by national experts in educational research. Reliability was established by trained faculty members who used the Active-Learning Inventory Tool to observe 9 pharmacy lectures. Instructors were then interviewed to elicit perceptions regarding active learning and asked to share their perceptions. Results. Per lecture, 13 (range: 4-34) episodes of active learning encompassing 3 (range: 2-5) different types of active learning occurred over 2.2 minutes (0.6-16) per episode. Both interobserver ($87%) and observer-instructor agreement ($68%) were high for these outcomes. Conclusions. The Active-Learning Inventory Tool is a valid and reliable tool to measure active learning in the classroom. Future studies are needed to determine the impact of the Active-Learning Inventory Tool on teaching and its usefulness in other disciplines. Keywords: active learning, assessment, peer observation, perceptions, faculty development The concept of student engagement is becoming more than just educational rhetoric. Active-learning techniques have emerged as strategies for instructors to promote engagement with both discipline material and learning.1 Reports by the National Survey of Student Engagement (NSSE) demonstrate that a high level of student engagement increases learning and retention of material particularly in underrepresented minority students.2,3 While instructors’ engagement with students may occasionally occur outside the classroom (eg, experiential learning activities, casual conversation), it most often occurs in the classroom. Student engagement in the classroom involves the student participating in a Didaktik triangle interaction between the instructor, fellow students, and the discipline material.4 The most challenging aspect of this relationship for instructors to establish is the student interaction and engagement with material beyond the basic level of knowledge and comprehension. We have begun to think of this learning process as ‘‘the continuum of engagement,’’ where students are presented with multiple pathways to engage in learning that must begin with being actively engaged in the classroom. The advantages of active learning are numerous. Integrating active-learning strategies into the classroom results in a strong model of teaching because active learning promotes the application of material while it is still being presented.5,6 Active-learning techniques engage students more deeply in the process of learning course material by encouraging critical thinking and fostering the development of self-directed learning.7,8 Use of activelearning techniques not only benefits students by allowing them the opportunity to practice skills and ask questions, but also benefits instructors by affording them the opportunity to assess the students’ understanding and remediate important points on a nearly ‘‘real time’’ basis. Classroom environments that include active learning engage students in their learning, encourage ‘‘doing’’ with understanding, provide the students with opportunities to revise and improve their own thinking (formative assessment), and help the students to connect the information from the classroom to practice in the outside world.9 These skills are critical to the personal, professional, and intellectual development of health care providers. The Accreditation Council for Pharmacy Education (ACPE) and the American Association of Colleges of Pharmacy (AACP) both encourage a doctor of pharmacy (PharmD) curriculum that directly involves students, facilitates the Corresponding Author: Jenny A. Van Amburgh, BPharm, PharmD. Associate Clinical Professor, Northeastern University, Bouve´ College of Health Sciences - School of Pharmacy, 206 Mugar Life Science Building, Boston, MA 02115. Tel: 617-373-7444. Fax:617-373-7655. E-mail: j.vanamburgh@neu.edu American Journal of Pharmaceutical Education 2007; 71 (5) Article 85. 1
American Journal of Pharmaceutical Education 2007;71 (5)Article 85 transition from"dependent to independent learner.and related to active learning.as well as classroom ass ment materials which emphasize active-learning techniques as a means to formatively monitor ongoing learning through Active learning can a course.All materials were reviewed in order to find create an environment that facilitates"learning in real types of active-learning techniques and methods that have time either in collaboration with others or independently. been descrbed to quantify and categorize these active and this is central to the philosophy of practice-oriente learning techniques.As our intentions for the use of this amning can b tool went beyond mere step he tin anuv iterat ng practice nange the tive le liz tive ntre h we use tad into the of the iall literature failed to identify an entory tool to quan facult risk,which was estimated based on the inter tify and characterize the use ofactive.le ing techn and ease of use of the ac ivity in the class om.Hvpoth by faculty members.Development of a standardized in esizing that a simpler tool would have greater ac ance ventory tool that is both valid and reliable provides th among faculty members we consciously y grouped similar ability to document the type and quantity ofactive leaming tems together and kept the tool as short as possible.The occurring in classes and establishes a basis for qualitativ proposed Active-Leamning Inventory Tool that was sent evaluation of active- ning techniques addition,an out to reviewers 2 inventory tool will help det nine the type of active learn widely accepte active-leam ng techniques and ranked ng th hem by degre oy th ser dmi a中 unt sign,impler rators as and ctivity nt tablish the of th In earch in thi ulte vide do tation for th ublished and hed e ely in the field of ing emphasis on meas uring nes in pharn education.The expe ts were asked to review the active education but to our knowledge there are no tools avail learning Inventory Tool and then comment on the use of able to obiectively document the use ofeffective teaching terminology and descrintions appropriateness of the spe techniques (such as active learning)to attain these out cific activities included,overall validity ofthe assessment comes.We therefore sought to develop a valid and reliable ease of use.and generalizability to other academic discr active-l ring inventory tool to quantify the use of activ plines nd prc general comments.The tool was sub e course .In additio we compared in ons of th ount and type ofactive leam and an ing act ers g th consid. While the ng tech ctive l ing.for the p of th in no as ents doing things and thinking risk The they are doing." In addition we considered active leam ing as successfully occurring in the classroom when 3 key and cons sensus of the authors.i5 The section of tool that components were observed:(1)the contert of the activity asked for qualitative comments about the use of active was explained,(2)the students were engaged in the activ learning was also clarified ty,and (3)there was closure to the activity via reflection After approval by the ortheastern University Insti We think of these elements as the CER(context.engage tutional Reviev Board the revised Active-Learning ment,reflection)components of successful active learning Inventory Iool stages METHODS and in 6 liv scn f the Active-Learning ses xpe prepared using articles,textbo an 3 pharmacy faculty membe ers)participated in this pha
transition from ‘‘dependent to independent learner,’’ and develops in graduates ‘‘the ability to integrate and apply learning to both the present practice of pharmacy and the advancement of the profession.’’10,11 Active learning can create an environment that facilitates ‘‘learning in real time’’ either in collaboration with others or independently, and this is central to the philosophy of practice-oriented education at our institution.12Lastly, active learning can be viewed as the first step along an experiential learning continuum that promotes more substantive learning outcomes. While active learning has been validated as an effective way to increase student learning, and is increasingly being incorporated into the classroom, a search of the literature failed to identify an ‘‘inventory tool’’ to quantify and characterize the use of active-learning techniques by faculty members. Development of a standardized inventory tool that is both valid and reliable provides the ability to document the type and quantity of active learning occurring in classes and establishes a basis for qualitative evaluation of active-learning techniques. In addition, an inventory tool will help determine the type of active learning that is best suited to teach or assess a particular level of knowledge, serve faculty members and program administrators as they seek to evaluate teaching skills and provide comments for improvement, and be valuable for those conducting research in this area. An additional advantage of having an instrument is to provide documentation for the increasing emphasis on measuring outcomes in pharmacy education, but to our knowledge there are no tools available to objectively document the use of effective teaching techniques (such as active learning) to attain these outcomes. We therefore sought to develop a valid and reliable active-learning inventory tool to quantify the use of active learning in large courses. In addition, we compared instructor perceptions of the amount and type of active learning used in the classroom to the amount and type quantified using the Active-Learning Inventory Tool as we considered potential faculty development uses for this tool. While there is no standard definition for active learning, for the purpose of this paper, we defined active learning as ‘‘...students doing things and thinking about what they are doing.’’6 In addition, we considered active learning as successfully occurring in the classroom when 3 key components were observed: (1) the context of the activity was explained, (2) the students were engaged in the activity, and (3) there was closure to the activity via reflection. We think of these elements as the CER (context, engagement, reflection) components of successful active learning. METHODS A draft of the Active-Learning Inventory Tool was prepared using articles, textbooks, and online references related to active learning, as well as classroom assessment materials which emphasize active-learning techniques as a means to formatively monitor ongoing learning through a course. All materials were reviewed in order to find types of active-learning techniques and methods that have been described to quantify and categorize these activelearning techniques. As our intentions for the use of this tool went beyond mere quantification of activities, we reviewed the literature on how to create change in faculty teaching practices.13,14 Change theory provided the context for conceptualizing the approach we used to develop, introduce, and debrief this tool to instructors. Each activelearning technique was initially categorized according to faculty risk, which was estimated based on the intensity and ease of use of the activity in the classroom. Hypothesizing that a simpler tool would have greater acceptance among faculty members, we consciously grouped similar items together and kept the tool as short as possible. The proposed Active-Learning Inventory Tool that was sent out to reviewers identified, and briefly explained 22 widely accepted active-learning techniques and ranked them by degree of faculty risk as determined by the amount of time and effort it takes to design, implement, and extract learning data from the activity. To establish the validity of the Active-Learning Inventory Tool, we consulted with expert reviewers who had published and researched extensively in the field of education. The experts were asked to review the ActiveLearning Inventory Tool and then comment on the use of terminology and descriptions, appropriateness of the specific activities included, overall validity of the assessment, ease of use, and generalizability to other academic disciplines, and provide general comments. The tool was subsequently modified based on the results of their written and verbal feedback, including reorganizing the rank order of activities based on complexity versus faculty risk. Based on expert feedback, coding schemes and descriptions of active-learning techniques were clarified and the rank order of techniques was changed to reflect complexity of the active-learning activity as opposed to faculty risk. The complexity of an activity was assigned based on a combination of findings from the literature review and consensus of the authors.15 The section of tool that asked for qualitative comments about the use of active learning was also clarified. After approval by the Northeastern University Institutional Review Board, the revised Active-Learning Inventory Tool was tested for reliability in 2 stages: assessment using 3 videotaped 1-hour pharmacy lectures and then in 6 live lectures in large, school of pharmacy courses. Four trained observers (1 educational expert and 3 pharmacy faculty members) participated in this phase of American Journal of Pharmaceutical Education 2007; 71 (5) Article 85. 2
American Journal of Pharmaceutical Education 2007;71(5)Article 85. n adult teach rved in each lecture and the revisions that o of the le ers participated in e sing common statistical testing such as the mon understanding ofthe definition of active learning and av erage measure intrac ss correlation Therefore to recognize the cer elements that would lead to su cent greement was calculated between each ohserver ful implementation ofan active-learning activity we felt for the 3 outcomes and between the observers as a whole that it was important to include faculty members with and and the instructor.Data analysis was performed using without prio extensive education in active learning to Excel and SPSS 6.11 (SAS.Cary.NC). ensure usability by our target audience of doctoral trained faculty members who may not have training in the con- RESULTS e Learning Inventory Tool was construc h r2005 fal ted to dD ord type of th spring 20 at t thre ourt mp ty of any uld a d g be no to 。3 pa d ask sed fo nt on the ality in the N=1) nd fall s 200 during the activity the rall class atm h and the in=4).and spring scmester 2006 (n =4)in the Thera perceived ease and skill of the instruc Through the pilot assessments 12 additional mod. ophysiology (n =2)courses Students were in the third. ifications were made to the active-Learning Inventory professional year (P3)of a doctor of pharmacy degre Tool.Most modifications pertained to"Code A:Question program with an approximate class size of 100.Immedi Answer, with other adjustments made for clarity ately following each lec ture.the reviewers met to com Changes to each draft of the tool are described in Table pare initial results and propose modi and included clarification of descriptions and a summary nvento including improve its The m the ver th Code A:Que 1n8 arning Inventory Tool d ir P nd the of 13( ntory Tool wa to the nex took ar m evaluation rage of 2.2 minutes (rang :0.6-16e ach to Seven of the instructors were interviewed following Three (range:2-5)different types of active leamning were their lecture using a scripted interview guide to elicit per observed per lecture ceptions of their lesson that included:their definition of Average percent apreement among faculty observers active learning.the perceived merits of active learning in was excellent for each outcome and is presented in figure the classroom,the types of active-learning activities used The percent agreement for the total number of active in the lecture, the rational for the use of the specifi ing episodes in all lectures was 88%(61%-100%) ch the estimated amoun the number of ss nme (67 act uired 0% gh not g the us agreement am e-vers ed is. All dat tory Tool ed (Table 2) from the instructor intervie coded and analyzed of active learning varied widely.but all using anal lyst triangulation with 3 inder ndent coders included elements of"doing essing "The pri The following outcomes were used to measure agree perceived merits of using active learning that were ment among the observers using the Active-Learning cited by instructors included better retention of material Inventory Tool and between the observers and each in (57%)and improved application and critical thinking structor:number of active-learning episodes tim (29%).The most frequently reported types of active learn pe r active-learning episode,and the number of different ng used were cases (100%),think/pair/share activities active-learning episodes included in each lecture.Given (66%),and the use ofa computer-based personal response the small number of episodes of active learning that were system(33%).Past exposure or familiarity with a particular
the investigation. Of the 4 observers, 2 had received prior formal training in adult teaching and learning. All observers participated in extensive discussions to develop a common understanding of the definition of active learning and to recognize the CER elements that would lead to successful implementation of an active-learning activity. We felt that it was important to include faculty members with and without prior extensive education in active learning to ensure usability by our target audience of doctoral trained faculty members who may not have training in the concepts of adult learning and active learning. Lectures were selected during the summer 2005, fall 2005, or spring 2006 semesters when at least three fourths of the observers could attend and when the instructor was willing to participate. Nine lectures (3 videotaped and 6 live) were used for reliability evaluation. Lectures were given in the summer (N 5 1) and fall semesters of 2005 (n 5 4), and spring semester 2006 (n 5 4) in the Therapeutics(n 5 6), Self-Care Therapeutics(n 5 1) and Pathophysiology (n 5 2) courses. Students were in the thirdprofessional year (P3) of a doctor of pharmacy degree program with an approximate class size of 100. Immediately following each lecture, the reviewers met to compare initial results and propose modifications to the Active-Learning Inventory Tool, including changes to improve its ease of use and clarity. The most difficult item to capture on the Active-Learning Inventory Tool was ‘‘Code A: Question & Answer.’’ Differences around this item were resolved by consensus and the Active-Learning Inventory Tool was revised accordingly prior to the next classroom evaluation. Seven of the instructors were interviewed following their lecture using a scripted interview guide to elicit perceptions of their lesson that included: their definition of active learning, the perceived merits of active learning in the classroom, the types of active-learning activities used in the lecture, the rationale for the use of the specific active-learning activities chosen, the estimated amount of class time that was devoted to active-learning activities, the estimated time required to prepare the lesson and active-learning activities, any perceived barriers to the use of active learning, and the impact of using active-learning techniques on the amount of content covered.16,17 All data from the instructor interviews were coded and analyzed using analyst triangulation with 3 independent coders. The following outcomes were used to measure agreement among the observers using the Active-Learning Inventory Tool and between the observers and each instructor: number of active-learning episodes used, time per active-learning episode, and the number of different active-learning episodes included in each lecture. Given the small number of episodes of active learning that were observed in each lecture and the revisions that occurred after some of the lectures, interrater reliability could not be estimated using common statistical testing such as the average measure intraclass correlation. Therefore, percent agreement was calculated between each observer for the 3 outcomes and between the observers as a whole and the instructor. Data analysis was performed using Excel and SPSS 6.11 (SAS, Cary, NC). RESULTS The Active Learning Inventory Tool was constructed to allow a trained peer observer to record the type, amount, length, and complexity of any observed activelearning teaching behaviors. Each active-learning activity is recorded as a separate ‘‘episode’’ and asks the observer to comment on the quality of the classroom environment during the activity, the overall class atmosphere, and the perceived ease and skill of the instructor. Through the 8 pilot assessments, 12 additional modifications were made to the Active-Learning Inventory Tool. Most modifications pertained to ‘‘Code A: Question & Answer,’’ with other adjustments made for clarity. Changes to each draft of the tool are described in Table 1 and included clarification of descriptions and a summary page for the reviewer’s comment. The frequency of modifications decreased over the development process. The final version of the Active-Learning Inventory Tool is presented in Appendix 1. Over these 9 lectures, an average of 13 (range: 4-34) episodes of active learning were observed that took an average of 2.2 minutes (range: 0.6-16) each to complete. Three (range: 2-5) different types of active learning were observed per lecture. Average percent agreement among faculty observers was excellent for each outcome and is presented in Figure 1. The percent agreement for the total number of activelearning episodes in all lectures was 88% (61%-100%), the number of different types of active learning observed was 90% (67%-100%) and the time per active-learning episode was 87% (64%-100%). Although not statistically significant, agreement among the observers improved over time as experience with the Active-Learning Inventory Tool increased (Table 2). Definitions of active learning varied widely, but all included elements of ‘‘doing’’ and ‘‘processing.’’ The primary perceived merits of using active learning that were cited by instructors included better retention of material (57%) and improved application and critical thinking (29%). The most frequently reported types of active learning used were cases (100%), think/pair/share activities (66%), and the use of a computer-based personal response system(33%).Pastexposureorfamiliaritywitha particular American Journal of Pharmaceutical Education 2007; 71 (5) Article 85. 3
American Journal of Pharmaceutical Education 2007;71 (5)Article 85 Table 1.Description of Significant Revisions to Active Learning Inventory Tool Area Modified Modification(s)Made Expert Reviewers'Comments During Validation .Changed 'Risk Level'to'Complexity Level" .Need to revise and clarify active-leaming activity list .RemovedDirected Reading ·Added'Free Write .'Cold Calling'was renamed to'Question&Answer'by either Problems Ientified/Incurred During Reliability Testin .Need to revise and clarify active leaming activity lis ·Clarified descriptor .Would like to capture mographic information and demographic and cassroom information to .Need to clarifyQuestions&Answer'activity .Removed voluntary answer vs.cold calling and included Al denotes students responded to the question;A2 denotes 1t0-A1 question/A2 denotes students were asked to respond AND given time but did not respond-will track Al and A2 for mbers but n time as conducted in I ss than I minute provided time(min)to process then respond. active-learning techniqu e was cited as the most commor (14%).Most instructors believed that any increase in the reason for its use.The reported amount of time time devoted to active-learning came at the expense of completing active-learning activities varied widely lecture content(86%). among instructors (range 10%-50%of total classroom Results showed general congruence between instruc- time).Most instructors(71%)reported that the use of tor perception and the Active-Learning Inventory Tool active learning required more preparation time.particu- observations with respect to the time spent completing larly if the technique was new to them,if they were un- active-learning activities.With the exception of instruc tors underreporting their use of student questioning,in- e use of active learning inc out the types remove milar to tha siz sing th Active-L number of active range.58-100%):number of different ypes of active learning used (78%:ran ge time 8654 ■Number of active DISCUSSION The Active-Learning Inventory Tool is the first tool that utilizes qualitative and quantitative information to tive learning be ev alid ability. Lecture Number engagem ith Figure 1.Percent agreement among ob sfor each lecture orme reflectio
active-learning technique was cited as the most common reason for its use. The reported amount of time spent completing active-learning activities varied widely among instructors (range 10%-50% of total classroom time). Most instructors (71%) reported that the use of active learning required more preparation time, particularly if the technique was new to them, if they were unfamiliar with active learning overall, or if this was a new lecture. Barriers to the use of active learning included lack of time (86%), need to remove lecture content (43%), lack of technology in the classroom (14%), and large class size (14%). Most instructors believed that any increase in the time devoted to active-learning came at the expense of lecture content (86%). Results showed general congruence between instructor perception and the Active-Learning Inventory Tool observations with respect to the time spent completing active-learning activities. With the exception of instructors underreporting their use of student questioning, instructor perceptions about the types and quantity of active learning used in lectures was similar to that observed using the Active-Learning Inventory Tool. Agreement among observers and instructors was good for all outcomes: number of active-learning episodes used (82%; range, 58%-100%); number of different types of active learning used (78%; range, 62%-100%), and average time per active-learning episode (68%; range, 50%-82%). DISCUSSION The Active-Learning Inventory Tool is the first tool that utilizes qualitative and quantitative information to capture the amount and type of active learning in the classroom that has been evaluated for validity and reliability. Through an extensive review of the literature on active engagement, consultation with experts in the field, and continual and thoughtful informed reflection, Table 1. Description of Significant Revisions to Active Learning Inventory Tool Area Modified Modification(s) Made Expert Reviewers’ Comments During Validation d Categorizing active learning based on ‘Risk Level’ is not the appropriate descriptor d Changed ‘Risk Level’ to ‘Complexity Level’ d Need to revise and clarify active-learning activity list d Removed ‘Directed Reading’ d Added ‘Free Write’ d ‘Cold Calling’ was renamed to ‘Question & Answer’ by either u voluntary or u cold calling d Clarified facilitator for Small Group Presentations Problems Identified/Incurred During Reliability Testing d Need to revise and clarify active learning activity list d Clarified descriptors d Removed ‘Pause Procedure’ d Would like to capture demographic information and classroom environment d Added instructor demographic and classroom information to The active-learning inventory tool d Need to clarify ‘Questions & Answer’ activity d Removed voluntary answer vs. cold calling and included A1 denotes students responded to the question; A2 denotes students were asked to respond and given time, but did not respond. This does not include rhetorical questions. d Further changed to: A1 denotes students responded to question/A2 denotes students were asked to respond AND given time but did not respond – will track A1 and A2 for numbers but not time as conducted in less than 1 minute. A3 denote a higher-order question, where students are provided time (.1 min) to process then respond. Figure 1. Percent agreement among observers for each lecture. American Journal of Pharmaceutical Education 2007; 71 (5) Article 85. 4
American Journal of Pharmaceutical Education 2007;71 (5)Article 85. Table 2.Agreement Between Observers and Instructors,Median (Range) Bercent Agreement and Instructors Namber of ctive-learin per 134.34 102 de 2.2(0.616 140.3-13) Number of different types of active-learning used per lecture 32.51 21.5 78(62-100) we have created a tool that can clearly measure the room.We e th ng Inventory Tool There are broad this inventory too i l al st eflection)to say active learr from these and other professions that prom ote the routine classroom.In addition,the tool was modified to provide inclusion ofactive-learning st tegies in professional cur qualitative feedback regarding (1 faculty approach in the riculum,a tool to evaluate active learning will be useful to activity,(2)quality of classroom environment during the instructors in various learning venues. activity.and (3)overall atmosphere.In order for users to In our pilot testing we found some interesting prelim- obtain maximum benefit from the Active-Learning In- inary themes.Although instructors perceived that they de. ventory Tool,observers should participate in a multi veloped full active-learning experiences, the Active pronged training session consisting of 3 components Learning Inventory Tool documented only engagemen on he vanous a earning and e absent. For examp h mis ens a cnts to to quc and will he achorheseac hon it o Th in nt includes training on how to n the he g in the tion involving discussion and p om content to condu tactivities They belie dtha and how to observe it.Observers should understand that active leaming increased retention and involvement.but successful active learning is more than just having an felt it was too time intensive to use when presenting com- activity during class;successful active learning involves plex materials.This is a key barrier that needs to be ex- CER.The last and perhaps most important segment of the plored further if the use of active leaming is to increase training is the feedback that observers are able to provide We were faced with many challenges as we worked to to colleagues to motivate them to reflect more deeply on design and modify this Tool.Since nomenclature for ac- Active-Leaming To several change aming Inventor serve ying the des of ce h active mg a but with oland ner and student earning in classe It will be nt to ring pharmacy students for practice in the mod. exnand the evaluation of this tool to other health.scie ern healthcare stem requires that we rethink pharmacy and humanitie s disciplines in order to fully comment on teaching methodology and go beyond the traditional lec. the generalizability of the Tool.The Active-Learning In- ture-based delivery of factual material to incorporate ventory tool primarily focuses on the quantitative eval- those methods that allow for effective application and uation of active learning rather than qualitative evaluation problem solving in the classroom.With the changing cli since quantitative evaluation is far easier for observers mate of accountability in higher education,it is no longe who are not experts in active learning to complete.We sufficient to say we are "good teachers";we need to have chose not to have 4 formally trained observers,but rather mechanisms that promote real change and growth in fac a mix of 2 formally trained in adult teaching and learning
we have created a tool that can clearly measure the amount and type of active learning occurring in a professional curriculum. This tool also helps to identify the gap between faculty perceptions of classroom activities and actual usage of active learning based on the 3 key components that need to be observed (context, engagement, and reflection) to say active learning was occurring in the classroom. In addition, the tool was modified to provide qualitative feedback regarding (1) faculty approach in the activity, (2) quality of classroom environment during the activity, and (3) overall atmosphere. In order for users to obtain maximum benefit from the Active-Learning Inventory Tool, observers should participate in a multipronged training session consisting of 3 components. First, an in-depth discussion of the various active-learning activities described in the tool. This ensures all observers have a similar understanding of what each of these activities should involve and will help them recognize the activity when it occurs in the classroom. The second component includes training on how to perform the observation involving discussion and practice of what to observe and how to observe it. Observers should understand that successful active learning is more than just having an activity during class; successful active learning involves CER. The last and perhaps most important segment of the training is the feedback that observers are able to provide to colleagues to motivate them to reflect more deeply on their use of active learning. In our development of the Active-Learning Inventory Tool, we did not observe all the active-learning activities listed on the tool, but with sufficient training and practice as described above, we feel that the tool can capture episodes of successful active learning in classes. Preparing pharmacy students for practice in the modern healthcare system requires that we rethink pharmacy teaching methodology and go beyond the traditional lecture-based delivery of factual material to incorporate those methods that allow for effective application and problem solving in the classroom. With the changing climate of accountability in higher education, it is no longer sufficient to say we are ‘‘good teachers’’; we need to have mechanisms that promote real change and growth in faculty teaching skills and that capture the reality of the classroom. We believe the Active-Learning Inventory Tool is a strong tool to serve both these functions. There are broad potential applications for this inventory tool in other professional disciplines as diverse as accounting and nursing education.18-20 Based on current educational standards from these and other professions that promote the routine inclusion of active-learning strategies in professional curriculum, a tool to evaluate active learning will be useful to instructors in various learning venues. In our pilot testing we found some interesting preliminary themes. Although instructors perceived that they developed full active-learning experiences, the Active Learning Inventory Tool documented only engagement – context and reflection were absent. For example, several instructors asked students to respond to questions, but the questions were perceived as rhetorical or could not be answered by students in the time allowed. Also, most of the instructors interviewed said they highly valued active learning in the classroom, but were reluctant to take class time away from content to conduct activities. They believed that active learning increased retention and involvement, but felt it was too time intensive to use when presenting complex materials. This is a key barrier that needs to be explored further if the use of active learning is to increase. We were faced with many challenges as we worked to design and modify this Tool. Since nomenclature for active-learning activities varies, we made several changes that focused on clarifying the description of certain active-learning activities. Also, in our experience, pharmacotherapeutics courses lend themselves easily to the use of cases and student questioning, but other disciplines may rely on different techniques. It will be important to expand the evaluation of this tool to other health, science, and humanities disciplines in order to fully comment on the generalizability of the Tool. The Active-Learning Inventory Tool primarily focuses on the quantitative evaluation of active learning rather than qualitative evaluation since quantitative evaluation is far easier for observers who are not experts in active learning to complete. We chose not to have 4 formally trained observers, but rather a mix of 2 formally trained in adult teaching and learning Table 2. Agreement Between Observers and Instructors, Median (Range) Observer Recorded Instructor Perception Percent Agreement Between Observers and Instructors Number of active-learning episodes per lecture 13 (4-34) 10 (2-40) 82 (58-100) Time per active-learning episode (minutes) 2.2 (0.6-16) 1.4 (0.3-13) 68 (50-82) Number of different types of active-learning used per lecture 3 (2-5) 2 (1-5) 78 (62-100) American Journal of Pharmaceutical Education 2007; 71 (5) Article 85. 5
American Journal of Pharmaceutical Education 2007;71 (5)Article 85 and 2 practice-based faculty members.We believed doing RefeRences this added to the external validity of the Tool's usefulness Qualters DM.Do students want to be active?Scp by any trained observer.Ideally,any trained observer could use this tool to increase awareness ofteaching tech- 2.K hG.A2001 niques or perhaps stimulate change in the amount of active learning used in a classroom.Our reliability was ted minority students benefit most from generally good,however,the low number of episodes of active learning and th revisions made to the Active caching as reflective practice.Crric Sd 2002:3-57 al measure ol ag enenthadt. JA.Active Lear ng:Creating Excitement in tested this tool for use in a nd its There are a number of areas in which the Active s critical thinking. I Technol Edue 1995-7-22-30 Learning Inventory Tool requires further evaluation Its role as a basis for providing feedback to instructors should be investigated.One potential valuable contribu- 9.Bransford JD,Brown AL,Cocking RR.How V P. n:Brain tion would be as part of a peer-evaluation process to raise instructor awareness of their usage of active learning and ture rch or faculty devel arriers to t aming inall dis 93.5737-8 the sults in mprov student eain large classes CONCLUSION eat:http://www.poe.neu.cdu/ A valid and reliable tool to measure the type and s IC In search of how amount of active learning in large pharmacy classroom was developed.Use and testing of this tool will be people change.4m Psychol.1992:47:1102-14 expanded to other university disciplines and other colleges and schools of pharmacy.Most importantly,we hope to (PofesionalandOrganato2a20i6 lopment Network,http:// increase dialog about the use of effective techniques to improve student learning in institutions across the country 15.Amg TA,Cross KP.Clas :A ACKNOWLEDGEMENTS 6.Patton M. to Use Oualitative Me thods in Evaluation We would like to thank ou pert review rs Charle C.Bonwell.PhD.President. well Associates s and Bogdon R.Iklen s Pmofessor Emeritus ofHistory Eric H Hohson PhD Asso ciate Dean for Academic Affairs and Assessment at South University;and Linda B.Nilson,PhD,Director,Office of 8 Accountine Education Change Commission.Obiectives of ducation fo n statement number one.Issues Teaching Effectiveness and Innovation at Clemson Univer sity,for their valuable feedback.We would also like to nting Education:Chartin thank Dr.Eric Hobson for the use of his active-learning let us 2000 lity an ns,the Nor Uni support thi and Dr. for critically reviewing the manuscnpt
and 2 practice-based faculty members. We believed doing this added to the external validity of the Tool’s usefulness by any trained observer. Ideally, any trained observer could use this tool to increase awareness of teaching techniques or perhaps stimulate change in the amount of active learning used in a classroom. Our reliability was generally good, however, the low number of episodes of active learning and the revisions made to the ActiveLearning Inventory Tool after each lecture did not allow us to calculate a formal measure of agreement. In addition, observers and instructors were not chosen randomly, nor blinded to the assessment. Lastly, we designed and tested this tool for use in a large class, and its appropriateness for smaller settings still needs to be evaluated. There are a number of areas in which the ActiveLearning Inventory Tool requires further evaluation. Its role as a basis for providing feedback to instructors should be investigated. One potential valuable contribution would be as part of a peer-evaluation process to raise instructor awareness of their usage of active learning and facilitate change in teaching techniques. It may also serve as the foundation for future research or faculty development programs to reduce barriers to the use of active learning in all disciplines and types of content. Lastly, we hope to explore the role of this tool as a change agent to increase use of active learning in large classes or to stimulate a dialog that results in improved student learning in large classes. CONCLUSION A valid and reliable tool to measure the type and amount of active learning in large pharmacy classrooms was developed. Use and testing of this tool will be expanded to other university disciplines and other colleges and schools of pharmacy. Most importantly, we hope to increase dialog about the use of effective techniques to improve student learning in institutions across the country. ACKNOWLEDGEMENTS We would like to thank our expert reviewers, Charles C. Bonwell, PhD, President, Bonwell Associates and Professor Emeritus of History; Eric H. Hobson PhD, Associate Dean for Academic Affairs and Assessment at South University; and Linda B. Nilson, PhD, Director, Office of Teaching Effectiveness and Innovation at Clemson University, for their valuable feedback. We would also like to thank Dr. Eric Hobson for the use of his active-learning activities chart, the instructors who were kind enough to let us use their classes to test reliability and determine their perceptions, the Northeastern University Provost for the funding to support this work, and Dr. Jennifer Trujillo for critically reviewing the manuscript. REFERENCES 1. Qualters DM. Do students want to be active? J Scholarship Teach Learn. 2001;2:51-60. 2. Kuh G. Assessing what really matters to student learning: inside the national survey of student engagement. Change. 2001;33:10-7. 3. Wasley P. Underrepresented minority students benefit most from engagement. Chronicle Higher Educ. 2006;53:A39. 4. Hudson B. Holding complexity and searching for meaning: teaching as reflective practice. J Curriculum Stud. 2002;34:43-57. 5. Moffett BS, Hill KB. The transition to active learning: a lived experience. Nurse Educator. 1997;22:44-7. 6. Bonwell CC, Eison JA. Active Learning: Creating Excitement in the Classroom. ASHE-ERIC Higher Education Report No. 1. Washington, DC: The George Washington University, School of Education and Human Development; 1991. 7. Gokhale A. Collaborative learning enhances critical thinking. J Technol Educ. 1995;7:22-30. 8. Brown MN, Freeman K. Distinguishing features of critical thinking classrooms. Teaching Higher Educ. 2000;5:301-9. 9. Bransford JD, Brown AL, Cocking RR. How People Learn: Brain, Mind, Experience and School. Washington, D.C.: National Academy Press; 2000. Available at: http://www.nap.edu/html/howpeople1/. Accessed January 04, 2005. 10. Background Paper II: Entry level, curricular outcomes, curricular content and educational process. Am J Pharm Educ. 1993;57:377-85. 11. Accreditation Council for Pharmacy Education. Accreditation Standards and Guidelines for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree. Adopted January 15, 2006. Available at: http://www.acpe-accredit.org/pdf/ACPE_ Revised_PharmD_Standards_Adopted_Jan152006.pdf. Accessed January 17, 2007. 12. Center for Work and Learning. Practice-Oriented Education at Northeastern University. Available at: http://www.poe.neu.edu/ about/whatispoe.html. Accessed January 06, 2005. 13. Prochaska JO, DiClemente CC, Norcross JC. In search of how people change. Am Psychol. 1992;47:1102-14. 14. Qualters DM, Sheahan T, Isaacs J. An electronic advice column to effect teaching culture change. To Improve the Academy (Professional and Organizational Development Network, http:// www.podnetwork.org). 2006;24:201-16 15. Angelo TA, Cross KP. Classroom Assessment Techniques: A Handbook for College Teachers. 2nd ed. San Francisco, Calif: Jossey Bass, Inc; 1993. 16. Patton M. How to Use Qualitative Methods in Evaluation. London, England: Sage Publications; 1987. 17. Bogdon R, Iklen S. Qualitative Research for Education: An Introduction to Theory and Methods. Boston, Mass: Allyn and Bacon; 1992. 18. Accounting Education Change Commission. Objectives of education for accountants: position statement number one. Issues Accounting Educ. 1990;5:307-12. 19. Albrecht WS, Sack RJ. Accounting Education: Charting the Course Through a Perilous Future. Accounting Education Series No. 16. Sarasota, Fla: American Accounting Association; 2000. 20. Commission on Collegiate Nursing Education. Standards for Accreditation of Baccalaureate and Graduate Nursing Education Programs. Available at: http://www.aacn.nche.edu/Accreditation/ standrds.htm. Accessed January 06, 2005. American Journal of Pharmaceutical Education 2007; 71 (5) Article 85. 6
American Journal of Pharmaceutical Education 2007;71(5)Article 85. Appendix 1.Final version of the Active Learning Inventory Tool2006 Van Amburgh,Devlin,Kirwin,Qualter A cold-o One ot from a particular lesson /s e to a prome the kodtoshawhlar9r9gro tudents write down everything they know about an announced topi E uring or after an in-dass presentation studonts wrte a quick response to the prompt "What was the muddiest point in nation on what students perceive they already know coly 1-2 pr ts to real life stuation CgonRh8"eg0eiocapiureertaleementothecours icipate in th re stu performance against applcable criteria extend to offer specific suggestions for improvement terial.-led by口Facutty vs.口Student 0 ngs or dia ams that show the nections that s make betveen a major c ncept presented and other concepts they have learned arios that re ire students to intehe skils to solve problems tha relate toc rio-based problem-solving activity using small groups to tackle specife questions/issues from larger list from cu 7
Appendix 1. Final version of the Active Learning Inventory Tool 2006 Van Amburgh, Devlin, Kirwin, Qualters American Journal of Pharmaceutical Education 2007; 71 (5) Article 85. 7
American Journal of Pharmaceutical Education 2007;71 (5)Article 85 Type/Lecation of Room: A Total tim0(mn0s】 ALOaeummary er res nt during the activit 8
American Journal of Pharmaceutical Education 2007; 71 (5) Article 85. 8