THE AMERICAN ECONOMIC REVIEW SEPTEMBER 2004 note:"The first day of training also included an examine the nature of the differential treatment introduction to employment discrimination, from many more angles equal employment opportunity, and a review of project design and methodology. This may I. Experimental I generate conscious or subconscious motives among auditors to generate data consistent or A. Creating a Bank of Resu inconsistent with their beliefs about race issues in America. As psychologists know very well, The first step of the experimental design is to these demand effects can be quite strong. It is generate templates for the resumes to be sent very difficult to insure that auditors will not The challenge is to produce a set of realistic and yant to do"a good job. " Since they know the representative resumes without using resumes goal of the experiment, they can alter their that belong to actual job seekers. To achieve behavior in front of employers to express (indi- this goal, we start with resumes of actual job rectly)their own views. Even a small belief by searchers but alter them sufficiently to create ently can result in measured differences in treat- structure and realism of the initial resumes with ment. This effect is further magnified by the fact out compromising their owners. that auditors are not in fact seeking jobs and are We begin with resumes posted on two job therefore more free to let their beliefs affect the search Web sites as the basis for our artificial sumes. While the resumes posted on these Finally, audit studies are extremely expen- Web sites may not be completely representative sive, making it difficult to generate large of the average job seeker, they provide a prac- enough samples to understand nuances and pos- tical approximation 12 We restrict ourselves to sible mitigating factors. Also, these budgetary people seeking employment in our experimental constraints worsen the problem of mismatched cities(Boston and Chicago). We also restrict auditor pairs. Cost considerations force the use ourselves to four occupational categories: sales of a limited number of pairs of auditors, mean- administrative support, clerical services, and ing that any one mismatched pair can easily customer services. Finally, we further restrict drive the results. In fact, these studies generally ourselves to resumes posted more than six tend to find significant differences in outcomes months prior to the start of the experiment. We across paIr purge the selected resumes of the person's name Our study circumvents these problems. First, and contact information because we only rely on resumes and not peo- During this process, we classify the resumes ple, we can be sure to generate comparability within each detailed occupational category into across race. In fact, since race is randomly as- two groups: high and low quality. In judging signed to each resume, the same resume will resume quality, we use criteria such as labor sometimes be associated with an African- market experience, career profile, existence of American name and sometimes with a White gaps in employment, and skills listed. Such a name. This guarantees that any differences we classification is admittedly subjective but it is find are caused solely by the race manipulation. made independently of any race assignment on econd, the use of paper resumes insulates us the resumes(which occurs later in the experi from demand effects. While the research assis- mental design). To further reinforce the quality tants know the purpose of the study, our proto- gap between the two sets of resumes, we add to col allows little room for conscious or each high-quality resume a subset of the follow subconscious deviations from the set proce- ing features: summer or while-at-school em- dures. Moreover, we can objectively measure ployment experience, volunteering experience, whether the randomization occurred as ex- extra computer skills, certification degrees, for pected. This kind of objective measurement is eign language skills, honors, or some military impossible in the of the previous audit ginal cost, we can send out a large number of rewww.careerbuilder.comandwww resumes. Besides giving us more precise esti ve found large variation in skill levels mates, this larger sample size also allows us to among people posting their resumes on these sites994 THE AMERICAN ECONOMIC REVIEW SEPTEMBER 2004 note: "The first day of training also included an introduction to employment discrimination, equal employment opportunity, and a review of project design and methodology." This may generate conscious or subconscious motives among auditors to generate data consistent or inconsistent with their beliefs about race issues in America. As psychologists know very well, these demand effects can be quite strong. It is very difficult to insure that auditors will not want to do "a good job." Since they know the goal of the experiment, they can alter their behavior in front of employers to express (indi￾rectly) their own views. Even a small belief by auditors that employers treat minorities differ￾ently can result in measured differences in treat￾ment. This effect is further magnified by the fact that auditors are not in fact seeking jobs and are therefore more free to let their beliefs affect the interview process. Finally, audit studies are extremely expen￾sive, making it difficult to generate large enough samples to understand nuances and pos￾sible mitigating factors. Also, these budgetary constraints worsen the problem of mismatched auditor pairs. Cost considerations force the use of a limited number of pairs of auditors, mean￾ing that any one mismatched pair can easily drive the results. In fact, these studies generally tend to find significant differences in outcomes across pairs. Our study circumvents these problems. First, because we only rely on resumes and not peo￾ple, we can be sure to generate comparability across race. In fact, since race is randomly as￾signed to each resume, the same resume will sometimes be associated with an African￾American name and sometimes with a White name. This guarantees that any differences we find are caused solely by the race manipulation. Second, the use of paper resumes insulates us from demand effects. While the research assis￾tants know the purpose of the study, our proto￾col allows little room for conscious or subconscious deviations from the set proce￾dures. Moreover, we can objectively measure whether the randomization occurred as ex￾pected. This kind of objective measurement is impossible in the case of the previous audit studies. Finally, because of relatively low mar￾ginal cost, we can send out a large number of resumes. Besides giving us more precise esti￾mates, this larger sample size also allows us to examine the nature of the differential treatment from many more angles. 11. Experimental Design A. Creating a Bank of Resumes The first step of the experimental design is to generate templates for the resumes to be sent. The challenge is to produce a set of realistic and representative resumes without using resumes that belong to actual job seekers. To achieve this goal, we start with resumes of actual job searchers but alter them sufficiently to create distinct resumes. The alterations maintain the structure and realism of the initial resumes with￾out compromising their owners. We begin with resumes posted on two job search Web sites as the basis for our artificial resumes." While the resumes posted on these Web sites may not be completely representative of the average job seeker, they provide a prac￾tical approximation.12 We restrict ourselves to people seeking employment in our experimental cities (Boston and Chicago). We also restrict ourselves to four occupational categories: sales, administrative support, clerical services, and customer services. Finally, we further restrict ourselves to resumes posted more than six months prior to the start of the experiment. We purge the selected resumes of the person's name and contact information. During this process, we classify the resumes within each detailed occupational category into two groups: high and low quality. In judging resume quality, we use criteria such as labor market experience, career profile, existence of gaps in employment, and skills listed. Such a classification is admittedly subjective but it is made independently of any race assignment on the resumes (which occurs later in the experi￾mental design). To further reinforce the quality gap between the two sets of resumes, we add to each high-quality resume a subset of the follow￾ing features: summer or while-at-school em￾ployment experience, volunteering experience, extra computer skills, certification degrees, for￾eign language skills, honors, or some military "The sites are www.careerbuilder.com and www. americasjobbank.com. l2 In practice, we found large variation in skill levels among people posting their resumes on these sites