American Political Science Review Vol.104,No.1 propositions.No study to date has been able to Version 1:Do you agree or disagree that the US distinguish between attitudes toward highly skilled should allow more highly skilled immi- immigrants and attitudes toward low-skilled immi- grants from other countries to come and grants,even though this distinction is a critical fea- live here?(emphasis added) ture of the theoretical story.Below we describe a Version 2:Do you agree or disagree that the US survey experiment aimed at addressing this short- should allow more low-skilled immi- coming and providing an explicit test of arguments grants from other countries to come and about how economic concerns shape attitudes toward live here?(emphasis added) immigration. Answer options (both versions): THE SURVEY EXPERIMENT Strongly Somewhat Neither agree disagree disagree nor disagree Design 2 3 Our experiment was embedded in the Cognitive Styles Somewhat Strongly Survey (CSS),a survey instrument designed to study agree agree opinions regarding trade and immigration.The CSS was administered by the research firm Knowledge The two question versions differed only in that they Networks (KN)and fielded between December described the immigrants'skill level as either highly 2007 and January 2008 to some 2.285 panelists who skilled or low-skilled.15 Accordingly,for half the re- were randomly drawn from the KN panel.Of these. spondents,referred to as the treatment group,we 1,601 responded to the invitation,yielding a final measured preferences over highly skilled immigration, stage completion rate of 70.1%.11 The KN panel is whereas for the other half,referred to as the control a probability-based panel where all members have group,we measured preferences over low-skilled immi- a known probability of selection.It covers both the gration.Randomization ensured that the two groups of online and offline U.S.populations aged 18 years respondents were (in expectation)identical in all other and older.The sampling procedure for the CSS thus observed and unobserved characteristics that may con- constitutes a two-stage probability design.12 The found a comparison across groups.16 recruitment rate for this study,reported by KN using The general distribution of preferences over both the AAPOR Response Rate 3(RR3)guidelines,was highly skilled and low-skilled immigrants is displayed 24.6%.13 The final respondent data were adjusted for in Figure 2.For both types of immigration the barplots the common sources of survey error (nonresponse, show the fraction of respondents answering each of coverage error,etc.)using poststratification weights.14 the five answer categories;the superimposed whiskers The rate of item nonresponse was very low,below 1% decode the upper.95 confidence interval derived from for the questions we use in the analysis below. the design-based variance estimator.Two features For the core experiment,we randomly allocated re- stand out in this graph.First,in line with previous spondents to two groups of equal size and presented studies,our survey once again confirms the profound each group with one of two versions of the survey ques- divide among the American public in opinions on tion about immigration: immigration.Pooling over both types of immigration, about 50%of the respondents oppose an increase in immigration,whereas about 25%favor it.Second and 11 All fielded sample cases had one e-mail reminder sent three days more importantly,our findings for the first time docu- ment the fact that preferences over immigration vary after the initial email invitation.No monetary incentive was used in the CSS study.Of the invited respondents,4.5%did break off before rather dramatically depending on the immigrants'skill the interview was completed. levels.Although more than 60%of the respondents 12 Panel members are randomly selected using random digit dialing (in the control group)state that they strongly disagree (RDD)sampling techniques on the sample frame consisting of the or somewhat disagree with an increase in low-skilled entire U.S.residential telephone population (both listed and un- immigration,only 40%of the respondents (in the listed phone numbers).Households are provided with access to the Internet and hardware if needed.In contrast to opt-in Web panels, treatment group)are opposed to an increase in highly unselected volunteers are not allowed to join the KN panel.A de- skilled immigration.7 Because of the randomization, tailed report about the KN recruitment methodology and the survey f panle 15 Notice that we stratified the random assignment by four education recruited at different times and committed to answer several surveys levels(described below)so that an equal number of respondents for a period of time.KN panelists must also complete profiling sur- within each education level received the two different versions of veys in order to become members of the panel.These differences the question. make directly comparing response rates between one-time surveys 16 We conducted extensive balance checks by comparing the distri- (such as simple RDD telephone or mail sample)and panel surveys butions of all our covariates in both groups.All tests confirmed that difficult and perhaps not illuminating.See Callegaro and DiSogra (as expected given the large sample size)randomization balanced (2008)for an extended description of how to compute response the distributions evenly.Results are available upon request. metrics for online panels. 14 Poststratification weights are raked to adjust to the demographic 17 In the preimplementation pilot testing.we created a third. "vanilla"version of the question that referred simply to "im- and geographic distributions from the March Supplement of the 2007 migrants",without mentioning skill levels,and we randomly as- Current Population Survey. signed respondents into a third group who answered this question 67American Political Science Review Vol. 104, No. 1 propositions. No study to date has been able to distinguish between attitudes toward highly skilled immigrants and attitudes toward low-skilled immi￾grants, even though this distinction is a critical fea￾ture of the theoretical story. Below we describe a survey experiment aimed at addressing this short￾coming and providing an explicit test of arguments about how economic concerns shape attitudes toward immigration. THE SURVEY EXPERIMENT Design Our experiment was embedded in the Cognitive Styles Survey (CSS), a survey instrument designed to study opinions regarding trade and immigration. The CSS was administered by the research firm Knowledge Networks (KN) and fielded between December 2007 and January 2008 to some 2,285 panelists who were randomly drawn from the KN panel. Of these, 1,601 responded to the invitation, yielding a final stage completion rate of 70.1 %.11 The KN panel is a probability-based panel where all members have a known probability of selection. It covers both the online and offline U.S. populations aged 18 years and older. The sampling procedure for the CSS thus constitutes a two-stage probability design.12 The recruitment rate for this study, reported by KN using the AAPOR Response Rate 3 (RR3) guidelines, was 24.6%.13 The final respondent data were adjusted for the common sources of survey error (nonresponse, coverage error, etc.) using poststratification weights.14 The rate of item nonresponse was very low, below 1% for the questions we use in the analysis below. For the core experiment, we randomly allocated re￾spondents to two groups of equal size and presented each group with one of two versions of the survey ques￾tion about immigration: 11 All fielded sample cases had one e-mail reminder sent three days after the initial email invitation. No monetary incentive was used in the CSS study. Of the invited respondents, 4.5 % did break off before the interview was completed. 12 Panel members are randomly selected using random digit dialing (RDD) sampling techniques on the sample frame consisting of the entire U.S. residential telephone population (both listed and un￾listed phone numbers). Households are provided with access to the Internet and hardware if needed. In contrast to opt-in Web panels, unselected volunteers are not allowed to join the KN panel. A de￾tailed report about the KN recruitment methodology and the survey administration is available from the authors upon request. 13 Notice that an online panel such as KN is composed of people recruited at different times and committed to answer several surveys for a period of time. KN panelists must also complete profiling sur￾veys in order to become members of the panel. These differences make directly comparing response rates between one-time surveys (such as simple RDD telephone or mail sample) and panel surveys difficult and perhaps not illuminating. See Callegaro and DiSogra (2008) for an extended description of how to compute response metrics for online panels. 14 Poststratification weights are raked to adjust to the demographic and geographic distributions from the March Supplement of the 2007 Current Population Survey. Version 1: Do you agree or disagree that the US should allow more highly skilled immi￾grants from other countries to come and live here? (emphasis added) Version 2: Do you agree or disagree that the US should allow more low-skilled immi￾grants from other countries to come and live here? (emphasis added) Answer options (both versions): Strongly Somewhat Neither agree disagree disagree nor disagree 12 3 Somewhat Strongly agree agree 4 5 The two question versions differed only in that they described the immigrants’ skill level as either highly skilled or low-skilled. 15 Accordingly, for half the re￾spondents, referred to as the treatment group, we measured preferences over highly skilled immigration, whereas for the other half, referred to as the control group, we measured preferences over low-skilled immi￾gration. Randomization ensured that the two groups of respondents were (in expectation) identical in all other observed and unobserved characteristics that may con￾found a comparison across groups.16 The general distribution of preferences over both highly skilled and low-skilled immigrants is displayed in Figure 2. For both types of immigration the barplots show the fraction of respondents answering each of the five answer categories; the superimposed whiskers decode the upper .95 confidence interval derived from the design-based variance estimator. Two features stand out in this graph. First, in line with previous studies, our survey once again confirms the profound divide among the American public in opinions on immigration. Pooling over both types of immigration, about 50% of the respondents oppose an increase in immigration, whereas about 25% favor it. Second and more importantly, our findings for the first time docu￾ment the fact that preferences over immigration vary rather dramatically depending on the immigrants’ skill levels. Although more than 60% of the respondents (in the control group) state that they strongly disagree or somewhat disagree with an increase in low-skilled immigration, only 40% of the respondents (in the treatment group) are opposed to an increase in highly skilled immigration.17 Because of the randomization, 15 Notice that we stratified the random assignment by four education levels (described below) so that an equal number of respondents within each education level received the two different versions of the question. 16 We conducted extensive balance checks by comparing the distri￾butions of all our covariates in both groups. All tests confirmed that (as expected given the large sample size) randomization balanced the distributions evenly. Results are available upon request. 17 In the preimplementation pilot testing, we created a third, “vanilla” version of the question that referred simply to “im￾migrants”, without mentioning skill levels, and we randomly as￾signed respondents into a third group who answered this question. 67