Daniel M.Butler and Hans J.G.Hassell 7 Follow-up Survey.Three days after the city official Attrition can also be a concern if it is related to treat- sent the email message,we emailed citizens for the ment assignment.In other words,if a given treatment is follow-up survey to measure the impact of the treat- causing people to systematically drop out of the survey, ment.Only citizens who took the baseline survey this can introduce bias.Columns 1 and 3 of Table 4 test and agreed to the follow-up survey were contacted. for whether treatment assignment is related to drop- In addition to the initial invitation,we sent two re- ping out of the survey.We again look at the results minder emails for those who had not taken the sur- both at the individual level(column 1)and the issue vey yet.The initial invitation and the reminder in- level (column 3).These probit regressions look at all vitations were spaced two to three days apart.Most the observations that were in the sampling frame when people who took the follow-up survey did so within we randomized (i.e.,because they completed the pre- a week of receiving the email from their city offi- treatment survey and were eligible for treatment).The cial.For our placebo design,we restrict the sample dependent variable is simply whether the observation to those who opened the email from the official and is in the final sample (i.e..because the respondent an- who did so before taking the follow-up survey(Nick- swered the question in the post-treatment survey).The erson 2005) results show that the missing observations(missing be cause of attrition)are missing independent of treat- INFORMATION ABOUT THE SAMPLE ment assignment.In other words,individuals (or the issues we asked about)that were assigned to the treat- We achieved a high follow-up rate on the post- ment group were not more or less likely to attrite from treatment survey (especially given that no incentives the study. were provided),with 68%of respondents(310 out of Finally,attrition can affect the population we learn 455)from the first round also taking the follow-up sur- about in the study.Columns 2 and 4 of Table 4 analyze vey.For the analysis,we use the 80%of respondents whether certain types of people were more likely to from this subset(244 out of 310)who also opened the drop out of the study.As Column 2 shows,there was no 4号元 email sent from the official(as tracked by MailChimp) systematic attrition at the individual level.There was before taking our survey.We use this subsample in some attrition at the issue level,with older individuals our analysis because they are the ones who were ex- and men being more likely to be in the final sample. posed to the intended message (either the treatment Table 5 provides a more general overview of our or placebo message).When we analyze the individual- sample.Our study investigates the reactions of individ- level data.we have 244 observations.When we look at uals who are reached by public officials.These individ- individuals issue priorities,we have 415 observations uals differ from the general population in systematic (408 of whom answered all the pre-treatment demo- ways and this is reflected in our sample.Our sample graphic questions).14 is older,politically interested,and highly likely to fol- Attrition can be a problem if it leads to imbalance low local politics.These individuals may respond differ- 是 between the treatment and placebo groups.We test for ently than the general public to messages from politi- balance by regressing the randomly assigned treatment cians.These differences,however,are intentional;we (1 treatment,0=placebo)on the demographics that are interested in politicians'ability to influence the au- were gathered in the first wave of the survey (and thus dience to whom they are regularly communicating. S5.501g measured pre-treatment).The independent variables include gender,education (six-point scale),age (six- point scale),level of political interest (four-point scale) RESULTS and how much they follow local politics(three-point scale).The wording for these questions is provided in Because we are interested in the ability of politicians to the Section SI.2 of the Supplementary Material.Be- change their constituents'priorities and to encourage cause we analyze the results at both the individual level their constituents to take political action,we estimate and at the issue level,Table 3 presents the balance tests the effect of the issue priority email treatment on two for both levels of the data(column 1 for the individual outcomes.First,we test whether the elected official's level and column 2 for the issue level).These probit message increases the priority of the issue for the re- regressions test the significance of each variable indi- spondent.In the pre-treatment survey,we asked indi- vidually and all the variables jointly (see the bottom of viduals about their attitudes on the issue,using a four- the table for the results of the joint significance tests) point scale(that included the degree to which the issue The variables fail to achieve statistical significance both is a priority).We include the same question on the post- individually and jointly.We have balance on these pre- treatment survey (for all the issues in that city)to ana- treatment characteristics. lyze the impact of the official's message.The question wording is provided in Section SI.2 of the Supplemen- 14 The sample size affects power.In Section SI.6 of the Supplemen- tary Material.Because these questions included four tary Material,we present simulations to investigate how much power categories,we use an ordered probit model to analyze we have for our specific sample size at different treatment effect this outcome. sizes (Coppock 2013).While not drastically underpowered,the re- As noted in the procedures above,when constituents sults show that the power for the analyses are roughly between 0.6 and 0.65 for the treatment effects we find.The results of the simu- were moveable on more than one issue.we random- lation,along with the R-code to produce them,are given in Section ized which issue the official would write about in his SI6 of the Supplementary Material. or her email message.For the analysis,we maximize 866Daniel M. Butler and Hans J.G. Hassell 7. Follow-up Survey. Three days after the city official sent the email message, we emailed citizens for the follow-up survey to measure the impact of the treat￾ment. Only citizens who took the baseline survey and agreed to the follow-up survey were contacted. In addition to the initial invitation, we sent two re￾minder emails for those who had not taken the sur￾vey yet. The initial invitation and the reminder in￾vitations were spaced two to three days apart. Most people who took the follow-up survey did so within a week of receiving the email from their city offi￾cial. For our placebo design, we restrict the sample to those who opened the email from the official and who did so before taking the follow-up survey (Nick￾erson 2005). INFORMATION ABOUT THE SAMPLE We achieved a high follow-up rate on the post￾treatment survey (especially given that no incentives were provided), with 68% of respondents (310 out of 455) from the first round also taking the follow-up sur￾vey. For the analysis, we use the 80% of respondents from this subset (244 out of 310) who also opened the email sent from the official (as tracked by MailChimp) before taking our survey. We use this subsample in our analysis because they are the ones who were ex￾posed to the intended message (either the treatment or placebo message). When we analyze the individual￾level data, we have 244 observations. When we look at individuals issue priorities, we have 415 observations (408 of whom answered all the pre-treatment demo￾graphic questions).14 Attrition can be a problem if it leads to imbalance between the treatment and placebo groups.We test for balance by regressing the randomly assigned treatment (1 = treatment, 0 = placebo) on the demographics that were gathered in the first wave of the survey (and thus measured pre-treatment). The independent variables include gender, education (six-point scale), age (six￾point scale),level of political interest (four-point scale), and how much they follow local politics (three-point scale). The wording for these questions is provided in the Section SI.2 of the Supplementary Material. Be￾cause we analyze the results at both the individual level and at the issue level,Table 3 presents the balance tests for both levels of the data (column 1 for the individual level and column 2 for the issue level). These probit regressions test the significance of each variable indi￾vidually and all the variables jointly (see the bottom of the table for the results of the joint significance tests). The variables fail to achieve statistical significance both individually and jointly. We have balance on these pre￾treatment characteristics. 14 The sample size affects power. In Section SI.6 of the Supplemen￾tary Material, we present simulations to investigate how much power we have for our specific sample size at different treatment effect sizes (Coppock 2013). While not drastically underpowered, the re￾sults show that the power for the analyses are roughly between 0.6 and 0.65 for the treatment effects we find. The results of the simu￾lation, along with the R-code to produce them, are given in Section SI.6 of the Supplementary Material. Attrition can also be a concern if it is related to treat￾ment assignment. In other words, if a given treatment is causing people to systematically drop out of the survey, this can introduce bias. Columns 1 and 3 of Table 4 test for whether treatment assignment is related to drop￾ping out of the survey. We again look at the results both at the individual level (column 1) and the issue level (column 3). These probit regressions look at all the observations that were in the sampling frame when we randomized (i.e., because they completed the pre￾treatment survey and were eligible for treatment). The dependent variable is simply whether the observation is in the final sample (i.e., because the respondent an￾swered the question in the post-treatment survey). The results show that the missing observations (missing be￾cause of attrition) are missing independent of treat￾ment assignment. In other words, individuals (or the issues we asked about) that were assigned to the treat￾ment group were not more or less likely to attrite from the study. Finally, attrition can affect the population we learn about in the study. Columns 2 and 4 of Table 4 analyze whether certain types of people were more likely to drop out of the study. As Column 2 shows, there was no systematic attrition at the individual level. There was some attrition at the issue level, with older individuals and men being more likely to be in the final sample. Table 5 provides a more general overview of our sample. Our study investigates the reactions of individ￾uals who are reached by public officials. These individ￾uals differ from the general population in systematic ways and this is reflected in our sample. Our sample is older, politically interested, and highly likely to fol￾low local politics.These individuals may respond differ￾ently than the general public to messages from politi￾cians. These differences, however, are intentional; we are interested in politicians’ ability to influence the au￾dience to whom they are regularly communicating. RESULTS Because we are interested in the ability of politicians to change their constituents’ priorities and to encourage their constituents to take political action, we estimate the effect of the issue priority email treatment on two outcomes. First, we test whether the elected official’s message increases the priority of the issue for the re￾spondent. In the pre-treatment survey, we asked indi￾viduals about their attitudes on the issue, using a four￾point scale (that included the degree to which the issue is a priority).We include the same question on the post￾treatment survey (for all the issues in that city) to ana￾lyze the impact of the official’s message. The question wording is provided in Section SI.2 of the Supplemen￾tary Material. Because these questions included four categories, we use an ordered probit model to analyze this outcome. As noted in the procedures above, when constituents were moveable on more than one issue, we random￾ized which issue the official would write about in his or her email message. For the analysis, we maximize 866 Downloaded from https://www.cambridge.org/core. Shanghai JiaoTong University, on 26 Oct 2018 at 03:53:05, subject to the Cambridge Core terms of use, available at https://www.cambridge.org/core/terms. https://doi.org/10.1017/S0003055418000473