Why use experiments for program evaluation?

David W. Nickersondavid.w.nickerson@gmail.comJared P. Barton jared_p_barton@yahoo.com

Broad Overview and Explanation

One of the challenges of managing fieldwork is evaluating the success of the program. Program implementers want to replicate successful programs, and also to learnfrom (and avoid repeating) unsuccessful programs. It is also valuable to understand howcostly achieving a goal is, so organizations can plan cost-effectively in the future.Experiments can help answer both questions of success and of cost-effectiveness, but todo so the research protocols need to be designed and implemented prior to the fieldwork.When nonprofit organizations evaluate their operations, they often rely uponobservational data passively collected during the course and at the end of their programs.For instance, suppose an organization selected precincts in which to mobilize voters for an upcoming election. The group selected precincts on the basis of the density of Asian-American population and the lack of secure apartment buildings to facilitate canvassing.A typical observational study would take careful notes on the activities in the selected precincts and then compare the rates of turnout of these particular precincts to theneighboring precincts. Suppose the group discovered that in its selected precincts turnoutwas 5 percentage points higher than in neighboring precincts -- even after "controllingfor" age and past turnout. Can the group safely conclude that its program was a success? Not necessarily. There may be other reasons that these precincts had higher turnout that are completely unrelated to the mobilization efforts of the group. For instance, perhaps the requirement that the precincts have few secure apartment buildings pushed the campaign into slightly wealthier neighborhoods with more single-familyhousing. Since income is positively correlated with voter turnout, it is possible that thetype of neighborhood selected is responsible for the 5 percentage point difference. Thus,the difference in turnout could be the result of unobserved factors and not the result of themobilization campaign. The potential causes could be even more subtle and unseen. For instance, suppose the Asian-American neighborhoods populated by the campaign aregentrifying. The higher voting rate might be a result of the changing demographics of thecommunity rather than the mobilization campaign. In short, it is impossible to knowwhether the boost in turnout was the result of the mobilization campaign.Moreover, the campaign managers do not know how much of the success can beattributed to each technology. Was it the door-to-door knocking? Posters? Telephonecalls? How much did each technology contribute to the success of the campaign? Werethe early efforts conducted in September as effective as the efforts in the week prior to the

election? Was it even worth working in the field a month prior to the election? These arequestions that an observational study is unable to answer. In contrast, a properly designed experiment can provide answers to each of thesequestions. At its most basic, an experiment manipulates the application of campaigntechniques using the following steps:1)The list of targets are identified and listed -- these are the people the campaignwould like to contact with the technique;2)The people listed as targets are randomly divided into a treatment group, where people are contacted using the technique, and a control group that is not contacted by the campaign (note: the division need NOT be 50-50);3)The campaign then goes about its business contacting the treatment group (andleaving the control group alone);4)The outcome variable (e.g., registration or turnout or support for a ballot proposition) is then measured.Because the assignment to treatment and control groups is random, they should becomposed identically. They should be the same age, the same education, the same votehistory, and the same income on average. Due to randomization, the treatment and controlgroup should also be identical with regards to matters that the campaign can't know suchas interest in the election, personal connections to the campaign, or convenience of driving to the polls on Election Day. The only difference between the two groups should be who receives the treatment. We say “should be” because the similarity of the twogroups due to randomness depends upon the size of the overall population; we need asufficiently large group to use this property in assuming sameness. Thus, to measure theeffectiveness of the campaign, the manager need only compare the rate of turnout in thetreatment group to the control group. If one voted at 45% and the other 37%, then thecampaign boosted turnout in the treatment group by 8%. Moreover, we can then calculatethe expenses of the campaign and determine how much it cost to generate each "new"vote. For instance, say the campaign that boosted turnout by 8 percentage points spent $5on each attempt to contact a member of the treatment group. Then, we could concludethat each new vote cost $62.50. The manager now has the ability to decide whether toemploy the same strategy in the next election knowing roughly how cost effective it wasin the past election.

Requirements

The requirements to run an experiment are not onerous:1)An identifiable subject population. For instance, Asian-American identified onthe VAN or streets in neighborhoods with a high density of Asian-Americans areclearly defined. People at a street fair or festival are not a well-defined group.2)A treatment that can be randomized. Mail, phone, and doors are very easy tomanipulate. A national TV campaign or web broadcasts would be difficult tostudy.3)The ability to measure the outcome variable of interest. Registration and turnoutare public records that can be gathered. Vote choice would probably require a post-election survey, as would attitudes about the electoral process.

4)The willingness to extract a control group and abide by the agreed upon protocol.Sadly, it is this last requirement that trips up most organizations, and causes the loss of many opportunities to learn from past field work. We recognize that your desire to learnfrom experiments needs to be balanced by the goals of your program. We will work withyou through the series of trade-offs and compromises that will inevitably come up.Experiments that are difficult to implement (politically or logistically) are often difficultfor organizations to execute properly and thus lose a learning opportunity for both theorganization and the evaluators. Thus, our overriding goal when designing experimentsto test your field operations is to tailor the study to fit your organization as closely as possible.What does this mean in terms of how you would go about conducting anexperiment with us? The process can be broken into a series of steps (note: the steps below assume that you are using the VAN and conducting phone or door-to-door canvassing):1)Tell us what you plan to do, how your organization is structured, the logisticaldetails behind your field operation, and any political constraints imposed bycoalition members, donors, or past history (e.g., we have to hit 1000 doors inPrecinct 5). This information will help us tailor an experiment to your need. Youshould also tell us what questions you would like answered in this election cycleand concerns you have about the experimental process, so we can be sure toaddress these issues up front.2)We will work with you to come up with an evaluation plan that imposes as few burdens on you as possible.3)You email a list of the places or people that you are targeting and will be subjectsin the experiment. If you are using the VAN, you simply request an export of your target list.4)We randomly assign the experimental subjects to the treatment and control groupsand return the assignments to you. If you are using the VAN, we will return a filewith the VANID, a flag for the treatment group, and perhaps a messageassignment as well.5)Upload the file onto the VAN.6)When pulling your walk or phone lists, in addition to the targeting (e.g., precinct5, Asian, under 50), select only the treatment individuals (you just check a box).7)If there is a messaging component to the experiment, make sure the walk or callsheet you formatted includes the randomly assigned message flag (e.g., “E” for early voting and “D” for Election Day) in addition to name and address/phone.8)Print the walk or call sheets.9)Perform the walking and calling.10)Be sure to enter in the contact information (e.g., not home, left message,contacted).11)The day after Election Day, send us: a) any information on how you targetedneighborhoods; b) any information on what canvassers/callers said; c) copies of printed materials like flyers, mailers, or pamphlets; d) all the contact information;e) reports of any problems or colorful anecdotes.