State Voices

–

2008 Experimental Evaluations in the States

1 of 8

State Voices, 501(c)(3) State TablesExperimental Evaluations 2008

Background on Evaluation Approach

State Voices is a non-profit organization that links together and provides support,shared services, information and resources to the sixteen active state 501(c)(3) voter

and civic engagement networks (or “tables”). Our goal is to foster long

-term,collaborative work to engage socially responsible citizens and historicallyunderrepresented communities in our democratic process. State Voices represents theculmination of more than five years of work organizing nonprofit 501(c)(3) votercollaboratives across the country that started before the 2004 elections.One of the top level outcomes of State Voices is expected to be 2-5% annual increases inthe state or local vote share (percentage of the overall vote, controlling for populationchange) of underrepresented or socially responsible voters, via support for aggressive,collaborative registration, education, election protection and GOTV campaigns. Ashared voter file measures this by tracking collective identification and contact data. Alltables syndicate a Catalist voter file and VAN interface to partners.State Voices also uses the shared file to support an innovative experimental evaluationregime that works to measure the specific impact of collaborative nonpartisan voterengagement efforts.

In 2008, fourteen of the sixteen state tables successfully worked with partners to build randomized, controlled experimental evaluations into their work.

With this system, no longer do organizations have to pour energy and effort intoturnout strategies and later wonder whether they were successful. By setting asiderandomly selected control groups, program effectiveness can be accurately andtransparently measured.

How to Conduct Experimental Evaluations

Voters receive a broad range of intense messages about voting from a wide variety of

formal and informal sources. Isolating the unique effect of a single organization’s effort

on the beliefs or behavior of targeted citizens is a daunting task. However, randomized,controlled experiments provide a straightforward means of accurately measuring themarginal effect of different engagement strategies through four easy steps:

State Voices

–

2008 Experimental Evaluations in the States

2 of 8



First, the organization decides which households to target (e.g., infrequentyoung voters living in a house district) and the tactics to engage these people(e.g., phone calls, door knocks, mail).



Second, the target population is randomly divided into a treatment group, whichis exposed to the campaign tactics, and a control group that does not receiveoutreach from the organization. Because the assignment into treatment andcontrol groups is entirely random, on average, they will possess the same pastturnout behavior, interest in the election, exposure to the media, and contactfrom other campaigns. The only difference between the two groups is theimpact of the campaign or tactics the organization intends to measure.



Third, the organization executes the engagement strategy to be tested keepingcareful track of which citizens are successfully contacted.



Finally, official state-by-state voter turnout records determine whether thetreatment or control groups voted at higher rates. Before and after surveys of the groups can help find the persuasion impact of an issue education campaign.Organizations conducting civic engagement work during elections are under manycompeting pressures. Working with small budgets and staff, organizers have to managevolunteers, data, and partner groups. Finding the time to work rigorous evaluation intothe day can be difficult. There are three key ingredients to

successful

experimentalevaluation of campaign activities:



First, all levels of the organization being studies be engaged and helped to be asenthusiastic about the process as possible. It is not enough that a handful of board members support effort, the organizers and volunteers on the groundneed to understand why it is important to carry out the experimental evaluation.



Second, the evaluation needs to fit well with the goals and structure of theorganization. Given all the pressures placed on organizers, if an evaluation isdifficult to implement, something will go wrong at some point during thecampaign. Ease of implementation greatly increases an experiment

’s chances

of success.



Finally, the expectations and responsibilities for conducting the experiment needto be explicitly laid out. Each step of the experimental process should be clearlystated and assigned to an individual who will be held accountable for performingthe duty. When designed well and supported by staff who really want to betterunderstand the impact of their work, experiments can happen with minimalhassle and yield huge gains in knowledge.

State Voices

–

2008 Experimental Evaluations in the States

3 of 8

State Table Case Studies

While controlled experiments offer unparalleled ability to make causal claims about theeffect of engagement strategies, there are a wide range of ways to implementexperiments and an infinite number of topics to study. Each state conducted usefulresearch, but the states below offer instructive examples of how to structure evaluationand the kinds of topics that groups can tackle during even the busiest election seasons.

Michigan

: The Michigan table, Michigan Voice, pursued a two-pronged evaluationstrategy in 2008. Working with Notre Dame Professor David Nickerson organizationsconducted two types of randomized experiments.



First, different

niche populations

including Arab Americans, Asian Americans(with 7 sub-populations), Native Americans, students, and members of anenvironmental group did mobilization experiments. Others measured tactics likefundraising on ballot initiatives, meeting attendance, vote-by-mail strategies,and the power of different types of issue information with voters.



Second, the table removed a

universal control group

from the voter file sharedby the organizations. Roughly 1.5 million people met the targeting criteriaagreed upon by organizations at the table. However, the table had the capacityto contact 1.2 million voters; a short fall of 300,000, a natural control group.These people did not appear in any query or count of the file and thereforewould not be contacted by any member of the coalition.Results of both approaches will be ready in early 2009. The universal control groupapproach is very unique as it: a) allowed

every

walk or call or mail list exported by anygroup to automatically be part of an experimental evaluation, and b) allows smallorganizations who alone would not have programs large enough measure to be part of acollective experiment.

Wisconsin:

The Wisconsin table, called the Wisconsin Civic Engagement Project, usedlocal ballot initiatives to advance socially responsible causes and energize localsupporters. In 2006, partners developed and ran successful local health care ballotinitiatives in Eau Claire County and the City of Racine. In 2008, nine municipalities ransimilar referenda that passed by an average of 82%. Wisconsin conducted two novelexperiments during its campaigns:



First, a long term experiment to determine whether petition signing itself leadsto greater participation in an election. Streets were randomized in the spring todetermine the cost-effectiveness of door-to-door signature gathering. These