You are on page 1of 4

Practice: Get quality data into your evaluator’s hands

Key Action: Manage logistics and communications for data collection

SAMPLE MATERIAL: Successfully Collecting Data From Comparison Groups

Purpose: This reading offers tips for collecting high-quality data
from comparison groups, an essential—and often
challenging—step in conducting a rigorous, quasi-
experimental evaluation. The focus is on building
relationships with comparison group teachers that
engender their trust and cooperation.

Source: Excerpted from a research paper presented by Edvantia
at the 2008 Annual Meeting of the American Evaluation
Association: Cowley, K. S., Chadwick, K. L., Hughes-
Webb, G. K., Jones, A. R., Voelkel, S., Samanta, D., &
O’Connor, C. (2008). The challenges of implementing
experimental and quasi-experimental designs in
Teaching American History (TAH) grant evaluations (pp.
8-10). Charleston, WV: Edvantia.

Note: Although this reading is based on a Teaching
American History grant evaluation, the information is
useful for any rigorous evaluation.

1

 On a related note.  K. If geographic distance and travel expenses prohibit evaluator on-site presence (as is the case with most of our TAH evaluations). 8-10).) 2 Cowley. x Example:Successfuldatacollectionreliesheavilyontheprojectdirectorsand/orkey staff. & O’Connor.whocanrequest.evaluatorshavestrivedtotranslateour languageandstandardsintoaformatthatourclientsandotherstakeholders (includingcomparisonteachers)willunderstand. S. Samanta.Forallprojects.Some of the same challenges that relate to teacher recruitment for evaluation activities overlap with the challenges evaluators face with collecting data from comparison group participants.. and then follow up with electronic communications if needed. yet not comply with data collection procedures. D. thus highlighting both the technical and people skills that evaluators must have to be effective. A. R. However. (Note: this personal contact is critical for sustaining active participation by treatment teachers as well..nudge.. One main issue with comparisons is that they may consent to be in the comparison group. collecting data from comparison groups has required evaluator flexibility and extensive use of persuasion skills. Hughes-Webb. then project directors and/or key staff (and even participating teachers) can provide personal contact that helps foster participation. G. personal contact is critical.buyinto. K.andactivelysupport. S.. (2008). Educatingprojectstaffabouttheimportanceofgooddatacollection. WV: Edvantia. Jones. Practice: Get quality data into your evaluator’s hands Key Action: Manage logistics and communications for data collection The Challenges of Implementing Experimental and Quasi-Experimental Designs in Teaching American History (TAH) Grant Evaluations CollectingDatafromComparisonGroups Collecting high-quality data from treatment and comparison groups is crucial for experimental and quasi-experimental TAH evaluations. It is often relatively easy to collect data from treatment teachers who have agreed to participate in all aspects of the TAH project. L. .. C.orevenmoreforcefullyaskthatcomparisonteachers complywithdatacollectionactivities. Helpingallstakeholdersunderstandthataccurateandcompletedatacollectionis keytoensuringstudyvalidityandtohelpingrevealprojectimpactisanecessityfor eachandeveryproject. Voelkel. Charleston. Chadwick. Evaluators must be sure to provide project staff with the proper message.Thisoftenrequirespatienceonbehalfof evaluatorsastheyhelpprojectstaffunderstandevaluationingeneral. The challenges of implementing experimental and quasi-experimental designs in Teaching American History (TAH) grant 2 evaluations (pp. K.sothat projectstaffcancommunicatetheimportanceofdatacollectiontobothcomparison andtreatmentgroups. including the evaluation..

Again. Jones. The challenges of implementing experimental and quasi-experimental designs in Teaching American History (TAH) grant 3 evaluations (pp.and theyhaveseenfirsthandthattherehasbeennonegative“fallout”fromother teacherswhoparticipateindatacollectionactivities. Charleston.etc.”resultinginteachersbeingeagertoparticipateinthetreatmentorat leastfeelingsafeenoughtoparticipateasacomparison.Asaresult. incentives may increase the likelihood that teachers will agree to serve as comparisons.Someprojects includelanguageintheagreementstoindicatethatcertainactivities/benefitscould bewithheldifparticipantsdonotcomplywithalldatacollectionactivities.Anotherprojectpaidpartofthestipendatthebeginningof theprojectyearandtheremainderattheconclusionoftheprojectyear. S. thinking data might be used against them in some way.formultiyearprojectsor districtswithseveralprojects. Voelkel.Inother words.hasbeenthereforanumber ofyears.a directorisalsothesocialstudiesdirectorofthedistrict..andsubsequentlyhasestablishedgoodrelationshipswiththehistoryand socialstudiesteachers..Evaluators must also deal with the challenge of trust—some teachers (or other stakeholders) may be distrustful of evaluation.graduatecredit.shewasabletouseherpersonalrelationshipsto persuadecomparisonteacherstocomplywithallrequestedactivities. D.theprojectsthemselvesmayhaveacquireda“good reputation.Forthoseprojectsinwhich participatingteachersrecruittheirowncomparisons.As noted earlier.  x Example:Ineachproject. Incentives. R.. A.classroommaterials. G.Inoneproject. K. L.Notonlydidcomparisonteachersreceivefreerefreshments.The influence or “clout” of the project director has also been critical to successful data collection. x Example:Someprojectdirectorshavemorecloutthanothers. WV: Edvantia. 8-10). .. Hughes-Webb. & O’Connor. K..thesamestaffhaveservedasthedirectorsfor severalTAHprojects. S..Too.Althoughparticipationin evaluationͲrelateddatacollectionactivitiescannotbemandatedbyevaluators.Still anotherprojectheldthefinaldatacollectionactivityforcomparisonsata restaurant. K.participatingteachersreceivesubstantialbenefits—free fieldtrips. x Example:TheTAHprojectthatseemstohavebeenmostsuccessfulwithfinancial stipendslinkedthestipendvoucherstothefinaldatacollectionactivity.Therefore. Incentives may also increase the likelihood that comparison teachers will fully complete all associated data collection activities. (2008). Usually.teachersinthedistrictsknowandtrustthem. Chadwick. Practice: Get quality data into your evaluator’s hands Key Action: Manage logistics and communications for data collection Projectdirector’sinfluence.  Trust. C. the reputation of the project director can help ease any anxieties. Samanta.the projectdirectorhasmoreofanopportunitytoexplicate/expectfullcompliance whenteachersagreetoparticipateastreatmentorcontrolteachers. x Example:Inseveralofourprojects.comparisonteachersmusthavecompletedthefinalactivitybeforethey receivedtheirstipend. evaluators have no direct influence over the comparison teachers. 3  Cowley.trusthasbeenmucheasierto establishbecausetheteachersinvolvedalreadyknowandworkwithoneanother.the arrangementalsoprovidedacontrolledsettingfortheposttestadministration.

.anddidnothavetheirownclassrooms. Charleston. The last challenges we encountered are associated with collecting student data from comparison teachers. Hughes-Webb. D. K.Inmostcases..Intheseinstances. L.althoughnostudentdatawereavailablefortheseteachers.theirstudents arenolongerintheappropriategradelevel. x Example:Insomeprojects. S.theseteachersheldresourceorspecialeducation positions.. The challenges of implementing experimental and quasi-experimental designs in Teaching American History (TAH) grant 4 evaluations (pp. Comparisonteacherswithoutstudentsorwithstudentsfromdifferentgrades. Practice: Get quality data into your evaluator’s hands Key Action: Manage logistics and communications for data collection Finally. WV: Edvantia. 4  Cowley.wedonotcollect studentdata.severalofthecomparisonteachers(andoccasionallya participatingteacher)endedupnothavinganystudentstocompletethepre/post knowledgetest. Voelkel..these teacherswereretainedascomparisonsandtheirownpre/postknowledgetest resultswereused.oneprojectprovidedcontrolsfromthefirst2yearsoftheprojectwiththe incentive/assurancethattheywouldbecometreatmentteachersinthethirdand finalyear. K. Jones. G. S. Chadwick. & O’Connor.wehaveencountered situationsinwhichteachers(bothtreatmentandcomparison)werereassignedto differentgradelevelsfromoneschoolyeartothenext.. C.  x Example:Inoneprojectinwhichthetreatmentyearextendsacrossschoolyears. We have faced situations in which comparison teachers did not have classroom students and where comparison teachers do not teach the targeted grade level. R.. 8-10). Samanta. (2008). . andthetreatmentisfocusedspecificallyononegradelevel. K.Giventhedifficultiesencountered infindinganadequatenumberofcomparisonsforseveraloftheprojects. A.Asaresult.