226 QUALITATIVE DESIGNS AND DATA COLLECTION Review: of relevant literature can also} bring focus to a study. What is already’ novi?" Unimawn? What are the cutting literature can presenta quandary in qualita tive inquiry because jt may bias the re= searcher’s thinking and reduce openness to ‘whatever emerges in the field. Thus, some times a literature review may not take place ‘until after data collection. Alternatively; the literature review may go on simultaneously With eldwork, permitting 2 creative inter- play among the processes of data collection; literature seviet, and researcher introspec: ‘Hon (Marshall and Rassman 1989:38-80).As with other qualitative design issues, irade-offs appear at every turn, for there are decided advantages and disadvantages '3 reviewing the literature before, during, of after fieldwork—or on a continual basis ‘throughout the study. ‘A specific example of possible variations in focus will illustrate the kinds of trade-offs involved in designing = study: Suppose some educators are interested in studying how a school program’ affects the social de velopment of school-age children. "They ‘want to know how the interactions of chil dren with others in the school setting cox tribute to the development of socal skills. ‘They believe that those social skills will Be different for different children, and they age not sure ofthe range of social interactions qualitative inquiry th tionsin program exper experiences to indivi ‘What then aretradeofsindetemining the final focus? child has social interact ‘many people. The first problem in focusing, then, i to determine how much ofthe social reality experienced by children we should those interactions that occur in the class- ‘oom, thereby increasing the scope of the study to' include interactions not only be- ween teacher and child, but. also. among peerain the classroom and between any vol lunteers and visitor othe classroom and th children, Broadening the scope ofthe study stil moze, we might decide to look at all of the socal relationships that children experi- tice in schools in this case we would move dministraiive staff. Broadening the of the study stl further, the educators decide that it is important to look at the social relationships children experience at ome as wel as at school so as to under- sland better how children experience and ‘A case could be made for the importance omes’clesr; immediately, that there are tradeoffs between breadth and depth. A highly focused study of the studentteather sume our entire bud get Bt allow. us to investigate the issue in Wwe might st tempt to look at ll sca relationships childgen expécience but to look at each them in «relatively cursory way-in ox chaps to explore which of those rl whether the school redesigned to have, trying to'directly affect social developm The trade-offs involved are the dlas- l-offs between breadth and depth, wernow tum to in more depth, Breadth Versus Depth Im some ways, the differences between quantitative and qualitative methods in- volve trade-offsbotween breadthana depth. ‘Qualitative methods permit inquiry into se- lected issues in great dept with care tention to detail, context, and nuance; that data collection need not be constrained by predetermined analytical categories con- {tibutes to the poténtial breadth of qualita- tive inquiry. Quantitative instruments, én the other hand, ask standardized questions ‘dat limit responses to predetermined cate- ‘breadth and depth). This has the f making it possible to measure the reactions of many respondents toa lim ited set of questions, thus facilitating com- pPatison and statistical aggzegation of the data. By contrast, qualitative methods typi Designing Qualitative Studies F227 tally produce'a wealth of detailed dota about ser uinerof people and cases. However , the breadth. versus ‘depth tradeoff also applies: ‘is how mach time.and effort we are any siigle person’sexpe- riences, So, for example, we conld look ata arrow. range. of experiences for a larger number of people or i broader range of ex- periences foi'a smaller number of people. Take the; case-of interviews, Interviewing, ‘with an instrument that provides tespon- ai largely open-ended stimuli typi developed an open-ended for elementary students consist ing of 20 questons atin 28 “What do you like most and "What don’t you like ‘These interviews took Between hal an hour and two hours depending on students" ages sand how articulate they wore. twould cer tainly have been possible to have longer in torviews. Indeed, Ihave conducted in-depth interviews with people thatran 610 16hours, over a period of a couple of days. On the other hand, it would have heen possible to ask fewer questions, make the interviews shorter/and probe in less depth. Or consider another example with fuller range of possibilities. [tis possibleto study a single individual overan extended period of time—for example, the study, in depth, of fone week in the life of one child. This in- volves gathering detailed information about every occusrence in that child's life and ev- ‘ery interaction involving that chile during the week ofthe study. With more focus, we ‘might study several children during that228 GL QUALITATIVE;DESIGNS AND DATA COLLECTION week bitt capture fewer events, With a stil ber of issues. The extreme case would be to spend all of oir resources and time asking a single question of as many childzen as we could interview given time and resource constraints. [No rule of thumb’ exists to tell a ree searchce précisély how to focus a study. "The extent to which a research or evalua Units of Analysis A design specifies the unit or units of analysis to be studied. Decisions about sam- ples both sample size and sampling strate- mary focus of data collection willbeon what {is happening to individuals ina setting and 18 Br0ups of people in a pro- s programs involves a differ may beinterestedin comparing demographic groups (males compared with females, Whites compared with African Americans) or programmatic groups (dropouts vs. people who complete pple who do well vs people people who experience group therapy vs. people who experience in- dlividual therapy). One or more groups are selected asthe unit of ‘some important character ‘peopleinto groupsand Istic has important implications gram. ‘A different unit-of analysis involves fo- ccusing on different parts ofa program. Dif- {ecent classrooms within a school might be making the dassroom the unit of Outpatientand inpatient programs ical facility could bestudied. The in- ‘of aprogram mightbe studied sop- the service delivery part of a sparate units of analysis. Entire programs can became the unit of analysis.In State end national prograrns where there are a number of local sites, the appropriate unit variations among individuals within pro- jects. Different units of analysis are not muti ally exclusive. However, each unit of analy- qualitative methods involve observations fand description focused directly on that analysis) For example, quality assurance effort ira health or mental health prograin ight foc only en thee eal iedens teatment. A cultural study may focus on celebrations, Saimpling can also involve time period strskegics, for example, continuous and on- ‘going. observation, versus fixed-interval sampling over continuous, monitoring are that fieldivorkers experience less fat and can esllect more information at sampling interval than they could on. tinuous observation routine” Johnson and ‘important approach ganizations, and communities may function in different ways at diferent times during the year. Of course in some programs there Designing Qualitative States 15-229 the kids want to be outside, go that's not an cfective time to gather data. In Aftican vil- lages, I was the difficulties of dats collection for every ‘There ae limits to how mutch ane can ap- ply logicin trying to caleulateall of the pos- sible consequences of saimpling options, whether the decision isabout which time pe: riods to sample or which activities to ob- ‘serve. The tricks to keep coming back to the criterion of usefulness. What data collected. during what time period describing what activities will most likely illuminate the in- quity? For evaluation, what focus of inquiry ‘will be mast useful? There are no perfect evaluation designs, only moze and less use- information about. Do they want bout the different experiences of Is in programs, or do they want to tions in program processes at different sites? Or both? Such differences230 QUALITATIVE DESIGNS AND DATA COLLECTION ibe Designing Qualitative Suds 231 transhition withidiéintended evaluationué-. pling 2s one of the core distinguishing, ers. coc 2 strategicthemes of qualitative inquiry. The Exhibit 55 presents some. alternative units of analysis. Clarity about, the unit of - nales for and the details ofhovr todesign a analysis is weded to select a study sample. study based on a purposeful sample... In Chapter'2; 1 identified purposeful sam- Exainiples of Uits of Analysis for Case Studies and Comparisons 1. Purposeful Sampling - © [ete porrecatcecion —Margaet Preston 3675 leamn a great deal about issued of central Pechapé nothing better ciptares the dite aes " ference between’ quantitative and qualita. Smportance to the purpose Hon-rich eases for study in depth. Informa: tion-tich cases are those from which one can for families in. po232 \SL_QUALITATIVEDESIGNS AND DATA COLLECTION influential book Within Our Renck Stephen Covey’ (2980) best-selling book The 7 Habe lisof Highly Efetive Peopleisbased on apur- ‘posefl, extreme group sampling strategy. “Saies of leadership Kave long focused on ‘identifying the charscterstis of highly suc- cessful leaders, asin Collins's (2001) ease studies of Il corporste executives in whom “extreme personal humility blends para- doxially with intense professional (p67), what he calls “Level 5 le highest level in his model In the early days outlier cases that provided important in- sights into directions researchers should take in combating AIDS. Sometimes cases of dramatic failure offer ‘powerful lessons, The legendary UCLA bas- ‘ketball coach John Wooden won 10 national championships from 1964 through 1975, an ‘unparalleled sports achievement, But the ‘game he remembered the most and said he semifinals (quoted in the Las Angeles Tres, made thousands of trades in many com- ‘modities over a long and distinguished cx- seer, but what he reports learning the most from was a highly unusual combination of mistakes in which he last more than $1 mil- lion ina few weeks of trading soybeans—an ‘extreme but iluminative case. He reports “An excellent applied research example {is Angela Browne's (1987) study When Bat- tered Women Kil. She conducted in-depth. studies of the most extreme cases of do- ‘mesti violence to elucidate the phenome: non of battering and abuse. The extreme ‘nature of the eases is what renders thom so ‘powerful. Browne's bookisan exemplar of, oalitative inquiry using purposeful sam- pling for applied research, In evaluation, the logic of extreme case sampling is that lessons may be learned. about unusual conditions or extreme out comes thatare relevant improving more typical programs. Let's suppose that we are interested in studying a national pro~ gram with hundreds of Tecal sites. We ‘know that many programs are operating, reasonably well, even quite well, and that coher programs verge on being disasters. ‘We alsa know that most programs are do- ing "OK" This information comes from Imowledgeable sources, who have made site visits to enough programs to have a ‘basic iden about what the variation is. The ‘question is this: How should programs be sampled for the study? If one wanted to precisely document the natural variation among programs, @ random sample ‘would be appropriate, preferably a ran- dom sample of sufficient size to be truly zepresentative and permit generalizations tothe total population of programs. How- ever, some information is already aveil- able on what program variation like. The sources and limited Hime, an evaluator sighs earn more by intensively studying tone ox more examples of really poor pro- grams and one or more examples of really excellent programs. The evaluation focus, then, becomes a question of understand- ing under what conditions programs get info trouble and under what conditions ‘programs exemplify excellence. programs or excellent programs. The re- searchers and intended usersinvolved inthe study think through what eases they could ve most from and thase are the eases single program the same strategy standing selected cases of special interest, forexample, unexpected dropouts or out- standing successes In a evsluation of the CCarbbesn Agricultucal Extension Project, ‘weed casestudies ofthe “outstanding ex: tergion agen” selected by peersin exch of ‘ight Caribbean countries to help the pro- gram develop curicultsm and standacds for lepproving extension practice, The sample was purpoefully “based,” not to make the program look good, but rather234 L_ QUALITATIVE DESIGNS AND DATA COLLECTION learn, from, those who. were exemplars of | nomenon of {g00d practice, In many instances, more can the normal ditbution of participants slatsicel fers, exreme case sampling cose on itios (he endpoints ofthe bel Shaped curve nota ditibation) hat ae | often ignored in aggregate data porting. | thnomethodologats use a form of ex: iy oethng pathologies or extrem fess of loneliness fealouy, or what fever phenomenon is’ of interest. Such extreme cases might not lend themselves to of | the ruloctive process of heuristic inquiry. Oni the other hand, if the experience of the heuristic: researcher and his or her nh cneserchers gules won't be snuch to study. Thus, the researcher seeks a sampleof suffcientintenityelucidaethe “phenomenon of interest ‘shudying a deviant sample to illumi ordinary. In essence, the logic of extreme samplingjis that extreme ses may be: ‘mation-rich cases precisely because, by ng unusual, they can iluminat unusual and the typical. In proposing anex- ~ not at the extreme. ‘treme group sample, asin all purposeful '. Intensity sampling in igram evaluation. Extreme succesies of un- ‘usual failures may be discredited as being too extreme or unusual to yield useful infor: the rationale and expected | tegy as wel as tonote its trem formation-rich cases that manifest the phe- ‘Thesamestrategy canbeapplied inapro- Dasigning Qualitative Studies 295 ating a matrix in which each person in the : sample wasas diferentas possible from oth- _Geogtaphic location, mobility health status, ‘nationality, and field of endeavor. The the- a matic pattems ofachievemnent thatemerged i+ from this divetsity allowed.us toconstructa ‘mode to illuminate the primary dimensions cof and factors imsthe:award’s impact. A gram has project sites spread around the” theine song emerged from all tie scattered state, ome in rural areas, somein urban ar- _noige. That's the povrer of maximum varia~ ‘eas, and somelin suburban areas. The evalu- tion (heterogeneity) sampling, ation lacks sufficient resources torandomly Thus, when selecting:a small, of select enough project sites io generalize great diversity, the data collection and anal= sate, The evaluator ean study a high-quality, detailed descriptions of each case, which are useful for docimenting niquenestes, and (2) important shared pate tome that eu across caes and derive their significance fom having emenged out of heterogeneity Both sre important findings in qualitative inquiry. efforts statewide using a m ‘geneity sampling stat 4, Homogeneous samples. - In direet contrast to maximum variation samplings the srat- small, homogeneous sam- purpose of which is to describe articular subgroup in depth. A pro- each community wasas different from every other commiunity on: Fellowship Program, the design focused on gg, We maximized sampleariaion by er236. HL QUALITATIVE DESIGNS AND DATA COLLECTION Focus group interviews are based typi cally ‘on homogeneous’ groups. Focis ‘groups invelve open-ended interviews with ‘groups af five to eight people on specially targeted or focused issues. The use of focus - ‘groups in evaluation will be discussed at ‘groups typically involves binging together ‘people of similar backgrouncls and e ‘ences to participate in 2 group interview about aijor issues that affect them. {5 Typicdl case sanipling. “tn describing 2 ‘typical cases. These cases are selected with the cooperation of key informants, such as program staff or knowledgeable partici suring survey data, a demographic analysis fof averages, or bther statistical data that about the experiences o} ‘The sample is illustrative participants definitive. When entire programs or communities are the unit of analysis, the processes and ef {fects described for the typical program may bbe used to provide a frame of reference for case studies of "poor" or “excellent” sites ‘When the typical site sampling strategy is used, the sites specifically selected because it isnot in any major way atypical, extreme, deviant, or intensely unusual. This iMluminates key issues that snust be consid- ered invany development project aimed at that kind of village ‘In evaluation and policy research terests of decision makers will shape the grams; but the programs they ‘aly waned ore ifomatish bot were ivhlch cases are fpleal—and what crtera are being used to define typicality. 6, Critica one sampling. Crit happen anywhere’ doesn’t happen there, ‘group is having sureall he groups pact on the devi Wie wating oe os er Designing Quaiteive Studs EE 2 evidénce producedin studying single crit- leak case: Physics provides a good example ject affected the rate of Speed. at which it ‘would fall, Rather than randomly sampling, He was able to demonstrate that, under ative in looking fr thom. For exam hypnotized. ‘pose national policyniakers wait to get local knows 2 lot about talk to? By asking a numbi cal case is to evaluate the regulati ‘community @fwell educated ciizene;ifthiey can’t undetstand the regulators, ‘as youaccumulatenew inforn wes. In most programs or § ‘key namesor incidents are mu tle repeatedly Those people or ever recommended a6 valuable by a number different informants, take on special imp tance. The chain of recommended inf ally diverge initially ‘might consider the critical case to be a com- ‘munity consisting of people with quite low levels of education: "Ifthey can understand the regulations, anyone can.” Identification of eriical cases depends on. recognition of the key dimensic fora critical ense. For exemple, ‘might come froma pasticul ‘gram location. Ifthe fundets of a new pro- gram are worried about recruit participants into a program it238 L_QUALITANIVEDESIGNS AND DATA COLLECTION innovative, companies: She began by:ask ng corporate experts for candidate compa- nies to study: Nominations snowballec she broadened her.inguity:and,then con vverged into:a small number pf come cases, ominated by number of diferent expert informants 8. Criterion sempling” Thelogicoferitrion | sampling isto review and study all cass | that meet some predetermined chilerion of | importance, a strategy co asourance efforts For range of partispatio outpatient program might be 0.6 weeks ‘Allcases at exceed weeks ne reviewed to find out why the expected range was ceeded and to make sue the cage was being appropriately handled. Ora quality assur ancestandard maybe thatall ptienisenter jg hoepital emergency room, who eect ina lifethrestening situation, receive care within Thurs Cases that exceed thi stan. viewed, incidents canbe conse fete ing. Fr xainpleallncdents of lent sbideln «ogre may Se aber of | indepth evaluation in a quality nosurance effor All foriner mental health tints who commit suicide within three months of re- lease may consttuté'a sample for in-depth, (qualitative study. Ina school setting, ll st- dents who are absent 25% of more of the time may merit the in-depth attention of ‘ase stuly. The point of criterion sampling tobesure to understand cases that are to be information rich because they ‘veal major system weaknesses tha targets of opportunity for program or sys- tem improvement, Criterion sampling can add an important qualitative component toa management in- formation sys 'monitoting system, Allcasesin thedata sys- © for in-depth, qualitative analysis. Criterion: “from standardized questio tern that exhibit certain predetermined éxite= rion characteristics are routinely identified sampling;also can be used to identify cases: depth follow-up, for ekamp! ongo- ing workplace discrimination. (This sttateyy- {ean only be used where:respondents fave willingly supplied contact information ) tion sampling Is theory-based sampling, ‘The researcher samples incidents; slices of tionand selection, representative ofthe phe- ‘romenan of interest. ‘When one is studying people, programs; ‘organizations, or communities, the popula- Hon of interest can be fairly readily deter- ‘mined. Constructs, however, donot have as clenr a frame of reference: For sampling operational instances of con= structs, there is no concrete target popule- lion... Mostly therfore, se are forced to Selecton a purposive isis those particular in- stances ofa construc hat past validity stud 'ee,convantinal practice, individual inition, ‘ronsultation with ritcally minded persons suggest oe the ses comzespondencet the ‘construct intrest Altomatvely,wecan use the same procedures ta select multiple opera- ‘ional reprasentations ofeach consinuc,cho- s2n beeause they overlap in representing the citi theoatical components af the con- teuctad Bacau they differ from eachother. ‘on ilevan dimensions. his second form of Sampling i called multiple opersiomlism, and it depends more hebvily‘on iriividual uelgment than doas theandom sampling of persons from a well- designated, target popu Jaton. Yet, such judgments, while inevitable, sroess vel understood than formalsampling methods andar largely ignored by sampling experts. (Cook, Leviton, and Shadish 1985: 164) Operational construct sainpling simply means that one samples for study” real- -For example, classic diffusion of theory (Rogers 1952) predicts adopters of same innovation will rent insignificant ways. from later ters. Doing cases studies.on early and Inte adopters then, would bean example of theory-based sampling. Such samples aze often necessarily purposefully selected be- ‘cauée the population of allay. and Iate adopters may notibe known, so: random. sampling is not an option. ‘Theoretical sampling is what grounded theorists define as “sampling on the bass of theemerging concepts, with theaimbeing to explore the dimensional range or varied conditions along which the properties of concepts vary” (Strauss and Corbin 1998: 73). In grounded theory, theoretical sam- to-use the constant comparative method of analysis. The two go hand in glove, connect- Designing QuailaticeStuties 1 239 data gathered during fieldwork, The con- stant comparative method involves sys tematically examining: and refining. varia- tions in-emergent and grounded concepts ‘Variations in the concept must be sampled . to rigorously compare, and: contrast. those variations. (See Chaptars 3 and 8 for more detailed discussions of grounded theory.) 10. Confirming ana disconfirming cases. In ‘the early part of qualitative filawork, the ‘evaluator is exploring— gathering data and ‘watching for patterns to emenge, Overtime, the exploratory process gives way tocontit~ tory fieldwork. This involves testing confirming. the .importance,, and: 1g of possible patterns, and checking ‘out the viability of emergent findings with new data and additional cases. This stage of Fieldwork reguires considerable rigor and, integrity onthe partof the evaluator inlook- ing for and sampling confirming as well as Aisconfirming cases. Confirmatory cases are additional exam ples that fitalreadyemergentpatterns; these. ~ ‘cases confirm'and elaborate the findings, fit. They are source of rival interpretations as well’as 2 way of placing boundaries around confirmed findings. They may be, “exceptions that prove the rule” or excep! tions that discongirm and alter what ap- peared to be primary patterns. ‘Thesouree of questions orideas tobe con- firmed or disconfirmed may be from stake- holders or previous scholarly literature rather than the evaluator’s leldwerk. An evaluation may in part serve the purpose of confirming or disconfirming stakeholders’ or scholars’ hypotheses, these having been. Identified during early, conceptual evalua- tor-stakeholder design discussions orlitera-‘Thinking about the challenge of finding Confirming and disconfiming exses empha sizesthe relationship between samplingand reseatch conclusions. The sample deter~ ‘mined what the evaluator will have some- thing to say about—thus the importance of sampling carefully and thoughtfully. 11, Stratified purposefl sampling, Strat: ‘might stratify by socioeconomic. status ‘within a larger population so as to make ‘generalizations and statistically valid com= parisons by social class as well as to gener~ ‘Purposeful samples can also be stratified and nested by combining types of purpose- fal sampling. So, for example, one might combine typical case sampling with maxi- ‘mum heterogencity sampling by taking a stratified purposeful sample of above aver- age, average, and below average cases. This represents less than a full maxima varia- tion sample, but moze than simple typical ‘ase sampling. The purpose of stratified ‘purposeful sampleis to capture major varia~ tions rather than to identify a common core, although the latter may also emerge in the that the sample sizes are likely to be too ‘small for generalization or statistical repre 240 QUALITATIVEDESIGNS AND DATA COLLECTION lever the date lend is a primary strengthiof ‘qualitative fieldwork strategies. This:per- mits the sample to:emerge during ficld- ‘work. ‘During fieldwork, it is impossible to ob- serve everything:-Decisions must be made about what activities to observe, which peo- pple to observe and interview, and when to collect data. These decisions cannot all be made in advance, The purposeful sampling strategies discussed above provide direction for sampling but often deperid on some knowledge ofthe setting being studied. Op- pportunistic, emergent sampling takes ad- ‘vantage of whatever unfolds as it unfolds. In Chapter 2, identified emergent flexi- ble designs as one of the core strategic themes of qualitativeinguiry and cited as an ‘exemplar the anthropologist Brackette F. Williams and her fieldwork on how Ameri- ‘eans view violence in America: 1 doimpromptu interviews. Idon'thave some target umber ofinterviewsin mind or prede- termined questions. It depends onthe parson bout Kiling or about death alle opportunity sampling. ng whace the data take me, where my question akeme, (personal interview) Few qualitative studies areas fully emergent and open-ended as the fieldwork of Wil- liams. Her approach exemplifies emergent opportunity sampling. 13, Purpossfil random sampling. A purpose- their report, the direcor and staff decided to bein collecting evaluation information more systematially, Bécause they” were sleiving for individualized outcomes, they ‘ejected the notion of basing the evaluation “ntrely on a standardized pre-postistru- ‘tent. They wanted to collect case histories and do in-depth case sis of clients, but ‘hey had very limited resources and time to evoteto such data collection. Ineec, staff at each progea site, many of whom serve 2000300 families yeu fltthatthey could only do 10 or 15 detailed, in-depth clinical ace histories each year. We systomatized thekind of information that would be going, into the case histories at each program site and then setup a random procedure fr se- lecting those clients whose: case histories ‘ould be recorded in depth, thefeby'sys- able to tell lgislators that the stares ee reporting eerandory sted was comprehensive. The credibility of systematic and randomly se- Designing Qualitative Studies 24 about why certain cases, were selected for ‘study butsuchasernplestilldoes not permit Statistical generalizations. 1A simpling politically important cases. Eval ‘uation is inherently and inevitably politieal (Gee Turpin 1989; Palumbo. 1987; Patton 19875). A variation on the critical case strat- egy involves selecting (or sometimes avoid ing) a politically fe site or unit of ‘analysis. For oample, a statewide program. ‘may havea local stein the district ofa state legislator whois particularly influential. By studying carefully the program in that dis- ‘tic, evaluation data may be more likely to attract attention and get used. This pending siege Sermomact Ths ‘would clearly be unethical. Rather, sam- pling politically important easesis simply a strategy for trying toinierease the usefulness and felevance-of information where re- sources permit the study of only a limited number of cases. ‘The same politcal perspective (broadly ‘speaking) may inform case sampling in ap- ‘pli orevenbasicresearch studies. A politi- ‘cal scientist or historian might select the election year 2000 Florida vote-counting case, the Clinton impeachment effort, [Nixon's Watergate crisis, or Reagan's Iran- CContra scandal for study not only because of {gists study ofa famous suicide would likely involve some attention during sampling to the public and political importance of the242 J QUALITATIVE DESIGNS AND DATA COLLECTION probably dé most common sampling strat ‘egy-—and the least desirable, Too often eval- ‘uatots using qualitative methods think that because the sample size they can study will "be td0 small to peraut generalizations, it dod matter how casesare picked, so they ‘might as well pick ones that are easy-to ac~ ‘cess and inexpensive 6 study. While conve Information-Rich Cases Exhibit 5.6 summarizes the 15 purposeful sampling strategies discussed above, plus a 116th approach—combination or mixed pur- ‘poseful sampling. For example, an extreme _group or maximum heterogeneity approach ‘may yield an initial potential sample size: . that i stil larger than the study can handle ‘The final selection, then, may be made ran- domly~a combination approach. Thus, these approaches are not mutually exclu sive, Each approach serves a somewhat dif ferent purpose. Because esearch and evalu- ations often serve multiple purposes, more: than one qualitative sampling strategy may ‘be necessary. In long,-term fieldwork, all of these strategies may be used at some point ‘The underlying principle that is common to all these strategies is selecting informa- tion-rich cases—tases from which one can learn a great deal about matters of impor- tance and therefore worthy of in-depth study. In the process of developing the research ‘design, the evaluator or researcher is trying toconsider and anticipate the kinds of argu sons-for site selections: or individual care sanipling noed to be carefully aricua and made explici’ Moreover itis import tobe open and clear about a study's iit tions, dati, to anticipate and address cite ‘sme that nay bemade ofa particular sam- pling strategy, especialy from people who think thatthe only high-quality samples ake dom ones ee ving weighed the evidence and cone ed the alternatives, evaluators and pri= stakeholders make heir sampling de- sions, sométimes painfully, but always ‘withthe recognition that there re a perfect designs. The sampling strategy must be se- lected to fit the purpose ofthe study the r- sources available, thequestionsbeing asked, and the constraints being faced. This holds {sub for sampling strategy as wells sample 5 Sample Size proaches instead of statistical formulas. Qualitative inquiry seems to work best for ‘the matter of sample size. I get letters, Eget calls. get e-mails, Is 10a ange enough sample to achieve maxi- ‘mum variation? started out to interview 20 people for two hours each, Bot 've lost 2 people. 8 large ough, or do Thaveto find 2 more? pe Random probability 1, Simple random 8 Criterion sampling 8, Theory Designing Qualittie Stes EE 243 £Sainpling Strategies244 QUALTTATIVEDESIGNS ANDDATA COLLECTION Continued want to study just one organization but in- terview 20 people inthe organization. [s my simple siz 1 0F 20 or both? My universal, certain, and confident reply to these questions is this: "It depends.” In-depth information from part. Suppose 100 people in a pro- ‘gram tobe evaluated. Irwould be necessary to randomly sample 80-of those people fidence lovel. If there are 500 people in progratn, 217 people must be sampled sampled (7%) to achieve a 95% confidence Javelin the generalization of findings. Atthe bother exteme, if only 50 people in the program, 44 mist be randomly seanpled (88%) to achieve a 95% level of confidence. (Gee Fitz-Gibbon and Morris [1987163] fora table on determining sample size from a sgiven population.) ‘The logic of purposeful sampling is quite different. The problem is, however, that the utlity and credibility of small purposeful ‘samples are often judged on the basis ofthe logic, purpose, and recommended sample izes of probability sampling, Instead, pur- to the purpoce and ational ofthe study: Does the sampling stzategy support the study's purpose? The sample, ike all other aspects of qualitative’ inquiry, must be judged in context—the same principle thet ysis and. presentation of to our understanding of how children think by obsérving his own two children at ength guistic programming (NLP) three renowned and highly pists: Milton Erickson, Fe Designing QualintceShuice SL 245 mulated their widely followed eight princi ples for organizational excellence by stucly- {ing 62 companies, very small sampleof the ‘thousands of companies one might study. overcome and challenges not overcome, Park and her husband made systematic ob- servations throughout the years. Eminent ‘medical anthropologist Oliver. Saks. re- ‘viewed the data and determined in his pref- ace to the book that more data are available ‘on the woman in this extraordinary case study than on any other autistic human be- ing who has ever lived. Her, then, is the signed an essay to write "How longdoes the paperhaveto “Long enough to cover the as- ignment.” Shuden!: “But how many pages?” Instructor: “Enough pages 10 do justice to Lhe subject—no more no less"246 1 QUALITATIVEDESIGNS ANDDATA COLLECTION Lincoln: ana Guba (1985)" recommend sample seletion “to the point of redun- ancy... IX purposeful sampling the size of ‘he sainple is determine by informational ‘insiderations. I the: purpose is to’ maxi- nize information; the sampling is termi- nated when 10 new inforiation is forth coming from new sampled units; thus redundancy the primary eitrion” (p. 202). ‘This strategy eaves the question of sam- plesize open, another example of the emer- ‘ent natureof qualitative inguiry. ‘There r= ‘mains, However, the practical problem of how tonegotiatean evaluation budgetorget 1 dissertation committee to approve a de- sign if you don’t have sosteides of sample size. Sampling tothe point of redundancy is. fan ideal, one that works best for basic e- search, unlimited timelines, and “uncan- strained resources. ‘Thesolutionisjudgméntand negotiation. recommend signs specify pected reasonable coverage ofthe phenome- fon given the purpose of the study and Blakeholder interests‘One may add to the ‘Simple as fieldwork uafolds. ‘Orie may change the sample if information emerges thatindicates the value of change. The de- sign should be witderstood tobe flexible nd emergent Ye, atthe beginning, forplanning, and budgetary purposes, one specifies = minimum expected samplesizeand builds a the orignal sampling approach and/or size. In the end, sample size adequacy, like all, aspects of research is subject to peer review, ‘consensuel validation, and judgment. What fs crucial is that the sampling procedures ‘and decisions be fully described, explained, ‘and justified so that information users and [peer reviewershave the appropriate context for judging the sample. The researcher or evaluator is obligated to; discuss how the sample affected the findings, the strengths, and” weakiesses of the Sampling "proce- dures, ahd any other design decisions that are relevant for interpreting and ‘under staring the reported -sésults. Exercising, care not to overgeneralize from purposeful samples, while maximizing to the fullsthe advantages of in-depth purposeful, sam. pling, will do much to, alleviate conceins about small sample size, 45 Emergent Designs and Protection of Human Subjects Emergent designs pose special institutional review boards (1 with approving reseazch protection of human subjects, Such bosrds typically wantto know, in advanee offield- ‘work, who will be interviewed and the pre- cise questions that willbe asked, Ifthe topic is fairly innocuous and the general line of ‘questioning relatively unobtrusive, an IRB 1y be willing to approve the framework of an emergent design with sample questions included, but without full sample specifica- tion and a formal interview instfument. ‘Another approach is to ask for approval {in stages. This means initially asking for approval for the general framework of the inguiry and specifically for the first ex- ploratory sage of fieldwork, including pro- ‘cedures for assuring confidentiality and in- formed consent, then returning periodically (eg. quazterly or annually) to update the design and its approval. ‘This is cumber- some or both the researcher and the IRB, but itis a way of meeting IRB mandates and sti lementing an emergent design. This jainly with iff and/or partici pnts and therefore eannot specify the full desgn at te beguning of the peta, process. “Methodological Mixes, ‘A study-may employ more than. on ssim- pling strategy, Itmay also inchide multiple ‘Triangulation: g Designing Qualitatice Stier SL 247 typesofdata, The chapters on interviewing, ‘of using multiple methods in research and ‘evaluation, ‘he method srust fllow the question ‘Campbell, many decades ago, promoted the concept of tiangulation—hat every method has is lintations, and multiple method are usually needed, Gene V. Glass eulogiing pioneesing methodologist Donald‘. Cainpbell quoted in Tashakkor and Ted (199822) Triangulation strengthens. a ‘study’ by combining methods. This can mean sing several kinds of methods or daa, including sing both quantitative and qualitative ap- proaches, Denzin (1978) z basic types of triangulation Infor, he use ofa variety locates you somewhere along a line in a di rection from the landmark, whereas with ‘wo landmarks (and your own position be- ing.the third point ofthe triangle) you can take bearings in two directions and locate ‘yourself at their intersection (Fielding and Fielding 1986:23). The term triangulation also ‘works metaphorically to call to mind the ‘World's strongest geometric shape—the tr- angle (eg, the orm used toconstruct geade- sicdomes la Buckminster Fuller), Thelogic, oftriangulation isbased on thepremisethat no single method ever adequately solves the problom of val causal fet. Becruse each rmathod reveals diferent aspects of empisicl, realty, multiple methods of observations rust be employed. This is termed trangula: tion. now afer as final methodologies rule the principle that multiple methods should be used inevery investigation (Denzin 1978b28) ‘Triangulation is ideal. It can also be ex- pensive. A study’s limited budget and time frame willaffectthe amount of triangulation, {hati practical, as will politcal constraints 1 important strategy for inquiry is y multiple methods, measures, x=- searchers, and perspectives— but to do so reasonably and practically2s ted t the thorough study of a reearch prob- lem, method secondary to the research question tsefand theunderlying worldview hardy enters thepicure exceptin ie mast ab- stract sense. (Tnehaleor and Taddio1 9822) A rich vatiety of methodological combi- nations can be employed to illuminate an inguiry question. terviewing, observation, and document analysis. Others rely more on interviews than observations, and vice versa. Studies ‘that use only one method are more vulnera~ ble toerrors linked to thatparticular method (e4, loaded interview questions, biased o ‘untrue responses) than studies that use ml- tiple methods in which different types of data provide cross-data validity checks. ‘Using multiple methods allows inquiry into | research question with “an arsenal of ‘methods that have nonoverlapping weak- strengths” (Brewer and Hunter 1989-17). ‘However, a common misunderstanding ‘may yield somewhat differen «cause different types of in to different real-world nus derstanding inconsistencies in findings ‘across different kinds of data can be illuminative. Finding such inconsistencies ‘ought notbe viewed as weakening the credi .sofferingoppor- ship between ing phenomenon und ‘THangulation within a qualitative in; [QUALITATIVE DISIGNS AND DATA COLLECTION ‘or examining how compéting theoretical perspectives inform’ a particular analysis (eg, the transcendental phenomenology of Huseerl vs. the hermeneutic phenomenal ogy of Heidegger). A study carvalso be de- signed to cut across inguiry approaches and ‘achieve triangulation by combining qualita- tive and quantitative methods; a strategy discussed and llistrated in thenext secticn, Mixing Data, Design, and Analysis Approaches Borrowing and combining distinct ele- ‘ments feom pure or coherent methodologi- calstrategies can generate creative mixed in Guiry strategies that illustrate variations on. the theme of triangulation. We begin by distinguishing measurement, design, and quasi-experimental) designs a calanalys ‘Measurement, design, and analysis alter- Designing Quottne Stes 249 Triangulation approach to research, it will help to examine alternative design possibilities for program evaluation, The examples low have been constructed under cial constraint that only one kind surement, design, and used in ench case, In pra possible mixes are much ‘cause any given study could inclide several ‘measurement approaches) varying design approaches, and varying diferent analytical _spproaches to achieve triangulation. style, high druguse), and whoarelikely an- didates for: delinquency (alienated. from dominant societal values, running with.a “bad” cxoved, and angry). The program con- sists of experiential education internships through which these individual tutoring. job placements that come while gaining work exposure, and ‘participation in peer group discussions aimed at changing health values, establish- ing a positive peer cultute, and increasing social integsation, Several evaluation ap- proaches are possible. PURE HYPOTHETICAL-DEDUCTIVE APPROACH TO EVALUATION: EXPERIMENTAL DESIGN, (QUANTITATIVE DATA, AND STATISTICAL ANALYSIS ‘The progam doesnot have suficent re ‘Sins Wicd Gunn Foon buch Coan250 QUALITATIVE DESIGNS ANDDATA COLLECTION sie with ads nto the program ona random basis and the remaining group r= Ceives no imediat teatmentinterventon. Bef fh Pog tains al one, yee liter all youth, both those in thé program tnd thovein the conto group, at adminis tered standatgiaed inotraments measuring Schoo! achievement, see! Sianation, and locus of sontol, Rates of Schou atendance ness, dug use,and de- Tnguancy are cbiained for each group. ‘When al datahayebeenevletd atthe end ofthe year comparsonsbetween theconiol Shd experimental groups areiade wsingin- ferential statistics. ‘PURE QUALITATIVE STRATEGY: NATURALISTIC INQUIRY, QUALITATIVE DATA, AND CONTENT ANALYSIS Procedures fr resting and selecting participants forthe program are determin tntirely by the staff. The evaluator finds =| in-depth in- convenient time to condi ‘hey are admitied into the pr students to describe what schot cally spend their time, how they appr their views about health, and their behav- jogs/attitudes with regard to delinquent and eriminal activity. In brief, participants social world. The gram activities, and youth rections, Th portunities for adit ‘views with participants to find view the program, what tencas they are having, and what they're do- ing. Near the'end of the program, in-depth interviews fare coneucted-with the partic ‘pants toleamyhatbehaviorshavedianged, theirexpecr identify the patterns of experiences partici- ppanis bring to the progeam what pateins ‘characterize their participation in the-pro- gram, and what pattems of change are ported by and observed in the participants. ‘MIXED FORM: EXPERIMENTAL DESIGN, QUALITATIVE DATA, AND CONTENT ANALYSIS As inthe pur xpi om, poten tal patients are randomly assigned to testa end conte rps dep terviewsaveconducte wi all youth bork thesein the tentmentgroupandthosein he contol group and both before he prog Begins and the end fhe progeam, Cone tentand themati anaes are performed 20 ‘thal the control and experimental group pat- temacanbe compared contated, (Fors Acted eample combing expeisiental fonols and snogiaphy, see Mawel, ‘Bashook, and Sandlow 1987.) ‘MIXED FORM: EXPERIMENTAL. DESIGN, QUALITATIVE DATA, AND STATISTICAL ANALYSIS Participants are randomly assigned to treatment and control groups, and in-depth interviews are conducted both before the ‘progiamandatitsend. These interview data, in raw form, are then given to a panel of judges, who rate each interview along sev= ood f sce n.school on = igh = 20) likelihood. ductive work self-esteem, and manifesta- tion of desired nutritional and health habits, Inferential statistics are then used 10 eom- ‘pare these two groups. Judges make the rat- ings. without knowlédge of which parti. pants were in which group, Outcomes ted scales are also staistically-related to background characteristics of participants. MIXED FORM: NATURALISTIC INQUIRY, QUALITATIVE DATA, AND STATISTICAL ANALYSIS. Asin the pure qualitative form, ‘are selected for the program on the whatever ctiteria staff inembera apply. In-depth interdicws ace conducted withall students before and at the end ofthe rogram: These data ae then submitted toa panel of judges, who at themn on aseriés of Aliriensiohssinllarto those listed in the pre- vious example. Change scoces are compited for eachindividual anid changes are statist- cally elated to backgudund characteristics of the students to determine ina gression fonmat which ciaracteristics of stents are ely to predic success in the program. In dition, chservat ae rated on a extent to which student as high ot low, the extent 0 «tions were formal orinformal, and the extent to which participants had input into pro- gram activities. Quantitative ratings of ac- tivities based on qualitative desri then aggregted to provide an overview of asin a ies 251 MIXED FORM: NATURALISTIC. INQUIRY, QUANTITATIVE DATA,’ AND STATISTICAL ANALYSIS. = ted Jor the program cording tostatferitarin. The evaluator enters ig without any predeter- mined categories of analysis 0 tions about important variable relationships. The evaluator observes im- pportantactivitis and evantsin the progiam, looking for the types ofbehaviors and inter- actions that will emerge. For each significant ‘ypeafbehavior or interaction observed, the evaluator creates a category and then ises & ‘ime and space sampling design to fount the requency with which those éategories of behavior and interaction are exhibited. The frequency of the rhanifestation of observed ‘behaviors and interactions are then statisti- cally elated toch characteristics as group size, duration of the activity, staf-student ratios, and social/physical density. Alternative Pure anid Mixed Strategies ‘hibit 7 sumunatizesthesixalternative design scenarios wwe created and just ce- ‘viewed for evaluation of "Operation Reach- Out” As these altemative designs illustrate, purity of approach is only one option. In- ering the case for maintaining the integrity and purty of qualitative and quantitative paradigms. The 12 themes of qualitative inquiry described in the second chapter (EX- hibit 21) do fit together as of naturalistic inquiry mesh well with ‘openness and depth of qualitative data,1 252 {QUALITATIVE DESIGNS AND DATA COLLECTION Measurement, Design, and Analysis: Pure and Mixed Combinations & ‘Natur f ea, flute | pee fee. oe ji ‘ia Jerre > ie pa 0 Sm toulyen ee case Genuine openness flows naturally from an inductive approach to analysis, pasticlarly ability of hoiétie understanding of unique human settings. ‘Likewise, iere i sa intel consistency findings. Guba and Lincoln (1988) have argued that the internal consistency and logic of each approach, or paradigm, mitigates against methodological mixing of different inquiry modes and data collection strote- ‘ses Their cautions are not to be dismissed, gh Mixing pars ifeestapposches is a matter of philosophical and ological controversy: Yet, the practical date in evaluation (Patton 1981) to gather the most relevant possible information for evaluation users outweighs concerns about. ‘methodological purity based on episte- ‘mological end philosophical arguments. The intellectual mandate tobe open to the world has to offer surely includes me odological openness. In practice, it is alto- gether possible, 2s wehaveseen, to combine ely (Patton more sophisticated and multifunctional designs, Advocates of methodological purity ar- gue that a single evaluator cantiot be both dleciuctiveand inductive at the same time, or cannot he testing predetermined hypotieses ppossible to research pred tions and test hypotheses and naturalistic pursuing: 4 program. In principle, this is no! different from a questionnaire that both fixed-choice and open-ended ques- tions. The extent to which a qualitative ap- roach is inductive or deduétive varies along acontinuum. As evaluation eldvvotk begins, the evaluator may be open to what. lever emerges from the data, a discovery or inductive approsch, Then, asthe inquiry e- veals patterns and major ditnensions of in- terest, the evaluator will bégin to focus on verifying and elucidating what appears to be emerging—a more dediuetvely oriented approach to data collection and analysis ‘Theextent towhicha study is naturalistic in design is alsoa matter of degree. This ap: plies particularly with gard to theextent to ‘which the investigator places conceptual constraints on or makes presuppositions ‘emerged from those more open-en¢ riences, sometimes. even manipulating something to see what happens. ‘experimental designs. Pure experiments are Designing QullutveStuaies 258 the ideal; quasi-experimental designs often represent what is possible and practical “Likewise, full participant observation over ‘an extended period af time's thequalitative ideal. In practice,-inany acceptable and ‘meaningful variations to qualitativeinguiry : canbe designed. ‘This spirit of adaptability and creativity. in designing studies is aimed at being prag- ‘matic, responsive to real-world conditions and, when doing evaluations, to meeting stakeholder information needs. Mixed ‘methods and strategies allow creative re- search adaptations to particular settings and questions, though certain designs pose con- straints that exclude other possibilities. I is not possible, for example, for a program to ‘operate as an experiment by assigning par tidipants to treatment and control groups while atthe same time operating the pro- gram under naturalistic inquiry conditions im which all eligible participants enter the ‘program (and thus there is no contre group and no random assignenent). Another in- ‘compatibility: Qualitative description’ can bbe converted into quantitative scales for ‘Purposes of statistical analysis, but it s-Aot possible to work the other way around and convert purely quantitative measures into detailed, quabitative deseriptions. Design and Methods Decisions Which research design is best? Which able, the politcal context, and the inte abilities biases ofthe researchers. Exhibit 5.81254 QUALITATIVE DRSIGNS AND DATA COLLECTION ‘exploratory sus open timelines summarizes the issues discussed in this chapter that must be addressed in designing aarudy. ‘In qualitative inquiry the problem of de- sign poses a paradox. The term design ug- at deg as very specie Duet rete nature see. ans plang for cvtsin broad coningeneis without weve indicating exacly what vil be mein eatin teach (ncn and Guba 1985226). A qualitative design needs to re- © main suificentiy open and flexibleto permit exploration of whatever the, phenomenon under study offers for inquiry. Qualitative designs continue to be entgent even alter data lletion begins The degree of exibil- ity and openness is, however, also a matte: ‘of great variation among desigas. What is certain is that different methods can produce quite different findings. The challenge isto Figure out which design and rebhodis are most appropriate, productive, and useful ina given situation, Martin Trow (2970) points out (quite nicely I think) the ference between arguments about wich ‘methods are most spproprate for studying | particular problem and arguments about ‘heintrinsicand universalsuperiarity ofone ‘method over another: Every cobbler thinks leather is he only thing ‘Most social scientists, including the present Designing Quaitetie States W255 Ihave thee favorite tesencch methods hich they ae familia and have some sing. And [suspect we mat choose 'o investigate problems hat seem vainerable to'attack dhrough these methods. ‘But we. should atleast ty tobe less parolial than set on with the business of attacking our problems withthe widestarray of conceptual tnd methodological tools thatwe possess and they demand, This doesnot preclude dlscus- sion and debste regarding he relative usefle ness of aillrent methods for the study of256 G_ QUALITATIVE DESIGNS AND DATA COLLECTION Professor, for the last four years I’ve been struggling to define my research problem, but it "The evaluation sage, Fialcalm, might jggest that all to often the methods deci animals together for his momentous ‘announcement. °"T have studied the matter at great ‘which, due to your past prejudices and lack of experience, you have ignored Then you 0 monn Guns Pon a Mel Cnn Sophisticated emergent design strategy Designing Qualitative Studies 257 2 EC very path we take leads to fantasies about the path not taken, Hialeolm willbe able to itake the sam rational deci- sion that I have made “Honey comes conveniently packaged in beautifully shaped prisms of the most deli- catetexture.I'sready to eat alidesdown the throat ever so easily, is a highly nutritious source of energy, digests smoothly, and Honey is readily available and requires no speciaf labor to produce since bees do all the ity make ta food beyond compare. ready to consume—no peeling, no killing, no same conscious decision thavemade. Ihave chosen to like haney.”