Dre kheth, pe yew (ZF) 3 Dilemmatics The Study of Research Choices and Dilemmas Joseph E. McGrath 0 Theresearch process cam be viewed as _ series of imterlocking choles, in which we y simultaneously ‘to maximize several conflicting desiderata. Viewed in that wa) the research processisto beregarded nots aset of problems be “solved,” but rather asa set of dilemmas to be “lived with”; and the series of interlocking choices isto be regarded not asa attempt to find the “right choices but as aneffort to keep from ‘becoming impaled on one or another horn of one or more of these dilemmas ‘From this perspective —from this “dlemmatic view of the research process”—a proper starting place fora discussion of methodology is: (@) to lay out the series of generic choice (0) to describe those choices in “dilemmatic™ tert ‘that i, in terms ofthe mutually incompatible goals involved, and in terms ofthe dilemmatic consequences involved in any of the available choice; and then (c) to discuss what the belea- ‘ered researcher can do, feel ed iy Toc eeped nth oa ‘mei ae cen en hr ‘ery of tons ad wth wrk some one Mar ad, ‘eset enh tcf manson onan70 Iudgment Call in Reach “The upshot of such a view of research is of course, rather unpolyanna. Not ony is there no one true method, o correct set of methodological choices that wll guarantee sucess there i not even a “best” strategy or set of choices for a given, problem, setting, and available set of resources. In fac, from the dilemmatic point of view, all research strategies ond ‘methods are seriously flawed, often with theie very stengthsin ‘epard to one desideratum functioning a serious weaknestesin tegard to other, equally important, goals. Indec ‘possible, in principle, 1 do “good” (that is, methodologically sound) research. And, of course, todo good research, in prac- tic, is evén harder than that (We area very long way from ‘converting “dilemmatis”into“ilemmetrcs" much es intoa full-fledged “dilemmatology." And there is no “dilemmagic™ ‘that will make the problems go away!) ‘A first confrontation with the dilemmatic view of research often leaves one very pessimistic, nat only about the state of the art of the fel, but also about the value ofthat particular field Dilemmatis is certainly not a polyanna philosophy. 1 is ‘extremely skeptical, though it need not be cynical. Lregarditas realistic, rather than pessimistic. Ise no merit at allinpretend- ing that our methods can deliver what we wish they could but ‘know they cannot, namely, to provide noncontingen certainty Lnperturbed by the methods of study and unperturbed over timel Perhaps someone might wantto make cas for ryingto {ool sponsors, agencies, or clients, in regard to what our ‘methods can and cannot do, But there is no rationale at believe, for trying to fool ourselves in these matters. This pint leads directly oa statement ofthe First and Second Rules of Dilernmatcs: RULE I: Always foce your methodological problems squarely: ‘or, Never turn your back on a Horned: Dilemma. RULE Ik: A wie researcher never rest of ‘That uel you ae about to st on may tun out tobe ‘an unecognized horn of another methodologies! Siemon. Dilemmas 78 oat Far 34 Te Ce of El ewe STRATEGIES, DESIGNS, AND METHODS AS STAGES OF THE RESEARCH PROCESS. We can regard the research process a8 a series of logically cordered—though chronologically chaotic—choices. Those ‘choices un from formulation ofthe problem, through design and execution ofa study, through analysis of results and their inerpretation. The series of choices s locally directional: Pan ‘must come before execution; data collection must come before ‘data analysis, Bu the set of choices i ystemicaly circular: starts witha problem, and gets back tothe problem. The end result ofthe process, however, never arrives back atthe exact, farting point, even if all goes well. So, the process really720. ude Call Research ‘should be regarded asaseres of spirals, rather than ssa closed circle, Figure 3.1 illustrates this, and ssa frame forthe rest of this materia ‘The labeling of the figure suggests that we can divide that G they offer a classification of designs into ), tre experimental (3), and quasi-experi- mental (17). Second, they describe four-tinde"of validity— internal, statistical conclusion, construct, and external Third, they provide alist of major classes of threats to each of those types of validity: that is, they ist classes of plausible rival hypotheses. Pu together, these constitute a23 (designs) by 32 (classes of threats to val ithfour types of validity) ‘Campbellian matrix, that represents a definitive treatment of, research design at this level ofan with the Campbellian 23x 32 mati the rest ofthis material is assumed throughout ‘Another, familiar distinction, elated to the Campbellian lassfication of designs, isthe distinction between “experimen- tal" and “correlational” studies (se Cronbach, 1957). Theis, refers to designs that compare average values of different ‘aches of eases (relative to variation in values within each batch) on some autribute. The second refers to designs that ‘examine the covariation ofthe values of two or more aributes, among the cases of a single batch. Actually, the two are ul mately convertible, one to the othe, since they are each special ‘cases of the same, underlying Baconian logicoftelations be ‘goes with Y!, invariably, then, X and Y are related. If Thave ‘reason to believe that X comes first, and/or that Xis not af fected by Y, and that all other pertinent matters ae aken into nated), then Ian infer that X led to Y, ‘and/or suficient condition for Y, of that X “causes” . Correlational designs (assuming selatively sophisticated ‘ones, ofcourse) are very good for finding out the functional82 © Judgment Cale in Research form ofthe X-Y relation (eg linea, curvilinear; for specity- ing value mapping between X and Y (how many units of increate in X will yeld one unit of increase in Y7}; and for determining the degree of predictability of ¥ from X (ie, the size of the correlation). But correlational designs are bhont instruments, relatively speaking, for interpreting the causal direction, f any, ofthe X-Y relation. Experimental orcompari- son designs have precisely the opposite virtues and weaknesses. ‘Good experimental designs ate excelent for examining the causal nexus by which X and Y are connected. But they are seldom ureful in assessing the functional form, or the value mappings of X on Y, contingent information about degree of predictability of Y from X. This set of contrasts points tosome major dilemmasin the design domain, ones that canbe examined beter if we first cr clasfications at a more microleve. AA single observation isnot science. All research requires multiple observations, though not necessarily multiple “cases.” Case studies use only one popula tion unit—one “A” unit inthe symbolsystem we ae using ere. Bat they involve extensive observation of tha one ate ‘AI research requires some form of aggregation of those ns. So-called “qualitative” studies involve multiple ‘observations are then mapped via numbers into magnitudes, fotder relations, of frequencies. And those observations must be ageregated in some way before they can be interpreted — ‘whether oF not that aggregation is done on some simple and ‘explicit bass, such as an average, or on some complex and Dilemmatics Oa ‘more implicit bass, suchas pattern expressed in words rather than in numbers. ‘The researcher isalways and continually faced with deciding how to aggregate such multiple observations. While itis ter ally true that no two observations are identical, the researcher ‘must decide which sets of two or more observations are to be {tealed ar ifthe observations in the set wee all alike (hati, to decide which observations will be treated as “replications”; and which two or more sets of observation are tobe treated as if the sets were different (hat is, which sets willbe treated ax “partitions” across which comparisons can be made). Ta correlating X with Y, fr example, the researcher decides to treat each individual “case” as different; but at the same time, decides that each individual was the “same” person when data were collected for attribute X and when data were col lected for attribute Yeven if those observational events ‘occurred years apart, and evenifthose events varied greatly, in time and content, from one case tothe next In comparison studies, the investigator decides that all eases within a given condition are to be considered “the same” replications—not literally tue, of course—and that te differ- cent sets of observations defined by the different combinations ‘of eaperimental conditions will be treated as diflerent—a5, ‘meaningful partiions ofthe dataset. Thee same and differ fence decisions, these replication and partitioning choices, go ‘on at several levels. They occu, for example, when we agare- ‘ate items within 2 single test If we score a 30item test by calculating number correct, varying from 0 to 30, we are implicitly treating the 30 items as replications. If, instead, we identity two factors (by whatever means theoretical orempit cal, one with 19 items and the other with 11 items, and compute two separate scores, we are partitioning that set of 30, ‘observations into wo batches that we will treat as different#4 Judgment Cals in Research (although we are sil rating all 19 items in one batch and all 1 items in the other as replications). We also make such replication and partitioning decisions when we agarczate over trials or observation periods. We decide which tme-ordered ‘sets of observations belong together, and which should be batched separately, and in doing so we are deciding, in effect, which time periods contain meaningfully different situa or chunks of behavior. ‘What is important to note here is that such same and differ. cent decisions are arburary and tentarive. They are arbitrary because any two observ it and different in others, and iis up tothe investigator vo decide which ofthese “respects are to be focused on. They ae ent tive, oF should be regarded as tentative, because i is often useful to take cases treated alike for one purpose and later partition among them for another purpose, and vice vers, ‘Analysis of variance and covariance makes this point wel. If ‘one does an analysis of variance, one treats cases within a cell ‘a “alike” If one then adds a covariance analysis, one would be, in effect, partitioning the case within each cell on that covariate. The reverse change can also be illustated from ANOVA, When categories of one oF more factors, or their interactions, show no differences, itis often usefalto combine them—thus treating as “same” what had previously been leeated as diferent. Such replication and partitioning deci- sions are the processes underlying many other decisions within the research process. Some ofthese will be examined next. UNCERTAINTY, NOSE, INFORMATION, AND I is worth examining research design decisions at ‘another, micro, level. This level has to do with how one actu- ally deais withthe various ateibutes or properties ofthe events fone wants to study. Thee questions are pertinent here: (1) Dilemmaticr O85 ‘What properties (variables) ae relevant to my problem? (2) ‘What should I do in regard o those properties with which lam, ‘most concerned? (3) What should | do about all the rest? What are the properties? In regard to the fist question: All properties of the events being studied that can or might vary ‘from one observation to another, aretbe proper subject of your ‘concer. That means all ofthe properties on which you could partition the set of observations—and that isaninfiniteset, or, for practical purposes, might as wel be. Each of these potential properties of the set of events can vary—that is each can take any one of two oF more values or level or sates, (Inthe ase of ‘a continuous" variable, we can regard itashavinga very large ‘number oflevels with very small differences between levels. AS ‘an aside, so-called “qualitative” data can take on only one of two “values” in regard to any one property: “present” or “absent or, more accurately, “observed” or“not observed." In all other regard, they are like any other observations of any property of an event) I we consider a “problem” or a “set of observations” as having a certain numberof relevant properties, P (1,2, 3, Dj; and if we regard each of them as potentially taking on any ‘one of some specific number of different levels or vas, V (with V being potentially a different number for each ofthe properties), then: The (otal numberof posible combinations ‘of values of properties that can occut—thats the total number of “diferent” events that can oceur—is given by: (V.)(V) (V3) (9 } If we simplify, by imagining that all properties have the Same number of posible different levels, that expression becomes (V?). For most problems, where pis substantial, (V") is a very large number, even if Vis only two. Research as dealing with information, noise, and uncer- ‘ay. Ifany given event or observationcantake onany one of86 © Judgment Cll in Research the (VP) values, that expressionis a measure ofthe Uncertainty in, orth Potential Information in, that problem. V?= U.I'we reduce that uncertainty (potential information) inthe “pob- lem by doing» study tat establishes relation between the tccurence of various values of X andthe co-occurresce of predictable values of, there is reduction in uncertainty “There are now fewer possible combinations of events tha can occur, That reduction in uncertainty, from (V?) 10 (VF })(a Substantial amount if Pislarge) ia statement of he Informs tion Ved of that study. ‘On the other hand, if we reduce (V?) by “eliminating” variables—by experimental contol, for example—we redece {he potential formation nour set observations, but we do not redace the uncertainty inthe real-work™ problem. When ‘we do this we then can find out about es. Tati, thee is kee {otetal information fa our set of observations, because we fave cit the scape (in the hope of gaining precision) and {hereby left some ofthe potential information ofthe problem outside the cope of our study. IK -we reducr the amount of potential information (¥") within our observations by allowing some properties vo vary bot ignoring them (hati, by not trying to ontrl them and ‘ot mearuring then), thai amount of potential information Wil fonction as noise” and it wl confound any “signal that we might have detected (Such as the XY relation we ae Investigating). This, oo, doesnot yield information; but rather ‘teonfound what aformaton could have bea learved. More wil be sid about these matters ate. In what ways can Treat the properties of most interest? In regard tothe second question asked at the start ofthis section: there are four things You can do in regard to any one property that is of interest in your study: (0) You canta particular property vr fey ast wilinaatore soo speak, but meature what valu in ake in each insane, Dienmais 047 “This called Tieetment Y here, anditie what one mua do ia segatd to one's dependent variable). (2) You ean scet caer inlade i the set events —or other- wine arrange the conditions of ebscrvation—so that all cases have the some (and predetermined) value on some particular propenty. This called TreamentK((or constant) here, andit ie what we mean when welll about holding someting con- slant or experimentally (as oppose o statistically) coating (2) You can deiberately cause one value ofthe propery to occur {for one wobec ofthe ses of observations and diferent (bt cqualy predetermined) value of that property to cur for “nother subset of those observations. Thi willbe called Treat ‘ment X, and iis what we mean when we alk about "manip lating” an independent variable (0 Ys can divide cases into two (or mor) subsets such a way thatthe owo sets are made equal on the average hough vary ing within se) on a patcslar property. This wll be called Treament M (or "matching? I ofcourse ca be doe fr sore th. two sets (as can Treatment), and for more than ‘one property (scan Treatment K, Treaiment X and Trea tment Y) alto can be done for both mean and variance (or, for that mater, for ay other parameter of the distribution of that property). But notice that Treatment M requires a prior {Treatment ¥ (vary and measee) on the matching property, and i requies a prior division into subset (a prttioning) on {he basis of some other property thanthose being matched on, for Treatment X) “These four treatments provide different things regarding “replication” and “partitioning” (se Figure 3.3). For Treat- sme K, the value of the properties is "same" for allcases within subset, and ako is the same for the different subsets. For ‘Treatment X, the value ofthe property isthe same forall cases of each subset, but difers—deliberately, and ina way known in ‘advance—from one subset to another. For M, while the aver- ‘age value is made equal for various subsets, the Individual88 © Judgment Calin Research values cam and will vary among the cases within subsets. For Y, ‘values of the property can and will vary among cases within each subset, and the average value can and pechaps wil difer between subses. (The later is often the question you are studying.) What can be dane about the other properties? Given avery large number of potentially relevant properties and limited resources, one can only provide Treatments X, K, Y,and M for ‘relatively small number of those properties. What can be ‘done about all the others? There are four ways to ‘other propertice”—all those that have not been specifically siven ¥ (meaturement), X (manipulation), K (held constant), ‘or M (matching) treatments, Those four ways to teat prope tiesin-general parallel the four treatments of specific propes ties, They are shown in Figure 3.4 "Note that in all feur ofthe specific treatments ofa property you end up knowing either the value of that property that ‘Secured foreach event or the average value that occurred for each subset. In the four “general” treatments of “all other” properties, you end up not knowing what values occurred in ‘any case oF in any subset of cases. “Two ofthe four general treatments are especially notable for present purposes. One is Treatment Z. which lets all ofthe ‘other properties vary freely, but ignores them. Unlike the other {reatmens, all of which offer advantages and disadvantages, Treatment Zis always bad, and is bad in all respects. Wis a ‘unmitigated bane. Note, also, that Treatment Z:is the general cate analog of Treatment Y, measurement, and ¥ isthe nearest thing we have 10 an unmitigated blessing (except for cost). ‘The other notable general treatment is Treatment R (for Randomization). involve assigning cases to subsets (defined by one or mote X-tteated or manipulated properties) on a random basis, Treatment R is the zine qua non for a “irve “experiment.” But iis by no means an unmixed blessing, much Dieses 088 pect Te Fos St Ng rom Ae Sein aie ind ‘wake omen hin ato Mee ‘eokeoon sng eet tac ni} ieee oo ‘Fewe34 Comptes Mode fo Tenant of Vat SOURCE sont tad Meet 1972 less a panacea forall research design problems, Randomiza- tion is crucial, and powerful, but, inspite ofits very good press, itis not Dilemmogic Indeed, it isat the core of some dilemmas, ‘as we will se later in this section. Randomization has at least four major weaknesses: (1) Wecannot always be api for tenia, practical, and/or chica reason (ee Cook and Campbell, 197).50 Judgment Cals in Reser © Wiens foe ra one ‘aly far on pone ce Canpbel and San there are several classes of threats to internal ality ae ‘ated by randomization ee Cook an Camp, 199) (0) Wie randomisation maker“ diferente met ey ‘ale, diferenes (on yon oper) eee unt ‘her hich er were randomly mnede yn eae urate iene oma oe Propey abd oe er iy shold mt expect mo ifeen ewe bse ‘very poi ropes een ou have eged tendon (0) Wile randomtnton incre the chances of bing Aiterene” between abc nach ofthe“ pein {able garneeshving aloof eas wih each ‘ube om ech an rey one of hone rei Tf th ton iy outcome tht each propery wil ey wl Sin ach sae ds nt wt (aking diere ‘Spiess int scrote ta css ihn tbe oe tobe treated a fie for aay pps od ht vr ion within abet uncom stndom ee c-aieSi tn ofthese poprishae any floss on the XY clan icing examin ten Trait R lla to inset (line tothe XY “sign andthe to ec te Shanes of eecing the NY nna even ict “ere ‘These point bring to the fore some of the dilemmas of research design. Firs, there is the R dilemma. Randomization both a cure and acurse,a bane anda boon. Onthe onc hand, randomization is costly, does not help reduce certain major ‘threats to internal validity, often posee practical and ethical ‘problems, does not guarantee comparability, and does guaran tcc high within subset variability (e., noise) On the other not disentangle causal connections for theX-Y relation, ‘The other treatment operations are also dlemmatic in their effects. The treatment operations (X and K) that give you the Diemmatics O° 9 ‘most logical leverage regarding the X-Y relation, by cutting. ‘down “noise,” are the very operations that it the scope of the ‘question, +0 that resulting information is very much con- Strained. (This has to-do with one aspect of externa validity.) So, theresa trade-off between scope (the amount of potential information in the problem) and precision (the amount of, ‘edvetion of noite). On the other hand, the treatment opera- tions (¥ and R) that allow broader generalization from results fare the very ones that incorporate much “noise into the infor mation that is contained i the st of observation, making it hard to detect *signa ft is there. This is another trade-off between scope (amount of information in the problem) and precision amount of noite inthe information). Together these pose the researcher withthe following choices: You can reduce noise, by cutting teope, so you can Fearn more about less. Or, you can leave scope broad, by accepting noise along with Signal, in which case you can learn less about more, AC the limit, i you constrain scope by manipulating and controling, more and more variables, there will be no potential informa- tion It, and you wil then be able to learn everything about nothing. That it thecase, infact, forthe esearch strategy called ‘Computer Simulations, where there are lot of X and K treat- iments, and some R treatments, but no Y treatments—thus 0 potential information at al. At the other limit, you can con- ‘rain less and ess—eschewing Xand K and M treatments—(0 maximize scope (and noise), using Y for some variables of interest and Z for all the rest. Here, at the limit, you can learn ‘or nothing about everything. This is more or less what happens in that research called Field Studies "The latter dilemma information versus noise, or scope versus precision—is one instance of a very pervasive set of, dilemmas within the research process. The general class of dilemmas can be characterized as standardization versus gen- ceralizablty (which i realy replication versus partitioning in "There isa direct conflict between maximizing (wo92 © udemen Calls in Reserch desierat. (a) Om the one hand Standardization of rclevan conditions (ime af ey color trays of made of mentee ad forth-becaue from such standadration we bape to precision by reducing nose (ve, redacng voiaos win el among cases Wate alte (0) On the ther and itis ually desirable to maximize the range of conditions ver Stich lation hasbeen ested by aring"evnt cond om acing toe of ence, te te ne from such heerogeney we hope to gan incensed feneracabiny wept fo those ang proper {hereby mun igen coniene inthe brendan Yous nest of the XY reaion we are asering There is another dicmn that was suggisted bu not cmphaied cali inthis eton on design as to do with eiding how many diferent combinations of condos ato te compared, and how many ester arto be obained foreach combination of conditions. Again, there adres cont between tw desea. On the one band is eways dele to inrete the numberof lene ofan independ arabe whofe ob tied nd indeed, desirable tet Ili eves of tp ndependent vari