Types of research design



experiments

Chapter 8 in Babbie & Mouton (2001) 

Introduction

to all research designs  All research designs have specific objectives they strive for  Have different strengths and limitations  Have validity considerations

Types of design - experiments

1

Validity considerations
When we say that a knowledge claim (or proposition) is valid, we make a JUDGEMENT about the extent to which relevant evidence supports that claim to be true  Is the interpretation of the evidence given the only possible one, or are there other plausible ones?  "Plausible rival hypotheses" = potential alternative explanations/claims  

e.g. New York City's "zero tolerance" crime fighting strategy in the 1980s and 1990s - the reverse of the "broken windows" effect
Types of design - experiments 2

The logic of causal social research in the controlled experiment
Explanatory rather than descriptive  Different from correlational research - one variable is manipulated (IV) and the effect of that manipulation observed on a second variable (DV)  If then . 


E.g. 


"Animals respond aggressively to crowding" (causal) "People with premarital sexual experience have more stable marriages" (noncausal)

Types of design - experiments

3

Three pairs of components:  Independent and dependent variables  Pre-testing and post-testing Prepost Experimental and control groups Types of design .experiments 4 .

to detect effects of the experiment itself Types of design .experiments 5 .Components  Variables  Dependent (DV)  Independent (IV)  Pre-testing PreO and post-testing postO X  Experimental  To and control groups offoff-set the effects of the experiment itself.

experiments 6 . it is manipulated The participants who receive one level of the IV are equivalent in all ways to those who receive other levels of the IV Types of design .The generic experimental design: R O1 R O3 X O2 O4   The IV is an active variable.

experiments 7 .Sampling  1. Selecting subjects to participate in the research  Careful sampling to ensure that results can be generalized from sample to population  The relationship found might only exist in the sample. need to ensure that it exists in the population  Probability sampling techniques Types of design .

g. age. How the sample is divided into two or more groups is important  to make the groups similar when they start off  randomization .similar to quota sampling procedures  match the groups in terms of the most relevant variables.equal chance  matching .Sampling  2. and race Types of design .experiments 8 . sex. e.

experiments 9 .Variations on the standard experimental design  One-shot One- case study X  No O real comparison Types of design .

like many Germans during the Nazi period.experiments 10 . under what conditions will the person defy the authority if s/he is asked to carry out actions clearly incompatible with basic moral standards? Types of design . important issue here: how could "ordinary" citizens.A famous one-group posttest-only design oneposttest     Milgram's study on obedience Obedience to authority The willingness of subjects to follow E's orders to give painful electrical shocks to another subject A real. do these incredibly cruel and brutal things? If a person is under allegiance to a legitimate authority.

OneOne-group pre-test post-test design prepost O1 X O2 Types of design .experiments 11 .

Example      We want to find out whether a family literacy programme enhances the cognitive development of preschoolpreschool-age children.a gain of 25 Types of design .experiments 12 . Find 20 families with a 4-year old child. enrol the family 4in a high-quality family literacy programme highAdminister a pretest to the 20 children . now they postscore 75 on the test .they score a mean of say 50 on the cognitive test The family participates in the programme for twelve months Administer a post-test to the 20 children.

Two claims/conclusions: 1 The children gained 25 points on average in terms of their cognitive performance the family literacy programme caused the gain in scores .rival explanations Types of design .experiments 13 2  VALIDITY .

experiments 14 .StaticStatic-group comparison X O O Types of design .

Evaluating research (experiments)  We know the structure of research  We understand designs  We know the requirements of "good" research  Then we can evaluate a study  Is it good? Can we believe its conclusions?  Back to plausible rival hypotheses Types of design .experiments 15 .

then the conclusions drawn are not supported.Validity in designs  If the design is not valid. it is like not doing research at all  Validity of designs come in two parts:  Internal  can validity validity the design sustain the conclusions?  External  can the conclusions be generalized to the population? Types of design .experiments 16 .

the higher the internal validity The conclusions drawn from experimental results may not accurately reflect hat has gone on in the experiment itself Types of design .Internal validity  Each design is only capable of supporting certain types of conclusions  e.experiments 17 .g. the more controlled the situation. only experiments can support conclusions about causality    Says nothing about if the results can be applied to the real world (generalization) Generally.

g.Sources of internal invalidity  These sources often discussed as part of experiments.g. but can be applied to all designs (e.experiments 18 . e. see reactivity)  History  Historical events may occur that will be confounded with the IV  Especially in field research (compare the control in a laboratory. nonsense syllables in memory studies Types of design .

over time  People Types of design . tired. bored.Maturation  Changes over time can be caused by a natural learning process naturally grow older.experiments 19 .

experiments 20 .Testing (reactivity) realize they are being studied. the "on stage" effects  People Types of design . and respond the way they think is appropriate The very act of studying something may change it  In qualitative research.

people became aware that they were in an experiment. chronic mental patients Types of design .The Hawthorne studies  Improved performance because of the researcher's presence . e.g. or that they were given special treatment  Especially for people who lack social contacts. residents of nursing homes.experiments 21 .

Placebo effect  When a person expects a treatment or experience to change her/him. the person changes.experiments 22 . or the power of suggestion Types of design . even when the "treatment" is know to be inert or ineffective  Medical research  "The bedside manner".

self-fulfilling prophecies of selfe.g.Experimenter expectancy      Pygmalion effect . They don't know what treatment the participant is getting Types of design .experiments 23 . teachers' expectancies about student achievement Experimenters may prejudge their results experimenter bias Double blind experiments: Both the researcher and the research participant are "blind" to the purpose of the study.

Types of design .experiments 24 .g.Instrumentation  Instruments with low reliability lead to inaccurate findings/missing phenomena human observers become more skilled over time (from pretest to posttest) and so report more accurate scores at later time points  e.

which would not occur in moderate scorers Types of design .Statistical regression to the mean  Studying extreme scores can lead to inflated differences.experiments 25 .

Selection biases  Selection subjects for the study.experiments 26 . and assigning them to E-group and C-group ECout for studies using volunteers  Look Types of design .

experiments 27 . comparing the effectiveness of family therapy with discussion groups for treatment of drug addiction addicts with the worst prognosis more likely to drop out of the discussion group will make it look like family therapy does less well than discussion groups.g. it creates a bias to those who did not    e.Attrition  Sometimes called experimental (or subject) mortality  If subjects drop out. because the "worst cases" were still in the family therapy group Types of design .

pass on some information about the treatment (IV) Types of design .experiments 28 .Diffusion or imitation of treatments  When subject can communicate to each other.

Compensation  In real life.experiments 29 . and provide some enhanced service to the children receiving usual care Types of design . compare usual day care for street children with an enhanced day treatment condition  service providers may very well complain about inequity. people may feel sorry for CCgroup who does not get "the treatment" try to give them something extra  e.g.

experiments 30 .Compensatory rivalry  C-group may "work harder" to compete better with the E-group E- Types of design .

they may very well become profoundly demoralized Types of design .Demoralization  Opposite  e. to compensatory rivalry  May feel deprived.experiments 31 . who receive "no treatment". and give up giving unemployed high school dropouts a second chance at completing matric via a special education programme  if we assign some of them to a control group.g.

or of a wider group?  To what populations.experiments 32 . and measurement variables can the finding be generalized? Types of design . treatment variables (IV's). settings.External validity  Can the findings of the study be generalized?  Do they speak only of our sample.

or manipulations Dependent variables.experiments 33 .External validity  Mainly questions about three aspects:    Research participants Independent variables. or outcomes    Says nothing about the truth of the result that we are generalizing External validity only has meaning once the internal validity of a study has been established Internal validity is the basic minimum without which an experiment is uninterpretable Types of design .

our interest is in the variables.experiments Types 34 . not the specific details of a piece of research  But studies differ in many ways. even if they study the same variables:       operational definitions of the variables subject population studied procedural details observers settings  Generally bigger samples with valid measures lead to better external validity of design .External validity Our interest in answering research questions is rarely restricted to the specific situation studied .

experiments 35 .Selecting a sample which does not represent the population well. will prevent generalization Interaction between the testing situation and the experimental stimulus When people have been sensitized to the issues by the pre-test preRespond differently to the questionnaires the second time (post-test) (postOperationalization Types of design .Sources of external invalidity      Subject selection .

Operationalization  We take a variable with wide scope and operationalize it in a narrow fashion we find the same results with a different operationalization of the same variable?  Will Types of design .experiments 36 .

e.Field experiments  "natural" . disaster research  Static-group comparison type Static Non-equivalent experimental and control Nongroups Types of design .experiments 37 .g.

Strengths and weaknesses  Strengths  Control  Manipulating the IV  Sorting out extraneous variables  Weaknesses  Articifiality  Expense  Limited .a generalization problem range of questions Types of design .experiments 38 .

Each of the planks we now depend on we will in turn have to replace. We depend on the relative soundness of all other planks while we replace a particularly weak one. No one of them is a foundation. no one of them is incorrigible" Types of design .experiments 39 .IN CONCLUSION  Donald  "in Campbell often cited Neurath's metaphor: science we are like sailors who must repair a rotting ship while it is afloat at sea. nor point of certainty.