You are on page 1of 52

A scientific approach to educational research?

What’s this all about?
• Understanding the assumptions behind the scientific approach to educational research • Understanding the power and limitations of experimental designs and of statistical tools • Understanding the potential of mixed methodologies and methods

• Supporting critical reading and personal research decision making

Positivist Assumptions
after Lincoln & Guba 1985
• There is a single reality that can be revealed; this reality will not be contested by fair minded individuals • Part of this can be studied independently; the whole is the sum of those parts • It is possible to study this objectively, independently of the researcher. • Results will apply at other times; in other places • Cause and effect can be distinguished • Inquiry is value-free

True of science?
• Science is about building models not mapping reality. Different models are possible from the same data • Data has error – so no picture of reality is error free (triangulate across forms of data, alternative studies etc) • 0+0 = 100 • Observing a system necessarily changes it Δx. Δp≈h • Sometimes causes are complex and not open to definition • Values determine what science is done and what is regarded as important

True of social science?
• If it’s not really true of physical science, it’s not likely to apply cleanly to social science BUT ALSO • ‘Scientific’ social research is dependent on the logic of statistical significance
– Meaning of statistical significance – Limitations of statistical significance

• Issues of sampling

Dependence on the logic of statistical significance • An example Group 1 – 10 people. average score=6 Group 2 – 10 people. average score=8 • Is this difference between the groups important? • FIRST STEP: is it likely to occur just by chance? .

• Would it be unusual for two random samples from one population to have means as different as 6 and 8? • If yes. • This is our NULL HYPOTHESIS Ho • The groups are like random samples from the same population. we would say that the difference is probably just down to chance • Difference is not statistically significant. we could assume that the two groups came from different populations • Difference is statistically significant. Accept Ho . Reject Ho • If no.Basic notion of significance • Assume no difference in outcome between the two groups.

Population 8 5 10 4 7 6 4 4 0 5 3 8 6 2 3 1 9 4 0 9 9 7 3 5 5 4 7 3 7 6 9 10 0 7 4 9 2 7 7 7 2 5 4 7 4 8 3 1 8 7 1 5 9 1 1 7 6 2 7 1 7 7 0 9 3 5 6 9 6 2 5 6 4 3 7 7 3 3 9 10 .

One random sample 8 5 10 4 7 6 4 4 0 5 3 8 6 2 3 1 9 4 0 9 9 7 3 5 5 4 7 3 7 6 9 10 0 7 4 9 2 7 7 7 2 5 4 7 4 8 3 1 8 7 1 5 9 1 1 7 6 2 7 1 7 7 0 9 3 5 6 9 6 2 5 6 4 3 7 7 3 3 9 10 Mean of red group = 7.1 .

mean of blue group = 5.9 Difference = 1.Two random groups 8 5 10 4 7 6 4 4 0 5 3 8 6 2 3 1 9 4 0 9 9 7 3 5 5 4 7 3 7 6 9 10 0 7 4 9 2 7 7 7 2 5 4 7 4 8 3 1 8 7 1 5 9 1 1 7 6 2 7 1 7 7 0 9 3 5 6 9 6 2 5 6 4 3 7 7 3 3 9 10 Mean of red group = 7.2 .1.

Distribution – difference of two means .

Distribution – difference of two means .

Not just group differences • Is there a correlation between two variables? • One group 10 people IQ and attitude scores are linked .

Correlation IQ and attitude to school 180 160 140 120 100 80 IQ score 60 40 0 10 20 attitude to school .

(2-tailed) N **.000 . 40 40 IQ and attitude to school 180 160 IQ s core attitude to s chool Pears on Correlation Sig. 140 120 100 80 IQ score 60 40 0 10 20 attitude to school .564** 1.000 .Correlation Correlations attitude to IQ s core s chool 1. Correlation is significant at the 0.000 .000 40 40 . (2-tailed) N Pears on Correlation Sig.01 level (2-tailed). .564** .

then our results probably aren’t just pairs of random numbers – there is a link between IQ and attitude • If is NOT unlikely.Statistically significant? • Is it unlikely that we would get this correlation if we just chose 40 pairs of numbers from a set of random numbers? • If it IS unlikely. then our result could well be nothing more than chance .

The importance of significance It may be significant but does it matter? .

The importance of significance It may be significant but does it matter? .

088 .000 5.196 .276 .000 10.000 r .444 .997 .Effect of sample size Critical Values of r for Rejecting the Null Hypothesis (r = 0) at the .632 .05 Level Given Sample Size n n 3 5 10 20 50 100 500 1. .062 .0196 Note: Values are taken from Table 13 in Pearson and Hartley (1962).0278 .878 .

What if? arguments • If we get a significant result with a sample size if 100 how small would the sample have to be for this to become non-significant? (If very small – fairly confident in rejecting Ho. say. if 90 (say) – not very confident in rejecting Ho) • If we get a non-significant result with a sample size if 20 how big would the sample have to be for this to become significant? (If 30. – not much confidence in Ho. if very large –fairly confident in Ho) .

Effect size Effect size  m1  m2  sd1 Effect Size =1 .the mean for one group coincides with boundary score for the top 16% of the other group .

Non-significant results .

Sampling Samples that allow statistical generalisation • • • • • random systematic stratified random cluster multi-stage Samples that don’t allow statistical generalisation • quota • convenience • snowball .

Sampling Samples that allow statistical generalisation • • • • • random systematic stratified random cluster multi-stage Samples that don’t allow statistical generalisation • quota • convenience • snowball .

Is your sample representative? • Since it’s quite difficult to get representative samples. . the answer may well be no.

The experiment • SM’s gold standard for revealing cause and effect relationships .

The experiment • Logic of experimental methodology • Avoiding threats to validity • Problems – Sampling and assignment – Models of causation • Solutions – Quasi-experimental designs – Mixed methods within an interpretive paradigm .

OXO • Is X the cause of any difference between the first and second observation? .

Issues of validity at the design stage – experimental designs Internal threats to validity • History • Maturation • Testing • Instrumentation • Selection • Statistical regression • Mortality .

External threats to validity • Interaction of selection bias and treatment • Interaction between testing and treatment • Reaction to being in an experiment .

True experiment R X - Oe Oc .

Why are experimental designs so useful? X Oe Oc R • They deal with treats to validity. • ….?? ..

So why bother with SM at all? Can be very powerful in revealing complex interactions and relationships .

.

Multiple regression • Dependent variable English O-level • Independent variables – DAT scores (VR.SP.18 . B.26 Adjusted R squared = 0.NA.66VR +2.35SP +0.MR.36 Standard error = 15.SR.64B +31.65LU + 0.CSA.I) – Creativity (Flu.Flex.Orig) Eng O level = 0.LU) – Attitudes ( A. ….VN.

teacher 72 40 .Chooses Stays on task 85 56 Persists at task Eager to learn Infant checklist Match pictures 43 48 Draws onjects Can use tools 59 49 83 Copy cross Copy square 58 Picks up songs Describes env Organised tale Listens to story Gives message Clear speech 1 59 42 70 72 58 2 3 4 40 5 6 43 Settles away 61 Receptive to sch 79 Integrates-peers 60 Plays .peers 64 Relates .

Cluster results .

So why bother with SM at all? Can be very powerful in revealing complex interactions and relationships So how do we get the benefits in spite of the problems? – through limiting the population – through non-inferential stats – through links with interpretive designs .

Limiting the population .

Non-inferential use of statistics .

Quasi experiments – no R O O X O O .

Using quantitative methods within an interpretive methodology .

Mixing approaches? • Paradigms as watertight boxes – simply a choice to be made • Paradigms as coherent systems that serve as the starting point for creative thinking .

One example • Using interpretive methodologies to explore possible threats to validity in a quasi-experimental design .

Interpretive Assumptions after Lincoln & Guba 1985 • Realities are multiple. and are individually constructed • The knower cannot be separated from the known • We can only make statements that are time and context bound • All entities are continually shaping each other • Inquiry is inevitably value-bound .

(Interp. Paradigm) .Creative thinking Linking qualitative and quantitative data • qualitative work gives rich exemplification of generalisable relationships established by statistical methods – (Sci Paradigm) • quantitative work establishes the generalisablity of hypotheses which emerge from a qualitative enquiry (Sci Paradigm) • qualitative and quantitative work are used together (iteratively) to deepen the understanding of the particular cases on which we have been working.

• NOT that quantitative work should be used to establish the generalisablity of hypotheses which emerge from a qualitative enquiry . .as if this is in some way a necessary step in order that the qualitative findings can be taken seriously.It’s … • NOT the purpose of qualitative work simply to give rich exemplification of generalisable relationships established by statistical methods – to give a human face to a statistical study. • BUT qualitative and quantitative work are used together (iteratively) to deepen the understanding of the particular cases on which we have been working.

Another example • Using scientific and interpretive methodologies iteratively to explore factors influencing learning in FE classrooms .

2 2.2 3.0 3.C1 site differences 3.5 Pth4 Prn GNVQBus ADMPA Student Negotiation > Shared Control 2.5 SpprtMS WrkbdAst 3.4 ASPsych CACHE Student Negotiation .5 2.0 2.0 AVCET&T Voc Path BTECHlth Engneer 1.0 ITSkills Connect2 Shared Control > Student Negotiation ELDrama ESOL LS Shared Control 2.6 2.4 2.8 3.

negotiation based on assignments.Explaining the High SC/Low SN grouping Support for Mature Students • Self assessment. individual learning plans agreed and reviewed by tutor and student Workbased assessment • Individual support from tutor (underground working) .

Explaining the High SC/Low SN grouping Workbased assessment • Different geographical placements Support for Mature Students and IT skills • Same room but ISOLATION • Different times ESOL • Same room. same times but • Different languages .

0 2.2 2.5 2.0 ITSkills Connect2 Shared Control > Student Negotiation ELDrama ESOL LS Shared Control 2.8 3.2 3.0 AVCET&T Voc Path BTECHlth Engneer 1.Does isolation feature elsewhere? 3.4 ASPsych CACHE Isolation was a feature of the ‘top left’ sites In 4 of the ‘bottom right’ sites isolation was ‘not at all evident in this site’ Student Negotiation Isolation (broadly defined) appeared to be a factor related to a site culture in which there was low student negotiation.4 2.5 Pth4 Prn GNVQBus ADMPA Student Negotiation > Shared Control 2.0 3.6 2. .5 SpprtMS WrkbdAst 3.