See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/371134299

A METHODOLOGICAL CRITIQUE OF FOCUS ON THE "AVERAGE

STUDENT" IN PSYCHOLOGICAL AND EDUCATIONAL RESEARCH

Presentation · May 2023

CITATIONS READS

0 13

1 author:

A. Imam
John Carroll University
65 PUBLICATIONS 289 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Class merger in stimulus equivalence View project

Pedagogy: Testing Effects and Assessment View project

All content following this page was uploaded by A. Imam on 29 May 2023.

The user has requested enhancement of the downloaded file.

A METHODOLOGICAL CRITIQUE
OF FOCUS ON THE "AVERAGE
STUDENT" IN PSYCHOLOGICAL
AND EDUCATIONAL RESEARCH
Abdulrazaq A. Imam, PhD
Behavior Labs Africa Consultancy & John Carroll University
Statistical considerations, not experimental
nor experiential necessity

Image Source: Google Trends

“AVERAGE PEOPLE OR STUDENTS
ARE ALSO GREAT. IT’S JUST THAT
THEY MAY NOT GET NOTICED AND
RECOGNIZED AS PUBLICLY.”

By Anuupadhyay
https://www.geeksforgeeks.org/is-it-okay-to-be-an-average-student/
Retrieved April 12, 2023
 So, why the focus on
the “average person or student”
in psychology and education?
 The answer is rooted historically
in the marriage of measurement and
inferential statistics broadly
and
of measurement error and statistical
control specifically
(see Cowles, 2001; Imam, 2021; Stigler, 1992)
Definition of Psychology and Education
✦ Psychology

✦ Now standard: The scientific study of behavior and mental processes

✦ Presumably, of the individual, human or animal
✦ It is what differentiates psychology from sociology or anthropology

✦ Education

✦ Less precise and more varied (see Arslan, 2018)

✦ But revolve around learning and acquiring knowledge (see Dragoescu, 2018)
✦ Again, by the individual, but exclusively (?) human
 Predominant methodology
✦ (Yet) the predominant experimental methodology:
✦ (is) focused on averages
✦ (due to) Large-N group designs
✦ Subjected to inferential statistics

✦ Mostly null hypothesis statistical testing (NHST)

✦ But there are also the estimation and Bayesian approaches that are
considerably less common
✦ Regardless, all
three approaches rely on computation of the mean to
make sense of data
 Descriptive and Inferential Statistics

The mean

The arithmetic average = the sum of numbers / N

The t-test and the F-test to test hypotheses
They estimate population parameters
 Topical Coverage in Research Methods Textbooks
Number of Pages (%) Devoted to Design and Statistics Topics
Quantitative
Group Single-
NHST Estimation Bayesian Qualitative Case Study
Book (edition) Designs Subject
Psychology
Christensen etal., 2011 (11e) 35 (66%) 3 (6%) 0 15 (28%) 40 (63%) 3 (5%) 21 (33%)
Cozby & Bates, 2012 (12e) 23 (77%) 1 (3%) 0 6 (20%) 59 (89%) 2 (3%) 5 (8%)
Nestor & Schutt, 2015 (2e) 12 (29%) 1 (2%) 0 28 (68%) 64 (71%) 0 26 (29%)
Rosnow & Rosenthal, 2008
83 (86%) 2 (2%) 0 11 (12%) 33 (89%) 0 4 (11%)
(6e)
Education
Ary etal., 2010 (8e) 64 (40%) 4 (3%) 0 91 (57%) 59 (53%) 46 (41%) 7 (6%)
Cohen etal., 2011 (7e) 56 (43%) 3 (2%) 0 70 (54%) 25 (60%) 14 (33%) 3 (7%)
Fraenkel & Wallen, 2009 (7e) 54 (28%) 7 (4%) 0 132 (68%) 33 (53%) 2 (3%) 27 (44%)
Lodico etal., 2010 (2e) 16 (6%) 0 0 80 (83%) 27 (73%) 0 10 (27%)
 The Ubiquity of the Mean
✦ The mean is required for the t-test, F-test, effect sizes r, d, and eta in
NHST (see, e.g., Cozby & Bates, 2015)
✦ The mean is required for margin of error and confidence intervals,
forest plots and meta-analysis in estimation (see, e.g., Cumming &
Calin-Jageman, 2017)
✦ The mean is required for the “point estimate along with an interval
that summarizes the width of the [posterior] distribution” in the
Bayesian approach(see Graf, 2020)
✦ In short, the three inferential statistical approaches rely
on the mean for their respective roles in interpreting
experimental results in psychology and education
 Key Elements of Experimental Research in
Psychology and Education
✦ Develop hypothesis/hypotheses (hopefully theory-informed)
✦ Adopt a design (using experimental and control groups)

✦ Recruit participants (mostly college students)

✦ Pilot study (sometimes added; unnecessary, see Sidman, 1960)

✦ Conduct study (i.e., collect data)

✦ Analyze data (mostly NHST)

✦ Report findings (conference and/or publications)

✦ Build theory (hopefully); more hypotheses

 How it should work: what’s missing
✦ Meeting basic statistical assumptions from the outset
✦ e.g., Specifying alpha and/or p-value

✦ e.g., Specifying relevant population (for estimating parameters)

✦ e.g., Ensuring the population meets normality requirements (rarely checked or

known)

✦ Power analysis (often ignored)

✦ Random sampling (relying on convenient samples largely)
✦ Random assignment (sometimes flimsy)
 Consequential Adverse Outcomes
for Psychology and Education
 p-Hacking (see Imam, 2018)

 Failures to replicate (see Cesario, 2014)

 Lack of representativeness and generalizability (see Imam 2021;

Jaffe, 2005)

 Lack of a cumulative science (see Branch, 2014)

 Literature awash with massive psychological and educational

averages (see Imam, 2022)
 All the statistical approaches render
"the average person or student" as the
putative "subject" of interest
via the widespread adoption of group
designs for experimental work
 In reality,
it is the individual person or student
who learns,
not some nonexistent “average person
or student.”
 What then?
A largely ignored alternative to group
designs that has a long and productive
history in psychology, education, and
medicine.

(see Bernard, 1927/1957; Moran & Malott, 2004; Sidman, 1960;

Tankersley et al., 2008)
 What’s that?
✦ Small-N or single-subject designs
✦ Focus solely on the individual, situation, or setting; mostly the individual
✦ Only a few of them at a time
✦ Each extensively exposed to various conditions of the relevant variables
✦ Each exposure lasts until measurement stability
✦ Reveals functional relationships between behavior and environmental
conditions
 In education, such environments can
range from the school, the classroom, or
teacher (variables) to teaching methods,
materials, and/or technology.
ILLUSTRATIVE
EXAMPLES
from Education
Completed assignments

27% 87% 38% 90%

(68%) (73%)

(10%)
(43%)

From Witt & Elliott, 1982, p. 159

From Munro & Stephenson, 2009, p. 798
From Munro & Stephenson, 2009, p. 798
From Munro & Stephenson, 2009, p. 798
From Bohan & Smyth, 2022, p. 290
From Bohan & Smyth, 2022, p. 291
From Bohan & Smyth, 2022, p. 292
Contrarian Approaches for Psychological
and Educational Research
Large-N Group Designs
✦ p-Hacking
✦ Failures to replicate
✦ Lack of representativeness
and generalizability
✦ Lack of a cumulative science
✦ Literature awash with
averages
Small-N Single Subject Designs
Utterly irrelevant
Replications built in
Replication ensures generality
Mute issue
Use does not define the
individual student
 Implications of Broader Adoption Small-N
✦ Would focus attention on mastery in education, instead of
perpetuating mediocrity
Long history in education with PSI
Most common in hard sciences
Instructional design advances with computers and AI

✦Would require a significant shift in how we approach

 asking questions
 collecting data
 analyzing and interpreting data
 making research and practice decisions
 In Conclusion
Nothing wrong with the average measure per se

Making sense of data is the culprit

But we are not condemned to live with the “average student” side effect
of heavy reliance on large-N group designs
Whose shorty implementation has been responsible for
methodological problems like replication failures
Demonstrably viable alternative in small-N single-subject designs

Should be more widely adopted in psychology and education (see Table

1 above)
Not doing so is like repeating the same mistake over and over and
expecting a different result.
Thank you for your attention!
 References

Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and
Social Psychology, 100, 407-425. doi: 10.1037/a0021524

Bernard, C. (1927/1957). An introduction to the study of experimental medicine. Dover Publications, Inc.

Bohan, C., & Smyth, S. (2022). The Caught Being Good Game in a Middle Primary School Class. Behavior Analysis: Research and Practice, 22, 283–297

Branch, M. (2014). Malignant side effects of null-hypothesis significance testing. Theory & Psychology, 24, 256-277.
https://doi.org/10.1177/0959354314525282

Cesario, J. (2014). Priming, replication, and the hardest science. Perspectives on Psychological Science, 9, 40-48. doi: 10.1177/1745691613513470

Christensen, L. B., Johnson, R. B., & Turner, L. A. (2011). Research methods, design, and analysis, 11e. Allyn & Bacon.

Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education, 7e. Routledge.

Cowles, M. (2001). Statistics in psychology: An historical perspective. Lawrence Erlbaum.

Cozby, P. C., & Bates, S. C. (2015). Methods in behavioral research, 12e. McGraw Hill.

Cumming, G., & Calin-Jageman, R. (2017). Introduction to the new statistics: Estimation, open science, and beyond. New York, NY: Routledge.

Graf, C. (2020). Performing a Bayesian Analysis by Hand: A step by step walkthrough of a real life Bayesian analysis. Towards Data Science Retrieved
19 April, 2023 from https://towardsdatascience.com/performing-a-bayesian-analysis-by-hand-c589ab992916
 References
Imam, A. A. (2018). Place of behavior analysis in the changing culture of replication and statistical reporting in
psychological science. European Journal of Behavior Analysis, 19, 2-10, doi:10.1080/15021149.2018.1463123

Imam A. A. (2021). Historically recontextualizing Sidman’s Tactics: how behavior analysis avoided psychology’s
methodological Ouroboros. Journal of the Experimental Analysis of Behavior, 115, 115–28.
https://doi.org/10.1002/jeab.661.

Imam, A. A. (2022). Remarkably reproducible psychological (memory) phenomena in the classroom: some
evidence for generality from small-N research. BMC Psychology, 10, 274-290.

Jaffe, E. (2005, September 10). How random is that? Observer, 18 (9).

https://www.psychologicalscience.org/observer/how-random-is-that

Moran, D. J., & Mallott, R. W. (Eds). (2004). Evidence-based Educational methods. Academic Press.

Munro, D. W., & Stephenson, J. (2009). The effects of response cards on student and teacher behavior during
vocabulary instruction. Journal of Applied Behavior Analysis, 42, 795-800.

Nestor, P. G., & Schutt, R. K. (2015). Research methods in psychology: Investigating human behavior. 2e. Sage
 References

Rosnow, R. L., & Rosenthal, R. (2008). Beginning behavioral research: A conceptual primer, 6e. Pearson Prentice
Hall.

Sidman, M. (1960). Tactics of scientific research: Evaluating experimental data in psychology. Authors Cooperative.

Stigler, S. M. (1992). A historical view of statistical concepts in psychology and educational research. American
Journal of Education, 101, 60-70. https://doi.org/ 10.1086/444032

Tankersley, M., Harjusola-Webb., & Landrum, T. J. (2008). Using single-subject research to establish evidence base
of special education. Intervention in School and Clinic, 44, 83-90. https://doi.org/10.1177/1053451208321600

Witt, J. C., & Elliott, S. N. (1982). The response cost lottery: A time efficient and effective classroom intervention.
Journal of School Psychology, 20, 115-161.

View publication stats