See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/325761416

Research limitations: the need for honesty and common sense

Article  in  Interactive Learning Environments · July 2018

DOI: 10.1080/10494820.2018.1486785

CITATIONS READS

2 1,944

1 author:

Sue Greener
University of Brighton
71 PUBLICATIONS   203 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

BSc Professional Development in Business - blend but mostly online View project

Teaching Research Methods in HE View project

All content following this page was uploaded by Sue Greener on 28 June 2018.

The user has requested enhancement of the downloaded file.

 Interactive Learning Environments

ISSN: 1049-4820 (Print) 1744-5191 (Online) Journal homepage: http://www.tandfonline.com/loi/nile20

Research limitations: the need for honesty and

common sense

Sue Greener

To cite this article: Sue Greener (2018) Research limitations: the need for honesty and common
sense, Interactive Learning Environments, 26:5, 567-568, DOI: 10.1080/10494820.2018.1486785

To link to this article: https://doi.org/10.1080/10494820.2018.1486785

Published online: 13 Jun 2018.

Submit your article to this journal

Article views: 15

View Crossmark data

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=nile20
INTERACTIVE LEARNING ENVIRONMENTS
2018, VOL. 26, NO. 5, 567–568
https://doi.org/10.1080/10494820.2018.1486785

EDITORIAL

Research limitations: the need for honesty and common sense

A quick look through the articles in this issue offers a handy instant view of the focus of current
research into learning with technologies. Educational researchers are overwhelmingly keen on
using technology to flip learning, to bring context into the classroom through Virtual and Augmented
Reality, and to use enquiry or problem-based scenarios and game-based activities for collaborative
and personal learning engagement. The concerns of teachers continue to include the development
of critical thinking, the impact or not of learning preferences, cognitive load, peer assessment and
what constitutes learner presence.
At a time when the Pandora’s Box of the world wide web is shaken by contrasting notions and impli-
cations of data privacy, it is not surprising to see Information Security, Data Management and Govern-
ance, and Data-Enabled Institutional Culture as three of the top ten IT issues for 2018 raised in Educause
Review (Grajek, 2018). Leading educational researchers are aware that Higher Education is being radi-
cally reshaped by digital technology, and this journal can provide evidence of debates within Higher
Education around the student’s and the organisation’s experience of digital enhancements for learn-
ing. This issue includes two calls for papers for future Special Issues relating to learning analytics and the
use of “big data”, and the application of crowd-sourcing within education.
But as we press forward with research-led and practitioner-led exploratory and explanatory
studies in this field, we should perhaps take a moment to consider how potentially misleading
such studies may easily be. Otherwise, we may fall into the trap that Higher Education teachers
constantly warn their students about: that of accepting data, especially self-report but also other
quantitative data, at face value. While it is unusual for this journal to publish small studies
founded on self-report data alone, particularly where a new application or course design has been
tested on only one learner cohort, it is not difficult to find reviews of literature in papers which
cite such work uncritically. A recent study of this behaviour in the tourism research field (Yüksel,
2017) found considerable response bias of varying kinds in the papers reviewed, but also found
that such papers could find further airing through citation. It is incumbent upon journals such as
this to emphasise the need for cautious statements where data may include bias, and for papers
always to include a statement of limitations, whether related to research design, methodology,
findings or the conclusions drawn. Such limitations may be included in the methodology section
or the conclusions, but without them we cannot be sure that authors have genuinely had regard
to potential areas of exclusion or bias which affect the results they report. It is unfortunately often
the case that it falls to peer reviewers to point out the need for limitations to be included.
Limitations generally fall into some common categories, and in a sense we can make a checklist for
authors here. Price and Murnan (2004) gave an excellent and detailed summary of possible research
limitations in their editorial for the American Journal of Health Education. They discussed limitations
affecting internal and external validity; internal validity relating to limitations of the study design and
its internal integrity, external validity relating to the outward generalisability of the reported results.
The construction of questions within a survey instrument, the layout and type of questions in surveys,
the level of understanding of the respondents, an over-dependence on convenience sampling and
general access difficulties, confounding variables, statistical inadequacies, low response rates,
absence of or impossibility of control groups, these all offer potential limitations in educational
research. But perhaps the most common issue for papers submitted to this journal is the “Hawthorne
effect”, the notion that an innovation or intervention may produce positive results not related to the

© 2018 Informa UK Limited, trading as Taylor & Francis Group

568 EDITORIAL

innovation but to the presence of an innovation and that the respondents are given the focus of
attention in a research study. All such limitations and more should of course be wisely considered
prior to the execution of research, but at very least must form part of the presentation of a study
for publication.
This is not just about being self-critical or particularly humble in presenting our research. Identify-
ing limitations, and explaining to the reader what impact these limitations have on the study results,
not only demonstrates rigour but also gives the authors a chance to identify clear directions for future
research. In fact this section or paragraph can be one of the most exciting parts to read in a paper, as
we get a sense of the challenges faced in the study and the forward thinking about prospective
improvements for further research which can be a rich seam of opportunity for a range of authors.
As they mine this seam for research opportunities, we should also remember that we are discussing
learning here. Despite the plethora of experimental and quasi-experimental designs we see, the
learner often has only one “go” at learning at a time. If we start to experiment with live learner
cohorts we can let them down. Researchers in the field of digital and interactive learning should
perhaps take to heart some lessons from early medical research and experiment first on themselves.
Learning communities of teachers, designers and trainers can espouse beta designs readily, and can
pilot or pioneer new designs before they lead to serious limitations of applicability and
generalizability.
The papers in this issue demonstrate clear and thoughtful understanding of research limitations,
an attribute and research practice which we, as editors, strongly recommend to potential contributors
to the journal.

References
Grajek, S. (2018). Technology and the remaking of higher education: A longer view. Educause Review, 53(1), 11–59.
Price, J. H., & Murnan, J. (2004). Research limitations and the necessity of reporting them. American Journal of Health
Education, 35(2), 66–67.
Yüksel, A. (2017, April). A critique of “Response Bias” in the tourism, travel and hospitality research. Tourism Management,
59, 376–384.

Sue Greener
University of Brighton, Brighton, England
S.L.Greener@brighton.ac.uk

View publication stats