International Journal of Research & Method in Education

ISSN: 1743-727X (Print) 1743-7288 (Online) Journal homepage: https://www.tandfonline.com/loi/cwse20

Learning management systems: a review of the

research methodology literature in Australia and
China

Darren Turnbull, Ritesh Chugh & Jo Luck

To cite this article: Darren Turnbull, Ritesh Chugh & Jo Luck (2020): Learning management
systems: a review of the research methodology literature in Australia and China, International
Journal of Research & Method in Education, DOI: 10.1080/1743727X.2020.1737002

To link to this article: https://doi.org/10.1080/1743727X.2020.1737002

© 2020 The Author(s). Published by Informa

UK Limited, trading as Taylor & Francis
Group

Published online: 05 Mar 2020.

Submit your article to this journal

Article views: 88

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=cwse20
INTERNATIONAL JOURNAL OF RESEARCH & METHOD IN EDUCATION
https://doi.org/10.1080/1743727X.2020.1737002

Learning management systems: a review of the research

methodology literature in Australia and China
Darren Turnbull , Ritesh Chugh and Jo Luck
School of Engineering and Technology, Central Queensland University, Rockhampton, Australia

ABSTRACT ARTICLE HISTORY

Education is seen as an important vehicle to foster relationships with Received 28 May 2019
countries in the Asia-Pacific region – particularly with Australia’s most Accepted 22 January 2020
important economic partner, China. There are many prior studies that
KEYWORDS
have explored the impact of Learning Management Systems (LMSs) on Learning management
users, through a variety of research designs. However, the diversity of system; e-learning; online
research designs used means that the nuances and subtleties of the learning; research
diverse approaches, particularly in the context of Australia and China are methodology; research
unknown. The purpose of this literature review is to compare the method; Australia; China
research designs employed by empirical studies of LMS usage in
Australia and China. This review explores the similarities and differences
in research methodologies (quantitative, qualitative or mixed methods),
research approaches (confirmatory or exploratory) and data collection
methods commonly used in a selection of contemporary studies from
both countries. The findings indicate Chinese studies employ
quantitative methodologies more than Australian studies; confirmatory
research is more prevalent in China than Australia; and that there is a
distinct preference for surveys in Chinese studies. This review will assist
the developers of future collaborative research projects between China
and other countries, to construct study designs that are compatible with
the identified research strengths.

Introduction
Traditional classroom methods of course delivery have been significantly transformed by the inte-
gration of e-learning technologies into course offerings (Bottino and Robotti 2007; Stokes 1999;
Sutherland et al. 2004). Learning Management Systems (LMSs) are defined as online learning technol-
ogies for the creation, management and delivery of course material (Sabharwal et al. 2018; Turnbull,
Chugh, and Luck 2019). In today’s ubiquitous digital environment, LMSs play an important role in
enhancing and facilitating teaching and learning. LMSs not only enable the delivery of instructions
and electronic resources to improve and augment student learning in a collaborative environment,
but also allow instructors to focus on designing meaningful pedagogical activities (Kattoua, Al-Lozi,
and Alrowwad 2016).
LMSs are an integral part of the learning experience for students in many post-secondary insti-
tutions in Australia and China. Chinese students represent a significant portion of the total overseas
student market in Australia. There are more than 1,37,000 Chinese students enrolled in Australian
institutions, mainly in the post-secondary sector (Department of Education and Training 2019). The

CONTACT Darren Turnbull d.turnbull@cqu.edu.au

© 2020 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group
This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://
creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the
original work is properly cited, and is not altered, transformed, or built upon in any way.
2 D. TURNBULL ET AL.

rise in importance of LMSs in post-secondary institutions has led to an increase in interest in empirical
research publications that investigate the impact of such systems on institutional stakeholders. This is
confirmed by Tharindu, Adams, and Williams (2013) who conducted a systematic review of the litera-
ture on MOOCs1 from 2008 to 2012 which found that the number of publications increased consist-
ently over that period. LMS research often provides interesting insights into the choice of appropriate
research methodology for particular LMS related studies. For example, a study into e-research
methods using web-based commenting tools suitable for doctoral and early career academics
explored how this approach could be used with online focus groups (Kilburn and Earley 2015). The
study suggested that such tools facilitate the engagement of research participants with the research
data during analysis, enabling participant input into the interpretation of the data. In another study
into online social learning, Kent and Rechavi (2018) concluded that the assessment of online learning
should be based on methodologies that are different from face-to face learning. They emphasized
that online learning is often viewed through the lens of static documents such as records of social
interactions without due consideration given to the dynamic interactions between learners in
online environments. Finally, Beach and McConnel (2019)’s study into the use of an eye-tracking
methodology to study teacher learning revealed some interesting insights into the sort of data
that can be obtained using this approach, such as behavioural patterns and decision-making strat-
egies. It is often useful to compare research designs between different studies to ascertain if there
are any factors that influence the choice of research method, such as country of origin. An integrative
literature review looked at research strategies employed in empirical studies about the use of LMSs in
e-learning management, but it did not compare research designs arising from different countries (Oli-
veira, Cunha, and Nakayama 2016), such as Australia and China.
An understanding of the reasons for differences between the research designs of LMS studies can
help consumers of empirical LMS research make a more informed determination of the quality and
relevance of the studies. Given the importance of the Australia-China post-secondary educational
relationship, a specific focus on Australia and China LMS studies would be of significant value to
key LMS stakeholders in Australia and China. As there are often commonalities between research
designs in different disciplines and countries, the results of this review would also be of interest to
a broader audience with an interest in cooperative research in China. Some of the research design
decisions that could impact on a LMS study’s usefulness include: (1) the research methodologies
adopted (quantitative, qualitative or mixed methods), (2) whether the research is exploratory or confi-
rmatory in design, and (3) the methods employed by researchers to gather empirical data. The purpose
of a study’s methodology is to assist the audience to understand the processes used to arrive at the
research findings (Brannen 2005). By identifying research methodologies as qualitative, quantitative
or mixed-methods, the limitations and advantages inherent in these approaches can be properly
attributed to the study findings. The choice of research approach combined with the intent of the
research question (exploratory or confirmative) should drive the choice of data collection and analysis
methods. How these three attributes of a study’s methodology are constructed establish the charac-
teristics and limitations of a study’s outputs. No prior study was found that has looked at the research
designs employed by empirical studies of the impact of LMSs on learner communities, particularly
those that compare research from two or more educational partners. This literature review compares
and contrasts 23 empirical studies (fourteen with an Australian focus, nine with a Chinese focus) with
the aim of identifying factors that may influence research design decisions. The main aim of this review
is to investigate the research methodologies, research approaches (confirmatory or exploratory) and
data collection methods used by empirical research studies on the perceptions of LMS usage in Aus-
tralia and China. A secondary aim of this review is to discern whether there are any significant differ-
ences in research methodologies employed by Australian and Chinese research projects, and to
suggest possible reasons for these differences. The research questions that drive this review are:
RQ1: What are the research methodologies (qualitative, quantitative or mixed-methods) used by empirical
research studies on the perceptions of LMSs in Australia and China?
 INTERNATIONAL JOURNAL OF RESEARCH & METHOD IN EDUCATION 3

RQ2: What are the confirmatory or exploratory research approaches used by empirical research studies on the
perceptions of LMSs in Australia and China?

RQ3: What are the data collection methods used by empirical research studies on the perceptions of LMSs in Aus-
tralia and China?

This paper is structured as follows. The next section outlines the methodology used to select papers
and the inclusion and exclusion criteria used in this selection. The subsequent section, ‘findings and
discussion’ provides an overview of the qualitative, quantitative and mixed-methods research meth-
odologies, exploratory and confirmatory approaches, and data selection techniques discovered in the
review. This is followed by ‘practical and managerial implications’, which provides recommendations
of how the review findings can benefit LMS decision makers. The paper concludes with a summary of
the important findings and identifies opportunities for future research.

Research methodology
The methodology adopted in this review, draws on the strengths of two main approaches to literature
reviews: narrative and systematic. According to Rother (2007), a narrative review is a critical analysis of
the literature of a chosen topic. Narrative reviews are broad in nature and do not describe in detail the
methods employed to collect and analyse the literature selected for examination (Henry et al. 2018). By
contrast, systematic reviews are characterized by explicit, transparent methods that can be reproduced
by other researchers (Collins and Fauser 2005). The strengths of systematic reviews include the narrow
focus of the research question, comprehensive search for evidence, and criterion-based selection of rel-
evant evidence. This review adopts a narrative approach to address the research aims and applies some
of the rigour of systematic review methodologies to the selection and inclusion of the published articles
referred to in this paper. Literature reviews drawing on the characteristics of systematic and narrative
approaches have been widely used in the analysis of a wide range of information technology-related
studies such as social media (Best, Manktelow, and Taylor 2014; Robinson et al. 2016; Sarmiento et al.
2018), online learning (McCutcheon et al. 2015; Sinclair et al. 2016) and human engagement with com-
puters (Boyle et al. 2012; Fleischer 2012).

Search inclusions and exclusions

The databases used in this review were A+ Education, Education Research Complete, ERIC, Vocational
Education plus (VOCEDplus), Teacher Reference Centre, Scootle, Vocations and Career Collection, and
Mental Measurements Yearbook. The databases were chosen because they covered educational jour-
nals. Only peer reviewed journal publications were included in the searches. Given the rapid
advances in LMS technology and use over the years, it was pertinent to limit the age of publications
included in this review to ensure that the included papers had a contemporary focus. Accordingly,
results were restricted to a five-year period (2014–2018) and only full text papers written in
English were included.

Selection process and search results

Based on the title and abstract, publications were initially screened for inclusion depending on their
relevance to LMS usage and impact. After reading each article, a short list of publications was com-
piled according to the following six criteria: the publication critiqued the use of LMSs in a post-sec-
ondary institution; the publication was based on empirical evidence, not expert accounts of other
studies; the publication was an empirical study, not an analysis of other works; the research method-
ology was identified and discussed; there was an adequate description of the context in which the
research was carried out; and the data analysis was sufficiently rigorous. Figure 1 depicts the
search strategy used in this review.
4 D. TURNBULL ET AL.

Figure 1. Search strategy.

The search strategy employed in this review yielded 40 publications about LMS systems in China
and 54 publications about LMS systems in Australia. It is noted here that despite being specifically
excluded using the search criteria, ‘no full text’ records were also returned by the database searches.
There were also cases of full text records being inaccessible due to connecting database authentica-
tion issues.
Each retrieved publication was examined for inclusion against each of the previously outlined six
selection criteria. A total of nine China-focussed and fourteen Australia-focussed publications were
shortlisted for further investigation. This sample size is justified as previous literature reviews have
 INTERNATIONAL JOURNAL OF RESEARCH & METHOD IN EDUCATION 5

similar sizes, such as McCutcheon et al. (2015)’s study of online, blended and face-to-face learning in
nursing education (19 papers); Du et al. (2013)’s investigation of web-based learning in nursing edu-
cation (9 papers); and Cook and Steinert (2013)’s review of online learning for faculty development
(20 papers). This review mainly considered research papers that were substantially based in either
Australia or China. However, a United States study by Sadykova (2014) was also included because
of its substantial focus on the insights of a Chinese student studying online at an American institution
while residing in China. Although some papers referred to other countries, the dominant focus of
each study determined their classification as Australia-focussed or China-focussed. One of the aims
of this review was to compare research methodologies employed in LMS studies in Australia and
China, so studies conducted in other countries were not considered to be relevant to this review.
While there were some research papers that compared LMSs in other countries such as the USA,
Japan and Malaysia, they were included in this review because of their substantive focus on China.
Accordingly, such papers were classified as China-focussed.
These articles were uploaded into NVivo to facilitate the analysis process.2 Within NVivo, two attri-
butes were assigned to each publication: the paper’s research methodology (qualitative, quantitative
or mixed methods); and its deductive approach (confirmatory or exploratory). The sections of each
paper’s text describing data collection methods were also coded in NVivo with one or more of the
following labels: focus groups, interviews, LMS data, observations, online discussions, online
diaries, and surveys.

Limitations of the review

It is possible that since our review was restricted to specific databases, educational articles from other
sources may have been omitted. The inclusion of more databases during the search process could
have increased the number of articles that met the search criteria. In addition, the databases used
in this review are outside the control of the authors in terms of the publications they have access
to, their search algorithms, and their currency. Accordingly, the articles retrieved in this review are
dependent on how each database executes the search query. Another limitation is the search restric-
tion of English-only publications from prominent English language databases. Expanding the search
to include Chinese language and mainland China databases could have further enriched this review.

Findings and discussion

In the process of analysing the Australia-focussed and China-focussed articles in this review, three
aspects of each paper’s research design were considered. These were: (1) Research methodology:
qualitative, quantitative or mixed methods; (2) Exploratory versus confirmatory design; and (3)
Data collection methods employed in the study. The following discussion is an overview of the litera-
ture review findings from these perspectives.

Research methodology: China researchers’ preference for quantitative studies

Figure 2 displays the breakdown of studies by category of study: quantitative, qualitative, or mixed
methods.
As can be seen from Figure 2, the Australian studies were predominantly qualitative (nine in total)
with five studies classified as mixed methods. The China-focussed studies by comparison, consisted
of four quantitative studies, one qualitative study and four mixed methods approaches. This is an
interesting difference that may have something to do with epistemological differences between
China-based and Australia-based researchers. The inference drawn from this could be that China-
based researchers conducting research in China may prefer quantitative methods because of a
belief that they isolate the author’s personal background and beliefs from the results of the research.
Maintaining ‘Guanxi’ or relationships in China is important to the research process (Kriz, Gummesson,
6 D. TURNBULL ET AL.

Figure 2. Choice of research methodology by country of focus.

and Quazi 2013) so Chinese researchers would be keen to avoid situations that might cause offense
to others.
There have been few attempts in the literature to compare education research methodologies in
China and other countries, and none that dealt explicitly with the use of LMSs in higher education.
However, one study focussing on the comparison of education research in China and the USA does
explicitly address the issue of differences in research methodologies between the two countries
(Zhao et al. 2008). In this study, the researchers compare articles from two prominent research pub-
lications in each country: the American Educational Research Journal (AERJ) and 教育研究 (Jiaoyu
Yanjiu) (JYYJ). In their analysis of papers from these two journals, Zhao et al. (2008) attempt to
discern differences in research methodologies adopted by papers published by each journal. They
state that Chinese researchers appear to favour qualitative research methods over quantitative
research methods. It should be noted that the majority of the papers published in the JYYJ were
based on domestic issues, so they can be considered China-focussed studies. However, the
authors note that the majority of Chinese papers they analysed were not empirical studies, but
rather personal reflections that did not employ systematic empirical methodologies. Only two of
these were classified as quantitative studies based on original data collection. Zhao et al. (2008)’s
findings that China-focussed studies rarely use quantitative methods is at odds with the findings
in this review where four Chinese studies had substantially used quantitative methods. This differ-
ence could be attributed to either an increasing preference for quantitative research methods
since the Zhao et al. (2008)’s review, or research design considerations for LMS studies that
require more quantitative approaches.
An analysis of the four quantitative research papers with a China focus, reveals that a high number
of participants is a common factor in the studies as shown in Table 1.

Table 1. Number of participants in quantitative studies.

Study Number of participants
Gómez-Rey, Barbera, and Fernández-Navarro (2016) 1175
Xu, Du, and Fan (2014) 307
Zhang et al. (2017) 245
Zhao and Mei (2016) 372
 INTERNATIONAL JOURNAL OF RESEARCH & METHOD IN EDUCATION 7

Research objectives often determine the data collection methods to be employed in a study. All
of the four studies utilized surveys as the vehicle to collect data. The survey data was then analysed by
statistical methods against the aims of the research project. Gómez-Rey, Barbera, and Fernández-
Navarro (2016)’s cultural dimensions study analysed surveys utilizing multiple correspondence analy-
sis and cluster analysis to explore patterns in student perceptions of online learning. Correlation
analysis was used on survey data in Xu, Du, and Fan (2014)’s exploration of emotional management
of online group work by Chinese students. Zhang et al. (2017) applied bootstrapping hypothesis
testing to establish the accuracy of their proposed model of Massive Open Online Course (MOOC)
adoption by Chinese learners, while Zhao and Mei (2016) applied T-tests and one-way ANOVAs to
survey data to establish if there was any difference in motivation to pursue online learning
between American and Chinese students. Clearly it is easier to perform a quantitative analysis of
responses to closed-ended questions than it is to thematically analyse responses to open-ended
questions. However, this does not explain why qualitative measures were not used to support
findings from the quantitative analysis. Table 2 shows the number of participants in the mixed
methods studies. These studies included quantitative analysis techniques that were applied to
data that was collected via surveys and in some cases, interviews.
The five studies with 70 or less participants (Sadykova (2014); Wang and Duan (2014); Khairudin
and Hamid (2015); Li et al. (2015); Ng and Angstmann (2017)) used mixed-methods approaches
that involved the collection of detailed qualitative data from all participants via surveys and/or inter-
views. The studies by Sridharan, Deng, and Kinshuk (2014) and MacKenzie, Muminovic, and Oerle-
mans (2017) had a qualitative component that focussed on a small number of participants who
were interviewed, and a quantitative component that analysed a much larger number of survey
responses. Yang and Wang (2014)’s survey-based research was substantially quantitative but
included a thematic analysis of answers to free response questions in the survey. The available popu-
lation of participants in each mixed-methods study may have influenced the choice of qualitative or
quantitative technique. However, this does not explain why the China-focussed quantitative studies
do not include a qualitative element. Quantitative research is often criticized for its dehumanizing
aspect, reducing a problem to numbers and statistical analysis (Schulze 2003). Qualitative research
can complement generalizations drawn from the analysis of statistical data by providing rich insights
into specific instances of the phenomena explored by a study.
A possible reason for the finding that all of the quantitative studies in this review were from China
and that there were no quantitative studies in Australia, is that only English language publications
were reviewed. China-based scholars submitting research written in English are likely to be attracted
to quantitative research methods due to a misguided belief that journals in English and in the USA in
particular, favour quantitative research methods. This is supported by Lyytinen et al. (2007) who
confirm the common held belief among European scholars that high impact journals prefer quanti-
tative studies despite the evidence that the investigated journals upheld a practice of accepting all
types of research. In addition, most Chinese university graduates have endured a university

Table 2. Number of participants in mixed methods studies.

Study Number of participants
Australia
Khairudin and Hamid (2015) 35
MacKenzie, Muminovic, and Oerlemans (2017) 42 (Two Tutors and 40 students)
Ng and Angstmann (2017) 59
Salmon et al. (2015) 129 Interviews (Qualitative) and 155 surveys (Quantitative)
Sridharan, Deng, and Kinshuk (2014) 29 Interviews (Qualitative) and 210 surveys (Quantitative)
China
Sadykova (2014) 12
Li et al. (2015) 18
Wang and Duan (2014) 70 (10 tutors and 50 students)
Yang and Wang (2014) 800 (for two surveys at different times)
8 D. TURNBULL ET AL.

admissions exam that places a premium on the mastery of mathematical concepts (Davey, De Lian,
and Higgins 2007), so the choice of analytical techniques that rely on established statistical tools may
be a more attractive option than qualitative approaches when the target publications are English
language journals.

Exploratory versus confirmatory design

Deductive reasoning is often considered a top down approach. Deductive reasoning starts with a
general theory, from which is extracted a hypothesis to be tested, and examines observable phenom-
ena against this hypothesis to determine the validity of the original theory (Trochim 2006). Figure 3
illustrates the stages of deductive reasoning that lead to a confirmation or denial of a theory under
consideration.
Inductive reasoning works in the opposite way. It starts with an analysis of observable phenom-
ena. This is followed by the identification of themes and commonalities within the data that leads
to the development of hypotheses to be tested. These hypotheses, if confirmed by subsequent obser-
vations, can lead to an established theory (Trochim 2006). Figure 4 depicts the broad stages of induc-
tive reasoning that leads to the development of a viable theory.
Deductive approaches to research are often associated with quantitative research, and inductive
reasoning with its qualitative nemesis. Accordingly, many researchers apply purely quantitative tech-
niques to confirmatory studies, and qualitative techniques to exploratory studies. It has been argued
that the analysis tools employed in a research project should not be driven by a chosen research
methodology (qualitative or quantitative) but rather by its confirmatory (deductive) or exploratory
(inductive) objective (Onwuegbuzie and Leech 2005). Onwuegbuzie and Leech (2005) also argue
that a mix of qualitative and quantitative analytical methods should be employed regardless of
the research objective. For example, in exploratory studies data analysis could use quantitative tech-
niques such as descriptive statistics or cluster analysis, and qualitative methods such as thematic
analysis. By contrast, confirmatory studies could utilize inferential statistics combined with confirma-
tory thematic analysis to analyse the collected data. In this way, the strengths of methods from both
qualitative and quantitative methodologies are combined to enrich data analysis.
It is useful to examine the papers in this review, based on their confirmatory or exploratory charac-
teristics. Figure 5 depicts the number of confirmatory and exploratory studies by country of origin.

Figure 3. Stages of deductive reasoning. Adapted from ‘Deduction and Induction’ (Trochim 2006).
 INTERNATIONAL JOURNAL OF RESEARCH & METHOD IN EDUCATION 9

Figure 4. Stages of inductive reasoning. Adapted from ‘Deduction and Induction’ (Trochim 2006).

Figure 5. Confirmatory and exploratory studies by country of focus.

There was a total of 22 empirical studies classified as either confirmatory or exploratory. The study
by Mirriahi et al. (2015) on leadership and change was excluded from the classification process
because the paper’s intent was to describe an existing situation rather than vindicate a hypothesis
or establish a new theory. Eighteen papers (five China-focussed and thirteen from Australia) were
classified as exploratory in nature. All four confirmatory studies were China-focussed.
It is interesting to note that there were no confirmatory studies among the Australia-focussed
studies. However, as previously reported, there were instances of both qualitative and mixed-
methods approaches employed in the research designs of the exploratory studies. For example,
the Australian study on the use of blended e-learning relied entirely on rich data extracted from inter-
views and focus groups to create a detailed description of several instances where trade-related
organizations had attempted to adopt blended learning into their training programmes (Callan,
10 D. TURNBULL ET AL.

Johnston, and Poulsen 2015). By contrast, Sridharan, Deng, and Kinshuk (2014) used interviews and
surveys in a mixed-methods approach to explore differences in perceptions between e-learners and
e-learning providers on what constitutes an effective learning environment. The lack of confirmatory
studies among the Australia-focussed articles could be explained by the fact that LMSs may have a
more established presence in Australia than they do in China, and there is a greater need to explore
the impact of these systems on specific situations and applications. Prominent examples in this
review of LMS user-impact studies from Australia, include the study of online student attrition
(Moore and Greenland 2017); Redmond, Devine, and Basson (2014)’s exploration of discipline differ-
entiation in online discussion participation; and the paper by Sheridan, Kotevski, and Dean (2014) on
learner perspectives of online assessments as a mechanism to engage reflective practice. The China-
focused exploratory studies in this review also examined specific applications of LMS use from a user
perspective, such as the study of online MBA programmes and their impact on working mothers
(Kibelloh and Bao 2014); the investigation into integrating MOOC and flipped classroom practice
in a traditional undergraduate course (Li et al. 2015); and the study of e-learning privacy and personal
information disclosure (Yang and Wang 2014). It is noted that in all of the Chinese exploratory studies,
there was a connection to institutions outside China either directly by the inclusion of a researcher
domiciled in an overseas institution, or through the affiliations of local staff. In a study of cross-cultural
collaborative research between British and Chinese academics (Easterby-Smith and Malina 1999)
revealed that ontological and epistemological perspectives tended to converge as experience of
the two groups working together increased. The influence of an outside researcher in the Chinese
studies could have contributed to the choice of an exploratory approach in the research design.
The adoption of a deductive approach to the four confirmatory studies appears to be due to two
factors: firstly a need to validate a theory developed outside China using data in a local context
(Gómez-Rey, Barbera, and Fernández-Navarro (2016); Zhang et al. (2017)); and secondly a desire to
replicate overseas studies within China using similar data collection instruments and methods of
analysis to establish the differences and similarities between two populations of learners (Xu, Du,
and Fan (2014); Zhao and Mei (2016)). One possible benefit of these approaches, is to determine if
foreign models and practices of LMS integration and use, are appropriate for a local Chinese
context. However, a valid criticism of the simple replication of foreign research designs in China is
that it cannot adequately account for the different context and populations in China-based phenom-
ena (Tsui 2004).

Data collection methods

A variety of data collection methods were used in the 23 studies of this review. These were: inter-
views, surveys, observations, focus groups, online discussion forums, and online diaries. Figure 6 dis-
plays the 23 studies and the data collection methods employed. Most studies in this review
principally use only a single data collection method: the most favoured being surveys and interviews.
The use of multiple data sources could strengthen the validity of the findings by ensuring the studies
had achieved ‘data saturation’. Data saturation is achieved when there is enough information to repli-
cate the study, no more new information can be attained, and further coding is not possible (Fusch
and Ness 2015). Of the four studies using multiple data collection strategies (Sadykova (2014); Srid-
haran, Deng, and Kinshuk (2014); Khairudin and Hamid (2015); Li et al. (2015)), interviews and obser-
vational data were used to confirm the results from survey responses. The benefits of triangulation
are well documented and include: increases in comprehensiveness and completeness; the confir-
mation of trends and identification of data inconsistencies; and improved validity and reliability
(Weyers, Strydom, and Huisamen 2014). The obvious disadvantage of including alternate data
sources, particularly if they are of the interview or observation type, is the time and cost required
to collect this data. Given that participants in LMS studies are likely to be geographically isolated,
it is not surprising that labour intensive data collection techniques (i.e. interviews, focus groups
and observations) were not more widely employed.
 INTERNATIONAL JOURNAL OF RESEARCH & METHOD IN EDUCATION 11

Figure 6. Data collection methods identified in the 23 empirical studies by country of focus.

It can also be useful to examine the chosen data collection techniques by their country of focus.
Figure 6 displays the number of data collection methods according to the country of focus.
Both the Australian and Chinese studies strongly favoured interviews and surveys as primary data
collection techniques. Both data collection techniques have their strengths and weaknesses. Inter-
views allow researchers to obtain complete answers to questions in a controlled environment but
are subject to subconscious bias (Alshenqeeti 2014). By contrast, surveys are capable of collecting
data from a wider audience, particularly if they are web-based, but may not adequately represent
the population studied if the response rate is low (Heiervang and Goodman 2011).
Of the 13 instances of data collection methods in the China-focussed studies, surveys were by far
the most popular method: the others being three instances of interviews, and one each of online
diaries, observations, and LMS data. The seven instances of survey use in the China-focussed
studies can be partly attributed to the confirmatory nature of four of the studies, as surveys are excel-
lent tools to efficiently collect large samples of data that can then be analysed against an established
hypothesis. Another possible explanation for the extensive use of surveys is the geographical dis-
persion of the participants in two of the exploratory China-focussed papers.
The other interesting finding displayed in Figure 6, is the lack of any focus groups used in the
China-focussed studies. One possible explanation for this could be cultural:
Chinese people are expected to avoid conflicts and aggressive ways of handling them to protect social face so
that others do not feel their competence is questioned. (Tjosvold and Sun 2002, 144)

Focus groups are moderated discussions of research participants on issues related to a research ques-
tion where opinions and beliefs are carefully recorded (Schneider et al. 2002). Focus groups by their
very nature can be hotbeds of dispute. The researchers of the China-focussed studies may have per-
ceived that the use of such data collection techniques would be ineffective in a Chinese context due a
perceived reluctance of participants to openly communicate. Young people for example, may be dis-
inclined to present views that might contradict older, more experienced colleagues (Joseph 2016). A
possible strategy to mitigate the reluctance of China-based researchers to use this data collection
technique, might be to use online video conferencing that can mask the identity of the participants
to each other, reducing the risk of participant inhibition (Murgado-Armenteros, Torres-Ruiz, and
Vega-Zamora 2012).
12 D. TURNBULL ET AL.

Practical and managerial implications

Educators and institutional administrators in Australia who regularly engage with their Chinese
counterparts already understand that cultural and language differences can impact significantly on
the success or otherwise of joint ventures that they may wish to pursue. Any project that is under-
taken between Australia and China is likely to have a research component to it: be it a student
exchange programme between two institutions, the delivery of an off-shore programme at a
Chinese institution, or the deployment of learning technology in China. For LMS administrators
and decision makers, the way in which information is gathered, analysed, and interpreted determines
the type and quality of information available to make decisions. As outlined in this review, LMS
research designs in China tend to favour quantitative techniques more than in Australia, are often
structured to replicate studies in other countries, and seek to avoid face-to-face data collection
methods such as focus groups. Accordingly, managers of joint Australia-China projects have to be
particularly careful in how they propose to set up the research components of their projects. Research
designs to be employed in collaborative studies between organizations in Australia and China should
be thoroughly discussed and agreed upon by both sides to ensure that research outputs generate
appropriate inputs for the decision-making processes of each organization.

Conclusion
In comparing the findings and significance of LMS research between countries such as Australia and
China, differences in methodologies employed, research approaches (confirmatory or exploratory),
and data collection methods are indicative of practices favoured by researchers in the country of
origin. For example, this review found that there were no purely quantitative LMS studies in Australia,
suggesting that researchers in that country found it necessary to incorporate at least some qualitative
analysis in their research designs. Differences in research approaches between countries can also
indicate whether LMS studies lean towards exploring new ground or testing established theories.
As found in this review, all of the Australia-focused studies were exploratory, while four of the
China-focused studies were confirmatory in design. This suggests a greater propensity in China to
investigate the applicability of prior studies to new situations than in Australia where LMS study
designs appear to be more aligned with a need to investigate emerging phenomena. However,
this literature review has two limitations which could be addressed by further investigation. The
first limitation is the use of English-language only articles from China-based sources. The investi-
gation of publications in Chinese may reveal a different pattern of methodology use than those
included in English language articles published in Western journals. Secondly, this review does not
identify important attributes of each empirical study, such as the geographic location in China or Aus-
tralia, the type of institution featured in the study (for example, university, school, or trade college), or
the characteristics of LMS researchers (gender, education, ethnicity and so forth). These attributes
may be contributing factors to the research design choices in each country. Further studies compar-
ing LMS research designs in Australia, China, and possibly other countries are needed to develop a
more in-depth understanding of the reasons for incorporating particular methodologies, approaches,
and data collection methods into LMS research designs.

Notes
1. MOOC stands for Massive Open Online Courses. Traditional MOOCs use LMSs as their medium to publish learning
materials (Tharindu, Adams, and Williams 2013, 13).
2. NVivo is a software programme used to assist in the collation and analysis of qualitative data.

Disclosure statement
No potential conflict of interest was reported by the author(s).
 INTERNATIONAL JOURNAL OF RESEARCH & METHOD IN EDUCATION 13

ORCID
Darren Turnbull http://orcid.org/0000-0003-0509-8564
Ritesh Chugh http://orcid.org/0000-0003-0061-7206
Jo Luck http://orcid.org/0000-0002-5478-7289

References
Alshenqeeti, Hamza. 2014. “Interviewing as a Data Collection Method: A Critical Review.” English Linguistics Research 3 (1):
39–45.
Beach, Pamela, and Jen McConnel. 2019. “Eye Tracking Methodology for Studying Teacher Learning: A Review of the
Research.” International Journal of Research & Method in Education 42 (5): 485–501. doi:10.1080/1743727X.2018.1496415.
Best, Paul, Roger Manktelow, and Brian Taylor. 2014. “Online Communication, Social Media and Adolescent Wellbeing: A
Systematic Narrative Review.” Children and Youth Services Review 41: 27–36. doi:10.1016/j.childyouth.2014.03.001.
Bottino, Rosa Maria, and Elisabetta Robotti. 2007. “Transforming Classroom Teaching & Learning Through Technology:
Analysis of a Case Study.” Journal of Educational Technology & Society 10 (4): 174–186.
Boyle, Elizabeth A., Thomas M. Connolly, Thomas Hainey, and James M. Boyle. 2012. “Engagement in Digital Entertainment
Games: A Systematic Review.” Computers in Human Behavior 28 (3): 771–780. doi:10.1016/j.chb.2011.11.020.
Brannen, Julia. 2005. “Mixing Methods: The Entry of Qualitative and Quantitative Approaches Into the Research Process.”
International Journal of Social Research Methodology 8 (3): 173–184. doi:10.1080/13645570500154642.
Callan, Victor James, Margaret Alison Johnston, and Alison Louise Poulsen. 2015. “How Organisations are Using Blended
E-Learning to Deliver More Flexible Approaches to Trade Training.” Journal of Vocational Education & Training 67 (3):
294–309. doi:10.1080/13636820.2015.1050445.
Collins, John A., and Bart C. J. M. Fauser. 2005. “Balancing the Strengths of Systematic and Narrative Reviews.” Human
Reproduction Update 11 (2): 103–104. doi:10.1093/humupd/dmh058.
Cook, David A., and Yyvonne Steinert. 2013. “Online Learning for Faculty Development: A Review of the Literature.”
Medical Teacher 35 (11): 930–937. doi:10.3109/0142159X.2013.827328.
Davey, Gareth, Chuan De Lian, and Louise Higgins. 2007. “The University Entrance Examination System in China.” Journal
of Further and Higher Education 31 (4): 385–396. doi:10.1080/03098770701625761.
Department of Education and Training. 2019. International Student Data Monthly Summary. Canberra: Australian
Government Department of Education and Training.
Du, S., Z. Liu, S. Liu, H. Yin, G. Xu, H. Zhang, and A. Wang. 2013. “Web-Based Distance Learning for Nurse Education: A
Systematic Review.” International Nursing Review 60 (2): 167–177. doi:10.1111/inr.12015.
Easterby-Smith, M., and D. Malina. 1999. “Cross-Cultural Collaborative Research: Toward Reflexivity.” Academy of
Management Journal 42 (1): 76–86. doi:10.2307/256875.
Fleischer, Håkan. 2012. “What is Our Current Understanding of One-To-One Computer Projects: A Systematic Narrative
Research Review.” Educational Research Review 7 (2): 107–122. doi:10.1016/j.edurev.2011.11.004.
Fusch, Patricia I., and Lawrence Ness. 2015. “Are We There Yet? Data Saturation in Qualitative Research.” Qualitative Report
20 (9): 1408–1416.
Gómez-Rey, Pilar, Elena Barbera, and Francisco Fernández-Navarro. 2016. “The Impact of Cultural Dimensions on Online
Learning.” Journal of Educational Technology & Society 19 (4): 225–238.
Heiervang, Einar, and Robert Goodman. 2011. “Advantages and Limitations of Web-Based Surveys: Evidence from a Child
Mental Health Survey.” Social Psychiatry and Psychiatric Epidemiology 46 (1): 69–76. doi:10.1007/s00127-009-0171-9.
Henry, Brandon Michael, Bendik Skinningsrud, Jens Vikse, Przemysław A. Pękala, Jerzy A. Walocha, Marious Loukas, R.
Shane Tubbs, and Krzysztof A. Tomaszewski. 2018. “Systematic Reviews Versus Narrative Reviews in Clinical
Anatomy: Methodological Approaches in the Era of Evidence-Based Anatomy.” Clinical Anatomy 31 (3): 364–367.
doi:10.1002/ca.23042.
Joseph, David. 2016. “How Do Focus Groups in China Differ?” Market Research in China. https://www.hubofchina.com/
2016/02/21/what-do-we-need-to-bear-in-mind-when-conducting-focus-groups-in-china/.
Kattoua, Tagreed, Musa Al-Lozi, and Ala’aldin Alrowwad. 2016. “A Review of Literature on E-Learning Systems in Higher
Education.” International Journal of Business Management & Economic Research 7 (5): 754–762.
Kent, Carmel, and Amit Rechavi. 2018. “Deconstructing Online Social Learning: Network Analysis of the Creation,
Consumption and Organization Types of Interactions.” International Journal of Research & Method in Education, 1–
22. doi:10.1080/1743727X.2018.1524867.
Khairudin, Norhaiza, and Mohd Noor Abdul Hamid. 2015. “The Development of a LMS Decision Making Model: Evaluating
the Importance of Non-Financial Measures in LMS Decision Making at Universities.” Paper presented at the
International Conference on E-Commerce, Kuching, October 20–22.
Kibelloh, Mboni, and Yukun Bao. 2014. “Can Online MBA Programmes Allow Professional Working Mothers to Balance
Work, Family, and Career Progression? A Case Study in China.” The Asia-Pacific Education Researcher 23 (2): 249–
259. doi:10.1007/s40299-013-0101-1.
14 D. TURNBULL ET AL.

Kilburn, Daniel, and Jonathan Earley. 2015. “Disqus Website-Based Commenting as an E-Research Method: Engaging
Doctoral and Early-Career Academic Learners in Educational Research.” International Journal of Research & Method
in Education 38 (3): 288–303. doi:10.1080/1743727X.2015.1026253.
Kriz, Anton, Evert Gummesson, and Ali Quazi. 2013. “Methodology Meets Culture.” International Journal of Cross Cultural
Management 14 (1): 27–46. doi:10.1177/1470595813493265.
Li, Yan, Muhua Zhang, Curtis J. Bonk, and Yuqing Guo. 2015. “Integrating MOOC and Flipped Classroom Practice in a
Traditional Undergraduate Course: Students’ Experience and Perceptions.” International Journal of Emerging
Technologies in Learning (iJET) 10 (6): 4–10. doi:10.3991/ijet.v10i6.4708.
Lyytinen, Kalle, Richard Baskerville, Juhani Iivari, and Dov Te’eni. 2007. “Why the Old World Cannot Publish? Overcoming
Challenges in Publishing High-Impact IS Research.” European Journal of Information Systems 16 (4): 317–326.
MacKenzie, Andrew, Milica Muminovic, and Karin Oerlemans. 2017. “The Intentional Use of Learning Management
Systems (LMS) to Improve Outcomes in Studio.” Journal of Problem Based Learning in Higher Education 5 (1): 47–63.
McCutcheon, K., M. Lohan, M. Traynor, and D. Martin. 2015. “A Systematic Review Evaluating the Impact of Online or
Blended Learning vs. Face-To-Face Learning of Clinical Skills in Undergraduate Nurse Education.” Journal of
Advanced Nursing 71 (2): 255–270. doi:10.1111/jan.12509.
Mirriahi, Negin, Dennis Alonzo, Simon McIntyre, Giedre Kligyte, and Bob Fox. 2015. “Blended Learning Innovations:
Leadership and Change in one Australian Institution.” International Journal of Education and Development Using
Information and Communication Technology 11 (1): 4–16.
Moore, Catherine, and Steven Greenland. 2017. “Employment-Driven Online Student Attrition and the Assessment Policy
Divide: An Australian Open-Access Higher Education Perspective.” Journal of Open, Flexible and Distance Learning 21
(1): 52–62.
Murgado-Armenteros, Eva María, Francisco José Torres-Ruiz, and Manuela Vega-Zamora. 2012. “Differences Between
Online and Face to Face Focus Groups: Viewed Through Two Appproaches.” Journal of Theoretical & Applied
Electronic Commerce Research 7 (2): 73–86. doi:10.4067/S0718-18762012000200008.
Ng, Wan, and Elizabeth Angstmann. 2017. “Promoting Physics Literacy Through Enquiry-Based Learning Online.” Journal
of Education in Science, Environment and Health 3 (2): 183–195.
Oliveira, Paulo Cristiano de, Cristiano José C. de A. Cunha, and Marina Keiko Nakayama. 2016. “Learning Management
Systems (LMS) and E-Learning Management: An Integrative Review and Research Agenda.” Journal of Information
Systems and Technology Management 13 (2): 157–180. doi:10.4301/s1807-17752016000200001.
Onwuegbuzie, Anthony J., and Nancy L. Leech. 2005. “On Becoming a Pragmatic Researcher: The Importance of
Combining Quantitative and Qualitative Research Methodologies.” International Journal of Social Research
Methodology 8 (5): 375–387. doi:10.1080/13645570500402447.
Redmond, Petrea, Jo Devine, and Marita Basson. 2014. “Exploring Discipline Differentiation in Online Discussion
Participation.” Australasian Journal of Educational Technology 30 (2): 122–135.
Robinson, Jo, Georgina Cox, Eleanor Bailey, Sarah Hetrick, Maria Rodrigues, Steve Fisher, and Helen Herrman. 2016. “Social
Media and Suicide Prevention: A Systematic Review.” Early Intervention in Psychiatry 10 (2): 103–121.
Rother, Edna Terezinha. 2007. “Revisão sistemática X revisão narrativa.” Acta Paulista de Enfermagem 20: v–vi.
Sabharwal, R., M. R. Hossain, R. Chugh, and M. Wells. 2018. “Learning Management Systems in the Workplace: A Literature
Review.” Paper presented at the 2018 IEEE International Conference on Teaching, Assessment, and Learning for
Engineering (TALE), 387–393. Wollongong, December 4–7.
Sadykova, Gulnara. 2014. “Mediating Knowledge Through Peer-To-Peer Interaction in a Multicultural Online Learning
Environment: A Case Study of International Students in the US.” International Review of Research in Open &
Distance Learning 15 (3): 25–49.
Salmon, Gilly, Janet Gregory, Kulari Lokuge Dona, and Bella Ross. 2015. “Experiential Online Development for Educators:
The Example of the Carpe Diem MOOC.” British Journal of Educational Technology 46 (3): 542–556.
Sarmiento, Irene G., Chelsea Olson, Geck Hong Yeo, Y. Anthony Chen, Catalina L. Toma, B. Bradford Brown, Amy Bellmore,
and Marie-Louise Mares. 2018. “How Does Social Media Use Relate to Adolescents’ Internalizing Symptoms?
Conclusions from a Systematic Narrative Review.” Adolescent Research Review 1–24. doi:10.1007/s40894-018-0095-2.
Schneider, Sid J., Jeffrey Kerwin, Joy Frechtling, and Benjamin A. Vivari. 2002. “Characteristics of the Discussion in Online
and Face-To-Face Focus Groups.” Social Science Computer Review 20 (1): 31–42.
Schulze, Salomé. 2003. “Views on the Combination of Quantitative and Qualitative Research Approaches.” Progressio 25
(2): 8–20.
Sheridan, Lynnaire, Suzanne Kotevski, and Bonnie Amelia Dean. 2014. “Learner Perspectives on Online Assessments as a
Mechanism to Engage in Reflective Practice.” Asia-Pacific Journal of Cooperative Education 15 (4): 335–345.
Sinclair, Peter M., Ashley Kable, Tracy Levett-Jones, and Debbie Booth. 2016. “The Effectiveness of Internet-Based E-
Learning on Clinician Behaviour and Patient Outcomes: A Systematic Review.” International Journal of Nursing
Studies 57: 70–81.
Sridharan, Bhavani, Hepu Deng, and Kinshuk. 2014. “Does Supply Always Come on the Heels of Demand? Matches and
Mismatches in E-Learning.” Issues in Educational Research 24 (3): 260–280.
Stokes, Peter J. 1999. “E-Learning: Education Businesses Transform Schooling.” Paper presented at The Forum on
Technology in Education: Envisioning the Future, Washington, December 1–2.
 INTERNATIONAL JOURNAL OF RESEARCH & METHOD IN EDUCATION 15

Sutherland, Rosamund, V. Armstrong, S. Barnes, Richard Brawn, N. Breeze, M. Gall, Sasha Matthewman, F. Olivero, A.
Taylor, and Pat Triggs. 2004. “Transforming Teaching and Learning: Embedding ICT Into Everyday Classroom
Practices.” Journal of Computer Assisted Learning 20 (6): 413–425.
Tharindu, Rekha, Andrew Adams, and Shirley Williams. 2013. “MOOCs: A Systematic Study of the Published Literature
2008–2012.” International Review of Research in Open and Distance Learning 14 (3), doi:10.19173/irrodl.v14i3.1455.
Tjosvold, Deon, and Haifa F. Sun. 2002. “Understanding Conflict Avoidance: Relationship, Motivations, Actions, and
Consequences.” International Journal of Conflict Management 13 (2): 142–164. doi:10.1108/eb022872.
Trochim, William M. K. 2006. “Deduction & Induction.” http://www.socialresearchmethods.net/kb/dedind.htm.
Tsui, Anne S. 2004. “Contributing to Global Management Knowledge: A Case for High Quality Indigenous Research.” Asia
Pacific Journal of Management 21 (4): 491–513. doi:10.1023/B:APJM.0000048715.35108.a7.
Turnbull, Darren, Ritesh Chugh, and Jo Luck. 2019. “Learning Management Systems: An Overview.” In Encyclopedia of
Education and Information Technologies, edited by A. Tatnall. Cham: Springer Nature. doi:10.1007/978-3-319-60013-
0_248-1.
Wang, Jingzheng, and Yuanbing Duan. 2014. “Conducting Web-Based Formative Assessment Reform for ODL Students: A
Case Study.” Journal of Language Teaching & Research 5 (3): 654–662. doi:10.4304/jltr.5.3.654-662.
Weyers, Mike, Herman Strydom, and Arnel Huisamen. 2014. “Triangulation in Social Work Research: The Theory and
Examples of Its Practical Application.” Social Work/Maatskaplike Werk 44 (2), doi:10.15270/44-2-251.
Xu, Jianzhong, Jianxia Du, and Xitao Fan. 2014. “Emotion Management in Online Groupwork Reported by Chinese
Students.” Educational Technology Research and Development 62 (6): 795–819. doi:10.1007/s11423-014-9359-0.
Yang, Fang, and Shudong Wang. 2014. “Students’ Perception Toward Personal Information and Privacy Disclosure in
E-Learning.” Turkish Online Journal of Educational Technology - TOJET 13 (1): 207–216.
Zhang, Min, Shuaijun Yin, Meifen Luo, and Weiwei Yan. 2017. “Learner Control, User Characteristics, Platform Difference,
and Their Role in Adoption Intention for MOOC Learning in China.” Australasian Journal of Educational Technology 33
(1): 114–133.
Zhao, Chunjuan, and Zongxiang Mei. 2016. “A Case Study of American and Chinese College Students’ Motivation
Differences in Online Learning Environment.” Journal of Education and Learning 5 (4): 104–112.
Zhao, Yong, Gaoming Zhang, Wenzhong Yang, David Kirkland, Xue Han, and Jianwei Zhang. 2008. “A Comparative Study
of Educational Research in China and the United States.” Asia Pacific Journal of Education 28 (1): 1–17. doi:10.1080/
02188790701849826.