Received July 28, 2019, accepted August 26, 2019, date of publication August 30, 2019, date of current

version September 13, 2019.

Digital Object Identifier 10.1109/ACCESS.2019.2938561

A Systematic Mapping Study of the Empirical

MOOC Literature
RASHEED ABUBAKAR RASHEED 1,2 , AMIRRUDIN KAMSIN 1, NOR ANIZA ABDULLAH 1, (Member, IEEE),
ABUBAKAR ZAKARI 1,4 , AND KHALID HARUNA 1,3
1 Facultyof Computer Science and Information Technology, University of Malaya, Kuala Lumpur 50603, Malaysia
2 Department of Software Engineering, Faculty of Computer Science and Information Technology, Bayero University, Kano 3011, Nigeria
3 Department of Computer Science, Faculty of Computer Science and Information Technology, Bayero University, Kano 3011, Nigeria
4 Department of Computer Science, Kano University of Science and Technology, Wudil 3244, Nigeria

Corresponding author: Nor Aniza Abdullah (noraniza@um.edu.my)

This work was supported by the University of Malaya Research Grant (UMRG) under Project RP043A-17HNE.

ABSTRACT Massive open online courses (MOOCs) have revolutionized todays education by offering a
global accessible form of online learning. Over the years, MOOCs have been an attractive research area and
have yielded an ample amount of research publications. However, the existing review studies in MOOCs
are characterized by short year coverage or focusing on a specific theme. As such, a systematic mapping
methodology was adopted to provide a fine-grain overview of MOOC research domain by identifying the
quantity, types of research, available results and publication trends in educational aspects of MOOCs from
2009 to 2018. Key findings show that I) MOOC research have been on the rise since MOOCs became
mainstream in 2011. II) MOOC research largely resides in the United States and few European countries.
II) Most of MOOC studies focused on addressing learners’ completion/dropout/retention. In addition,
we proposed some recommendations for future research on MOOCs.

INDEX TERMS E-learning, distance education, massive open online course, MOOC, systematic mapping
study.

I. INTRODUCTION
Since 2008, Massive Open Online Courses (MOOCs) have
gained popularity [1] and revolutionized the whole edu-
cational sector by offering new opportunities in education
through online courses targeting large scale global accessi-
bility in open online resources [2]. This new form of online
learning is said to be an expansion of e-learning and distance
education [3]. ‘The New York Times’ magazine declared the
year 2012 as ‘‘The Year of the MOOC’’ [4], due to its evolve-
ment from an education axiom to revolutionizing education
at global scale. It was estimated that MOOCs have more
than 58 million registered users after nearly a decade, and
hundreds of universities are offering thousands of courses that
resides on different platform such as the popular Cousera,
edX, Udacity, FutureLearn, XeutangX; and others such as
MirindaX which offers MOOCs in Spanish [5].
The democratic distinction of MOOCs in offering free
access to online materials and certificates, in particular, dur-
The associate editor coordinating the review of this article and approving
it for publication was Utku Kose.
ing its first years have changed along the line as some of the
key providers of MOOC courses such as Coursera and edX
have gradually changed their business model by dropping
their offerings of free certificates [6], and in some courses
users who do not pay money may not be able to access the
whole package of course materials [7].
The initiation of MOOCs have created an entirely new
domain for research, outlining several avenues of research
opportunities for educational institutions and e-learning
researchers and practitioners [8], [9]. Over the years, MOOCs
emerged as three distinct types namely: cMOOCs, xMOOCs
and hybrid MOOCs. In cMOOCs environment, users share
information and engage in a joint teaching and learning
experience, whereas xMOOCs is structured so that each
user is either a learner or a teacher. Hybrid MOOCs later
emerged with the aim of affording the benefits of cMOOC
and xMOOC learning experiences.
Through this time, MOOCs have become an attractive
research area and have yield countless publications. Various
themes and issues with MOOCs have progressively emerged
thereby reshaping and redefining researchers focuses and

VOLUME 7, 2019 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/ 124809
 R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature
interests in the research area. For example, there has
been a substantial growth in research pertaining learners’
dropout/completion related issues e.g. [10]–[13], pedagogical
design of MOOCs e.g. [14], issues on learners satisfaction
with MOOCs for learning e.g. [15] and learners engagement
on MOOC platforms e.g. [16], [17]. In addition, there have
been a considerable number of review studies on MOOCs,
demonstrating the gradual rise and development of MOOC
research by offering categorization and characterization of
various emergent themes, topics, trends, research interests,
issues/challenges etc. However, most of these review studies
are characterized by short year coverage, typically reviewing
MOOC publications of 2 to 3 years’ span, for example,
the review studies of [9], [18], [19]; or characterized by
focusing on a specific subject/issue, for example the review
on personalization of MOOCs by [20]. To date, there has
not been a generalized review study that covers the entire
MOOCs by providing a universal picture of MOOCs since
MOOCs became mainstream nearly a decade ago.
As MOOCs have been in existence since before 2008 when
it got global attention, there have also been MOOC publica-
tions in other disciplines such as business, as well as other
publications that investigate issues such as software engi-
neers, librarian issues, platform development etc. Hence, our
study aims to focus only on educational aspect publications
of MOOCs.
This study tempts to fill this gap by extensively analyzing
relevant empirical MOOC studies published from the last
decade. This study attempts to motivate and aid researchers
in understanding new trends in MOOCs, in terms of research
methods, research focus, methods of data collection, data
analysis frequently used, publication trends, key publication
venues as well as proposing future research recommenda-
tions. We adopted an evidence-based systematic mapping
methodology that ensures coverage of state-of-the-art lit-
erature, and followed an unbiased and systematic evalua-
tion process in order to ensure transparency and coverage
of the relevant empirical studies in educational aspects of
MOOCs over the last decade. This study begins with the
formation of a systematic mapping process consisting of a
systematic literature search; selection process which includes
inclusion/exclusion criteria; and data extraction and synthesis
strategies. We also considered the overall publication trends
that defines and shapes MOOC research domain. Finally,
we offer some research recommendations and proposed a
number solutions guided by the findings of this study and also
by considering other online learning research areas such as
blended learning.
II. RELATED WORK
Over the years, MOOCs have attracted numerous types of
reviews such as systematic reviews and critical reviews.
The first major review study on MOOCs in 2013 examines
published MOOC literature from 2008 to 2012, and cate-
gorized the literature into eight different interest areas as:
introductory, case studies, concept, educational theory, par-
124810
ticipants focus, provider focus, technology and others [19].
The findings offered an insight into MOOCs research mainly
focusing on learners. In 2013, however, the popularity and
worldwide growth of MOOCs possibly led researchers in
reviewing MOOC literature to hypothesize where research
on MOOCs is heading, by revealing five research themes
that could form a framework of future researches on MOOCs
as: MOOC design and curriculum; students engagement and
learning success; self-regulated learning and social learning;
motivation, attitude and success criteria; and social network
analysis and networked learning [21]. Although MOOCs
have emerged as a way of offering quality education glob-
ally, there were concerns over its efficiency. A review study
examining MOOC design, delivery and assessment found that
most MOOCs provide the same content for all learners and
therefore are not learner centric; the effectiveness of tutoring
is generally poor; and peer-based evaluation and feedbacks
are usually unprofessional [22]
In 2014, research was carried out to examine the challenges
and motivations of using MOOCs from learners perspec-
tives [23]. Personal curiosity about MOOCs, personal chal-
lenge, extending knowledge, and obtaining certificate were
the four main motives behind enrollment in MOOC courses.
Personal rewards, curiosity, and sense of autism were the
three main primary motives for MOOC instructors. However,
the main challenges facing MOOC instructors were found to
be: fostering an online interaction, time and cost challenges,
lack of physical contact with the students (live audience)
and assessment challenges. Researchers carried out another
review in the same year to examine MOOCs influence on
higher education using disruptive innovation theory [24]. The
study underlined how MOOCs have affected different aspects
of research on MOOCs such as the delivery methods, business
models and the common pedagogical approaches. Moreover,
the authors [24] suggested the need to investigate the level and
extent of disruption brought by MOOCs on today’s education.
In the same year also, a number of reviews aimed at identi-
fying trending themes on MOOC literature was carried out.
For instance, [25] explored MOOC literature from 2008 to
2014 by reviewing the history of MOOCs; MOOCs design;
different types of MOOCs; assessment and challenges in
MOOCs. Another example is the study of [26] that examines
MOOC empirical studies from 2009 to 2013 which captured
empirical MOOCs in two phases. The study by [26] identi-
fied themes in phase one (2009 to 2011) which involves the
development of technological experiments and innovations in
MOOCs, and connectivism theory in cMOOCs. The second
phase involves the emergence of themes such as xMOOCs;
learning analytics and evaluation/assessments in MOOCs;
and development of MOOCs pedagogy and platforms. Sub-
sequently, themes such as learners’ dropout rates emerged.
In 2015, [20] explored the literature on personalization
in MOOCs. The study identified the needs, current states
and efforts made to personalize learning in MOOCs. Per-
sonalized assessment and feedback; personalized learning
path; personalized forum thread; and recommendation
VOLUME 7, 2019
R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature
service for learning materials were used for implement-
ing personalization in MOOCs [20]. Another review study
in the same year examined methodological approaches
adopted in MOOCs by reviewing MOOC literature from
2008 to 2014 [27]. The review study shows that previ-
ous MOOC studies have focused more on case studies
and theoretical research, as a result, neglecting research
on MOOC methodologies and research paradigms. Simi-
larly, [28] reviewed MOOC literature from an interdisci-
plinary perspective from 2013 to 2015 and found that MOOC
researchers were from computer science and education
fields.
In 2016, MOOC research methods, publication outlets,
authors location and citations from 2013 to 2015 was exam-
ined. The authors found that MOOC research papers were not
widely cited; most of MOOC researchers are from Europe
and North America; MOOC researchers commonly use quan-
titative research methods; and there are relatively very few
research focusing on MOOC instructors [9].
In 2017, researchers found interviews, surveys and log
files as the most commonly adopted data collection methods
in empirical MOOC studies through examining the research
methods employed in MOOC research [8]. Another review
study in the same year found that research patterns and trends
in MOOC studies encompass three main research areas: the-
ories and models and theories, learners’ characteristics and
instructional design [29].
In 2018, researchers examined MOOC discussion forums
by reviewing the literature from 2013 – 2017 [30]. The study
focused on covering two analysis perspectives: descriptive
analysis and content analysis. The researchers found that
majority of the selected MOOC articles focused on research
exploration, and few directly targeted MOOC learners. On the
other hand, the content analysis of the papers stretches a sum-
mary of the main challenges of research on discussion forums
in MOOCs. Moreover, some interesting review studies exam-
ines issues relating to the impact of MOOCs on social groups
and population identity. For instance, [31] examined how
MOOCs benefit the socially privileged in comparison to
underprivileged social groups.
Although MOOCs were instigated to enhance social
mobility, researchers have suggested that MOOCs mainly
benefit privileged groups and could even widen the gap
between privileged and unprivileged populations. Also,
researchers have suggested certain types of MOOCs that
might serve less privileged learners more while specifying
FIGURE 1. Systematic mapping process.
VOLUME 7, 2019
formal barriers that might make MOOCs less accessible to
underprivileged learners. In addition, the rising concern that
MOOCs need technological infrastructure as well as learners
to have language fluency and digital skill has given rise to
the doubt on the linkage between MOOCs and the possible
solution for digital divide. Reference [32] argued that learn-
ers’ literacy on technologies plays a key role in sustaining a
regular access to online based learning resources. The study
examined the trend of MOOCs in the sense of having both
the positive and negative influence in providing solutions for
narrowing digital gap [32].
Researchers also reviewed a number of MOOC extant
literature which focused on analyzing peer assessments, com-
pletion rates and blended learning with regard to MOOCs
from multiple perspectives [33]. In addition, 2018 witnessed
another familiar review study that examines research meth-
ods, research topics and publication trends of the empirical
MOOC literature from 2014 to 2016 [18].
Majority of the review studies on MOOCs highlighted
above centers around learners, instructors and instructional
design. It is quite interesting to know that the unprece-
dented interest and involvement of the general public on
MOOCs innovation has led to a number of other unfamiliar
review studies such as the reviews on MOOC related media
reports [34], and the portrayal of MOOCs in Irish newspapers
aimed at better understanding the public discourse around
MOOCs [35].
The review studies above show that research trends and
topics are rapidly evolving. Although majority of MOOC
studies at the beginning were largely theoretical and concep-
tual studies, as years pass by, more empirical studies, themes
and topics have emerged. This study aims to compose an
overview of the empirical MOOC literature from the last
decade.
III. RESEARCH METHODS
A. SYSTEMATIC MAPPING PROCESS
A systematic mapping study typically explores existing lit-
erature to provide an overview of the results and types of
research that has been carried out in a research domain. Sys-
tematic mapping studies offer an overview of a research area
by identifying the quantity, types of research and available
results within it. It typically shows publication trends by map-
ping the frequencies of publications over time. In addition,
it provides a fine-grain overview of the research domain.
Petersen et al. [36] stated that ‘‘A systematic mapping study
124811
 R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature
TABLE 1. Research questions.
provides a structure of the type of research reports and results
that have been published by categorizing them and often
gives a visual summary, the map, of its results’’. On the
other hand, a systematic literature review explores relevant
available research in a research domain and reviews them in-
depth, evaluates, interprets and describes their methodology
and results [37].
Systematic mapping studies are typically popular in soft-
ware engineering domain. Over the years, researchers have
adopted and followed Petersen’s et al. [36] systematic map-
ping study guidelines for conducting such reviews. Although,
systematic mapping studies were not limited to software
engineering domain, several other research domains have
acknowledged such methodology and researchers in those
domains have carried out interesting and useful systematic
mapping reviews by adopting Petersen’s guidelines. This
approach has proven to be beneficial in providing a fine-
grain description of a research domain or a sub-domain
e.g. [38]–[40]. Another common practice of recent in con-
ducting systematic mapping studies is by adopting Petersen’s
systematic mapping study guidelines [36] in synergy with
kitchenham’s systematic literature review guidelines [41].
This approach is largely adopted to further enrich and
make systematic mapping studies more complete, and to
derive meaningful conclusions e.g. (see [42]). Therefore, our
study adopts Petersen’s systematic mapping study guide-
lines [36] in collaboration with kitchenham’s systematic lit-
erature review guidelines [41].
A systematic mapping process mainly consists of five
component phases as illustrated in figure 1. The first phase
is the definition of research questions - RQs. The next phase
involves the definition of a ‘search strategy’ and the process
of selecting relevant studies. The thirds phase involves the
screening and refinement of the selected studies, usually
124812
based on a set of inclusion/exclusion criteria. The fourth
phase involves the classification scheme. A classification
scheme is the blueprint or anatomy of the systematic map-
ping study. The classification is usually initiated based on
the observation from the existing baseline studies and also
from the knowledge of the researchers in a research area.
The final phase consists of selecting the final selection of
studies and mapping them into the classification scheme
(usually a bubble plot graph) to demonstrate research trends.
Figure 1 gives an illustration of the systematic mapping
process.
B. RESEARCH QUESTIONS
The aim of this study is to identify, understand and provide
a fine-grain representation of the current state of MOOC
research. As such, our focus is on how and what MOOC
researchers mainly investigate; what MOOC researchers
focus on; how has MOOC research progressed over the years;
and the methodology that MOOC researchers adopted in
offering feasible solutions. In addition, the formulation of our
research questions was also inspired from other influential
MOOC studies such as [18] and [8]. Similarly, we examined
the significance of a formulated research question and its
potential in defining or expanding a new avenue in MOOC
research. The research questions are presented in table 1.
C. LITERATURE SEARCH PROCESS
Four (4) electronic databases: Web of Science (WoS), Science
Direct, IEEEXplore and ACM Digital Library were con-
sulted for MOOC literature. We did not consider Springerlink
database because we don’t have full access to the elec-
tronic articles. Secondly, most of the articles in Springer-
link were inaccessible; and only displaying article’s title
VOLUME 7, 2019
R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature
TABLE 2. Database study selection process.
and abstract. In addition, almost all of the influential and
relevant Springerlink articles were accessible via Web of Sci-
ence (WoS) database. Similarly, we also decide not to include
Google Scholar, as it is characterised by overlapping and
low precision results compared to other databases. Moreover,
Google Scholar does not provide ‘advanced search option’;
and most of the relevant journals deemed as ‘high impact’
that are found in Google Scholar were accessible in the four
electronic databses mentioned above.
Formulation of precise search terms is necessary in obtain-
ing the relevant studies in any systematic reviews as well as
systematic mapping studies. For MOOCs, the search string
was straight forward - ‘‘MOOCs’’ or ‘‘Massive Online Open
Course(s)’’ (e.g. see [18]). ‘Massive Open Online Course(s)’
and ‘MOOCs’ were used in screen titles, abstracts and key-
words on the studies from the identified databases (ACM Dig-
ital Library, IEEEXplore, Science Direct and Web of Science
(WoS)) for the literature extraction. As different databases
have different interfaces for command search and advanced
search, our defined search terms were carefully keyed into
each of the four databases that best suits each search interface
for optimization and extraction of all the relevant studies.
Table 2 shows the search process with the initial results and
final results from each database. Table 2 also provides a
summary of the search process and the number of studies
obtained.
It is crucial to indicate that we conducted a decade study by
examining studies from 2009 – 2018, as decade studies have
been a common practice for conducting systematic mapping
studies.
1) HARMONIZATION
For efficiency concerns in the selection process, we began
by integrating and cleaning the obtained results from any
duplicate identified through the title, author and year. We used
Microsoft Excel in identifying and eliminating these dupli-
cates. The obtained studies were distributed equally among
three members of our team of researchers. Full text versions
of the documents were downloaded and then uploaded to
a shared Google drive. Specifically, each researcher read
through the article first to decide whether the article is
deemed fit to be consideration.
2) VOTING
We adopted voting procedure inspired by [43]. The vot-
ing exercise would ensure that each article has undergone
VOLUME 7, 2019
thorough check by strictly considering the inclusion and
exclusion criteria set for the selection process, deemed as
relevant or irrelevant. The inclusion and exclusion criteria
(see Table 3) guided the decision making for those papers
that were not immediately agreed. In addition, a workshop
was set to resolve conflicting studies and reaching conclu-
sions on the selection process disagreements. After the initial
voting, the selection was reviewed by a third researcher for
confirmation. In some cases, the most experienced member
of the research team on MOOCs was called upon to decide
the inclusion or exclusion of a study, as empirical studies
in MOOCs cover a wide and diverse set of dimensions and
methods. In addition, the most experienced member of the
research team helps the other researchers in fully extracting
the required information and the categorization of each study
article.
3) INTEGRATION
Finally, we excluded the studies that were deemed irrele-
vant/not qualified to be in the final list of our studies. It is vital
mentioning that we have excluded many IEEE conference
papers, as many of these studies have not met our inclusion
criteria.
4) INCLUSION AND EXCLUSION FRAMEWORK
Inclusion and exclusion criteria (see Table 3), was employed
as a framework for selecting relevant studies from our MOOC
literature pool for answering our research questions. The
inclusion and exclusion criteria was carefully discussed,
designed and agreed among all the researchers. In total,
we obtained 311 empirical studies. Table 3 provides a break-
down of the inclusion and exclusion criteria.
Inclusion/Exclusion criteria has been one of the key strate-
gical guideline found both in Petersen’s et al. [36] and
Kitchenham’s et al. [44] systematic mapping and systematic
review guidelines respectively. In addition, it is an established
practice for synthesizing and filtering the unwanted/irrelevant
set of studies in a given domain/sub-domain, thereby fully
focusing a study on a given direction.
D. CLASSIFICATION PROCESS
We developed a classification scheme from Petersen [36]
guidelines. From the 311 finalized studies, the researchers
examined the title, abstracts for keywords and concepts in
that reflects the research method employed in the paper,
124813
 R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature

TABLE 3. Inclusion and exclusion criteria.

research focus, contributions of the paper (contribution facet), liarity of our empirical research, we only consider evaluation,
research facet and publication outlets such as (publication validation and solution research facets for this study. Ref-
year, geographical distribution (country), publication fora, erence [36] define evaluation research as ‘‘Techniques are
affiliation etc.). In addition, researchers may opt to read other implemented in practice and an evaluation of the technique
parts of the paper such as introduction and conclusions to is conducted. That means, it is shown how the technique
fully identify the characteristics of the research paper. In some is implemented in practice (solution implementation) and
cases, the researchers had to thoroughly read the whole paper what are the consequences of the implementation in terms
and study the figures in order to deduct and identify the of benefits and drawbacks (implementation evaluation). This
various characteristics of the paper for classification. The also includes to identify problems in industry’’; validation
characteristics of each paper is documented on a printed data research as ‘‘Techniques investigated are novel and have
extraction form. Figure 2 illustrates the classification process. not yet been implemented in practice. Techniques used are
The first classification involves categorizing the selected for example experiments, i.e., work done in the lab’’; and
studies into the types of research methods employed. The solution proposal as ‘‘A solution for a problem is proposed,
three main research methods employed in empirical MOOCs the solution can be either novel or a significant extension of
are qualitative, quantitative and mixed research methods. an existing technique. The potential benefits and the applica-
Subsequently, we extended the classification/categorization bility of the solution is shown by a small example or a good
by identifying the commonly used data collection methods line of argumentation.’’
and the commonly used data analysis methods employed in
our selected studies.
The second classification involves classifying the selected
studies based on the study focuses. We found that
most MOOC studies have been focusing on learners,
tutors/instructors, and instructional design. In addition,
we also identified the specific topical focus of the MOOC
study such as dropout/completion, assessments/evaluation,
motivation etc. For example, the most common topical focus
in MOOCs is dropout of learners (see [3]).
The third classification inspired by [36] explores and
classify primary selected studies according to the existing
types of research approaches. Reference [36] suggested that
research facets or study types comprise of validation research
evaluation research, philosophical paper, solution proposal,
opinion papers and experience report. Considering the pecu- FIGURE 2. Classification process.

124814 VOLUME 7, 2019

R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature
The fourth is identifying the contribution facet inspired
by [36], which mainly consists of the various contributions
that our selected studies proposed. From our knowledge of
MOOCs and inspiration from other MOOC studies such
as [43] and [45], we classified contribution facets into the-
ory, tool/system, lesson learned, model, algorithm, approach,
framework, method and suggestions/guidelines/advice. The
classification of our contribution facets and more importantly
the ‘lesson learned’ contribution, - which entails the set of
outcomes from obtained results of a study, was inspired from
the study of [43], which categorized contribution facets into
model, theory, framework, guidelines, advice, lesson learned
and tool following the guidelines from the study of [45].
Lastly, we identified where MOOC studies were mostly
published by examining our selected studies publication
dates, publication fora, citation counts and geographical
distributions.
IV. RESULTS
RQ1: What are the research methods used in MOOC studies?
Research methods can be categorized into quantitative,
qualitative and mixed methods. Quantitative research meth-
ods are used in quantifying a problem by way of numerical
data – numbers. Quantitative data is typically statistical and
more structured than qualitative. Example of the most com-
mon quantitative data collection method is surveys, through a
structured questionnaire. Secondly, qualitative method seeks
to describe a topic rather than to quantify it. Although qualita-
tive provides an in-depth understanding of the research ques-
tion, but the data is harder to analyze or make conclusions
compared to quantitative methods. Qualitative data collection
methods vary using unstructured or semi-structured tech-
niques. Example of the common qualitative methods include
interviews, focus group discussions, ethnography, and obser-
vations. The sample size of the respondents is relatively small
compared to quantitative, and are usually expected to fill a
given quota. It is important to note that our Study research
paradigms can be different from qualitative, quantitative and
mixed methods. For instance, learner analytics and social
network analysis can take a different form of categorization.
However, we categorized each study that best fit into our three
research paradigms.
Based on our analysis of the general research methods
used in the 311 selected studies, 158 studies (53%) used
quantitative research methods, 64 studies (17%) used qual-
itative and 89 studies (30%) were mixed methods (using both
quantitative and qualitative) as shown in figure 3.
FIGURE 3. Research methods.
VOLUME 7, 2019
RQ1.1 What are the number of data collection methods
used in MOOC studies?
From our results, we categorized the data collection meth-
ods into one, two, three and more than three. Majority of
the studies use one data collection method (234 = 75%).
Secondly, (57 = 18%) of the studies used two data collection
methods, but only (12 = 4%) and (8 = 3%) used 3 and more
than 3 data collection methods respectively. We also found
few studies that used four or five different method of data
collection which were automatically categorized under the
‘more than three’ method of data collection. For example,
the study of [46] that reports on tools for pioneering online
design teaching in a MOOC format employs a series of
surveys and interviews for data collection. Figure 4 gives an
illustration of the number of data collection figures.
RQ1.2 What are the data collection methods used in
MOOC studies?
Data collection in MOOCs employs numerous methods.
In this study, we are interested in identifying the method
employed in collecting/eliciting MOOC empirical data, and
not the type of data collected. We identified the most
frequently used data collection methods which are seven
in number. From all the data collection methods, survey
(n = 175) was the most frequently used data collection
method. Although, some of our selected studies mentioned
‘survey’, while other studies were more specific in describ-
ing the survey as ‘questionnaire’ or ‘questionnaire survey’.
In addition, we found that most of the studies that employed
survey data collection method uses descriptive statistics and
regression method of data analysis. Interviewing has also
been employed in MOOC empirical research with (n = 84).
We found only two studies that conducted focus group inter-
views while the remaining were standard interviews. Thirdly,
a large number of empirical MOOC studies elicited their data
from their MOOC platforms (n = 96). We also noted that
most of these studies obtained the dataset from Coursera,
Udacity, EdX, FutureLearn and XeutangX MOOCs platform.
Other common methods of data collection found in MOOCs
are: discussion forum (33), observation (14), learner analytics
(12) and ethnography (3).
RQ1.3 What are the data analysis methods used in
MOOC studies?
Data analysis method involves how the empirical data
was analyzed. We found that MOOC empirical studies have
employed a wide range of methods, techniques, approaches,
tools, tests and frameworks for analyzing empirical data.
124815
 R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature
FIGURE 4. Number of data collection methods.
FIGURE 5. Data collection methods.
Although, researchers in different domains and even in a
number of influential MOOC studies (e.g. see [18]) have used
the term ‘data analysis methods’ referring to the collection
of different statistical methods, approaches, techniques, etc.
used in analyzing empirical data. Researchers have been
using these terms interchangeably depending on the con-
text of the usage. Therefore, we also adopted a similar
terminology for these varied categories of research meth-
ods, approaches, techniques etc. as ‘data analysis meth-
ods’, accompanied with their corresponding number of times
used (frequency (f )) in our selected studies. We categorized
linear regression, logical regression, multivariate and cor-
relation under ‘inferential statistics’. We also found super-
vised machine learning approaches used in analyzing MOOC
empirical data that include support vector machine (SVM)
algorithm (f = 5), neural networks algorithm (f = 1),
decision tree algorithm (f = 2) and random forest algo-
rithm (f = 1). Similarly, we found unsupervised machine
learning approaches such as k-means clustering unsupervised
machine learning approaches. We also categorized structural
equation modelling (SEM), partial least square, path analysis,
latent class analysis, cluster analysis under ‘causal analysis’.
We also categorized the less frequently used data analy-
sis methods as ‘others’. The ‘others’ data analysis meth-
ods, approaches, techniques or tests include cross validation,
search grid, semantic analysis, propensity score matching,
local interpretable model-agnostic explanations (LIME), Del-
phi method, robustness analysis, mapping analysis, survival
analysis, frequency analysis, typology analysis, pedagogical
analysis, Bourdieu’s framework of social and cultural capital,
massive open online proctor (MOOP) framework, and root-
mean square error (RMSE). The data analysis methods are
presented in table 4 below.
124816
RQ2: What are the research focuses in MOOC studies?
In terms of the general focus of MOOC empirical stud-
ies, we categorized the research focuses into student-focused
(n = 202), instructor-focused (n = 33), design-focused
(n = 37), context/impact (n = 28), and others (n = 19) see
figure 6. Apart from identifying the general focus of empirical
MOOC delivery, we also examined a more in-depth picture or
detailed of these focuses. As majority of the studies focused
on MOOC learners, these studies mainly examines learn-
ers’ dropout, learners’ engagement, learners’ experience with
MOOC, learners’ behavior with MOOCs, learners’ motiva-
tion to use MOOCs, communication/interaction, assessment
and evaluation of MOOC learners and satisfaction with the
use of MOOCs. In addition, other rare focuses such learners’
academic dishonesty (cheating), self-regulation, communica-
tion patterns were among the ‘other’ specific focal topics in
MOOCs.
Instructor focus mainly investigates instructors experience
of designing and developing MOOCs, instructors’ roles in
MOOCs, instructors teaching presence, communication pat-
terns, assessments and evaluation of learners and satisfaction.
Meanwhile design focus studies primarily centers on creation
of affective online community, pedagogical design quality
and design collaboration activities.
Impact/context focus studies mainly investigates MOOCs
in higher education, MOOCs in developing countries, effec-
tiveness and flexibility of MOOCs, and adoption of MOOCs.
Other (Others) related focus studies examines issues per-
taining to MOOCs and higher institutions [9] such as profes-
sional development, underserved students, MOOCs for the
disabled, MOOCs for the elderly people, social media in
MOOCs, cultural/religious dimensions in MOOCs, MOOCs
in underdeveloped regions/countries etc.
VOLUME 7, 2019
R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature
TABLE 4. Data analysis methods (methods, techniques, approaches, frameworks tools and tests) in MOOC studies.
FIGURE 6. Research focuses.
RQ2.1: What are the research topics in MOOC studies?
Based on our knowledge in MOOC research and also
inspiration from other related studies (e.g. see [18]), we were
able to identify 18 key topics in MOOC research. We also
categorized the infrequent MOOC research topics as ‘others’.
The ‘others’ consist of topics such as gender in MOOCs, pro-
fessional development, k-12, underserved students, blended
learning in MOOCs, learners with disability, technologi-
cal competence on MOOCs, demographic distributions of
VOLUME 7, 2019
MOOC users, sustainability of MOOCs etc. Figure 7 gives
a representation of the research topics in MOOCs based on
our selected primary studies.
RQ3:What type of contribution (contribution facet) did
MOOC studies proposed?
1) 107 of the selected studies reported on the lessons
learned from the qualitative, quantitative or mixed
methods used without proposing any system, tool or
124817
 R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature
FIGURE 7. Research topics in MOOCs.
model. For instance, from the results of [47] exper-
iment, it was learned that learners motivation and
goals shape how learners conceptualized the pur-
pose of MOOCs. The contribution primarily entails
the lessons learned from the experiment or study
(See figure 8).
2) 65 of the selected studies proposed suggestions/
guidelines/advice. Therefore, we categorized sugges-
tions, guidelines and advice as a single contribution
facet category. An example is the study of [48] which
proposed a suggestion/guideline/advice stating that the
issue of dropout should be addressed from two perspec-
tives: retention as a problem but also as an opportunity.
Although some of the studies declared and proposed
suggestions based on their research outcome, others
suggested suggestions or broadly confirmed an advice
as a conclusion or contribution and not based on the
outcome of the study (See figure 8).
3) Third, 28 of the selected studies have proposed a
framework for MOOCs. An example is the study
of [49] which proposed a framework for determining
the instructional quality of MOOCs (See figure 8).
4) 58 of the selected studies proposed a model. An exam-
ple of a proposed model in the selected MOOCs studies
can be found in the studies of [50] which signifies
that learner control, user characteristics, platform dif-
ference, and their relationships in learners adoption
intention of MOOCs (See figure 8).
5) 19 of the selected studies proposed a system/tool.
An example of a proposed system is the study of [51],
that proposed a mobile MOOC learning system called
OneMind which detects the type, presence and inten-
124818
sity of divided attention through camera-based heart
tracking MOOC (See figure 8).
6) Six (6) of the selected studies propose theoretical con-
tributions. For example, the study of [2] proposed a
theory that ‘‘to Learners with a different regulatory-
focus (promotion-focus or prevention-focus), different
types of advocates (promotion-oriented or prevention
oriented advocates) yield different levels of motiva-
tion in learners engaged in MOOC Learning’’ (See
figure 8).
7) Three (3) of the selected studies proposed an
approach. An example of a proposed approach in
a MOOC study can be found in the study of [52]
which proposed a big data modelling and text
mining approach to improve MOOC analysis (See
figure 8).
8) 21 of the selected studies proposed a method/technique.
An example is the study of [53] that proposed a k-
means clustering method for grouping MOOC learners
based on their interaction with their course material
(See figure 8).
9) Four (4) of the selected studies proposed an algorithm.
An example is the study of [54] which proposed
a text detection algorithm in MOOC videos (See
figure 8).
RQ4: What research approaches MOOC studies have been
focusing on?
From our selected studies as discussed earlier in section 3,
research facets were classified into evaluation research, val-
idation research and solution proposal. Most of the studies
(169 = 54%) were evaluation research studies, (79 = 20%)
VOLUME 7, 2019
R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature
FIGURE 8. Contribution facet.
FIGURE 9. Research facet.
FIGURE 10. Publication channel.
were solution proposal studies and (63 = 26%) were valida-
tion research studies illustrated in figure 9.
RQ5: Where were MOOC studies most published?
A. PUBLICATION CHANNEL
Overall, our selected studies consist of 173 (56%) journal
articles, and 138 (44%) conference papers totaling the sum
of 311 MOOC publications as illustrated in figure 10.
B. PUBLICATION TREND
Although, MOOCs offering of global free online educa-
tion began in 2008, research activities started significantly
in 2011, with only two relevant selected studies. However,
we did not found any relevant study in 2012. One possible
explanation of that is researchers were skeptical about the
recognition or acceptability of MOOCs globally. An evidence
of this skepticism comes from one of our selected study
in 2011 by [55], which was one of the first few empirical
studies that sets to investigate the position of MOOCs in
higher education as: ‘is it really a pedagogy for humans
or pedagogy in abundance?’. Nevertheless, research activ-
ities changed in subsequent years with MOOCs experienc-
VOLUME 7, 2019
ing an increase in research activities in 2013, and a sudden
upsurge in 2014 and 2015; from five (5) relevant publications
in 2013 to 41 publications in 2014 and reaching its climax
in 2016 with 73 publications. The main reason of this upsurge
in research activities is that in October 2011, Stanford Univer-
sity in the United States started its first two MOOCs (artifi-
cial intelligence-AI and machine learning-ML) that attracted
global attention and recognition which led to 2012 declared
as the ‘year of MOOC’. The declaration of 2012 as the year
of MOOC possibly restored assurance in this new promising
instructional approach (MOOCs), thereby easing and moti-
vating e-learning researchers and practitioners for MOOC
research from 2013. The year 2017 and 2018 have witnessed
a fairly even amount of publications of 73 and 70 in 2017 and
2018 respectively. Figure 11 gives a representation of MOOC
publication trends over the years.
It is imperative to highlight research topic trends in
MOOCs across the years in order to fully understand
the specific type of researches that MOOC practition-
ers and researchers have been pursuing over the years.
Figure 12 gives a summary of the research topics with respect
to the years in MOOCs, and also shows the gradual evolution
of MOOC research over the years.
124819
 R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature
FIGURE 11. Publication trend.
As the year 2013 witnessed some progress in MOOC
research activities, 2014 witnessed an even greater amount
of MOOC publications with 41 publications compared to
only five publications in 2013. Also, studies in 2014 began
focusing on a number of emergent themes and issues in
MOOCs such as learners’ experience, engagement, behavior
and other non-learner related aspects such as instructional
design of MOOCs. In 2015, other research dimensions such
as dropout of MOOC courses emerged, possibly due to learn-
ers’ dropout rate. Figure 12 illustration shows that research
on dropout have been on the rise since 2014. This raise
a major concern on MOOCs as many studies and reports
show that the average completion rate of MOOC learners
is embarrassingly very low (figures of 10% retention are
widely cited) e.g. see [56], [57]. Researchers have been
battling with MOOC learners dropout by employing var-
ious intervention, motivational approaches and strategies
such as the ‘follow-feature’ in FutureLearn MOOC platform
(see [10]). Although, there is no formal definition of dropouts
in MOOCs, some researchers tentatively consider dropout
as whenever a learner does not engage in the final week’s
activities of a course, or does not submit the final assign-
ment of a course [10]. Contrary to that, some consider a
dropout as when a learner becomes inactive for up to two
weeks [10].
Similarly, research on engagement have been on the rise
identically to dropout research. One possible explanation
might be that engagement is another way of committing learn-
ers to their respective MOOC courses, thereby increasing
the likelihood of completion or reducing the likelihood of a
dropout. Arguably, engagement, motivation and dropout are
not mutually exclusive, they cross boundaries. Additionally,
the number of publications focusing on MOOC instructional
design have doubled from three (3) studies in 2014 to six
(6) studies in 2015, and the number has doubled again to
12 studies in 2016. However, these figures dropped to eight
(8) in both 2017 and 2018. One possible explanation of
this development might be the quest for fully understanding
124820
the structure and design of MOOC instructions, as MOOC
instructors are possibly novice in teaching MOOC courses
to their respective learners. Other areas of research that
have been fairly even across the years are ‘satisfaction’,
‘self-regulation’, ‘learner behavior’ and ‘communication/
interaction’. Although self-regulation has been ever present
in technology-mediated modes of instruction (e.g. blended
learning), MOOC research activities have focused more on
learners’ dropout.
Another finding from Figure 12 shows a decline in research
on learners’ motivation in MOOCs. One possibility is that
motivation has less significant impact on MOOC learners as
compared to dropout, engagement and instructional design.
Other areas of research in MOOCs termed as ‘others’ involve
topics such as learners study patterns, demographic distribu-
tion of MOOC learners, gender etc. Figure 12 clearly shows
that the quantity of research has been fairly consistent on
these less frequently MOOC topics. We anticipate that one
of these less frequent areas of research might emerge and
gain more research attention in the future such as learners’
self-regulation, or research on learners’ patterns which has
appeared twice in 2018.
C. GEOGRAPHICAL DISTRIBUTION
From our selected studies, we have identified 57 unique
countries in MOOC research. United States with 124 pub-
lications is the most active country in MOOC research, fol-
lowed by China with 39, Spain with 34, and United Kingdom
with 32. We also found Mexico as the most active South-
American country with 6 relevant studies, whereas other
South-American countries such as Uruguay, Brazil, Ecuador
are having one (1), two (2) and four (4) relevant studies
respectively. Spain, United Kingdom, Holland and Germany
were found to be the most active European countries. Sim-
ilarly, China and Taiwan have been the front runners for
MOOC research in Asia, while Australia is having 20 relevant
publications and New Zealand having only one (1). Togo,
Tanzania and South Africa were the only African countries
VOLUME 7, 2019
R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature

FIGURE 12. Bubble plot of year-wise empirical MOOC research topics.

TABLE 5. Most represented countries in MOOC publications. D. PUBLICATION FORA

From our selected studies, we found 31 different journals
and 27 different conferences. With respect to the publication
venues in which MOOC studies were published,
table 6 highlights the most represented journals in MOOC
publications. International Review of Research in Open and
Distributed Learning (IRRODL), Computers & Education
and Computers in Human Behavior were the top publishers of
MOOC research with 33, 25, and 13 journal articles respec-
tively. Similarly, Table 7 shows the most active conferences
on MOOCs. ACM and IEEE conferences have been the hall-
mark for publishing high impact MOOC conference papers.
ACM conference on Learning @ Scale (L@S), International
Conference on Learning Analytics and Knowledge (LAK)
were the most active conferences on MOOCs issuing 19 and
18 MOOC conference papers respectively. IEEE Global engi-
neering education conference (EDUCON) and IEEE frontiers
in education conference were the most active IEEE confer-
ences for MOOC, with seven (7) conference papers each.

found in our selected studies with each having one (1) publi- E. CITATION IMPACT
cation. Surprisingly, Saudi Arabia is the only country from the As expected, citations are highly influenced by publication
Arabian Peninsula with a MOOC publication. Table 5 shows date (among other factors), and papers published earlier tend
the countries being the most active in MOOC research. to have more citations. Similarly, journal articles tend to have

VOLUME 7, 2019 124821

 R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature
TABLE 6. Most represented MOOC journals.
TABLE 7. Most represented MOOC conferences.
higher citations compared to conference papers. From our
selected studies, we found a number of MOOC studies that
have been very influential in terms of citations. We obtained
124822
TABLE 8. Most influential MOOC journal articles.
the citation counts of each of our selected study from Google
Scholar citation count, which is subject to change at any
moment. Google Scholar offers a sound measure of how
much a paper has been cited. We identified 12 journal articles
having over 100 citations, and eight (8) conference papers
that equally have over 100 citations. Table 8 and table 9 pro-
vides a list of the most influential MOOC journals articles
and conference papers respectively based on the number of
citations of each journal article or conference paper.
VOLUME 7, 2019
R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature
TABLE 9. Most influential MOOC conference papers.
F. TOP INSTITUTIONS
From our selected studies, we identified 241 unique insti-
tutions around the globe. All the institutions are academic
institutions. Additionally, most of these institutions are in
the United States where MOOCs originated. Table 10 dis-
plays the most active institutions in MOOC research with
respect to their number of publications. Stanford Univer-
sity, Pennsylvania State University, Harvard University and
Massachusetts Institute of Technology (MIT) are the leading
institutions in MOOC research. We found only one institution
from China - Tsinghua University China, among the top
institutions in MOOC research. One possibility of Tsinghua
University been among the top institutions in MOOC research
is due to XeutangX MOOC –the world’s first Chinese MOOC
platform.
V. DISCUSSION
This study adopted Petersen’s systematic mapping study as
the primary guideline in providing an overview of the exist-
ing empirical research in educational aspects of MOOCs.
In addition, this study also looked at kitchenham’s systematic
literature review as a secondary guideline to further enrich the
results and ensure completeness of the study.
Out of 4248 studies from our initial search from the four
electronic databases, 1279 papers were identified on the basis
of relevance, of which 311 were finally selected based on ful-
filling the inclusion/exclusion criteria. The 311 studies exam-
VOLUME 7, 2019
TABLE 10. Most active institutions in MOOC research.
ined in this study reveals several interesting trends and issues
related to research in MOOCs. Our study found that research
activities on MOOCs have been on a continuous rise over the
years with an average growth of 30 publications from rep-
utable journals and proceedings per year. Although, MOOC
publications have slightly dropped in 2018, we strongly
believe that MOOCs would continue to be adopted across the
globe presumably in Europe and Asia.
Our study has revealed that MOOC research activities
started substantially in 2013 and increased steadily until it
reached its peak in 2016. A large number of these studies were
published in the United States. MOOC conference papers
were mostly from ACM and IEEE, and were mainly affiliated
to Universities in the United States and few European coun-
tries. Similarly, all our 311 selected studies were affiliated to
academic institutions. We have not found any selected study
affiliated to an industry.
Despite a decade since MOOCs became globally accessi-
ble, our study has found that MOOC research and adoption is
only growing substantially within the United States and few
other countries such as Spain, China and United Kingdom,
and has been relatively scarce or non-existent in developing
or underdeveloped regions such as Africa. Therefore, future
research should investigate the barriers to MOOCs adoption
in other parts of the globe, for instance, in Africa.
Our study shows that the most utilized research method in
MOOCs is quantitative with over 50% from our primary stud-
ies. We also identified nine (9) types of the contribution facet
which are lessons learned, method/technique, model, frame-
work, tool/system, algorithm, suggestions/guidelines/advice,
approach and theory. We also found that the most com-
monly used data collection methods are survey, interviews
and dataset from MOOC platforms. Furthermore, MOOC
researchers mostly use quantitative research methods. One
124823
 R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature
possible explanation of this finding is from the assertion that
quantitative research are more preferable compared to other
research methods because of easy access to data [71].
We also found that most of our selected studies have
focused on MOOC learners; and a large portion of MOOC
research specifically focused on learner dropout, as several
studies have underlined dropout as a headline challenge in
MOOCs. Even though, researchers have proposed various
strategies, techniques and intervention approaches for com-
bating learners’ dropout in MOOCs, research on learner
dropouts have always been an active research area over the
years. Arguably, many other related research themes such as
learner engagement, interaction and motivation to learn in
MOOCs were also research topics directly/indirectly aiming
at learners’ completion/dropout issue.
Our study has found that MOOC researchers and practi-
tioners have focused less on important research themes/issues
such as learners’ self-regulation. Self-regulation has been an
inherent problem that hinders several technology-mediated
mode of instructions in achieving true excellence. Self-
regulation related challenges are one of the obvious group of
challenges that learners face in online learning domains.
In addition our study also shows that MOOC research
community have not specifically looked at other types of
learners’ self-regulation behavior such as procrastination.
Since procrastination is considered a psychological behav-
ior [72], majority of the research activities on procrastina-
tion were from the psychological domain. Arguably, MOOC
researchers’ attention have been solely on the issue of learn-
ers’ dropout, that is why learners’ self-regulation issue has
received very little research attention in MOOCs. As such,
there is needed for more investigation and refocusing on
learners’ self-regulation behavior as well as providing effec-
tive interventions to other forms of self-regulation behav-
ior such as learners’ procrastination behavior. In addition,
future research should examine the possibility of stimulating
learners’ self-regulation behavior through external scaffolds
e.g. integrating social identity groups. Furthermore, future
research should also look at the possibility of combating
dropouts through stimulating learners’ self-regulation and
procrastination behavior interventions. Research is also war-
ranted to further investigate the underlying issues of suffer-
ing, anxiety and seclusion that MOOC learners face.
We have found that relatively few studies or efforts
were made in offering a personalized and recommenda-
tion approaches in MOOCs. More research is warranted
to explore the possibility of offering personalization and
recommendations in MOOCs; as personalization in other
domains (e.g. [73]) and recommendations in e-learning sys-
tems (e.g. [74]) has proven to be an effective means of
lessening technological complexity, connecting and engaging
students to online learning platforms.
Over the years, the community of inquiry online learning
framework has received a significant amount of attention
revision (e.g. [75]). Community of inquiry has received
considerable amount of attention in MOOCs. Online learn-
124824
ing researchers in general have struggled to remodeled the
framework by tentatively adding an additional construct
or conceptualizing the three original social, teaching and
cognitive presences. Learning presence proposals from the
empirical studies of [76], [77], regulatory presence from the
study of [78] has opened a new research dimensions due
to the importance of self-regulation in technology-mediated
environments. Therefore, more research is warranted to
investigate the proposed tentative variables/constructs of the
community of inquiry framework in relation to MOOCs.
Other proposed presences such as, autonomy presence [79]
and instructor presences are worth investigating in relation to
MOOCs.
VI. LIMITATIONS
Our study is limited with the employment of a rich selection
criteria and methodology in order to only consider articles
deemed as ‘high impact’. Secondly, we used four elec-
tronic databases for the extraction of MOOC studies, as such
results/data may slightly differ if we had consulted other
MOOC studies that were not in any of our four databases,
or even in any database such as ‘learning with MOOCs’ and
‘EMOOCs’ conference papers. Similarly, we only consid-
ered studies published in peer-reviewed journals; and studies
published in English language. There might be some MOOC
studies that were written in other languages than English,
which might be of importance to our studies. There might
also be some MOOC studies in book chapters, master’s
dissertations, technical reports or studies that are partially
or fully unavailable online that were not included in this
study. Another important shortcoming of our study is the
employment of voting exercise in which some decisions
are taken based on the researchers’ perceptions and view-
points. As such, some studies may be excluded based on
the researchers’ viewpoints whereas in some cases, these
excluded studies may be considered using a different inclu-
sion and exclusion criteria framework or other researchers’
viewpoints. Nevertheless, we strongly believe that our study
can serve as a roadmap towards the continuity of MOOCs
research.
VII. CONCLUSION
Even though our research foundation has been rich, it is
very difficult to make clear cut conclusions due to the
rapid advancements in technology and the complex nature
of human behavior especially in online learning environ-
ments such as MOOCs. Our study examines frequently used
research methods, data collection methods, data analysis
methods, research focuses, authors’ affiliations, publication
avenues, publication journals and conferences in MOOCs.
We learned that MOOC research and adoption has been on
a continuous rise over the years especially in the United States
and few European countries. We learned that MOOCs have
been relatively less researched and adopted in other parts of
the globe especially in developing or underdeveloped regions.
VOLUME 7, 2019
R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature
We learned that MOOC researchers and practitioners have
paid more attention to the issues of learners’ dropouts,
and have literally abandoned research on learners’ behavior
themes such as self-regulation. Our study has also identified
research trends that would better equip and trigger MOOC
researchers and practitioners in building upon their MOOC
research through exploring the areas in which knowledge is
as yet weak and inconclusive.
REFERENCES
[1] C. King, A. Robinson, and J. Vickers, ‘‘Online education: Targeted MOOC
captivates students,’’ Nature, vol. 505, p. 26, Jan. 2014.
[2] J. Zhang, ‘‘Can MOOCs be interesting to students? An experimental
investigation from regulatory focus perspective,’’ Comput. Edu., vol. 95,
pp. 340–351, Apr. 2016.
[3] O. Almatrafi, A. Johri, and H. Rangwala, ‘‘Needle in a haystack: Iden-
tifying learner posts that require urgent response in MOOC discussion
forums,’’ Comput. Edu., vol. 118, pp. 1–9, Mar. 2018.
[4] L. Pappano, The Year of the MOOC, vol. 2. New York, NY, USA: The New
York Times, 2012.
[5] D. Shah. (2016). Monetization Over Massiveness: Breaking Down MOOCs
by the Numbers in 2016. Accessed: Jul. 25, 2017. [Online]. Available:
https://www.edsurge.com/
[6] Y. Shi, X. Li, A. Haller, and J. Campbell, ‘‘Knowledge pricing structures
on MOOC platform—A use case analysis on edX,’’ presented at the 22nd
Pacific Asia Conf. Inf. Syst., Japan, 2018.
[7] M. Cook, ‘‘State of the MOOC 2016: A year of massive landscape change
for massive open online courses,’’ Online Course Rep., 2016.
[8] R. Deng and P. Benckendorff, ‘‘A contemporary review of research meth-
ods adopted to understand students’ and instructors’ use of massive open
online courses (MOOCs),’’ Int. J. Inf. Edu. Technol., vol. 7, pp. 601–607,
Aug. 2017.
[9] G. Veletsianos and P. Shepherdson, ‘‘A systematic analysis and synthesis
of the empirical MOOC literature published in 2013–2015,’’ Int. Rev. Res.
Open Distrib. Learn., vol. 17, p. 2, Mar. 2016.
[10] A. S. Sunar, S. White, N. A. Abdullah, and H. C. Davis, ‘‘How learners’
interactions sustain engagement: A MOOC case study,’’ IEEE Trans.
Learn. Technol., vol. 10, no. 4, pp. 475–487, Oct./Dec. 2017.
[11] S. Halawa, D. Greene, and J. Mitchell, ‘‘Dropout prediction in MOOCs
using learner activity features,’’ in Proc. 2nd Eur. MOOC Stakeholder
Summit, 2014, pp. 58–65.
[12] W. Xing, X. Chen, J. Stein, and M. Marcinkowski, ‘‘Temporal predication
of dropouts in MOOCs: Reaching the low hanging fruit through stacking
generalization,’’ Comput. Hum. Behav., vol. 58, pp. 119–129, May 2016.
[13] C. P. Rosé, R. Carlson, D. Yang, M. Wen, L. Resnick, P. Goldman, and
J. Sherer, ‘‘Social factors that contribute to attrition in MOOCs,’’ in Proc.
1st ACM Conf. Learn. Scale Conf., Mar. 2014, pp. 197–198.
[14] C. B. Garcíal, C. U. Artur, I. Ubieto-Artur, L. H. Ara, P. B. Guillén,
Á. F. Blanco, and M. Lacleta, ‘‘Designing and implementing a massive
open online course: Lessons learnt,’’ in Proc. 5th Int. Conf. Technol.
Ecosyst. Enhancing Multiculturality, Oct. 2017, p. 81.
[15] A. M. F. Yousef, U. Wahid, M. A. Chatti, U. Schroeder, and
M. Wosnitza, ‘‘The effect of peer assessment rubrics on learners’ satis-
faction and performance within a blended MOOC environment,’’ in Proc.
CSEDU, May 2015, pp. 148–159.
[16] P. J. Guo, J. Kim, and R. Rubin, ‘‘How video production affects student
engagement: An empirical study of MOOC videos,’’ in Proc. 1st ACM
Conf. Learn. Scale Conf., Mar. 2014, pp. 41–50.
[17] I. Nawrot and A. Doucet, ‘‘Building engagement for MOOC students:
Introducing support for time management on online learning platforms,’’
in Proc. 23rd Int. Conf. World Wide Web, Apr. 2014, pp. 1077–1082.
[18] M. Zhu, A. Sari, and M. M. Lee, ‘‘A systematic review of research meth-
ods and topics of the empirical MOOC literature (2014–2016),’’ Internet
Higher Edu., vol. 37, pp. 31–39, Apr. 2018.
[19] T. R. Liyanagunawardena, A. A. Adams, and S. A. Williams, ‘‘MOOCs:
A systematic study of the published literature 2008–2012,’’ Int. Rev. Res.
Open Distrib. Learn., vol. 14, pp. 202–227, Jul. 2013.
[20] A. S. Sunar, N. A. Abdullah, S. White, and H. Davis, ‘‘Personalisation
in MOOCs: A critical literature review,’’ in Proc. Int. Conf. Comput.
Supported Educ., May 2015, pp. 152–168.
VOLUME 7, 2019
[21] D. Gasevic, V. Kovanovic, S. Joksimovic, and G. Siemens, ‘‘Where is
research on massive open online courses headed? A data analysis of the
MOOC research initiative,’’ Int. Rev. Res. Open Distrib. Learn., vol. 15,
p. 5, Oct. 2014.
[22] T. Daradoumis, R. Bassi, F. Xhafa, and S. Caballé, ‘‘A review on massive E-
learning (MOOC) design, delivery and assessment,’’ in Proc. 8th Int. Conf.
P2P, Parallel, Grid, Cloud Internet Comput., Oct. 2013, pp. 208–213.
[23] K. F. Hew and W. S. Cheung, ‘‘Students’ and instructors’ use of massive
open online courses (MOOCs): Motivations and challenges,’’ Educ. Res.
Rev., vol. 12, pp. 45–58, Jun. 2014.
[24] J. Jacoby, ‘‘The disruptive potential of the massive open online course: A
literature review,’’ J. Open, Flexible, Distance Learn., vol. 18, pp. 73–85,
Mar. 2014.
[25] A. M. F. Yousef, M. A. Chatti, U. Schroeder, M. Wosnitza, and H. Jakobs,
‘‘The state of MOOCs from 2008 to 2014: A critical analysis and
future visions,’’ in Proc. Int. Conf. Comput. Supported Edu., Dec. 2014,
pp. 305–327.
[26] M. Ebben and J. S. Murphy, ‘‘Unpacking MOOC scholarly discourse: A
review of nascent MOOC scholarship,’’ Learn., Media Technol., vol. 39,
no. 3, pp. 328–345, 2014.
[27] J. E. Raffaghelli, S. Cucchiara, and D. Persico, ‘‘Methodological
approaches in MOOC research: Retracing the myth of P roteus,’’ Brit. J.
Educ. Technol., vol. 46, pp. 488–509, Mar. 2015.
[28] G. Veletsianos and P. Shepherdson, ‘‘Who studies MOOCs? Interdisci-
plinarity in MOOC research and its changes over time,’’ Int. Rev. Res. Open
Distrib. Learn., vol. 16, p. 3, Jun. 2015.
[29] A. Bozkurt, E. Akgün-Özbek, and O. Zawacki-Richter, ‘‘Trends and pat-
terns in massive open online courses: Review and content analysis of
research on MOOCs (2008–2015),’’ Int. Rev. Res. Open Distrib. Learn.,
vol. 18, p. 5, Aug. 2017.
[30] O. Almatrafi and A. Johri, ‘‘Systematic review of discussion forums in
massive open online courses (MOOCs),’’ IEEE Trans. Learn. Technol., to
be published.
[31] K. van de Oudeweetering and O. Agirdag, ‘‘MOOCS as accelerators of
social mobility? A systematic review,’’ J. Educ. Technol. Soc., vol. 21,
pp. 1–11, Jan. 2018.
[32] J. Lee, A. Hong, and J. Hwang, ‘‘A review of massive open online courses:
MOOC’s approach to bridge the digital divide,’’ in Proc. 22nd ITS Biennial
Conf. Int. Telecommun. Soc. Beyond Boundaries, Challenges Bus., Policy
Soc., Seoul, South Korea, 2018.
[33] M. Lu, H. Zhao, Y. Guo, K. Wang, and Z. Huang, ‘‘A review of the recent
studies on MOOCs,’’ in Proc. 13th Int. Conf. Comput. Sci. Educ. (ICCSE),
2018, pp. 1–5.
[34] M. Brown, E. Costello, E. Donlon, M. Giolla-Mhichil, C. Kirwan, V.
Kovanović, D. Gaševic, S. Joksimovic, G. Siemens, and M. Hatala, ‘‘Part 2:
Role media exposure on MOOC development,’’ presented at the Int. Open
Distance Learn. Conf. Dublin, 2016.
[35] V. Kovanović, S. Joksimović, D. Gašević, G. Siemens, and M. Hatala,
‘‘What public media reveals about MOOC s: A systematic analysis of news
reports,’’ Brit. J. Educ. Technol., vol. 46, no. 3, pp. 510–527, May 2015.
[36] K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson, ‘‘Systematic mapping
studies in software engineering,’’ Ease, vol. 8, pp. 68–77, Jun. 2008.
[37] S. Keele, ‘‘Guidelines for performing systematic literature reviews in
software engineering,’’ Softw. Eng. Group, School Comput. Sci. Math.,
Keele Univ., Keele, U.K., 2007.
[38] A. Abdelmaboud, D. N. A. Jawawi, I. Ghani, A. Elsafi, and
B. Kitchenham, ‘‘Quality of service approaches in cloud computing:
A systematic mapping study,’’ J. Syst. Softw., vol. 101, pp. 159–179,
Mar. 2015.
[39] E. Cavalcante, J. Pereira, M. P. Alves, P. Maia, R. Moura, T. Batista,
F. C. Delicato, and P. F. Pires, ‘‘On the interplay of Internet of Things
and cloud computing: A systematic mapping study,’’ Comput. Commun.,
vol. 89, pp. 17–33, Sep. 2016.
[40] A. Fernandez, J. Black, M. Jones, L. Wilson, L. Salvador-Carulla,
T. Astell-Burt, and D. Black, ‘‘Flooding and mental health: A systematic
mapping review,’’ PloS One, vol. 10, Apr. 2015, Art. no. e0119929.
[41] B. Kitchenham and P. Brereton, ‘‘A systematic review of systematic review
process research in software engineering,’’ Inf. Softw. Technol., vol. 55,
pp. 2049–2075, Dec. 2013.
[42] A. Zakari, S. P. Lee, K. A. Alam, and R. Ahmad, ‘‘Software fault localisa-
tion: A systematic mapping study,’’ IET Softw., vol. 13, no. 1, pp. 60–74,
Feb. 2018.
124825
 R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature
[43] M. Kuhrmann, P. Diebold, and J. Münch, ‘‘Software process improvement:
A systematic mapping study on the state of the art,’’ PeerJ Comput. Sci.,
vol. 2, p. e62, May 2016.
[44] B. Kitchenham, O. P. Brereton, D. Budgen, M. Turner, J. Bailey, and
S. Linkman, ‘‘Systematic literature reviews in software engineering—
A systematic literature review,’’ Inf. Softw. Technol., vol. 51, pp. 7–15,
Jan. 2009.
[45] M. Shaw, ‘‘Writing good software engineering research papers,’’ in Proc.
25th Int. Conf. Softw. Eng., May 2003, pp. 726–736.
[46] J. Daalhuizen and J. Schoormans, ‘‘Pioneering online design teaching in
a MOOC format: tools for facilitating experiential learning,’’ Int. J. Des.,
vol. 12, p. 2, Jan. 2018.
[47] A. Littlejohn, N. Hood, C. Milligan, and P. Mustain, ‘‘Learning in MOOCs:
Motivations and self-regulated learning in MOOCs,’’ Internet Higher
Educ., vol. 29, pp. 40–48, Apr. 2016.
[48] S. Zheng, M. B. Rosson, P. C. Shih, and J. M. Carroll, ‘‘Understand-
ing student motivation, behaviors and perceptions in MOOCs,’’ in Proc.
18th ACM Conf. Comput. Supported Cooperat. Work Social Comput.,
Mar. 2015, pp. 1882–1895.
[49] A. Margaryan, M. Bianco, and A. Littlejohn, ‘‘Instructional quality of
massive open online courses (MOOCs),’’ Comput. Edu., vol. 80, pp. 77–83,
Jan. 2015.
[50] M. Zhang, S. Yin, M. Luo, and W. Yan, ‘‘Learner control, user character-
istics, platform difference, and their role in adoption intention for MOOC
learning in China,’’ Australas. J. Educ. Technol., vol. 33, p. 1, Apr. 2017.
[51] X. Xiao and J. Wang, ‘‘Undertanding and detecting divided attention in
mobile MOOC learning,’’ in Proc. CHI Conf. Hum. Factors Comput. Syst.,
May 2017, pp. 2411–2415.
[52] A. Maté, E. de Gregorio, J. Cámara, and J. Trujillo, ‘‘Improving massive
open online courses analysis by applying modeling and text mining: A case
study,’’ in Proc. Int. Conf. Conceptual Model., 2013, pp. 29–38.
[53] S. Arora, M. Goel, A. S. Sabitha, and D. Mehrotra, ‘‘Learner groups in
massive open online courses,’’ Amer. J. Distance Edu., vol. 31, pp. 80–97,
Apr. 2017.
[54] H. Zhong and Y. Ma, ‘‘Fast and robust text detection in MOOCs videos,’’
in Proc. Int. Conf. Distance Edu. Learn., May 2018, pp. 98–102.
[55] R. Kop, H. Fournier, and J. S. F. Mak, ‘‘A pedagogy of abundance or a
pedagogy to support human beings? Participant support on massive open
online courses,’’ Int. Rev. Res. Open Distrib. Learn., vol. 12, pp. 74–93,
Apr. 2011.
[56] K. S. Hone and G. R. El Said, ‘‘Exploring the factors affecting MOOC
retention: A survey study,’’ Comput. Edu., vol. 98, pp. 157–168, Jul. 2016.
[57] D. Davis, I. Jivet, R. F. Kizilcec, G. Chen, C. Hauff, and G.-J. Houben,
‘‘Follow the successful crowd: raising MOOC completion rates through
social comparison at scale,’’ in Proc. 7th Int. Learn. Analytics Knowl.
Conf., Mar. 2017, pp. 454–463.
[58] R. Kop, ‘‘The challenges to connectivist learning on open online networks:
Learning experiences during a massive open online course,’’ Int. Rev. Res.
Open Distrib. Learn., vol. 12, pp. 19–38, Mar. 2011.
[59] K. Jordan, ‘‘Initial trends in enrolment and completion of massive open
online courses,’’ Int. Rev. Res. Open Distrib. Learn., vol. 15, pp. 1–15,
Jan. 2014.
[60] K. M. Alraimi, H. Zo, and A. P. Ciganek, ‘‘Understanding the MOOCs
continuance: The role of openness and reputation,’’ Comput. Educ., vol. 80,
pp. 28–38, Jan. 2015.
[61] K. Jordan, ‘‘Massive open online course completion rates revisited: Assess-
ment, length and attrition,’’ Int. Rev. Res. Open Distrib. Learn., vol. 16, p. 5,
Jun. 2015.
[62] N. Gillani and R. Eynon, ‘‘Communication patterns in massively open
online courses,’’ Internet Higher Edu., vol. 23, pp. 18–26, Oct. 2014.
[63] B. Toven-Lindsey, R. A. Rhoads, and J. B. Lozano, ‘‘Virtually unlimited
classrooms: Pedagogical practices in massive open online courses,’’ Inter-
net Higher Edu., vol. 24, pp. 1–12, Jan. 2015.
[64] M. Barak, A. Watted, and H. Haick, ‘‘Motivation to learn in massive open
online courses: Examining aspects of language and social engagement,’’
Comput. Edu., vol. 94, pp. 49–60, Mar. 2016.
[65] R. F. Kizilcec, M. Pérez-Sanagustín, and J. J. Maldonado, ‘‘Self-regulated
learning strategies predict learner behavior and goal attainment in Massive
Open Online Courses,’’ Comput. Educ., vol. 104, pp. 18–33, Jan. 2017.
[66] R. F. Kizilcec, C. Piech, and E. Schneider, ‘‘Deconstructing disengage-
ment: Analyzing learner subpopulations in massive open online courses,’’
in Proc. 3rd Int. Conf. Learn. Analytics Knowl., Apr. 2013, pp. 170–179.
[67] D. Clow, ‘‘MOOCs and the funnel of participation,’’ in Proc. 3rd Int. Conf.
Learn. Analytics Knowl., Apr. 2013, pp. 185–189.
124826
[68] P. J. Guo and K. Reinecke, ‘‘Demographic differences in how students
navigate through MOOCs,’’ in Proc. 1st ACM Conf. Learn. Scale Conf.,
Mar. 2014, pp. 21–30.
[69] D. Coetzee, A. Fox, M. A. Hearst, and B. Hartmann, ‘‘Should your MOOC
forum use a reputation system?,’’ in Proc. 17th ACM Conf. Comput.
Supported Cooperat. Work Social Comput., Feb. 2014, pp. 1176–1187.
[70] J. Huang, A. Dasgupta, A. Ghosh, J. Manning, and M. Sanders, ‘‘Super-
poster behavior in MOOC forums,’’ in Proc. 1st ACM Conf. Learn. Scale
Conf., Mar. 2014, pp. 117–126.
[71] G. Veletsianos, A. Collier, and E. Schneider, ‘‘Digging deeper into learn-
ers’ experiences in MOOCs: Participation in social networks outside of
MOOCs, notetaking and contexts surrounding content consumption,’’ Brit.
J. Educ. Technol., vol. 46, pp. 570–587, May 2015.
[72] W. van Eerde and K. B. Klingsieck, ‘‘Overcoming procrastination? A
meta-analysis of intervention studies,’’ Educ. Res. Rev., vol. 25, pp. 73–85,
Nov. 2018.
[73] M. M. Rahman and N. A. Abdullah, ‘‘A personalized group-based recom-
mendation approach for Web search in E-learning,’’ IEEE Access, vol. 6,
pp. 34166–34178, 2018.
[74] K. I. Ghauth and N. A. Abdullah, ‘‘The effect of incorporating good
learners’ ratings e-Learn.content-based recommender System,’’ J. Educ.
Technol. Soc., vol. 14, p. 248, Apr. 2011.
[75] V. Kovanović, S. Joksimovic, O. Poquet, T. Hennis, P. de Vries, M. Hatala,
S. Dawson, G. Siemens, and D. Gaševic, ‘‘Examining communities of
inquiry in massive open online courses: The role of study strategies,’’
Internet Higher Edu., vol. 40, pp. 20–43, Jan. 2019.
[76] P. Shea, S. Hayes, S. U. Smith, J. Vickers, T. Bidjerano, A. Pickett,
M. Gozza-Cohen, J. Wilde, and S. Jian, ‘‘Learning presence: Additional
research on a new conceptual element within the community of inquiry
(CoI) framework,’’ Internet Higher Edu., vol. 15, pp. 89–95, Mar. 2012.
[77] P. Shea, S. Hayes, S. Uzuner-Smith, M. Gozza-Cohen, J. Vickers,
and T. Bidjerano, ‘‘Reconceptualizing the community of inquiry frame-
work: An exploratory analysis,’’ Internet Higher Edu., vol. 23, pp. 9–17,
Oct. 2014.
[78] S. Kilis and Z. Yıldırım, ‘‘Investigation of community of inquiry frame-
work in regard to self-regulation, metacognition and motivation,’’ Comput.
Edu., vol. 126, pp. 53–64, Nov. 2018.
[79] J. Y. Lam, ‘‘Autonomy presence in the extended community of inquiry,’’
Int. J. Continuing Edu. Lifelong Learn., vol. 8, p. 39, Dec. 2015.
RASHEED ABUBAKAR RASHEED received the
B.Sc. degree in computer science from Bayero
University, Kano, Nigeria, in 2008, the M.Sc.
degree in management information systems from
Coventry University, U.K., in 2011, and the M.Sc.
degree in software engineering from the Univer-
sity of Portsmouth, U.K., in 2015. He is currently
pursuing the Ph.D. degree with the University
of Malaya, Malaysia. He is also a Lecturer with
the Faculty of Computer Science and Informa-
tion Technology, Bayero University, Kano. His research interests include
e-learning, blended learning, personalized and adaptive learning, recom-
mender systems, and software verification.
AMIRRUDIN KAMSIN received the B.I.T.
degree in management from the University of
Malaya, the M.Sc. degree in computer anima-
tion from Bournemouth University, U.K., and the
Ph.D. degree from University College London
(UCL). He is currently a Senior Lecturer with
the Faculty of Computer Science and Information
Technology, University of Malaya, Malaysia. His
research interests include human–computer inter-
action (HCI), authentication systems, e-learning,
mobile applications, serious game, augmented reality, and mobile health
services.
VOLUME 7, 2019
R. A. Rasheed et al.: Systematic Mapping Study of the Empirical MOOC Literature

NOR ANIZA ABDULLAH received the bachelor’s
degree (Hons.) in computer science from the Uni-
versity of Malaya, the master’s degree in inter-
active multimedia from Westminster University,
London, and the Ph.D. degree in computer sci-
ence from Southampton University, U.K. She is
currently an Associate Professor with the Faculty
of Computer Science and Information Technology,
University of Malaya, Malaysia. She has authored
or coauthored more than 60 refereed publications
in international journals, book chapters, and conferences. She has supervised
several master’s and Ph.D. students in the University of Malaya. She also co-
supervised several Master by Research students in the Moratuwa University
of Sri Lanka. Her research interests include personalized and adaptive learn-
ing, recommender system, decision support systems, big data analytics, and
content-based image/video retrieval. She serves as a Reviewer for several
ISI-indexed journals.
KHALID HARUNA received the B.Sc. and
M.Sc. degrees in computer science from Bayero
University, Kano, Nigeria, in 2011 and 2015,
respectively, and the Ph.D. degree in information
systems from the University of Malaya, Malaysia,
in 2018. He is currently a Lecturer with the Depart-
ment of Computer Science, Bayero University. His
research interests include recommender systems,
the Internet of Things, semantic web, big data ana-
lytics, sentiment analysis, soft set, and e-learning
systems.

ABUBAKAR ZAKARI received the master’s

degree in computer networks from Middlesex Uni-
versity, London, in 2014, and the Ph.D. degree
in software engineering from the University of
Malaya, Malaysia, in 2019. His current research
interests include software testing, software fault
localization, dynamic software update, and graph
theory.

VOLUME 7, 2019 124827