Computers and Education: Artificial Intelligence 2 (2021) 100037

Contents lists available at ScienceDirect

Computers and Education: Artificial Intelligence

journal homepage: www.sciencedirect.com/journal/computers-and-education-artificial-intelligence

Automatic analysis of cognitive presence in online discussions: An approach

using deep learning and explainable artificial intelligence
Yuanyuan Hu a, *, Rafael Ferreira Mello b, c, Dragan Gašević d, e, f
a
Faculty of Engineering, University of Auckland, Auckland, New Zealand
b
Departamento de Computação, Universidade Federal Rural de Pernambuco, Recife, Brazil
c
Cesar School, Recife, Brazil
d
Faculty of Information Technology, Monash University, Melbourne, Australia
e
School of Informatics, University of Edinburgh, UK
f
Faculty of Computing and Information Technology, King Abdulaziz University, Saudi Arabia

A R T I C L E I N F O A B S T R A C T

Keywords: This paper proposes the adoption of a deep learning method to automate the categorisation of online discussion
Cognitive presence messages according to the phases of cognitive presence, a fundamental construct from the widely used Com­
Deep learning munity of Inquiry (CoI) framework of online learning. We investigated not only the performance of a deep
Explainable artificial intelligence
learning classifier but also its generalisability and interpretability, using explainable artificial intelligence al­
Online discussion
gorithms. In the study, we compared a Convolution Neural Network (CNN) model with the previous approaches
reported on the literature based on random forest classifiers and linguistics features of psychological processes
and cohesion. The CNN classifier trained and tested on the individual data set reached results up to Cohen’s κ of
0.528, demonstrating a similar performance to those of the random forest classifiers. Also, the generalisability
outcomes of the CNN classifiers across two disciplinary courses were similar to the results of the random forest
approach. Finally, the visualisations of explainable artificial intelligence provide novel insights into identifying
the phases of cognitive presence by word-level relevant indicators, as a complement to the feature importance
analysis from the random forest. Thus, we envisage combining the deep learning method and the conventional
machine learning algorithms (e.g. random forest) as complementary approaches to classify the phases of
cognitive presence.

1. Introduction them, cognitive presence is a central construct to seize the development of

Asynchronous online discussions play a key role in aiding social
interaction within educational environments, especially in higher edu­
cation (Anderson & Dron, 2010). They promote course participation,
collaboration, and even reduce drop-out rates (Rivard, 2013). Although
the online discussions have several benefits, it is crucial to provide tools
and methods to assist the instructors to monitor contributions and
promote understanding of the process of knowledge (co-)construction in
a group of students (Garrison, Cleveland-Innes, & Fung, 2010).
In this context, the Community of Inquiry (CoI) framework (Garrison
et al., 1999) is a widely validated model that provides different di­
mensions to analyse the dynamic and social nature of modern online
learning. CoI defines three presences that explain students and in­
structors’ social interactions in online learning environments. Within
the critical, in-depth thinking and argumentation skills of the students
(Garrison et al., 2001). The CoI framework also defines coding schemes
for the categorisation of discussion messages of students according to
each presence. For instance, cognitive presence is operationalized using
a four-phase inquiry process (Garrison et al., 2001): 1) Triggering event,
where a problem or dilemma is identified and conceptualized, 2)
Exploration, in which students’ explore and brainstorm potential solu­
tions to the issue at hand, 3) Integration, during which students (co-)
construct new knowledge by synthesizing the existing information, and
4) Resolution, in which students evaluate the newly-created knowledge
through hypothesis testing or vicarious application to the problem/­
dilemma that triggered the learning cycle. Although commonly used in
research for content analysis, the manual coding process is
labour-intensive, time-consuming, and not practical to be performed on

* Corresponding author.
E-mail addresses: y.hu@auckland.ac.nz (Y. Hu), rafael.mello@ufrpe.br (R. Ferreira Mello), dragan.gasevic@monash.edu (D. Gašević).

https://doi.org/10.1016/j.caeai.2021.100037
Received 26 April 2021; Received in revised form 23 August 2021; Accepted 10 October 2021
Available online 29 October 2021
2666-920X/© 2021 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Y. Hu et al.
ongoing discussions to assist instructors in their teaching (Kovanović,
Gašević, & Hatala, 2014).
Several automated and semi-automated methods have been pro­
posed in the recent literature (Barbosa et al., 2020; Ferreira et al., 2018;
Kovanović et al., 2014b, 2016; Neto et al., 2018; Rolim et al., 2019;
Waters et al., 2015). While early approaches applied a combination of
word and phrase counts features with black-box machine learning al­
gorithms (i.e., Neural Networks and Support Vector Machine)
(Kovanović, Joksimović, et al., 2014; Mcklin, 2004), recent studies
proposed the adoption of features based on psychological processes,
writing cohesiveness, and discussion structure compound with decision
tree algorithms (Barbosa et al., 2020; Kovanović, Joksimović, et al.,
2014; Neto et al., 2018). The main advantages of the second approach
are the description of important indicators of different cognitive pres­
ence phases and the improvement in the generalisability of the created
classifiers (Kovanović et al., 2016).
Advances in natural language processing have led to the develop­
ment of more accurate text classification techniques based on deep
neural networks (Lai et al., 2015). The main idea of these methods is that
the classifier tends to reach better results, even for different contexts, as
the database increases. Moreover, explainable artificial intelligence is a
recent method for unpacking the results of deep neural networks clas­
sifiers (Miller, 2019; Samek & Müller, 2019). The combination of these
approaches could provide alternative analysis not explored before in
terms of the analysis of CoI constrains.
Therefore, this paper proposes a new method for automatic catego­
risation of online discussion messages using deep machine learning and
explainable artificial intelligence. Our study analyses the efficiency and
the generalisability of this approach in the context of cognitive presence.
The results showed that this approach reached results comparable to the
previous approaches reported on the literature (Barbosa et al., 2021;
Kovanović et al., 2016; Neto et al., 2018). Moreover, the explainable
artificial intelligence visualisations shine new insights into the feature
importance for this problem. The results and their implications are
further discussed in this paper.
2. Background
2.1. The Community of Inquiry and cognitive presence
The Community of Inquiry (CoI) framework proposed by Garrison
et al. (1999) is a social-constructivist model used to analyse the inter­
active educational experience in online learning environments. It de­
scribes educational engagement that occurs in a learning community, in
which “a group of individuals who collaboratively engage in purposeful
critical discourse and reflection to construct personal meaning and confirm
mutual understanding” (Garrison & Anderson, 2011, p. 2). The CoI
framework has been widely used and validated for multidisciplinary
courses to understand the participants’ engagement in online discussion
forums (Garrison & Anderson, 2011; Garrison, Anderson, & Archer,
2010; Hu et al., 2020; Liu & Yang, 2014). This model defines three di­
mensions, called presences, to mould the learning experience: (i)
Cognitive presence explains the critical reflection of the knowledge
(re)construction, and problem-solving processes in the learning com­
munity (Garrison et al., 2001); (ii) Social presence analyses the ca­
pacity of the participants to humanise their relationships during an
online discussion. It considers how social interactions could be used to
model the social climate within a group of learners (i.e., cohesion, af­
fective, and interactive dimensions) (Rourke et al., 1999); (iii) Teaching
presence concerns the teaching role before (i.e., course design) and
during (i.e., facilitation and direct instruction) the course (Anderson
et al., 2001).
Although all three presences contribute to the understanding of
learning experience, cognitive presence is the primary element of the
CoI model in terms of the operationalisation of the practical inquiry
cycle of social-constructivist learning within online environments
2
Computers and Education: Artificial Intelligence 2 (2021) 100037
(Garrison et al., 2001). Within cognitive presence, this cycle is repre­
sented in four phases:
1. Triggering event initiates the cycle of critical inquiry. In this phase, a
student or the instructor proposes a problem or dilemma to trigger
the discussion. It could be a problem statement or a question asked
by a student;
2. Exploration involves an exchange of findings and reflections of initial
ideas. Based on the interactions, the participants propose possible
solutions for the given problem or explanations for the questions;
3. Integration is the phase in which the participants synthesise coherent
solutions and knowledge constructed on the ideas and the informa­
tion shared in the previous phases. In other words, the participants
construct structured solutions to the given problem/question by
integrating various ideas about the solution;
4. Resolution is the phase in which the participants assess the newly-
constructed knowledge through hypothesis testing or real-world
application in relation to the problem/dilemma that initially trig­
gered the discussion.
In this study, we focuses on the analysis of cognitive presence since it
is the ‘primary issue’ of students’ learning evidence to be explored
before the other dimensions (Rourke & Kanuka, 2009). The other two
presences of the CoI will be investigated in our future research. Thus, the
discussion messages that do not fall into any of the above phases are
classified as the Other. Also, the discussion messages that reveal in­
dicators of two phases are classified into the higher one, as the rule of
coding up. Moreover, examples of student discussion messages explored
in this study were provided in Table 1 for a better understanding of each
cognitive phase.
2.2. Content analysis of online discussions with CoI
Quantitative Content Analysis (QCA) (Rourke et al., 2001) is broadly
applied to assess social and cognitive presences by assigning the units of
the online discussion transcripts to a class and counting the number of
the units in each class (Bauer, 2007). The CoI model defines three QCA
coding schemes, one for each presence. Although the CoI model has
extensively been adopted in education research, content analysis has
been essentially employed for retrospective analysis, after the courses
are finished, without much impact on the actual student learning and
outcomes (Strijbos, 2011).
In this regard, the recent literature reports several studies on the
adoption of automated methods (i.e., machine learning techniques) to
assess CoI presences, intending to drive instructional interventions and
enhance student learning outcomes (Kovanović, Gašević, & Hatala,
2014; Rolim et al., 2019). Initially, the methods focused on the use of
combinations of traditional text mining features (i.e., bag-of-words
vectors) and classification algorithms. For instance, Mcklin (2004) and
Kovanović, Joksimović, et al. (2014) proposed the application of a
bag-of-words (n-gram) and supervised algorithms based on Neural
Networks and Support Vector Machines (SVMs), respectively, to auto­
matically classify online discussion messages into the phases of cognitive
presence. The methods reached Cohen’s κ of 0.31 (Mcklin, 2004) and
0.41 (Kovanović, Joksimović, et al., 2014). Moreover, previous works
also combined word and document embeddings with LIWC and
contextual features to categorise online discussion messages according
to the cognitive presence phases (Hayati et al., 2019). Although this
study used embeddings to enhance the feature vectors, the paper
adopted black-box algorithms in the analysis, such as SVM. the authors
did not provide details about the number of messages in the evaluation
data, but the best classifier proposed reached Cohen’s κ of 0.68.
The shortcomings of the initial approaches (bag-of-words and black-
box classification algorithms) are the lack of: (i) generalisability,
implicating that the creation of classification models is dependent on the
domain, and (ii) interpretability, which does not provide information on
Y. Hu et al.
Table 1
Examples of student discussion messages for each phase of cognitive presence
from two courses.
Phase An example of student discussion messages
Other Hi XXX I really appreciate your suggestions. I will keep
that in mind for future presentations.
An interesting start and I think I’m going to enjoy the
next 8 weeks.
Trigger I noticed you defined organizational model related
more on process actors do you think any other
important factors can influence the organizational
model study as well?
Have you managed to argue the above (that there is no
argument) in standard form?
Exploration From my understanding TagViewer transcribes audio
and links it to video. Is there future work for the
TagViewer to interpret changes in facial responses for
example? Even though body language doesn’t have
associated audio it does say quite a bit and it would be
nice to transcribe that as well.
It’s not a disagreement but discussion! Technically I
think the words have different meanings but in a
religious context they also have a connection so maybe
we’re both right!
Integration … I’ve liked using iterative processes as you describe
however I don’t believe we should ignore process
improvement tools that can be integrated into the
process to build better software. I believe the patterns
and questionnaires described by the author’s are some
of these tools. If we consider usability what your are
describing was almost 70% less effective than using the
authors patterns …
… I think it was legitimate as he was acting in the
course of his duties. Your professional role applies
constraints which would not apply if you were
discussing matters as an individual; therefore, in the
example, the appeal to authority is appropriate …
Resolution … Agile is by its definition supposed to be flexible and
dynamic in nature. As a result I find it ironic that there
would be a sticking point on literature Vs real world. I
would think the approach should be flexible based on
the client and developer skill sets. Software
development is a complex human interaction
regardless of the project size …
… Another way to test it would be to see if similar
positions eg heads of industry are also held by more
left-handed people than statistics would suggest. It
would be incredibly difficult to iron out other possible
factors - all the other things that help to make people
successful, like good schooling, access to health care
and a stable family environment, but if it were
possible, could we scientifically show that being left-
handed is a factor? I’m assuming that if we could we
would scientifically show that it’s pure coincidence …
why a specific class was chosen for each message (Kovanović et al.,
2016). To overcome these problems, the previous methods reported on
the literature used domain-independent features and decision trees,
especially the random forests algorithm. In this direction, Kovanović
et al. (2016), Neto et al. (2018), and Barbosa et al. (2020) adopted
random forests and features based on linguistic resources (Coh-metrix
(McNamara et al., 2014) and LIWC (Tausczik & Pennebaker, 2010)),
latent semantic analysis, named entity recognition, and discussion
structures (Waters et al., 2015), to identify the phases of cognitive
presence. These studies reached Cohen’s κ of 0.63 (Kovanović et al.,
2016), 0.72 (Neto et al., 2018), and 0.53 (Barbosa et al., 2020) when
applied to discussion messages written in English, Portuguese and across
these two languages, respectively.
2.3. Deep learning and explainable artificial intelligence
Deep learning algorithms have gained notoriety due to the
outstanding results achieved in computer vision tasks (Russakovsky
Computers and Education: Artificial Intelligence 2 (2021) 100037
et al., 2015), but its application rapidly spread to other domains as
Natural Language Processing (NLP) (Young et al., 2018). These algo­
rithms rely on the idea that the large number of connections generated
Course on deep neural networks could simulate the same learning process as the
Software
human brain (Goodfellow et al., 2016).
course In terms of NLP, deep learning has played an increasing role in tasks
Philosophy related to topic modelling/classification (ALRashdi & O’Keefe, 2019)
MOOC and machine translation (Yang et al., 2020). The Convolutional Neural
Software
Network (CNN), as a class of the deep neural networks, has been
course
commonly used for the text classification tasks, especially for small text
documents such as Twitter and online discussions posts (Lai et al., 2015;
Philosophy Wei et al., 2017). Some promising studies demonstrated that CNN out­
MOOC performs traditional machine learning algorithms in this context. For
Software
course
instance, Kamath et al. (2018) compared the results of a deep learning
architecture against random forests, SVM, Logistic Regression and
Multilayer Perceptron (MLP) in two text classification tasks. The final
results showed that the deep learning algorithm achieved results up to
50% better than the other ones. Similarly, Guo et al. (2019) compared
Philosophy
MOOC different architectures of CNN with Adaboost, a state-of-the-art decision
tree algorithm, with the goal of identifying urgency in online discussion
messages (Guo et al., 2019). In this context, CNN outperformed Ada­
Software boost by up to 14%.
course
Despite the promising results, the inability to explain the function­
ality or reasons of the prediction output from the deep learning algo­
rithms is one of the core barriers that Artificial Intelligence is
encountering (Barredo Arrieta et al., 2020). It is also one of the main
reasons why the white-box algorithms (e.g. random forests) can be
Philosophy preferable in educational research, notably in the previous studies of the
MOOC automatic classifier for the cognitive presence (Barbosa et al., 2020;
Kovanović et al., 2016; Neto et al., 2018), compared to the deep learning
neural networks. However, the field of eXplainable Artificial Intelli­
gence (XAI) aims to create machine learning techniques that enable
Software
course human audiences to easily understand, reasonably clarify, and appro­
priately trust the AI methods and implementations (Gunning, 2017). XAI
can potentially complement the adoption of deep learning algorithms
since it unpacks the model decisions. In learning analytics, explain­
ability of the automatic analysis approach and its product is also a sig­
Philosophy nificant goal (Gašević et al., 2017). It should provide scholars and
MOOC educators insights into the research questions, for example, the in­
dicators of different cognitive phases revealed in the online discussion
messages in our study.
3. Research questions
As discussed in Section 2.1, cognitive presence has a crucial role in
analysing knowledge construction and critical thinking in online
learning environments. It assesses “the extent to which the participants
in any particular configuration of a community of inquiry are able to
construct meaning through sustained communication” (Garrison et al.,
1999, p. 89). Although several studies proposed automatic methods for
classifying discussion transcripts according to the phases of cognitive
presence, to our knowledge, no publication applied deep learning
techniques for the automatic content analysis of cognitive presence.
Hence, our first research question is:
Research Question 1 (RQ1):
To what extent can accurately deep learning methods classify online
discussion messages according to the phases of cognitive presence? Can deep
learning outperform traditional machine learning techniques for this task?
One of the main limitations reported in the literature is the lack of
generalisability of the existing classifiers of cognitive presence (Barbosa
et al., 2020; Kovanović et al., 2016; Neto et al., 2018). As deep learning
methods are based on the content of the text, it could lead to overfitting
of the generated model. Therefore, our second research question is:
3
Y. Hu et al.
Research Question 2 (RQ2):
To what extent does the use of deep learning methods affects the gen­
eralisability of a classifier for cognitive presence?
Finally, Section 2.2 highlights the need to not only to provide ac­
curate classification of discussion messages but also to explain the
reason why messages were classified in one of the cognitive presence
phases. Traditionally, deep learning method are categorised as black-
box algorithms. However, eXplainable Artificial Intelligence (XAI)
(Miller, 2019; Samek & Müller, 2019) is a method to unpack the deep
learning model with visual representations of the output. This is a novel
approach never adopted in this context. We hypothesise that XAI could
provide additional information on the development of cognitive pres­
ence phases. Thus, our last research question is:
Research Question 3 (RQ3):
What are the important features revealed from the deep learning with XAI
for identifying cognitive presence phases in online discussions? What are the
differences between the information provided by deep learning with XAI and
previous machine learning methods used in the classification of cognitive
presence?
4. Methods
4.1. Data description
In this study, we utilised two data sets coming from the online dis­
cussion forums of two different courses. The software engineering data
set is from a fully online, masters-level, for-credit course, which was
applied in the previous automatic classifier studies (Barbosa et al., 2020;
Kovanović et al., 2016). The Logical and Critical Thinking data set is
from an introductory Philosophy MOOC, which was used to validate the
classification rubric of cognitive presence in the MOOC setting (Hu et al.,
2020).
4.1.1. The software engineering course data set
The software engineering course data set (SoftwareSet) consisted of
1747 discussion messages1 generated by 81 students from six offerings
of the course between 2008 and 2011, on the Moodle LMS, at a Canadian
public university. The course was a research-intensive course that con­
tained 14 different topics in software engineering, such as software re­
quirements, design, and maintenance. The students were asked to share
a video presentation about a course-related research paper, post a new
thread in the discussion forum, and invite other students to engage in the
debate about the presentation. The students’ participation in the threads
accounted for 15% of the final course mark.
All the 1747 messages were manually classified by two expert coders
(98.1% agreement, Cohen’s κ of 0.974, which is a largely used measure
to evaluate the inter-rater agreement when two coders are involved in
the process (Ben-David, 2008; Cohen, 1960)) into five phases of cogni­
tive presence according to Garrison et al.’s coding scheme (Garrison
et al., 1999). Both coders are experts in educational technology research
with extensive experience with CoI framework. Moreover, one of them
has expertise in the course (software engineering), and the other has a
background in psychology and education. The disagreements (32 mes­
sages) were resolved by discussions of the coders. The third column in
Table 2 illustrates the distribution of the cognitive presence phases in
SoftwareSet.
4.1.2. The logical and critical thinking MOOC data set
The logical and critical thinking MOOC data set (MOOCSet) was
1
In both of our data sets, an individual message refers to a starting post or its
replies from each discussion thread.
4
Computers and Education: Artificial Intelligence 2 (2021) 100037
Table 2
Distribution of cognitive presence phases for both data sets.
ID Phases SoftwareSet MOOCSet Total (%)
Messages (%) Messages (%)
1 Other 140 (8.01%) 85 (5.75%) 225 (6.97%)
2 Triggering 308 (17.63%) 279 (18.86%) 587
event (18.20%)
3 Exploration 684 (39.15%) 835 (56.46%) 1519
(47.09%)
4 Integration 508 (29.08%) 244 (16.50%) 752
(23.31%)
5 Resolution 107 (6.12%) 36 (2.43%) 143 (4.43%)
Total 1747 (100%) 1479 (100%) 3226
(100%)
from an archived offering of the Philosophy MOOC, which was designed
and taught by a course-design team at a New Zealand’s university, on
the FutureLearn platform. This course focused on the basic concepts of
logical and critical thinking, how to build good arguments, and how to
link the concepts with daily life. The MOOCSet consists of 1917 dis­
cussion messages that were randomly selected from the forums under
the eight-weekly topics of the archived offering. The offering had 12,311
messages in total generated by 1000 registered learners.
The 1917 messages from the LCT MOOC were manually classified by
three trained coders (77.15% agreement, Fleiss’s κ of 0.763, which is a
combination of multiple uses of the Cohen’s κ measure applied when
more the two coders participated in the annotation process (Falotico &
Quatto, 2015)) based on a revised classification rubric of cognitive
presence for the MOOC settings (Hu et al., 2020). This study uses only
the messages classified in the same category by all three coders (1479
messages). The forth column in Table 2 displays the distribution of the
cognitive presence phases in MOOCSet. The three coders were
post-graduate students from the Philosophy Department, who were also
the teaching assistants of the LCT MOOC.
4.1.3. Comparison between the two data sets
The discussion messages in both of the data sets share some simi­
larities. One pattern is that the distribution ratio of messages across the
four phases of cognitive presence and messages coded as other has a
similar trend. For instance, the messages classified into Exploration
accounted for the largest proportion, and the Resolution for the least in
both data sets. Also, the messages coded as Other, which is called None-
cognitive presence in the study of the LCT MOOC set, had the second
lowest proportion. Another similarity between the two data sets is that
all the messages were contributed by the learners without the in­
structors’ participation. The learners from both of the courses raised
threads in terms of questions and comments around different topics in
the videos and articles.
Some differences are also revealed between the two data sets. Firstly,
they were from two disciplines (i.e., software engineering and philoso­
phy); therefore, the vocabularies utilised are distinct. Secondly, the
types of the two courses where the messages originate from were
different. The software engineering course was a small-scale, for-credit,
university course, whereas the LCT MOOC was an open and free course
with the participation of the large number of learners. The learners’
demographics and motivations in the MOOC are diverse, which is
distinct from the for-credit, university courses.
Moreover, the details of the distribution across the four phases of
cognitive presence and Other messages were not exactly the same
although the trend was similar. We used the Mann-Whitney U Test to
investigate whether the differences of the distribution were significant
between the two data sets (McKnight & Najab, 2010). The result (p <
.001, U = 1146776, r = 1747&1479, with two-sided hypothesis) indi­
cated that the distributions of the four phases of cognitive presence were
statistically different between the data sets. We are aware that these
differences could affect the outcomes of our experiments, most notably
Y. Hu et al.
the between-course generalisability of the automatic classifier for
cognitive presence.
4.2. Deep learning architectures
In this study, we used an Inception CNN of naïve form (Szegedy et al.,
2015), with the application of pre-trained word-embedding models as
word representations to build a classifier for analysing phases of
cognitive presence. The architectures of this approach is explained in
Section 4.2.2.
4.2.1. Word embeddings
Word embedding is an NLP technique that uses continuous real
number vectors to represent texts (i.e. words, phrases, sentences and
documents). These vector representations are learned from very large
corpora, and are composed of meaningful features associated with po­
sitions in a high-dimensional space. The main idea of word embeddings
is that semantically similar words or phrases can be located very closely
in the space, which also means they have similar representation vectors.
We used the GloVe pre-trained model (Pennington et al., 2014) to covert
the text messages into embedding vectors as the inputs for our deep
learning neural networks. For the out-of-vocabulary words (77 words)
that GloVe did not contain, we randomly initialised their vectors, which
is a common approach in the literature. During the data pre-processing,
we removed the punctuation and numbers in each sentence, and also
performed case-folding and lemmatisation. The max length of tokens for
each message was 1939, which contained the complete information for
all the messages from both data sets. A matrix of shape D × L × N was
generated as the input variables of our neural networks, in which D
denotes word embedding dimension,2 L the max length of tokens, and N
the number of messages for training.
4.2.2. Convolutional neural networks
The architecture of the naïve-form Inception CNN (Szegedy et al.,
2015) used in this study was based on an architecture3 applied in the
classification of discussions on the Quora platform with high perfor­
mance. Fig. 1 depicts the details of the architecture. It consists of a
spatial dropout layer, five convolution layers with max poolings, a batch
normalisation process, and a fully-connected layer. Initially, the word
embedding matrix, as the vector representation of the input messages, is
fed into the spatial dropout neuron with five different filters. Instead of
feeding the matrix into the convolution layers directly, the use of the
spatial dropout can be effective for preventing neural works from
over-fitting (Lee & Lee, 2020). After the dropout layer, the five convo­
lution layers in one dimension operate the five matrices produced from
the dropout layer separately. Then, five max pooling layers retain the
important information from the convolution layers but reduce the
dimensionality, respectively. After concatenation of the five layers, a
new matrix is produced. Next, the new matrix passes through the batch
normalisation for improving training speed and validation accuracy
before the fully-connected layer computing the prediction probability
for each class. Finally, the network outputs one of the five phases of
cognitive presence using the activation function of softmax. Table 3
demonstrates the values of the optimal hyper-parameters implemented
for all the experiments.
4.3. Explainable artificial intelligence
We are interested in the relevance scores between the input variables
and the identified categories on word level. In other words, we wonder
which words in the messages are important for each cognitive phase, to
2
The 100-dimension pre-trained vectors were used in our experiments. http
s://nlp.stanford.edu/projects/glove/.
3
https://www.kaggle.com/christofhenkel/inceptioncnn-with-flip.
5
Computers and Education: Artificial Intelligence 2 (2021) 100037
understand the deep learning algorithm’s decisions. We applied
Gradient-based Sensitivity Analysis (SA), as an example of eXplainable
AI to visualise the importance of words in the discussion messages for
identifying the phases of cognitive presence. Similar methods were used
to explain deep learning neural networks in NLP tasks (Arras, Horn,
et al., 2017,b). Our trained neural network has a prediction function fc
for each category c and an input vector x, which consists of dimension
d using word embeddings. We use xi,t to denote the ith element of the
word embedding vector, which represents the tth word in a message.
Gradient-based SA (Gevrey et al., 2003) assigns the relevance scores Ri,t
by calculating squared partial derivatives:
( )2
∂fc
Ri,t = (x) (1)
∂xi,t
Theses derivations can be computed by using standard gradient
backpropagation (Rumelhart et al., 1986) in most neural networks. SA
can also be a decomposition of the squared gradient norm if we sum up
the relevance scores of the entire input dimension d:
∑
‖∇xfc (x)‖ 22 = Ri,t (2)
d
In this study, we employed the SmoothGrad method from the
iNNvestigate library (Alber et al., 2019) as it can effectively improve the
gradient-based sensitivity maps by averaging the gradient over number
of inputs with added noise (Smilkov et al., 2017). The relevance scores
of SA are positive values and were between 0 and 1 in our study. We
plotted heatmaps to visualise the word-level relevance scores for each
cognitive phase. The words highlighted in red denote strong relevance
(0.5 < R ≤ 1) to a target phase, while words in blues indicate low
relevance (0 < R ≤ 0.5).
4.4. Evaluation metrics
We used the metrics of accuracy, Cohen’s κ, F1 score, weighted-aver­
aged F1 score, and error rate of each class to measure the performance of
our CNN and RF classifiers. Accuracy score and Cohen’s κ are commonly
used to evaluate the performance of a supervised machine-learning
classifier (i.e., input variables have been pre-classified), in educational
research (Barbosa et al., 2020; Kovanović et al., 2014b, 2016; Neto et al.,
2018; Wang et al., 2015; Waters et al., 2015). Accuracy score is defined as
the number of true positives (TPc) plus the number of false positives
(FPc) over the sum of all the true and false predictions for each class c.
∑ ∑
TP + FP
accuracy = ∑ ∑ c c ∑c c ∑ (3)
c TPc + c FPc + c NPc + c FNc
Cohen’s κ (Cohen, 1960) was first introduced to measure the reli­
ability between two human coders on the classification tasks, lessening
the agreement caused by chance. It can also be used to evaluate the
degree of agreement between the actual and predicted labels in machine
learning tasks. Cohen’s κ is defined as
P0 − Pe
κ= (4)
1 − Pe
where P0 is the agreement probability of all classes. Pc is the agreement
probability by chance, which is computed as the columns and rows
marginal probabilities in the confusion matrix (Ben-David, 2008).
Macro- and micro-averaged scores are often adopted to compute the
overall performance of multi-class classification problems.
Macro-averaged method considers all classes as equally important,
whereas micro-averaged regards the different contributions of each
class, valuing the minority class the least (Van Asch, 2013).
Macro-Averaged Precision (macro AP) is defined as the mean value of
precision score (Pc) of each class c.
Y. Hu et al. Computers and Education: Artificial Intelligence 2 (2021) 100037

Fig. 1. Architecture of the naïve-form Inception CNN used in this study.

4.5. Data processing

Table 3
The optimal hyper-parameters for all experiments in this
To address research question 1, we trained two classifiers based on
study.
SoftwareSet (Classifier 1) and MOOCSet (Classifier 2), respectively. The
Hyper-parameters Values 10-fold cross-validation (CV) method was adopted to randomly divide
Kernel size (1, 2, 3, 5, 7) the entire sample data into 10 non-overlapping folds of approximately
Number of filters 36 equal size. The k-fold CV approach has been commonly used in the
Hidden size 128
machine learning tasks to reduce the risk of overfitting, aiming for more
Batch size 36
Epoch 300 valid result. In every CV loop, the combination of nine-fold data was the
Optimizer Adam training set, and the remainder was the validation set. Moreover, we
Learning rate 0.005 repeated the training process with 10-fold CV six times by different
seeds. Thus, sixty weights file with performance outcomes were gener­
∑ ated in order to select the best classifier for identifying the cognitive
c Pc
macroAP = (5) presence phases by using one of the course data set (SoftwareSet or
the number of c
MOOCSet). The distribution of training and test data in the experiments
Also, macro-Averaged Recall (macro AR) is defined as the mean value for research question 1 can be seen in Table 4.
of recall score (Rc) of each class c. To address research question 2, we used the best weights of the clas­
∑ sifiers trained by one dataset to validate on the other one. For example,
c Rc Classifier 1 was applied to test on the entire MOOCSet (1479 messages),
macroAR = (6)
the number of c
and Classifier 2 on the entire SoftwareSet (1747 messages). Moreover,
Macro-averaged F1 score is defined as the harmonic mean of macroAP we trained three new CNN classifiers by concatenation of the two data
and marcoAR. sets, and evaluated the best models on the partial set of SoftwareSet
(Classifier 3), MOOCSet (Classifier 4) and both (Classifier 5). For Classifier
macroAP × macroAR
macroF1 = 2 × (7) 3, the SoftwareSet was split into training and test parts in the ratio of 9 :
macroAP + macroAR
1 by using stratified sampling method. This split before the data
We also used Weighted-averaged F1 score, which has been applied in concatenation was to ensure that the same distribution of instances from
the text classification tasks in recent years (Chakravarthi et al., 2020), in five categories of cognitive presence contained in the training and test
regard to class imbalance issues in our data set, especially the impact of sets. After combing the training part (90% of 1747 messages) of the
the minority instances (e.g., the Resolution phase). Weighted-averaged F1 SoftwareSet with the entire MOOCSet, we performed the same training
score4 is defined as the harmonic mean of precision and recall scores process as in the experiments for research question 1. The model weights
weighted (Pc, Rc) by the proportion of true instances for each class (Wc). with the best performing outcomes was selected to validate on the test
∑ part of SoftwareSet (10% of 1747 messages). Similarly, the MOOCSet
weightedAP = Pc × Wc (8)
was split before the concatenation and training process in the
c

∑
weightedAR = Rc × Wc (9)
Table 4
c
Distribution of training and test data in the experiments.
weightedAP × weightedAR RQ Experiment Classifier Training Test size
weightedF1 = 2 × (10) size
weightedAP + weightedAR
1 Train & validate on 1 1747 1747 (10%)
To evaluate the class-level performance, the error rate was used with
SoftwareSet (90%)
the confusion matrices as the number of false positives (FPc) over the Train & validate on MOOCSet 2 1479 1479 (10%)
number of true positives (TPc) plus false positives (FPc) for each class. (90%)
2 Validate Classifier 1 on 1 1747 1479 (100%)
FPc MOOCSet (90%)
error ​ rate ​ of ​ each ​ class ​ c = (11)
FPc + TPc Validate Classifier 2 on 2 1479 1747 (100%)
SoftwareSet (90%)
Train on both but validate on 3 3051 1747 (10%)
SoftwareSet
Train on both but validate on 4 3078 1479 (10%)
MOOCSet
4 Train & validate on both 5 2903 323 (10% of
https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_s
each set)
core.html/#sklearn.metrics.f1_score.

6
Y. Hu et al. Computers and Education: Artificial Intelligence 2 (2021) 100037

experiment of Classifier 4. In the final experiment for Classifier 5, we split Table 5

both of the SoftwareSet and MOOCSet in the ratio 9:1, and then com­ Outcomes of the CNN classifiers for the individual course set compared to the RF
bined each of the 90% data as a new training set, and the 10% as a test classifiers (RFs).
set. The details of training-test split of the experiments for research Experiment Accuracy Kappa Weighted Macro
question 2 is also listed in Table 4. (SD) (SD) F1 (SD) F1 (SD)
Finally, to address research question 3, we generated several XAI Classifier 1 Train & test on 0.632 0.475 0.649 0.541
heatmaps to visualise the word-level relevance to every cognitive pres­ the SoftwareSet (0.03) (0.05) (0.03) (0.04)
ence phase by the SmootGrad sensitivity analysis method. The relevance (CNN)
Classifier 2 Train & test on 0.707 0.528 0.723 0.467
scores were obtained from the best performing weights and the training
the LCT (0.03) (0.06) (0.03) (0.06)
data of the experiments. MOOCSet
The source ode of the experiments in this study is publicly available (CNN)
at github.com/yuanhu19/CognitivePresence_Cnn_XAI repository. RF Train & test on 0.642 0.479 0.667 0.530
classifier the SoftwareSet (0.04) (0.05) (0.05) (0.03)
1 for (RF)
4.6. Random forests classifiers RQ1
RF Train & test on 0.720 0.488 0.751 0.487
In order to compare the proposed method, we evaluated the Random classifier the LCT (0.04) (0.08) (0.05) (0.06)
Forests (RF) approach derived from the state-of-art model for classifi­ 2 for MOOCSet (RF)
RQ1
cation of cognitive presence phases by Kovanović et al. (2016), Neto
et al. (2018) and Farrow et al. (2019). We used 199 classification fea­
tures extracted from the textual information in the two course data sets, the best case. We can see that the Cohen’s κ coefficient values obtained
106 features by Coh-metrix (McNamara et al., 2014) and 93 features by in both classifiers fell in the ‘moderate’ agreement level (Landis & Koch,
LIWC (Linguistic Inquiry Word Count) (Tausczik & Pennebaker, 2010) 1977). Also, the confusion matrices for both CNN experiments are
for English language (Kovanović et al., 2016). In order to maintain the shown in Figs. 2 and 3. They demonstrates that Classifier 1 (SoftwareSet)
consistency of the classification features with our CNN classifiers, the had the lowest error rate in the Integration phase, whereas Classifier 2
contextual features, such as the message depth and number of replies (MOOCSet) performed the best in the Triggering event phase. Besides,
and so on, were excluded from the classification features in the RF the Exploration phase revealed the second lowest error rater, and the
classifiers. Similarly, the class rebalancing method was not included. Resolution obtained the highest (i.e. greater than 0.9) in the CNN clas­
The sample data was also pre-processed to remove numbers, and to sifiers for both course data sets.
perform case-folding and lemmatisation, in all of RF classifier Compared to the performance of the RF classifiers, CNN Classifier 1
approaches. (SoftwareSet) reached similar accuracy, Cohen’s κ and macro F1 scores
After the pre-processing and feature extraction, the training-test split with minor differences (lower than 0.011). CNN Classifier 2 (MOOCSet)
in the RF classifiers were exactly the same with the CNN classifiers of displayed a better Cohen’s κ and slightly lower accuracy and macro F1
each experiment (as shown in Table 4). The 10-fold CV was used to scores than the RF classifier 2, with minor differences (lower than
select the best performing RF models by the fine-tuning of two important 0.025). We can see the weighted F1 scores decreased in both of the CNN
parameters, ntree (i.e. the number of decision trees built) and mtry (i.e. classifiers.
the number of classification features randomly selected by each tree). To
select the optimal ntree value, we examined 100 to 1300 sampled with
every 100 interval. For each ntree value, a 10-fold CV was used to 5.2. Evaluation across different data sets – RQ2
perform 30 different numbers, which were randomly selected from 1 to
199, to fine-tune the mtry value. The optimal RF classifiers were created This section reports the performance of the CNN classifiers that were
with the best mtry and ntree. In the RF classifier of classifiers for research trained on a course data set but validated on another one for answering
question 1, a final 10-fold CV was conducted with the entire sample data the second research question. Table 6 displays the evaluation outcomes of
to report the best performance and SD value. While in the RF classifiers the cross-course experiments. The weights of the best performing clas­
for research question 2, we repeated the training-test loop 10 times with sifiers (1 and 2), as reported in Section 5.1, were applied in the first two
the optimal RF classifiers, as a consistent approach to compare with our experiments.
CNN classifiers. The optimal parameters in the experiments were: (i) CNN Classifier 1 achieved the accuracy of 0.416, Cohen’s κ of 0.116,
ntree of 900, mtry of 179 in the RF classifiers of Classifier 1, (ii) ntree of weighted F1 of 0.447 and macro F1 score of 0.190, when the training set
700, mtry of 118 in the RF classifiers of Classifier 2, and (iii) ntree of 500, was the SoftwareSet and test set was the MOOCSet. The confusion ma­
mtry of 160 in the RF classifiers of Classifier 3, 4 and 5. trix of this case (Fig. 4) illustrates that the classifier trained on Soft­
wareSet had lower error rates on the prediction of Other and Exploration
5. Results in the MOOCSet, and obtained higher error rates on the other three
phases. The confusions indicate that many LCT MOOC messages of the
5.1. Evaluation for individual data set – RQ1 Triggering-event and Exploration phases were misclassified into the
Other category. Also, many Integration-phase messages from the
This subsection reports the performance of the CNN classifiers and MOOCSet were misclassified into Exploration phase by the Classifier 1
RF classifiers trained and tested on the individual course data set learned from the SoftwareSet.
(SoftwareSet or LCT MOOCSet) for answering the first research question. CNN Classifier 2, as trained on MOOCSet and tested on the Soft­
Table 5 displays the outcome metrics of both the CNN classifiers and the wareSet, reached an accuracy of 0.421, Cohen’s κ of 0.175, weighted F1
RF classifiers. In the experiment on the SoftwareSet, the best performing of 0.462 and macro F1 of 0.276. It obtained an overall higher perfor­
CNN Classifier 1 achieved accuracy of 0.632 (SD = 0.03), Cohen’s κ of mance than the previous experiment. The confusion matrix in Fig. 5
0.475 (SD = 0.05), weighted F1 score of 0.649 (SD = 0.03), and macro F1 shows that the lowest error rate appeared at the prediction of the Inte­
score of 0.541 (SD = 0.04). The CNN Classifier 2, which was trained and gration phase in the SoftwareSet by using the classifier trained on the
tested on the LCT MOOCSet, showed an overall better performance MOOCSet. The error rates for identifying the other four categories were
except the macro F1 score than Classifier 1. Classifier 2 reached an ac­ greater than 0.65, demonstrating a pool performance. Moreover, the
curacy of 0.707 (SD = 0.03), Cohen’s κ of 0.475 (SD = 0.06), weighted majority of the misclassification occurred between the Exploration and
F1 score of 0.723 (SD = 0.03) and macro F1 score of 0.467 (SD = 0.06), in Integration phase.

7
Y. Hu et al. Computers and Education: Artificial Intelligence 2 (2021) 100037

Fig. 2. Confusion Matrix by training and testing on SoftwareSet.

Fig. 3. Confusion Matrix by training and testing on LCT MOOCSet.

We also examined the performance of the CNN classifiers trained on

Table 6
the combination of the two course data sets by three experiments.
Outcomes of the CNN classifiers for cross-course validation compared to the RF
Table 7 displays the outcomes of the CNN classifiers trained on the
classifiers (RFs).
combined set and tested on partial SoftwareSet, MOOCSet and both,
Experiment Accuracy Kappa Weighted Macro respectively. We can see that Classifier 4 performed better than Classifiers
F1 F1
3 and 5. Our CNN classifiers of the combined-course experiments had
Classifier 1 Train on 0.416 0.116 0.447 0.190 lower performance than the RF classifiers. Moreover, the overall results
SoftwareSet but
of the combined-course experiments surpassed the outcomes of the
test on MOOCSet
(CNN) cross-course validations (Table 6), but were lower than the results in the
Classifier 2 Train on MOOCSet 0.421 0.175 0.462 0.276 individual-course experiments (Table 5).
but test on
SoftwareSet (CNN)
RF classifier Train on 0.454 0.188 0.453 0.262 5.3. Visualisation by the explainable AI – RQ3
1 for RQ2 SoftwareSet but
test on MOOCSet The results of the XAI heatmaps were generated from the experi­
(RF)
ments of the best performing classifiers trained and tested on the indi­
RF classifier Train on MOOCSet 0.419 0.079 0.512 0.206
2 for RQ2 but test on vidual course data set, SoftwareSet or LCT MOOCSet. These two
SoftwareSet (RF) classifiers can demonstrate purer comparative outcomes of the word-
level predictive indicators of the cognitive presence phases between
the two data sets, than the classifiers trained in the combined-course
In both of the cross-course experiments, the Cohen’s κ values were at experiments. Due to the page limitation, we provided four messages
a ‘slight-agreement’ level (Landis & Koch, 1977). Also, both of them classified into Exploration (Fig. 6) or Integration (Fig. 7) phase from
reached a zero accuracy for the classification of the Resolution phase. both the data sets, as the examples of the XAI visualisations.
Outcomes of the RF classifiers for both experiments were also listed in We can observe that the word constructs and accommodate were
Table 6. The CNN classifier trained on the MOOCSet and tested on the highly relevant with classification of the Exploration phase in Softwar­
SoftwareSet had a better performance than the RF classifier with the eSet, followed by the programming methodology and function as moder­
same condition, whereas the CNN classifier reached slightly lower out­ ately positive (Fig. 6a). While in MOOCSet (Fig. 6b), the word conclusion
comes than the RF approach for the other way around (i.e., trained on was revealed a high relevance to identify the Exploration phase, and the
the SoftwareSet and tested on the MOOCSet). words agree, example, even, premises and true performed moderate

8
Y. Hu et al.
Table 7
Outcomes of the CNN classifiers for combined-course data set validation.
Experiment
Classifier 3 Train on both but test on SoftwareSet (CNN)
Classifier 4 Train on both but test on MOOCSet (CNN)
Classifier 5 Train & test on both (CNN)
RF classifier 3 Train on both but test on SoftwareSet (RF)
RF classifier 4 Train on both but test on MOOCSet (RF)
RF classifier 5 Train & test on both (RF)
relevance. In comparison with the positive associations, the majority of
the lowest relevant words in both the data sets were ‘stop-words’, which
refers to the most commonly used words in a language, such as the, to, a,
is in Fig. 6. Apart from the stop words, there were some nouns high­
lighted as low relevance to the Exploration phase in the SoftwareSet
message (Fig. 6a), namely programmer and adaptation.
We also provided two sample messages classified into the Integration
phase from the two data sets in Fig. 7. In the message from the Soft­
wareSet (Fig. 7a), the highly relevant words for classification of the
Integration phase contained implementation, i, fold, and long, and the
moderately relevant words included corresponding, exception, aspects,
and methodology. While the low-relevance words were presents, actually,
reports, and however and also the stop-words. Comparatively in the
MOOCSet message (Fig. 7b), the words that indicate strong relevance
were correlation, between, associated, relation, lateralization, and func­
tional, whereas the low-relevance words were divergent, necessarily,
hemisphere(s), specifically, cite and so on.
6. Discussion
6.1. The performance comparison between our CNN classifiers and RF
baslines – RQ1
The CNN classifiers presented in this study reached Cohen’s κ of
0.475 and 0.528 for the SoftwareSet and LCT MOOCSet, respectively,
which demonstrates a moderate inter-rater agreement (Landis & Koch,
1977). These results were similar to the performance of the RF classifiers
which used random forests with LIWC and Coh-Metrix features, and the
outcomes of the experiments without the over-sampling methods re­
ported in the previous studies reported on the literature (Kovanović
et al., 2016; Farrow et al., 2019; Barbosa et al., 2020). In addition, the
CNN classifiers obtained a smaller error rate in the Integration phase
(Figs. 4 and 5) compared to the previous works (Barbosa et al., 2020;
Farrow et al., 2019). Conceptually, the Integration phase can be char­
acterised by a knowledge construction process where the students
formulate new concepts and solutions (Garrison et al., 2001). The error
reduction in this phase could be related to the adoption of pre-trained
Fig. 4. Confusion Matrix with SoftwareSet as the training set and MOOCSet as the test set.
9
Computers and Education: Artificial Intelligence 2 (2021) 100037
Accuracy Kappa Weighted F1 Macro F1
0.463 0.194 0.501 0.327
0.628 0.359 0.649 0.404
0.582 0.366 0.603 0.398
0.528 0.281 0.550 0.419
0.679 0.440 0.719 0.504
0.624 0.408 0.662 0.444
word embeddings, which include an extensive vocabulary compared
to the traditional machine learning methods (Pennington et al., 2014).
6.2. Generalisability of our CNN classifiers across different data sets –
RQ2
Another important finding of this study concerns the analysis of the
generalisability of a method based on textual-related features. While the
literature suggests that the adoption of linguistic resources (i.e., LIWC
and Coh-Metrix) and structural information (i.e., discussion context
features) (Kovanović et al., 2016; Farrow et al., 2020; Barbosa et al.,
2020) can increase the generalisability of a classifier, our results
demonstrate that the use of the CNN classifiers on cross-course valida­
tions can reach similar results (Table 6). This finding implies that the
deep learning approaches with word embeddings can have potential to
increase the generlisability if we can build neural networks including the
sequential structures of the discussion messages, such as Recurrent
Neural Networks (Graves et al., 2013) and Transformers (Wolf et al.,
2020), in future work. Also, the generalisability could be improved by
using the advanced pre-trained models (e.g. BERT (Devlin et al., 2018))
and fine-tuning them with the corpora that contain the domain-specific
vocabularies in the sample data sets.
Random forests classifiers achieved better results when the training
step was performed in the combined data set (the SoftwareSet and LCT
MOOCSet). A possible reason for this could be found in the fact that the
optimal parameters (i.e. ntree and mtry) were fine-tuned separately in
the RF classifiers for the individual-course and combined-course ex­
periments. However, we used the same hyper-parameters, such as the
number of filters, batch sizes and kernel sizes, in all of our CNN exper­
iments to maintain the consistency. This result provides us a hint that the
hyper-parameters should be adjusted according to the change of the data
set.
6.3. Important features revealed from the explainable AI – RQ3
This paper also introduced some examples of the eXplainable Arti­
ficial Intelligence (XAI) visualisations to investigate the indicators for
Y. Hu et al. Computers and Education: Artificial Intelligence 2 (2021) 100037

Fig. 5. Confusion Matrix with MOOCSet as the training set and SoftwareSet as the test set.

Fig. 6. Two XAI visualisations for the classification of Exploration phase in the two data sets. High relevance is mapped to red, low relevance to blue. The colour
opacity is normalised to the maximum relevance scores per message. The redder the word is, the more important it is to identify the phase. The bluer the word is, the
lower the relevance. (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

the phases of cognitive presence through the training of deep neural
networks. Interestingly, the first four high relevant words we mentioned
in the MOOCSet message (Fig. 7b), and the word corresponding in the
SoftwareSet message (Fig. 7a), can be the vocabularies for ‘connecting
ideas’ and ‘convergence’, which are the important indicators to identify
the Integration phase in the coding scheme of cognitive presence
(Garrison et al., 2001; Hu et al., 2020). Besides, most the high-relevance
words in the examples of the Exploration phases (Figs. 6 and 7) were the
course-relevant concepts previously used in the discussions, which is
align to the definition of this phase (Garrison et al., 2001). These find­
ings suggest that the use of expressions, which indicate relationships
between different subjects, ideas or arguments in a discussion message,
can be significant features to identify the Integration phase. Also, in
shorter messages, the use of specific nouns or noun phrases, which are
the core concepts delivered in the courses, could be important indicators
to identify the Exploration phase.
We also found the important features provided by our deep learning
with XAI method were word-level based, which differs from what the
previous works have found by the Mean Decrease Gini (MDG) inter­
pretation (Barbosa et al., 2020; Kovanović et al., 2016). The previous
works proposed more abstract features, such as the degree of lexical
diversity, the degree of meaningfulness content words and so on.
Educational researchers and course instructors have to seek the ex­
pressions matched with definitions of those abstract features in the
Coh-Metrix and LIWC tools for each message. This work is still
time-consuming and nontransparent. In comparison, the word-level
explanations of XAI are perceivable and understandable. Nevertheless,
these two explainable methods have some similarities. For instance,
Kovanović et al. (2016) suggests that messages, which had higher se­
mantic similarities with their previous ones, tend to be more relevant to
the higher phases (i.e., Integration and Resolution). This information
could be connected with the results in our XAI visualisations as it reflects
that the expressions in terms of associating ideas in the messages can be
strong indicators for the Integration phase. Similarly, the XAI could shine
a light on the topic of a question asked during the discussion since the
number of questions marks can be strong indicators of Triggering event
messages according to the previous studies (Barbosa et al., 2020;
Kovanović et al., 2016). A divergence from the previous work is also
reflected. Several important indicators proposed in the previous work
were related to the stop-words (i.e., number of personal pronouns,
prepositions, indefinite pronouns, pronouns in the third person singu­
lar); however, the XAI highlighted many of them as low-relevance
indicators.
Therefore, we suggest that the deep learning method (CNN and XAI),
and the previous approach (random forests and MDG (Barbosa et al.,
2020; Farrow et al., 2020; Kovanović et al., 2016)) could be applied as
complementary approaches. On the one hand, it is possible to propose a
compound classifier to benefit from the best predictive power of each
approach since random forests better predicts the Triggering event and
the Exploration phases, and CNN obtained more favourable results for
the Integration phase. On the other hand, XAI could provide additional
insights (i.e., word-level positive and negative indicators) on the most
relevant features extracted by MDG.
7. Final remarks
This paper has three contributions. First, the introduction of a deep
learning model to automatically classify online discussion messages into
cognitive presence phases. The results reached Cohen’s κ values of
0.498, a moderate inter-rater agreement. Second, the proposed

10
Y. Hu et al.
Fig. 7. Two XAI visualisations for the classification of Integration phase in the two data sets. High relevance is mapped to red, low relevance to blue. The colour
opacity is normalised to the maximum relevance scores per message. The redder the word is, the more important it is to identify the phase. The bluer the word is, the
lower the relevance is. (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)
approach was applied to several scenarios to evaluate the general­
isability power of this solution. This investigation concluded that deep
learning models produce results similar to those of the random forests
classifiers. Finally, this study adopted an explainable artificial intelli­
gence visualisation method to provide insights into the indicators of
different cognitive presence phases.
The study’s practical implications include providing more informa­
tion to support decision-making from instructors in relation to the stu­
dents’ participation in the online discussion. While previous works
focused on provide structural information based on the text (lexical
diversity and readability of the text), the XAI visualisation proposed
produce insights on the relevant content for the discussion. The com­
bination of both approaches produces a richer analysis for the
instructors.
Despite promising results, some limitations can be identified, such as
the small number of message examples used in the current study, which
could have limited the potential of the deep learning models used. Next,
the data came from only two courses (fully online masters course about
software engineering and a MOOC course on philosophy), therefore,
results might have been affected by the intricacies of their course de­
signs. Moreover, the discussion structural features, such as the conver­
sational sequences, were not involved in the construction of the deep
learning classifiers. Finally, we did not explore the potential of
combining the results of the deep learning and traditional machine
learning algorithms. Future work should aim to conduct experimenta­
tion with the approach and possible evaluation with larger sample sizes
composed of data from different domains and languages. We also seek to
optimise the approach proposed in this study by analysing the perfor­
mance of different deep learning models and combine them with
traditional machine learning algorithms.
11
Computers and Education: Artificial Intelligence 2 (2021) 100037
Declaration of competing interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence
the work reported in this paper.
References
Alber, M., Lapuschkin, S., Seegerer, P., Hägele, M., Schütt, K. T., Montavon, G.,
Samek, W., Müller, K. R., Dähne, S., & Kindermans, P. J. (2019). INNvestigate neural
networks. Journal of Machine Learning Research, 20, Article 04260. arXiv:1808.
ALRashdi, R., & O’Keefe, S. (2019). Deep learning and word embeddings for tweet
classification for crisis response. arXiv preprint arXiv:1903.11024.
Anderson, T., & Dron, J. (2010). Three generations of distance education pedagogy. In
The international review of research in open and distance learning (Vol. 12, pp. 80–97).
URL: http://www.irrodl.org/index.php/irrodl/article/view/890/.
Anderson, T., Rourke, L., Garrison, D. R., & Archer, W. (2001). Assessing teaching
presence in a computer conferencing context. Journal of Asynchronous Learning
Networks, 5, 1–17.
Arras, L., Horn, F., Montavon, G., Müller, K. R., & Samek, W. (2017). ” what is relevant in
a text document?”: An interpretable machine learning approach. PLoS One, 12,
Article e0181142.
Arras, L., Montavon, G., Müller, K. R., & Samek, W. (2017b). Explaining recurrent neural
network predictions in sentiment analysis. arXiv preprint arXiv:1706.07206.
Barbosa, G., Camelo, R., Cavalcanti, A. P., Miranda, P., Mello, R. F., Kovanović, V., &
Gašević, D. (2020). Towards automatic cross-language classification of cognitive
presence in online discussions. In Proceedings of the tenth international conference on
learning analytics & knowledge (pp. 605–614).
Barbosa, A., Ferreira, M., Ferreira Mello, R., Dueire Lins, R., & Gasevic, D. (2021). The
impact of automatic text translation on classification of online discussions for social
and cognitive presences. In LAK21: 11th international learning analytics and knowledge
conference (pp. 77–87).
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A.,
Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., & Herrera, F. (2020).
Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and
challenges toward responsible AI. Information Fusion, 58, 82–115.
Bauer, M. W. (2007). Content analysis. an introduction to its methodology–by klaus
krippendorff from words to numbers. narrative, data and social science–by roberto
franzosi. British Journal of Sociology, 58, 329–331.
Y. Hu et al.
Ben-David, A. (2008). About the relationship between roc curves and cohen’s kappa.
Engineering Applications of Artificial Intelligence, 21, 874–882.
Chakravarthi, B. R., Priyadharshini, R., Muralidaran, V., Suryawanshi, S., Jose, N.,
Sherly, E., & McCrae, J. P. (2020). Overview of the track on sentiment analysis for
dravidian languages in code-mixed text. In Forum for information retrieval evaluation
(pp. 21–24).
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and
Psychological Measurement, 20, 37–46.
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv:
1810.04805.
Falotico, R., & Quatto, P. (2015). Fleiss’ kappa statistic without paradoxes. Quality and
Quantity, 49, 463–470.
Farrow, E., Moore, J., & Gasevic, D. (2019). Analysing discussion forum data: A replication
study avoiding data contamination. https://doi.org/10.1145/3303772.3303779
Farrow, E., Moore, J., & Gašević, D. (2020). Dialogue attributes that inform depth and
quality of participation in course discussion forums. In The 10th international
conference on learning analytics and knowledge (pp. 129–134). Frankfurt, Germany:
LAK ’20).
Ferreira, R., Kovanović, V., Gašević, D., & Rolim, V. (2018). Towards combined network
and text analytics of student discourse in online discussions. In Artificial intelligence in
education (pp. 111–126). Cham, Switzerland: Springer International Publishing.
https://doi.org/10.1007/978-3-319-93843-1_9.
Garrison, D. R., & Anderson, T. (2011). E-learning in the 21st century: A framework for
research and practice (2nd ed. ed). New York, NY, USA: Routledge.
Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based
environment: Computer conferencing in higher education. The Internet and Higher
Education, 2, 87–105. https://doi.org/10.1016/S1096-7516(00)_00016-6
Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence,
and computer conferencing in distance education. American Journal of Distance
Education, 15, 7–23. https://doi.org/10.1080/08923640109527071
Garrison, D. R., Anderson, T., & Archer, W. (2010). The first decade of the community of
inquiry framework: A retrospective. The Internet and Higher Education, 13, 5–9.
https://doi.org/10.1016/j.iheduc.2009.10.003
Garrison, D. R., Cleveland-Innes, M., & Fung, T. S. (2010). Exploring causal relationships
among teaching, cognitive and social presence: Student perceptions of the
community of inquiry framework. Internet and Higher Education, 13, 31–36.
Gašević, D., Kovanović, V., & Joksimović, S. (2017). Piecing the learning analytics
puzzle: A consolidated model of a field of research and practice. Learning: Research
and Practice, 3, 63–78.
Gevrey, M., Dimopoulos, I., & Lek, S. (2003). Review and comparison of methods to
study the contribution of variables in artificial neural network models. Ecological
Modelling, 160, 249–264. https://doi.org/10.1016/S0304-3800(02)00257-0
Goodfellow, I., Bengio, Y., Courville, A., & Bengio, Y. (2016). Deep learning (Vol. 1).
Cambridge: MIT press.
Graves, A., Mohamed, A.r., & Hinton, G. (2013). Speech recognition with deep recurrent
neural networks. In 2013 IEEE international conference on acoustics, speech and signal
processing (pp. 6645–6649). IEEE.
Gunning, D. (2017). Explainable artificial intelligence (xai). Defense Advanced Research
Projects Agency (DARPA). nd Web 2.
Guo, S. X., Sun, X., Wang, S. X., Gao, Y., & Feng, J. (2019). Attention-based character-
word hybrid neural networks with semantic and structural information for
identifying of urgent posts in MOOC discussion forums. IEEE Access, 7,
120522–120532.
Hayati, H., Chanaa, A., Idrissi, M. K., & Bennani, S. (2019). Doc2vec &naïve bayes:
Learners’ cognitive presence assessment through asynchronous online discussion tq
transcripts. International Journal of Emerging Technologies in Learning, 14.
Hu, Y., Donald, C., Giacaman, N., & Zhu, Z. (2020). Towards automated analysis of
cognitive presence in MOOC discussions: A manual classification study. In The 10th
international conference on learning analytics and knowledge (pp. 135–140). Frankfurt,
Germany: LAK ’20).
Kamath, C. N., Bukhari, S. S., & Dengel, A. (2018). Comparative study between
traditional machine learning and deep learning approaches for text classification. In
Proceedings of the ACM symposium on document engineering 2018 (pp. 1–11).
Kovanović, V., Gašević, D., & Hatala, M. (2014a). Learning analytics for communities of
inquiry. Journal of Learning Analytics, 1, 195–198.
Kovanović, V., Joksimović, S., Gašević, D., & Hatala, M. (2014b). Automated cognitive
presence detection in online discussion transcripts. Workshop Proceedings of LAK
2014, Indianapolis, IN, USA. URL: http://ceur-ws.org/Vol-1137/.
Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., & Siemens, G.
(2016). Towards automated content analysis of discussion transcripts: A cognitive
presence case. In Proceedings of the sixth international conference on learning analytics
and knowledge (pp. 15–24). New York, NY, USA: Association for Computing
Machinery. https://doi.org/10.1145/2883851.2883950.
Lai, S., Xu, L., Liu, K., & Zhao, J. (2015). Recurrent convolutional neural networks for
text classification. In Proceedings of the Twenty-ninth AAAI conference on artificial
intelligence (pp. 2267–2273). Austin, Texas, USA.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical
data. Biometrics.
Lee, S., & Lee, C. (2020). Revisiting spatial dropout for regularizing convolutional neural
networks. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-020-
09054-7
12
Computers and Education: Artificial Intelligence 2 (2021) 100037
Liu, C. J., & Yang, S. C. (2014). Using the community of inquiry model to investigate
students’ knowledge construction in asynchronous online discussions. Education
computing research, 51, 327–354. https://doi.org/10.2190/EC.51.3.d
Mcklin, T. E. (2004). Analyzing cognitive presence in online courses using an artificial neural
network. Atlanta, GA, USA: Ph.D. thesis.. URL: https://scholarworks.gsu.
edu/msit_diss/1/. aAI3190967.
McKnight, P. E., & Najab, J. (2010). Mann-whitney u test. The Corsini Encyclopedia of
Psychology, 1–1.
McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z. (2014). Automated evaluation
of text and discourse with Coh-Metrix. Cambridge University Press.
Miller, T. (2019). Explanation in artificial intelligence: Insights from the social sciences.
Artificial Intelligence, 267, 1–38.
Neto, V., Rolim, V., Ferreira, R., Kovanović, V., Gašević, D., Lins, R. D., & Lins, R. (2018).
Automated analysis of cognitive presence in online discussions written in
Portuguese. In European conference on technology enhanced learning (pp. 245–261).
Springer. Springer International Publishing. https://doi.org/10.1007/978-3-319-
98572-5_19. URL:.
Pennington, J., Socher, R., & Manning, C. D. (2014). Glove: Global vectors for word
representation. In Proceedings of the 2014 conference on empirical methods in natural
language processing (pp. 1532–1543). EMNLP).
Rivard, R. (2013). Measuring the mooc dropout rate. Higher Education, 8, 2013.
Rolim, V., Ferreira, R., Kovanović, V., & Gašević, D. (2019). Analysing social presence in
online discussions through network and text analytics. In Proceedings of the 19th IEEE
international conference on advanced learning technologies (ICALT’19) (pp. 163–167).
New Jersey, USA: IEEE. https://doi.org/10.1109/ICALT.2019.00058.
Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (1999). Assessing social presence
in asynchronous text-based computer conferencing. The Journal of Distance
Education, 14, 50–71.
Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (2001). Methodological issues in
the content analysis of computer conference transcripts. International Journal of
Artificial Intelligence in Education (IJAIED), 12, 8–22.
Rourke, L., & Kanuka, H. (2009). Learning in communities of inquiry: A review of the
literature (winner 2009 best research article award). International Journal of E-
Learning & Distance Education/Revue internationale du e-learning et la formation à
distance, 23, 19–48.
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by
back-propagating errors. Nature, 323, 533–536.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,
Karpathy, A., Khosla, A., Bernstein, M., et al. (2015). Imagenet large scale visual
recognition challenge. International Journal of Computer Vision, 115, 211–252.
Samek, W., & Müller, K. R. (2019). Towards explainable artificial intelligence. In
Explainable AI: Interpreting, explaining and visualizing deep learning (pp. 5–22).
Springer.
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017). Smoothgrad:
Removing noise by adding noise. arXiv preprint arXiv:1706.03825.
Strijbos, J. W. (2011). Assessment of (computer-supported) collaborative learning. IEEE
Transactions on Learning Technologies, 4, 59–73.
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D.,
Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. In
Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern
Recognition 07-12-June (pp. 1–9). https://doi.org/10.1109/CVPR.2015.7298594.
arXiv:1409.4842.
Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: Liwc
and computerized text analysis methods. Journal of Language and Social Psychology,
29, 24–54.
Van Asch, V. (2013). Macro-and micro-averaged evaluation measures (Vol. 49). Belgium:
CLiPS.
Wang, X., Yang, D., Wen, M., Koedinger, K., & Rosé, C. P. (2015). Investigating how
student’s cognitive behavior in mooc discussion forums affect learning gains. International
Educational Data Mining Society.
Waters, Z., Kovanović, V., Kitto, K., & Gašević, D. (2015). Structure matters: Adoption of
structured classification approach in the context of cognitive presence classification.
In Information retrieval technology (pp. 227–238). Cham, Switzerland: Springer
International Publishing. https://doi.org/10.1007/978-3-319-28940-3_18.
Wei, X., Lin, H., Yang, L., & Yu, Y. (2017). A convolution-LSTM-based deep neural
network for cross-domain MOOC forum post classification. Information, 8, 1–16.
https://doi.org/10.3390/info8030092
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T.,
Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y.,
Plu, J., Xu, C., Le Scao, T., Gugger, S., … Rush, A. (2020). Transformers: State-of-the-
art natural language processing. In Proceedings of the 2020 conference on empirical
methods in natural language processing: System demonstrations (pp. 38–45). Association
for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-demos.6.
Online https://www.aclweb.org/anthology/2020.emnlp-demos.6.
Yang, S., Wang, Y., & Chu, X. (2020). A survey of deep learning techniques for neural
machine translation, Article 07526. arXiv:2002.
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018). Recent trends in deep learning
based natural language processing. Ieee Computational intelligenCe Magazine, 13,
55–75.