Prediction and Localization of Student

Engagement in the Wild

Amanjot Kaur Aamir Mustafa Love Mehta Abhinav Dhall
Learning Affect & Semantic Imaging Analysis Group,
Indian Institute of Technology Ropar
{amanjot.kaur, 2014csb1018, abhinav}@iitrpr.ac.in aamir 281btech14@nitsri.ac.in

Abstract—Digital revolution has transformed the traditional

teaching procedures, students are going online to access study
materials. It is realised that analysis of student engagement
in an e-learning environment would facilitate effective task
accomplishment and learning. Well known social cues of en- /HYHO
gagement/disengagement can be inferred from facial expressions,
body movements and gaze patterns. In this paper, student’s
response to various stimuli (educational videos) are recorded and
cues are extracted to estimate variations in engagement level. We
study the association of a subject’s behavioral cues with his/her
engagement level, as annotated by labelers. We have localized
engaging/non-engaging parts in the stimuli videos using a deep
multiple instance learning based framework, which can give use-
ful insight into designing Massive Open Online Courses (MOOCs)
video material. Recognizing the lack of any publicly available
dataset in the domain of user engagement, a new ‘in the wild’
dataset is curated. The dataset: Engagement in the Wild contains
264 videos captured from 91 subjects, which is approximately
16.5 hours of recording. Detailed baseline results using different
classifiers ranging from traditional machine learning to deep
learning based approaches are evaluated on the database. Subject
independent analysis is performed and the task of engagement /HYHO
prediction is modeled as a weakly supervised learning problem.
The dataset is manually annotated by different labelers and
the correlation studies between annotated and predicted labels Fig. 1. Frames from our engagement database (with low to high
of videos by different classifiers are reported. This dataset engagement [0-3]) with varied recording environments.
creation is an effort to facilitate research in various e-learning
environments such as intelligent tutoring systems, MOOCs, and
others. Download available at http://iitrpr.ac.in/lasii.
Index Terms—Dataset for Student Engagement, Engagement sistence and concentration. Several traditional methods and
Detection, Engagement Localization, E-learning Environment measures introduced in the literature to assess the level of
engagement have their own relevance. Different methods such
I. INTRODUCTION as self-reports: participants answer the set of questions related
Human beings require time and persistent effort to perform to their experience such as level of engagement, interest and
a task such as learning. On the same lines, the researchers have so on [3]. An observational study done by the external expert:
argued the importance of continuous effort or engagement to Focus on the behavioral analysis of students.
accomplish the learning task [1]. With the advent of digital Traditional measurement methods are not sufficient to measure
technologies, the traditional classroom teaching is transformed engagement in all the contexts. So automatic engagement
into advanced learning environments such as an Intelligent assessments for the digitally transformed learning environment
Tutoring System (ITS) and e-learning environment such as are required. These techniques analyze various facial cues,
Massive Open Online Courses (MOOCs). As a result, student body posture and well known social cues of engagement
engagement in the e-learning environment has also adapted in and disengagement captured automatically using affective
its own way. computing techniques. These techniques are sensitive to the
In the literature, Student engagement is defined as a com- engagement levels variations over time. Moreover, automatic
plex structure with multi-dimensions and components. Vari- measures can facilitate timely intervention to change the
ous other works in literature have classified engagement in course of fading engagement level [4].
different ways. It is evident that student engagement plays a key role in
Fredricks et al. [2] as behavioral engagement component, the learning process, so its automatic measurement is also
which explains student involvement in terms of effort, per- important in today’s digital learning environment. The paper

978-1-5386-6602-9/18/$31.00 ©2018 IEEE

 Name Features Subj. Method
D’Mello et al. [12] FACS & Log 28
Grafsgaard et al. [3] FACS 67 Regression
Whitehill et al. [13] FACS 34 Regression
Grafsgaard et al. [14] FACS, Log - HMM
Gupta et al. [15] Facial Features 112 DNN
Xian et al. [11] PPG 18 RBF+SVM
Arroyo et al. [16] Video 60 Regression
&Physiological
Bosch et al. [17] Facial Features 137 Regression
Body Posture
TABLE I
I MPORTANT WORKS FOR AUTOMATIC ENGAGEMENT DETECTION .
proposes an automatic pipeline for predicting user’s engage-
ment, while they are watching educational videos such as the
ones in Massive Open Online Courses (MOOCs). Secondly, A
new ‘Engagement Detection in the wild’ video-based database
(Figure 1) is also introduced. User engagement analysis is
of importance to learning environments, which involves in-
teraction between humans and machines. Generally, a course
instructor is able to assess the engagement of students in a
classroom environment based on their facial expression. Other
well known social cues such as yawning, glued eyes, and body
posture are also handy in analyzing the behavioral change
of the participants in the natural environment. However, this
assessment becomes tricky, when students are watching study
material such as videos, on their computers and mobile phones
in different environments. These different environments can be
their room, office, cafeteria, outdoor and so on.
This work is related to the estimation of perceived engage-
ment by the external observer and it is appropriate for the
e-learning environment. Various examples of environments in
which the engagement of user is a vital component to analyze
are: e-learning, health sector such as conducting autism-related
studies and analyzing driver engagement in automated cars etc.
Automatic engagement prediction can be based on various
kinds of data modalities. It is argued that the student response
can be used as an indicator of engagement in intelligent
tutoring systems [5]. Another approach is based on features
extracted on the basis of facial movements [6], [7]. Automated
measures such as response time of a student to problems
and test quizzes are used in ITS [8]. Physiological and
neurological measures, requiring specialized sensors, such as
electroencephalogram, heart rate, and skin response have been
used to measure engagement [9]–[11].
Table I summarizes important engagement detection ap-
proaches. Xiang et al. have studied the impact of distractions
or multitasking in MOOCs through Mobile apps [11]. In
this work, along with recorded photoplethysmography (PPM)
signals post the lecture, self-reports and quiz related to video
content was asked to 18 subjects who watched two stimuli as
lecture videos.
In an interesting work, Whitehill et al. analyzed the behav-
ioral engagement and defines four levels of engagement. The
analysis is performed on the Facial Features (FACS) with
the help of classifier [13]. In another experiment, affective
states pertaining to engagement such as Interest, Confidence,
Excitement and Frustration (1-5 scale) were used. In this
experiment, video and physiological sensor signals were used
to capture different modalities to analyze affective states of
students while solving mathematical problems of SAT. Dataset
was self-annotated and linear regression was used to regress
the value of different affective states [16]. In another study
conducted by Bosch et al. in the real world settings, various
affective states of engagement such as boredom, engaged,
concentration, confusion, frustration and delight were used.
Features were in the form of facial expressions and body
movements. Regressors such as BayesNet, clustering, regres-
sion were used to regress different dimensions of the network
[17].
Surveys such as [18] [19] discuss the important work done in
the field of affective computing. Based on the knowledge in
affective computing, researchers have recently tried to predict
user engagement. El Kaliouby et al. [20] argue that user
engagement prediction is a challenging problem and has many
applications in autism-related studies. Furthermore, Tawari et
al. [21] found that a car driver’s affective state recognition
in terms of the engagement level provides important cues in
detecting drowsiness levels while driving. Navarathna et al.
[22] proposed an interesting pipeline for analyzing a movie’s
affect by predicting viewer’s engagement in the movie. User
engagement level recognition is inevitable in e-learning and is
useful in detecting states such as fatigue, lack of interest and
difficulty in understanding the content.
Temporally constrained and dynamic nature of engagement
as a affective state signifies the fact that it doesn’t remain
same, moreover, varies between low and high. Pattern of user
engagement can give various interesting insights such as: for
how long users concentrate, at which point in a lecture video
the user loses interest etc. This is useful for successful delivery
of educational content and deciding evaluation criteria for
user engagement [23].The number of datasets related to user
engagement in the online learning environment released are
limited [13], [24], [25]. With the advent of deep learning
frameworks, larger sized databases representing diverse set-
tings are required. The availability of datasets related to user
engagement especially student engagement in online courses
would help in understanding the problems faced by students
such as: loss of interest, fatigue, boredom etc in the online
learning environment.
In this paper, we introduce a database for common bench-
marking and development of student engagement assessment
methods, which work on‘in the wild’ conditions. Here ‘in the
wild’ refer to different backgrounds, occlusion, illumination
and head poses etc. Video level engagement prediction task
is modeled as a weakly labeled learning problem. Multiple
Instance Learning (MIL) is followed as the paradigm for this
weakly supervised learning. MIL has been found extensively
useful for tasks, in which the dataset has weak or noisy labels
[26] and has been recently used in affect analysis tasks such as
facial expression prediction [27] and image emotion prediction
[28]. MIL assumes that the data could be presented with the
help of a set of sub-instances or bag of sub-instances as {Xi },
where each bag consists of sub-instances {xij } and the weak
labels {yi } are present at the bag level only [29], where i is
the number of instances or bags and j is the number of sub
instances in a particular bag.
Our work is based on Deep Multi-Instance Learning
(DMIL) framework, which makes use of complex features
that can be learned by deep neural networks along with MIL
technique for a weakly supervised problem.Sikka et al. [30]
studied pain localization in a video in a weakly supervised
fashion using the MIL framework. Tax et al. [27] proposed
an MIL based framework to select the concept frames using
clustering technique for multi-instance learning.
The main contribution of this work is to formulate student
engagement prediction and localization as a MIL problem
and derive the baseline scores based on DMIL. The rationale
behind MIL is that labeling of engagement at frequent intervals
in user videos is expensive and noisy.
The rest of the paper is organized as follows: In section II,
we introduce the dataset and details about the data collection
and annotation. In section III, we discuss the methodology
used for student engagement prediction and localization. In
section IV, we describe the experimental setup and the results
obtained respectively.
II. DATA C OLLECTION
In this section, we discuss the data recording paradigm
used in this work. The experiment has two parts: i) Video
recordings of the subjects, while they watch the stimuli videos
(each video around 5 minutes long) and ii) A verbal feedback
section, in which the subject is given 15 seconds to speak
about the video. In those 15 seconds the subjects are asked to
express their views about the video, especially if they found
the video interesting?, would they like to watch the video
again?, which part of the video was most/least engaging?,
any comments/suggestions on how the video could have been
made more engaging etc. Based on the content of the stimuli
videos, we choose purely educational videos Learn the Korean
Language in 5 minutes, a pictorial video (Tips to learn
faster) and (How to write a research paper). This was aimed
to capture both focused and enjoyable settings which allow
natural variations. In the experiment section, as a baseline, we
explore the video signals only.
The dataset contains 91 subjects (27 females and 64 males).
The age range of the subjects is between 19-27 years. A total
of 264 videos are collected, each approximately 5 minutes
long. The dataset is collected in unconstrained environment i.e.
at different locations such as the computer lab, hostel rooms,
open ground etc. Inorder to have more uncontrolled real life
scenarios, we also include a video conferencing based setup. In
this, the subject watched the stimuli on their computer (full)
screen and in parallel a Skype-based video session was set
up. We captured the video region of the Skype application
at the other end of the Skype video-call. It was made sure
that the subject was not disturbed by the video recording. The
usefulness of recording over Skype is that frame drop, network
latency and disturbance (of the subject) can help in simulating
different environments.
The videos are captured at a resolution of 640 x 480 pixels
at 30 fps using a Microsoft Lifecam wide-angle F2.0 camera
(excluding the Skype-based recordings). The audio signal is
recorded using a wideband microphone, which is part of
the camera. Apowersoft Screen Recorder software is used to
capture the Skype videos.
The dataset and it’s accompanying baseline code are pub-
licly available to facilitate open research. The dataset also
forms the part of the engagegent prediction in the wild
challenge as part of the Emotion Recognition in the Wild
challenge 20181 . The consent of the subjects has been taken
following the due process. To the best of our knowledge, this
is one of the first publicly available user engagement datasets
being recorded in diverse conditions.
A. Data Annotation
A team of 5 annotators labelled the engagement intensity
of the subjects in videos. The labelers were instructed to label
the videos on the basis of their engagement intensity (from
facial expressions) ranging from 0 to 3. Audio for the videos
was turned off during labeling. The engagement categories are
inspired by the work of Whitehill et al. [13]. The engagement
intensity mapping is as follows: Not Engaged means that
the subject is completely disengaged, e.g. the subject seems
uninterested and looks away from the screen frequently. Less
Engaged applies to be barely engaged e.g. the subject barely
opens his/her eyes, moves restlessly in the chair. Engaged
applies that the subjects seem engaged in the content, e.g.
the subject seems to like the content and is interacting with
the video. Highly Engaged means that the subject seems to
be fully engaged, e.g. the subject was glued to the screen and
was focused. Note that the labels above were used to make the
labellers understand the steps of the engagement scale (0-3).
The Train, Validation and Test partitions contain 147, 48
videos and 67 videos, respectively. There is a natural progres-
sion in the intensity of engagement, therefore, we are looking
at the task of engagement problem as a regression problem.
For judging the annotator's reliability, we computed weighted
Cohen's K with quadratic weights as the performance metric
(similar to [13], [15]) as the labels are not categories but
intensities of engagement {0, 1, 2, 3}. Any annotator whose
agreement coefficient is less than 0.4 is marked less reliable
and the label is ignored. After denoising, the labels are
averaged and rounded off to the nearest integer to give a
ground truth engagement rating to the video. Example images
for engagement labels are shown in Figure 1.
B. Weighted Labels
Annotation process should be such that there should be less
subjective bias introduced while labeling the videos. In this
work, low agreement among the annotators and subjective bias
in the annotation process is addressed based on the technique
1 https://sites.google.com/view/emotiw2018

;<=> ?;= ;=@
DEFG,HF/I'%$:3'#
A''./B':3%5/B'$C-30
678'-/)'9:'"&'
Fig. 2. Proposed deep multi-instance network: LBP-TOP features are extracted from facial video segments. Linear regression using DNN is
employed and instance level responses are ranked. Finally, max and mean pooling are used to predict the subject engagement intensity.
introduced in the work of Joshi et al. [31]. The notion of the
credibility of the user is defined in the work as consistency
of the annotator in giving engagement label to the same clip
which is shown at different time. We have used the same
notion of credibility to calculate the weight of rating of each
annotator based on the deviation in rating given to the same
clips shown at different time stamps. This gives the weighted
labels and serves the purpose to penalize the raters who are
prone to give more erroneous ratings.
III. P ROPOSED F RAMEWORK
Our framework performs subject’s engagement detection
and localization. The steps are as follows: The face and
facial landmark positions [32] are detected in each frame. The
video is then sub-sampled into small segments (instances) as
discussed in detail in Section III-A. Features are extracted for a
window of frames. The window then slides, next set of frames
are extracted to compute features. This stage is discussed in
Section III-B and III-C. The engagement prediction and local-
ization are computed with the deep MIL network using mean
and top-k pooling for regression. The details are discussed
in Section III-D. Another model to process the sequence
of segments is constructed using Long Short Term Memory
(LSTM) based network using mean pooling. The details are
discussed in III-D
A. Pre-Processing
Let v = [f1 , f2 , f3 , · · · , fn ] be a video clip of a subject in
response to a stimuli, where f denotes a video frame and n
are the total number of frames. The OpenFace library [32] is
used to track the facial landmarks first in order to register the
facial region, which is then cropped to a final resolution of
112 × 112. In order to speed up the experiments, each video
is therefore sampled at a different frame rate (they recorded
in varied conditions with different cameras) so that all have
same number of frames.
B. LBP-TOP Feature Extraction
We are interested in computing spatio-temporal features to
capture the changes in the facial region. Local Binary Patterns
,-./0/!"#$%"&'#
;
!"#$%"&'()'*+'"$
12'3%*'/-4/%55/!"#$%"&'#
J!D/D-##'#
from Three Orthogonal Planes (LBP-TOP) [33] is a standard
spatio-temporal texture descriptor and is used in our work.
1) Shorter segment: A sliding window vf =
[f1 , f2 , · · · , fm ] of m frames is taken at a time
and LBP-TOP features are extracted. A smaller value
of m would fail to efficiently approximate the entire
expression while a larger value of m would yield results
with poor frame-wise specificity. We therefore deem
m=50 (approx. 10 seconds in the sampled video) as
the appropriate length of each window for expression
recognition. After extracting features for the first m
frames, the sliding window is moved by p frames. The
value of p is empirically set to 25 (overlap of 50%) as
it was found to provide adequate accuracy.
C. Eye Gaze and Head Pose Feature Extraction
Student activities give cues to infer learning centered affec-
tive states such as engagement and others. In this paper, social
signal in the form of eye gaze movement and head movement
features are extracted using OpenFace toolkit.
1) Head Pose (HP): The movement of head an indicator of
engagement intensity. OpenFace returns 3 points related
to head movement in 3D space. These points give the
location of head w.r.t camera. Dynamics of these points
give insights about head movement of a subject.
2) Eye Gaze (EG): The gaze of left and right eye is
extracted using Open Face for every frame in the videos.
It returns 3 dimensional direction vector for both the
eyes. The movement of eyes give cues for the engage-
ment level of the student such as closed eyes, looking
away from the screen could be negative indicators of the
engagement level.
Various challenges involved in constructing video level repre-
sentation can be summarized as limited number of videos,
large decision unit (label is assigned to the whole video).
Aforementioned challenges highlight the fact that the whole
video contains lots of indicator of engagement in its short
temporal segments which in turn acts as a motivation to
construct segment level features using frame level features as
mentioned earlier. Each frame is represented as 9-dimensional
 "#$%*)/$)"0*1234
!234,+,25.
H-$%#0,&
1'+,*%"603
2%3$%0*&
!"#$%&'%(%'& )%#*+"%,4&
)%#*+"%,&- 2*#*6,*61#'& EFG
./%0!#1% $8$%0*,&9
,*77%(: $%#0
!"#$%&'($)* +,-. "#$%*)/$)"0*1234 "#$%*)/$)"0*1234
!234,+,25. !234,+,6.
Fig. 3. Different approaches discussed in experiments to regress engagement level of a video. Approach ’A’ is Multi instance learning
approach using Kmeans Clustering. Approach ’B’ is related with single video level and representation and Approach ’C’ is related to Deep
Multi-Instance learning approach.
feature vector. The statistical feature representation is useful
to capture the dynamics of various social cues present in the
form of eye gaze feature and head movement. We have applied
various moment function in the form of standard deviation
and mean on eye gaze and head pose as different modalities.
For each modality segment level features are constructed as
follows. Eye gaze is returned as a 6-dimensional feature
vector to capture the eye gaze behavior by the OpenFace
utility. Subsequently, two-moment functions such as mean
and standard deviation are applied on the set of frames to
construct a segment level features. As a result, eye gaze
movement for each segment is represented using 2*6=12
dimensional feature vector. Head pose modality represents the
movement of the head. It is a 3-dimensional vector, returned
by OpenFace. Following the application of moment functions,
segment level feature is of 2*3=6 dimensional feature vector
representing dynamics of head movement for a segment. These
features are concatenated for each segment which makes it
an 18-dimensional feature vector as shown in Figure III.
For dimensionality reduction, Principal Component Analysis
(PCA) is applied on a normalized segment level features of
all the videos.
D. Deep Multi-Instance Network for Engagement Localization
and Prediction
We design a deep multi-instance network for the task of
user engagement level prediction and localization. Figure II
shows the proposed network architecture with multiple dense
regression layers, one ranking layer and one multi-instance
loss layer. Pooling functions play an important role in MIL
deep networks to fuse instance-level outputs. Two pooling
schemes are employed for combining multiple instances, i)
the top k-pooling based multi-instance learning taking only
the largest ten elements (k = 10) from the ranking layer and
ii) mean-pooling based multi-instance learning taking mean of
all the elements. The details of these schemes are given later
in the section.
A%'#B%'& A%3"%,,8",4&
7#*#&- 1'+,*%"& 25A:&C#D%,6#0& G
A673%
$%#0:&$87%
2%3$%0*& A#078$&
$%#04&567%8& !8"%,*& C
'%(%'&)%#*+"%, A%3"%,,8"
!2,+,6.
;<<:&=2>?:&
F
;%%/?@=
1) Top-l Pooling based Multi-instance Learning: A general
MIL framework for classification problems defines two kinds
of bags, positive and negative. In MIL, if at least one instance
is positive then the bag is assigned a positive label [34].
In our dataset, since the ground truth labels are given
for the entire video spanning 5 minutes, we take the first l
largest intensities of instances from the ranking layer instead
of adopting the general multi-instance learning assumption
of considering the largest contributing instance. The hyper-
parameter l has been chosen from the following set of values
l ∈ {1, 5, 10, 20, 30} and is empirically set as 10.
2) Mean Pooling based Multi-instance Learning: The en-
gagement intensity fluctuates among various instances over
the span of the video. We adopted another scheme of pooling
in which the final predicted intensity for ith bag is taken to
be the mean of all the intensities from the ranking layer i.e.
1
∑
M
M xij , where xij represents the j th segment of ith video.
j=1
3) Sequence Network: LSTM based framework is success-
ful in sequence based prediction task. It motivates us to
design a network which leverage insights from the sequence of
segments. The intuition behind using the LSTM based network
is that engagement level varies across the segments. The
annotators gives attention to different segments of the recorded
video and gives label on the basis of perceived engagement
level which implies different segments contribute differently
to the overall annotation of the video.
The Figure 4 presents the proposed architecture of the se-
quence based architecture. In this network the first layer is
LSTM layer which has 32 hidden units. The purpose of this
layer is to compute an activation value for the sequence of
segments of a video. This maps the multi segment represen-
tation of a video to a the feature vector which represents the
activation for different segments of the video. It is followed
by flatten layer which serves the purpose to make a single
feature vector of activations of all the segments of the video.
This feature vector is passed to another module which consists
of three dense layers with sigmoid activations followed by
average pooling. This module is responsible to learn the
function which maps the segment level activations to the
engagement level of a video.
IV. E XPERIMENTS
The details about the dataset which was used for experi-
mentation can be found in Section II. We followed subject
independent cross-validation for dataset split. In this section,
we discuss the two approaches followed to predict the engage-
ment level of individuals and result comparisons are shown in
Section V.
A. Multiple Instance Learning Approach
The following section explains the workflow ’A’ of Figure
III.
In this approach a temporal scanning window is taken and
features are extracted for each window segment of the video.
The training data is in the form of bags, B = {Xi , yi }N i=1 ,
where Xi = {xij }M j=1 , y i ∈ Y and M (number of instances
per video) is equal to 150 and N is the total number of videos
in the training dataset. T
The segments obtained for each sequence are labeled using
the following two approaches:
1) Noisy/ Weak Labeling: All the segments in the training
data are assigned the same label as that of their corresponding
video sequence. This strategy is similar to that employed in
previous works [35], [36].
2) Relabeling segments using Kmeans clustering: In this
approach, the segments are divided into k clusters using
the Kmeans clustering over the features. The hyperparam-
eter k is selected from the following set of values k ∈
{5, 10, 15, 20, 25, 30, 35, 40}. Segments falling in a cluster are
relabeled as follows:
i) All the segments in a cluster are relabeled by the mode
of the labels in that cluster. ii) All the segments in the cluster
are relabeled by the mean of the labels in that cluster.
3) Regressors: Several state of the art machine learning
models are trained for the regression task. i) Support Vector
Regression (SVR): This classifier works on the relabeled seg-
ments and regress the value of the video as the mean value of
all the segments belonging to that particular video. A Gaussian
kernel is trained by performing a grid search on the selection
of parameters C and sigma. The mse is obtained for the best
combination of C and Sigma for different mode of segment
relabeling techniques as mentioned earlier. For mode analysis
C and Sigma is (1,1) whereas for mean analysis C and Sigma
is (1,4).
BayesianRidge Regressor is used to regress the engagement
level of a video from the regressed values of its segments with
different parameter settings.
B. Traditional Machine Learning approach for single video
level representation
In this paradigm, the video level representation is con-
structed using segment level feature representation which is
explained as workflow B of Figure III. . The 18 dimensional
statistical feature representation is extracted using standard
deviation and mean of the segment. It is followed by dimen-
sionality reduction using PCA and as a result each segment is
represented using 4 dimensional feature vector which is nor-
malized. The video level feature is constructed as an average of
all 4 dimensional feature vector of all the segments. As a result
to each video is represented as a 4 dimensional feature vector
which is passed to a regressor. In our case RandomForest (RF)
Regressor is optimized using cross validation based on grid
search applied on hyperparameters. Using metric performance
the depth of the tree is selected as 5 and no of trees parameter
is selected as 10.
C. Deep Multi Instance Learning
In this section, the workflow C of Figure III is introduced.
DNN:
The architecture for Deep MIL approach is shown in Figure
II. Here segment level LBPTOP features are extracted. These
features are then passed to a neural network with four dense
layers, obtaining a single feature for each segment. To obtain
a single label for the entire video, segment-wise features are
then passed through a Deep MIL loss layer using two pooling
schemes, namely Top-l and Mean as mentioned in Section
III-D.
We also evaluated Rectified Linear Unit (ReLU): f (x) =
max(0, x), activation function the most frequently used activa-
tion function as it allows gradient flow for all positive values.
Proposed Method: The architecture for LSTM based Deep
MIL approach is shown in Figure 4 The eye gaze and head
pose features of the segment are fused for feature fusion as
explained in section III-A. These features are passed through
the LSTM layer which returns activation for each segment
of the video, passed to the flatten layer and then flattened
feature vector is passed to the network of three dense layers
followed by average pooling which gives the regressed value
of engagement level of a video.
D. Metrics
Various performance metrics used in this work are Mean
Square Error (MSE) and Pearson Correlation Coefficient
(PCC). The analysis performed on weighted labels is reported
using weighted mean square error (MSE wgt) and weighted
Pearson Correlation Coefficient (PCC wgt).
V. R ESULTS AND D ISCUSSIONS
In this paper, we have presented the efficacy of different
approaches based on short temporal segments to regress the
engagement level of the videos. There are various challenges
involved in the introduced video dataset. Firstly, the imbalance
in the distribution of training and test samples for different
engagement levels. Secondly, the total number of labeled video
samples across training and test sets is 267 which very less.
Thirdly, the length of every labeled video in the dataset is
5mins on an average which is a much larger decision unit.
This makes the problem even more complex. Lastly, forming
 !"#$% &'() 89: 89: 8;;
*+,%%-"
,-'.%*'/01/%22/!"#$%"&'#

A'%5/B0#'/C/DE'/F%G' H!I/I0##
345'0/)'67'"&'
!"#$%"&'()'*+'"$
<''=/>'7.%2/>'$?0.@

Fig. 4. The proposed deep multi-instance network pipeline. First, Eye-Gaze and Head Pose features are extracted from facial video segments.
Second, the sequence of segments is passed through a LSTM layer followed by 3 dense layers, finally average pooling layer is applied.

discriminative temporal segment level features from frame Method MSE PCC MSE wgt PCC wgt
level features and in turn, forming discriminative video level RF 0.09 0.17 0.05 0.26
feature using segment level features using statistical or deep LSTM 0.09 0.05 0.06 0.10
DNN 0.15 0.01 0.08 0.01
learning based methods.
We have given baseline results for the dataset using dif- TABLE II
ferent approaches with feature fusion of the Eye Gaze and P ERFORMANCE COMPARISON OF DIFFERENT APPROACHES FOR THE
FUSED FEATURES OF H EAD P OSE AND E YE G AZE BASED ON NON
Head Pose Movement in Table II. The results show better WEIGHTED AND WEIGHTED LABELS
performance on Random Forest (RF) regressors with single
video representation constructed using segment level features Method Relabeling Mode MSE MSE wgt
as given in Figure III as workflow ’B’. The LSTM based SVR Noisy 0.15 0.09
SVR Mode 0.15 0.09
deep network also shows comparable results whereas the
SVR Mean 0.15 0.09
conventional DNN based network fails to capture the dynamics
of the hierarchy of features used at different levels. Another TABLE III
BASELINE PERFORMANCE COMPARISON OF DIFFERENT ML APPROACHES
perspective is analyzing the results based on weighted labels. FOR LBPTOP. H ERE R ELABELING REFERS TO THE METHOD OF
As given in Table II, clearly the performance metrics improves SEGMENT- WISE LABELING .
in the weighted labels approach. We have performed subject
independence cross validation and reported the average results Method Relabeling Mode MSE MSE wgt
SVR Noisy 0.09 0.09
in all the cases.
SVR Mode 0.15 0.09
We also conducted experiments for the LBPTOP features with SVR Mean 0.15 0.09
different regressors like SVR, Bayesian Ridge Regressor and
TABLE IV
Random Forest Regressor to predict engagement level. We BASELINE P ERFORMANCE COMPARISON OF DIFFERENT RELABELING
also used various relabeling techniques like mean, median TECHNIQUES FOR F USED FEATURE OF H EAD P OSE AND E YE G AZE .
and mode over k-means clusters in combination with these
regression algorithms. The corresponding results of SVR are
shown in Table III. It is clear from the results that this
technique doesn’t work as it predict almost the same label results on the new dataset using various techniques. In au-
for all the videos so the MSE is low but the PCC is nearly 0 tomatic engagement detection, most of the current techniques
for all the combinations. used like self-reports, teacher introspective evaluation are cum-
Similarly in Table IV, where different relabeling techniques bersome and lack the ability to localize student engagement.
are applied in combination with different regressors on the There is a need to come up with an intelligent predictive tech-
feature fusion of Eye Gaze and Head Pose Movement. We nique for this problem. An important requirement to achieve
observed similar results as in the case of LBPTOP features. that is the availability of a real-world engagement dataset
The MSE and MSE wgt metrics are small but PCC is also with diverse subjects. In prior literature, different modalities
almost 0 which implies the failure of this modeling technique. such as student response, audience movie rating, psychological
This can be explained by arguing about less number of samples and neurological sensor readings etc. have been used for
and imbalanced distribution of the sample labels. Another engagement prediction. However, we have worked just with
common observation among the failed techniques is the same facial and head movement features of the subject and shown
label prediction for all the instances. that the segment based RF model approach and LSTM based
deep sequence network gives a good baseline performance.
VI. CONCLUSION & FUTURE SCOPE We also tried to solve the problem of long ( 5mins) labeled
In this paper, we presented engagement dataset that we have videos using various temporal segment labeling techniques.
prepared in the wild settings. We have also provided baseline The results show that these techniques do not work well with
either the LBPTOP or facial features as mentioned earlier. The
presented dataset can be used for further impactful research
work in this area. Also, the proposed predictive methods can
be extensively used in designing study material for MOOCs
and ITS to substantially decrease the dropout rates.
VII. ACKNOWLEDGMENTS
We would like to thank Nvidia for providing the Titan X
card for research.
R EFERENCES
[1] M. E. Nicholls, K. M. Loveless, N. A. Thomas, T. Loetscher, and
O. Churches, “Some participants may be better than others: Sustained
attention and motivation are higher early in semester,” The Quarterly
Journal of Experimental Psychology, vol. 68, no. 1, pp. 10–18, 2015.
[2] J. A. Fredricks, P. C. Blumenfeld, and A. H. Paris, “School engagement:
Potential of the concept, state of the evidence,” Review of educational
research, vol. 74, no. 1, pp. 59–109, 2004.
[3] J. F. Grafsgaard, R. M. Fulton, K. E. Boyer, E. N. Wiebe, and
J. C. Lester, “Multimodal analysis of the implicit affective channel in
computer-mediated textual communication,” in Proceedings of the 14th
ACM international conference on Multimodal interaction. ACM, 2012,
pp. 145–152.
[4] S. D’Mello, E. Dieterle, and A. Duckworth, “Advanced, analytic, auto-
mated (aaa) measurement of engagement during learning,” Educational
psychologist, vol. 52, no. 2, pp. 104–123, 2017.
[5] K. R. Koedinger, J. R. Anderson, W. H. Hadley, and M. A. Mark,
“Intelligent tutoring goes to school in the big city,” 1997.
[6] S. K. D’Mello and A. Graesser, “Multimodal semi-automated affect
detection from conversational cues, gross body language, and facial
features,” User Modeling and User-Adapted Interaction, vol. 20, no. 2,
pp. 147–187, 2010.
[7] X. Xiao, P. Pham, and J. Wang, “Dynamics of affective states during
mooc learning,” in International Conference on Artificial Intelligence in
Education. Springer, 2017, pp. 586–589.
[8] E. Joseph, “Engagement tracing: using response times to model student
disengagement,” Artificial intelligence in education: Supporting learning
through intelligent and socially informed technology, vol. 125, p. 88,
2005.
[9] B. S. Goldberg, R. A. Sottilare, K. W. Brawner, and H. K. Holden,
“Predicting learner engagement during well-defined and ill-defined
computer-based intercultural interactions,” in International Conference
on Affective Computing and Intelligent Interaction. Springer, 2011, pp.
538–547.
[10] M. Chaouachi, P. Chalfoun, I. Jraidi, and C. Frasson, “Affect and
mental engagement: towards adaptability for intelligent systems,” in
Proceedings of the 23rd International FLAIRS Conference. AAAI Press,
2010.
[11] X. Xiao and J. Wang, “Undertanding and detecting divided attention in
mobile mooc learning,” in Proceedings of the 2017 CHI Conference on
Human Factors in Computing Systems. ACM, 2017, pp. 2411–2415.
[12] S. K. D’Mello, S. Craig, and A. Graesser, “Multimethod assessment of
affective experience and expression during deep learning.” International
Journal of Learning Technology, vol. 4(3-4), pp. 165–187, 2009.
[13] J. Whitehill, Z. Serpell, Y.-C. Lin, A. Foster, and J. R. Movellan, “The
faces of engagement: Automatic recognition of student engagementfrom
facial expressions,” IEEE Transactions on Affective Computing, vol. 5,
no. 1, pp. 86–98, 2014.
[14] J. F. Grafsgaard, K. E. Boyer, and J. C. Lester, “Predicting facial indi-
cators of confusion with hidden markov models,” in International Con-
ference on Affective Computing and Intelligent Interaction. Springer,
2011, pp. 97–106.
[15] A. Gupta and V. Balasubramanian, “Daisee: Towards user engagement
recognition in the wild,” arXiv preprint arXiv:1609.01885, 2016.
[16] C. R. Beal, R. Walles, I. Arroyo, and B. P. Woolf, “On-line tutoring
for math achievement testing: A controlled evaluation,” Journal of
Interactive Online Learning, vol. 6, no. 1, pp. 43–55, 2007.
[17] N. Bosch, S. K. D’Mello, R. S. Baker, J. Ocumpaugh, V. Shute,
M. Ventura, L. Wang, and W. Zhao, “Detecting student emotions in
computer-enabled classrooms.” in IJCAI, 2016, pp. 4125–4129.
[18] Z. Zeng, M. Pantic, G. Roisman, and T. Huang, “A Survey of Affect
Recognition Methods: Audio, Visual, and Spontaneous Expressions,”
IEEE Transaction on Pattern Analysis & Machine Intelligence, vol. 31,
no. 1, pp. 39–58, 2009.
[19] C. A. Corneanu, M. Oliu, J. F. Cohn, and S. Escalera, “Survey on rgb, 3d,
thermal, and multimodal approaches for facial expression recognition:
history, trends, and affect-related applications,” IEEE Transactions on
Pattern Analysis and Machine Intelligence, 2016.
[20] R. El Kaliouby, R. Picard, and S. BARON-COHEN, “Affective com-
puting and autism,” Annals of the New York Academy of Sciences, vol.
1093, no. 1, pp. 228–248, 2006.
[21] A. Tawari, S. Sivaraman, M. M. Trivedi, T. Shannon, and M. Tippelhofer,
“Looking-in and looking-out vision for urban intelligent assistance:
Estimation of driver attentive state and dynamic surround for safe
merging and braking,” in Intelligent Vehicles Symposium Proceedings,
2014 IEEE. IEEE, 2014, pp. 115–120.
[22] R. Navarathna, P. Carr, P. Lucey, and I. Matthews, “Estimating audience
engagement to predict movie ratings,” IEEE Transactions on Affective
Computing, 2017.
[23] S. Wu, M.-A. Rizoiu, and L. Xie, “Beyond views: Measuring
and predicting engagement on youtube videos,” arXiv preprint
arXiv:1709.02541, 2017.
[24] S. Attfield, G. Kazai, M. Lalmas, and B. Piwowarski, “Towards a science
of user engagement (position paper),” in WSDM workshop on user
modelling for Web applications, 2011, pp. 9–12.
[25] B. Sun, Q. Wei, J. He, L. Yu, and X. Zhu, “Bnu-lsved: a multimodal
spontaneous expression database in educational environment,” in Optics
and Photonics for Information Processing X, vol. 9970. International
Society for Optics and Photonics, 2016, p. 997016.
[26] J. Amores, “Multiple instance classification: Review, taxonomy and
comparative study,” Artificial Intelligence, vol. 201, pp. 81–105, 2013.
[27] D. M. Tax, E. Hendriks, M. F. Valstar, and M. Pantic, “The detection
of concept frames using clustering multi-instance learning,” in Pattern
Recognition (ICPR), 2010 20th International Conference on. IEEE,
2010, pp. 2917–2920.
[28] T. Rao, M. Xu, H. Liu, J. Wang, and I. Burnett, “Multi-scale blocks
based image emotion classification using multiple instance learning,”
in Image Processing (ICIP), 2016 IEEE International Conference on,
2016.
[29] J. Wu, Y. Yu, C. Huang, and K. Yu, “Deep multiple instance learning for
image classification and auto-annotation,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2015, pp.
3460–3469.
[30] K. Sikka, A. Dhall, and M. Bartlett, “Weakly supervised pain lo-
calization using multiple instance learning,” in Automatic Face and
Gesture Recognition (FG), 2013 10th IEEE International Conference
and Workshops on. IEEE, 2013, pp. 1–8.
[31] 22nd International Conference on Pattern Recogni-
tion, ICPR 2014, Stockholm, Sweden, August 24-28,
2014. IEEE Computer Society, 2014. [Online]. Available:
http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=6966883
[32] T. Baltrušaitis, P. Robinson, and L.-P. Morency, “Openface: an open
source facial behavior analysis toolkit,” in Applications of Computer
Vision (WACV), 2016 IEEE Winter Conference on. IEEE, 2016, pp.
1–10.
[33] G. Zhao and M. Pietikainen, “Dynamic texture recognition using local
binary patterns with an application to facial expressions,” IEEE trans-
actions on pattern analysis and machine intelligence, vol. 29, no. 6, pp.
915–928, 2007.
[34] T. G. Dietterich, R. H. Lathrop, and T. Lozano-Pérez, “Solving the
multiple instance problem with axis-parallel rectangles,” Artificial in-
telligence, vol. 89, no. 1, pp. 31–71, 1997.
[35] A. B. Ashraf, S. Lucey, J. F. Cohn, T. Chen, Z. Ambadar, K. M. Prkachin,
and P. E. Solomon, “The painful face–pain expression recognition using
active appearance models,” Image and vision computing, vol. 27, no. 12,
pp. 1788–1796, 2009.
[36] P. Lucey, J. Howlett, J. Cohn, S. Lucey, S. Sridharan, and Z. Ambadar,
“Improving pain recognition through better utilisation of temporal
information,” in International conference on auditory-visual speech
processing, vol. 2008. NIH Public Access, 2008, p. 167.