Using Programming Process Data to Detect Differences in
Students' Patterns of Programming
Adam S. Carter
Humboldt State University
Department of Computer Science
1 Harpst ST, Arcata, CA 95521 USA
Adam.Carter@Humboldt.edu
ABSTRACT
Analyzing the process data of students as they complete
programming assignments has the potential to provide computing
educators with insights into their students and the processes by
which they learn to program. In prior work, we developed a
statistical model that accurately predicts students' homework
grades. In this paper, we investigate the relationship between the
paths that students take through the programming states on which
our statistical model is based, and their overall course achievement.
Examining the frequency of the most common transition paths
revealed significant differences between students who earned A's,
B's, and C's in a CS 2 course. Our results indicate that a) students
of differing achievement levels approach programming tasks
differently, and b) these differences can be automatically detected,
opening up the possibility that they could be leveraged for
pedagogical gain.
CCS Concepts
• Social and professional topics➝Professional topics➝
Computing education➝Computer science education
• General and reference➝Cross-computing tools and
techniques➝Metrics.
Keywords
Programming State Model; Predictive measures; Learning
analytics; Educational data mining
1. INTRODUCTION
Given the relative ease of automatically collecting learning process
data, it is no surprise that computing education researchers are
exploring new ways of gaining insights into students' programming
behaviors and course achievement through IDE log data. To date,
researchers have focused mainly on describing general
programming behaviors (e.g. [2, 10]) and developing predictive
models (e.g. [9, 18]). In prior work (see [6]), we contributed to this
research by developing the Programming State Model (PSM),
which characterizes students' programming in terms one of eleven
programming states. For a dataset consisting of CS2 students, a
statistical model based on the normalized time spent in each PSM
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than ACM must be
honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions
from Permissions@acm.org.
SIGCSE '17, March 08-11, 2017, Seattle, WA, USA
© 2017 ACM. ISBN 978-1-4503-4698-6/17/03…$15.00
DOI: http://dx.doi.org/10.1145/3017680.3017785
Christopher D. Hundhausen
School of Electrical Engineering and Computer Science
Washington State University
Pullman, WA 99164 USA
hundhaus@wsu.edu
state was found to correlate moderately well with student
achievement. This paper continues our investigation into the PSM's
usefulness by considering the ways in which students transition
between PSM states. To this end, we pose the following research
question:
RQ: What is the relationship between a student's sequence of
transitions through the PSM and his or her classroom
performance?
To address this research question, we present two analyses of
students' programming data collected in a CS 2 course. We identify
the seventeen most common PSM transition sequences and relate
them to course performance. We find that, on average, lower
performing students compile their code less often and spend more
time editing syntactically incorrect code. Furthermore, when
examining transition sequences on a day-to-day basis for the four
days leading up to an assignment's due date, we find significant
differences in PSM transition sequences between A, B, and C-level
students. These findings provide further evidence that students of
differing levels of achievement do indeed engage in different
programming activities as they complete their programming
assignments, and that these differences can be automatically
detected.
2. BACKGROUND AND RELATED WORK
Given the perceived difficulty of the computing discipline and its
notorious reputation for low retention rates, computing education
researchers are understandably keen to better identify connections
between learner characteristics, learning outcomes, and learning
behaviors. Past researchers have considered a learner's background
(e.g. [5, 11]), prior knowledge (e.g [5, 13]), cognitive abilities (e.g.
[15]), time-on-task (e.g. [16]) and learning attitudes (e.g. [4]) as
possible correlates to learning outcomes and behaviors. While this
line of research shares our interest in identifying associations
between students and course outcomes, it differs from our work in
that it is typically based on periodic assessment, rather than on a
continuous stream of learning process data automatically collected
as students complete their coursework.
A more recent line of research has focused on leveraging a
continuous stream of data for the purpose of identifying positive
learning patterns [3, 17]. In computing education research, these
data have been used to describe compilation behavior (e.g. [2, 10])
and to identify behaviors associated with eventual success or failure
in a course (e.g. [1, 6, 9, 18]). Contrary to most of the work
chronicled by Ihantola et a. [8], the method presented in this paper
does not require an entire semester's worth of data in order to
achieve predictive accuracy. Indeed, the results presented in
Section 4 require only a single day's worth of data to achieve a
statistically significant level of predictive accuracy.
 Table 1: PSM States
State Description
Idle The student is not currently programming (default state
after 3 minutes of inactivity)
UU The student has yet to compile his or her project
YN Editing syntactically correct project, last debug
unsuccessful
YU Editing syntactically correct project, last debug
successful
NU Editing syntactically incorrect project, last debug
successful
NN Editing syntactically incorrect project, last debug
unsuccessful
DU Debugging project when last debug did not result in a
runtime exception
DN Debugging project when last debug resulted in a
runtime exception
RU Running a project outside of debug mode when last
debug attempt did not result in a runtime exception
RN Running a project outside of debug mode when last
debug attempt resulted in a runtime exception
R/ Running a project that does not compile; current run
represents last successful compilation
In Carter et al. [6], we introduced the Programming State Model.
Rather than describe students' behaviors in terms of keystrokes (e.g.
[12]) or compilation attempts (e.g. [9, 10, 18]), the PSM
categorizes students programming within a two-dimensional space
that captures both a student's current activity (e.g. editing,
debugging) and the correctness of the student's most recently
compiled programing solution. Table 1 lists all possible PSM
states. In prior work, we explored the effectiveness of a statistical
model derived from the normalized time spent in each PSM state.
Our model correlates moderately well with course achievement (r
= .67). However, rather than focusing on time spent in each PSM
state, this paper takes a different approach by examining the paths
that students take within the PSM. In particular, we investigate the
relationship between paths and course outcomes.
3. CONNECTING PATHS TO COURSE
PERFORMANCE
Our first analysis focuses on the transitions between PSM states.
To this end, we identify common PSM sequences, attempt to place
them temporally within the context of a given programming
assignment, and examine how they might relate to a student's
programming performance within a given course.
3.1 Participants
This study is based on student programming activity during the
spring 2014 offering of CptS 122, the CS2 course at Washington
State University. Taught in the C++ programming language, this
course enrolled 140 students, 129 of whom finished the course and
received a grade. Of those who received a grade, 108 students (100
men, 8 women) consented to release their data for this study.
Transition sequences were then compared to students' final grades,
which were based on the grades received on programming
assignments (35%), labs (10%), participation (5%), in-class quizzes
(10%), midterm exams (20%), and a final exam (20%).
Table 2: Activity Sequences
Chain Description
YU-RU-YU Running a semantically unknown program
outside of debug mode
YU-DU-YU Running a semantically unknown correct
program in debug mode
YN-RN-YN Running a semantically incorrect program
outside of debug mode
YN-DN-YN Running a semantically incorrect program in
debug mode
NU- R/-NU Attempting to execute a syntactically incorrect
program
NN- R/-NN Attempting to execute a syntactically incorrect
program
Table 3: Correctness Sequences
Chain Description
YU-NU Failed compilation attempt
YU-DU-YN A runtime exception in debug mode occurred
YN-DN-YU A runtime exception in debug mode did not occur
Data Collection
Data was collected through the OSBIDE social programming
environment (see [7]), a plug-in to the Visual Studio IDE. In total,
2,781,061 events over seven programming assignments were
analyzed for this study. In analyzing our dataset, we identified
21,215 unique PSM sequences generated by students as they
completed programming tasks.
3.2 Identifying Common PSM Transitions
We begin by identifying commonly-occurring chains of transitions
within the PSM. While the PSM contains ten states (see Table 1),
there are few edges that connect states, owing to the logical
constraints of the programming process. For example, it is
impossible for a student to perform runtime debugging on a
syntactically incorrect program. These constraints give rise to
several characteristic transition sequences, which we break into two
categories:
Activity Sequences are those that indicate that a student is
switching between editing and running their application, with no
change in the underlying correctness of their project. Table 2 lists
activity sequences.
Correctness Sequences are those that give rise to a change in either
the syntactic or semantic correctness of a student's solution. Table
3 lists the correctness sequences.
In addition to the activity and correctness sequences listed in Tables
2 and 3, we mined our dataset for commonly occurring sequences.
Excluding the previously identified sequences, which were found
to be the most common patterns, we identified an additional eight
sequences that occurred at least 1,000 times throughout the
semester. These sequences are listed in Table 4.
It is important to note that because sequences were identified using
post-hoc analysis, we are unable to assign a meaning to any specific
sequence. It is entirely possible, and indeed likely, that a single
sequence (e.g. Idle-YN-Idle) occurs in differing circumstances and
therefore is associated with multiple high-level problem solving
activities. We plan to address this limitation in future research.
 Table 4: Other Frequently Occurring Sequences 3.3 Associating Transitions with Performance
Chain Description In order to tie PSM transition sequences to learning outcomes, we
captured the number of times that a sequence occurred for students
Idle-YN- A programming session a) that contains only in each grade band (Table 5). An analysis of variance on the data
Idle editing activities and b) whose last debug attempt
detects significant differences in the frequencies of several of the
resulted in a runtime exception.
sequences experienced by students who achieved different final
Idle-YU- A programming session a) that contains only grades in the course. However, notice that the number of sequences
Idle editing activities and b) whose last compilation (Table 5, last row) also differs significantly between grade levels.
attempt was successful.
This indicates that any observed significant differences may be the
Idle-NU- A programming session a) that contains only result of merely spending more time programming, and not an
Idle editing activities and b) whose last compilation actual difference in programming behavior as indicated by a given
attempt was unsuccessful. programming sequence. To test for this possibility, we normalized
Idle-UU- A programming session a) that contains only our data set by the number of transitions generated. Performing a
Idle editing activities and b) has never been second ANOVA on the normalized data tells a much different story.
compiled. Table 6 lists the transitions that remain significant after the
NU-R/-YU Attempting to run a syntactically incorrect normalization process.
project, then making the project syntactically
correct. 4. EXPLORING TRANSITION
YN-RN-DN Attempting to run a project whose last debug SEQUENCES OVER TIME
attempt was unsuccessful, followed by a The results presented in the prior section indicate a weak
debugging session. relationship between the total number of transitions generated and
YU-RU- Attempting to run a project whose last debug course achievement. In this section, we explore whether differences
DU-YU attempt was successful, followed by a debugging might exist when we also factor in a sequence’s proximity to an
session. assignment's due date. For example, it seems plausible that F-level
DN-YU-DU Performing a debug session that did not result in students might encounter "troublesome" transition sequences closer
a runtime exception, followed by another debug to an assignment's due date, whereas A-level students might spend
session. the same time testing their program. To investigate this possibility,
we use the same data set as was used in the prior analysis to
examine the transition sequences generated on a particular day. As
Table 5: Frequency of Sequence by Final Course Grade assignment periods were of differing lengths, we recorded the
(* = sig at p < 0.05) frequency relative to each assignment's due date. In total, we
Sequence Count examined the 18 transition sequences identified in Section 3 for
F D C B A every student for the 21 days leading up to each of the seven
YU-RU-YU* 48 66 122 127 173 programming assignments.
YN-RN-YN* 20 41 69 59 108 For each transition sequence considered (see Tables 2-4), we
NU-R/-NU* 95 123 227 169 201 constructed 17 graphs that track the frequency of a given transition
NN-R/-NN* 4 9 15 10 17
chain as a programming assignment's date approaches. From these
YU-DU-YU* 19 60 51 60 64
YN-DN-YN* 19 37 65 60 68 graphs, we identified eight sequences whose graphs appear to be
YU-NU-YU 11 15 21 28 18 influenced by student achievement: YU-RU (Running a
YU-DU-YU-YN 1 2 3 3 3 semantically unknown program outside of debug mode), NU-R/
YN-DN-YU 5 8 20 10 13 (Attempting to execute a syntactically incorrect program), NN-R/
Idle-YN-Idle* 13 22 33 39 61 (Attempting to execute a syntactically incorrect program), YU-DU
Idle-YU-Idle* 32 51 72 86 110 (Running a semantically unknown correct program in debug
NU-R/-YU* 9 14 25 23 28 mode), NU-R/-YU (Attempting to run a syntactically incorrect
Idle-NU-Idle* 7 10 13 10 14
project, then making the project syntactically correct), Idle-NU (A
Idle-UU-Idle* 4 6 6 6 6
RN-DN-YN* 4 11 19 17 25 programming session on a syntactically incorrect project that
YU-RU-DU* 6 10 15 15 21 contains only editing activities), YN-DN-YU (A runtime exception
DN-YU-DU* 3 10 9 12 11 in debug mode did not occur), and YU-RU-DU (Attempting to run
YN-DN-YU- 1 2 4 2 4 a project whose last debug attempt was successful, followed by a
DU* debugging session). Furthermore, we noticed that the majority of
Number of 23 409 616 610 812 students' activity occurred during the four days leading up to an
Transitions* 1 assignment's due date. For the sake of brevity, we therefore limit
our statistical analysis to the four days leading up to each
Table 6: Significant Normalized Transitions by Final homework's due date.
Course Grade (p < 0.05). Subscript a Indicate Focusing on the data on the four days preceding an assignment due
Significantly Higher Rate than Subscript b Using Post-Hoc date, we next performed a three-way analysis of variance
Bonferroni Analysis (ANOVA), inputting letter grade, day, and transition as predictor
Sequence F D C B A variables and frequency as the outcome variable. Statistical
analysis reveals significant main effects on letter grade (F(4,199) =
Idle-YN 4.0% b 5.2% 4.9% 6.6% 7.1% a
9.23, p < 0.01, partial 𝜂2 =0.01), day (F(4,199) = 19.17, p < 0.01,
Idle-NU 5.1%a 2.5% 2.8% 2.1%b 1.8% b partial 𝜂2 =0.02), and transition (F(7,199) = 21.29, p < 0.04, partial
Idle-UU 3.7% a 1.7% 1.6% b 1.6% b 1.0% b 𝜂2 =0.04). In addition, statistical analysis detects a significant two-
way interaction between letter grade and transition ((F(28,199) =
16.85, p < 0.01, partial 𝜂2 = 0.12) and a statistically significant
interaction between all three predictor variables ((F(140,199) =
1.29, p = 0.01, partial 𝜂2 = 0.05).
In the following subsections, we consider the significant
differences in the three-way interaction by performing post hoc
pairwise analysis for the four days leading up to an assignment due
date.
4.1 Patterns of Behavior Four Days Before
Due Date
We first consider the interaction between letter grade and transition
sequences generated four days prior to the due date (see Figure 1).
Statistical analysis identifies the following significant differences
(p < 0.05) between the groups:
 YU-RU sequences were most common among A students.
 YU-DU sequences were most common among B students.
 Idle-NU sequences were most common among C students.
 YU-RU-DU sequences were most common among C students.
Recall that sequences containing the letters "R" and "D" involve
program execution. Therefore, these findings would seem to
indicate that four days before the due date, A-, B-, and C-level
students are in the testing phase of software development.
However, unlike A- and B-level students, we also see that C-level
students are spending more time working on an error-riddled
project, as indicated by the number of Idle-NU sequences
generated. Interestingly, we see that B-level students are apparently
using the debugger on a much more frequent basis than A-level
students. Investigating the cause of this difference would be a
worthwhile activity for future research.
4.2 Patterns of Behavior Three Days Before
Due Date
We next consider the interaction between letter grade and
transitions sequences generated three days prior to the due date (see
Figure 2). On this day, we begin to see increased activity from F
students, who generated the second most YU- RU sequences.
Statistical analysis finds the following differences (p < 0.05)
between the groups:
 For YU-RU, A students again generated more transitions than
all other grade levels.
 F students generated significantly more YU-RU transitions
than B and D students.
 For YU-DU, B students again generated more transitions than
all other grade levels.
 For both Idle-NU and YU-RU-DU, C students generated more
transitions than all other grade levels.
It would seem as though the patterns of sequence generation
identified in the prior section continue. Again, we see A-level
students adopting a strategy consisting mainly of YU-RU
sequences, B-level students with YU-DU, and C-level students
with Idle-NU and YU-RU-DU. Unlike the previous day, we see an
increase in YU-RU sequences generated by F-level students.
However, other sequence activity remains minimal.
4.3 Patterns of Behavior Two Days Before
Due Date
As illustrated in Figure 3, the behavioral trends of A-, B-, and C-
level students remained steady two days before assignment due
dates. That is, A-level students generated the most YU-RU
transitions, B-level students generated the most YU-DU transitions,
and C-level students generated the most Idle-NU and YU-RU-DU
transitions. However, the overall increase of activity for all students
makes it more difficult for statistical tests to detect significant (p <
0.05) differences between certain groups:
 YU-RU transitions continue to be significantly more likely to
be generated by A students.
 B students are more likely to generate YU-DU transitions
when compared to A, D, and F students.
 B students are more likely to generate NU-R/-YU transitions
when compared to A, C, and F students.
 C students are more likely to generate Idle-NU transitions
when compared to A, B, and F students.
 C students are more likely to generate YU-RU-DU transitions
when compared to A, B, and F students.
Again, we see well-defined patterns of behavior among A-, B-, and
C-level students. In contrast, while D- and F-level students'
transition frequencies increased, they lack a common pattern,
which prevents us from identifying transitions unique to their
problem-solving behaviors.
4.4 Patterns of Behavior on Day Before Due
Date
Once again, we witness the three peaks of activity characteristic of
A-, B-, and C-level students (see Figure 4). Statistical analysis
reveals the following significant (p < 0.05) findings:
 A and F students were more likely to generate YU-RU
transitions than B and C students.
 B students continued to produce more YU-DU transitions than
any other grade level.
 Additionally, B students produced more NU-R/-YU
transitions than A and C students.
 Likewise, C students continued to produce more Idle-NU
transitions than any other grade. D students produced more
than A students.
 Again, C students produced the most YU-RU-DU transitions.
While we continue to see the same pattern of sequence generation
from A-, B-, and C-level students, we also witness a slight
reduction in B-level students' generation of NU-R/-YU sequences.
Two days prior to the due date, B-level students generated an
average of 67 of these transitions whereas on the day before the due
date, B-level students generated 44. This might indicate that B-
level students have transitioned from working through compiler
and runtime issues, and are focusing more attention on runtime
behavior.
5. DISCUSSION
In Section 3, we explored the relationship between the total number
of PSM transitions and course outcomes. In particular, we
considered the relationship between specific sequences of
transitions and course outcomes. While the frequency of individual
sequences differed significantly between grade level, such
differences coincided with an overall increase in student effort.
This suggests that individual sequence differences may merely be
a proxy for overall effort. However, even when accounting for time
on task, statistical analysis revealed significant differences between
students for three sequences: Idle-YN, Idle-NU, and Idle-UU.
It is interesting to note that the three significant transitions represent
programming sessions focused on editing activities. Based on these
results, the key difference between the programming behaviors of
students of differing levels of course achievement can be
 Figure 1: Interaction between Letter Grade and Transition
Four Days Before Due Date
Figure 3: Interaction between Letter Grade and Transition
Two Days Before Due Date
characterized as follows: A- and B-level students tend to expend
more of their effort editing programs that have had recent runtime
exceptions, whereas F-level students tend to expend more of their
effort editing programs that either have syntax errors, or that have
never been compiled. As C- and D-level students' behaviors do not
differ significantly from either extreme (F or A/B), we might
conclude that C- and D-level students exhibit behavior somewhere
between the groups. In all cases, our analysis shows that editing a
project without periodically exploring its compilation and runtime
behavior appears to be a suboptimal solution strategy.
In Section 4, we examined the weak but significant interaction
between the eight most common PSM transitions for the four days
leading up to a programming assignment due date. For each day
observed, we see clear differences in behavior between A, B, and
C students: A-level students appear to favor a strategy that involves
transitioning between editing and running outside of debug mode;
B-level students favor an approach that involves transitioning
between (a) editing and running in debug mode and (b) editing a
Figure 2: Interaction between Letter Grade and Transition
Three Days Before Due Date
Figure 4: Interaction between Letter Grade and Transition
One Day before Due Date
syntactically incorrect project, attempting to run the project, and
then addressing the syntax error; and C-level students tend to
transition between (a) editing a syntactically incorrect project and
(b) editing a syntactically incorrect project, running the project
outside of debug mode, then immediately running the project in
debug mode. Other than the fact that D- and F-level students
generated significantly fewer transitions than the other groups, no
distinctive behavior for these groups could be identified. Given
this, it would appear that a simple time on task measure would be
sufficient to distinguish A/B/C students from D/F students.
It is also notable that frequent debugging, a behavior widely
considered to be a beneficial practice, is most commonly associated
with B-level students. Interestingly, the frequency of debugging
behavior for A-level students does not differ significantly from that
of C-, D-, and F-level students. Instead, A-level students often run
without debug mode, presumably making it more difficult to
diagnose issues that occur during runtime. In future work, we
would like to further investigate this finding.
Like A- and B-level students, we find that C-level students also
frequently run their code. However, in contrast to higher achievers,
we find that C-level students spend more time working on code that
does not compile. This might suggest that C-level students
encounter more compiler errors than A- or B-level students,
perhaps due to possessing less knowledge of programming
constructs. We leave this as an open question for future work.
Lastly, we underscore that one must exercise caution in interpreting
these results. First, it is possible that these results are specific to
the course considered in this paper and do not generalize across
populations. Second, it is probable that a given transition sequence
(e.g. YU-RU) represents several disparate problem-solving
activities. This observation provides motivation for performing
follow-up lab observations of students as they complete their
programming assignments. Such observations, common in Human
Activity Recognition (e.g. [14]), would allow us to ascribe
intentionality to the various transition sequences. We believe that
making such a connection will pave the way for a rich descriptive
analysis of students' programming processes—one that can gain
deeper insights into the differences identified in this study.
6. CONCLUSION
In this paper, we considered the relationship between trajectories
through the PSM and their achievement in a CS2 course. Statistical
analyses revealed significant associations between PSM sequences
and student achievement. The research presented in this paper
makes the following research contributions:
First, this research contributes a new method for detecting a
student's programming ability. This method, based on tracing PSM
programming sequences, can be used as a basis for dynamically
tailoring learning interventions that better serve students as they
learn to program.
Second, we provide further empirical evidence that students of
differing levels of achievement approach the process of solving
programming assignments differently. In future work, we plan to
qualitatively investigate how these different problem solving
strategies manifest themselves in students' code.
Lastly, while we were able to identify differences between students,
our research highlights a key deficiency of using automatically
generated log data--namely, that such a methodology is incapable
of connecting a student's underlying intent to his or her
programming processes. For this reason, future researchers would
do well to augment automatic data collection with alternate sources
of data that can get at learners' intent, such as lab observations
and/or retrospective interviews.
7. ACKNOWLEDGMENTS
This project is funded by the National Science Foundation under
grant no. IIS-1321045.
8. REFERENCES
[1] Ahadi, A. et al. 2015. Exploring Machine Learning
Methods to Automatically Identify Students in Need of
Assistance. Proceedings of the Eleventh Annual
International Conference on International Computing
Education Research (Omaha, NE, USA, 2015).
[2] Altadmri, A. and Brown, N.C.C. 2015. 37 Million
Compilations: Investigating Novice Programming
Mistakes in Large-Scale Student Data. Proceedings of the
46th ACM Technical Symposium on Computer Science
Education (Kansas City, MO, USA, 2015), 522–527.
[3] Baker, R.S.J. and Siemens, G. 2014. Educational data
mining and learning analytics. The Cambridge Handbook
of the Learning Sciences. Cambridge University Press.
253–274.
[4] Bergin, S. et al. 2005. Examining the role of self-regulated
learning on introductory programming performance. Proc.
2005 ACM International Computing Education Research
Workshop. ACM Press. 81–86.
[5] Bransford, J. et al. eds. 1999. How people learn: Brain,
mind, experience, and school. National Academy Press.
[6] Carter, A.S. et al. 2015. The Normalized Programming
State Model: Predicting Student Performance in
Computing Courses Based on Programming Behavior.
Proceedings of the Eleventh Annual International
Conference on International Computing Education
Research. ACM. 141–150.
[7] Carter, A.S. and Hundhausen, C.D. 2015. The Design of a
Programming Environment to Support Greater Social
Awareness and Participation in Early Computing Courses.
J. Comput. Sci. Coll. 31, 1 (Oct. 2015), 143–153.
[8] Ihantola, P. et al. 2015. Educational Data Mining and
Learning Analytics in Programming: Literature Review
and Case Studies. Proceedings of the 2015 ITiCSE on
Working Group Reports (New York, NY, USA, 2015), 41–
63.
[9] Jadud, M.C. 2006. Methods and Tools for Exploring
Novice Compilation Behaviour. Proceedings of the Second
International Workshop on Computing Education
Research. ACM. 73–84.
[10] Jadud, M.C. and Dorn, B. 2015. Aggregate Compilation
Behavior: Findings and Implications from 27,698 Users.
Proceedings of the Eleventh Annual International
Conference on International Computing Education
Research (Omaha, NE, USA, 2015).
[11] Jeske, D. et al. 2014. Learner characteristics predict
performance and confidence in e-Learning: An analysis of
user behavior and self-evaluation. Journal of Interactive
Learning Research. 25, 4 (2014), 509–529.
[12] Leinonen, J. et al. 2016. Automatic Inference of
Programming Performance and Experience from Typing
Patterns. Proceedings of the 47th ACM Technical
Symposium on Computing Science Education (Memphis,
TN, USA, 2016).
[13] Ma, W. et al. 2014. Intelligent tutoring systems and
learning outcomes: A meta-analytic survey. Journal of
Educational Psychology. 106, 2007 (2014), 901–918.
[14] Minor, B. et al. 2015. Data-Driven Activity Prediction:
Algorithms, Evaluation Methodology, and Applications.
Proceedings of the 21th ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining
(2015).
[15] Schunk, D.H. 2012. Learning theories: An educational
perspective. Merrill Prentice Hall.
[16] Slavin, R.E. 2011. Educational psychology: Theory and
practice. Pearson Education.
[17] U.S. Department of Education, Office of Educational
Technology 2012. Enhancing Teaching and Learning
through Educational Data Mining and Learning Analytics:
An Issue Brief.
[18] Watson, C. et al. 2013. Predicting Performance in an
Introductory Programming Course by Logging and
Analyzing Student Programming Behavior. Proceedings of
the 2013 IEEE 13th International Conference on Advanced
Learning Technologies (2013), 319–323.