An Affective and Cognitive Tutoring System for Learning Programming Maria Lucia Barrén-Estrada’, Ramén Zstarain-Cabada'™, Francisco Gonzilez. Hemindez', Raiil Oramas Bustillos”, and Carlos A. Reyes-Garcia® * Instuto Teenoogico de Culiacin, Juande Dios Baz sf, eo. Guadalupe, Culiaca, Sinaloa $0220, Mexico (barron,rzatarain)@: teutiacan. edu. mx * Universidad de Oecidente, Cancera a Cliaancte, km, 1.5, Callaeén Sinaloa 80054, Mexico * Instituto Nacional de Astofisica, Optica Y Blecusnica (INAOF), Luis Envigue Eno no. 1, Sia. Ma. Tonal, Pucbla 72840, Mexico karganxiginaoep. mx Abstract. In this paper, we present a mulplatform and Intelligent Tutoring System for learning Java (lava Sensei). The learning system combines state-of-the-art action selection, motivation through emotions, a moder rec- ‘ommendation mechanism, and multimodal instructional and selection learning. Tava Sensei architecture works witha collection of modules and processes, each, ‘with its own effective representations and algorithms. The leaning system was implemented under different learning methodologies like problem-solving for the pedagogical module, knowledge space for the expert module, and overlays for the student module. One of the main contributions of this work was the integration of cognitive and allctve information in a behavioral graph which is used by a learning companion to show emotions and empathy to the student, Java Sensei was tested with diferent groups of university students with which ‘we obained postive results. In addition to providing a detailed description of, the implementation and evaluation of Java Sensei, we also provide some pro- posals of furare work in our intelligent tutoring systems Keywords: Intelligent tutoring systems + Affective computing * Artificial neural networks * Fuzzy logic - Mobile leaming Knowledge spaces 1 Introduction Learning a programming language is the first big challenge for a student of computer science or a related field [1]. Some of the issues that contribute to this obstacle re: the teaching methods employed by the instructor, the difficult nature of computer pro- gramming, the study methods, abilities and attitudes employed by the student, the nature of the art of programming, the lack of prior knowledge of novice students, and the psychological influence that the student suffers from society [2-5 ‘Many researchers have proposed and developed methodologies and tools to help students leam programming. For example the ACM has SIGCSE (Special Interest © Springer International Publishing Switzerand 2015 (. Pichardo Lagunas et al. (Eds): MICAL 2015, Part Il, LNAI 9414, pp. 171-182, 2015. DOK 10.1007978-3-319-27101-9_12172 ML. Barrén-Estrada etal Group on Computer Science Education) where issues related to the development, implementation and/or evaluation of programs and curricula are discussed [6]. Today, in the field of Artificial Intelligence, affect recognition is an emerging research area of significant theoretical and applied awareness to a number of academic ficlds including machine Ieaming, computational linguistics, computer vision and speech and language, neuroscience, educational psychology, and pedagogy [7]. ‘The link between affect computing and student learning has been a topic of increasing attention in recent years. Due to the importance of this connection of emotions and learning, one of the current aims of many intelligent tutoring systems is, (o recognize and manage student emotional states by means of affective mediations. Intelligent Tutoring Systems (ITS) try to simulate a human tutor to provide personal- ized instruction taking into account not only cognitive aspects of students [8] but also affective elements [9]. ‘This paper presents the implementation of an Intelligent Learning Environment (ILE) for Java programming. This system works in a flexible and interactive multi- platform environment that considers both cognitive and affective states of students. ‘The paper is organized as follows: Sect. 2 describes related work of the main topics. Section 3 shows and explains the architecture of the ILE Java Sensei. Affect recog- nition and feedback is illustrated in Sect. 4. Section 5 presents the experiments and results produced with the system. Conclusions and future work are discussed in Sect. 6 2 Related Work Research on intelligent tutoring systems for programming language has focused on different programming languages like Java, C-C#, or PHP, Wiggins ct al. [10] implemented The JavaTutor System, an affective tutoring system that uses natural language dialogues for human-to-human or human-to-computer dialogues using ‘machine learning techniques. The system takes into account cognitive aspects, where it applies the ACT-R theory of cognition for knowledge representation. Affect focused on nonverbal behavior obtained through bodily expressions and detection is performed by different hardware tools. CSTutor is a tool [11] to help students learn programming in CA, The tool incorporates anchored instruction for the domain representation of the programming language and game theory, instead of affect recognition, for motivation improvement. The PHP ITS was developed to teach students to program in PHP scripting language [12]. The tutor makes an analysis written by the student and translates to the equivalent of an abstract syntax tree (AST) code. Then, the AST is used to create events or trigger actions. The facts that exist after all AST nodes are processed ate called the final state, To check the correctness of a program, the tutor analyzes whether all predicates in the goal for a particular task are present in the final state. Through this method, the student writes the solution of the task defined in its own, way, as long as the final state of hislher program meets the specified target. Tutor (13) is an intelligent learning environment (ILE) where students perform exercises programming in C. The idea of this system is to facilitate the effectiveness of Teaming when students navigate through complex problems and thereby provide an advanced education system to enhance the learning process, This ILE allows teachers‘An Aifective and Cognitive Tutoring System 173 to create new programming exercises and manage them from the system interface. The domain in the system is adjustable for each solution of each exercise, From the per- spective of a student the CTutor interface is a tool with controls to represent the solution of the exercises. The tool uses a constructivist pedagogical orientation where the authors give special weight to feedback as the main source of information; moreover, its domain is formed based on a constraint-based model. On the other hand, CTutor does not support affect recognition, 3 Java Sensei Architecture In this section we present the main architecture of Java Sensei: the affective environ- ‘ment for Jearning Java programming. Figure 1 shows the different layers and com- ponents of our architecture. Sensei Java architecture is designed in seven layers with a client-server structure with a fat client. The aim of developing a thick client is to reduce the processing load on the server. Next, we explain the function of each layer. ‘© Web Layer: The Web Layer is the layer of the client, and is made up of modules that need to run in the web browser when the user interacts with the system. This layer contains the following components’ — Web Interface: This component represents the web user interface, so. this ‘component is responsible for interaction with the customer, the use of external libraries and centralizing interactions with other layers, mainly with the Service Layer. Example-Tracing Engine: This element is the semantic and visual interpretation of all courses in the system, Within this component, the following tasks are performed: creation of tracing trees, extraction of tree information, and inter- action with the Web Interface to build the GUI. In addition, the events generated by the example-tracing logic and notification of state changes are managed, Pedagogical Agent: This component creates the visual representation of a learning companion and it is based on the pedagogical information obtained from the server layer. The actions performed are: extraction of pedagogical data for the visual representation of the agent and extraction of information of affective information from the student using the wee created by Engine Example-Tracing. Camera in Background: This component manages the capture of the array of photographs in the background during exercise sessions in the system. It is responsible for starting and stopping the photography sessions, clear the ‘memory of « photograph queue, and manage the time between each capture; also it is responsible for converting the array of photographs into an understandable format for the Server Layer. © Service Layer: Il is the layer where the components related to the handling of ‘external requests to the server are Iocated, In this layer the components must be174 ML. Barrén-Bstrada etal Fig. 1. Java Sensei Architecture accessible from the Web and use the HTTP communication protocol. This layer holds the next components: = Restful Services: This component is responsible for receiving incoming web requests from the Web Layer. lis role is to accept requests and ensure its integrity and completeness. It also gives them a format so they can be worked on by the components comprising the Server Layer. ‘© Server Layer: The Server Layer isthe layer that contains the modules of the server, which is composed of modules that need a strong interaction with the Domain Layer and Student Layer; also it makes decisions using a high burden of processing and‘An Affective and Cognitive Tutoring System 175 memory resources. Within this layer there are some key components for making instructional decisions. = Tutoring Module: 1s responsible for taking appropriate cognitive and alfective educational decisions for the student as: assessing the quality of student response, assessing the overall ability of the student and use the Affective Framework component for obtaining affective information. = Adaptation Module: It uses the perceptions and preferences of the student in relation to the exercises and help resources fo build a recommendation system. = Affective Framework: This component represents the affective student assess- ment, I¢uses cognitive and alfective information from the students provided by the Tutoring Model. It uses a back propagation neural network to recognize emotions through the face and uses a fuzzy logic system for affective peda- ‘eogical strategy, ‘© Domain Layer: This layer contains one component: The Content Manager, which is responsible for interacting with the Server Layer and Data Management Layer. The ‘Content Manager represents the tutor domain or knowledge of the Java language. ‘This knowledge is represented using the theory of knowledge spaces which is ‘encoded using ISON files. ‘© Student Layer: It represents the student's knowledge compared to the domain model. This layer contains one component: The Student Profile Management. The tasks of the component are: calculate the overall ability of the student (the total number of correct exercises) and quality of the current response (the number of ‘errors thatthe student makes in an exercise); verify changes in the domain model to upgrade the overlay student model; and store student preferences regarding exer- cises and resources. The component also interacts with the Data Management Layer to store/retrieve student data in JSON format. © Data Management Layer: This layer contains two components: Entities and Plain Objects. The component Entities are data structures that encapsulate domain and stu- {dent information that are used in the Data, Domain, and Student Layers respectively. ‘The component Plain Objects contains information in plan text in JSON format. ‘© Data layer: This layer manages the data for the Domain and Student Modules represented in ISON and MongoDB formats 4 Affective Recognition and Feedback Figure 2 shows the steps followed in the recognition and affective feedback from the system. In the first step the ILE Java Sensei receives the data and the student's col- lection of images from the browser which ate later sent to the ILE. In the second step the ILE starts a cycle with several student faces being sent to the neural network, and decides, at the end of the eyele, the current emotion. This current emotion is obtained based on the most frequent student emotion. In the thd step the ILE sends the updated student model to the fuzzy system which performs the fuzzy rules to output the tutor actions. In the last step, the tutor actions are sent to the browser to build the actions of the pedagogical agent176 ML. Barrén-Bstrada etal Pedagogical Agent: The pedagogical agent is responsible for transmitting messages that a human tutor would perform, For this, it has been provided with both facial expressions and dialogues that can communicate to the student ‘These expressions are based on facial expressions investigated by Ekman [14]. Positions of the eyebrow, mouth, and eye opening are taken into account, These positions are entered as parameters in the FaceGen Modeller [15] software. In Fig. 3 we observe the four facial expressions made by the pedagogical agent of Java Sensei These emotions were based on the pedagogical agent working with AutoTutor [16] Fuzzy System and Neural Network: ‘The fuzzy logic system seeks to represent the ‘way a human tutor acts when he/she faces various scenatios related with cognitive and cmaotional situations of a student. These situations have to do with the predefined actions. The fuzzy system uses the emotional values obtained by the neural network which recognizes the affective state of the student when he/she works with the ILE Java Sensei. The fuzzy logic system works with four input fuzzy variables and three output ones, A total of 144 fuzzy rules were constructed, To define these rules we use Fuzzy Control Language, a standardized language for ereation of fuzzy rules. The imple- ‘mentation was done with jFuzeyLogic [17] ‘The emotion recognition system was built in three steps: the first one was an implementation to extract features from face images in a corpus used to train the neural network. The second step was the implementation of the neural network. We used the Java-based algorithms implemented in NeuroPH [18] to implement classification by using a neural network (feed-forward method). ‘The third step integrated feature extraction and emotion recognition with the fuzzy system. For training and testing the neural network, we used the corpus RAFD (Radboud Faces Database), which is a database with 8040 different facial expressions that contain a set of 67 models including men and women. In the training process we obtained a successful rate of 86 % which we consider is good enough for our system. Once the emotional state is extracted from the student, the emotional state is sent to the fuzzy system. Adaptation Module: The ILE must be able to adapt to the student, The ILE collects the rating produced by the student concerning the exercises and resources that the student visits. The rating of one exercise is measured using Likert scale [19] where one indicates total disagreement with the exercise and five indicates total agreement. At the beginning all the exercises are rated with a value of two. We used a recommendation system ereated with Apache Mahout [20]. This tool helps to find the next problem the stuclent must solve. The system implements the K-Nearest Neighbor (k-NN) approach and the Pearson Correlation. As more users are included, the rating system generates, the best recommendations and thereby it gets more adapted to the changes. Pedagogical Module: The system uses the pedagogical model known as “problem solving” [21] where the student learns as he solves problems with a certain structure. ‘The system uses three types of strategies for solving problems. Type 1 is used to evaluate theoretical concepts with “true-false” exercises. Type 2 questions present a complete program or piece of code and ask the student to determine the output of the‘An Aifective and Cognitive Tutoring System 177 Student Model : oad Fig. 2. Affect recognition and response Enchanted Skeptical Surprised Neutral Fig. 3. Pedagogical agent code. Type 3 is a combination of the two previous types and allows creating more complex exercises involving “n” number of steps to reach the solution Expert Module: This model contains the expert knowledge that the student wants to learn and practice. The system represents the knowledge using JSON files (a variation of XML files). Knowledge representation uses the knowledge space theory [22]. The expert model represents six basic skills that students must master to be modeled by a graph representing the Knowledge Space. These skills are: Introduction to Java, Variables and calculations, selection, iteration, methods, and arrays178 ML. Barrén-Estrada etal Example-Tracing Engine: An ILE combines hypermedia techniques with other methods used by Intelligent Tutoring Systems (ITS). We applied Example-Tracing [23, 24] in order to implement the ITS in Java Sensei. Example-Tracing ITS has the advantage of providing step-by-step guiding to the students and providing with mul- tiple strategies to the problem solution, including optimal or sub-optimal solution oF riss-conception management, Example-Tracing exercises can be easly adapted with the pedagogical model implemented with “problem solving” and they are represented by a graph that is traversed by the student (Fig. 4). One of the main contributions of our work is the integration of cognitive and affective information in a behavior graph which is used by the pedagogical agent to show emotion and empathy to the student. Next, we describe the algorithm to implement the affective-example-tracing (a) The ILE loads the JSON file with the corresponding exercise (b) From the JSON file, a behavior graph is created and an initial step is executed. (6) As students traverse the graph (by clicking with the mouse), different options can be performed and different paths are traversed. The options are: i. Initial step (IS) fi, Exvor step (ES) iii. Optimal step (OS) iv. Sub-optimal step (SOS) v. Final Optimal step (FOS) vi. Final Sub-optimal step (FSS) (a) Every step triggers two events i. Actual emotion evaluation ii, ‘Tutor action evaluation, Every time a student chooses some of the options in the graph, the engine calls the fuzzy logic system to get the tutor action (Feedback, intervention, ete.) and to get the emotion information so the pedagogical agent can build the graph, Figure 4 illustrates aan example. At the left we can observe how the student solved the problem in an ‘optimal way. At the right the student made a mistake, 5 Experiments and Results In order to test the ILE, we made different experiments with different kind of students of the Instituto Tecnolégico de Culiacén. The system was first rested with graduate students in our research lab. This test has the aim to test the functionality of the ILE. Next, we show (Fig. 5) a small session of a student browsing in the ILE. To access the ILE, the student must use a browser with a PC, laptop, or mobile device (e.g. smartphone), and a Facebook account allowing the use of the Web cam. The left part of the example shows a problem with the student making a mistake, atthe end of the code. In the right part of the problem the student finish successfully the exercise.‘An Affective and Cognitive Tutoring System 179 ave Optimal step Erroneous Step Fig. 4 Affective examplesrscing An errors shown n the exercise sucess inthe solution 1! Fig. S. Affective examplesracing ‘The following evaluation was applied to Computer Systems students (Engineering) taking currently a Java course, In this case we consider two aspects: the software func- tionality and the software effectiveness. With respect to software functionality we compared! Java Sensei against three similar ILEs: JavaTutor System [25], JITS [26] y {Latte [27]. We considered 9 features to evaluate [6] and Liker scale from 1 to 5, where 1 represents an incomplete feature and 5 represents a complete feature the ILE accomplish. ‘Table | shows the results obtained from the software functionality evaluation, ‘The results show that Java Sensei and JavaTutor System are the ILEs that get better results in the evaluation, The evaluated systems meet almost every feature of an ILEJITS system except for three features (Mixed Initiative) and 6 (self-improvement). Feature 7 (Affect recognition and Handling) which is a recent area causing that few systems incorporate this feature. The same applies to features 8 and 9. It is expected that modern ITS migrate to the Web and Mobile platforms, For evaluation of the effectiveness of the system, we conducted a test with a group ‘of 20 students in the subject Object Oriented Programming (OOP), looking to measure the learning outcome of the ILE JavaSensei on the topic of inheritance. The selection of students was not performed randomly, but by availability of participants. To access the180 ML. Barton-Estrada eta ‘Table 1. Comparisons results in software functionality a Feature JavaSensel | JavaTutor stem [JTS [Late Generativity 3 5 44 ‘Suident and expert modeling | 5 5 ss Mixed initiative 2 2 tit Interactive learning 4 5 as Instructional modeling 5 5 ss Selfimproving I T rit fect recognition and Randling | 5 tit Web platform | 5 5 rit Mobile platform 5 1 tit Toul a 3a 23 (24 ILE, students used a web browser with support for HTMLS and front camera in desktops, laptops, and smartphone, entering the system from their Facebook account (Gee left part of Fig. 6). Fig. 6. Students using Java Sensei ig. 7. Pretest and postest results‘An Aifective and Cognitive Tutoring System 181 ‘The evaluation design methodology consisted of measuring the knowledge of two groups of student: one experimental group who learns by using the ILE and one control, group who leams by using traditional teaching. The time given to the pretest was 15 min (see right part of Fig. 6), the time for the ILE session was 70 min, and the time for the posttest was 20 min, Figure 7 shows the evaluation of both tests, The results in the postiest indicate that the experimental group obtained better scores (in most cases) than the control group. 6 Conclusions and Future Work Tne ILE has been implemented with open source code in programming languages and software libraries like Java, Javascript, MongoDB, JSON, jFuzzyLogic, Apache Mahout, Weka, jQuery, jQueryMobile among, others, and can be accessed ferent platforms (operating systems, web browser, computers, etc.). Right now, of programming Java professors is applying more tests to the ILE with students from ur university (Instituto Teenolégico de Culiacén). For future work, we want to include zgamification techniques in order to work more the motivational part of the system. We also want to work with emotions that affect the process of learning (engagement, boring, frustration, etc). Last we are developing a new version oriented to more collaboration work among students and authoring new exercises by instructors. Funding. The work described in this paper is fully supported by a grant from the DGEST (ireceién General de Educaci6n Superior Teenoldgica) in Mexico under the program “Projects, of Scientific Research and ‘Technological Innovation”. References 1. Rodrigo, MMT, Baker, RSJ. Jadud, M.C., Amara, A.CM,, Dy, "T, Espejo-Lahoz, MB. V., Lim, S.AL., Pascua, SAMS., Sugay, 1.0., Tabanao, E'S. Affective and behavioral predictors of novice programmer achievement. SIGCSE Bull. 41, 156-160 (2009) 2. Mauthiasdétir, A: How to teach programming languages to novice students? Lecturing or pot. In: International Conference on Computer Systems and Technologies-CompSysTech (2008) 4. Jenkins, Ts On the dlifcuty of leering to program. In; Proceedings of the 3rd Annual ‘Conference ofthe LTSN Centre for Information and Computer Sciences (2002) 4. Gores, A., Mendes A.J: Learning t program-ificulties and soluins, In: Intemational Conference on Engineering Eestion-l 'S. Berard, M., Bich, E.: Enhancing the metacognitive skill of novice programmers through collaborative learning. Metacogaition: Fundaaments, Applications, and Trends, pp. 277-298, Springer, Berlin (2015) 6, SIGCSE 2015 [cited 2015}. hup:/mww.sigese.org! 7. Picard, Ri: Affective computing: from laughter to IEEE. 11-17 @010) §. Woolf, B.P: Building Intllgent Interactive Tutors. Morgan Kaufmann Publishers, Amherst (2009) ‘Teams, Affect. Comput. 1182 MLL. Barrén-Estrada etal 9, Haley, IM. Lajoie, SP., Frasson, C., Hall, N.C. An integrated emotion-aware framework {or intelligent tuoring systems. In: Conati,C., Heffernan, N., Mitrovie, A.. Venleja, M. {eds,) ATED 2015. LNCS, vol. 9112, pp. 616-619. Springer, Heidelberg (2015) 10. Wiggins, LB., Boyer, K-E, Baikadi, A, Bren-Can, A., Grafsgaard, JP, He, ELY., Wiebe, [Nz JavaTutor: an intelligent tutoring system that adapts to cognitive and lective states uring computer programming. In; Paper Presented at the Proceedings of the 46th ACM ‘Technical Symposium on Computer Science Education 2015) 11. Budi, HL Jim, Rs Incorporating anchored learing in a C# intelligent tutoring system. In: Paper presented at the Consortia Proceedings of the 2Ist Intemational Conference on ‘Computers in Education, Indonesia: Asia-Pacific Society for Computes in Education 2013) 12, Weragama, D., Reye, J: Designing the knowledge hase for a PHP ator. In: Cert, S.A. Clancey. W.., Papadourakis, G., Panourpia, K. (eds) TTS 2012. LNCS, vol. 7315, 1p. 628-629, Springer, Heidelberg (2012) 13. Kose, U., Deperioglu, O.: Intelligent learning environments within blended learning for ensuring effective C programming course. Int. J. Arti Intell. App. 3(1), 20 (2012) 14, Ekman, P.: Facial expressions, Handb. Cognit. Emot, 16, 301-320 (1999) 15. Todorov, A., Oosterhof, N.N.: Modeling social perception of faces [socal sciences), IEEE. Sig. Process. Mag. 282), 117-122 2011) 16. D'Mello, ., etl: AutoTutor detects and responds to learners affective and cognitive states. In: Workshop on Emotional and Cognitive Issues at the Intemational Conference on Intelligent Tutoring Systems (2008) 17. Cingolani, P., Alealé-Fez, J: jFuzzyLogie: a java brary to design fuzzy logic controllers according to the standard fr fuzzy contcol programming. Int. J. Comput, Intell. Syst. 6 (up), 61-15 2013) 18. Sevarae, Z: Neuroph-Java neural network framework. Accessed July 2015 19, Likert, RA method of constsctng an atude scale, Sealing: a Sourcebook for Behavioral ‘Scientists, pp. 233-243, Aldine, Chicago (2012) 20. Anil, R, Dunning, 'T, Friedman, B.: Mahout in Action, pp. 145-183. Manning, Shelter Istand (2011) 21. Kumar, R, etal. Comparison of algorithms for automs ‘models. In: Educational Data Mining (2014) 22, Doignon, 1P., Falmagne, J..: Knowledge Spaces. Springer, Belin (1999) 23. Aleven, V., et al: A new paradigm for intelligent turoring systems: example-tracing tutors. Int J. Amif, Intell Educ. 19), 105-154 (2009) 24, Aleven, V., MeLaren, BM., Sewall, I, Koedinger, K.R.: The cognitive tutor authoring tals, (CTAT: preliminary evaluation of efficiency guns. In: Tkeda, M, Ashley, K.D., Chan, T-W. {eds,) ITS 2006. LNCS, vol. 4083, pp. 61-70. Springer, Heidelberg (2006) 25. Wallis, M. JavaTutor-a remotely collaborative, realtime distributed inteligent catoring system for introductory Java computer programming qualitative analysis (2011) Sykes, ERR, Franck, F. An intelligent tutoring system prototype for leaming to program Java TM, IEEE Xplore. In: Proceedings of the 3rd IEEE Intemational Conference on ‘Advanced Learning Technologies (2003) 27. Holland, J, Mitrovi, A., Manin, Bx LATTE: a constrant-hased ‘Tutor for Java (2009) cally building exampletracing tutor 26,