INTELLIGENT TUTORIAL SYSTEM & ITS ENHANCEMENT USING EMOTIONAL FEEDBACK

Soumya Bose 11EC65R09, VIPES

SCOPE
Emergence of Intelligent Tutoring System (ITS) Central framework of ITS. Different Modules New Generation ITS (NGITS) Need of emotional feedback Facial Emotion Analysis Conclusions Future Work

EXISTING TUTORING SYSTEMS (CAI)

E-learning Web Based Tutorials using Audio/Video.

Advantages: Low Cost. Learn when you need. Not constrained by geographical location.
Disadvantages: Based on Simple Computer Aided Instructions No student teacher feedback. Lack of understanding. Appears to be boring.
3

INTELLIGENT TUTORIAL SYSTEM

Outgrowth of Computer Aided Instructions with added intelligence. System where teacher-student communication strengthens learning process. Tracks students performance. Adaptive system where next state of instruction is dependant upon students performance.

CENTRAL FRAMEWORK OF ITS

Traditionally ITS has the following components:

DOMAIN MODULE

PEDAGOGICAL MODULE

STUDENT MODULE

TASK ENVIRONMENT

Fig.1:

ITS Framework

DOMAIN MODULE
Knowledge database. Depends upon the domain in which the ITS is intended to instruct. Domain module is prepared with a view of cognitive psychology of human skill acquisition.

Knowledge DECLARATIVE or PROCEDURAL

6

Example

An ITS to tutor school mathematics.

Q. How to evaluate an expression with different arithmetic operands? Declarative:
Priority1. ( operation ) Priority2. / Priority3. x Priority4. + Priority5. -

Procedural:
Evaluation: (3-4) x 18 / 9 + 5 - 8 Step1. A= (3-4) = -1

Step2.
Step3. Step4. Step5.

B= 18 / 9 = 2
C= A x B = -1 x 2 = -2 D= C+5 = -2+5 = 3 Ans.= 3-8 = -5
7

Combination of both provides effective learning process

TASK ENVIRONMENT

Interface for student - teacher communication GUI may serve the purpose
Problem Statement Tutor Instructions

Tutor displays instructions. Student gives input through a text editor.

Output Window

Students Worksheet

A simulation response to the student forms a feedback to the student. Tutor Feedback helps student in reasoning.
Fig.2: Typical Task Environment

PEDAGOGICAL / TUTORING MODEL

Structuring the instructions. At curriculum level it is sequence of information. At problem solving level it can intervene to advise students.
Next instructions should be on the basis of present state (can be modeled as a tree data structure). Present state includes (Logical decision): Current stage of domain module. Knowledge state of student.

INSTRUCTION STRUCTURES
1. Primitive linear structure
C
1

C
2

C
3

C
4

C
n

C
3

C
4

2. Branched structure

C
1

C
2

C
3

C
7

C
n

C
6

C
8

3. Multilevel structure

C
1

C
2

C
5

C
n

C
3

C
4

C
6

C
7

C
8

C1
0

C
n

Cn is quantum of domain knowledge / information in the nth stage

C
9

C1
1

C1
2

C1
3

C1
4

C
n

10

STUDENT MODULE
Record of students knowledge state.
Student module is dynamic: (Modeling is complex).
e.g: Identifying Operands Identifying Operators Doing operations Evaluating Expressions Solving Equations

Knowledge state is changing

Statistical methods used for estimating students knowledge state (by psychophysicist Green and Swets, 1973)

Intelligence of Domain Expert module assess students performance

11

ITS DESIGN FLOW

TUTORING SYSTEM Modeler
Predicted and Preferred behavior

Relations and Student Prototypes

Update Model

Expert Simulator

Knowledge Base
Explanation Data

Student Model

Problem Solving Situation

Students Current state

Tutor
Problem Information
Data Request

Advice &Explanation

Problem

Problem Data

Student

12

A TYPICAL ITS
Problem Statement 1.Evaluate the expression: Z= (3-4) x 18 / 9 + 5 8 Tutor Instructions
st Hint: Calculate 1operationoperation Problempriority donest priority are YouSolved!nd operations Evaluate 2 prioritypriority 1st have the operation. Wrong! Now 1 nd independent an done. 2 Operation whichmodified priority dependant. the is alsostindependent Write downCalculate 1st expression Operation.now on the1modified operation Calculate operation expressionoperation. Calculate 1st operation of the first priority operation

Output Window 1. 2. 3. 4. 5. A = -3 B=2

Students Worksheet 5 - 8 = -3 18/9 = 2 3 - 4 = -1 -1 x 2 3 2 x2= -1 3 =1-2 -2 3 = -5

C= -1 Expression Wrong -2 6. -5

13

PRACTICALLY IMPLEMENTED SYSTEMS

1. The PUMP Algebra Tutor (PAT) (Anderson, Corbett, Koedinger and Pelletier, 1995): Used for tutoring introductory algebra in Pittsburg Schools.

Fig.3.: The PAT GUI (Courtesy: Ref.4)

Fig.4.: The SHERLOCK interface (Courtesy: Lawrence Elbaum Associates)

2. The SHERLOCK Project (Lesgold, Laioie, Bunzo and Eggan, 1992; Katz and Lesgold, 1993): It is a practice environment for electronics troubleshooting commissioned by Air-Force.

14

SHORTCOMING OF CURRENT SYSTEMS

Q. What is the value of force acting on a body of mass 3Kg and moving with a retardation of 4m/s2? Ans. : Ans. : -12N -12N Confident Not-Sure

Evaluation by Current systems (Based on knowledge state): In either of the above cases student will be assumed to have a knowledge of FORCE.
15

NEW GENERATION ITS

Aimed at developing more adaptive tutors Expert Module evaluates both: Knowledge state Emotional / Mental state Measuring mental state of learner by bio signals analysis: Facial expressions Signals from Brain Electro-dermal signals ECG signals, etc.

16

FACIAL EMOTION ANALYSIS

Challenges:

Involves lot of real time image processing. Time is the constraint. Should be processed in parallel with knowledge evaluation.

Hardware / Software co-design approach is adopted for fast processing.

17

BASIC STEPS
IMAGE GRABBER PRE PROCESSING

GENETIC SOBEL ALGORITHM / COARSE LAPLACIAN WAVELET REGION TRANSFORM EDGE SEGMENTATION

FEATURE EXTRACTION

EDGE DETECTION

INTERPRETATION

MATCHING
18

Object Segmentation and Labeling

Labeling : Smoothing small variations in intensity. Segmentation : Finding edges or sharp transitions.

Smoothening

with linear resistive network blurs edges of objects.

Resistive

Fuse networks are used to label and segmenting the image.

19

Resistive Fuse Network

The resistive fuse acts as a linear resistor for |Vdiff| < Voff

Acts as an open circuit for |Vdiff| > Voff

Fig.5.: Fuse Resistor

The change in voltage at each node can be calculated from Kirchoffs current law:

Vout2,2(t+1) Vout2,2(t) = v[ G(Vout i,j Vout2,2) + (Vin2,2- Vout2,2) ]

i,jN 2,2

Fig.6.: Segmentation Circuit

20

The equation

v[ G(Vout i,j Vout2,2) + (Vin2,2- Vout2,2)

i,jN 2,2

can be realized in FPGA:

Fig.7.: Raster Scan of the image (Courtesy: Ref.7)

e = sample pixel a i = neighbor pixel Smem = pixel data Vin is stored Dmem = Output data Vout is stored LUT1 performs evaluation LUT2 performs G (fuse value) per channel (RGB)
Fig.8.: FPGA implementation (Courtesy: Ref.7)

21

Gabor Wavelet Transform (GWT)

Wavelet function is multiplication of a harmonic function and gaussian function
determines frequency and the direction

GWT is performed over the segmented object for feature extraction

22
Fig.9.: GWTs of the face (Courtesy: Ref.7)

Matching & Interpretation

Edges of features (eye / lips / nose): Matching: Genetic Algorithm (GA) GA is an iterative process . In each step shapes are matched with known curves
Fig.10: Matching eye with an ellipse (Courtesy: Ref.8)

Sobel / Gaussian operator

Termination occurs when error is minimized.

Fig.11.: Matching lips with an irregular ellipse (Courtesy: Ref.8)

Final matched curve parameters (like major axis/ minor axis) are matched with known values to predict emotions: Happy 23 Sad Frustration, etc.

CONCLUSIONS
ITS has been proved efficient and stronger than simple CAI Involves student in sustained reasoning activity. Problem solving tutor helps conceptual understanding as well as solving real life problems related to a domain. High level GUI attracts students for learning The ability to read the mental state of the learner through facial emotional analysis: Increases Adaptability Repeated instructions can be delivered on the basis of mental satisfaction Helps student in sound understanding

However the accuracy of the emotional analysis can be improved significantly adding voice information
The main drawback is it is an one-to-one process
24

FUTURE DIRECTIONS
Different bio-signals processing: Assess mental state of the student more correctly. Electro dermal signal(GSC): Human skin is a weak conductor of electricity.

Fig.12: Galvanic Skin Response (Courtesy: Springer Images)

Brain Signals (EEG): Higher frequency beta waves (15-25Hz) and low theta waves implies seriousness. ECG can be analyzed to detect stress, low confidence of the student. Real time processing several bio-signals will make the design complex. But even if half the ability of real human tutor is realized the payoff to the society will be substantial.

25

REFERENCES
1. Abdolhossein Sarrafzadeh, Hamid Gholam Hosseini, Chao Fan, Scott P. Overmyer; Facial Expression Analysis for Estimating Learners Emotional State in Intelligent Tutoring Systems; Proceedings of the The 3rd IEEE International Conference on Advanced Learning Technologies (ICALT03); 2003 Morteza BIGLARI-ABHARI, Abbas BIGDELI; FPGA Implementation of Facial Expression Analysis For Intelligent Tutoring Systems; Proceedings of the II International Conference on Multimedia and Information & Communication Technologies in Education; ICTE 2003, Spain Sunandan Chakraborty, Devshri Roy, Anupam Basu; Development of Knowledge Based Intelligent Tutoring System; Indian Institute of Technology, Kharagpur, India; 2001 M.Helander, T. K. Landauer, P. Prabhu (Eds), Elsevier Science B. V.; Intelligent Tutoring Systems, Handbook of Human-Computer Interaction, Second Edition; 1997 Arjen Hoekstra and Joris Janssen; Linking Bio-signals to Transfer of Knowledge Towards Mind-reading ECAs; Faculty of Electrical Engineering, Mathematics and Computer Science University of Twente, The Netherlands Teppei NAKANO, Hiroshi ANDO, Hideaki ISHIZU, Takashi MORIE, Atsushi IWATA; Coarse Image Region Segmentation Using Resistive-fuse Networks Implemented in FPGA; Graduate School of Life Science and Systems Engineering,Kyushu Institute of Technology; Graduate School of Advanced Sciences of Matter, Hiroshima University; Hiroshima Prefecture Industrial Research Institute T. Nakano, T. Morie and A. Iwata; A Face/Object Recognition System Using FPGA Implementation of Coarse Region Segmentation; SICE Annual Conference in Fukui, Fukui University, Japan; 2003 M. Karthigayan, M. Rizon, R. Nagarajan and Sazali Yaacob; Genetic Algorithm and Neural 26 Network for Face Emotion Recognition; School of Mechatronics Engineering, Universiti Malaysia Perlis (UNIMAP); 2006

2.

3. 4. 5.

6.

7.

8.