Scribd Logo
Upload

Welcome to Scribd!

  • Good5 Us Patent PDF

    Uploaded by

    Shailesh Patil
    8 views19 pages

    Original Title

    good5 us patent.pdf

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document

    Copyright:

    © All Rights Reserved
    Download as PDF or read online from Scribd
    Flag for inappropriate content
    8 views19 pages

    Good5 Us Patent PDF

    Uploaded by

    Shailesh Patil

    Copyright:

    © All Rights Reserved
    Download as PDF or read online from Scribd
    Flag for inappropriate content
    of 19
    cu») United States US 202000462771 2) Patent Application Publication (0) Pub. No.: US 2020/0046277 Al Dolsma et al. (43) Pub. Date: Feb. 13, 2020 (54) INTERACTIVE AND ADAPTIVE LEARNING AND NEUROCOGNITIVE DISORDER DIAGNOSIS SYSTEMS USING FACE TRACKING AND EMOTION DETECTION WITH ASSOCIATED METHODS. (71) Applicant: Yuen Lee Viola LAM, Hong Kong (aK) (72) Inventors: Johan Matthijs Dotsma, Hong Kong (HK); Yuen Lee Viola Lam, Hong Kong (HK) ey @y (86) Appl.Nox 16(48s,702 PCT Filed: Jul §, 2017 PCT No, $371 (XD, (2) Date: PCTNB2017/057631, Aug. 14, 2019 Related U.S. Application Data Provisional application No. 62/488,684, fled on Feb 14, 2017, provisional application No. 62/520,$42, filed on Jun, 15, 2017. (60 SC = Publication Classification (1) Inc. AGB S16 (2006.01) G06K 9700 (2006.01), G09B S700 (2006.01), G01. 25/63 (2006.01), AGB S205 (2006.01), Gr01.23/60 (2006.01), a AGIB 5/165 (2013.01); GO6K 900315 (GOi3 01): AerB 5/163 (2017.08): G10L 2860 (2013.01): G10L 263 2013.01): A618 $7168 (201301), 4618 S205 (2013.01); G09B S700 (201301), 6 ABSTRACT A system for delivering leaming programmes comprising ‘optical sensors for capturing a subject's facial expression, eye movements, pointof-gaze, and head pose during a Teaming session; a data repository comprising task data entities: a module for estimating the subject’ affective and cognitive siates using the captured sensory datas and. & ‘meddle fr selecting a task data entity for presentment w the subject alter each completion ofa task data entity based on 4 probability ofthe subject's understanding ofthe associated knowledge; wherein the probability ofthe subjet’s under- standing is computed using the subjects estimated affective cognitive states. Patent Application Publication Feb. 13,2020 Sheet 1 of 5 US 2020/0046277 AL US 2020/0046277 AI Feb. 13, 2020 Sheet 2 of 5 Patent Application Publication sme ead ne t / Loz Oz wD Patent Application Publication Feb. 13, 2020 Sheet 3 of 5 US 2020/0046277 AI Sogert e» “TabiewDesitop App {Communication Module User Interface) Gurricalum Gioud Service {(eteractive Learning and Training System) 308 316 test oatdomiet 318 anaoire FIG. 3 US 2020/0046277 AI Feb. 13, 2020 Sheet 4 of 5 Patent Application Publication doo} yoe9F 095 J>POHN -~ 00} 385) W124 sy v 7 / f ror r\ / Sov so | zo Lov aiep se Feb. 13,2020 Sheet Sof 5 US 2020/0046277 AI Patent Application Publication sola (a BEBE 1h WAS Dew abpeyA jeuewy eine FOS 981 205 levsiew 2np99 70S 981 20S jeusiew 91"9] 70S 4821 208 leusiew ana FOS 9821 205 [weed < 1 wis om obpemOnE] oBpapsouy wrewog 005 anew wonseno 505] Ipis pue ebpeynouy £05 wdeou09 10g iis pue @peymouy £05 ydaou0g Los 2214 aBpaprouy) pue ebpeyrouy £05 ydeouog 105 pis. IS pue abpayrouy E05 yeoueg 05 US 2020/0046277 AI INTERACTIVE AND ADAPTIVE LEARNING ‘AND NEUROCOGNITIVE DISORDEI DIAGNOSIS SYSTEMS USING FACE TRACKING AND EMOTION DETECTION WITH ASSOCIATED METHODS ‘CROSS-REFERENCES TO RELATED "APPLICATIONS, 10001} This application claims priority to U.S. patent pplication Ser. No. 151647,272 fled Jul. 12, 2017, US. Patent Application No. 624458.654 filled Feb. 14,2017, and US. Patent Application No. 62/520,542 fled Jun. 15, 3017: the disclosures of which are incorporated by reference thie entirety FIELD OF THE INVENTION [0002] The present invention relates wenerlly to methods and systems for providing and delivery of educational programmes and traning, including corporate taining, aca- ‘demic tutoring, in-class and ott-of-class learnings. Partict larly, the present invention relates to the customization of tests and assessment of leaming progress through the use of ‘emotion detection and analysis. The present invention also relates to methods and systems for providing and delivery of medical services and elderly cre, particularly inthe area of neurocognitive disorder diagnosis BACKGROUND OF THE INVENTION 10003] In the current education system, especially ia South-Fast Asia, pressure to excel in schools keeps mount- ing. In a resulorented based society, the student needs 10 achieve high grades to have a decent opportunity to enter prestigious academia and obtain advance degrees. The School system continues to rely heavily on class. based Fecturing combined with paper based examinations. Hence, there is limited support for personalized learning. Furher- more, the knowledge of each student is only evaluated ‘during examinations held afew times per year. 10008} These shortcomings have le to the rise of tutoring ‘centers where there are more personal atlentions and die Jogues between the teachers/tutors and students. Any deviae tions from the learning path, or knowledge gaps can then be remedied directly as such. However, in the tutoring industy. ‘good quality teacherstulors ure in searce supply and teacher! tutor teaining is offen done informally and easually on the Job, Pressure on fees has frher increased pressure on teacherstutors 10 take up more and more related adainis- trative tasks such as course material preparation, class Scheduling, and other logistics, reducing effecting teaching [0005] The lst decades have seen an increased foeus on the diagnosis and understanding of psyehological disorders such as autism spectrum and attention deficit hyperactivity disorder (ADHD). Patents have high expectations on the ability of educators to spot these disorders in the stidents While diagnosing those disorders is dificult and requires professional judgement by the subject matter experts [0006] To. address aforementioned issues, it wold be desirable to ave an intelligent learning and training system that models the alective and cognitive states of the student, to assist the teacheritrainer in providing personalized Feb. 13, 2020 struction, monitor the student's mental health and mini 7 acinistrative tasks to let the leocherltsiner focus on ‘eaching/raining, [007] The abovementioned dilculty of providing per- sonalized leaning, shortage of qualified techerftrainer, and ‘unmet needs for psychological disorders eury diagnosis ca also be commonly found in corporate training setings [008] "Regarding psychological disorders carly diagnoss, a8 the populations of most developed counties are all getting older, medical and elderly care systems are ine Ing stretched for resourees and care providers. One of the areas of medical and elderly care that often receives the least attention is the neurocognitive disorder disgnosis, preven- tion, and treatment. Even though early and accurate diag- ‘nosis of the various types of neurocognitive disorders ean lead to effective traiments, similar to the academic and corporate training settings, accessibility to qualified profes- sionals is an issue SUMMARY OF THE INVENTION [0009] ‘The present invention provides a method and a system for delivering and managing interactive and adaptive eaming and training programmes using a combination of sensing of the studeat subject's gestures, emotions, and ‘movements, and quantitative measuremenis of test results and leaning progress. It is also an objective ofthe present Jnvention t provide such method and system applicable in workplace performance monitoring and appraisal asses ‘ment. Its stil another objective ofthe present invention to provide a method and a system for neurocognitive disorder iggnosis in. general medical services environment and elderly care environment [0010] In accordance to one aspect of the present inven- ‘ion, the system estimates the affective state and cognitive sate of the subject by image and/or video capturing and analyzing the sujet’ facial expression, eye movements, point-of-gaze, and head pose, and physiologic detection, ‘uch as tetile pressure exerted on a tactile sensing device, subject's handriting, and tone of voice during a sampling time window. The image or video capture can he performed by using built-in or peripheral cameras in desktop comput- er, laptop computers, sblet computers, andr smartphones used by the subject, andior ether optical sensing devices. ‘The expired images andr videos are then analyzed using ‘machine vision techniques. For example, stalled eye move- ‘ments, out-of focus point-of ga7e, anda tilted hood pose are signals indicating lack of interest and attention toward the Sbjct matters being presented inthe test questions: while a strong twetile pressure detected is signal indicating anxiety lack of eonfidence, andor frusteation in te subject ‘matters being presented in learning of taining seston, [011] In accordance to one embodiment, selected perfor: ‘mance data and behavioral data from the subject ate also collected in determining the subjects understanding of the Jeaming materials. These selected performance data and behavioral data include, but not limited to, correctness of answers, number of suecessfil and unsuccessiul attempls, foumber of toggling between given answer choices, and response speed to test questions of certain types, subject ‘matters, andor dificuly levels, and working stops toward a solution. For example, the subject's excessive togaling between given choices and slaw response speed in answer: ing a test question indicate doubts and hesitaions on the answer (0 the fest question. The subjet’s working steps US 2020/0046277 AI toward a solution toa test problem are captured for matching ‘withthe model solution snd in turn provides insight to the subject’ understanding ofthe materials 10012] The alletve state and cognitive state estimation and performance data are primarily used in gauging the subject's understanding of ad interests in the materials ‘covered in a learning or taining programme. While a single ‘estimation is used in providing a snapshot assessment ofthe subject's progress in the learning of training programmes ‘and prediction ofthe subject's test results on the materials, riuiple estimations are used in providing an assessment history and trends of the subject’s progress in the leaming or training programme and traits (eg. strengths and weak- nesses, study pattems and habits) of the subject, Furr ore, the tends ofthe subjet’s progres in the leaming or teaining programme and tris of the subject, and the est- mated affective sates and cognitive states of the subject are used inthe modeling of the leaming o taining programme in terms of choice of subject matter materials, delivery methods, and administration, [0013] Tn accordance to another aspect of the present ‘invention, the method and system for delivering and man- aging interactive and adaptive learsing and training pro- rammes logically stractore the lecture materials and the livery mechanism data in a learning aod training pro- framme as Domain Knowledge, and its data are stored in 8 Domain Kaowledge repository. A Domain Knowledge repository comprises one oF more Concept objects and one ‘or more Task objects. Each Concept object comprises ne oF ‘more Knowledge and Skil items. The Knowledge and Skill items are ordered by difficulty level, and ‘wo or more Knowledge and Skill items can be linked to form a Cut= riculum, Inthe case where the present invention is applied ina schoo, Curriculum defined by the present invention is the equivalence of the school curriculum and there is one- to-ane relationship between a Knowledge and Skil item and ‘a eson in the school curriculum, The Concept objets aa, be linked to form a logical tree data structure for vse in @ Task selection process. {0014} Bach Task object has various lecture content mate- Fins, and is associated with one or more Concept objects in ‘8 Curriculum. In accordance to one embodiment, & Task ‘Object can be classified as: Basic Task, Interactive Task, oF ‘Task with an Underlying Cognitive or Expert Model. Each Basie Task comprises one or more lecture notes, ilustra- tions, test questions and answers designed to assess whether the subject has read all the materials, and instructional Videos with embedded test questions and answers. Each Interactive Task comprises one or more problem-solving ‘exereses each comprises one or more steps designed 10 aide the subject in deriving the solutions to problems. Each Step provides an answer, common misconceptions, and hints. ‘The steps are in the onder designed to follow the dalivery flow of lecture, Each Task with an Underlying ‘Cognitive or Expen Model comprises one or more problem= solving exercises and each comprises one or more heuristic rules andior constraints for simulating problem-solving ‘exereise steps delivered in synchronous with a student subjects leaming progress. This allow a tailored sealold- ing (eg. providing guidance and/or hints) for each student subject hased on 3 point in a problem set or space presented jn the problem-solving exerise, [0015] In accordance to another aspect of the present invention, the method and system for delivering and man- Feb. 13, 2020 ‘aging interactive and adaptive lear ‘srammes logically builds om top of the Domain ls {wo models of operation: Student Model and Training ‘Model. Under the Stadeat Model, the system executes exch of one or more of the Task objects associated with a Curriculum ia « Domain Knowledge in a learning session for a student subject, During the execution of the Task objects, the system measures the student subject's perfor ‘mance and obtain the stadent subject's performance metrics in cach Task such as: the numbers of suecessfil and unste- cessful attempys to questions in the Task, number of hints requested, and the time spent in completing the Task. The performance metrics obtained, along with the information of the Task object, such as its dilficuty level, are fed into a logistic rogrssion mathematical model of each Concept ‘object associated with the Task object. This also called the knowledge tace ofthe student subject, whichis the ealen- Jation of a probability of understanding of the material in the ‘Concept objet by the student subject. The advantages ofthe Student Model include thatthe execution ofthe Task objects ‘an adap to the changing ability ofthe stadent subject For ron-limiting example, following the Student Model. the system can estimate the amount of learning achieved by the sfudent, estimate how much learing gain ean be expected for a next Task, and provide a prediction of the student subjeet’s performance in an upcoming test. These data are thon used in the Training Model and enable hypothesis ‘esting (© make further improvement to the system, evaluate teachertainer quality and lecture material quality [0016] Under the Training Model the system receives the ata collected from the exccution of the Task objects under the Student Model and the Domain Knovsledge foe making decisions on the leaning or taining sirtegy and providing feedbacks to the student subject or teachertrainer, Under the ‘Training Model, the system is mainly responsible for exeeut- ing the followings 1) Define the entry point for the fist Task. Initially all indicators for Knowledge and Skil items are st to defaults, \whiel are infered from data in either an application form filled hy the student subject or teachentrsiner or an initial assessment of the student subject by the teachertrainer Select the sequence of Tasks to execute. To select the next ‘Task, the system's ainer module has to seareh through a logical toe data structure of Concept objects, locate a ‘Knowledge and Skill with the lowest skill evel and then use 4 question matrix to lookup the eorresponding Task items that mateh the traits (eg. strengths and weaknesses, study pattems and habits) of the student subject. Once selected, the ‘necessary lecture content materials pulled from the Domain ‘Knowledge, and send to the system's communication mod- ‘ule foe delivery presentation ia the systems communication sodule user interlace, 2, Provide feedback. While the stadent subject is working ‘ona Task object being executed, the system's tier module ‘monitors the time spent on each Task step. When a limit is ‘exceed, Feedback is provided asa function of the curent affective state ofthe student subject. Por example, this ean be an encouraging, empathetic, or challenging message selected from a generic ist, rit isa dedicated hint from the Domain Knowledge. 3.) Drive the system’s pedagogical agent, The systems ‘miner module matches the current allective slate of the student subject with the available states in the pedagogical ‘agent, Besides providing the alective state information, ext US 2020/0046277 AI messages can be sent tothe system’s communication mod- tle for rendering the pedagogical agent in user interface. 4.) Decide when a Concepts mastered. As describe earlier, under the Student Model, the system estimates the student subject's probability of understanding of the materials ia ‘each Concept, Based on a predetemnined threshold (e., 95%), the teachertrainee ean decide when a Concept is mastered 5.) Flag student subjeet’s behavior that is recognized to be related to mental disorders. For example, when the system's ‘exccution under the Student Model shows anomalies in the sensor dala compared to a known historical context and ‘exhibits significant lower leaming progress, the system under the Training Model raises a warning notice to the teachertainer. [also provides more detailed information on ‘common markers of disorders such as Attention Deficit, Hyperactivity Disorder (ADHD) and Autism Spectrum Dis- ‘onder (ASD). 10017] The present invention can also be applied in medi ‘al assessment for cognitive disorders, such us Alzheimers" ‘dementia and autism ADHD. In accordance to one embod ment, provided is a neurocognitive disorder diagnosis sys- tem for administering a cognitive test administered to patient subject. The system monitors ane estimates the patient subject’ affective state and cognitive state using the collected and analyzed sensory data on patient subject's {aia expression, eye movemeals, point-of-a2ze, head pose, voiee, speech clarity, ection Gime, andlor touch responses similar to the aforesaid system for delivering and managing Jneractive and adaptive learning and training programmes. [0018] "The cognitive test materials can also be based on the patient subject’s distanced event knowledge and recent ‘event knowledge so to assess the patient subject's states of Jong-term memory and short-term memory through memory recall time and accuracy as art of the patient subject's test performance data, The patient subjects alective state and ‘cognitive state estimation, along withthe patie subject's ‘countive lest performance data during the cognitive test are Teedback to the system to drive the course ofthe eognitive test, adaptively change the cognitive test materials, and influence the paticat suhjet’s emotions. 10019] "The neurocognitive disorder diagnosis system pro- Vides a real-time diagnosis that is less prone to human eto The patient subject’s affective state and cognitive state ‘estimation can also be matched and used alongside with MRI data on the patient subjects brain ativty in further study. 0020] |The goal of neurocognitive disorder diggnosissys- tem is tenable the early detection of cognitive disorders, particularly among elderiy in elderly care lites such as Fetirement homes, through the periodic administrations of ‘cognitive tests using this system. Another goal isto enable the tacking of treatments, and in turn drive the adjustments jn the course of the treatments, medications, ad frequencies of dactor’s visit BRIEF DESCRIPTION OF THE DRAWINGS 10021] Embodiments of the invention are described in more detail hereinafier with reference to the deawings, ia whieh [0022] FIG. 1 depicts a schematic diagram ofa system for delivering and managing interactive and adaptive learning fand training programmes in accordance to one embodiment ‘of the present inveation Feb. 13, 2020 [0023] FIG. 2 depicts logical data flow diagram of the System for delivering and managing interactive ae adaptive Teaming and taining programmes; [024]. FIG. 3 depicts aa setvity diagram of a method for Selivering and managing interactive and adaptive learning ‘and training programmes in aeconlanee to one embodiment of the present invention: [0028] FIG. 4 depicts a flow diagram of an iterative ‘machine learning workflow used by the system ia ealculat- ing a probability of understanding of lecture materials by the student subject; snd [0026] FIG. § ilustrates a logical data structure used by the system for delivering and managing interactive and ‘adapive leaming and taining programmes in accordance to ‘one embodiment of the present invention. DETAILED DESCRIPTION [0027] In the following description, methods and systems {or delivering and managing learing and ining. pro- ‘rammes, neurocognitive disorder diagnosis, and the likes fare set forth as preferred examples. It will be appareat to those skilled in the aft that modifications, inching addi- jons andlor substitutions may be made without departing from the scope and sprit ofthe invention, Specific details :may be omitted so as not to obscure the invention; however, the disclosure i written to enable one skilled inthe art 10 practice the teachings herein without undue experiments. tion. [0028] In accordance to various embodiments of the pres- ent invention, the method and system for delivering and ‘managing interactive and adaptive leaning and taining programmes uses a combination of sensing of the student subject's gestures, emotions, and movements, and quantita- tive measurements of tet resulis and learaing progress. [029] In accordance to one aspect of the present inven- ‘ion, the system estimates the affective state and cognitive state of the subject by image and/or video capturing and analyzing the sujet’ facial expression, eye movements, point-of-gaze, and head pose, and haptic feedback, such as tactile pressure exerced ona tactile sensing device during a sampling time window. The image or video capture can be performed by using built-in or peripheral cameras in desktop computers, lupiop computers, let computers, andior Smartphones used by the subject, andor other optical sen ing devices. The captured images and/or videos are thea analyzed using machine vision techniques. For example, stalled eye movements, out-of-focus point-of-gaze, and 3 filted head pose are signals indicating lack of interest and attention toward the learning materials being presented in the learning or training session; while a strong tactile pressure detected is a signal indicating anxiety, lack of fonfdence, andr fusion in the subject matters being asked in atest question [030] In acconlance to one embodiment, selected perfor: ‘mance data and behavioral data from the subject ate also collected in the affective state and cognitive tate estimation. These selected. performance data and. behavioral data include, but not limited to, correctness of answers, number fof successfal and unsuccessful attempts, toggling between given answer choices, and response speed 1 fest questions fof certain types, subject matters, andlor difficulty levels, working steps toward solution, and the subject's handwrit jing and tone of voice. Far example, the subjct's repeated togaling between given choices and slow response sped US 2020/0046277 AI ‘answering atest question indicating doubts and hesitations ‘on the answer to the test question. The subject's working steps toward solution toa test problem are captured for matching with the model solution and ia tum provides insight to the subject's understanding of the lecture materi- al, 10031] In accordance to various embodiments, the system Jor delivering and mangging interstive and adaptive learn- Jing and tesining programmes comprises a sensor banding module implemented by a combination of software and finnware executed in general purposed and. specially designed computer procestors, The sensor handling modile ‘manages the Various sensors employed by the system. The sensor handling module isin elecirical andlor data eomm- nications with various electronic sensing devices including, but not limited to, optical and toveh sensing devices; input devices including, But not Timited 1, keyboard, mouse, pointing device, sty, and electron pens imoge capturing vices: and cameras 0032} Daring the operation of the system, inpot sensory data are continuously colleted at various sampling rates and averages of samples of input sensory data are computed. In fonder to handle the different sampling rates of different sensing devices, a reference rate is chosen (eg. § Hz). A slower sampling rate input sensory data i interpolated With zero order hold and then sampled at the reference mite. A higher sampling rate input sensory data is subsampled at the reference rate. Afler the sample rate aljgnmeat,a trace ofthe Jast few seconds is kept in memory afte which the average is calculated. Effectively this produces 1 moving average of am input sensory data and acts as alow pas filter to remove 10033] Eye Movements, Point-of-Gaze, and Head Pose Detection 10034) Inoneembediment, a low-cost optical sensor bull in in a computing device (eg. subject facing camera in @ tablet computer) is used, Ata rate of minimal 5 Hz, images ate obtained from the sensor. Each image is then processed by faceleye tracking and analysis systems known inthe ar. ‘The three-dimensional (3D) head orientation is measured in Euler angles (pitch, yaw, and roll). To measure the poinl- ‘ofgaze, a 3D vector is assumed from the origin of the ‘optical Sensor tothe center ofthe pupil ofthe user, second 3D vector is determined from the center of the eye-ball 0 the pupil. These two veetors are then used to calculate the point of gaze. A calibration step helps to compensate for ‘ollsets (subject position behind the screea, camera position relative to the screen) Using this data, the planar coordinate ‘of the gaze on the computer sereen can be derive. 10035] Facial Expressions and Emotions Determination [0036] In another embodiment, the images and/or videos ‘captured as mentioned above, are processed to identify key landmarks on the face such as eyes, tip of the nase, comers ‘ofthe mouth. The regions between these landmarks are then analyzed and classified into facial expressions such a ‘attention, brow urow, brow mse, cheek raise, chin raise,

    You might also like