This full text paper was peer-reviewed at the direction of IEEE Instrumentation and Measurement Society prior to the

acceptance and publication.

A Synthetic Instrument for Diagnosis and

Performance Measurement of Individuals with
Visual Sequential Memory Deficit
Ayşe Rumeysa Mohammed1 , Ammar Rashed1 , Shervin Shirmohammadi1,2
1
College of Engineering and Natural Sciences, Multimedia Systems Lab
Istanbul Sehir University, Istanbul, Turkey
{aysemus | ammarrashed}@std.sehir.edu.tr, shervinshirmohammadi@sehir.edu.tr
2
Distributed and Collaborative Virtual Environments Research Lab (DISCOVER Lab)
University of Ottawa, Ottawa, Canada
shervin@discover.uottawa.ca

Abstract—When performing tasks such as remembering let-
ters, numbers, objects, or shapes, a persons Visual Sequential
Memory (VSM) plays a crucial role, especially when the order
of the tasks is important. Lack of VSM makes the persons life
more challenging, possibly leading to dyslexia and dyscalculia.
As such, it is important to detect and treat Visual Sequential
Memory Deficit (VSMD). But current clinical methods have a
low rate of diagnosis, and also offer limited hours to persons
being treated in the clinics. In this paper, we propose an Origami
based Serious Game, called Memori, as a synthetic instrument
for the diagnosis, performance measurement, and treatment of
people with VSMD. We illustrate the rationale behind using
Origami, the design process of our game, and its implementation.
Our preliminarily performance evaluations with 24 adults reveal
a 13% improvement of memory and 1.00 score increase in
performance while a slight decrease occurred in attentiveness from
2.33 to 2.02 for people who use our tool.
Keywords—Visual Sequential Memory Diagnosis and Perfor-
mance Measurement; Serious Games; Origami.
I. I NTRODUCTION
Serious Games are games designed for any purpose other
than just entertainment Originated in the 1970s [1]. They
are often used in training and education. For example, many
serious games have been proposed and successfully used for
the treatment of children with Autism Spectrum Disorders [2]
[3]. The effectiveness of serious games for education has been
confirmed in many studies, such as [4].
In the instrumentation and measurement community, serious
games have been used for a variety of medical measurement
applications. For example, [5] proposes a therapeutic serious
game using a natural user interface materialized by Kinect
to motivate patients with upper limb disabilities to perform
rehabilitation, while measuring arm motion metrics in an
unobtrusive way. MeMaPads [6] is an exergaming system de-
signed for overweight or obese children. It promotes exercising
through playing, and consists of a number of sensory mounted
pads used to interact with serious games. Another work [7]
uses serious games to engage disabled children in therapy
while also tracking their motion data through multiple sensors
in a noninvasive manner. As a last example, [8] proposes a
framework that adds context awareness, by capturing temporal
and spatial context as well as the person’s vital parameters,
to serious games. This will assist the user to find the most
appropriate health services according to his/her needs.
VSM indicates a person’s competency of remembering
letters, numbers, objects or shapes in the correct order [9].
VSM is crucial as it affects various aspects of life since we
use this ability in our daily lives while remembering names,
phone numbers, directions, recipes, etc. A person with VSMD;
i.e., a person whose VSM is not working as well as a healthy
person, could experience problems in various aspects of life
and be affected by different disorders such as dyslexia [10]
and dyscalculia [11].
Individuals who experience difficulty in VSM often show
it when having difficulty remembering numbers in sequence,
sequencing letters in similar words such as was/saw or re-
verse/reserve, copying from a book to a notebook, or recalling
events and tasks in series. In the absence of such abilities,
possible problems arise: dyslexia [?], dyscalculia [11], and
reading difficulties [12].
In this work, we present a serious game, called Memori, as
a synthetic instrument that can run on any general computing
hardware, such as laptop, tablet, smartphone, PC, etc., to di-
agnose, measure performance, and perform therapy for people
with VSMD, especially children who find games attractive.
To the best of our knowledge, while there are some games
available for visual memory, dyslexia and dyscalculia treat-
ment, as will be shown in Section II, there is no serious
game directly targeting VSM. In this paper, we will show our
implementation, preliminary results and analysis.
Our research questions are as follows:
1) Does our game succesfully diagnose people with
VSMD?
2) Does our game succesfully treat people with VSMD?
3) Is the instruction medium better in Paper or PC?
The rest of this paper is organized as follows: in Section
II we cover related work, in Section III we present the game

978-1-5090-2984-6/17/$31.00 ©2017 IEEE

design, while in Section IV we show our implementation, and III. I MPLEMENTATION
in Section V we discuss our results and limitations. Finally,
we conclude our paper. A typical environment for origami is a desk or table
that serves the scene aesthetically while receiving shadows
II. R ELATED W ORK
casted by the paper object, immersing the gameplay from a
A detailed description of the related work in terms of casual 2D origami schemes to much more realistic perspective
Current Clinical Methods, Serious Games for Visual Memory, view of each step. For this purpose, we have designed and
Serious Games for Dyslexia and Dyscalculia, Origami in implemented a Serious Game called Memori, using Unity3D
Therapy and Education, and Origami as a Diagnostic Test has exported to WebGL, and playable in any web browser and any
been covered in [13]. We do not repeat the first 3 here, but to platform. Our game presents the user with various Origami
make this paper self-contained, it is important to cover again shapes, each designed to act for the specific goal of tutorial,
the last two, as discussed next almost verbatim: diagnosis, or therapy, as discussed next.
A. Origami for VSM in Therapy and Education
While playing Origami, a child will invoke his/her VSM A. Tutorial
quite heavily. Gross suggests that while doing Origami, one
needs use the following skills: following a sequence of direc- Tutorial starts with introducing the player to the game
tions, listening skills, reading skills, language skills, writing environment and controlling mechanism. The player can play
skills, social skills, mathematics, spatial relationships, memory the whole animation in one shot or go step by step as many
and concentration [14]. According to Kaplan, performing a times as he wants until he feels comfortable with the game
folding act requires following instructions, solving problems, environment, see Figure 1. The Tutorial scene has to have a
learning the steps in the proper order, thinking skills, selfim- very simple origami shape which in our case is “Shirt”. The
age, and success experience [15]. Other studies have come player is also given a piece of paper to do the shape while
to similar conclusions, that Origami requires skills that are watching the steps.
directly related to VSM [16][17].
Origami has been used for therapeutic and educational
purposes by many researchers in the past. Origami has also
been shown to benefit learning disabled, dyslexic, physically
or emotionally handicapped students [18]. Origami also helps
with fine motor control and hand writing, attentiveness, se-
quential memory, and specifically helps children with learning
disorders to develop fine motor skills, observation skills, and
visual sequential memory [19]. A child suffering from dyscal-
culia who cannot comprehend or manipulate numbers, forms
or anything related to those can also benefit from Origami
by learning to follow the sequence and order of things [11].
Origami is also utilized in pediatrics for mental health, clinical
fine/visual motor skills, memory, and sequencing [20].
B. Origami as a Diagnostic Test
For diagnosis, Origami can evaluate which basic difficulties
a child might have in the acquisition of topological concepts
and spatial orientation [21][19]. Brown, et. al. [19] used
Origami as a tool to diagnose children with VSMD. 64
children of 8-9 years old were shown origami folds such as
Basic House, Sailboat, and Duck, then asked to duplicate. If
the child is unable to exert success, that indicates poor VSM
skills.
The above work all show very promising effects that Fig. 1. Some steps of T-shirt origami
Origami applications can have on VSM, for both diagnosis and
treatment. In this paper, we are revisiting such applications of
Origami, in the context of today’s technology and gaming en- B. Diagnosis
vironments. We design a platform-independent mobile serious
game called Memori that shows to a child the step-by-step In diagnosis the player has one option which is to play the
order of making various Origami shapes. Using the child’s whole animation, see Figure 2. That is he cannot go step by
performance, one can then diagnose the child for VSMD, as step. After repeating the animation n times, the player is given
well as treat children with VSMD at clinics and homes. a piece of paper to do the shape.

Fig. 2. Some steps of House origami
C. Therapy
In therapy, the player has both options of playing the whole
animation, or going step by step. In Figure 3, we can see the
Windmill origami for the therapy level.
Fig. 3. Some steps of Windmill origami
For Tutorial and therapy stages, the game scene has a Next
and a Back buttons for going between steps, as shown in
Figure 4. In Diagnosis and Therapy stages, there is a Play
button to play the whole animation. Should the player go away
from the main scope of the scene, there is a Reset Camera
button to reset the view to default position without changing
any of the game status (or current step in the shape).
Fig. 4. Game scene showing Next, Back, and Reset Camera Buttons
IV. P ERFORMANCE E VALUATION
We carried out experiments to test various subjects and
see if our tool can diagnose any of them with VSMD, and
also to see if the game when used as therapy can improve
the performance of the subjects. These are explained next,
starting with the game itself. The participants were recruited
from Istanbul Sehir University. Our experiment was conducted
for 4 weeks.
A. The Test Game
For testing, Memori was set to present animations of various
origami shapes such as T-shirt, Basic House, Sailboat, Duck,
Windmill and Heart. During the Tutorial, the player gets
used to the game environment, game progression and folding
activity. Then for each origami shape, the whole animation of
folding from the first step till the last one is played. Player
is allowed to watch the animation as many times as needed.
However, once player’s folding starts, it is not to be watched
again. It is obligatory to remember the sequence of steps and
fold the origami figure correctly.
B. Participants
24 adults (ages range between 18 and 27) have participated
in this study. No subject had ADHD or any physical disabilities
and none of them were under medication during the game play
that have affected their attention.
C. Procedures
Participants were provided a room with no distractions.
ASUS brand laptop with Core i5 is placed on a table. The
game is displayed via Google Chrome. Subjects were asked
to play. In total 6 origami shapes (including the tutorial) were
folded in 2 sessions (1 session per week).
Before starting the game, participants answered a ques-
tionnaire about demographic information, visual memory, and
physical abilities (regarding fine motor skills). Each of them
were given a participant ID to preserve their privacy. 4
participants were randomly selected to be assigned to one
group that received the demonstration by instructor. Everyone
received instructions and presentment about how to play the
game.
 For the non-computer game, the instructor showed the folds • Memory. Number of correct steps the participant folded.
and the participants repeated them from their memory. For the • Attentiveness. A variable defined by how many times the
computer-based game, the same procedure was applied except origami animation watched.
origami animation was played on the web browser. • Medium. Performances of each medium; Paper and PC.
D. Experimentation Stage V. E XPERIMENTAL R ESULTS & D ISCUSSION
As explained earlier, the game consists of Tutorial, Diagno- In order to diagnose people with VSMD, we defined a
sis and Therapy parts. performance score and test it with the data set given in
1) Tutorial: This level aims at familiarizing the user with [19]. After testing, we were able to determine a threshold
the game environment and folding process. On average, it takes which if the participant is below that, it indicates that the
128 seconds to finish the tutorial. Only for this level, time player is suffering from VSMD. In Table I, we can see
duration for each fold is saved. the corresponding performances for participants. Based on
2) Diagnosis: In this part, there are 3 origamis; Basic the testing dataset, the threshold is calculated as 0.2. Any
House, Sailboat, and Duck. The difficulty, i.e. number and participant with performance score of 0.2 or less is diagnosed
complexity of folding steps, increases in the corresponding with VSMD. In our experiment, participant 7301, 7338 and
order. Along the diagnosis protocol, users must play the whole 7868 are detected.
origami folding animation and then try to construct the shape.
3) Therapy: In this level, players will see the whole ani- TABLE I
P ERFORMANCE SCORE FOR EACH PARTICIPANT
mation and do the folding. If they are unable to finish all the
steps then by using Next and Back buttons, they can navigate Participant ID Performance Medium
through the animation and complete the folding activity. 5473 0.96 PC
1061 0.39 PC
E. Data Processing 7301∗ 0.14 PC
Number of successive correct folds and time spent for 3996 0.31 PC
3682 0.88 PC
one origami are used to diagnose for VSM. Therefore these 7338∗ 0.11 PC
parameters are noted down for analysis. 8249 0.31 PC
We defined an equation, as shown below in Eqn (1). 3225 0.32 PC
7590 0.22 PC
s 2414 0.21 PC
P erf ormance = r
∗ 100 (1) 5507 1.03 PC
t∗n 7452 0.21 PC
5835 0.47 PC
where s is the correct number of folds for the participant, r
2502 0.41 PC
is the total number of folds for one origami, t is the time that 6385 0.80 Paper
the participant takes to finish the folding activity and lastly n 9828 1.04 Paper
is the number of times the participant watched the animation. 1164 0.78 Paper
This equation gives participants a score to determine the 4947 0.33 PC
3464 0.73 PC
performance for each participant. Similar to the experiment
2430 0.59 PC
in [19], we used the number of correct folds and the times 7868∗ 0.20 PC
animation is watched as features. Distinctly we decided to 2772 0.68 Paper
add another variable to the equation: time. Since time is an 8542 0.23 PC
important feature for performance [22] and additionally it is 5697 0.47 PC
seen as a crucial tool for visual objects like animation [23], * indicates diagnosed participants
we inserted time as a supplementary variable for our equation.
Later, we tested the performance equation on the dataset in Some participants, i.e. 5835 and 5507, took more time
[19], and specified the threshold for diagnosing VSMD. to fold not because they didn’t remember the sequence, but
To compare the performance in two different mediums, we because they wanted the folds to be perfect and geometrically
selected 8 people who showed the resembling characteristics equal. Therefore for future work, we believe that we should
based on the pre-survey we conducted before playing our define another variable indicating correctness of the final shape
game. Then among these people, 4 of them were randomly for the performance equation. One important limitation of the
assigned to one group which received the instructions by the experiment was that there was no time limit. Hence some
instructor and the rest were assigned to the other group. participants used trial-and-error method for figuring out the
fold. Next step in our research, we inted to settle a max time
F. Measurements for each origami.
Our main interest was the performance of each participant. During our research we diagnosed 3 participants with
However we needed few more variables to answer the research VSMD. Nonetheless, participants 7590, 2414, 7452 and 8542
questions in Section I: have scores which are very close to the threshold value,
• Performance. As defined in Eqn 1. see Table I. Since we conducted our experiment only for 2
sessions, we think that it is a significant limitation to our
results. Having more sessions will provide additional data
which will result in more accurate diagnosis.
Table II, shows the comparison of attentiveness, memory,
and performance for two sessions. Attentiveness refers to
the quick respond defined by the times origami animation is
watched. As can be seen, the average time is decreased slightly
from 2.33 to 2.02. Memory is assigned as the percentages of
the average number of correct folds. There is a remarkable
increase in memory percentage which indicates successful
progress for future sessions as well. Likewise, same trend is
observed in performance. The increment from 3.16 to 4.16
shows promising effect in imroving VSM skills.
TABLE II
E XPERIMENT R ESULTS
Attentiveness Memory Performance
Session 1 2.33 %67 3.16
Session 2 2.02 %80 4.16
Lastly, we wanted to see if the instruction medium is better
in paper or PC. We compared the variables, see Table III. For
the non-computer game, almost all the players preferred to see
the instructions only once, but for the computer based game,
they watched the animations till they feel confident enough
to start folding. Although this helped them to increase the
correct number of folds, it affected our results in attentiveness
and therefore performance.
TABLE III
M EDIUM C OMPARISON
Attentiveness Memory Performance
Paper 1.10 %74 0.75
PC 1.70 %83 0.61
VI. C ONCLUSION
In this paper, we tested the serious game Memori for
diagnosing and treating people with VSMD. During our ex-
periments, we observed improvements in performance. Our
preliminary results show promising enhancement in VSM
skills.
Although the results of game medium for Paper was better
than PC, our game has many advantages; (i) can be done in
the natural environment of the player, (ii) time saving, (iii)
financially economical since the game runs on web browser,
its accesibility is quite high.
ACKNOWLEDGMENT
We would like to thank Assistant Professor Reyyan Bilge for
her guidance. We want to also acknowledge the support from
Shady A. Mohammed who helped us during the experiments.
This study is developed under the grant from Euro-
pean Project “Intelligent Serious games for Social and Cog-
nitive Competence”. For further information please visit
www.isg4competence.com
R EFERENCES
[1] J. Leng, Handbook of Research on Computational Science and Engi-
neering: Theory and Practice: Theory and Practice. IGI Global, 2011,
vol. 2.
[2] H. M. Zakari, M. Ma, and D. Simmons, “A review of serious games
for children with autism spectrum disorders (asd),” in International
Conference on Serious Games Development and Applications. Springer,
2014, pp. 93–106.
[3] H. A. M. Noor, F. Shahbodin, and N. C. Pee, “Serious game for autism
children: review of literature,” World Academy of Science, Engineering
and Technology, International Journal of Social, Behavioral, Educa-
tional, Economic, Business and Industrial Engineering, vol. 6, no. 4,
pp. 554–559, 2012.
[4] C. Girard, J. Ecalle, and A. Magnan, “Serious games as new educational
tools: how effective are they? a meta-analysis of recent studies,” Journal
of Computer Assisted Learning, vol. 29, no. 3, pp. 207–219, 2013.
[5] V. Viegas, O. Postolache, J. Pereira, and P. Girỳo, “Nui therapeutic
serious games with metrics validation based on wearable devices,” in
Instrumentation and Measurement Technology Conference Proceedings
(I2MTC), 2016 IEEE International. IEEE, 2016, pp. 1–6.
[6] A. Karime, B. Hafidh, A. Khaldi, J. M. Aljaam, and A. El Saddik,
“Memapads: Enhancing children’s well-being through a physically inter-
active memory and math games,” in Instrumentation and Measurement
Technology Conference (I2MTC), 2012 IEEE International. IEEE,
2012, pp. 2563–2566.
[7] M. A. Rahman, M. Ahmed, A. Qamar, D. Hossain, and S. Basalamah,
“Modeling therapy rehabilitation sessions using non-invasive serious
games,” in Medical Measurements and Applications (MeMeA), 2014
IEEE International Symposium on. IEEE, 2014, pp. 1–4.
[8] S. Hardy, A. El Saddik, S. Göbel, and R. Steinmetz, “Context aware
serious games framework for sport and health,” in Medical Measure-
ments and Applications Proceedings (MeMeA), 2011 IEEE International
Workshop on. IEEE, 2011, pp. 248–252.
[9] D. C. Giles and C. D. Terrell, “Visual sequential memory and spelling
ability,” Educational Psychology, vol. 17, no. 3, pp. 245–253, 1997.
[10] N. K. Goulandris and M. Snowling, “Visual memory deficits: A plausible
cause of developmental dyslexia? evidence from a single case study,”
Cognitive Neuropsychology, vol. 8, no. 2, pp. 127–154, 1991.
[11] I. Lanoux, “Les mathematiques, pas a plis,” in Proc. First International
Conference Origami Education Therapy (COET ’91), Birmingham, UK,
oct 1997, pp. 291–295.
[12] K. Hermann, Reading disability: A medical study of word-blindness and
related handicaps. Munksgaard, 1959.
[13] A. R. Mohammed, A. Rashed, and S. Shirmohammadi, “Memori: A
serious game for diagnosing and treating visual sequential memory
deficit,” in 5th IEEE Conference on Serious Games and Applications
for Health (SeGAH’17), Perth, Australia, april 2017, pp. 1–7.
[14] G. M. Gross, “Using origami in the classroom,” in Proc. First Interna-
tional Conference Origami Education Therapy (COET ’91), Birming-
ham, UK, oct 1991, pp. 95–100.
[15] D. Kaplan, “Paperfolding - plus,” in Proc. First International Conference
Origami Education Therapy (COET ’91), Birmingham, UK, oct 1991,
pp. 154–158.
[16] M. B. Brown, “Teaching techniques for the child with a learning
disability - origami, a learning disability aid,” in Proc. First International
Conference Origami Education Therapy (COET ’91), Birmingham, UK,
oct 1991, pp. 282–290.
[17] N. Robinson, “Thoughts on origami and therapy,” in Proc. First Inter-
national Conference Origami Education Therapy (COET ’91), Birming-
ham, UK, oct 1991, pp. 416–439.
[18] F. Temko, “Paperfolding in schools,” in Proc. First International Con-
ference Origami Education Therapy (COET ’91), Birmingham, UK, oct
1991, pp. 223–228.
[19] M. B. Brown and M. Loomis, “Origami as a diagnostic test,” in Proc.
First International Conference Origami Education Therapy (COET ’91),
Birmingham, UK, oct 1991, pp. 264–281.
[20] S. Gold, “Origami as a therapeutic tool,” in Proc. First International
Conference Origami Education Therapy (COET ’91), Birmingham, UK,
oct 1991, pp. 383–385.
[21] M. Paparo, “Origami as diagnosis, recovery and socialization in the
learning of reading and writing,” in Proc. First International Conference
Origami Education Therapy (COET ’91), Birmingham, UK, oct 1991,
pp. 363–368.
[22] B. Bruno, J. Bart, O. Lubos, D. Florent, S. Victor, and V. S. J. Serge,
“Clinical evolution or familiarization?: Time analysis of serious games
exercises to assess the learning effect,” in 2015 International Conference
on Virtual Rehabilitation (ICVR), June 2015, pp. 166–167.
[23] A. Ferko, Z. ernekov, J. Dadov, V. Major, D. Onailov, E. ikudov,
R. varba, M. Valkov, I. Varhankov, M. Vataha, M. Vesel, and E. Dukov,
“Schola ludus, serious games, and measurement of interestingness,”
in 2011 14th International Conference on Interactive Collaborative
Learning, Sept 2011, pp. 557–558.
[24] R. Van Eck, “Digital game-based learning: It’s not just the digital natives
who are restless,” EDUCAUSE review, vol. 41, no. 2, p. 16, 2006.
[25] A. R. Cano, B. Fernández-Manjón, and Á. J. Garcı́a-Tejedor, “Glaid:
Designing a game learning analytics model to analyze the learning
process in users with intellectual disabilities.”