Measuring Effectiveness of Memory Through Affect

Khamis Faraj Alarabi Abdul Wahab Izza Karim Department of Computer Science Faculty (Kulliyyah) of Information and Communication Technology International Islamic University Malaysia P.O. Box 10, 50728 Kuala Lumpur, Malaysia
Abstract The efforts to measure memory effectiveness have been on the top of research not only in psychology but in cognitive science and engineering as well. Much work has been done by psychologists to understand and model human memory. Atkinson and Shiffrin in [1] introduced multi-store model of memory that consists of sensory memory, short term memory (STM) and long term memory (LTM). More recent Squire and Zola in [17] suggested that LTM is divided into two main classes; declarative and non-declarative memory; each class is subdivided into more specific memories. Since emotions have great influence on effectiveness of memory and they can be measured using EEG signals in terms of valencearousal mapping, this paper proposes new technologies focus on how memory effectiveness can be measured by analyzing dynamic emotion trajectory. Preliminary results of our experiments showed potential of using electroencephalogram (EEG) in measuring the effectiveness of memory through affect. Key words: Implicit memory, Memory test, Emotion recognition, Affect, Memory effectiveness. learning. Human behavior, however, controlled by the information retrieved from the one’s previous experience. Most facts about multiple memory systems have come from findings of studies on brain damaged patients. In [17] and [14], amnesic patients found to perform almost as well as control patients in some tasks classified as implicit memory and perform much less on tasks classifies as explicit memory. The difference in patient performance has led to suggestion that, there are two separate memory systems, one related to explicit and the other to implicit. To understand how information is processed in both explicit and implicit memory, several memory tests have been applied by psychologists. In more recent, the development in brain monitoring techniques such as EEG and FMRI has led to renew the research not only on general framework of memory systems, but to go deeply to localize the brain structures. In this paper we are trying to establish an experiment to measure the implicit memory and to understand how information is encoded and retrieved. Human memory The Atkinson and Shiffrin model described that human memory consists of three stores, sensory memory (store), short-term (STM) and long-term memory (LTM). Sensory store is the first memory stage, where the model supposes that the environmental stimuli are captured by sensors (sense organs), and stored for hundreds of milliseconds in sensory memory. If the stimuli were interesting for the subject, the information about these stimuli is then transferred to the next stage of memory, the Working memory (sometimes called shortterm memory STM). The second stage of information processing in memory is Short-term memory; that is a temporary storage of information. Alan Baddely, in[3], uses STM to refer to the simple temporary storage of information, versus WM,



Human memory has been considered as one of the most interesting research areas in cognitive science. Plethora of research papers have been published on it. Hermman Ebbinghaus was the first who achieved an experimental study on human memory and information processing. Findings from his work on measuring memory have encouraged many others to develop new approaches to understand how memory works. In recent decades, Atkinson and Shiffrin [1], suggested a multistore modal describes how the information is processed in human memory. This model has been the base of more advanced research on memory systems and human

which implies a combination of storage and manipulation. STM has a limited capacity and limited time. Information in STM lasts up to 30 seconds and 7±2 is the maximum number of stimuli that can be processed at once. To keep information for longer time, information has to be rehearsed before it can be transferred and stored into Long-term memory (LTM). If the information left STM before transferring to LTM, the information will be lost forever [1]. Information intended to be transferred into LTM has to be encoded. There are, however, several types of encoding depending on the type of information, (i.e. acoustic and semantic, Konrad, 1972 and Winkensn 1976) and (semantic and visual, Baddeley, 2012). Short-Term Working memory Because of the large demand on working memory, Andres Ericsson and W. Kintsche in [7] proposed a general framework of WM and called it Short-term Working Memory (ST-WM), that includes a mechanism based on skilled use of LTM and referred to it as Longterm Working Memory (LT-WM). The information in LT-WM is stored in stable forms that can be accessed only by means of retrieval cues in ST-WM. For a particular learning process, individuals can acquire over time a set of patterns of information in LTM that allows them to present the information in terms of already learned patterns rather than individual elements.

categories of LTM. Distinguish between these categories has come first from the study on amnesic patients [17]. The operation of capturing environmental stimuli, coding, storing and retrieving information is known as human information processing. That is a combination of human learning and human behavior.


Long-term Memory

In learning process, the person acquires the information, encodes it and stores it in LTM in ways when needed; the information can be easily retrieved. There is a belief among psychologists that there are multiple forms of LTM, each has its own brain structure and processes the information in different manner. These memory forms are classified in two main categories: Declarative memory and Non-declarative memory. Declarative memory (known also as explicit memory) refers to the form of memory which can be consciously recalled. It is divided into two separate systems, episodic and semantic [19]. Episodic memory refers to those memories about specific incidents in our past which allow us to recollect and respond to a particular piece of information. In contrast, semantic memory refers to the generic knowledge of the world and memory of meaning which underlies the conscious recollection [2]. Non-declarative memory (known also as implicit memory) refers to the part of LTM where it is difficult to verbally express the knowledge; rather it can be expressed as a change in behavior without any conscious recollection [2]. Larry R. Squire and S. Zola in [17], introduced taxonomy of Long-term memory systems together with specific brain structures involved in each system, fig(1).

The third stage is Long-term memory. That is a memory storage used to store information for a long time (from view seconds to a long life). LTM has almost unlimited capacity and can retain Information for unlimited time. As will be explained in details later, there are several

Fig. (1): LTM systems and their brain structures.


Human learning

Learning has been defined by many scientists from different points of view. Ruth Weiss in [20] defined learning as “making of memory”, Mayer and Moreno defined it as a combination of neural processes, referred to as a knowledge acquisition process, bringing memories into mind, forming associations, retaining and using them [11]. More recent definition was introduced by J. Ormred is that, learning is the long-term change in mental representation or associations as results of experience [12]. There are, however, two ways that control learning process, they are very different and produce different results, conscious versus unconscious phenomenology [6]. Due to these two different controls, learning process can be divided into conscious learning and unconscious learning. 3.1 Explicit learning Awareness is the main key to distinguish between conscious and unconscious learning. Conscious learning (which known also as explicit learning), occurs when a person deliberately pays attention to the stimuli and intends to acquire information [5]. Scientists suppose that in any given time, what in our working memory is what we are conscious of [20]. This supports that, the explicit memory information is manipulated in working memory. 3.2 Implicit learning In contrast of conscious learning, unconscious (known also as implicit) learning broadly defined as learning without awareness, which means; learning is implicit if a person acquires new information without intending to do so and in such a way that the resulting knowledge is difficult to verbally expressed (i.e. riding a bike) [5]. There is argue, however, that implicit learning does not occur without attention and that attention is important condition for any kind of learning [22]. Therefore, it is sometimes called subconscious instead of unconscious learning. 3.3 Explicit versus implicit memory Memory is made by learning [20]. While there are two different types of learning, consequently, there will be two types of memories; explicit memory and implicit memory. Explicit memory is a memory of information that individuals were acquired with their awareness and they were conscious of [16]. In contrast, implicit memory made without awareness, or with sub-awareness [22]. The different types of knowledge are suggested to be represented (or reside) in different brain regions [21]. LTM of explicit memory (declarative and episodic) has been claimed to be represented in various areas of brain

(particularly the Frontal and Temporal Lobes and also in the Parietal and Occipital Lobes), while LTM of implicit memory has also been claimed to be stored in various areas of brain, that include Basal Forebrain, Striatum, Amygdala and Cerebellum [9]. As stated in [21], damage in declarative system can impair certain forms of implicit learning and the damage to procedural system does not necessarily impair it. This is evidence that explicit memory and implicit memory are stored in different brain region. 4. Memory test In recent years, memory performance has been the most interesting research not only in Psychology but in Engineering and Neuroscience too. Much work has been done to measure both explicit and implicit memories. According to H. Roediger III[ 14], explicit memory can be measured by explicit tests such as free recall and recognition, which reflect conscious recollection of the past. Implicit tests such as word-stem completion and word fragment completion are used to test implicit memory, where they can measure the transfer (or priming) from past experience on tasks that do not need conscious recollection of recent task to perform. As stated in [14], Warrington and Weiskrantz have performed much work on memory tests. In their (1970, experiment 2), they have tested 20 subjects, 4 of them were amnesic. Results of the experiment showed that the performance of control subjects in explicit memory tests was much better than that of amnesic patients. In implicit memory tests, the performance of amnesic patients was a slightly less than that of control subjects. These results support the suggestion that, implicit learning can be impaired by the damage in declarative systems, while it is not necessarily affected by the damage to procedural systems [21]. The experiment of Warrington and Weiskrantz has been repeated many times with different subjects and tasks (e.g. Jacoby and Witherspoon, 1982; Graf, Squire and Mandler, 1984) [14]. Rajaram and Roediger in [13] performed their experiment to compare four verbal implicit memory tests (word identification, word fragment completion, word-stem completion and anagram solution) and to contrast them with free recall as explicit memory test. They used three forms of stimuli; those were visual, auditory and pictorial. The results from this experiment showed that, on the four implicit tests the biggest priming was from visual stimuli, followed by the priming from auditory. The priming from pictorial stimuli was the lowest. In addition, on explicit (free recall) test there was better recall of pictures than words.

On the other hand and more recent, J. Berry et al. have argued against the memory model that dissociated the explicit and implicit memory. They assumed a singlesystem model of priming and recognition instead. Results from their replicated experiments, indicated that, when there was no priming, there was no evidence of existing of recognition; and concluded the recognition and priming are generally associated [4]. 4.1 Memory measurement through emotions The previous studies indicated that emotions can be detected through the brain signals, which can be acquired using EEG machines. Some algorithms have then to be applied on these signals; firstly for features extraction (such as KDD) and secondly for recognition (such as MLP). Since years, scientists have been investigating ways to measure the emotions. As stated in [10], emotions can be mapped using two-dimensional (valence-arousal ) scaling to classify the emotion experience, where the emotion components are represented by two dimensions; the vertical axis represents the arousal (which expresses the reaction to stimuli) and the horizontal axis represents the valence (which indicates how emotion experience is negative or positive). Using such model, four emotions (Fear, Happy, Sad and Calm) can be recognized. Emotions, however, have a great impact on memory. The findings from [8] indicated that memory associated with emotional experience can enhance conscious free recall (declarative memory) of information. This supports the assumption that memory measure can be correlated to emotion recognition. On the other hand, experimental studies showed that recall is mood dependency, where in [18], experiments showed that angry men in word recognition test could recall more negative words than others. In summation, LTM is influenced by emotional status in both information encoding and information retrieval phases; LTM then can be measured using valence-arousal mapping.

5.1 Experiment Methodology Participants Four pre-school children, in the age of 4-7 were participated in this experiment. They were chosen as they are able to name and differentiate between colors, know Latin alphabets and able to name basic shapes. Subjects were able to understand and follow the experimenter instructions. Consent was signed by one of their parents to participate in the experiment.

Stimuli Because participants were pre-school children, stimuli were chosen to be simple and easy to recognize. A-Z Latin alphabets presented in Font 48 were used as stimuli for Alphabet recognition test. Five colored panels against white were used as stimuli for color recognition. Five shapes were used as a memory test stimuli. Protocol Subjects were individually tested. Each subject was seated in a calm room against a computer screen. The 8-channel EEG machine electrodes were put on the scalp of the subject regarding to international 10-20 system. The experiment started with first phase of testing eyes-closed /eyes-open resting condition for one minute each and four 1-minute emotions (Fear, Happy, Sad and Calm) as a reference test. The second phase was alphabet recognition test, where subject was exposed to an alphabet and has to recognize (read) it. Third phase was on color test. Subject was exposed to one of five (Red, Black, Blue, Yellow and Green) colored panel and has to recognize (name) the color. In the fourth phase, subject has to detect left from right hand. The latest phase was about memory test; the subject was exposed to one of five (Square, triangle, circle, star and hexagon) shapes, then subject has to recognize the presented shape among the five. EEG brain signals were captured during the ten minutes time of tests. Kernel Density Estimation (KDE) algorithm was applied to the acquired signals to extract the features, for emotion recognition Multi-layer perceptron (MLP) algorithm used for classifying the extracted features. The results were in terms of point coordinates to be applied for valence and arousal model. The output of MLP was normalized to be (1,1), (-1,-1), (-1,1) (0,0) to represent emotions Happy, Sad, Fear and Calm respectively.

5. Emotion recognition through experimental study For early detection of dyslexic pre-school children, we have established an experiment to analyze and understand the behavior of dyslexic children through affective space model. We describe the experiment in this paper not to talk about dyslexia but to suggest that it is possible to measure long term memory using the same approach.

6. Results and discussion Results of this experiment can be discussed from several points of view. In this paper we discussed the findings that related to memory test and the impact of emotion on memory. As described in fig.3 (a, b, c and d) results explain the movement (trajectory) of dynamic emotion during memory test for four subjects represented on affective space model. The two-dimensional valance-arousal model is divided into four quadrants as in Fig. 2 to map normalized values of emotion [10].

This figure shows that emotion trajectory located in quadrant (-1,1), which represents emotion of Fear and Angry. 2 1.5 1 0.5 0 -1.5 -1 -0.5 -0.5 0 -1 Fig.3(b): subject -2. 0.5

This emotion path started at area of (-1,1) and kept changing between (-1,1) and (-1,-1) to represent emotion trajectory from angry to sad. 1 0 -1 -0.5 -1 -2 Fig.2: The two-dimensional valance-arousal model Since the test was word recognition, the dynamic emotion path was affected by the operation of information retrieval. By applying the same approach during the information acquisition (encoding) and information retrieval for emotional status, the influence of emotion on long term memory can be measured through affect. These preliminary results can be used in future work as a base to establish an experiment for explicit and implicit memory tests. Fig. 3(c): subject -3. Although the emotion path started at area of Happy, it has changed to area of (-1,-1) and (-1,1) to represents the change of emotion due to subjects status during the test. 0.8 0.6 0.4 0.2 1.5 1 Fig 3. (d): subject -4. 0.5 0 -1 -0.5 -0.5 Fig. 3(a): subject -1. 0 Emotion trajectory for this subject started and remained in the area of (1,1), which reflected emotion of happy. 0 0 0.1 0.2 0.3 0.4 0.5 0 0.5 1

Conclusion The primary findings of this work showed that memory effectiveness is notably influenced by emotions, which means that implicit and explicit memory tests can be carried out using the same experimental approach. According to the assumption that explicit memory has its own system which is different of the implicit memory system, in future work each system should be tested separately from the other. 7. References [1] Atkinson, R. , Shriffin, R.,“Storage and retrieval processes in Long Term Memory”, Stanford, 1971. [2] Baddeley, A., “Psychology of memory”, John Wiley & sons, ISBN 0-470-09141-X, 2004. [3] Baddely, A.; ”Working memory: Theories, Models, and controversies”, Annual reviews of psychology,63, 2012. [4] Berry, C. et al.,”Can ‘pure’ implicit memory be isolated? a test of a single-system model of recognition and repetition priming”, Canadian Journal of experimental psychology, Vol. 64, No. 4, 241-255, 2010. [5] Cleeremans, A. and Dienes, Z., “Computational models if implicit learning”, Cambridge handbook of computational psychology, 396-421, 2008. [6] Deines, Z.,”Conscious versus unconscious learning of structure”, Statistical learning and language acquisition, Mouton de Gruyter publisher, pp. 337-364, 2012. [7] Ericsson, K. and Kintsche, K.,” Long-Term Working Memory”, Psychological Review, Vol 102(2), Apr 1995, 211-245. [8] Frank, J.& C. Tomaz, “Enhancement of declarative memory associated with emotional content in Brazilian sample”, Brazilian Journal of medical and biological research, 33:1483-1489, 2000. [9] Hulstijn, J. , “Theoretical and empirical issues in the study of implicit and explicit second language learning”, SSLA, 27, 129-140, USA, 2005.

[10] Lewis, M. et al., “Handbook of emotion”, the Guilford press, London, 3rd ed., 2008. [11] Mayer, R. & Moreno, R., “Nine ways to reduce cognitive load in multimedia learning”, educational psychologist, 38:1,43-52,2003. [12] Ormrod, J., “Human learning”, 6th ed, Pearson, 2012.pp4. [13] Rajaram,S. and Roediger III, “Direct comparison of four implicit memory tests”, Journal of experimental psychology, vol. 19, no. 4, 765-776,1993. [14] Roediger, H.,” Implicit memory: retention without remembering”, American psychologist, 45, 1043-1056. 1990. [15] Scachter, Daniel L.,” Psychology”,2nd Ed., Madison avenue New York, NY 10010: Worth publisher, 2011. [16] Schacter, D., “Implicit memory: History and current status”, journal of experimental psychology, Vol. 13, No. 3, 501-518, 1987. [17] Squire, L. and Zola, ” Structure and function of declarative and non-declarative memory systems”, Proc. Natl. Acad. Sci. USA, Vol. 93, pp. 13515–13522, November 1996. [18] Taylor, L., “Relationship between affect and memory: motivation-based selective generation”, J Pers Soc Psychol., 62(5):876-82, may 1992. [19] Tulving, E., “Episodic and semantic memory”, EDS, Organization of memory, New York academic press, 381402, 1972. [20] Weiss, Ruth ”Memory and learning”, Training & Development, Vol. 54 Issue 10, p46, 5p, 2000. [21] Williams, John N., “Implicit learning in second language acquisition”, the new handbook of second language acquisition, 319-353,2009. [22] Zainudin A. et al, “Unconscious learning processes: mental integration of verbal and pictorial instructional materials”, SpringerPlus 2013.