IMPACT OF LITERACY ON VISUAL PROCESSING

Reading in the brain

1. The visual word form

area: myth or reality?

Stanislas Dehaene

Collège de France,
and
INSERM-CEA
Cognitive Neuroimaging Unit
NeuroSpin Center, Saclay, France
www.unicog.org
Beyond allowing the acquistion of new knowledge (through
reading) and the external storage of information (through
handwritten notes, books, computers, etc.), literacy induces three
main changes in the brain networks dedicated to language
• Indeed, literacy acquistion not only (i) allows spoken language areas to be activated by
written inputs, but also (ii) modifies the spoken language system itself through two
mechanisms. On the one hand, (ii.a) literacy improves phonological coding (in the planum
temporale) and, on the other hand (ii.b), it makes, in some listening situations,
orthographic representations to be activated top-down by spoken inputs (in the left
occipito-temporal cortex). In addition, (iii), literacy induces anatomical changes, including
in intra- and inter-hemispheric connectivity
ACTIVATION OF SPOKEN LANGUAGE AREAS
BY WRITTEN INPUTS
• The written words activate the cohort of brain areas dedicated to spoken language (with
the exception of the primary auditory cortex). These activations of the spoken language
network from the visual modality reach, in most regions, an intensity equivalent to that
evoked by hearing the spoken language and thus leads to a significant overlap of
activation of brain regions both by writing and by speech. This is not really surprising,
since the purpose of reading is precisely to restore the spoken language from vision,
giving us a sustainable external memory.
Deahene, S., Pegado, F., Braga, L. W., Ventura, P., Nunes, G., Jobert, A., Dehaene-Lambertz, G., Kolinsky, R., Morais, J., & Cohen, L. (2010).
How Learning to Read Changes the Cortical Networks for Vision and Language. Science, 330, 1359-1364.
CHANGES IN THE PROCESSING OF SPOKEN
LANGUAGE
Changes in the planum temporale: a modification of
phonological representations?
Deahene, S., Pegado, F., Braga, L. W., Ventura, P., Nunes, G., Jobert, A., Dehaene-Lambertz, G., Kolinsky, R., Morais, J., & Cohen, L. (2010).
How Learning to Read Changes the Cortical Networks for Vision and Language. Science, 330, 1359-1364.
• The PT is a region of the upper temporal cortex located just behind the Heschl gyrus,
which is involved in the coding of phonemes. This is a key region for the categorical
perception of speech which is also activated by silent lip-reading and is sensitive to the
congruence between a phoneme and the simultaneous visual presentation of a letter, an
effect which is absent in people with dyslexia.
• The increased activation of the PT with literacy may therefore reflect a refinement of
phonological representations: phoneme awareness (Morais et al., 1979 – PURSO 
URSO)
• More recently, a brain imaging study showed that in children, the activation of PT is
correlated not only with the level of reading, but also with vocabulary, verbal memory and
phoneme awareness [Monzalvo et al., 2012]
• So the increase in fMRI activation of PT observed in literate adults [Dehane et al., 2010;
Science] may reflect the change, during the reading acquisition process, of phonological
representations by orthographic knowledge and / or metaphonological knowledge
• It remains to be determined whether literacy affects not only phonological awareness, but
also the implicit phonological representations used in speech perception. In fact, illiterate
adults have a sophisticated implicit phonology: they discriminate, as well as literates,
syllables like / pa / and / ba / [Adrian, Alegria, & Morais., 1995]
• Like literates, they have an auditory lexicon phonologically restructured with an internal
encoding of spoken words modulated by the frequency of occurrence of words and the
number of phonologically close words.

Ventura, P., Kolinsky, R., Fernandes, S., Querido, L., & Morais, J. (2007). Lexical restructuring in the absence of literacy. Cognition, 105, 334-361.
Ventura, P., Kolinsky, R., Querido, L., Fernandes, S., & Morais, J. (2007). Is phonological encoding in naming influenced by literacy? Journal of Psycholinguistic Research,
36, 341-360.
• However, they may differ in the precision of phoneme category boundaries. Illiterates
have a categorical perception of speech: as literates, they negligentiate irrelevant
acoustic variations (intra-categorical) in the discrimination of syllables such as /ba / and
/da /. However, illiterate people displayed a less precise categorical boundary

Serniclaes, W., Ventura, P., Morais, J., & Kolinsky, R. (2005). Categorical perception of speech sounds in illiterate adults. Cognition, 98, B35-B44.
Activation of VWFA by spoken language: «top-down»
access to orthography
• These observations are consistent with many behavioral data showing that orthographic
representations are recruited even in purely auditory situations.
• This is true both in the metaphonological processing (Ventura, P., Kolinsky, R., Brito-Mendes, C., & Morais, J.
(2001). Mental representations of the syllable internal structure are influenced by orthography. Language and Cognitive Processes, 16, 393-
418); /bar/ /mel/; vs. /pez/ /fur/; and in the lexical-semantic processing. In lexical decision
tasks, especially, there is an effect of orthographic consistency, with faster responses to
words containing a rhyme that can be written in only a single way (eg, “ler" ) than to words
including a rhyme that can be written differently in other words of the language (eg, “mel”)
– (Ventura, P., Morais, J., Pattamadilok, C., & Kolinsky, R., (2004). The locus of the orthographic consistency effect in auditory word
recognition. Language and Cognitive Processes, 19, 57-95)
• Psycholinguists discuss the question of whether such effects indicate that orthographic
representations are activated in a "top-down" manner during the processing of spoken
words or if learning of reading has altered phonological representations. Brain imaging
studies suggest that the two phenomena coexist.
• The increase in fMRI activation in the PT of literate compared to illiterate adults [Dehaene
et al., 2010; Science], as well as data from electroencephalography techniques [Perre et
al., 2009] and transcranial magnetic stimulation [Pattamadilok et al., 2010] showing that
phonological areas (eg, the supramarginal gyrus left) are the source of the effects of
orthographic consistency, support the idea that phonological coding is improved by
literacy
• The activation of the VWFA suggests an additional recruitment of orthographic
representations which is observed only when access to the orthographic code is useful for
the task, for example in situations of auditory lexical decision, and not during passive
listening to words [Dehaene et al., 2010; Science]; more recent data suggest a
recruitment of orthographic representations even during passive listening
• The links between spelling and phonology also vary with age. Both behavioral and brain
imaging studies showed that the effects of spelling are more widespread among children
during the early years of learning to read, compared with adults [ behavioral: Ventura, P., Morais, J., &
Kolinsky, R. (2007). The development of the orthographic consistency effect in speech recognition. From sublexical to lexical involvement.
]. This likely reflects the intensive use of phoneme-grapheme and
Cognition, 105, 547-576.
grapheme-phoneme correspondences at this early stage of literacy, which would help
establish an active flow of information between orthographic and phonological
representations.
Anatomical changes in the circuits of spoken
language
Thiebaut de Schotten et al. (in press)
• The inter-hemispheric connectivity is also modified by literacy, with a thickening of the
splenium or isthmus of the corpus callosum [Carreiras et al., 2009; Castro-Caldas et al., 1999; Peterson et al.,
2007] . This would allow the improvement of inter-hemispheric transfer of phonological or
visual information.
• Note that, in addition, literacy leads to an increase in the density of gray matter in several
regions involved in reading, including the angular gyrus, the left supramarginal gyrus and
back of the superior temporal gyrus [Carreiras et al., 2009; Castro-Caldas et al., 1999; Peterson et al., 2007]
 USING REGRESSION WITHY READING ABILITY, WE
EVALUATED THE PRECISE MOMENT AT WHICH EVOKED
RESPONSES WERE MODULATED BY READING ABILITY

Pegado, F., Comerlato, E., Ventura, F., Jobert, A., Nakamura, K., Buiatti, M., Ventura, P., Dehaene-Lambertz, G., Kolinsky, R., Morais, J., Braga, L., Cohen, L., & Dehaene, S. (2014).
Timing the impact of literacy on visual processing. Proceedings of the National Academy of Sciences. In press.
 The ability to read correlates with enhanced early visual responses in
the post-P1 time window (~140-180ms)/enhanced visual responses in
direct proportion to the reading fluency of participants

Pegado, F., Comerlato, E., Ventura, F., Jobert, A., Nakamura, K., Buiatti, M., Ventura, P., Dehaene-Lambertz, G., Kolinsky, R., Morais, J., Braga, L., Cohen, L., & Dehaene, S.
(2014). Timing the impact of literacy on visual processing. Proceedings of the National Academy of Sciences. In press.
• Reading practice considerably increases electrophysiological responses at a later stage
of visual processing (post-N1, ~200-240ms)
 • Literacy induces enhanced left-lateralization of visual processing at the N1 (~176 ms)
stages over occipito-temporal region for letter strings and false fonts

• Increased activation in ventral occipito-temporal córtex, left-lateralized for strings and a

trend to right-lateralization for faces

Pegado, F., Comerlato, E., Ventura, F., Jobert, A., Nakamura, K., Buiatti, M., Ventura, P., Dehaene-Lambertz, G., Kolinsky, R., Morais, J., Braga, L., Cohen, L., & Dehaene, S. (2014).
Timing the impact of literacy on visual processing. Proceedings of the National Academy of Sciences. In press.
 • Reading acquistion leads to an enhancement of repetition suppression (evaluated by
subtracting evoked responses to pairs of “different” minus “identical” trials), indicating an
improved visual discrimination of exemplars within the same categories. The effect
occured over the left occipito-temporal region at an early stage of visual processing
(~100-150 ms)

Pegado, F., Comerlato, E., Ventura, F., Jobert, A., Nakamura, K., Buiatti, M., Ventura, P., Dehaene-Lambertz, G., Kolinsky, R., Morais, J., Braga, L., Cohen, L., & Dehaene, S. (2014).
Timing the impact of literacy on visual processing. Proceedings of the National Academy of Sciences. In press.
 • In order to test the influence of literacy on mirror invariance, we subtracted the evoked
potentials values for “mirror” minus “identical” pair trials. If the illiterate visual system
treated mirror images as identical, then this contrast should be small in illiterates and
should increase with literacy. Reading does indeed enhance mirror discrimination in the
left occipito-temporal region (~100-150ms)

Pegado, F., Comerlato, E., Ventura, F., Jobert, A., Nakamura, K., Buiatti, M., Ventura, P., Dehaene-Lambertz, G., Kolinsky, R., Morais, J., Braga, L., Cohen, L., & Dehaene, S. (2014).
Timing the impact of literacy on visual processing. Proceedings of the National Academy of Sciences. In press.
PLAUT & BHERMANN (2013)
 Gabay, Dundas, Plaut, & Behrmann (2017)

• Vários dados apontam para a ideia de que as palavras e as

faces são reconhecidas por mecanismos independentes:
palavras na Visual Word Form Area (VWFA) no VOT do
hemisfério esquerdo, enquanto que faces na Fusiform Face
Area (FFA); numa região aproximadamente homóloga no
hemisfério direito.
• Plaut & Behrmann consideram que estes 2 domínios são
interdependentes, estruturalmente e funcionalmente
• Indivíduos com alexia apresentam problemas no
reconhecimento de faces
• Indivíduos com prosopagnoisa apresentam problemas no
reconhecimento de palavras
• Neste estudo: o que acontece ao processamento de faces em
indivíduos adultos com dislexia?
EVA DUNDAS PHD

Eva Dundas David Plaut Marlene Behrmann

EXPERT FACE PROCESSING
 Holistic/configurational style has been identified as a hallmark of expertise in the
perception of faces (Farah, Wilson, Drain, & Tanaka, 1998; Maurer, Le Grand, & Mondloch, 2002; Young, Helawell, & Hay, 1987),
involving relatively little part-based decomposition but instead a strong integration
among face parts.
 Objects are typically identified at the category level (e.g.,“dog”; Rosch, Mervis, Gray, Johnson, & Boyes-
Braem, 1976). However, for faces, it is often necessary to identify it at the level of the
individual (e.g.,“Bob”) rather than detect a face or identify a face at the more general level
of race or gender.
THE PROBLEM WITH FACES

 [Humpty Dumpty to Alice]

"You’re so like other
people…the two eyes
(marking their place in the
air with his thumb), nose in
the middle, mouth under. It’s
always the same. Now if
you had the two eyes on the
same side of the nose, for
instance – or the mouth at
the top – that would be
some help"
THE MOST PROMINENT DEMONSTRATION OF HOLISTIC FACE
PROCESSING IS THE COMPOSITE ILLUSION

From Rossion (2003)

THE MOST PROMINENT DEMONSTRATION OF HOLISTIC FACE
PROCESSING IS THE COMPOSITE ILLUSION

From Rossion (2003)

HP IN FACES
• Sequential same-diferent matching task (classic in the face domain)
• Is the bottom-part of S2 the same as in S1?

“different”

“same”

From Ventura, Fernandes, Cohen, et al. (2013)

• Manipulation of alignment and congruence of the up/bottom-parts

• HP – stronger congruence effect (CE) in the aligned vs. misaligned condition
FROM FACES TO OBJECTS OF VISUAL EXPERTISE
• The same paradigm has been applied to the recognition of many other non-face objects,
such as cars (Gauthier, Curran, Curby, & Collins, 2003) , chess (Boggan et al., 2012), fingerprints (Busey & Vanderkolk,
2005) , greebles (an artificial stimulus type, Gauthier & Tarr, 1997), musical notation (Yong & Gauthier, 2010) , etc and
holistic processing has been found to indicate perceptual expertise with these objects.

It appears that highly familiar multipart objects of

expertise that co-opt fusiform face-area circuitry
become characterized by behavior hallmarks of face
processing (e.g., holistic processing; e.g. Parkinson
& Wheatley, 2015
EXPERT WORD PERCEPTION AND HP

 In contrast to expert face and object perception, the relationship between expert word
perception and holistic processing has been more controversial, with some evidence
suggesting that
 word reading is primarily part-based,
 other evidence suggesting that it is primarily holistic,
 and some that it relies on the interplay of both holistic and
part-based processing.
PART-BASED PROCESSING OF WORDS

 In terms of task demand, recognition of words and letters/characters requires basic-level

categorization (A. C.-N. Wong & Gauthier, 2007; Zhang & Cottrell, 2004) .
 For all writing systems, words differ from each other in terms of the number and identity
of their constituent parts (e.g., how many and what letters there are and their order in a
word).
 The detailed spatial relationships between parts (e.g., the distance between letters) seem
less important for identifying a word.
 Text also exhibits a different pattern of neural selectivity compared with faces and other
objects of expertise.
From Wong & Gauthier (2007)
HOLISTIC PROCESSING OF WORDS

 Different paradigms have been used to examine if words are processed holistically, and
results are mixed.
• The famous word superiority effect highlights the role of whole-word representations in the
identification of letters (Reicher, 1969; Wheeler, 1970) .
• Word identification can sometimes even bypass recognition of its constituent letters (Jordan, Thomas, &
Scott-Brown, 1999; Perea, Dunabeitia, & Carreiras, 2008).

• Word shape processing has also been shown to play an important role in efficient reading (Grainger &
Whitney, 2004; Osswald, Humphreys, & Olson, 2002; Pelli & Tillman, 2007).
 Since the majority of studies examining holistic
processing of text have used paradigms different from
those used for faces and other objects of subordinate
level expertise, it is difficult to make a direct comparison
across domains.
HP IN VISUAL WORDS (WONG, BUKACH, YUEN, ET
AL., 2011)
• Sequential same-diferent matching task
• Is the left-part of S2 the same as in S1?

Manipulation of alignment and of congruence

HP IN VISUAL WORDS (WONG, BUKACH, YUEN, ET
AL., 2011)
• Sequential same-diferent matching task
• Is the left-part of S2 the same as in S1?

• HP – stronger congruence effect (CE) in the aligned vs. misaligned condition, i.e.
• Alignment x Congruence interaction
VENTURA ET AL. (2017)
THE LOCUS OF HOLISTIC PROCESSING OF VISUAL
WORDS

 Although the composite paradigm has been used on faces

and words and holistic processing seems to be relevant in
both domains, the locus of holistic processing of words is
still unclear. In the present study, we examined it.
PORTUGUESE MATERIAL (EXAMPLES); CV.CV
PORTUGUESE WORDS

Congruent Incongruent
study test study test
same bife bife bife bico
different bife saco bife safe
 COURIER

400 ms

1000 ms

 2500 ms
 NOTERA

400 ms

1000 ms

 2500 ms
 MIXED CASE

400 ms

1000 ms

 2500 ms
TRABALHO REALIZADO ESTE
SEMESTRE
• Existe um efeito holísitco para palavras escritas e esse efeito não depende de
características superficiais da palavra – depende de representações lexicais abstractas
• Quisemos avaliar se esse processamento holístico tem um papel de facto funcional na
leitura de palavras
• Tarefa compósita com palavras courier
• Tarefa de decisão lexical (palavras e pseudo-palavras que variavam no número de letras
e na frequência; no caso de pseudo-palavras, frequência das base words))
• Podemos extrair um índices de acesso ao léxico na prova de decisão lexical: efeito de
frequência (diferença entre RT para palavras pouco frequentes e RT para palavras muito
frequentes; menor efeito de frequência indica acesso mais eficaz ao léxico
• Se o processamento holísitco tem um papel funcional na leitura de palavras, então o
tamanho do efeito compósito deve correlacionar-se negativamente com o efeito de
frequência; VERIFICADO
RICHLER ET AL. (2009)
RICHLER ET AL. (2009)
• Maior efeito de congruência para ziggerins quando precedidos de palavra alinhada
(tratada holisticamente) do que quando precedidos de palavra desalinhada (não tratada
holisticamente)
ALI-MIS*CONG-INC; LS Means
Current effect: F(1, 47)=4,0499, p=,04992
Effective hypothesis decomposition
Vertical bars denote 0,95 confidence intervals
4,5

4,0

3,5

3,0
DV_1

2,5

2,0

1,5

1,0 CONG-INC
1 2 1
CONG-INC
ALI-MIS 2