Structure, Meaning, and Constituency in

Visual Narrative Comprehension

A dissertation

W
submitted by

Neil Thomas Cohn

IE
In partial fulfillment of the requirements

for the degree of

EV

Doctor of Philosophy

in
PR

Psychology

TUFTS UNIVERSITY
May

2012

© 2012, Neil Cohn

Advisors: Ray Jackendoff, Gina Kuperberg, Phillip Holcomb

 UMI Number: 3512103

All rights reserved

INFORMATION TO ALL USERS

The quality of this reproduction is dependent on the quality of the copy submitted.

In the unlikely event that the author did not send a complete manuscript
and there are missing pages, these will be noted. Also, if material had to be removed,
a note will indicate the deletion.

W
IE
UMI 3512103
Copyright 2012 by ProQuest LLC.
EV
All rights reserved. This edition of the work is protected against
unauthorized copying under Title 17, United States Code.
PR

ProQuest LLC.
789 East Eisenhower Parkway
P.O. Box 1346
Ann Arbor, MI 48106 - 1346
Structure, Meaning, and Constituency in Visual Narrative Comprehension

W
IE
EV
PR

ii
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

Abstract

Narrative has been formally studied for at least two millennia, dating back

to the writings of Aristotle. While most theories began by describing the

construction of plotlines in theatre, most contemporary research on the structure

and comprehension of narratives has examined the discourse of spoken language.

However, visual narratives in the form of sequential images have also been

pervasive throughout history, whether drawn on cave walls, painted on pottery, or

W
printed in contemporary comic books and strips. Yet, compared with the study of

discourse in verbal language, the study of sequential image comprehension has

IE
been relatively impoverished. Just what are the structures motivating visual

narratives and how are they processed?

EV

This thesis will explore this question using experiments guided by an

overall theory that sequential images at the narrative level are structured and
PR

processed analogously to sequences of words at the sentence level. The main idea

is that a narrative “grammar” organizes the structure of sequential images in the

visual language used in comics, similar to the way that syntax organizes words

into coherent sentences. We focus here on two salient parts of this analogy. First,

this thesis will explore the idea that visual narrative comprehension involves a

system of narrative structure and a system of semantic coherence that contribute

to comprehension. This correspondence is akin to the interaction between syntax

and semantics at the sentence level. This hypothesis will be examined in an

experiment in Chapter 1. Second, it explores the idea that narrative structure is a

iii
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

hierarchic system that organizes images into groupings of constituents, analogous

to the phrase structures of syntax in sentences. This aspect of narrative structure

will be explored in two experiments in Chapter 2. In conclusion, Closing Remarks

will briefly discuss the overall implications for the analogy between narrative

structure in sequential images and syntax in sentences.

W
IE
EV
PR

iv
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

Acknowledgments

I would first like to thank my advisors, Ray Jackendoff, Gina Kuperberg,

and Phil Holcomb for believing in a research project that had yet to find a place in

the broader studies of human cognition. My parents, Leigh and Lindsey Cohn,

and my brother, Charlie Cohn, have been immensely supportive throughout my

years pursuing this work, as have my friends and colleagues, Laura Babbitt, Ian

Christian Blanche, Tali Ditman, Steve Maher, Ted M. Mason, Kathy Midgley,

Priya Mitra, Kana Okano, John Pacheco, Martin Paczynski, Agatha Trindade, Eva

W
Wittenberg, and Cherry Yum. Finally, my cousins Elizabeth Hall and Christian,

Henry and Emlen Lease are thanked for providing a place to set research aside, as
IE
are my students in the Tufts Soo Bahk Do Club: John Sotherland, Thomas Nikiper,

Martin Paczynski, Jason Rosenbaum, Negin Toosi, Madeline Gardner, Eric Sinski,
EV

and Matthew Cardarelli.

More direct to the experiments in this dissertation, Chelsey Ott, Patrick

PR

Bender, and Adrienne Ing are thanked for helping with making stimuli and

running the experiment in Chapter 1. Suzi Grossman, Chelsey Ott, and Patrick

Bender are thanked for helping to make stimuli and running the experiments in

Chapter 2. Fantagraphics Books is thanked for their generous donation of The

Complete Peanuts. This work was supported by NIMH (R01 MH071635),

NICHD (HD25889) and NARSAD (with the Sidney Baer Trust), as well as

funding from the Tufts Center for Cognitive Studies.

v
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

Table of Contents

Abstract iii  
Acknowledgments v  
Table of Contents vi  
List of Tables ix  
List of Figures ix  
Preface 1  
Chapter 1: Structure and Meaning in Visual Narrative Comprehension 4  
Semantic approaches 4  
Verbal narrative 4  
Visual narrative 5  
Structural Approaches 6  

W
Verbal narrative 7  
Interface of semantics and structure in verbal narrative 8  
Narrative structure in sequential images
IE 9  
Structure and semantics in processing 13  
ERPs in language processing 14  
ERPs in the visual-graphic domain 17  
Processing semantics and structure in visual narrative 19  
EV
Outstanding questions 21  
Methods 27  
Stimuli 27  
Objective measures of coherence 30  
PR

Rating Study 1: Complete sequences 31  

Rating Study 2: Cropped sequences 32  
Creation of final stimulus set 32  
Participants in the ERP/Self-paced viewing study 34  
Procedure 34  
ERP Recordings 35  
Behavioral Data Analysis 35  
ERP Data Analysis 37  
Results 37  
Behavioral results 38  
Ratings 38  
Viewing times 39  
ERP results 41  
Critical Panel 41  
Critical Panel +1 45  
Discussion 50  
Interaction between narrative structure and semantics 56  

vi
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

Prototypical and non-prototypical correspondences between narrative

structure and semantics 59  
Conclusion 62  
Chapter 2: Constituency in Visual Narrative Comprehension 63  
Linear coherence relationships in verbal and visual narrative 64  
Constituent structure in verbal narrative 65  
Constituent Structure in visual narrative 66  
Experiment 1: Self-paced viewing 71  
Methods 77  
Construction and Norming of Stimuli 78  
Segmentation study: Verification of narrative boundaries 78  
Coherence relationships between pairs of panels Before, At and After the
narrative boundary 81  
Construction of stimulus set for self-paced comic viewing experiment 83  
Self-paced viewing experiment 85  

W
Participants 85  
Procedure 85  
Data Analysis 86  
Coherence ratings 86  
Viewing times
IE 86  
Results 90  
Coherence ratings 90  
EV
Viewing times 91  
Disruptions vs. No-Disruptions 91  
Effect of crossing a narrative constituent boundary 92  
Effect of Disruption later in a sequence 93  
PR

Effect of Disruption panels 94  

Experiment 1 Discussion 95  
Disruption vs. No-Disruption 95  
Effect of crossing a narrative constituent boundary 96  
Effect of Disruption later in a sequence 98  
Effect of Disruption panels 99  
Conclusions 100  
Experiment 2: Event-Related Potentials 101  
ERPs in verbal language 101  
ERPs in other domains 104  
Open questions and the present study 107  
Methods 109  
Participants 109  
Stimuli 110  
Procedure 110  
ERP Recordings 111  
Behavioral Data Analysis 111  

vii
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

ERP Data Analysis 111  

Results 112  
Ratings 112  
ERP Results 113  
Experiment 2: Discussion 116  
General Discussion 121  
Additional questions 124  
Broader implications 126  
Conclusion 128  
Closing Remarks 129  
Graphic References 132  
References 132  

W
IE
EV
PR

viii
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

List of Tables

Table 1. Results of ANOVAs at the Critical Panel. 42

Table 2. Results of ANOVAs at Critical Panel +1. 49

Table 3. Results of ANOVAs comparing each disruption panel. 116

List of Figures

W
Figure 1. Comic strip with prototypical narrative categories. 10

Figure 2. Comic strip with non-prototypical correspondences of narrative to

IE
meaning. 11
EV
Figure 3. Image sequences that manipulate the contributions of semantic

coherence and narrative structure in a critical panel. 22

Figure 4. Electrode montage for analysis of ERP data. 36

PR

Figure 5. Mean ratings of coherence for visual sequences. 38

Figure 6. Mean viewing times to Critical Panels and subsequent panel

(Critical Panel+1). 40

Figure 7. Waveforms for Critical Panels. 41

Figure 8. Waveforms for Critical Panels +1. 47

Figure 9. Sequential images involving an embedded constituent. 61

Figure 10. Constituent structure in a visual narrative 69

Figure 11. Various constituent structure patterns in visual narratives 80

Figure 12. Mean rates of linear coherence changes 83

ix
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

Figure 13. Sequence types with disruption panels. 89

Figure 14. Mean viewing times to sequences types. 92

Figure 15. Mean viewing times to disruption panels. 94

Figure 16. Waveforms comparing amplitudes of disruption panels. 114

W
IE
EV
PR

x
Structure, Meaning, and Constituency in Visual Narrative Comprehension

Structure, Meaning, and Constituency in

Visual Narrative Comprehension

W
IE
EV
PR

1
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

Preface

Narrative has been formally studied for at least two millennia, dating back

to the writings of Aristotle. While most theories began by describing the

construction of plotlines in theatre, most contemporary research on the structure

and comprehension of narratives has examined the discourse of spoken language.

However, visual narratives in the form of sequential images have also been

pervasive throughout history, whether drawn on cave walls, painted on pottery, or

printed in contemporary comic books and strips (Kunzle, 1973; McCloud, 1993).

W
Yet, compared with the study of discourse in verbal language, the study of

sequential image comprehension has been relatively impoverished. Just what are
IE
the structures motivating visual narratives and how are they processed?

This thesis will explore this question using experiments guided by an

EV

overall theory that sequential images at the narrative level are structured and

processed analogously to sequences of words at the sentence level. The main idea
PR

is that a narrative “grammar” organizes the structure of sequential images in the

visual language used in comics, similar to the way that syntax organizes words

into coherent sentences. We focus here on two salient parts of this analogy. First,

sequential images at a narrative level involve the interaction between two

independent components of narrative structure and semantic coherence, analogous

to the interaction between syntactic structure and semantics at the sentence level.

Second, like syntactic structure, this narrative structure hierarchically organizes

panels into constituents.

2
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

Outright, it is important to stress that this is an analogy only, and that

words in sentences and images in visual narratives differ prominently. Foremost,

images convey far more information than words, perhaps even at the level of a

whole sentence or more. Because of this, the narrative structure guiding

sequential images may be the same structure guiding sequential sentences in

verbal discourse, and this similarity will be explored throughout. Nevertheless,

the analogy between visual narrative and sentences serves as a useful framework

to inform and interpret the study of sequential image comprehension, both for

W
their similarities and their differences.

First, this thesis will explore the idea that visual narrative comprehension
IE
involves a system of narrative structure and a system of semantic coherence that

contribute to comprehension. This correspondence is akin to the interaction

EV
between syntax and semantics at the sentence level. This hypothesis will be

examined in an experiment in Chapter 1. Second, it explores the idea that

PR

narrative structure is a hierarchic system that organizes images into groupings of

constituents, analogous to the phrase structures of syntax in sentences. This aspect

of narrative structure will be explored in two experiments in Chapter 2.

In conclusion, Closing Remarks will briefly discuss the overall

implications for the analogy between narrative structure in sequential images and

syntax in sentences.

3
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

Chapter 1: Structure and Meaning in Visual Narrative Comprehension

A prominent line of research on language has examined the interaction

between structure and meaning. At the sentence level, theories have debated

whether sentences are guided more by syntax (e.g., Chomsky, 1957, 1965, 1995),

by semantics (e.g., Lakoff, 1971; Langacker, 1987), or through a combination of

both (e.g., Culicover & Jackendoff, 2005; Jackendoff, 2002; Sadock, 1991).

Similar distinctions have been drawn beyond the level of individual sentences,

between narrative and meaning. While this distinction at the narrative level has

W
mostly been articulated for verbal discourse, we have argued that an analogous

relationship occurs between structure and meaning in the visual domain, most
IE
prominently in the sequential images used in comics (Cohn, In Press; Cohn,

Paczynski, Jackendoff, Holcomb, & Kuperberg, 2012). Below, we review

EV

approaches to verbal and visual narrative that focus on semantics, structure, or

their combination. We then introduce a new theory of narrative structure in

PR

sequential images that argues that the comprehension of visual narratives, as

found in comics, relies on the interaction between independent components of

structure and meaning.

Semantic approaches

Verbal narrative

Semantic approaches to discourse emphasize the establishment of

“coherence” across sentences or a broader discourse—the continuity that allows

an utterance or discourse to be meaningful (Halliday & Hasan, 1976). Coherence

4
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

can be established across sentences using a global semantic field or script

(Bransford & Johnson, 1972; Brown & Yule, 1983; Schank & Abelson, 1977; van

Dijk, 1977) or through local connections between each sentence (Halliday &

Hasan, 1976; Hobbs, 1985; Kehler, 2002; Mann & Thompson, 1987). On the

local sentence-to-sentence level, these different semantic relationships include

dimensions of reference (Haviland & Clark, 1974), temporal and event structure

(Mandler, 1986; Speer & Zacks, 2005; Zwaan, 1996), causation (Black & Bower,

1979; Keenan, Baillet, & Brown, 1984; Trabasso, Secco, & van den Broek, 1984),

W
spatial structure (Clark, 1972; Linde & Labov, 1975; Morrow, Greenspan, &

Bower, 1987), and intentionality (Lichtenstein & Brewer, 1980; Schank &
IE
Abelson, 1977). While reading a discourse, readers integrate these domains

together to construct a “situation model”—a mental representation of the situation

EV
described in the discourse (van Dijk & Kintsch, 1983). When readers progress

through a discourse, they continually update their situation model with new
PR

information by mapping coherence relations between sentences (Gernsbacher,

1990). When changes occur to any of the coherence domains—such as a shift in a

discourse from one spatial location to another—readers must update their

situation model with this new information, incurring a cost in processing (Zwaan,

Langston, & Graesser, 1995; Zwaan & Radvansky, 1998).

Visual narrative

Theories of visual narratives have looked at the same types of linear

coherence relationships as theories of verbal discourse. For example, “transitions”

between “panels”—the encapsulated units of images in comics—have also been

5
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

characterized across dimensions of time, causation, characters, environments, and

scenes (Cohn, 2003; McCloud, 1993; Saraceni, 2000, 2001; Stainbrook, 2003).

These transitions allow readers to inferentially “fill in the gaps” between adjacent

panels. Greater discontinuity between panels requires more inference, such as

when an incoming panel introduces new information or does not share the

semantic field of a previous panel (Saraceni, 2000, 2001). While coherence

relations have not been directly tested in experimental settings on static visual

narratives, some evidence for these coherence relationships have appeared in

W
studies of film. While watching a film, viewers are consciously able to identify

changes in time, characters, and spatial location (Magliano, Miller, & Zwaan,
IE
2001; Zacks, Speer, & Reynolds, 2009), and are the most sensitive to changes

between full scenes—where film shots shift from one action to another at
EV
different times and spaces (Magliano & Zacks, 2011). Additionally, viewers can

make predictions about subsequent events in a film based on semantic cues

PR

between shots and by drawing on global scripts about film storytelling (Magliano,

Dijkstra, & Zwaan, 1996). Thus, like in verbal discourse, viewers of films draw

upon semantic relationships at both a local and global level.

Structural Approaches

There is also substantial evidence that narratives extend beyond semantic

relationships between and across units. Some of the earliest theories of narrative

posited that narrative units play roles relating to a global structure. Dating back to

Aristotle’s “Beginning–Middle–End” pattern for theatre (Butcher, 1902), classical

approaches agreed that narratives are organized into a basic canonical pattern

6
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

(Freytag, 1894; Yamazaki, 1984), an observation that has permeated subsequent

research in both verbal and visual narratives.

Verbal narrative

More recent structural theories of narrative drew inspiration from the

linguistic theories of generative grammar at the syntactic level (e.g. Chomsky,

1957, 1965). Some approaches in this vein have modeled the structure of

conversational discourse (Clark, 1996; Hinds, 1976; Labov, 1972; Labov &

Waletzky, 1967). However, the generative theories of narrative that gained the

W
most attention were “story grammars” from the late 1970s. These models

hypothesized that hierarchic story structures were based around characters’

IE
achievements of goals (e.g. Mandler & Johnson, 1977; Rumelhart, 1975; Stein &

Glenn, 1979; Thorndyke, 1977). Story grammar research found considerable

EV

evidence that participants do prefer stories organized into canonical orders. Much

of this research used memory paradigms that asked participants to recall aspects
PR

of stories that they just read. On the whole, participants were better able to recall

the order of events in a story when it was presented in canonical order than when

the sentences of the story were inverted (Mandler, 1978, 1984; Mandler &

DeForest, 1979), taken out of their temporal order (Mandler & Johnson, 1977), or

fully scrambled (Mandler, 1984). In fact, a correlation was found between the

accuracy of recalling a story and the degree to which it was altered away from the

canonical order (Stein & Nezworski, 1978). These studies supported that readers

prefer certain canonical narrative patterns that extend beyond the scope of general

semantic relationships between sentences.

7
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

Interface of semantics and structure in verbal narrative

Though story grammar research provided evidence that readers draw upon

a global structure in their comprehension of texts, several of these models failed

to provide a clear distinction between semantic and structural information. While

story grammars set out to describe structural aspects of narratives, many critics

argued that their categories simply described semantic relationships at a global

scale (Black & Wilensky, 1979; de Beaugrande, 1982). Indeed, the goal-directed

focus of story grammar categories directly described semantic notions of

W
intentions and script-based events, just applied to stories (Bower, Black, & Turner,

1979; Lichtenstein & Brewer, 1980; Schank & Abelson, 1977). The use of scripts
IE
in particular could have been a confounding factor, because order of events in a

script could have been driving the effects of the narrative. For example, if a story
EV

rearranges the order of events describing a person dining at a restaurant, a

semantic “restaurant script” (including the routine of dining and the elements
PR

involved) may have more of an impact on the recall of those events than the

information they extract from the narrative (Ohtsuka & Brewer, 1992).

Some researchers attempted to overcome these confounds by more clearly

distinguishing between the presentation of a narrative and the underlying events

(Brewer & Lichtenstein, 1981, 1982; Johnson-Laird, 1983; van Dijk, 1977), an

idea with precedents in earlier theories of narrative (Chatman, 1978; Genette,

1980; Tomashevsky, 1965). In particular, the work of Brewer and colleagues

(Brewer, 1985; Brewer & Lichtenstein, 1981, 1982; Ohtsuka & Brewer, 1992)

emphasized that this separation allows for different mappings between narrative

8
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

and events. First, narratives can evoke different affective states in a reader based

on how much information (events) is provided or withheld (narrative). They

found that participants reported different affective responses to reading story

(surprise, suspense, and curiosity) depending on what information was omitted or

provided by a narrative (Brewer & Lichtenstein, 1981). Second, depictions of

time require a separation of narrative and events, or else flashbacks would

actually show time travel, instead of merely presenting events out of order from

their actual instantiation. Brewer and colleagues found that participants are more

W
accurate at comprehending stories where the order of events is isomorphic to its

presentation than when events are presented backwards, as flashbacks, as

IE
embedded inside each other, or as flashforwards (Ohtsuka & Brewer, 1992).

Together, these studies provided evidence that the understanding of events is

EV
separate from their presentation in narrative.

Narrative structure in sequential images

PR

Brewer and colleagues’ work on narrative stressed that discourse

comprehension arises not just from semantic relationships or an overall structure,

but from their interaction. Despite this insight, their research did not fully posit

just what the structure of narrative itself comprises outside of this interaction.

What would a formal model of narrative structure look like that also interfaces

with semantics?

Recently, Cohn (In Press) has proposed a theory of narrative structure for

sequential images that makes a clear division between narrative structure and

meaning. In this model, individual panels have categorical roles that are

9
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

structured hierarchically by general recursive rules. While the characteristics of

this hierarchic structure will be addressed in depth in other studies (see Chapter 2),

here we will focus on narrative categories and the relationship between narrative

(structure) and semantic (meaning) structures.

NS: [Arc Establisher Initial Peak Release ]

W
ES: [Event {Prep HOLD(BALL){Prep REACH}} {Head THROW} {Coda LOOK}]

Figure 1. Comic strip with prototypical narrative categories. (NS=Narrative

Structure, ES=Event structure).

IE
EV

In this model, prototypical narrative categories correspond to aspects of

events, as in Figure 1. A narrative sequence often begins with an Establisher,

PR

which sets up the characters and interaction involved in the sequence. As in the

first panel, where the boy passively holds a bowling ball, these panels most often

depict passive states, since they function to simply introduce the characters. Next,

an Initial panel begins the actions of the sequence—prototypically depicting

preparatory actions (such as the preparation to roll the bowling ball). The

narrative climaxes in the Peak, showing the main events of the sequence. In

Figure 1, the Peak is the penultimate panel, where the boy throws the ball. A

Release then shows the aftermath or coda of the event in the Peak, as in the final

panel of the sequence, where the ball stops rolling (to comedic effect).

10
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

Together, this ordering of categories forms a canonical narrative arc. In

this case, narrative categories map directly to events. Discourse theories have

proposed that prototypical mappings like this should be easier to comprehend than

narratives that are not isomorphic to the events they describe (Enkvist, 1981;

Ohtsuka & Brewer, 1992). However, correspondences between events and

narrative are not always prototypically mapped, and it is important to stress that

narrative categories are influenced by a panel’s content and its context in a

sequence. According to Cohn’s theory, in prototypical cases, the intrinsic

W
semantic characteristics shown in a panel provide cues to its narrative category

(such as Initial = preparation). In non-prototypical cases though, context may play

IE
a greater role in determining a panel’s role in the narrative structure.
EV
PR

NS: [Arc Establisher Initial Peak Release]

ES: [Event {Prep {Prep SURPRISE} {Process CHASE}}{Intercept BE HIT}{Coda HIDE}]

Figure 2. Comic strip with non-prototypical correspondences of narrative to

meaning (Schulz 1951). (NS=Narrative Structure, ES=Event structure).

Consider Figure 2, which depicts Snoopy chasing a hockey puck on ice,

only to be run over by the boys playing hockey. This sequence does follow the

canonical narrative pattern progressing from an Establisher to an Initial,

climaxing in a Peak, and then ending with a Release. In this case, the semantics of
11
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

each panel do not match the prototypical correspondences for narrative categories

described above. At the start, the Establisher in this sequence does not show a

passive state. This panel depicts a puck in the process of sliding and Snoopy

surprised by it, but it still does functionally set up the interaction between them.

Next, the Initial is not a preparatory action—it shows Snoopy already engaged in

the process of chasing the puck. This is not necessarily an “initiating” event, but it

acts as a narrative Initial because of its relationship to the Peak. The subsequent

Peak panel does reach a climax of the narrative, but it does not show the

W
culmination of Snoopy’s actions that are prepared by the Initial. Rather, the boys

interrupt Snoopy’s actions, a different type of culmination. Finally, the Release

IE
shows a reaction to the events in the Peak. This depicts an entirely new event of

hiding, not, for example, Snoopy being run over on the ground as direct the
EV
consequence of the Peak’s actions. In this way, narrative categories do not always

map to prototypical events, and both content and context factor into the
PR

recognition of narrative roles.1

Thus, this approach allows for narrative structure and semantics to operate

on different levels, with varying degrees of prototypicality in the correspondences

between them. Given this theoretical framework, how might we find evidence for

such separation between structure and semantics in the processing of visual

narratives?

1
It is also worth noting that this sequence does not involve goal-directed actions. Is Snoopy’s goal

12
 Structure, Meaning, and Constituency in Visual Narrative Comprehension

Structure and semantics in processing

We have drawn a broad analogy between the interaction of narrative

structure and semantics in models of narrative with the interaction of syntactic

structure and semantics at the level of sentences. It is important to stress though

that the nature and constraints of narrative structure in sequential images and of

syntactic structure in sentences differ substantially, especially given that images

contain more information than words. Indeed, images are closer to conveying the

amount of information found in clauses or sentences of a discourse. Nevertheless,

W
in looking for ways to differentiate narrative structure from semantic structure in

the processing of narrative sequential images, it is useful to examine analogous

IE
work on the differentiation of syntactic and semantic structure at the sentence

level.
EV

Experiments looking for evidence of a separation between syntactic and

semantic structure at the sentence level extend back to the early days of
PR

psycholinguistics (e.g., Marks & Miller, 1964; Miller & Isard, 1963). Over the

past thirty years though, event-related potentials (ERPs) have emerged as an

insightful technique for examining the neurocognitive processing of sentences,

because they directly index the underlying neural processes with excellent

temporal resolution. Several ERP effects have been correlated with semantic and

structural aspects of processing. We first provide an overview of the ERP

components most associated with semantic and structural processing in language

at the sentence level. We then describe how similar ERP effects appear to

13