Multimodality Studies in International Contexts Co... - (6 A Toolkit For The Analysis of Modal Intensity Strategies Instructors... )

6 A Toolkit for the Analysis
of Modal Intensity
Strategies Instructors Use to Facilitate
Student Engagement in Learning
Chloë Grace Fogarty-Bourget, Jesse Pirini

and Natasha Artemeva
Student engagement can be broadly defined as “the direct, measurable in

volvement or participation in learning activities” (Gettinger & Walter,
2012, p. 653) and is regarded as key to students’ overall academic success
(Dunne & Owen, 2013). Student engagement is increasingly being viewed
as a social phenomenon generated by student involvement in classroom
activities and through interaction between students and instructors (e.g.,
Chi & Wylie, 2014). In line with this social perspective, we define student
engagement as a state of sustained mutual activity wherein participants (i.e.,
the students and instructor), through a series of communicative exchanges,
including instances of directed attention, involvement, and participation, co
operate toward accomplishing a shared task (Fogarty-Bourget, 2019).
Here, we focus on the embodied strategies that instructors use in an at
tempt to direct students’ attention to key content and to involve them in
the subject matter, or, in other words, to facilitate student engagement. The
purpose of the present research is to identify and analyze such strategies
rather than evaluate their effectiveness or students’ reactions to them. In
this study, we draw on concepts from Multimodal (Inter)action Analy
Copyright © 2023. Taylor & Francis Group. All rights reserved.
sis (MIA) (Norris, 2004, 2020) to analyze the embodied actions of the
instructors in two face-to-face teaching contexts: university mathematics
lectures and high school tutoring sessions in English literature.
We apply the concept of modal density (Norris, 2004) developed within
the framework of MIA to attend to levels of attention and awareness in so
cial interactions. Modal density (see Bernad-Mechó, this volume) refers to
both how complex and intense an action is, recognizing that more complex
and intense actions are loci of attention. As we discuss below, processes for
analyzing the complexity of actions are well defined, but processes for ana
lyzing their intensity require further investigation. We draw on markedness
theory (Jakobson, 1975; Trubetzkoy, 1936), specifically, contemporary
notions of markedness (e.g., Andersen, 2001), to further the definition of
DOI: 10.4324/9781003367550-9
Multimodality Studies in International Contexts : Contemporary Trends and Challenges, edited by Rocca, Liliana Vásquez, et al.,
Taylor & Francis Group, 2023. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/oculcarleton-ebooks/detail.action?docID=30965718.
Created from oculcarleton-ebooks on 2024-01-31 19:03:27.
112 Chloë Grace Fogarty-Bourget et al.
modal intensity and propose a toolkit for multimodal analysis. We dis

cuss two distinct but related issues: further development of the concept of
modal intensity and its use in multimodal analysis, and the identification
and analysis of the teaching practices (strategies) used by instructors with
the intention to facilitate student engagement in learning.
Our investigation has been guided by the following research questions:
1 What teaching strategies do instructors use when attempting to focus

students’ attention and involve them in subject matter in face-to-face
teaching contexts? And, what role does modal density play in these
teaching strategies?
2 How can modal intensity be identified and described in interaction?
And, how can instances of heightened modal intensity be analyzed?
We propose that when instructors display heightened attention toward an

action, or heightened modal density, they do so, in part, to draw the atten
tion of students toward that action (e.g., a question posed or written on the
board, written notes, a chunk of text). People are adept at perceiving the
attention of others during social interaction (Norris, 2004) and, therefore,
when instructors increase the attention they display toward an action, stu
dents can recognize the increase. We identify a range of strategies instruc
tors use to make their actions “stand out” to students, and thus potentially
focus student attention and encourage their involvement in learning. Some
of these strategies include the simultaneous use of speech, writing, and
movement, the manipulation of gaze directionality, facial display, textual
devices, gesture, and vocal pitch. Our research offers practical implications
for instructors, and theoretical contributions to analyzing modal density,
and thus attention/awareness, in interaction.
Theoretical and Analytical Framework

Multimodal (Inter)action Analysis
MIA (Bernad-Mechó, this volume; Norris, 2004, 2020) is a framework for

the analysis of social interaction, wherein the mediated action (Vygotsky,
1934/1987; Wertsch, 1991) is the central unit of analysis. In MIA, the
mediated action is analyzed in terms of lower- and higher-level actions.
Lower-level actions are the smallest meaningful units of action – for exam
ple, a single gesture. Higher-level actions are “chunks” of interaction with
recognizable openings and closings – for example, a university lecture –
which typically nest within one another. Thus, within a mathematics lec
ture there are many nested higher-level actions such as brief interactions
between students or the class-wide solving of a problem. Here, we use
MIA to analyze instructor actions, showing how certain actions can be
Toolkit for the Analysis of Modal Intensity 113
emphasized (marked) through various qualitative features. These marked

lower-level actions are used by instructors to demonstrate or attract in
creased attention to a higher-level action, which may have important im
plications for student engagement in learning.
Modal Density: Complexity and Intensity
Mode in MIA is defined as a “system of mediated action with regulari

ties” (Norris, 2020, p. 16) built up through concrete lower-level actions.
The lower-level action is not synonymous with mode but can be viewed
as building up a mode through use over time. This perspective allows re
searchers to consider how a single action relates to a history of actions.
From the MIA perspective, modes are viewed as always multiple, occur
ring simultaneously with other modes and nested within other modes. We
will return to this theoretical discussion of mode in later sections, but we
introduce it here to preface our discussion of modal density as a tool for
analyzing interactional attention/awareness.
Social actors naturally develop a sense of which actions a person is dis
playing more or less attention to. We gauge interactional attention, not by
reading minds but by reading actions to perceive what others are display
ing attention toward (see Norris, 2004) as they interact with other people
and the environment. Modal density is an analytical tool for determining
to which actions people are displaying more or less attention. It is a com
pound notion made up of both the complexity and intensity of lower-level
actions in a higher-level action (Norris, 2004). Modal complexity refers
to the quantity of lower-level actions being simultaneously produced, and
how they are inter-related in a higher-level action. Modal intensity refers
to the relative strength, weight, or importance of a particular mode within
a higher-level action.
High modal density can be achieved with high modal complexity, inten
sity, or both. The more complex or intense the modes of communication
are (i.e., the higher the modal density), the more attention a social ac
tor pays to the action(s) being produced (Norris, 2020). Essentially, high
modal density becomes a locus of attention in interaction. In a teaching
context, instructors may increase the modal density of an action to at
tract the attention of students to prompt their involvement in classroom
activities and thus facilitate student engagement in learning (see Fogarty-
Bourget, 2019).
The concept of modal density has been used to study shifts in attention
and awareness of social actors interacting in various contexts. However,
while the definition of modal complexity is apparent, definitions of modal
intensity are less clear and difficult to apply analytically. Norris (2004,
2019) offers three strategies to identify a mode that is intense: (1) A mode
that structures a higher-level action is of high intensity. For example, the
activities of people dancing while watching a band are structured by the

mode of music. If that mode is removed, then their actions would drasti
cally change. (2) If a higher-level action would not be possible had a certain
mode not been produced, it is of high intensity. For example, children sit
ting in a circle playing a board game would not be able to produce that
action without being in close proximity. Therefore, the mode of proxemics
is of high intensity. (3) All embodied modes of the participant/s are focused
on one mode. For example, someone hunched over a table while writing
focuses all their embodied modes (posture, gaze, arm movements) on the
mode of writing, thus it is highly intense for this person.
All three strategies highlight the inter-relatedness of lower-level actions
and attempt to tease apart their reliance upon one another in producing a
higher-level action. Strategies (1) and (2) rely on hypothetically removing
a lower-level action from a higher-level action and considering the magni
tude of this change. A large change indicates high modal intensity, and a
small or insignificant change indicates low modal intensity. It is likely that
hypothetical changes to any number of modes would significantly alter a
higher-level action, and it is difficult to systematically replicate the deter
mined magnitude of these changes across researchers. Strategy (3) pro
vides an approach that is grounded in concrete data, analyzing the way
multiple embodied modes relate. Norris (2004) points out that intensity
is “determined by the situation, the social actors, and other social and en
vironmental factors involved” (Norris, 2004, p. 79). In keeping with this
argument, we seek to offer an additional analytical approach to modal in
tensity that relies on the analysis of lower-level actions, and their relation
ship in context to other actions being produced. Further, in the next section
we propose a comprehensive definition of modal intensity that, we hope,
explicates the concept as it occurs across communicative contexts. In doing
so, we hope to avoid relying on hypotheticals, and contribute to further de
veloping MIA as a systematic approach to the analysis of multimodal data.
Markedness in Interaction
To develop a comprehensive definition of modal intensity and advance our

understanding of modal density, we draw on the concept of markedness
(Andersen, 2001; Brekhus, 1998; Trubetzkoy, 1936). Specifically, we rely
on the Durkheimian (1915/1965) notions of markedness and principles of
perception (Wertheimer, 1912) to further elucidate the concept of modal
intensity and provide tangible criteria for analysis.
The theory of markedness in linguistics was first introduced by Trubetz
koy in the 1930s (Jakobson, 1975) to explain the interconnectedness of
correlative phonemes in terms of binary opposition. Stemming from struc
turalist phonological theory (Andersen, 2001; Jakobson, 1975; Trubetz
koy, 1936), the original concept describes phonemes as being distinguished
by features that are either positively or negatively specified or marked. In

common parlance, markedness is generally equated with notability of an
object or phenomenon: unmarked refers to the ordinary, frequent, and
unexceptional, whereas marked refers to the extraordinary, infrequent, ex
ceptional, and salient. Because markedness is used to describe the way an
object or phenomenon is perceived in relation to context, it is associated
with directionality of attention. In other words, that which is marked be
comes the focus of attention by virtue of standing out (Brekhus, 1998) as
attention “catching” (van Leeuwen, 2005, p. 182) or “attention-worthy”
(Zerubavel, 1997, p. 51).
Since its introduction, the notion of markedness and the accompanying
terms “marked” and “unmarked” have become associated with the speci
fication of linguistic forms in terms of frequency, semantic or structural
complexity, and distribution (see Andersen, 2001). In cognitive linguistics,
the theory is used in relation to cognitive organization and visual percep
tion largely as a result of the parallels Trubetzkoy drew between marked/
unmarked and the figure-ground relation in gestalt psychology (a principle
of perception that asserts that humans organize stimuli into figure, the ob
ject of focus, and ground, i.e., background) (Andersen, 2001; Wertheimer,
1912). While numerous conceptualizations of markedness theory exist, a
broad distinction can be made between those based on a binary contrast
model (i.e., an item is either marked or not by a particular quality) and a
trinary model where both upper and lower poles are considered marked
in relation to the unmarked norm. This trinary model is based on Dur
kheim’s (1915/1965) concept of the sacred where extremes that stand out
as remarkably “above” or remarkably “below” the norm are considered
marked (Brekhus, 1998, pp. 36–37).
We suggest that concepts derived from markedness theory, particularly
those of the Durkheimian influence and/or stemming from notions of
figure-ground organization (Wertheimer, 1912), should be seen as com
plementary to the concept of modal density and can be used to inform
investigations of modal intensity in interaction. By drawing on these
concepts, we offer the following definition of modal intensity (Fogarty-

Bourget, 2019):
the phenomenon of heightened pragmatic salience occurring as a re

sult of an action being made prominent relative to surrounding or co-
occurring actions that have become recognized as regular, expected, or
standard due to frequency of occurrence.
(p. 28)
That is, lower-level actions produced with heightened modal intensity

are understood as marked in relation to those that regularly occur and
become established as the unmarked “norm” within the context of the
interaction. A lower-level action may be perceived as more prominent due

to both spatial and/or temporal distribution and qualitative features that
appear to be above or below the norm (i.e., actions produced previously
and/or concurrently within the interaction), including size, speed, flow of
movement, loudness, pitch, or force. Lower-level actions that are accen
tuated, made prominent, or foregrounded (i.e., are of high modal inten
sity) indicate higher attention toward/awareness of the higher-level action
being produced. As indicated previously, we suggest that in instructional
contexts, actions of high modal intensity are more attention-catching and
thus, more likely to be focused on or attended to by social actors involved
in the interaction. To demonstrate how the proposed definition can be use
ful in the identification and description of high modal density, we use it
in our analysis of video data drawn from two instructional contexts. The
methods used to conduct this research are described in the next section.
Two Studies: Multimodal Investigations of Teaching Practices

in Two Contexts
We draw on two “separate but complementary” (Schryer & Spoel, 2005,
p. 249) multimodality studies to investigate the phenomenon of modal
density. The first study (Fogarty-Bourget, 2019) is an investigation of stu
dent engagement in university mathematics lectures and the second (Pirini,
2015) is a study of peer-tutor interaction in one-to-one high school tutor
ing sessions. Both studies received institutional ethics approvals, partici
pants’ informed consent was provided, and permission to reproduce the
likeness of study participants was obtained.
Background
University Mathematics Lecturing

The main way undergraduate mathematics is taught in lecture-style univer
sity classes involves the instructor writing on the chalkboard (or whiteboard)
while providing a running commentary on what is being written, turning
to the students in the classroom and talking about what has been written,
moving from one part of the board to the other, gesturing to students and
different parts of the board during explanations, and so on. This style of
teaching has been shown to be a typified, regularized way of teaching math
ematics to university students in different countries and different languages
worldwide (e.g., Artemeva & Fox, 2011; Fox & Artemeva, 2012).
While teaching, instructors spend much of the lecture shifting between
writing on the chalkboard while articulating what is being written (math
ematical commentary) and turning to the students to talk about what has
been written (metacommentary, see Artemeva & Fox, 2011). The students
in the classroom attend to the lecture, take notes, and interact with each
other and the instructor. Throughout the lesson, the instructor attempts to
engage students in the lecture by using strategies to focus their attention,
involve them in problem solving, and elicit feedback (see Fogarty-Bourget,
2019).
High School Tutoring
One-to-one tutoring is a common form of extracurricular support for high

school students, targeting increased understanding, self-confidence, and
grades (Ireson & Rushforth, 2005), which has been shown to improve stu
dents’ academic performance and self-esteem (Anania, 1983; Cohen et al.,
1982). Students are involved in a form of guided participation (Rogoff,
1990) with the tutor, where they participate in cultural practices such as
analyzing a text and are guided to take on more responsibility for the prac
tice over time. Directing a student’s attention within the tutoring session
toward key aspects of the learning task is of central importance to provid
ing guidance (Pirini, 2016; Rogoff, 1990).
Data Collection and Analysis
Both studies used qualitative methods to collect rich, multimodal data.

In the study of mathematics lecturing, the data include video recordings
of lectures, semi-structured interviews, informal follow-up interviews, and
detailed field notes. In total, 11 university mathematics instructors ranging
in age and levels of experience participated in the study. The video excerpt
used in this chapter comes from a recording of a 2.5-hour calculus lecture
delivered to first-year engineering and physics students by a mathematics
instructor with decades of experience who is also the recipient of multiple
teaching awards. The lecture was recorded in a lecture hall of a mid-size
North American university.
In the study of high school tutoring, the data include video recordings of
tutoring sessions and field notes. In total, three experienced tutors and nine
students participated in the study. The video excerpts used in this chapter
come from a one-hour tutoring session delivered to an upper-level high
school student preparing for an English exam on Richard III by Shake
speare. The tutor holds an undergraduate English degree and has several
years of tutoring experience. The tutoring session was recorded in the stu
dent’s home in New Zealand, where the student and tutor regularly held
tutoring sessions.
The video data in both studies were initially analyzed using MIA. The
studies were conducted separately, with slightly different approaches to
data analysis. In the study of university mathematics lecturing, first, the

video was descriptively coded at the higher-level action of speaker turns
(i.e., an utterance bracketed by pauses, change in topic, and/or change in
speaker). From this initial phase of descriptive coding, a pattern emerged
in which the instructor engaged in cyclical episodes of problem solving
with the students in the classroom. The episodes consisted of the instructor
first introducing a problem, then writing the problem on the chalkboard,
then turning to the students to involve them in the problem-solving process
by asking for solutions to the problem, and finally writing the solution on
the chalkboard. One such episode was isolated and transcribed for further
analysis. The excerpt was selected based on quality of the recording (i.e.,
the participant was clearly audible and visible and the writing on the board
was clear and legible) and was otherwise comparable to other problem-
solving cycles identified in the recording. To investigate the strategies that
the instructor used to try to engage students in the subject matter, and
to understand the role that modal density played in the interaction, each
lower-level action was descriptively coded. Next, a second phase of cod
ing was applied in order to identify instances of modal complexity (made
evident by a higher concentration of lower-level actions co-occurring). Fi
nally, using the qualitative features described in the previous section (e.g.,
distribution, size, speed, etc.) as a start-list of possible codes, the excerpt
was analyzed for instances of modal intensity. The coding was facilitated
by qualitative coding software, ELAN (Max Planck Institute for Psycho
linguistics, 2019) and NVivo 11 (QSR International, 2017), which allowed
the transcription of speech, embodied activity, and so on to be applied di
rectly to the media timeline and synched with the video footage. A member
check was conducted in which the participants of the study reviewed the
raw data and final manuscript to ensure the overall accounts were realistic
and accurate.
In the study of high school tutoring sessions, first, higher-level actions
throughout the video were described, resulting in a list of actions. From
this list and with reference to the video, actions wherein the tutor provided
some form of guidance to the student were identified. Those that included
lower-level actions produced with high modal intensity were selected using
the same qualitative features applied for the study of mathematics lectures,
and were then transcribed according to Norris (2004, 2019). In brief, this
process involves producing a transcript for each participant by capturing a
screenshot of each change in lower-level action for each mode (e.g., gesture,
gaze, posture). Some actions change quite frequently, such as gesture, gaze,
head movement, while others are more stable, such as posture, proxemics,
and layout (Pirini, 2016). Stable actions require fewer screenshots, while
those changing frequently require a finer level of detail. The final tran
scripts were constructed by selecting screenshots that most clearly reflect
the higher-level action as it progresses (Norris, 2004). The mode of spoken
language is transcribed over the screenshots, with overlapping text indi

cating overlapping talk. The speech analysis program Praat (Boersma &
Weenink, 2012) was used where detailed pitch analysis was required. In the
next section, we present selected findings of our analysis.
Findings and Discussion

Focusing students’ attention on key subject matter and eliciting their par
ticipation and involvement in learning activities have been identified as
principal strategies used by instructors to facilitate student engagement in
classroom contexts (see Fogarty-Bourget, 2019). The analyses of the video
excerpts revealed many instances of heightened modal density used by the
educator participants (referred to hereafter as the instructor and the tutor)
in the hopes of engaging the students in learning by attempting either to
focus students’ attention or to involve them in the subject matter. Interest
ingly, despite the differences in teaching contexts, the analyses revealed
many examples of the instructor and tutor using similar strategies in their
attempts to facilitate student engagement in learning. The results of our
analysis were consistent across both contexts; however, for the purposes of
this chapter, we include a series of different examples from the two studies
to illustrate how modal density, and modal intensity in particular, may be
analyzed more systematically than is currently reflected in the literature.
The following examples demonstrate how the modal density of a higher-
level action can be increased through the use of (1) heightened modal com
plexity; (2) distributional scarcity; (3) textual devices; (4) manipulation of
facial display; (5) foreground gestures; and (6) pitch and backchanneling.
In doing so, we present a novel approach to the analysis of heightened
modal intensity through these means.
Modal Complexity
The first example (Figure 6.1) shows how the instructor attempts to involve
the students in problem solving by heightening the modal complexity of the

higher-level action of writing mathematical notation on the chalkboard.
The first frame shows the instructor writing mathematical notation on
the chalkboard (Figure 6.1, Frame 1). He is oriented toward the chalk
board with his gaze focused on his writing as he completes the line of
notation he has just articulated. In this moment, the instructor is interact
ing with the students primarily through the modes of writing and spa
tial location (standing in front of them). When the time comes for the
instructor to complete the equation, he begins to articulate the mathemati
cal process once again (“is going to give you-”) and shifts his gaze to the
line above (Figure 6.1, Frames 2–3). Instead of immediately articulating
the answer, the instructor elongates the personal pronoun “you-”, gestures
Figure 6.1 An example of modal complexity being used as an involvement strat

egy. Photograph by the authors.
to the line above, looks over his shoulder slightly in the direction of the
students (Figure 6.1, Frames 3–4), and utters the answer with a rising in
tonation as though seeking confirmation from the students (“minus eight
een?”) (Figure 6.1, Frame 4). The instructor shifts from interacting with
the students primarily through writing to interacting with the students
through speech, gesture, gaze, and, to a limited degree, facial display, thus
heightening the modal complexity (and, therefore, the modal density) of
the higher-level action. By posing a question to the students, gesturing to
focus their attention, and shifting his gaze in their direction, the instructor
attempts to involve the students in the problem-solving process.
Modal Intensity
In addition to modal complexity (see Figure 6.1), high modal density can be
established through modal intensity. The following examples show how the
instructor and the tutor attempt to direct the students’ attention and involve
them in the subject matter by using instances of heightened modal intensity.
Distributional Scarcity
Figure 6.2 shows an instance in which the instructor has just posed a ques
tion to the students (“so which one is max and which one is min?”). The in
structor turns to the students and awaits a response. As he waits, he stands
with his hands at his sides without speaking or moving his body; in this
moment, the only movements he makes are shifts in his gaze as he looks
back and forth across the room at the students seated in the lecture hall.
In this example the mode of gaze takes on high modal intensity, in part,
because the higher-level action of waiting that the instructor is producing
(i.e., the instructor’s prolonged stillness, silence, and bodily orientation)
also stands out as attention-catching. During mathematics lecturing, in
structors are in near constant motion, their gaze directionality continu
ally shifting between the chalkboard, classroom equipment, students, their
notes, etc. (Artemeva & Fox, 2011; Fox & Artemeva, 2012), and they
spend much of the lecture speaking. Thus, instances in which instructors
stand silently facing the students are rare (distributionally scarce), and,
therefore, stand out as particularly salient.
In the mathematics study, Fogarty-Bourget (2019) observed that when
instructors stopped speaking but were still writing on the chalkboard, the
students could hear the sounds of the chalk on the board and continued to
Figure 6.2 An example of high modal intensity occurring as a result of distribu

tional scarcity of a higher-level action, in this case the instructor stand
ing motionless and silent. Photograph by the authors.
take notes; however, when the chalk also stopped, a particular silence was
created which caused the students to look up from their notes. In various
classroom contexts as well, teachers’ silence is marked and used to capture
and focus students’ attention (Jaworski & Sachdev, 1998). In the example
in Figure 6.2, the higher-level action of the instructor’s prolonged stillness,
(total) silence, and bodily orientation were notable and directed to attract
students’ attention. Throughout this pragmatically salient moment, the
only changes occurring in the instructor’s positioning were shifts in gaze.
Because in this instance all other modes remained constant (unchanging),
the instructor’s shifts in gaze stood out as prominent lower-level actions.
That is, the mode of gaze in this context can be perceived as markedly more
intense than earlier interactions which typically involved speech, gesture,
and/or writing in addition to shifts in gaze (see, for instance, Figure 6.1).
This exemplifies how, unlike modal complexity, certain actions (in this case,
shifts in gaze) can be made more prominent as a result of fewer actions co-
occurring simultaneously. Figure 6.2 also illustrates how certain actions (in
this case, the instructor waiting) can assume heightened modal intensity
by occurring only infrequently, below the norm of distribution based on
pre-established contextual expectations. This same strategy for increasing
modal intensity is commonly used in interactions with text and writing.
Textual Devices
In interactions with text and writing, heightened modal intensity can be

constructed through textual devices such as italicizing, and/or underlining,
provided either by writers or readers who may attempt to make certain
parts of the text stand out by underlining or highlighting segments of text.
However, for heightened modal intensity to be achieved in this way, such
devices should be used sparingly (distributional scarcity) to function suc
cessfully as an attention-catching strategy.
Analyzing instances of heightened modal intensity resulting from dis
tributional scarcity includes calculating the number of times a particular
action occurs within an interaction (e.g., within a video recording of a math

ematics lecture) or, ideally, within a series of comparable interactions (e.g.,
within a corpus of video recordings of mathematics lectures). Throughout
our data, we observed many instances of the participants working to facili
tate student engagement in learning through interactions with key pieces
of text and writing. For instance, it was not uncommon for the partici
pants to underline an important section of text or writing multiple times
to emphasize it for the students as essential. Not only does this strategy
make the content more visually salient, but by underlining multiple times,
the instructors increase the modal intensity of the action through repetition
(frequency) while demonstrating their attention toward the segment, mark
ing it attention-worthy.
Facial Display
In addition to frequency, heightened modal intensity can be the result of

the way a lower-level action is performed. As previously discussed, ac
tions can be accentuated or made more attention-catching through the
manipulation of certain features. Figure 6.3 shows an instance in which
the instructor is attempting to evoke a response from the students by ma
nipulating his facial display, thus accentuating the higher-level action of
looking to the students for a response.
After having posed a question (Figure 6.2), the instructor turns to face
the students. Initially, he gazes around the room with his eyebrows slightly
raised and his lips pressed shut; however, his facial expression is fairly re
laxed (Figure 6.3, Frame 1). Not being met with a response, he continues to
look around the room expectantly; he widens his eyes, raises his eyebrows
higher, and purses his lips more tightly together (Figure 6.3, Frame 2). By
pursing his mouth shut, the instructor demonstrates his silence to students.
By minimizing the size of his mouth while widening his eyes and raising
his eyebrows, the instructor makes his inquisitive gaze appear larger and
more intense. Finally, the instructor fixes his gaze on a student who looks as
though they may have a response to the question. To encourage the student
to respond, he focuses his gaze on the student, and exaggerates his expression
by widening his eyes, raising his eyebrows, and tightening his mouth further
(Figure 6.3, Frame 3). After a moment, the student offers a muffled response.
This example demonstrates how the mode of gaze can be made more
intense through changes in facial display that are markedly above or be
low the norm (Brekhus, 1998), in this case, by raising the eyebrows and
widening the eyes so they appear larger than normal and by tightening the
mouth, so it appears smaller than normal. By maintaining his focused gaze
with a high degree of modal intensity and demonstrating his unwillingness
to resume his turn at talk (emphasized by his tightly closed mouth), the
instructor applies a degree of pressure on the students to take up a turn
at talk and thus become actively involved in the problem-solving process.
Figure 6.3 An example of facial display impacting the modal intensity of gaze.
Photograph by the authors.
Foreground Gestures
Gesture can be used with a high degree of modal intensity. Cooperrider

(2017) makes a distinction between foreground gestures, which appear to
have heightened communicative intent, and background gestures, which “are
a kind of unmarked default against which foreground gestures stand out”
(p. 191). Cooperrider offers four hallmarks of foreground status, any one of
which is sufficient to qualify a gesture as foregrounded. These hallmarks are
as follows: the gesture is (1) used concurrently with a spoken demonstrative
(e.g., this, that, these); (2) produced in absence of speech, or sequentially with
speech (i.e., speech trails off before production of the gesture, or resumes
after the gesture is produced); (3) co-organized with speaker gaze (i.e., the
speaker looks at the gesture or in the direction being indicated by the ges
ture); and/or (4) produced with a visible degree of effort made evident by the
size (larger gestures require greater effort) and precision of the action.
The next example (Figure 6.4) shows the tutor using a foreground gesture
in an attempt to facilitate student engagement in learning. In Frame 1, the
tutor acknowledges the student’s comment “mm hmm”. She then points and
looks (co-organizing gesture and gaze) to a section of the text as she asks,
“But did you read what Richard says?” (Figure 6.4, Frames 2–3). The stu
dent responds by saying “no that’s” and reaching out his hand to the paper to
start reading (Figure 6.4, Frames 3–5). The tutor lifts the hand used for point
ing but does not return it to a position of rest (cf. Sacks & Schegloff, 2002).
She continues to point (showing effort) and says, “kay that that bit’s key”
making two rapid pointing motions that coincide with the demonstratives
“that that” (Figure 6.4, Frames 6–7), before returning her hand to her chest
(Frame 8). Pointing gestures are commonly held in position briefly (Kendon,
2004), but in this case the tutor points two more times as the student draws
the text closer. The first point indicates the text and is coproduced with the
tutor’s spoken utterance to specify “what Richard says”. The second and
third instances of pointing further indicate the text and emphasize its impor
tance. Thus, not only does the deictic stand out as foregrounded, the modal
intensity of the higher-level action is also established through repetition of

speech (that, that) and gesture (three consecutive points).
Vocal Pitch
Thus far, we have discussed the phenomenon of heightened modal intensity

as produced through embodied activity, text, and writing. Additionally,
sonic or vocal action can be marked through “increased loudness, pitch
or duration” (van Leeuwen, 2005, p. 182). In the final example, the tutor
uses changes in pitch to make particular utterances more pragmatically
salient than others. The tutor regularly uses backchannels (Edmonson,
1981) to indicate her support and attentiveness (Hirschman, 1994/1973)
Figure 6.4 An example of a foreground gesture and repetition being used as a

strategy to direct a student’s attention and involve him in reading a key
piece of text. Photograph by the authors.
to the student. Usually, these backchannels take the form of an unmarked

mmm hmmm with a characteristic rising pitch, but occasionally, when the
student has provided a response that is on the mark, the tutor produces
backchannels marked with a falling pitch. The software application Praat
(Boersma & Weenink, 2012) was used to compare the pitch contours of
each type of backchannel in order to illustrate how high modal intensity
manifests in the change of pitch. In the first 20 minutes of the session the
tutor produces nine mmm hmmms, which all display the same rising pitch
contour (Figure 6.5). When producing these backchannels, the tutor’s fa
cial features are neutral, and she produces only small nodding motions.
Figure 6.5 Screenshot and pitch analysis of typical mmm hmmm backchannel.

Photograph by the authors.
By contrast, Figure 6.6 shows screenshots of the two instances of

backchannels marked by a change of pitch contour. In the first instance
(A), the tutor says, hmmm good, and in the second instance (B) she says
mmm mmm good. In both instances the tutor drops her pitch on the
mmm. Comparatively, the tutor is also more expressive when produc
ing these backchannels. In Figure 6.6(A), the tutor smiles prominently
and raises her eyebrows as she produces the utterance. In Figure 6.6(B)
she nods deeply, producing a larger angle at her chin and raises her
eyebrows.
Figure 6.6 Screenshots and pitch analyses of the tutor producing backchannels

marked by falling pitch contours. Photograph by the authors.
The backchannels produced with a falling pitch that diverge from the
established norm appear to be used to encourage the student to continue
their line of thought and draw attention to their correct response. The
change in pitch is an instance of high modal intensity, whereby the deep
pitch emphasizes the backchannel. Comparative analysis using software
applications such as Pratt to identify changes in pitch, volume, and/or du
ration is yet another way that modal intensity can be analyzed.
Conclusions
In this chapter, we draw on the findings of two complementary studies of
teaching to explore the role of modal density in instructor strategies used
to facilitate student engagement in learning. At the same time, we further
develop the concept of modal density (Bernad-Mechó, this volume; Norris,
2004) by proposing a comprehensive definition of modal intensity that can
be practically applied to the multimodal analysis of interaction. We de
fine modal intensity as the phenomenon of heightened pragmatic salience
occurring as a result of an action being made prominent relative to sur
rounding or co-occurring actions that have become recognized as regular,
expected, or standard due to frequency of occurrence within a particular
context. Through MIA (Norris, 2004, 2020) of video recordings of an
instructor teaching university mathematics in a lecture hall and a tutor in
a one-to-one tutoring session of high school English, we sought answers
to the first research question: what teaching strategies do instructors use
when attempting to focus students’ attention and involve them in subject
matter in face-to-face teaching contexts? And, what role does modal den
sity play in these teaching strategies?
Our analysis revealed that the educators used a range of strategies in
an attempt to facilitate student engagement in learning. In addition to
teaching relevant content, collaborating in problem solving, and asking
questions (see Fogarty-Bourget, 2019; Pirini, 2015), the participants in our
study used a variety of strategies to focus students’ attention, and involve
them in the subject matter. In university mathematics lecturing, instructors
can use combinations of gesture, gaze directionality, and intonation while
writing, thereby constructing modal complexity to involve students in the
mathematical notation on the chalkboard. While facing the students, in
stances of silence and stillness, which are distributionally scarce in this type
of teaching, can be used as a strategy to capture students’ attention. Dur
ing these moments, changes in gaze directionality and facial display take
on heightened pragmatic salience. Instructors appear to use these combi
nations of silence, stillness, and gaze directionality as a strategy to elicit

responses from students seated in the classroom, involving them in the
problem-solving process.
Foreground gestures (Cooperrider, 2017) can also be used strategically
to encourage students’ active involvement in problem solving and direct
students’ attention to key subject matter. Repetition of gesture and speech
can be used to draw attention to important sections of text as can a strate
gic use of textual devices. Finally, vocal qualities such as pitch can be used
to emphasize backchanneling to encourage students to pursue a certain
line of thinking and thus guiding them toward correct interpretations of
the content.
It is important to reiterate that in both studies, our focus was on the

educators rather than on the students. While we cannot say for sure what
role modal density plays in students’ engagement, we have observed that
instances of heightened modal density were used by the instructor and
the tutor for the purpose of emphasis, and to capture and maintain stu
dents’ attention and promote students’ active involvement in learning. The
phenomenon of heightened modal intensity was identified throughout em
pirical data specifically at points when the participants were attempting to
highlight important subject matter for the students and encourage involve
ment. It is possible that instances of heightened modal intensity provide a
cue to students letting them know what is required of them in the shared
task at hand, and in that way may play a key role in facilitating student
engagement. Moreover, our study focuses on two very different teaching
contexts. While investigating the ways that contexts may shape the phe
nomena of interest was beyond our scope, such research could offer further
insights.
The second question we asked was, how can modal intensity be identi
fied and described in interaction? And, how can instances of heightened
modal intensity be analyzed? The methods described here offer a guide for
conducting MIA (Norris, 2004, 2020) of video data. By drawing on no
tions derived from markedness theory to inform our analysis, we present
a preliminary toolbox for the identification and description of lower-level
actions which contribute to increased modal intensity and, thus, modal
density. We have revealed that the following qualitative features can be
used to heighten the modal intensity of lower-level actions:
Distribution: an action can stand out if it is produced notably more or

less frequently than those that regularly occur in a given context. Thus,
modal intensity can be heightened through repetition or distributional
scarcity of an action;
Textual Devices: textual modes such as print or writing can be emphasized
using devices such as underlining, highlighting, bolding (which also en

tails value), italicizing, and/or circling depending on the distribution of
their application (see Distribution);
Size: actions such as gestures, facial display, and other bodily movements
can be emphasized by manipulating the size (larger or smaller) of their
production. Disembodied modes such as text, writing, and brush strokes
can also be intensified through size;
Foreground Status: gestures that meet at least one of the four hallmarks
(Cooperrider, 2017) qualify as having foreground status. Such gestures
are attention-catching and stand out as a “critical part” of a message
(p. 179);
Vocal Qualities: utterances can be emphasized through changes in v ocal

qualities such as pitch, stress, volume, and/or duration (see van L
eeuwen,
2005).
This preliminary list of qualitative features acts as a first step toward a

comprehensive account of characteristics that contribute to heightened
modal intensity. A myriad of other qualities, which can be used to con
struct heightened modal intensity in interaction such as speed, proximity,
color, force, and others not mentioned here, have yet to be documented, in
cluding those which may be unique to specific modes, contexts, disciplines,
cultures, etc. We, therefore, issue a call for more studies of this nature to
broaden our understanding of modal intensity as it occurs in interaction
and lend empirical support to these and other features which increase the
pragmatic salience of actions in various contexts, and educational settings
in particular.
References
Anania, J. (1983). The influence of instructional conditions on student learning
and achievement. Evaluation in Education: An International Review Series, 7(1),
3–76.
Andersen, H. (2001). Markedness and the theory of linguistic change. In H. An
dersen (Ed.), Actualization: Linguistic change in progress (pp. 21–57). John
Benjamins.
Artemeva, N., & Fox, J. (2011). The writing’s on the board: The global and the
local in teaching undergraduate mathematics through chalk talk. Written Com-
munication, 28(4), 1–35. https://doi.org/10.1177/0741088311419630
Boersma, P., & Weenink, D. (2012). Praat: Doing phonetics by computer [Com
puter program]. Version 5.3.23.
Brekhus, W. (1998). A sociology of the Unmarked: Redirecting our focus. Socio-
logical Theory, 16(1), 34–51. https://doi.org/10.1111/0735-2751.00041
Chi, M. T. H., & Wylie, R. (2014). The ICAP framework: Linking cognitive en
gagement to active learning outcomes. Educational Psychologist, 49(4), 219–243.

https://doi.org/10.1080/00461520.2014.965823
Cohen, P. A., Kulik, J. A., & Kulik, C. L. C. (1982). Educational outcomes of tutor
ing: A meta-analysis of findings. American Educational Research Journal, 19(2),
237–248. https://doi.org/10.3102/00028312019002237
Cooperrider, K. (2017). Foreground gesture, background gesture. Gesture, 16(2),
176–202. https://doi.org/10.1075/gest.16.2.02coo
Dunne, E., & Owen, D. (Eds.) (2013). The student engagement handbook: Practice
in higher education. Emerald Group Publishing.
Durkheim, É. (1965). The elementary forms of religious life (J. W. Swain, Trans.).
New York Free Press (Original work published 1915).
Edmonson, W. (1981). Spoken discourse: A model for analysis. Longman.
Fogarty-Bourget, C. G. (2019). Facilitating student engagement in undergraduate

mathematics lectures: A multimodal investigation [Doctoral thesis]. Carleton
University. https://doi.org/10.22215/etd/2019-13616
Fox, J., & Artemeva, N. (2012). The cinematic art of teaching university math
ematics: Chalk talk as embodied practice. Multimodal Communication, 1(1),
83–103. https://doi.org/10.1515/mc-2012-0007
Gettinger, M., & Walter, M. J. (2012). Classroom strategies to enhance academic
engaged time. In S. Christenson, A. Reschly, & C. Wylie (Eds.), Handbook of
research on student engagement (pp. 653–673). Springer.
Hirschman, L. (1994/1973). Female-male differences in conversational inter
action. Language in Society, 23(3), 427–442. http://www.jstor.org/stable/
4168538
Ireson, J., & Rushforth, K. (2005). Mapping and evaluating shadow education.
ESRC Research Project RES-000-23-0117. Institute of Education, University of
London.
Jakobson, R. (Ed.). (1975). N. S. Trubetzkoy’s letters and notes. Mouton.
Jaworski, A., & Sachdev, I. (1998). Beliefs about silence in the classroom. Language
and Education, 12(4), 273–292. https://doi.org/10.1080/09500789808666754
Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge University
Press. https://doi.org/10.1177/0261927X06289694
Max Planck Institute for Psycholinguistics. (2019, March 30). ELAN software –
The Language Archive. https://tla.mpi.nl/tools/tla-tools/elan
Norris, S. (2004). Analyzing multimodal interaction: A methodological frame-
work. Taylor & Francis.
Norris, S. (2019). Systematically working with multimodal data: Research meth-
ods in multimodal discourse analysis. John Wiley & Sons.
Norris, S. (2020). Multimodal theory and methodology: For the analysis of (inter)
action and identity. Routledge.
Pirini, J. (2015). Research into tutoring: Exploring agency and intersubjectivity
[Doctoral thesis]. AUT University.
Pirini, J. (2016). Intersubjectivity and Materiality: A multimodal perspective. Mul-
timodal Communication, 5(1), 1–14. https://doi.org/10.1515/mc-2016-0003
QSR International. (2017). NVivo (Version 11) [qualitative data analysis software].
Rogoff, B. (1990). Apprenticeship in thinking: Cognitive development in social
context. Oxford University Press.

Sacks, H., & Schegloff, E. A. (2002). Home position. Gesture, 2(2), 133–146.
https://doi.org/10.1075/gest.2.2.02sac
Schryer, C. F. & Spoel, P. (2005). Genre theory, health-care discourse, and profes
sional identity formation. Journal of Business and Technical Communication,
19(3), 249–278. https://doi.org/10.1177/1050651905275625
Trubetzkoy, N. S. (1936). Essai d’une théorie des oppositions phonologiques
[A theory of phonological oppositions]. Journal de Psychologie, 33, 5–18.
van Leeuwen, T. (2005). Introducing social semiotics. Taylor & Francis.
Vygotsky, L. S. (1987). Thinking and speech. In R. W. Rieber & A. S. Carton (Eds.)
The collected works of L. S. Vygotsky (Vol. 1), Problems of general psychology
(pp. 39-285). Plenum Press (Original work published 1934).
Wertheimer, M. (1912). Experimentelle Studien über das Sehen von Bewegung [Ex
perimental studies on the perception of motion]. Zeitschrift Für Psychologie,
61(1), 161–265.
Wertsch, J. V. (1991). Voices of the mind: A sociocultural approach to mediated
action. Harvard University Press.
Zerubavel, E. (1997). Social mindscapes: An invitation to cognitive sociology. Har
vard University Press.

Multimodality Studies in International Contexts Co... - (6 A Toolkit For The Analysis of Modal Intensity Strategies Instructors... )

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Multimodality Studies in International Contexts Co... - (6 A Toolkit For The Analysis of Modal Intensity Strategies Instructors... )

Uploaded by

Copyright:

Available Formats

6 A Toolkit for the Analysis

Chloë Grace Fogarty-Bourget, Jesse Pirini

Student engagement can be broadly defined as “the direct, measurable in­

modal intensity and propose a toolkit for multimodal analysis. We dis­

1 What teaching strategies do instructors use when attempting to focus

We propose that when instructors display heightened attention toward an

Theoretical and Analytical Framework

Multimodal (Inter)action Analysis

MIA (Bernad-Mechó, this volume; Norris, 2004, 2020) is a framework for

emphasized (marked) through various qualitative features. These marked

Modal Density: Complexity and Intensity

Mode in MIA is defined as a “system of mediated action with regulari­

activities of people dancing while watching a band are structured by the

To develop a comprehensive definition of modal intensity and advance our

by features that are either positively or negatively specified or marked. In

concepts, we offer the following definition of modal intensity (Fogarty-­

the phenomenon of heightened pragmatic salience occurring as a re­

That is, lower-level actions produced with heightened modal intensity

interaction. A lower-level action may be perceived as more prominent due

Two Studies: Multimodal Investigations of Teaching Practices

University Mathematics Lecturing

High School Tutoring

One-to-one tutoring is a common form of extracurricular support for high

Data Collection and Analysis

Both studies used qualitative methods to collect rich, multimodal data.

data analysis. In the study of university mathematics lecturing, first, the

language is transcribed over the screenshots, with overlapping text indi­

Findings and Discussion

the students in problem solving by heightening the modal complexity of the

Figure 6.1 An example of modal complexity being used as an involvement strat­

Figure 6.2 An example of high modal intensity occurring as a result of distribu­

In interactions with text and writing, heightened modal intensity can be

action occurs within an interaction (e.g., within a video recording of a math­

In addition to frequency, heightened modal intensity can be the result of

Gesture can be used with a high degree of modal intensity. Cooperrider

intensity of the higher-level action is also established through repetition of

Thus far, we have discussed the phenomenon of heightened modal intensity

Figure 6.4 An example of a foreground gesture and repetition being used as a

piece of text. Photograph by the authors.

to the student. Usually, these backchannels take the form of an unmarked

Figure 6.5 Screenshot and pitch analysis of typical mmm hmmm backchannel.

By contrast, Figure 6.6 shows screenshots of the two instances of

Figure 6.6 Screenshots and pitch analyses of the tutor producing backchannels

nations of silence, stillness, and gaze directionality as a strategy to elicit

It is important to reiterate that in both studies, our focus was on the

Distribution: an action can stand out if it is produced notably more or

using devices such as underlining, highlighting, bolding (which also en­

Vocal Qualities: utterances can be emphasized through changes in v­ ocal

This preliminary list of qualitative features acts as a first step toward a

gagement to active learning outcomes. Educational Psychologist, 49(4), 219–243.

Fogarty-Bourget, C. G. (2019). Facilitating student engagement in undergraduate

context. Oxford University Press.

You might also like

Student engagement can be broadly defined as “the direct, measurable in

modal intensity and propose a toolkit for multimodal analysis. We dis

Mode in MIA is defined as a “system of mediated action with regulari

concepts, we offer the following definition of modal intensity (Fogarty-

the phenomenon of heightened pragmatic salience occurring as a re

language is transcribed over the screenshots, with overlapping text indi

Figure 6.1 An example of modal complexity being used as an involvement strat

Figure 6.2 An example of high modal intensity occurring as a result of distribu

action occurs within an interaction (e.g., within a video recording of a math

Figure 6.4 An example of a foreground gesture and repetition being used as a

Figure 6.5 Screenshot and pitch analysis of typical mmm hmmm backchannel.

Figure 6.6 Screenshots and pitch analyses of the tutor producing backchannels

using devices such as underlining, highlighting, bolding (which also en

Vocal Qualities: utterances can be emphasized through changes in v ocal