Professional Documents
Culture Documents
Transcription systems for gestures, speech, prosody, postures, and gaze 1037
Sassenberg, Uta, Ingo Helmich and Hedda Lausberg 2010. Awareness of emotions: Movement be-
haviour as indicator of implicit emotional processes in participants with and without alexity-
hmia. In: Haack, Wiese, Abraham, Chiarcos (eds.), Proceedings of KogWis: 10th Biannual
Meeting of the German Society for Cognitive Science, 169. Potsdam, Germany.
Scheflen, Albert E. 1973. Communicational Structure: Analysis of a Psychotherapy Transaction.
Bloomington: Indiana University Press.
Scheflen, Albert E. 1974. How Behaviour Means. New York: Anchor/Doubleday.
Seyfeddinipur, Mandana, Sotaro Kita and Peter Indefrey 2008. How speakers interrupt them-
selves in managing problems in speaking: Evidence for self-repairs. Cognition 108: 837–842.
Sirigu, Angela, Jean-Rene Duhamel, Laurent Cohen, Bernard Pillon, Bruno Dubois and Yves
Agid 1996. The mental representation of hand movements after parietal cortex damage.
Science 273: 1564–1568.
Skomroch, Harald, Robert Rein, Katharina Hogrefe, Georg Goldenberg and Hedda Lausberg
2010. Gesture production in the right and left hemispheres during narration of short movies.
Conference Proceedings, International Society for Gesture Studies Frankfurt/Oder, Germany,
25–30 July 2010.
Ulrich, Gerald 1977. Videoanalytische Methoden zur Erfassung averbaler Verhaltensparameter
bei depressiven Syndromen. Pharmakopsychiatrie 10: 176–182.
Ulrich, Gerald and K. Harms 1985. Video analysis of the non-verbal behaviour of depressed pa-
tients before and after treatment. Journal of Affective Disorders 9: 63–67.
Wagner Cook, Susan and Susan Goldin-Meadow 2006. The role of gesture in learning: Do children
use their hand to change their minds? Journal of Cognition and Development 7(2): 211–232.
Wallbott, Harald G. 1989. Movement quality changes in psychopathological disorders. In B. Kir-
kaldy (Ed.), Normalities and abnormalities in human movement. Medicine and Sport Science
29: 128–146.
Wartenburger, Isabell, Esther Kühn, Uta Sassenberg, Manja Foth, Elizabeth A. Franz and Elke
van der Meer 2010. On the relationship between fluid intelligence, gesture production, and
brain structure. Intelligence 38: 193–201.
Willke, S. 1995. Die therapeutische Beziehung in psychoanalytisch orientierten Anamnesen und
Psychotherapien mit neurotischen, psychosomatischen und psychiatrischen Patienten. DFG-
Bericht Wi 1213/1–1.
Müller, Cienki, Fricke, Ladewig, McNeill, Teßendorf (eds.) 2013, Body – Language – Communication (HSK 38.1), de Gruyter, 1037–1059
Unauthenticated
Download Date | 5/17/16 9:42 PM
1038 V. Methods
Abstract
The chapter presents a concise overview of existing transcription system for gestures,
speech, prosody, postures, and gaze aiming at a presentation of their theoretical back-
grounds and methodological approaches. After a short introduction discussing the under-
standing of the term “transcription”, the article first focuses on transcription systems in
modern gesture research and discusses systems from the field of gestures research, non-
verbal communication, conversation analysis, and artificial agents (e.g., Birdwhistell
1970; Sager 2001; Martell 2002; Bressem this volume; Gut et al. 2002; Kipp, Neff, and Al-
brecht 2007; Lausberg and Sloetjes 2009; HIAT 1, GAT). Afterwards, the paper presents
well-known transcription systems for speech from the field of linguistics, conversation,
and discourse analysis (e.g., IPA, HIAT, CHAT, DT, GAT). Apart from systems
for describing speech, the article also focuses on systems for the transcription of prosody.
In doing so, the paper discusses prosodic descriptions within the field of conversation
analysis, discourse analysis, and linguistics (HIAT2, GAT2, TSM, ToBI, PROLAB,
INTSINT, SAMPROSA). The last sections of the paper focus on the transcription of
body posture and gaze (e.g., Birdwhistell 1970; Ehlich and Rehbein 1982; Goodwin
1981; Schöps in preparation; Wallbott 1998).
1. Introduction
The term transcription goes back to the Latin word trānsscrı̄bere meaning ‘to overwrite’
or ‘to rewrite’ (Bußmann 1990: 187). In a linguistic understanding, transcription refers
to the notation of spoken language in written form. More specifically, it is understood as
the reproduction of communicative events using alphabetic resources and specific sym-
bols while capturing the characteristics and specifics of spoken language (Dittmar
2004). Transcription must be understood as a scientific working method directed to
the analytical needs of a scientist by freezing oral communication and making it acces-
sible to thorough inspection (Redder 2001: 1038; see also Bohle this volume for a gen-
eral discussion). However, today, transcription is not only restricted to linguistics.
Investigating interaction and communication is part of a number of scientific disciplines,
such as ethnography, sociology, psychology, neurology, and biology for instance, which
face comparable problems and obstacles in making communicative behavior analyz-
able. Accordingly, the term transcription is no longer restricted to the notation of spo-
ken language, but also includes the notation of bodily behavior, such as gesture, posture,
and gaze.
Unauthenticated
Download Date | 5/17/16 9:42 PM
68. Transcription systems for gestures, speech, prosody, postures, and gaze 1039
For each of the section, the notational system includes a “basic notational logic”
(Birdwhistell 1970: 258), which is combined with indicators to capture the differing var-
iants producible by the 8 sections of the body. The description is based on articulatory
aspects, such as muscular tension as well joints of articulation. Altogether, the system
offers 400 signs for the description of bodily motion. It includes descriptions for hand
and finger activities, and bi-manual gestures, yet only rarely includes the notation of
movement.
Unauthenticated
Download Date | 5/17/16 9:42 PM
1040 V. Methods
The temporal structure of gestures is described according to the beginnings and endings
of movement. Assuming only a restricted number of possible movements for the pro-
duction of gestures, the direction of movement is described as being horizontal, vertical
or diagonal, for instance. Moreover, the system mentions differences in the quality of
movements and differentiates movements as slow, fast, or discontinuous (Sager 2001:
28–29). Signifikanzpunkte are described on the basis of two principles. The principle
center of rotation records the position of arms and hands through the centers of rotation
allowing the movement (shoulder, upper arm, elbow, wrist). The principle body levels
registers movements in the various centers of rotation relative to three body levels (ver-
tical axis, sagittal axis, transversal axis), allowing for different degrees of freedom for
movement (e.g., pronation or supination of the hand). Apart from the position and
movement of hands and arms, the system includes a description of hand shapes.
Seven communicatively relevant types of hand configurations, derivable from two
types of movements of the hand, are differentiated (e.g., cupped hand) (Sager 2001:
41–42).
Unauthenticated
Download Date | 5/17/16 9:42 PM
68. Transcription systems for gestures, speech, prosody, postures, and gaze 1041
FORM pursues a strictly hierarchical and technical set up due to its aspect of computa-
tional processing and its applicability in research on artificial agents. The system is
designed for the use with the annotation program ANVIL (Kipp 2004).
For the positions of the hand, the notational system draws on the gesture space intro-
duced by McNeill, which divides the gesture space “into sectors using a system of
concentric squares.” (McNeill 1992: 86) (see Bressem this volume for the notational
system)
Unauthenticated
Download Date | 5/17/16 9:42 PM
1042 V. Methods
McNeill’s system uses the written speech transcription as the basis for coding ges-
tures. Gestures are annotated into the speech transcription by inserting brackets for
the beginning and end of a gesture. The scheme includes the description of hand con-
figuration, orientation, position in gesture space, and movement. The gesture space con-
sists of a system of concentric squares dividing the space in front of the speaker into
three basic areas (center, periphery, and extreme periphery) (see McNeill 1992: 89).
Hand configurations are based on the labeling of hand shapes in American Sign Lan-
guage (ASL) using the “ASL shape that the gesture mostly resembles.” (McNeill
1992: 86) Orientation of the hand is coded according to the gesture space and palm ori-
entation. Gestural movements, such as shape, direction, and trajectory are accounted
for in a descriptive fashion without providing strict guidelines.
Configurations of the hand are described with the taxonomic notation systems of
HamNoSys (Prillwitz et al. 1989) and FORM (Martell 2002). Movements are coded
for shape, direction, and modifiers, that is, size, speed, and number of repetitions. In
addition symmetry of hands is coded. For the combination of gestures into complex
units, the Conversational Gesture Transcription system distinguishes sequences of pre-
cedence and overlap (Gut et al. 2002: 6). The functional classification of gestures is
based on a four-part classification distinguishing various degrees of overlap between
gestural and verbal meaning.
2.2.3. Kipp, Neff, and Albrecht (2007): A transcription and coding scheme for the
automatic generation and animation of character-specific hand/arm gestures
Offering a scheme developed “for the specific purpose of automatically generating and
animating character-specific hand/arm gestures, but with potential general value”
Unauthenticated
Download Date | 5/17/16 9:42 PM
68. Transcription systems for gestures, speech, prosody, postures, and gaze 1043
(Kipp, Neff, and Albrecht 2007: 1), the scheme operates on the concept of a gesture lex-
icon made up of lexemes, that is “prototypes of recurring gesture patterns where certain
formational features remain constant over instances and need not be annotated for
every single occurrence.” (Kipp, Neff, and Albrecht 2007: 4)
The scheme is implemented and used within the ANVIL annotation tool (Kipp
2004) and consists of adding annotation elements to a track in which each element
is described with a pre-assigned set of attributes, which capture the most essential
parts of a gesture. The annotation scheme includes the spatial form of gestures in
which gesture phases, phrases, and units (Kendon 1980, 2004) are described along
with handedness, path of movement, position, as well as hand shape, and distance of
hands. Hand shapes are coded using a taxonomic classification of 9 types of configura-
tions. Gestures’ membership to a lexical category is determined by the lexeme that de-
fines the hand shape, palm orientation, and exact trajectory. Typical lexemes include:
raised index finger, cup (open hand), finger ring or progressive (circular movement)
(Kipp, Neff, and Albrecht 2007: 14). In a last step, the relation of speech and gesture
is captured.
Unauthenticated
Download Date | 5/17/16 9:42 PM
1044 V. Methods
Module (i) refers to the kinetic features of a hand movement, i.e., execution of move-
ment vs. no movement, trajectory and dynamics of movements, location of acting as
well as contact with body or not. For the characterization of the dynamic aspects of
movements, NEUROGES uses Laban notation (1950). Module (ii) allows for the cod-
ing of bimanual relation (for instance in touch vs. separate, symmetrical vs. complemen-
tary, independent vs. dominance). Module (iii) brings in the functional aspects and
determines the meaning of gestures based on a specific combination of kinetic features
(hand shape, orientation, path of movement, effort and others), which define the
various gesture types.
(i) only partly to specific elements of the movement potential (e.g. raising hand),
(ii) to the expressional quality including the movement potential, and
(iii) the summery of complex movements and actions (e.g., waving).
The transcription furthermore includes a rough record of the on- and offsets as well as
the length of movements and only a rudimentary description of form or function. The
relevance of the gestural component and its transcription is thereby always dependent
on its relevance for the verbal communication (Ehlich and Rehbein 1979a: 315). The
verbal modality is the constitutive background for the transcription of gestures, so
that gestures are transcribed on commentary lines dependent on the verbal utterance.
Unauthenticated
Download Date | 5/17/16 9:42 PM
68. Transcription systems for gestures, speech, prosody, postures, and gaze 1045
kinesic behavior. Bodily behavior is noted in commentary lines dependent on the verbal
utterance and inclusive of a rudimentary coding of on- and offsets of movement se-
quences. Bodily behavior is yet only of interest if it obviously influences the verbal
and communicative orientations of the speakers and addressees.
The Gesprächsanalytische Transkriptionssystem (GAT, Selting, Auer, Barden, et al.
1998; Selting, Auer, Barth-Weingarten, et al. 2009) also only includes behavioral as-
pects, such as proxemic, kinesic, gesture, and gaze in the transcription of face-to-face
interaction if it contributes to the “(un)ambiguousness of other predominantly verbal
levels of activities.” (Selting, Auer, Barth-Weingarten, et al. 2009: 26) Regarding ges-
tures, the Gesprächsanalytisches Transkriptionssystem (Selting, Auer, Barden, et al.
1998) lists deictic gestures, illustrators, and emblems and includes a rough description
of on- and offsets as well as apex, that is, peaks, of gestural movement sequences.
The description is behavior-oriented and tries to be as little interpretative as possible.
The Gesprächsanalytisches Transkriptionssystem offers differing degrees of detailed-
ness in the transcription, as it sets apart basic vs. fine-grained transcripts. Basic tran-
scripts usually include an interpretive characterization of the gestures within the line
containing the verbal transcription. Fine-grained transcripts list gestures in a separate
line under the simultaneously occurring verbal activity. For illustrative purposes and in
cases of special importance of the nonverbal activities, the Gesprächsanalytisches Trans-
kriptionssystem also mentions the inclusion of pictures in the transcript (Selting, Auer,
Barden, et al. 1998: 28). Its newest revision, the Gesprächsanalytisches Transkriptions-
system 2, mentions that new conventions for the transcription of visual components of
communication are being designed (Selting, Auer, Barth-Weingarten, et al. 2009: 356)
due to the growing interest and importance of visual aspects of communication within
the field of interaction analysis.
This section has presented notation and transcription systems for gestures, which
range from a focus on:
(i) form, to
(ii) form and function, to
(iii) rudimentary descriptions.
The presented systems primarily differ in the aspect of whether a) gestures’ form can
and should be separated from possible meanings and functions (e.g., Birdwhistell
1970; Martell 2002; Bressem this volume) or b) whether a separation of form, meaning,
and function is not useful for a transcription of gestures (e.g., Gut et al. 2002; McNeill
1992, 2005). These diverging foci thereby go along with the theoretical assumption that
gestures can either be broken down into separate components, which may combine with
other features or not. Furthermore, the role of speech in the process of notation or tran-
scription is different in the systems presented above. While the verbal utterance is of
particular importance for some of the systems (e.g., Ehlich and Rehbein 1979b; Selting,
Auer, Barden, et al. 1998; Selting, Auer, Barth-Weingarten, et al. 2009), others exclude
the verbal modality in parts completely from the notational process (e.g., Bressem this
volume). A further difference in the presented systems is the integration of annotation
software. While especially recent systems use the advantages of annotation software for
the process of notation and transcription (e.g., Kipp, Neff, and Albrecht 2007; Lausberg
and Sloetjes 2009; Martell 2002), others rely on conventional and longstanding methods
Unauthenticated
Download Date | 5/17/16 9:42 PM
1046 V. Methods
of transcribing gesture with the use of word documents. Yet, the most important differ-
ence is the clarification and integration of the system within a theoretical and method-
ological framework as well as their implications, which, for most systems, are not
presented as articulately as is necessary.
Unauthenticated
Download Date | 5/17/16 9:42 PM
68. Transcription systems for gestures, speech, prosody, postures, and gaze 1047
Unauthenticated
Download Date | 5/17/16 9:42 PM
1048 V. Methods
transcript as an observational datum. A rich inventory for the reproduction of turns and
their sequential progression is thus characteristic for the system. The system uses the
“eye dialect,” a standard orthography onomasiologically adapted to the phonetic real-
ization of the expression. The system’s inventory of signs is based on the Latin alphabet.
The format of transcription is sequentially organized and turns of speakers are, analog
to their linear progression, ordered in chronological order. The system represents
simultaneous utterances of more than one speaker by using brackets at the time the
overlap occurs. The end of verbal units/turns is marked by standard orthography for
interrogative sentences. The system also includes prosodic aspects of utterances, such
as remarkable changes in pitch, changes in the intonations contour, lengthening,
emphasis, changes in tempo, and pauses. Nonverbal events, such as gestures, mimics,
breathing, and coughing for instance, are represented in commentary lines in double
parenthesis (e.g. coughing). In its newest revision (Jefferson 2002), the system also
includes guidelines for a computer-aided transcription.
Unauthenticated
Download Date | 5/17/16 9:42 PM
68. Transcription systems for gestures, speech, prosody, postures, and gaze 1049
While all of the presented systems aim at a representation of spoken language, the
systems differ from each other in a range of aspects. The most obvious difference is the
diverging format of representation for spoken language. Systems may use notation scores,
that is, an endless line (Ehlich and Rehbein 1979b) or single lines for single speakers and
turns (e.g., Selting, Auer, Barden, et al. 1998; Selting, Auer, Barth-Weingarten, et al. 2009).
Furthermore, the systems differ in their basic unit of analysis: turn (e.g., Jefferson 1984;
Selting, Auer, Barden, et al. 1998; Selting, Auer, Barth-Weingarten, et al. 2009) vs. utter-
ance as a whole (MacWhinney 2000). Going along with this is a differentiation in the
segmentation of verbal units, varying from sounds (International Phonetic Association
2005) to intonations units for instance (Du Bois 1991). In addition, the systems include
prosodic aspects as well as other forms of bodily behavior to varying degrees.
4. Transcription of prosody
Transcription systems for prosody generally capture two main types of phenomena:
a) the division of utterances into prosodically-marked chunks, units or phrases and
b) the representation of prominence along with aspects such as pitch movement,
reset or rhythmic change for instance. But, the size and type of prosodic units vary
considerably in the different systems, thus resulting in different prosodic transcriptions.
Unauthenticated
Download Date | 5/17/16 9:42 PM
1050 V. Methods
Unauthenticated
Download Date | 5/17/16 9:42 PM
68. Transcription systems for gestures, speech, prosody, postures, and gaze 1051
Unauthenticated
Download Date | 5/17/16 9:42 PM
1052 V. Methods
The preceding overview has shown that the systems not only vary in their theoretical
and methodological tradition, but also in their focus on a transcription of prosody. In
general, the proposed systems can be classified according to common and differing
Unauthenticated
Download Date | 5/17/16 9:42 PM
68. Transcription systems for gestures, speech, prosody, postures, and gaze 1053
parameters (Llisterri 1994). Regarding the representation of prosodic events, the sys-
tems can be classified into multi-tiered (the Tones and Break Indices, the International
Transcription System for Intonation) or one-tiered systems (e.g., the International Pho-
netic Alphabet, the Gesprächsanalytisches Transkriptionssystem 2, the HalbInter-
pretative ArbeitsTranskriptionen 2). The systems further differ regarding their aspects
of machine readable symbols (e.g., the Speech Assessment Method Phonetic Alphabet,
SAMSINT or SAM Prosodic Transcription) vs. non-machine readable symbols (e.g., the
Gesprächsanalytisches Transkriptionssystem 2, the Tones and Break Indices, the Ge-
sprächsanalytisches Transkriptionssystem, the HalbInterpretative ArbeitsTranskriptio-
nen, PROLAB). In addition, the systems differ in whether they are theory-driven
systems, that is a) based on a conception of the phonetics-phonology interface or b) data-
driven systems, i.e., defined by the needs and the practices which are known to be
relevant in order to explain the discursive or the interactional behavior of the speakers (Llis-
terri 1994).
Unauthenticated
Download Date | 5/17/16 9:42 PM
1054 V. Methods
head, arms, hands), which are described according to their movement abilities (up, down,
back for the shoulders for instance), Wallbott’s system allows for a rough anatomical
description of various body postures.
Although primarily developed for the notation of dance, aspects of the Laban nota-
tion (Laban 1950) are nowadays used in a range of transcription systems (e.g., Davis
1979; Greenbaum and Rosenfeld 1980; Kendon 2004; Lausberg and Sloetjes 2009).
This system includes a basic segmentation of the skeletal system, basic kinesiological
terms (e.g., rotations), spatial terms (e.g., straight vs. circular paths) and object relations
(e.g., touch), which allow for a detailed notation of body posture and bodily movement.
Recently, new transcription systems aiming at a combination of describing form and
function of body postures are postulated. Schöps (in preparation), for instance, presents
a system including basic postures (standing, laying down, sitting) as well as body parts
and movement categories, along with different predicates for the transcription of the
used body configurations (e.g., spread for arms and legs). The Body Action and Posture
Coding system by Dael and Scherer (2012) approaches the transcription and coding of
body posture on an “anatomical level (different articulations of body parts), a form
level (direction and orientation of movement), and a functional level (communicative
and self-regulatory functions)” (Dael and Scherer 2012).
7. Conclusion
This overview of notation and transcription system for speech and bodily behavior has
shown that a range of proposals exists, all of which try to account for the reproduction
of verbal and bodily behavior in written forms. It became apparent that the individual
Unauthenticated
Download Date | 5/17/16 9:42 PM
68. Transcription systems for gestures, speech, prosody, postures, and gaze 1055
8. References
Argyle, Michael 1975. Bodily Communication. London: Methuen.
Argyle, Michael and Mark Cook 1976. Gaze and Mutual Gaze. Cambridge: Cambridge University
Press.
Auer, Peter, Elizabeth Couper-Kuhlen and Frank Müller 1999. Language in Time: The Rhythm
and Tempo of Spoken Interaction. New York: Oxford University Press.
Avanzi, Mathieu, Anne Lacheret-Dujour and Bernard Victorri 2008. ANALOR. A tool for semi-
automatic annotation of French prosodic structure. Paper presented at the Interspeech 2008,
Campinas, Brazil, May 6–9.
Battison, Robin 1974. Phonological deletion in American Sign Language. Sign Language Studies
5: 1–19.
Becker, Karin 2004. Zur Morphologie redebegleitender Gesten. MA thesis, Department of Philos-
ophy and Humanities, Free University Berlin.
Beckman, Mary E. and Gayle Ayers Elam 1997. Guidelines for ToBI labelling. Retrieved
15.07.2011, from http://www.ling.ohio-state.edu/research/phonetics/E_ToBI/
Birdwhistell, Ray 1970. Kinesics and Context. Essays on Body, Motion, Communication. Philadel-
phia: University of Pennsylvania Press.
Bohle, Ulrike 2007. Das Wort Ergreifen – das Wort Übergeben: Explorative Studie zur Rolle Re-
debegleitender Gesten in der Organisation des Sprecherwechsels. Berlin: Weidler.
Bohle, Ulrike this volume Approaching notation, coding, and analysis from a conversational ana-
lysis point of view. In: Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David
McNeill and Sedinha Teßendorf (eds.), Body – Language – Communication: An International
Handbook on Multimodality in Human Interaction. (Handbooks of Linguistics and Communi-
cation Science 38.1.) Berlin: De Gruyter Mouton.
Bressem, Jana this volume. A linguistic perspective on the notation of form features in gestures. In:
Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill and Sedinha
Teßendorf (eds.), Body – Language – Communication: An International Handbook on Multi-
modality in Human Interaction. (Handbooks of Linguistics and Communication Science 38.1.)
Berlin: De Gruyter Mouton.
Bressem, Jana and Silva H. Ladewig 2011. Rethinking gesture phases: Articulatory features of ges-
tural movement? Semiotica 184(1/4): 53–91.
Brinker, Klaus and Sven F. Sager 1989. Linguistische Gesprächsanalyse. Berlin: Erich Schmidt.
Bußmann, Hadumod 1990. Lexikon der Sprachwissenschaft. 2nd revised edition. Stuttgart,
Germany: Alfred Kröner.
Campione, Estelle and Jean Vèronis 2001. Semi-automatic tagging of intonation in French spoken
corpora. In: Paul Rayson, Andrew Wilson, Tony McEnery, Andrew Hardie and Shereen Khoja
(eds.), Proceedings of the Corpus Linguistics’ 2001 Conference, 90–99. Lancaster, U.K.: Lancas-
ter University, UCREL.
Couper-Kuhlen, Elizabeth 1993. English Speech Rhythm: Form and Function in Everyday Verbal
Interaction. Amsterdam: John Benjamins.
Crystal, David 1969. Prosodic Systems and Intonation in English. Cambridge: Cambridge Univer-
sity Press.
Unauthenticated
Download Date | 5/17/16 9:42 PM
1056 V. Methods
Dael, Nele and Klaus R. Scherer 2012. The Body Action and Posture coding system (BAP):
Development and reliability. Journal of Nonverbal Behavior, 36, 97–121.
Davis, Martha 1979. Laban analysis of nonverbal communication. In: Shirley Weitz (ed.), Nonver-
bal Communication: Readings with Commentary, 182–206. New York: Oxford University Press.
Dittmar, Norbert 2004. Transkription: Ein Leitfaden mit Aufgaben für Studenten, Forscher und
Laien. Heidelberg: VS Verlag für Sozialwissenschaften.
Du Bois, John W. 1991. Transcription design principles for spoken discourse research. Pragmatics
1(1): 71–106.
Du Bois, John W., Susanna Cumming, Stephan Schuetze-Coburn and Danae Paolino 1992. Dis-
course transcription. Santa Barbara Papers in Linguistics 4. University of California, Santa Bar-
bara, Department of Linguistics.
Ehlich, Konrad and Jochen Rehbein 1976. Halbinterpretative Arbeitstranskriptionen (HIAT 1).
Linguistische Berichte 25: 21–41.
Ehlich, Konrad and Jochen Rehbein 1979a. Erweiterte halbinterpretative Arbeitstranskriptionen
(HIAT2). Linguistische Berichte 59: 51–75.
Ehlich, Konrad and Jochen Rehbein 1979b. Zur Notierung nonverbaler Kommunikation für dis-
kursanalytische Zwecke (HIAT2). In: Peter Winkler (ed.), Methoden der Analyse von Face-to-
Face-Situationen, 302–329. Stuttgart: Metzler.
Ehlich, Konrad and Jochen Rehbein 1982. Augenkommunikation. Methodenreflextion und Beis-
pielanalyse. Amsterdam: John Benjamins.
Eibl-Eibesfeldt, Irenäus 1971. Transcultural patterns of ritualized contact behavior. In: Aristide H.
Esser (ed.), Behavior and Environment. The Use of Space by Animals and Men, 238–246. New
York: Plenum.
Ekman, Paul and Wallace V. Friesen 1969. The repertoire of nonverbal behavior: Categories,
origins, usage, and coding. Semiotica 1 49–98.
Ekman, Paul and Wallace V. Friesen 1978. Facial Action Coding System (FACS): A Technique for
the Measurement of Facial Action. Palo Alto, CA: Consulting Psychologists Press.
Frey, Siegfried, Hans Peter Hirsbrunner, Jonathan Pool and William Daw 1981. Das Berner Sys-
tem zur Untersuchung nonverbaler Interaktion: I. Die Erhebung des Rohdatenprotokolls; II.
Die Auswertung von Zeitreihen visuell-auditiver Information. In: Peter Winkler (ed.), Metho-
den der Analyse von Face-to-Face-Situationen, 203–268. Stuttgart: Metzler.
Frey, Siegfried, Hans Peter Hirsbrunner and Ulrich Jorns 1982. Time-series notation: A coding
principle for the unified assessment of speech and movement in communication research. In:
Ernest W. B. Hess-Lüttich (ed.), Multimodal Communication: Vol. I Semiotic Problems of Its
Notation, 30–58. Tübingen: Narr.
Fricke, Ellen 2007. Origo, Geste und Raum: Lokaldeixis im Deutschen. Berlin: Walter de Gruyter.
Fricke, Ellen 2012. Grammatik multimodal: Wie Wörter und Gesten zusammenwirken. Berlin:
De Gruyter.
Garcia, Jesus, Ulrike Gut and Antonio Galves 2002. Vocale – a semi-automatic annotation tool for
prosodic research. Proceedings of Speech Prosody 2002.
Gibbon, Dafydd 1988. Intonation and discourse. In: Janos S. Petöfi (ed.), Text and Discourse Con-
stitution, 3–25. Berlin: De Gruyter.
Gibbon, Dafydd, Inge Mertins and Roger K. Moore 2000. Handbook of Multimodal and Spoken
Dialogue Systems: Resources, Terminology, and Product Evaluation. Norwell, Massachusetts,
USA: Kluwer Academic.
Goodwin, Charles 1981. Conversational Organization: Interaction between Speakers and Hearers.
New York: Academic Press.
Greenbaum, Paul E. and Howard Rosenfeld 1980. Varieties of touching in greetings: Sequential
structure and sex-related differences. Journal of Nonverbal Behavior 5(1): 13–25.
Grice, Martin, Stefan Baumann and Ralf Benzmüller 2005. German intonation in autosegmental-
metrical phonology. In: Sun-Ah Jun (ed.), Prosodic Typology: The Phonology of Intonation
and Phrasing, 55–83. Oxford: Oxford University Press.
Unauthenticated
Download Date | 5/17/16 9:42 PM
68. Transcription systems for gestures, speech, prosody, postures, and gaze 1057
Gumperz, John and Norine Berenz 1993. Transcribing conversational exchanges. In: Jane A. Ed-
wards and Martin D. Lampert (eds.), Talking Data: Transcription and Coding in Discourse
Research, 91–121. Hillsdale, NJ: Lawrence Erlbaum.
Gussenhoven, Carlos, Toni Rietveld and Jaques Terken 2005. Transcription of Dutch intonation.
In: Sun-Ah Jun (ed.), Prosodic Typology: The Phonology of Intonation and Phrasing, 118–145.
Oxford: Oxford University Press.
Gut, Ulrike, Karin Looks, Alexandra Thies and Dafydd Gibbon 2002. Cogest: Conversational ges-
ture transcription system version 1.0. Fakultät für Linguistik und Literaturwissenschaft, Uni-
versität Bielefeld, ModeLex Tech. Rep 1.
Hager, Joseph C., Paul Ekman and Wallace V. Friesen 2002. Facial action coding system. Re-
trieved 15.07.2011, from http://face-and-emotion.com/dataface/facs/guide/InvGuideTOC.html
Hall, Alan T. 1963. A system for the notation of proxemic behavior. American Anthropologist
65(5): 1003–1026.
Hirsbrunner, Hans-Peter, Siegfried Frey and Robert Crawford 1987. Movement in human inter-
action: Description, parameter formation and analysis. In: Aron W. Siegman and Stanley Feld-
stein (eds.), Nonverbal Behavior and Communication, 99–140. Hillsdale, NJ: Lawrence
Erlbaum.
Hirst, Daniel 1991. Intonation models: Towards a third generation. In: Actes du XIIe‘me Congre‘s
International des Sciences Phone´tiques, 305–310. Aix-en-Provence, France: Université de
Provence, Service des Publications.
Hirst, Daniel and Albert Di Cristo 1998. Intonation Systems: A Survey of Twenty Languages. Cam-
bridge: Cambridge University Press.
International Phonetic Association 2005. Handbook of the International Phonetic Association: A
Guide to the Use of the International Phonetic Alphabet. Cambridge: Cambridge University
Press.
Jefferson, Gail 1984. On stepwise transition from talk about a trouble to inappropriately next-
positioned matters. In: Maxwell J. Atkinson and John Heritage (eds.), Structures of Social
Action: Studies in Conversation Analysis, 191–222. Cambridge: Cambridge University Press.
Jefferson, Gail 2002. Is “no” an acknowledgment token? Comparing American and British uses of
(+)/(-) tokens. Journal of Pragmatics 34(10/11): 1345–1383.
Kallmeyer, Werner and Reinhold Schmitt 1996. Forcieren oder: Die verschärfte Gangart. Zur
Analyse von Kooperationsformen im Gespräch. In: Werner Kallmeyer (ed.), Gesprächsrhe-
torik: Rhetorische Verfahren im Gesprächsprozeß, 19–118. Tübingen: Narr.
Kendon, Adam 1967. Some functions of gaze-direction in social interaction. Acta Psychologica 26:
22–63.
Kendon, Adam 1972. Some relationship between body motion and speech In: Aron W. Siegman,
and Benjamin Pope (eds.), Studies in Dyadic Communication, 177–216. Elmsford, NY: Perga-
mon Press.
Kendon, Adam 1980. Gesture and speech: Two aspects of the process of utterance. In: Mary
Ritchie Key (ed.), Nonverbal Communication and Language, 207–288. The Hague: Mouton.
Kendon, Adam 2004. Gesture. Visible Action as Utterance. Cambridge: Cambridge University
Press.
Kendon, Adam and A. Ferber 1973. A description of some human greetings. In: R. P. Michael
and J. H. Crook (eds.), Comparative ecology and behavior of primates, 591–668. New York:
Academic Press.
Kipp, Michael 2004. Gesture Generation by Imitation: From Human Behavior to Computer Char-
acter Animation. Boca Raton, FL: Dissertation.com.
Kipp, Michael, Michael Neff and Irene Albrecht 2007. An annotation scheme for conversational
gestures: How to economically capture timing and form. Journal on Language Resources and
Evaluation – Special Issue on Multimodal Corpora 41(3/4): 325–339.
Klima, Edward and Ursula Bellugi 1979. The Signs of Language. Cambridge, MA: Harvard
University Press.
Unauthenticated
Download Date | 5/17/16 9:42 PM
1058 V. Methods
Knowles, Gerry, Briony Williams and L. Taylor 1996. A Corpus of Formal British English Speech.
London: Longman.
Kohler, Klaus J. 1991. A model of German intonation. Arbeitsberichte des Instituts für Phonetik
und digitale Sprachverarbeitung der Universität Kiel 25: 295–360.
Kohler, Klaus J. 1995. ToBIG and PROLAB: Two prosodic transcription systems for German
compared. Paper presented at the Conference ICPhS Stockholm, 13 August 1995.
Kohler, Klaus J., Matthias Pätzold and Adrian P. Simpson 1995. From Scenario to Segment: The
Controlled Elicitation, Transcription, Segmentation and Labelling of Spontaneous Speech.
Kiel, Germany: Institut für Phonetik und Digitale Sprachverarbeitung, IPDS, Universität Kiel.
Laban, Rudolph von 1950. The Mastery of Movement on the Stage. London: Macdonald and
Evans.
Ladewig, Silva H. and Jana Bressem forthcoming. New insights into the medium ‘hand’: Discover-
ing recurrent structures in gestures. Semiotica.
Lausberg, Hedda and Han Sloetjes 2009. Coding gestural behavior with the NEUROGES –
ELAN system. Behavioral Research Methods 41(3): 841–849.
Llisterri, Joaquim 1994. Prosody encoding survery. MULTEXT-LRE Project 62-050.
MacWhinney, Brian 2000. The CHILDES Project: Tools for Analyzing Talk, Volume 2: The Da-
tabase. Hillsdale, NJ: Lawrence Erlbaum.
Martell, Craig 2002. Form: An extensible, kinematically-based gesture annotation scheme. Paper
presented at International Conference on Language Resources and Evaluation. European
Language Resources Association.
Martell, Craig 2005. FORM: An experiment in the annotation of the kinematics of gesture. Ph.D.
dissertation, Department of Computer and Information Sciences, University of Pennsylvania.
Martell, Craig and Joshua Kroll no date. Corpus-based gesture analysis: An extension of the
FORM dataset for the automatic detection of phases in a gesture. Unpublished manuscript.
McNeill, David 1992. Hand and Mind. What Gestures Reveal about Thought. Chicago: University
of Chicago Press.
McNeill, David 2005. Gesture and Thought. Chicago: University of Chicago Press.
McNeill, David and Susan D. Duncan 2000. Growth points in thinking-for-speaking. In: David
McNeill (ed.), Language and Gesture, 141–161. Cambridge: Cambridge University Press.
Mertens, Pier 2004. The prosogram: Semi-automatic transcription of prosody based on a tonal per-
ception model. Paper presented at Speech Prosody 2004, March 23–26, 2004, Nara, Japan.
Müller, Cornelia 1998. Redebegleitende Gesten: Kulturgeschichte – Theorie – Sprachvergleich. Ber-
lin: Arno Spitz.
Müller, Cornelia 2010. Wie Gesten bedeuten. Eine kognitiv-linguistische und sequenzanalytische
Perspektive. In: Sprache und Literatur 41(1): 37–68.
Müller, Cornelia, Hedda Lausberg, Ellen Fricke and Katja Liebal 2005. Towards a grammar of
gesture: evolution, brain, and linguistic structures. Berlin: Antrag im Rahmen der Förderinitia-
tive “Schlüsselthemen der Geisteswissenschaften Programm zur Förderung fachübergreifender
und internationaler Zusammenarbeit”.
Müller, Cornelia, Jana Bressem and Silva H. Ladewig this volume. Towards a grammar of ges-
tures: A form-based view. In: Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig
and David McNeill (eds.), Body – Language – Communication: An International Handbook
on Multimodality in Human Interaction. Handbooks of Linguistics and Communication
Science (38.1). Berlin and Boston: De Gruyter Mouton.
O’Connor, Joseph Desmond and Gordon Frederick Arnold 1973. Intonation of Colloquial
English. London: Longman.
Pierrehumbert, Janet B. 1980. The Phonology and Phonetics of English Intonation. Boston: Mas-
sachusetts Institute of Technology Press.
Prillwitz, Siegmund, Regina Leven, Heiko Zienert, Thomas Hanke and Jan Henning 1989. Ham-
NoSys Version 2.0 Hamburger Notationssystem für Gebärdensprachen: Eine Einführung. Ham-
burg: Signum Verlag.
Unauthenticated
Download Date | 5/17/16 9:42 PM
68. Transcription systems for gestures, speech, prosody, postures, and gaze 1059
Redder, Angelika 2001. Aufbau und Gestaltung von Transkriptionssystemen. In: Klaus Brinker,
Gerd Antos, Wolfgang Heinemann and Sven F. Sager (eds.), Text und Gesprächslinguistik.
Ein Internationales Handbuch Zeitgenössischer Forschung, 1038–1059. (Handbücher zur
Sprach- und Kommunikationswissenschaft 16.2.) Berlin: De Gruyter.
Roach, Peter, Gerry Knowles, Tamas Varadi and Simon Arnfield 1993. Marsec: A machine-read-
able spoken English corpus. Journal of the International Phonetic Association 23(2): 47–54.
Sager, Sven F. 2001. Probleme der Transkription nonverbalen Verhaltens. In: Klaus Brinker, Gerd
Antos, Wolfgang Heinemann and Svend F. Sager (eds.), Text und Gesprächslinguistik. Ein In-
ternationales Handbuch Zeitgenössischer Forschung, 1069–1085. (Handbücher zur Sprach- und
Kommunikationswissenschaft 16.2.) Berlin: De Gruyter.
Sager, Svend F. and Kristin Bührig 2005. Nonverbale Kommunikation im Gespräch–Editorial. In:
Kristin Bührig and Svend F. Sager (eds.), Osnabrücker Beiträge zur Sprachtheorie 70: Nonver-
bale Kommunikation im Gespräch, 5–17. Oldenberg: Redaktion Obst.
Scheflen, Albert 1965. The significance of posture in communication systems. Psychiatry 27: 316–331.
Scherer, Klaus R. 1970. Non-Verbale Kommunikation: Ansätze zur Beobachtung und Analyse der
Aussersprachlichen Aspekte von Interaktionsverhalten. Hamburg: Buske.
Scherer, Klaus R. 1979. Die Funktionen des nonverbalen Verhaltens im Gespräch. In: Klaus R.
Scherer and Harald G. Wallbott (eds.), Nonverbale Kommunikation: Forschungsberichte zum
Interaktionsverhalten, 25–32. Weinheim: Beltz.
Schmitt, Reinhold (ed.) 2007. Koordination: Analysen zur Multimodalen Interaktion. Tübingen:
Narr.
Schneider, Wolfgang 2001. Der Transkriptionseditor HIAT-DOS. Gesprächsforschung-Online
Zeitschrift zur Verbalen Interaktion 2: 29–33.
Schönherr, Beatrix 1997. Syntax – Prosodie – Nonverbale Kommunikation. Empirische Untersu-
chungen zur Interaktion Sprachlicher und Parasprachlicher Ausdrucksmittel im Gespräch. Tü-
bingen, Germany: Niemeyer.
Schöps, Doris in preparation. Körperhaltung als Zeichen am Beispiel des DEFA-Films. Disserta-
tion, Technische Universität Berlin.
Selting, Margret, Peter Auer, Birgit Barden, Jörg R. Bergmann, Elizabeth Couper-Kuhlen, Sus-
anne Günther, Christoph Meier, Uta Quasthoff, Peter Schlobinski and Susanne Uhmann
1998. Gesprächsanalytisches Transkriptionssystem (GAT). Linguistische Berichte 173: 91–122.
Selting, Margret, Peter Auer, Dagmar Barth-Weingarten, Jörg Bergmann, Pia Bergmann, Karin
Birkner, Elizabeth Couper-Kuhlen, Arnulf Deppermann, Peter Gilles, Susanne Günthner,
et al. 2009. Gesprächsanalytisches Transkriptionssystem 2 (GAT 2). Gesprächsforschung –
Online Zeitschrift zur Verbalen Interaktion 10: 353–402.
Silverman, Kim, Mary Beckman, John Pitrelli, Mori Ostendorf, Colin Wightman, Patti Price, Janet
Pierrehumbert and Julia Hirschberg 1992. ToBI: A standard for labeling English prosody.
Proceedings of ICSLP-1992, 867–870.
Stokoe, William 1960. Sign Language Structure. Buffalo, NY: Buffalo University Press.
Wallbott, Harald 1998. Ausdruck von Emotionen in Körperbewegungen und Körperhaltungen. In:
Caroline Schmauser and Thomas Noll (eds.), Körperbewegungen und ihre Bedeutung, 121–136.
Berlin: Arno Spitz.
Weinrich, Lotte 1992. Verbale und Nonverbale Strategien in Fernsehgesprächen: Eine Explorative
Studie. Tübingen: Niemeyer.
Wells, John, William Barry, Martin Grice, Adrian Fourcin and Dafydd Gibbon 1992. Standard
computer-compatible transcription. Technical Report No. SAM Stage Report Sen.3 SAM
UCL-037. London: University College London.
Zwitserlood, Ingeborg, Asli Özyürek and Pamela Perniss 2008. Annotation of sign and gesture
cross-linguistically. Paper presented at the 3rd Workshop on the Representation and Proces-
sing of Sign Languages, Marrakesh.
Unauthenticated
Download Date | 5/17/16 9:42 PM