Macrocognition Metrics and Scenarios:

Design and Evaluation for Real-World

Teams First Edition Miller
Visit to download the full and correct content document:
https://textbookfull.com/product/macrocognition-metrics-and-scenarios-design-and-ev
aluation-for-real-world-teams-first-edition-miller/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...

Org Design for Design Orgs: Building and Managing In-

House Design Teams 1st Edition Peter Merholz

https://textbookfull.com/product/org-design-for-design-orgs-
building-and-managing-in-house-design-teams-1st-edition-peter-
merholz/

Color Design Workbook A Real World Guide To Using Color

In Graphic Design Sean Adams

https://textbookfull.com/product/color-design-workbook-a-real-
world-guide-to-using-color-in-graphic-design-sean-adams/

Membrane desalination: from nanoscale to real world

applications First Edition Sapalidis

https://textbookfull.com/product/membrane-desalination-from-
nanoscale-to-real-world-applications-first-edition-sapalidis/

Systemic Racism: Making Liberty, Justice, and Democracy

Real 1st Edition Ruth Thompson-Miller

https://textbookfull.com/product/systemic-racism-making-liberty-
justice-and-democracy-real-1st-edition-ruth-thompson-miller/
Prediction Methods for Blood Glucose Concentration
Design Use and Evaluation 1st Edition Harald
Kirchsteiger

https://textbookfull.com/product/prediction-methods-for-blood-
glucose-concentration-design-use-and-evaluation-1st-edition-
harald-kirchsteiger/

Layout Workbook Revised and Updated A real world guide

to building pages in graphic design Puhalla

https://textbookfull.com/product/layout-workbook-revised-and-
updated-a-real-world-guide-to-building-pages-in-graphic-design-
puhalla/

Managing Kubernetes operating Kubernetes clusters in

the real world First Edition Burns

https://textbookfull.com/product/managing-kubernetes-operating-
kubernetes-clusters-in-the-real-world-first-edition-burns/

World Music Concise Edition Terry E. Miller And Andrew

Shahriari

https://textbookfull.com/product/world-music-concise-edition-
terry-e-miller-and-andrew-shahriari/

Applied Computational Thinking with Python Algorithm

design for complex real world problems 2nd Edition
Sofía De Jesús | Dayrene Martinez

https://textbookfull.com/product/applied-computational-thinking-
with-python-algorithm-design-for-complex-real-world-problems-2nd-
edition-sofia-de-jesus-dayrene-martinez/
Macrocognition Metrics and Scenarios
Macrocognition Metrics and
Scenarios
Design and Evaluation for Real-World Teams

Edited by

Emily S. Patterson
Ohio State University, USA

&

Janet E. Miller
Air Force Research Laboratory, USA
First published 2010 by Ashgate Publishing

Published 2016 by Routledge

2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN
711 Third Avenue, New York, NY 10017, USA

Routledge is an imprint of the Taylor & Francis Group, an informa business

Copyright © Emily S. Patterson and Janet E. Miller 2010

Emily S. Patterson and Janet E. Miller have asserted their right under the Copyright, Designs
and Patents Act, 1988, to be identified as the editors of this work.

All rights reserved. No part of this book may be reprinted or reproduced or utilised in any
form or by any electronic, mechanical, or other means, now known or hereafter invented,
including photocopying and recording, or in any information storage or retrieval system,
without permission in writing from the publishers.

Notice:
Product or corporate names may be trademarks or registered trademarks, and are used only
for identification and explanation without intent to infringe.

British Library Cataloguing in Publication Data

Macrocognition metrics and scenarios : design and evaluation for real-world teams.
1. Group problem solving. 2. Teams in the workplace--Evaluation.
I. Patterson, Emily S. II. Miller, Janet E.
658.4'036-dc22

Library of Congress Cataloging-in-Publication Data

Macrocognition metrics and scenarios : design and evaluation for real-world teams /
edited by by emily s. Patterson and Janet e. Miller.
p. cm.
Includes bibliographical references and index.
ISBN 978-0-7546-7578-5 (hardback) 1. Group decision making--Evaluation.
2. Cognition--Measurement. 3. Teams in the workplace--Evaluation. I. Patterson, Emily
S., 1972- II. Miller, Janet E., 1952-
HM746.M33 2010
658.4'036072--dc22
2010000799

ISBN: 978-0-7546-7578-5 (hbk)

ISBN: 978-1-3155-9317-3 (ebk)
 Contents

List of Figures vii

List of Tables ix
Acknowledgments   xi
List of Contributors   xiii
Preface by Emily S. Patterson, Janet E. Miller, Emilie M. Roth, and
David D. Woods   xxiii

Part 1 Theoretical Foundations

1 Theory → Concepts → Measures but Policies → Metrics   3

Robert R. Hoffman

2 Some Challenges for Macrocognitive Measurement   11

Robert R. Hoffman

3 Measuring Macrocognition in Teams: Some Insights for

Navigating the Complexities   29
C. Shawn Burke, Eduardo Salas, Kimberly Smith-Jentsch,
and Michael A. Rosen

Part II Macrocognition Measures for Real-world

Teams

4 Macrocognitive Measures for Evaluating Cognitive Work   47

Gary Klein

5 Measuring Attributes of Rigor in Information Analysis   65

Daniel J. Zelik, Emily S. Patterson, and David D. Woods

6 Assessing Expertise When Performance Exceeds Perfection   85

James Shanteau, Brian Friel, Rick P. Thomas, John Raacke,
and David J. Weiss

7 Demand Calibration in Multitask Environments: Interactions

of Micro and Macrocognition   95
John D. Lee

8 Assessment of Intent in Macrocognitive Systems   109

Lawrence G. Shattuck
vi Macrocognition Metrics and Scenarios

9 Survey of Healthcare Teamwork Rating Tools: Reliability,

Validity, Ease of Use, and Diagnostic Efficacy   123
Barbara Künzle, Yan Xiao, Anne M. Miller, and Colin Mackenzie

10 Measurement Approaches for Transfers of Work During

Handoffs    137
Emily S. Patterson and Robert L. Wears

11 The Pragmatics of Communication-based Methods for

Measuring Macrocognition   161
Nancy J. Cooke and Jamie C. Gorman

12 From Data, to Information, to Knowledge: Measuring

Knowledge Building in the Context of Collaborative
Cognition   179
Stephen M. Fiore, John Elias, Eduardo Salas, Norman W. Warner,
and Michael P. Letsky

Part III Scenario-based Evaluation Approaches

13 Forging New Evaluation Paradigms: Beyond Statistical

Generalization   203
Emilie M. Roth and Robert G. Eggleston

14 Facets of Complexity in Situated Work   221

Emily S. Patterson, Emilie M. Roth, and David D. Woods

15 Evaluating the Resilience of a Human-Computer Decision-

making Team: A Methodology for Decision-Centered Testing  253
Scott S. Potter and Robert Rousseau

16 Synthetic Task Environments: Measuring Macrocognition   271

John M. Flach, Daniel Schwartz, April M. Courtice, Kyle Behymer,
and Wayne Shebilske

17 System Evaluation Using the Cognitive Performance

Indicators   285
Sterling L. Wiggins and Donald A. Cox

Index   303
 List of Figures

Figure 1.1 This concept map explains the derivation going from observations
to classifications, to conceptual measurables, to measures, to
measurements, and finally to metrics   8
Figure 4.1 Macrocognition model   48
Figure 5.1 Attributes of rigorous analysis organized with respect to
indicators of low, moderate, and high rigor   68
Figure 5.2 Scoring the rigor of the analysis processes of teams in the first
example, with differences between raters highlighted   74
Figure 5.3 Scoring the rigor of the analysis processes of participants in
the second example, with brief rationales for each rating   76
Figure 5.4 Graphical representation of the rigor attribute scores shown in
Figure 5.3   77
Figure 6.1 Mean log CWS values for three density levels for restricted/
non-restricted airspace   88
Figure 6.2 CWS scores for low-density condition for trials for which
there were no errors    90
Figure 6.3 Mean Time-Through-Sector (TTS) values for low-density
condition for trials for which there were no errors    90
Figure 7.1 Multilevel description of multitask demand management    100
Figure 8.1 The role of intent in a macrocognitive system    113
Figure 8.2 Possible outcomes when local agents respond to the possible
future scenarios posed by the remote supervisors    119
Figure 11.1 Push-to-talk communication system   164
Figure 11.2 Illustration of low-pass filter applied to sound pressure
activity to generate a communication flow log   165
Figure 11.3 Chat interface   165
Figure 11.4 A. Summary of communication events over one hour;
B. Temporal layout of the same communication events
over one hour   168
Figure 11.5 A. Hypothesized model of team collaboration; B. Data
interpreted in light of the hypothesized model   169
Figure 11.6 Tradeoff between data expense and data value for different
types of data and measures   173
Figure 11.7 Strategy of progressive deepening of communication
analysis   173
Figure 12.1 Knowledge-building process in macrocognition   193
viii Macrocognition Metrics and Scenarios

Figure 15.1 Applying cognitive pressure against a hypothesized edge

vs. generic cognitive pressure. The breaking point at the
edge will be significantly different from the limiting case
(not at an edge)   259
Figure 15.2 Specific region of the Functional Abstraction Network for
the micro-world work domain highlighting the multiple
supported goals in need of observability   262
Figure 15.3 Decision-making results for the three DSSs. Percent correct
for each of the scenarios is plotted for each of the durations
used in the experiment. The three graphs are vertically
aligned to permit visual comparisons across the DSSs   264
Figure 15.4 Confidence intervals for the three DSSs. These were computed
at the performance threshold (60% correct). Note that
the confidence bars are staggered in the graph to aid in
discrimination   266
Figure 17.1 Finding from NWS system evaluation conducted by the
Klein team   297
Figure 17.2 Finding from NWS system evaluation conducted by the
Klein team (cont.)   297
Figure 17.3 Demand-Capacity Management System   299
 List of Tables

Table P.1 Overview of macrocognition functions and measures   xxix

Table 2.1 Measures listed in the NAVSEA 2003 Human Performance
Technical Metrics Guide   13
Table 2.2 Measures and predictions from a study of performance using
a new work method   20
Table 2.3 Multiple macrocognitive measures for intelligence analysis
performance   21
Table 2.4 Some of the dimensions of difficulty and their implications for
macrocognitive measurement   23
Table 3.1 Specification of macrocognitive dimensions and subdimensions 33
Table 3.2 Potential methods used to assess macrocognition   39
Table 7.1 Types of control for each time horizon page   102
Table 9.1 Overview of observational rating tools for teamwork   126
Table 10.1 Alternative conceptual framings for handoffs   139
Table 10.2 Potential outcome measures for a handoff   144
Table 10.3 Standardized handoff content across all hospital settings   145
Table 10.4 Standardized handoff content for four operating room settings 146
Table 11.1 Classification of communication measures   167
Table 11.2 Minimum and preferable communication data characteristics
for collecting OPD for examining team collaboration   172
Table 11.3 Common pitfalls in the analysis of communications   175
Table 11.4 Guidelines for the effective analysis of team communications 175
Table 12.1 Definitions and examples of data, information, and knowledge 191
Table 13.1 Mismatches between traditional evaluation paradigms and the
goals and pragmatic constraints of joint cognitive system
evaluations and how they are addressed in the work-centered
evaluation approach   215
Table 14.1 Levels of scenario descriptions   225
Table 14.2 Factors that increase scenario complexity   231
Table 14.3 Clusters of complicating factors   246
Table 15.1 Response thresholds for the three DSSs and the three scenarios265
Table 17.1 Sample of the expert practitioners and various approaches
studied   288
Table 17.2 The cognitive performance indicators   290
This page has been left blank intentionally
 Acknowledgments

Support for book editing and a two-day conference on Macrocognitive Metrics

in Columbus, Ohio, at The Ohio State University on June 14, 2007, was provided
by the Air Force Research Laboratory (S110000012). The views expressed in this
book are those of the authors and do not necessarily represent those of the Air Force.
We thank Christopher P. Nemeth for sharing a book index from Healthcare Team
Communication, Michael W. Smith for creating the book index, and Fernando
Bernal for designing the book cover. We thank all of the chapter reviewers, and
in particular Robert Hoffman, for his thoughtful and detailed reviews of many of
the book chapters.
This page has been left blank intentionally
 List of Contributors

Kyle Behymer is currently a Senior Researcher at 361 Interactive, LLC. Mr.

Behymer received his M.S. in Human Factors Psychology in 2005 from Wright
State University. From 2003–2006, he was a Research Psychologist at JST
Applications in the defense and space industry. Mr. Behymer’s research interests
are human factors psychology and computer information systems.

C. Shawn Burke is a Research Scientist at the Institute for Simulation and

Training at the University of Central Florida. Dr. Burke earned her doctorate in
Industrial and Organizational Psychology from George Mason University. Her
expertise includes teams and their leadership, team adaptability, team training,
measurement, evaluation, and team effectiveness. Dr. Burke has published more
than 60 journal articles and book chapters related to these topics and has had work
accepted at more than 100 peer-reviewed conferences.

Nancy J. Cooke is a Professor of Applied Psychology at Arizona State University

and is Science Director of the Cognitive Engineering Research Institute in Mesa,
AZ. She is also Section Editor of Human Factors, a member-at-large of the
Human Factors and Ergonomics Society’s Executive Council, a member of the
National Research Council’s Committee on Human Systems Integration, and a
member of the US Air Force Scientific Advisory Board. Dr. Cooke specializes in
the development, application, and evaluation of methodologies to elicit and assess
individual and team cognition.

April B. Courtice is a Research Psychologist at the Air Force Research Laboratory.

Ms. Courtice received her M.S. in Psychology from Wright State University in
2009. She is a member of the Human Factors and Ergonomics Society and her
main research interest is cognitive engineering and decision making.

Donald A. Cox is a Senior Scientist at Klein Associates Division of Applied

Research Associates. Mr. Cox completed his M.Sc. in Computer Science at the
University of Calgary. Mr. Cox has been working with computers for over 25 years
in design, implementation, evaluation, and researching of systems and development
methods with a special interest in qualitative methods. At Klein Associates, Mr.
Cox is currently involved in projects evaluating advanced decision support for
planning, making ethnographic insights more available to system developers,
and characterizing the impact of new technology of submarine command team
functioning.
xiv Macrocognition Metrics and Scenarios

Robert G. Eggleston is a Principal Research Scientist affiliated with the Air Force
Research Laboratory with over 30 years of fundamental and applied research
experience on human performance in complex work. He is currently leading the
theoretical and applied development of work-centered support technology from
a joint cognitive systems perspective, addressing the contextualized work needs
over the range of orderly (simple, complicated), to unruly (complex and chaotic)
military work problem situations.

John Elias is a Graduate Student in the Modeling and Simulation program at

the University of Central Florida’s Institute for Simulation and Training. His
education includes studies in molecular biology, literature, and philosophy. His
chief research interests lie in the interdisciplinary cognitive sciences, with focus
specifically on intersubjectivity and its relation to interaction.

Stephen M. Fiore is an Assistant Professor of Cognitive Sciences at the

University of Central Florida with the Department of Philosophy and Director of
the Cognitive Sciences Laboratory at UCF’s Institute for Simulation and Training.
Dr. Fiore earned his doctorate in Cognitive Psychology from the University of
Pittsburgh. His multidisciplinary research interests incorporate aspects of the
cognitive, social, and computational sciences in the investigation of learning and
performance in individuals and teams. Dr. Fiore has published more than 100
scholarly publications in the area of learning, memory, and problem solving at the
individual and the group level.

John M. Flach is Professor and Chair of the Department of Psychology at Wright

State University. He received his Ph.D. in Human Experimental Psychology from
The Ohio State University in 1984. His current research interests are perceptual-
motor skill, manual control, decision making, cognitive systems engineering,
human-machine interface design (particularly in the domains of aviation, medicine,
and virtual reality), and ecological psychology.

Brian Friel is an Associate Professor in the Psychology Department at Delaware

State University. He holds a Ph.D. in Experimental Psychology from Kansas State
University and a M.S. in Cognitive Psychology from the University of Oklahoma.
His research interests include mental processes in decision making, measures of
performance, reading and language processes, and bilingualism.

Jamie C. Gorman is a Postdoctoral Research Associate at Arizona State University

and the Cognitive Engineering Research Institute. He completed his Ph.D. in
Cognitive Psychology at New Mexico State University. His research interests
are team cognition, communication, and the application of dynamical systems
methods. Dr. Gorman is currently conducting research on team coordination and
collaboration, real-time dynamics, and the deterministic structure of language.
 List of Contributors xv

Robert R. Hoffman is a Senior Research Scientist at the Institute for Human

and Machine Cognition. Dr. Hoffman is recognized as one of the world leaders
in the field of cognitive systems engineering and human-centered computing. He
is a Fellow of the Association for Psychological Science and a Fulbright Scholar.
His Ph.D. is in Experimental Psychology from the University of Cincinnati,
where he received McMicken Scholar, Psi Chi, and Delta Tau Kappa Honors.
Following a Postdoctoral Associateship at the Center for Research on Human
Learning at the University of Minnesota, Dr. Hoffman joined the faculty of the
Institute for Advanced Psychological Studies at Adelphi University. He began
his career as a psycholinguist, and founded the journal Metaphor and Symbol.
His subsequent research leveraged the psycholinguistics background in the study
of methods for eliciting the knowledge of domain experts. Hoffman has been
recognized internationally in disciplines including psychology, remote sensing,
weather forecasting, and artificial intelligence, for his research on human factors
in remote sensing, his work in the psychology of expertise and the methodology
of cognitive task analysis, and for his work on HCC issues intelligent systems
technology and the design of macrocognitive work systems. Dr. Hoffman is a Co-
Editor for the Department on Human-Centered Computing in IEEE: Intelligent
Systems. He is Editor for the book series, Expertise: Research and Applications.
He is a co-founder and Track Editor for the Journal of Cognitive Engineering and
Decision Making. His major current projects involve evaluating the effectiveness
of knowledge management, and performance measurement for macrocognitive
work systems.

Gary Klein is a Principal Scientist at Applied Research Associates. He was

instrumental in founding the field of Naturalistic Decision Making. Dr. Klein
received his Ph.D. in Experimental Psychology from the University of Pittsburgh
in 1969. He was an Assistant Professor of Psychology at Oakland University
(1970–1974) and worked as a Research Psychologist for the US Air Force (1974–
1978). The R&D company he founded in 1978, Klein Associates, was acquired
by Applied Research Associates in 2005. He has written: Sources of Power: How
People Make Decisions (1998); The Power of Intuition (2004); Working Minds:
A Practitioner’s Guide to Cognitive Task Analysis (Crandall et al., 2006); and
Streetlights and Shadows: Searching for the Keys to Adaptive Decision Making
(2009). Dr. Klein developed a Recognition-Primed Decision (RPD) model to
describe how people actually make decisions in natural settings. He also developed
methods of Cognitive Task Analysis for uncovering the tacit knowledge that goes
into decision making. He was selected as a Fellow of Division 19 of the American
Psychological Association in 2006, and in 2008 he received the Jack A. Kraft
Innovator Award from the Human Factors and Ergonomics Society.

Barbara Künzle is a Work and Organizational Psychologist at the Swiss Federal

Institute of Technology, Zurich, Switzerland and holds a Ph.D. in Psychology. She
did her Ph.D. research on anesthesia teams with the Department of Management,
xvi Macrocognition Metrics and Scenarios

Technology. She published several articles on leadership and coordination modes.

Her current research is focused on leadership, coordination, and teamwork in
healthcare teams.

John D. Lee is a Professor in the Department of Industrial and Systems Engineering

at the University of Wisconsin, Madison. Previously he was with the University of
Iowa, and was the Director of Human Factors Research at the National Advanced
Driving Simulator. Before moving to the University of Iowa, he was a Research
Scientist at the Battelle Human Factors Transportation Center for six years. He
received his Ph.D. from the Department of Mechanical and Industrial Engineering
at the University of Illinois. Professor Lee’s research focuses on the safety and
acceptance of complex human-machine systems by considering how technology
mediates attention. Specific research interests include trust in technology, advanced
driver assistance systems, and driver distraction.

Michael P. Letsky is currently Program Officer for the Collaboration and

Knowledge Interoperability Program at the Office of Naval Research (ONR),
Arlington, VA. He manages a research program of academic grants and innovative
small business projects seeking to understand team cognition and team performance.
He previously worked for the Army Research Institute of the Behavioral and Social
Sciences where he developed their long-range strategic research plan and also
served on the Army Science Board on Highly Maneuverable Forces. Dr. Letsky’s
education includes a B.S. in Electrical Engineering (Northeastern University), an
MBA and DBA in Operations Research (George Washington University).

Colin Mackenzie is a Physician, graduating from Aberdeen University, Scotland

in 1968. He is currently Professor of Anesthesiology, Associate Professor of
Physiology, University of Maryland School of Medicine. His research interests
include human factors in emergencies and trauma resuscitation. He has been
continuously funded by federal grants for the past 18 years, and has over 120 peer-
reviewed publications.

Anne M. Miller is an Assistant Professor in the School of Nursing at Vanderbilt

University. She received her R.N. (Royal Melbourne Hospital) and her Ph.D.
in Psychology at the University of Queensland. She has numerous scientific
publications and has served as Principal Investigator on a number of funded
grants. Her current research interests are resilience engineering for healthcare
environments, clinical decision making and continuity of clinical care, and clinical
information design.

Janet E. Miller is a Senior Electronics Engineer at the Air Force Research

Laboratory, Sensors Directorate at Wright-Patterson Air Force Base, Dayton,
Ohio. Dr. Miller received a Ph.D. in Industrial and Systems Engineering from The
Ohio State University in 2002. Currently, Dr. Miller is advancing the state-of-the-
 List of Contributors xvii

art in macrocognition research and application in military situations as well as in

trust in complex, analytic technologies.

Emily S. Patterson is an Assistant Professor in the Health Information

Management and Systems Division at The Ohio State University Medical
Center, School of Allied Medical Professions. Dr. Patterson received a Ph.D. in
Industrial and Systems Engineering from The Ohio State University in 1999. Her
professional focus is in applying human factors knowledge methods to improve the
design of complex, socio-technical settings, and particularly in human-computer
interaction and computer-supported cooperative work. She has served as Principal
Investigator on a number of federal grants and contracts. In 2004, Dr. Patterson
received the Alexander C. Williams, Jr., Design Award from the Human Factors
and Ergonomics Society, and in 2008 she received the Lumley Research Award
from The Ohio State University. Dr. Patterson currently serves as Associate Editor
for IEEE Systems, Man, and Cybernetics Part A, on the advisory board for the Joint
Commission Journal on Quality and Patient Safety, and formerly on the editorial
board for the Human Factors journal. She currently serves as an advisory board
member to the Center for Innovation of the National Board of Medical Examiners
and formerly to the Joint Commission International Center for Patient Safety.

Scott S. Potter is a Principal Scientist with Charles River Analytics. He completed

his Ph.D. in Cognitive Systems Engineering at The Ohio State University. He
has 15 years experience in applying cognitive work analysis as the underlying
basis for the design of decision-support tools for complex, dynamic, high-risk
environments. Before joining Charles River Analytics in April of 2007, he was
Principal Engineer at Aegis Research Corp. for seven years; for the six years prior,
he was a Principal Engineer with Carnegie Group.

John Raacke is the Associate Dean in the College of Arts & Sciences and an
Assistant Professor of Psychology at the University of North Carolina at
Pembroke. He completed his Ph.D. in Experimental/Cognitive Psychology at
Kansas State University with emphasis in Judgment and Decision-Making. Dr.
Raacke’s research has examined decision making in a number of applied areas
of psychology. Most recently, Dr. Raacke’s studies have dealt with investigating
expertise, team development in dynamic environments, juror decision making, the
use of conditional probabilities, and social networking.

Michael A. Rosen is a Doctoral Candidate in the Applied Experimental and

Human Factors Psychology program at the University of Central Florida and since
fall 2004 has been a Graduate Research Assistant at the Institute for Simulation
and Training, where he won the Student Researcher of the Year in 2006. His
research interests include individual and team decision making and problem
solving, human-computer interaction, team performance, and training in high-
stakes domains such as healthcare and the military.
xviii Macrocognition Metrics and Scenarios

Emilie M. Roth is the Owner and Principal Scientist of Roth Cognitive

Engineering. She has been involved in cognitive systems analysis and design in a
variety of domains including nuclear power plant operations, railroad operations,
military command and control, medical operating rooms, and intelligence
analysis. She received her Ph.D. in Cognitive Psychology from the University
of Illinois at Champaign-Urbana in 1980. She currently serves on the editorial
board of the journals Human Factors and Le Travail Humain, and is Editor of the
Design of Complex and Joint Cognitive Systems track of the Journal of Cognitive
Engineering and Decision Making.

Robert Rousseau is an Emeritus Professor at Université Laval and currently

Director of C3, a firm specializing in Research and Development (R&D) in
cognitive systems engineering. As a Senior Defence Scientist at DRDC Valcartier
between 2003 and 2006, he led projects on cognitive issues in evaluation of
Decision Support Systems (DSS), decision making in command and control (C2),
and situation awareness. He has extensive expertise on cognitive aspects of work
automation, information systems implementation, change management and team
decision making. He was for many years Chair of the School of Psychology at
Université Laval.

Eduardo Salas is Trustee Chair and Professor of Psychology at the University of

Central Florida where he also holds an appointment as Program Director for the
Human Systems Integration Research Department at the Institute for Simulation
and Training (IST). Before joining IST, he was a Senior Research Psychologist
and Head of the Training Technology Development Branch of Naval Air Warfare
Center Training Systems Division for 15 years. Dr. Salas has co-authored over 300
journal articles and book chapters and has co-edited 19 books. His expertise includes
teamwork, team training strategies, training effectiveness, decision making under
stress, and performance measurement tools. Dr. Salas is a Fellow of the American
Psychological Association, the Human Factors and Ergonomics Society, President-
Elect of the Society for Industrial and Organizational Psychology, and a recipient
of the Meritorious Civil Service Award from the Department of the Navy.

Daniel Schwartz is a Consumer Psychologist at Procter & Gamble. He received

his Ph.D. in Experimental Psychology from Wright State University. From 2005
until 2008, he was a Research Fellow at the Air Force Research Laboratory. Dr.
Schwartz’s current interests are consumer understanding, product research and
development, and social media.

James Shanteau is a University Distinguished Professor of Psychology and

Commerce Bank Distinguished Graduate Faculty. Professor Shanteau received his
Ph.D. in Experimental Psychology from University of California, San Diego. Since
joining the faculty at Kansas State University, he has held visiting appointments
at Universities of Michigan, Oregon, Colorado, Cornell, Toulouse, and National
 List of Contributors xix

Science Foundation. His research interests include studies of expertise (especially

medical decision making) and studies of consumer healthcare choices (especially
organ donation and transplantation). Professor Shanteau has received over $5.9
million from agencies such as National Institute of Mental Health, Division of Organ
Transplantation, National Institute of Child Health and Human Development, and
National Science Foundation. His publications include over 75 articles in referred
journals, 10 books, 65 book chapters, 6 encyclopedia entries, 7 monographs, 21
proceedings papers, 14 technical reports, and 3 computer programs. He served
on grant-review panels for the National Institute of Health, Environmental
Protection Agency, and National Science Foundation. He served on a National
Research Council Committee to redefine mental retardation standards, and on a
DHHS Advisory Committee on Organ Transplantation (ACOT). He is a Fellow
of the American Psychological Association and a Charter Fellow of the American
Psychological Society.

Lawrence G. Shattuck is a Senior Lecturer and Director of the Human Systems

Integration Program at the Naval Postgraduate School in Monterey, CA. Dr.
Shattuck received his Ph.D. in Cognitive Systems Engineering from the Ohio State
University in 1995. He is a retired Army Colonel with 30 years of service. During
his last assignment on active duty (1995–2005) he was a Professor and Director
of the Engineering Psychology Program at the United States Military Academy,
West Point, NY. His research interests include communication of intent, military
command and control, human-computer interaction, and human error.

Wayne Shebilske is a Professor in Psychology at Wright State University. Dr.

Shebilske received his Ph.D. in Psychology at the University of Wisconsin,
Madison, in 1974. After nine years on the faculty of the University of Virginia
Psychology Department, Dr. Shebilske served as Study Director for the Committee
on Vision at the National Academy of Sciences. Dr. Shebilske was a Professor at
Texas A&M University where he was the Graduate Program Advisor. In 1992,
he was awarded a National Research Council Senior Research Associateship at
the Armstrong Laboratory Intelligent Training Branch. Dr. Shebilske has worked
closely with government, military, and private agencies identifying critical issues
for study in many areas, including design and development of aerospace systems,
visual display equipment, medical devices, standards for pilots and drivers,
automated instruction for complex skills, and virtual reality systems.

Kimberly Smith-Jentsch is an Assistant Professor in Industrial and Organizational

Psychology at the University of Central Florida. She received her Ph.D. in
Industrial and Organizational Psychology from University of South Florida in
1994. She has earned a number of awards, including the M. Scott Myers Award
for Applied Research in the Workplace (2001), the Dr. Arthur E. Bisson Award
for Naval Technology Achievement (2000), and the NAVAIR Senior Scientist
xx Macrocognition Metrics and Scenarios

Award (2000). Her research interests include mentoring, teamwork, leadership,

assertiveness, training and development, and performance measurement.

Rick P. Thomas is an Assistant Professor in Psychology at the University of

Oklahoma. His primary research focus is in the emerging field of cognitive decision
theory. This field integrates theory and methods from cognitive and ecological
psychology with behavioral decision theory. Most of Professor Thomas’s research
in cognitive decision theory relies on computational models of memory and
knowledge acquisition to account for judgment and decision-making phenomena.
For example, he is currently investigating how a modified version of MINERVA-
DM, HyGene, accounts for the effects of hypothesis generation on probability
judgments, confidence judgments, and hypothesis testing (Thomas, Dougherty,
Springler & Harbison, 2008; http://www.thehygeneproject.org/). Professor Thomas
also does work concerning the study of expertise (Thomas et al., 2001; http://www.
k-state.edu/psych/cws/team.htm), primarily in the areas of performance evaluation
and the development of decision-support tools. For example, he has conducted
both basic and applied research to develop and evaluate decision-support tools to
aid pilot navigation through convective weather (Burgess & Thomas, 2004; Elgin
& Thomas, 2004).

Norman W. Warner is a Senior Scientist at the Naval Air Systems Command,

Patuxent River, MD. His Ph.D. is from the University of South Dakota in Human
Factors Engineering. His expertise is in automated decision support, human decision
making, and team collaboration. Over the past 28 years, he has conducted research
with a variety of organizations (for example, Naval Air Systems Command, Office
of Naval Research, Air Force and Army). Dr. Warner was awarded two patents
in decision-support technology and published over 25 peer-reviewed articles.
Currently, his research focus is on understanding team decision making and the
cognitive mechanisms used during asynchronous, distributed team collaboration.

Robert L. Wears is an Emergency Physician, Professor in the Department of

Emergency Medicine at the University of Florida, and Visiting Professor in the
Clinical Safety Research Unit at Imperial College London. He serves on the board
of directors of the Emergency Medicine Patient Safety Foundation, and on the
editorial board for Annals of Emergency Medicine. He is also on the editorial
board of Human Factors and Ergonomics, the Journal of Patient Safety, and the
International Journal of Risk and Safety in Medicine. Dr. Wears has been an active
writer and researcher with interests in technical work studies, joint cognitive
systems, and particularly the impact of information technology on safety and
resilient performance. His work has been funded by the Agency for Healthcare
Research and Quality, the National Patient Safety Foundation, the Emergency
Medicine Foundation, the Society for Academic Emergency Medicine, the Army
Research Laboratory, and the Florida Agency for Health Care Administration.
 List of Contributors xxi

David J. Weiss, the co-developer of the CWS index, received his bachelor’s
degree from the University of Pennsylvania and his doctorate from University of
California, San Diego. He has spent his entire teaching career in the Psychology
Department at California State University, Los Angeles. He has published
extensively in the area of judgment and decision making, and also has developed
several new statistical procedures. Much of his empirical work has employed
functional measurement methodology, and his textbook Analysis of Variance
and Functional Measurement (Oxford University Press, 2006) includes the only
elementary exposition of that approach. The text also includes the CALSTAT suite
of user-friendly computer programs he developed. Together with spouse Jie W.
Weiss, he edited A Science of Decision Making: The Legacy of Ward Edwards
(Oxford University Press, 2009).

Sterling L. Wiggins is a Principal Scientist at the Klein Associates Division of

Applied Research Associates. He completed his M.A. in Education with a focus
on Learning, Design, and Technology at Stanford University. Mr. Wiggins has
15 years experience in research and development at the convergence of humans,
technology, and work in high-consequence environments. Mr. Wiggins is currently
active in projects advancing human sensemaking through system design, and
developing augmented reality applications for small unit leaders in the military.

David D. Woods is a Professor at Ohio State University in the Department of

Integrated Systems Engineering. He received his Ph.D. in Psychology from
Purdue University in 1979. From his initial work following the Three Mile Island
accident in nuclear power, to studies of coordination breakdowns between people
and automation in aviation accidents, to his role in today’s national debates about
patient safety, he has studied how human and team cognition contributes to success
and failure in complex, high-risk systems. He is author of Behind Human Error,
received the Jack A. Kraft Innovator Award from Human Factors and Ergonomic
Society for advancing cognitive engineering and its application to safer systems,
and received a Laurels Award from Aviation Week and Space Technology (1995)
on the human factors of highly-automated cockpits. He has served on a National
Academy of Engineering/Institute of Medicine Study Panel applying engineering
to improve healthcare systems and on a National Research Council panel on
research to define the future of the national air transportation system.

Yan Xiao is directing patient safety research at Baylor Health Care System,
Dallas, TX. After finishing doctoral training in Human Factors from University
of Toronto in 1994, Dr. Xiao joined University of Maryland School of Medicine
to conduct research in communication, coordination, information technology,
team performance, and clinical alarms. Most recently he was a Tenured Professor
of Anesthesiology at University of Maryland School of Medicine in Baltimore,
MD. Dr. Xiao is the author of more than 130 journal and proceeding articles
and book chapters. His current research has been supported by National Science
xxii Macrocognition Metrics and Scenarios

Foundation, National Patient Safety Foundation, and Agency for Healthcare

Research and Quality. His current research interests include team communication
and technology-supported coordination.

Daniel J. Zelik is a Doctoral Candidate in the Cognitive Systems Engineering

program at The Ohio State University. He has experience analyzing and modeling
human performance as a co-op student at NASA Johnson Space Center. He
currently investigates how professional intelligence analysts cope with data
overload and infuse rigor into their analytical processes, as well as strategies for
creating insight into computer-supported information analysis activity.
 Preface
Macrocognition: Where Do We Stand?
Emily S. Patterson, Janet E. Miller, Emilie M. Roth, and David D. Woods

Real-world settings require effective teamwork to accomplish the objectives of an

organization. Drastic improvements in performance are anticipated by many over
the next decade via the implementation of technological innovations that facilitate
teamwork. With many of the anticipated innovations, the concept of a team
will be substantially broadened to include physically and temporally distributed
members, ad hoc groups that form and then disband once an objective has been
met, and anonymous interactions with others who have common or interacting
goals without any sense of shared identity.
A daunting challenge is accurately predicting which innovations will truly
improve team performance in a real-world setting, particularly for primarily
cognitive (as opposed to physical, which are usually directly observable) activities.
Most of the traditional human performance and cognitive process measures
were developed for individuals working on relatively simple tasks without
sophisticated computerized decision aids or communication networks. Without
suitable measures and evaluation methodologies, how can we predict in advance
whether the benefits of a proposed innovation justify the costs of implementation
and maintenance? This book advances the state-of-the-art in team process and
performance measurement, specifically for real-world teams in complex settings,
to evaluate the usefulness of a proposed innovation and compiles recent exciting
advances by prominent macrocognition researchers.

What is Macrocognition?

The short definition of macrocognition is ‘the way we think in complex situations’

(Klein et al., 2003; Chapter 4). The ‘macro’ in macrocognition emphasizes that
cognition in real-world settings occurs across multiple individuals using tools and
shared resources, as contrasted with ‘micro’cognition done ‘between the ears’ of
an isolated individual at a discrete point in time (Woods, 1987; Woods & Roth,
1986; Roth et al., 1986).
Complexity in modern work settings is likely to increase in the future due to a
number of interrelated drivers. Technological and communication innovations are
the primary motivation for explicitly considering the ‘macro’ level of cognition for
study in its own right. For instance, recent technological innovations have provided
xxiv Macrocognition Metrics and Scenarios

new opportunities to extend cognitive capacity, including advanced sensors,

automation, alerting algorithms, digital display formats (‘visual analytics’), and
massive digital data storage capacity. Advances in communications technologies
are already transforming how cognitive work is conducted, including new
communication formats such as Instant Messaging, Chat, and Twitter; dedicated
portable handheld devices like Kindles and BlackBerries; additional features
on existing devices such as cellular phones with embedded digital cameras,
Global Positioning Systems (GPS), and email and Internet connectivity; shared
information repositories that can be accessed and updated real-time simultaneously
by multiple users like Wikipedia and blogs; and access to specialized expertise that
is not within organizational or personal networks, such as Ask-An-Expert services.
Due to the technological and communication innovations as well as other factors,
the complexity of real-world environments is anticipated to increase, with more
interconnections (‘coupling’), a faster pace of change, more integration of diverse
software packages and hardware, and higher expectations for productivity (‘faster
better cheaper’).
Macrocognition is a relatively new term—in fact, new enough that some still
question whether the term is needed—and the associated theoretical frameworks,
concepts, terms, and definitions are a rapidly moving target. This chapter takes
stock of the macrocognition research in this book and answers the question: Where
do we stand? An important contribution of the creation of this book is clarifying
the scope of macrocognition and identifying areas of consensus and disagreement
among the community.

Historical Influences on Macrocognition Research

The term macrocognition can be viewed as a natural evolution and aggregation of

several lines of research that have mutually influenced each other for the last few
decades:

• Cognitive Systems Engineering (CSE) was introduced as a field in the

1980s with a focus on ‘cognitive adaptations to complexity’ (Rasmussen
& Lind, 1981; Schraagen et al., in press) and ‘coping with complexity’
(Woods, 1988).
• Ethnographics observations of ‘cognition in the wild’ (Hutchins, 1995)
revealed a reliance on specialized tools to ‘externalize cognition’ and make
aspects of cognition public in order to aid coordination across individuals
in well-defined roles.
• ‘Situated action’ research (Suchman, 1987) emphasized how events in the
world (‘event-driven’ as opposed to ‘self-paced’ worlds) challenge the
implementation of plans, thereby suggesting that plans are better viewed as
‘resources for action’ rather than hard constraints on activity.
 Preface xxv

• A substantial body of research on ‘naturalistic decision making’ and

decision making under uncertainty has revealed a number of phenomena,
including that decision making in complex settings differs substantially
from decision making in traditional psychology laboratory-based studies
with simpler tasks (Klein et al., 1993; Orasanu & Connelly, 1993), that
expert practitioners take ‘shortcuts’ from recognizing a situation to
implementing default procedural responses without evaluating alternative
plans (Rasmussen et al., 1991), and that commitments to irrevocable courses
of action are often strategically delayed to increase flexibility, particularly
in overconstrained situations (Pearl, 1982).
• Research on anomaly response in high-consequence worlds such as NASA
Johnson Space Center revealed that ‘safing’ activities were often performed
in parallel with activities aimed at diagnosing problems and that low-
probability, high-consequence explanations for data were often treated as
the actual explanation in parallel with the highest probability explanation
(Chapter 8 in Woods & Hollnagel, 2006).
• Research on human expertise has emphasized the importance of differences
between novice and expert strategies in problem solving (Feltovich et
al., 1997) and the importance of knowledge building to foster shared
understandings (‘shared mental models’, ’team situation awareness’) across
team members (Letsky et al., 2008; Salas & Fiore, 2004).
• One way to frame shared resources in a distributed setting is as potential
bottlenecks in cognitive processing, such as attention, memory, workload,
goals, and coordination bottlenecks (Roesler & Woods, 2007).
• An important distinction between macrocognition and more classic cognitive
models (for example, the observe-orient-decide-act (OODA) loop frequently
used by the US military, which was developed by John Boyd) is that there
is nearly always historical cognitive activity that shapes how cognitive work
is framed. For example, problems have already been framed, hypotheses
already exist as likely explanations for situations, plans are already in
progress, and actions are already being executed. Therefore, the emphasis of
the macrocognitive functions is on ‘RE’ doing the cognitive work as a result
of triggers to reconsider, revise, reframe, or otherwise change the default
activity space. For example, although there are occasions where new plans
need to be generated that are different than plans in progress and do not
match the typical procedures employed, this is not the dominating case for
(re-)planning as a macrocognitive function (Klein et al., 2003).

In macrocognition, everything relates to everything, and simplified theoretical

frameworks cannot reduce the complexity of the phenomena under study.
Therefore, while focusing on a particular activity like executing a procedure can
be important for gaining traction in conducting research, there will still be a need
to consider interactions with other activities, such as monitoring the environment
for changes, adapting a procedure to fit the dynamic situation, interpreting the
xxvi Macrocognition Metrics and Scenarios

guidance of distant supervisors as to the intent of an original procedure that cannot

be executed in order to adapt it, assessing whether a plan in progress can be done
within the anticipated window of opportunity, coordinating the implementation of
multiple procedures, and timing activities so that they do not negatively interact
with other activities (Woods & Hollnagel, 2006; Hollnagel & Woods, 2005).
In specifying the scope of macrocognition, there is a clear consensus that the core
value of the community of researchers is that cognitive work is complex (note that
this definition is inclusive of research conducted both in an actual, naturalistic work
setting as well as a humans-in-the-loop laboratory setting where complex problems
are simulated). Common elements for the research compiled in this book are:

• complex environment;
• embedded in a social context;
• artifacts require technical expertise;
• event-driven;
• dynamic demands;
• conflicting goals;
• organizational constraints;
• high consequences for failure.

There is similarly consensus among the researchers that macrocognition can

be simultaneously considered at three interacting (macro) levels of analysis, even
when there is usually an emphasis on the middle layer:

1. growth and decay of effective expertise and associated knowledge in

individuals;
2. joint activity conducted in teams; and
3. factors that facilitate or inhibit organizational adaptability and resilience.

Finally, one of the main contributions of this book has been to advance the
debate on what are the core functions in (macro)cognitive work. Although there is
unlikely to be a final, comprehensive list mutually agreed upon by the community
anytime soon, we believe that these functions and definitions represent an advance
in clarity and consensus from pioneering work published in Klein et al. (2003);
(see Chapter 4, Figure 4.1 for an updated figure summarizing macrocognitive
functions and supporting processes):

• Detecting: Noticing that events may be taking an unexpected (positive

or negative) direction that require explanation and may signal a need or
opportunity to reframe how a situation is conceptualized (sensemaking)
and/or revise ongoing plans (planning) in progress (executing). (Related
terms: detecting problems, monitoring, observe, anomaly recognition,
situation awareness, problem detection, reframing).
 Preface xxvii

• Sensemaking: Collecting, corroborating, and assembling information and

assessing how the information maps onto potential explanations; includes
generating new potential hypotheses to consider and revisiting previously
discarded hypotheses in the face of new evidence. (Related terms: orient,
analysis, assessment, situation assessment, situation awareness, explanation
assessment, hypothesis exploration, synthesis, conceptualization,
reframing.)
• Planning: Adaptively responding to changes in objectives from supervisors
and peers, obstacles, opportunities, events, or changes in predicted future
trajectories; when ready-to-hand default plans are not applicable to the
situation, this can include creating a new strategy for achieving one or more
goals or desired end states. (Related terms: replanning, flexecution, action
formulation, means-ends analysis, problem solving.)
• Executing: Converting a prespecified plan into actions within a window
of opportunity (Related terms: adapting, implementation, action, act); this
includes adapting procedures based on incomplete guidance to an evolving
situation where multiple procedures need to be coordinated, procedures
which have been started may not always be completed, or when steps in a
procedure may occur out of sequence or interact with other actions.
• Deciding: A level of commitment to one or more options that may constrain
the ability to reverse courses of action. Decision making is inherently
a continuous process conducted under time pressure that involves re-
examining embedded default decisions in ongoing plan trajectories for
the predicted impact on meeting objectives, including whether to sacrifice
decisions to which agents were previously committed based on considering
trade-offs. This function may involve a single ‘decision maker’ or require
consensus across distributed actors with different stances toward decisions.
(Related terms: decision making, decide, choice, critical thinking,
committing to a decision.)
• Coordinating: Managing interdependencies across multiple individuals
acting in roles that have common, overlapping, or interacting goals.
(Related terms: collaboration, leadership, resource allocation, tracking
interdependencies, communication, negotiation, teamwork.)

In summary, our view of the scope of macrocognition is:

• joint activity distributed over time and space;

• coordinated to meet complex, dynamic demands;
• in an uncertain, event-driven environment with conflicting goals and high
consequences for failure;
• made possible by effective expertise in roles;
• shaped by organizational (blunt end) constraints;
• that produces emergent phenomena.
xxviii Macrocognition Metrics and Scenarios

How This Book is Organized

The book is divided into three sections.

Part 1, Theoretical Foundations lays the groundwork for how measurement
theory in general is relevant to developing macrocognitive measures. In Chapter
1, Robert R. Hoffman provides an overview of terminology from measurement
theory literatures, including providing an example of moving from a theoretical
concept to an operational definition of measurement. He also argues that there
is a fundamental difference between the terms ‘measures’ and ‘metrics’ in that
‘metrics express policy that is shaped by goals and value judgments—things that
are external to the measurement process itself’ (Chapter 1, italics in original).
In Chapter 2, Robert R. Hoffman describes challenges for macrocognitive
measurement. He provides a review of lessons learned from prior attempts at
measurement, including Hits, Errors, Accuracy, Time (HEAT) measures, Human
Reliability Analysis (HRA), Cognitive Reliability and Error Analysis Method
(CREAM), and usability and learnability questionnaires. He provides a list of
desirable characteristics for macrocognitive measures, and encourages us to do
a broad exploration of possible measurement approaches: ‘Measurement for
complex cognitive systems, and for macrocognition, will certainly involve totally
new approaches, new kinds of scales, and new analytical techniques.’ (Chapter 2).
In Chapter 3, Eduardo Salas and colleagues provide a number of insights based on
their wealth of experiences measuring teamwork processes. They start by providing
guidelines for developing any measure, including: ‘(1) what to measure, (2) what
tools to use (that is, how to measure it), (3) when and from whom to collect the
information, (4) how to get useful responses (for example, maximize validity,
while limiting obtrusiveness), and (5) how to index the responses’ (Chapter 3).
In developing macrocognition measures in particular, their insights include clear
distinctions between macrocognition levels, unit of analysis, processes, emergent
states, dimensions, moderators, and outcomes.
Part 2, Macrocognition Measures for Real-World Teams details recent advances
in macrocognition measures (see Table P.1 for an overview of the nine chapters
that provide macrocognition measures). We believe that all of the measures
detailed in these chapters could be immediately applied in evaluations of proposed
technological innovations, or at least provide a significant jumpstart in developing
measures appropriate for a particular evaluation.
Part 3, Scenario-Based Evaluation Approaches describes advances in
methodologies well suited for enhancing team performance at a macro-level
of analysis in complex, dynamic settings. In Chapter 13, Emilie M. Roth and
Robert G. Eggleston propose a work-centered evaluation to address mismatches
between traditional evaluation paradigms, including usability testing and
controlled laboratory experimentation, and evaluation goals and pragmatic
constraints for studies done with a macrocognitive focus. In Chapter 14, Emily
S. Patterson, Emilie M. Roth, and David D. Woods provide guidance on how to
design evaluation scenarios that are sufficiently complex to represent the problem
Another random document with
no related content on Scribd:
“Solani grieves because he fears that harm may come to the
stranger,” he replied.
“Who would bring harm to Bomba, except perhaps Japazy?” asked
the jungle lad. “And Japazy is not here. Unless Japazy has come
since you brought the last meat to Bomba,” he added.
“Japazy is not here,” replied Solani. “But the people of the tribe are
fearful of his wrath if he should come and find the stranger here.
They think that Japazy will not be glad to hear the stranger’s words
about his father and his mother. They fear that Japazy may be angry
because they did not kill the stranger as soon as he landed on
Jaguar Island.”
Bomba’s heart sank. Was it possible that the elders, believing that
Japazy had killed the boy’s father, were fearful that he had come to
take vengeance on their chief?
To be sure, he was only a boy, but he had that very day
demonstrated that he had strength, skill, and courage that any
warrior of the tribe would have found it hard to match.
“It cannot be that the elders of the tribe would do harm to a stranger
who came with his palms out and put himself into their keeping,” said
Bomba. “That would bring on the people the wrath of the gods.”
“They do not fear the wrath of the gods as much as they fear the
wrath of Japazy,” returned Solani. “The gods are very far away.
Japazy is near.”
No better proof than this statement, simply made, could have been
afforded of the tremendous power exercised by the crafty half-breed
over the superstitious people through the strength of his personality
and his talk of ghosts and demons.
“What do the elders of the tribe say about Bomba?” questioned the
jungle lad, with an assumption of carelessness that he was far from
feeling.
“They meet to-night to take counsel,” answered Solani. “Then it will
be known what they think about the stranger and what they will do
with him.”
“Where do they meet?” asked Bomba, who was already forming a
plan of action.
“In the house of Abino, my father,” was the answer.
“Is that the house where Bomba was taken before he was brought to
the house of Japazy?” Bomba further queried.
“Yes,” replied Solani.
“It is well,” said Bomba. “Their words will be good words and they will
do no harm to the stranger that came to them in peace.”
Solani looked very doubtful as to that, but he said no more, and
Bomba himself did not prolong the conversation. He was too busy
with his thoughts.
He could readily understand that, in order to curry favor with the
tyrant, they might make a sacrifice of Bomba. It was for him to
forestall this, if possible, and to learn their plans before they were put
into execution.
“Has the stranger ever seen a cooanaradi?” asked Solani
unexpectedly, as he started to gather up the dishes and the
remnants of food.
“Yes, Bomba has seen them and slain them,” answered the jungle
boy. “They are the most wicked snakes of the jungle. Why does
Solani ask?”
But Solani acted as though he had already said too much and did
not reply. He hastily finished his task and disappeared. But as he
went out of the door he turned and looked once more at Bomba with
a long, lingering glance as though he never expected to see him
again.
The look made Bomba uneasy. What did Solani know that he had
been afraid to say? That he meant to warn him to be on his guard
was evident. Probably even in going that far the native lad had done
what he had been forbidden to do.
Bomba waited until it was fully dark. It was then within an hour of
moonrise, and darkness was needed for the plan that he had in
mind.
He went stealthily through the house, examining every room by the
light of the torch he had plucked from the wall. He thought that
perhaps some sentries were on guard to watch his movements. But
the dwelling appeared to be deserted. It had probably been decided
that any one who had braved so many perils in coming to Jaguar
Island would not be inclined to leave it before his mission had been
accomplished.
Having ascertained to his satisfaction that no one was lurking in the
shadows, he went back to his room and carefully examined his
weapons. Having found that they were in perfect order, he
extinguished the torch, after one last loving look at the pictured face
upon the wall, and slipped out of the house like a shadow.
There was scarcely any one abroad. Doubtless the fear that a jaguar
might be lurking about kept the inhabitants indoors after dark. Even if
Bomba should be met by any one, the probability was that in the
darkness he would pass as one of their own people.
With the keen observation that had been sharpened by his
experience of the jungle, he had carefully noted the location of
Abino’s house and now had no difficulty in finding it.
A flickering light from the torches within shone through the open
windows. Avoiding the part that fronted on the straggling street,
Bomba glided round to the back of the house and crouched under
one of the openings that served as windows. That the conference
was already in session was evident from the hum of conversation
that came to the unseen listener.
The first word that Bomba could distinguish was the last that he had
heard from the lips of Solani, “cooanaradi.” The coincidence sent a
shiver through his veins. That word of evil omen, the word that stood
for the fiercest and deadliest snake in the Amazonian jungle! What
significance did it have in a conversation that presumably concerned
itself with Bomba?
The house was made of rough logs with many crevices. Applying his
eyes to one of these, Bomba could see what was going on in the
room without the risk of discovery that would be involved in lifting his
head above the window sill.
In the room were gathered the same elders with whom he had talked
in the afternoon. To their number had been added the warriors,
Sunka and Boshot, who had slain the jaguar in the morning.
An expression of deep perplexity was visible on all the dark faces. It
was plain that the problem with which they had to deal was giving
them a great deal of concern.
Bomba listened intently, trying not to lose a syllable. Abino was
speaking.
“It is certain that Japazy will not wish to see the stranger,” he said.
“Who would wish to see the son of the man he had killed?”
“Is it known to Abino that Japazy killed the stranger’s father?” asked
Boshot.
“Japazy has not told it to Abino,” was the answer. “But when Japazy
has taken too much intoxicating drink he has talked sometimes to
himself and Abino’s ears have been open to his words. The stranger
thinks his father’s name was Bartow. When Japazy’s speech has
been thick he has spoken the name of Bartow and made motions
with his knife. Then he has said, ‘Bartow is dead.’ Then Japazy has
smiled as if he were glad.”
“Did Japazy kill the stranger’s mother, too?” asked Sunka.
“Abino does not know,” the native replied. “Japazy has not spoken of
that. But many times he has gone to look at the woman’s face on the
wall, and when he turned away Japazy’s face was net good to see.”
“It may be that Japazy has questions he would ask of the stranger,
as well as hear the stranger’s questions,” put in one of the elders.
“Japazy may be angry if he comes back and finds the stranger dead
so that he may not speak with him.”
“He need not know that the stranger has been here,” observed
another of the group. “We can put the people under silence so that
they will not speak of his coming.”
“That would be better than to have the stranger kill Japazy,”
remarked the eldest of the company. “The stranger speaks smooth
words, but who knows what black thoughts are in his heart? And if
he wounds but does not kill Japazy, what will Japazy do to us
because we did not kill the stranger when he came among us?”
That this consideration had weight was evident from the grave
expression of his hearers.
At this moment there was a terrific roar from the volcano and the
earth shook so violently under Bomba’s feet that he was thrown
headlong. For several minutes the quake continued and then
gradually subsided.
When Bomba peered again through the crevice he could note
consternation on all faces.
“Tamura is angry,” declared Abino solemnly. “To-day he has been
more angry than for many moons. Why is that? It is because to-day
the stranger came. He does not like the stranger. He wants that he
should die. Tamura has spoken.”
A chill went through Bomba’s veins. It had been very unlucky for him
that the convulsion should have come at the very moment that they
were debating his fate. Now, as he looked from face to face, he
could read his doom. To their minds the god of the mountain had
spoken, and it only remained for them to bow to the decree.
There were whispers now, as though what they were discussing was
too horrid to be spoken of aloud. Strain his ears as he might, Bomba
could catch but a single sentence:
“He shall die by the creeping death!”
 CHAPTER XXIII
THE FIRE STICK SPEAKS

With the utterance of their ominous words the elderly natives rose to
their feet, and it was evident that the meeting was about to break up.
It would never do to be caught lurking there, and Bomba was off like
a flash. The darkness had swallowed him up before any of the group
had come through the door.
Gliding along as silently as a ghost, he reached the dwelling of
Japazy. There was no sign of any one having been there in his
absence. Everything was as he had left it, and a deathlike silence
wrapped the house like a garment.
Reaching his room he threw himself on the bed to think. And if he
ever needed to think swiftly and think wisely, it was then.
Now he knew why Solani had looked at him so sadly and lingeringly,
as one looks on the face of the dead. The native boy had already
sensed the turn that things were taking and felt that in fact, if not in
words, the stranger’s death had been already decreed.
But sometimes there is a long way between a decree and its
execution, and Bomba, despite the awful peril in which he stood, was
by no means ready to concede that he was as good as dead. Again
and again he had faced death and escaped by a hair’s breadth. And
the quick wit and indomitable daring that had saved him then were
his now.
He rapidly reviewed the possibilities in the case. Two courses of
action seemed open to him. He could attempt to escape, to leave
this terrible island, cross the river and reach the mainland, where he
would be safe. Or he could stay and face the enemies who were
determined upon his death and try in some way to defeat their
purpose.
But if he adopted the first course, that would be an end to his chance
of finding speech with Japazy. It would be putting aside his last
chance to get some knowledge of his parents, of answering the
questions that for months past had tormented him.
The second course, Bomba had to admit to himself, was so
foolhardy as almost to smack of insanity. What chance would he
have to make head against the warriors of the tribe if it came to a
battle? He could doubtless kill some of them, but in the end they
would overwhelm him by sheer force of numbers.
But was there not a third course that offered some chance of
success? Suppose he did indeed leave the village of Japazy, but still
remain on the island?
He felt sure that Japazy had gone to the sunken city with the towers
of gold. Doubtless that was his treasure house, to which he had
managed in some way to find access and to which he went from time
to time to gather more riches. Only in that way could the priceless
objects already in this house be accounted for.
But if he had gone there, he had gone by the river and would have to
return in the same way. Bomba knew from the description of
Hondura the general direction of the sunken city. It lay to the north.
Why might not Bomba lurk about the river bank to the north and
descry Japazy’s coming? Then he could boldly make himself known
and get the interview that he had come so far to seek.
While he was immersed in these thoughts he became conscious of a
slight rustling. It was so slight that none but a trained ear like that of
Bomba’s would have detected it at all.
But he heard it and was about to leap from the bed to investigate,
when he suddenly stiffened and lay as though carved from stone.
For in a beam of moonlight that filtered through the window he had
caught sight of what looked like a long black rope dragging itself over
the sill of the door.
But the long black rope was alive!
Bomba knew what it was, knew even before he saw the glint of the
evil eyes in the wicked head. It was a cooanaradi, the most deadly
and terrible snake of the South American jungle, a species that
sometimes grows to twelve feet in length, is noted for its ferocity and
speed and its eagerness to attack man even when retreat is
possible. And its bite means certain death.
A cold sweat broke over Bomba. He did not dare move a muscle.
A cooanaradi! Now Bomba knew what Solani had meant. Now he
fathomed the meaning of that ominous phrase he had heard at the
conference of the elders:
“He shall die by the creeping death!”
The captive reptile had been brought to the house and silently
released. The Indians knew that it would find its way to Bomba’s
room by the smell of the food that had twice been served in it. Once
in the room with Bomba, the cooanaradi could be trusted to do the
rest.
The infernal cunning of the scheme came to Bomba like a flash. If
Japazy on his return should be vexed at not having spoken with
Bomba, the natives could point out that they were not responsible—
that the snake in some way had found its way into the house and
killed the lad. That would clear them from blame.
If, on the other hand, Japazy should be relieved at finding the boy
dead, the perpetrators could proudly boast their part in the deed and
count on the gratitude and perhaps the gifts of the master.
There was a horrid, slithering sound as the scaly body was dragged
over the floor. The deathlike stillness in which Bomba lay had thus
far prevented his detection. Luckily the bed lay in the shadow.
The moonlight came into the room at an angle, and it was
intercepted by the branches of a tree outside so that its beams were
cast only at intervals here and there. It was only when the rays
caught the long, black body that Bomba was able to follow the
course of the reptile about the room. At other times he had to guess
its location by the horrid rustling that accompanied its progress.
There was a possible chance, Bomba thought, that the snake, not
finding the food it sought, would curl up after a while and go to sleep.
But this hope was dissipated as he caught a glimpse of the evil eyes
that were darting in every direction. Nothing was further from that
messenger of evil than sleep.
It was only a matter of time when the reptile would reach the bed.
Then the head would be upreared. It was beyond possibility that it
would not detect the lad lying there, no matter how still he kept.
Then would come a dart like lightning and the terrible fangs, dripping
with poison, would be imbedded in the boy’s face or throat, and a
very few minutes later Bomba would be dead.
His weapons? The bow and arrow were out of the question. Long
before he could string his arrow the snake would be upon him. The
knife? That might serve at close quarters, but that meant he would
be bitten first.
The fire stick!
Slowly, imperceptibly, Bomba’s right hand stole to his pouch. It
gripped the butt of the revolver. Quite as slowly his hand came back
holding the weapon.
The slithering grew more distinct. The monster was drawing nearer.
There was a moment’s pause. Then a long rope seemed to rise from
the floor and Bomba found himself looking into the eyes of the
cooanaradi.
The fire stick spoke!
 CHAPTER XXIV
THE VOLCANO’S ROAR

Simultaneously with the crack of the revolver Bomba threw himself

from the further side of the bed to the floor.
There was a terrible thrashing about of the reptile’s body, and
Bomba knew that it had been hit. But whether it was mortally or only
slightly wounded he could not tell.
He held the revolver in readiness for a second shot, fearing every
moment to see that dreadful head reappear.
But no second shot was necessary. A few more moments and the
convulsive movements ceased.
The cooanaradi was dead!
With infinite caution Bomba tiptoed to the other side of the bed. Now
there was no doubt. The reptile’s head was practically gone,
shattered into fragments by that first shot.
Bomba slumped down on the bed, hardly able to believe in his good
fortune. He caressed the revolver lovingly and slipped in another
cartridge to take the place of the one that had done such good
service.
He had scarcely had time to collect his scattered thoughts before
there was a noise outside as of people running. He looked out of the
window and saw a number of shadowy forms, thrown into relief by
the torches that they carried.
Some of the foremost halted under the window and Bomba could
recognize Abino, Sunka, and Boshot, besides a couple of the elders
who had been at the conference.
They had reached there so quickly following the shot that Bomba felt
sure that they had been lurking in the vicinity, waiting for the
cooanaradi to get in its deadly work.
Bomba smiled grimly to himself as he noted that they stopped at the
window. He well knew why. Not one of them cared to come into a
house in which they believed a cooanaradi was wandering around.
Then, like a flash, the thought came to him that he might turn what
threatened to be a calamity into a blessing. Why should not the dead
serpent prove a protector, a guardian of the dwelling? As long as
they believed it was at large not one would risk stepping foot inside
the door.
With this belief on the part of the Indians to aid him, Bomba might be
able to hold the place until Japazy returned. At any rate, he would be
able to choose his own time for leaving.
There was a whispered consultation beneath the window and then
Abino called:
“Is the stranger still awake?”
Bomba waited for a moment and then leaned out of the window.
“He is awake. Does Abino want to speak words with him? Bomba’s
ears are open.”
“Abino and the elders heard a great noise, and they feared that harm
had come to the stranger,” the native returned. “They have come to
help him, if he is in trouble.”
To help him! The men who had sent the serpent to strike him down!
Bomba’s blood boiled. But he restrained himself with an effort, and
his tone was silken as he replied:
“It is good of Abino and the elders to want to help Bomba. But he is
not in trouble. But why do Abino and the elders stand beneath the
window? Will they not come in and talk with Bomba?”
In the moonlight he could see the movement of horrified recoil at the
suggestion, and again he smiled grimly to himself.
“The stranger needs sleep,” replied Abino, after a momentary pause.
“It is not well for us to talk with him when his eyes are heavy.”
“Bomba is not tired, and he would be glad to speak words with Abino
and the elders,” returned the lad. “He is lonely. There is none here
for him to talk to except the cooanaradi.”
There was a gasp of amazement.
“The cooanaradi!” came in a chorus from startled throats.
“Yes,” replied Bomba, with an ostentatious yawn. “The cooanaradi
came in to talk with Bomba. Bomba talked to him. Bomba knows
how to talk with the cooanaradi. He will not hurt Bomba. Bomba has
his hand on him now,” and he reached down and touched the grisly
coils. “He will not hurt any one who does not mean harm to Bomba.”
A yell of fright came from the superstitious natives. What magic was
this of which the stranger was master, this mysterious lad who spoke
of the dreaded scourge of the jungle as though it were a cooing
dove, who could place his hand upon it and still remain alive?
They turned to retreat in a flight of panic when Bomba’s voice stayed
them.
“Abino will send food to Bomba in the morning?” he inquired.
There was a moment’s pause, and then Abino replied in a trembling
voice:
“Abino would not dare have Solani come into the house where a
cooanaradi is.”
Bomba seemed to consider this.
“It is well,” he replied. “Solani does not know how to talk to the
cooanaradi as Bomba does. But he can bring food beneath the
window in a basket and Bomba will draw it up.”
“Solani shall come,” promised Abino.
Their figures melted away in the distance and Bomba chuckled as he
turned away. His reputation as a wizard was made. They were as
afraid of him as they were of death. Now he could rest in tranquillity
of soul until Japazy returned. For the first time in his life a cooanaradi
had proved a friend—an involuntary friend, to be sure, but none the
less a friend.
Bomba threw himself on the bed and in a few minutes was fast
asleep.
For hours he slept.
Then came a tremendous roar, a fearful explosion that sounded like
the crack of doom!
Bomba sprang from his bed and rushed to the window.
The heavens were a glare of red, and the glare was fiercest over in
the direction where Tamura lay.
 CHAPTER XXV
THE ISLAND SINKS

From the crater of the volcano a great mass of flame and smoke
belched toward the sky. At the peak rivers were forming, rivers of
molten lava, which, even as Bomba looked, began to flow down the
sides of the mountain in every direction.
The fearful explosions still continued, and with every one, that
dreadful column of flame mounted higher and the rivers of redhot
lava increased in volume. The earth was rocking in the throes of an
earthquake.
While Bomba was still trying to gather his wits together and
determine what he should do, the house of Japazy split apart. Half of
it went down in a deafening crash, and the other half, in which
Bomba’s room was located, swayed crazily, threatening at every
instant to fall to the ground.
Bomba slung his bow over his shoulder and rushed to the picture
hanging on the wall. He took it down—even in that moment of
frenzied haste he handled it reverently—and slipped the cord by
which it had been suspended about his neck, covering the picture
under the puma skin that stretched across his breast.
If he were doomed to die in this terrible cataclysm, he would die with
that picture next his heart!
He glanced down the corridor and saw that it was blocked with
debris. His only exit was the window. Luckily, it was not more than
twelve feet from the ground, and Bomba dropped down safely.
He was not a moment too soon, for he had barely got out of reach
before the rest of the building fell. The dwelling of Japazy, that had
witnessed nobody knew how many grim tragedies, was now in ruins.
In the dim dawn that was coming up in the eastern sky, Bomba could
see that most of the other houses in the town had shared the same
fate. People were rushing about in fright, men, women and children,
fleeing they knew not where, their only thought to get away from the
fiery rivers of lava that were now pouring down rapidly toward the
doomed town.
The earth was opening in great rents, which sometimes widened and
at other times snapped shut, engulfing houses, trees and sometimes
hapless human beings, whose screams of terror were dreadful to
hear.
Apart from these more spectacular demonstrations of the unchained
forces of nature, there was a sense of gradual sinking that was
sickening and ominous in the extreme.
No one paid any attention to Bomba, so engrossed was every one in
trying to find a path to safety.
The lad sniffed the air as a pointer does to get his sense of direction.
Then, when he knew that he was headed for the north end of the
island, he plunged into the woods and went as straight as an arrow
to his goal.
There was terrible need for haste, as the rivers of lava were now
pouring into the woods, blasting into ashes whatever they touched. If
one of those rivers intercepted his path, it would be impossible to
cross it.
The air was like that from a furnace, and Bomba was drenched with
sweat as he made his way through the jungle. Showers of hot ashes
were falling all about him. Great forest trees lay athwart his path,
hurled to the ground as though they had been saplings.
Over and over came those deafening roars that seemed to split his
eardrums, and ever the rivers of lava swelled in volume. The hot
blasts that came from them scorched his throat and lungs and
seared his eyes. He felt as though he were moving in a nightmare.
Now he was not alone. On every side the jungle was full of fleeing
animals and reptiles. There were hundreds of them, the jaguars
which had given the island its name, boa constrictors, dragging their
huge lengths through the brush, jararacas, cooanaradis.
They brushed against Bomba, bumped into him, but not one of them
gave him the slightest attention. Every hostile and venomous instinct
was swallowed up in panic, in the terror to escape. They did not
know where they were going. They did not care. All they thought of
was to get away from that roaring mountain, from those terrible rivers
of fire that threatened to overwhelm them.
Unexpectedly Bomba found himself on the edge of the river, the
same surging, turbulent stream that had brought him to Jaguar
Island. He was surprised at this, as he had calculated that he had at
least a mile further to go before he reached its banks.
And then he knew!
The river was creeping inland. It had already covered the ground to
the width of a mile. The island was sinking!
With frantic haste Bomba gathered together three logs, tore some
creepers from the trees and lashed the logs together. Even in the
brief time it took him to do this the water had deepened until it was
up to his neck.
He pulled himself up on the rude and fragile raft and committed
himself to the river. Soon the logs were in the grip of the current
where they tossed about so violently that Bomba could not keep his
feet. He threw himself flat and with his hands through interstices of
the logs held on for his life.
He had little fear this time of the caymans. They themselves would
be cowering with terror at this phenomenon that was shaking the bed
of the stream as well as the dry land above its level. They would
make no hostile move. Even the jaguars, whose heads now dotted
the stream everywhere, could swim to the mainland without any
danger from an attack by the alligators.
The current carried the raft toward the further side of the river, and
before long it struck heavily against a jutting point of land. With one
spring Bomba landed on the point, and a moment later stood safe on
the fringe of the jungle.
As he looked toward the island there came a stupendous crash that
dwarfed everything that had preceded. Once more Tamura belched
forth a great torrent of flame. Then the whole island sank out of sight
and where it had been were only miles of widely tossing waters.
Bomba viewed the terrible spectacle with awe and consternation.
The island where he had hoped to learn so much no longer existed.
But he was safe! His life had been spared—and that counted for
much.
Then another comforting thought came to him. The island of Japazy
had sunk. But, as far as he knew, Japazy himself still lived. He would
find the half-breed yet.
What fruit was borne by that determination will be seen in the next
volumes of this series: “Bomba the Jungle Boy in the Abandoned
City; or, A Treasure Ten Thousand Years Old.”
“Japazy still lives,” murmured Bomba to himself. “Bomba will seek
him out.”
He took reverently from beneath the puma skin the portrait of the
lovely woman. Again those beautiful eyes looked into his.
“Mother,” he whispered. “Mother!”
His eyes blurred. He pressed his lips to the picture and replaced it
next his heart.
Then he struck into the jungle.
THE END
 THE BOMBA BOOKS
By ROY ROCKWOOD
12mo. Cloth. Illustrated. With Colored jacket.
Price 50 cents per volume. Postage 10 cents additional.
Bomba lived far back in the
jungles of the Amazon with a
half-demented naturalist who told
the lad nothing of his past. The
jungle boy was a lover of birds,
and hunted animals with a bow
and arrow and his trusty
machete. He had only a primitive
education, and his daring
adventures will be followed with
breathless interest by thousands.

1. BOMBA THE JUNGLE BOY

2. BOMBA THE JUNGLE BOY
AT THE MOVING MOUNTAIN
3. BOMBA THE JUNGLE BOY
AT THE GIANT CATARACT
4. BOMBA THE JUNGLE BOY
ON JAGUAR ISLAND
5. BOMBA THE JUNGLE BOY
IN THE ABANDONED CITY
6. BOMBA THE JUNGLE BOY
ON TERROR TRAIL
7. BOMBA THE JUNGLE BOY
IN THE SWAMP OF DEATH
8. BOMBA THE JUNGLE BOY
AMONG THE SLAVES
9. BOMBA THE JUNGLE BOY
ON THE UNDERGROUND
RIVER
 10. BOMBA THE JUNGLE BOY
AND THE LOST EXPLORERS
11. BOMBA THE JUNGLE BOY
IN A STRANGE LAND
12. BOMBA THE JUNGLE BOY
AMONG THE PYGMIES
13. BOMBA THE JUNGLE BOY
AND THE CANNIBALS
14. BOMBA THE JUNGLE BOY
AND THE PAINTED HUNTERS
15. BOMBA THE JUNGLE BOY
AND THE RIVER DEMONS
16. BOMBA THE JUNGLE BOY
AND THE HOSTILE CHIEFTAIN

These books may be purchased wherever books are sold

Send for Our Free Illustrated Catalogue

CUPPLES & LEON COMPANY, Publishers

New York
 THE BOB DEXTER SERIES
By WILLARD F. BAKER
Author of “The Boy Ranchers Series”
12mo. Cloth. Illustrated. Jacket in colors. Price 50 cents
per volume. Postage 10 cents additional.
This is a new line of stories for
boys, by the author of the Boy
Ranchers series. The Bob Dexter
books are detective stories, yet
they are without the objectionable
features of the impossible
characters and absurd situations
that mark so many of the books
in that class. These stories deal
with the up-to-date adventures of
a normal, healthy lad who has a
great desire to solve mysteries,
and the volumes relate in an
entertaining way how he does it.

1. BOB DEXTER AND THE CLUBHOUSE MYSTERY or

The Missing Golden Eagle
This story tells how the Boys’ Athletic Club was despoiled
of its trophies
in a strange manner, and how, among other things stolen,
was the Golden
Eagle mascot.
2. BOB DEXTER AND THE BEACON BEACH MYSTERY
or The Wreck of the Sea Hawk
When Bob and his chum went to Beacon Beach for their
vacation, they
were plunged into a strange series of events, not the least