OBTE Monograph 01NOV2010

UNCLASSIFIED
Prepared for
The U.S. Army
Asymmetric
Warfare Group
Implications for an Integrated

Approach to Values-Based
Requirements
Copy edited by G. Riccio

November 2010
For individual chapters, see also:
www.scribd.com/collections/2715963
March 2010
UNCLASSIFIED
U.S. Army Asymmetric Warfare Group
2282 Morrison St., Ste. 5355 Ft. Meade, MD 20755-5355

AWG CUOPS awgawaccuops@conus.army.mil
AWG Portal https://portal.awg.army.mil
Asymmetric Warfare Group
ABSTRACT
This monograph documents the scientific and technical initiative conducted from March 2008 to
March 2010 sponsored by the Asymmetric Warfare Group (AWG). The purpose of this effort
was to help define and develop Outcomes-Based Training and Education (OBTE). The key
contributions of the initiative are that (a) the gap between Army instruction and Full Spectrum
Operations can be viewed in terms of the relationship between singular focus on tangible, nearterm objectives and the resulting effect on intangible attributes valued by Army leaders; (b)
everything that is done in Army training and education has an impact on this relationship whether
one addresses it or not; (c) this impact can be understood and described, allowing leadership to
make informed decisions in terms of verifiable observations and valid scientific reasoning; (d)
OBTE is an approach that can be utilized to inform leaders at all levels on the direction their unit
is moving in the development of leadership traits, problem solving skills and intangible attributes;
(e) the approach can be verified and validated; and (f) it can bring specifications for tasks,
conditions, and standards into a closer and more complete correspondence with doctrinal
requirements for training and education in the Army. These contributions are believed to lay a
foundation for continued development of training and education practices to support the
complexity of military operations in Full Spectrum Operations.
An Initiative in Outcomes-Based Training and Education

Implications for an Integrated Approach to Values-Based Requirements
Edited by
Gary Riccio
The Wexford Group International
Fred Diedrich
Aptima, Inc.
Michael Cortes
Prepared for
March 2010
Outcomes Based Training & Education
ii
ACKNOWLEDGMENTS
.
We gratefully acknowledge the sponsorship and support of the Asymmetric Warfare Group for
the opportunity to work on the issues and opportunities in Outcomes-Based Training and
Education (OBTE). In particular, we acknowledge COL Robert Shaw and LTC Michael
Richardson whose vision made this broad scientific investigation possible. In addition, we
appreciate the support and assistance of numerous individuals within the Asymmetric Warfare
Group, including LTC Richard Thewes, CSM Ray Devens, MAJ Jim Glover, CSM (Ret.) Hugh
Roberts, SGM (Ret.) Wayne Newberry, SGM (Ret.) John Porter, SGM (Ret.) Ed Rogers, SGM
(Ret.) John Hale, MSG (Ret.) Chris Crider, MSG (Ret.) Gary Keeney, and COL (Ret.) William
Ivey. We are especially indebted to CSM (Ret.) Morgan Darwin, MSG (Ret.) Scott Flanagan, and
LTC (Ret.) Blaise Cornell-dEchert for their guidance and advice in all phases of this
investigation.
Among the many others outside the AWG who have had a significant impact on our thinking, we
are especially grateful to COL Casey Haskins from whom we first learned about a systemic
approach to instruction based on long-term outcomes and associated best practices in military
leadership. We would like to thank LTC (Ret.) Michael Lerario for introducing COL Haskins and
Scott Flanagan to each in the context of their respective endeavors in OBTE. The resulting crossfertilization has been highly influential in the ongoing development of OBTE. Others outside the
AWG who have had an impact on our thinking about OBTE include BG (Ret.) Jim Schwitters,
LTC (Ret.) Kevin McEnery, Pam Hicks, Mac MacAllister, Dr. Bob Bauer, COL (Ret.) Jim
Connolly, Dr. Tom Duffy, Gary Rauchfuss, MAJ (Ret.) Don Vandergriff, and MAJ Chad Foster.
We would like to recognize Dr. Jean MacMillan of Aptima who provided advice during various
stages of the project.
More generally, we are indebted to numerous personnel in the United States Army Training and
Doctrine Command (TRADOC) who commented on various presentations of the data and ideas
compiled in this manuscript. We also are indebted to personnel in various units of the Army
Research Institute for the Behavioral and Social Sciences (ARI) who, over the last five years,
have provided us with opportunities to develop the theory and methods utilized in this
investigation. Among these, we would like to give special thanks to Dr. Scott Graham, Dr.
William Bickley, Dr. Jennifer Tucker, Dr. Scott Beal, and Dr. Jay Goodwin. In addition, we are
indebted to numerous personnel at Fort Benning, Fort Jackson, and Fort Sill for peer review and
support of data collection in Initial Entry Training. Finally, we would like to thank personnel at
Fort Knox who are associated with the Army Reconnaissance Course, and personnel at the United
States Military Academy (West Point) who are associated with the Department of Military
Instruction, who are continuing to develop comprehensive organizational and instructional
practices for OBTE.
iii
Riccio, Diedrich, & Cortes (Eds.)
EXECUTIVE SUMMARY
Requirement
This report describes research sponsored by the U.S. Army Asymmetric Warfare Group (AWG).
The mission of the AWG is to provide operational advisory assistance in support of Army and
Joint Force Commanders to enhance the combat effectiveness of the operating force and enable
the defeat of asymmetric threats. The AWG executes missions, responsibilities, and functions
required to enhance the capabilities of U.S. units by making them more adept at quickly
identifying and attacking enemy vulnerabilities, and by preparing them for a broader spectrum of
threats. In March of 2008, the AWG initiated a scientific and technical effort to help define and
develop Outcomes-Based Training and Education (OBTE). The objective of this research was to
enable the integration and development of OBTE as instructional capabilities, in both the
Generating Force and the Operating Force, that are closely linked with the needs of Full Spectrum
Operations. Derived requirements implicit in this broad agenda include an integrated approach to
leadership, education, training and self-development that can be institutionalized through
verification, validation, and the attendant capabilities for continuous improvement.
The AWG analysis of stakeholder requirements for Army training can be summarized in three
conjectures. These conjectures provide a framework within which to develop claims about the
manifestations and impact of OBTE in particular situations, to identify the kind of evidence with
respect to which the claims can either be vitiated or substantiated, and to identify tasks within
which the evidence can be collected.
Conjecture 1: Instruction can be designed, developed and implemented with respect to basic
skills that lead to better understanding of a situation including relationships among elements
within a relevant situation, that motivate and guide engagement with that situation, and that
afford opportunities to influence the situation instead of merely responding to it. There is a
need to identify this level of analysis for instruction and learning that fosters the development
of basic Soldier skills necessary for success in Full Spectrum Operations.
Conjecture 2: Instruction can be designed and implemented in verifiable ways that apply to
all levels of leadership and leader development for a curriculum or program of instruction. A
general approach to Army training and education can be developed that applies to all
interactions between an instructor and a learner, as well as to self-development, without
sacrificing validity with respect to Full Spectrum Operations. There is a need for a verifiable
and valid approach that can be utilized by instructors and learners together or by learners on
their own across all types of learning events.
Conjecture 3: Soldiers do not rise to the occasion in combat; they fall back to their highest
level of training and education. The highest level of training and education should relate
directly and unambiguously to the most important requirements in Army Doctrine, such as
Army Values and Warrior Ethos. There currently are no doctrinally codified standards for
such values-based requirements. There is a need for values-based standards that can be
applied to any and all learning events.
Procedure
The research design is consistent with a state-of-the-art Service System Development (SSD)
process area of the Capability Maturity Model Integration for Services (CMMI-SVC) and also
reflects best practices in scientific support of social and programmatic decision-making.
iv
Accordingly, it builds on the activities of the AWG to develop and analyze stakeholder
requirements (Specific Goal 1 of SSD) and to develop a service system that meets such
requirements (Specific Goal 2 of SSD). The focus of the investigation was to verify and validate
OBTE as a service system (Specific Goal 3 of SSD). Specific practices associated with SG 3
include performance of peer reviews, verification of selected service system components, and
validation of the service system. Peer review was addressed primarily in the context of surveys
administered by the AWG. Verification was conducted primarily through field-based
observations in Initial Entry Training (IET) at Fort Jackson, Fort Benning, and Fort Sill that
employed measures specifically developed to enable formative assessment of instructor and
student behavior. The empirical approach to validation focused on surveys of Soldiers who had
been deployed to a combat environment after being exposed to an early version of a field course
in marksmanship that demonstrated instructional principles of OBTE. Validation also was
addressed through development of grounded theory for OBTE based on established lines of
scholarship in the behavioral and social sciences. Consistent with spiral development, verification
and validation also informs the ongoing refinement of stakeholder requirements and continuous
improvement of OBTE as a mature service system.
Findings
The overall goal of OBTE is to prepare Soldiers to function well in full spectrum operations, and
it offers a variety of strategies to enable progress toward such outcomes. More specifically, it
focuses on the role of instructors in the development of Soldiers through a field-based course in
marksmanship (Combat Applications Training Course or CATC) and supporting workshops for
instructors and leaders. The first source of evidence about OBTE is self-report data obtained from
Soldiers who had participated in CATC (Chapter 6). Results from the surveys indicate that
participants are generally positive when asked about their experiences in CATC and that they
hold the instructional approach in high regard. Moreover, their comments tend to focus on
outcomes around which the approach is organized (e.g., problem solving, intangibles,
understanding and awareness, and combat performance). This suggests that they understand the
intent of OBTE. In addition, when asked about longer-term impact, respondents indicated that the
AWG course had an influence on their behavior when in theater, that it influenced their training
approaches with their own units, and that they experienced growth with respect to core outcomes.
Our overall impressions from the survey are that CATC, as an exemplar of OBTE, was perceived
as efficacious with respect to developmental outcomes involving improved cognition, emotion,
and motivation in the context of skill training such as marksmanship and weapons operation.
The second source of evidence about OBTE is a series of observations of the extent to which
CATC participants subsequently employed the approach as instructors (Chapter 8). Specifically,
this evaluation focused on Drill Sergeants (DS) when instructing Privates in rifle marksmanship
as part of Initial Entry Training at Fort Benning and Fort Jackson. Such observations and ratings
of instructor behavior and instructor-student interactions indicate whether OBTE is verifiable in
instructional practice. Based on the ratings of DS and Privates, correlations were computed to
assess whether highly rated DS (i.e., DS who were using an OBTE approach) were associated
with Privates who were also rated highly. The results of this analysis revealed a significant
positive relationship between highly rated DS and highly rated Privates. The significant
correlation suggests that DS whose behavior exhibited greater application of OBTE were more
successful as instructors. The practical implication is that any intervention that enhances
instructor behavior with respect to OBTE may result in better training and thus may improve
performance of Privates.
The third source of evidence is from an investigation at Fort Sill similar to the one conducted at
Fort Benning and Fort Jackson (Chapter 9). In this investigation, however, observations and
ratings of instructor behavior and instructor-student interactions were made before and after
instructors were exposed to OBTE through participation in CATC, AWG workshops and
briefings on OBTE, non-AWG briefings on OBTE and, in some cases, other applications of
OBTE beyond rifle marksmanship. The evidence from this pre-post comparison suggests that DS
changed their instructional behavior to be more consistent with OBTE after being familiarized
with it. DS altered elements of their instruction to be more learner-centered, and they asked
Privates more questions to promote deliberate thought and problem solving. They also attempted
to involve Peer Coaches as collaborative problem solvers, for example, in working to group and
zero during marksmanship instruction. Critically, these changes were accompanied by changes in
the behavior and performance of the Privates. Their marksmanship and performance on items
central to OBTE principles improved. These changes occurred with no apparent decreases in
safety despite increased handling of weapons by Privates.
The fourth source of evidence is from a survey of personnel exposed to OBTE in units at all
levels of the organization (e.g., from Commander to Drill Sergeant) at Fort Sill to gain an
understanding of their familiarity with OBTE and attitudes toward it (Chapter 9). Based on this
field-based investigation, we addressed the following questions: (a) Are there differences in the
way Drill Sergeants (DS) instruct Privates in Basic Rifle Marksmanship (BRM) following
exposure of DS to OBTE? (b) Are there differences in the behavior of Soldiers following
exposure of their DS to OBTE? (c) Are there changes in attitudes toward OBTE and
understanding of it following exposure to OBTE? The results suggest that the adoption of OBTE
has successfully increased the extent to which respondents believe that outcomes such as the
intangibles can be addressed and are addressed in Basic Training. This result is striking given
the interpretation of related results from the survey. The implication of other results is that DS are
taking the initiative to create situations favorable to OBTE because they believe the principles of
OBTE are important and in spite of the fact they dont believe the principles relate directly to
current training objectives.
The fifth source of evidence about OBTE as a service system is self-report data obtained from
Soldiers who had an opportunity to implement OBTE with their own units upon deployment in
the contemporary operating environment (Chapter 10). Such data increase our understanding of
the potential operational value of OBTE to the Army. In particular, they provide indications about
the influence of OBTE on the competency of Soldiers in their job as Soldiers in lethal
engagements with the enemy. The results from the post-deployment survey provide further
evidence that: (a) OBTE can influence development of cognition and behavior, beyond tactical
marksmanship skills, that transfer to combat situations; (b) compared to other training within the
Army, some Soldiers perceive OBTE to be an approach they prefer and are willing to employ in
training others; and (c) the focus on OBTE principles during training helps Soldiers develop with
respect to outcomes typically not addressed in marksmanship training (e.g., confidence, situation
awareness, judgment, and thinking skills). The reported improvement in these areas also suggests
that, more often than not, Soldiers can come to understand that the intent of OBTE is to develop
personal attributes and competencies beyond marksmanship and weapons operation.
The sixth source of evidence was from the application of OBTE to a classroom environment in
which we focused on the role of assessments that employ self-report measures as well as
measures of observable behavior (Chapter 13). Our objectives were (a) to demonstrate that best
practices of OBTE could be verified through measures that can be applied in the classroom, and
(b) to assess behavior of instructors and students and provide actionable feedback to support
student learning and improvement of instruction with respect to the principles and practices of
vi
OBTE. As the result of the experience with OBTE in the classroom, we have come to believe that
the use of formative measures in an iterative longitudinal 360 review can be one of the most
valuable tools for instructor education.
The seventh source of evidence is from measures of subjective workload that were explored in
the classroom environment (Chapter 13). Our findings indicate that workload assessments can
provide valuable additional information about engagement of students in the learning
environment. The workload data show that cognitive load is a multidimensional construct that
influences learning and motivation in complex ways. For example, instructors should be as
concerned about excessively low workload as about excessively high workload. Higher workload
is not necessarily a problem if it reflects that students are engaged. A key observation is that selfreported frustration went down as effort and mental demand went up. This result vitiates the
assumption that OBTE is necessarily problematic because it may require too much of students.
The eighth source of evidence was from the development of grounded theory, for OBTE, which
was informed by several concurrent activities during the investigation (Chapters 3, 4, and 5). In
particular, we interviewed progenitors and practitioners of OBTE and discussed relevant
scientific research with them, attended AWG workshops and seminars on OBTE, reviewed
documents from the AWG, observed AWG instructors in CATC, participated in CATC, and
interviewed stakeholders. The purpose of the grounded theory is to make connections that enable
increasingly thick descriptions and elaborations of the principles and practices of OBTE. We
began our inquiry with the critical assumption that there is deep wisdom to be understood in the
best practices of good training and in the education that changes lives. We believe this tacit
knowledge in the Army can be made concrete through scientific inquiry and through the
relationships that it reveals with findings and theories in a several independent lines of established
scholarship. The constant comparison of our findings and insights with theory and evidence from
the scientific literature revealed an abundance of research that is relevant to the current state and
future development of OBTE. Integrating this research, we identified three pillars for a scientific
foundation to OBTE. This foundation supplements our observations of OBTE by reinforcing the
conclusion that the claims of OBTE are credible, transferable, dependable, and confirmable.
Collectively, the evidence suggests considerable potential for OBTE in the Army, especially if it
becomes explicitly more directed toward staff and faculty development and quality assurance. It
is noteworthy that our findings suggest that OBTE can motivate individuals to take ownership of
their own learning and development, and that it can increase self-efficacy for teaching and
developing others. The approach motivates a greater interest in instructor-student interactions and
how these interactions affect progress toward developmental outcomes while satisfying coursespecific learning objectives. While the findings show that there is both an impact of OBTE and
considerable room for improvement in IET, the measures of OBTE provide actionable
information about the gaps and shortfalls. The findings and associated research have implications
for all the factors that influence institutional learning including student diversity and background,
capacity for self development, instructional design and development, instructor education, reward
and recognition, individual assessment and program evaluation, leadership in an institution of
learning, leader development, and the relevance science of measurement.
Utilization of Findings
The most important contributions of the AWGs initiative in Outcomes-Based Training &
Education and the associated research are the claims and evidence that (a) the gap between Army
instruction and Full Spectrum Operations can be viewed in terms of the relationship between
singular focus on tangible, near-term objectives and the resulting effect on intangible attributes
vii
valued by Army leaders; (b) everything that is done in Army training and education has an impact
on this relationship whether one addresses it or not; (c) this impact can be understood and
described, allowing leadership to make informed decisions in terms of verifiable observations and
valid scientific reasoning; (d) OBTE is an approach that can be utilized to inform leaders at all
levels on the direction their unit is moving in the development of leadership traits, problem
solving skills and intangible attributes; (e) the approach can be verified and validated; and (f) it
can bring specifications for tasks, conditions, and standards into a closer and more complete
correspondence with doctrinal requirements for training and education in the Army.
Given our approach to development of OBTE as an integrated instructional service system, it is
critical to understand what instructors should do and what they believe they should influence in
particular programs of instruction. Toward that end, we developed a set of measures for instructor
behavior that exemplifies OBTE and that can be applied to any program of instruction (Chapters
2, Appendix A). We also developed measures for student behavior that complement the instructor
measures and that help reveal the efficacy of instructor-student interactions (Chapters 7 and 12,
Appendix B). We thus defined OBTE as a verifiable and replicable instructional service system,
and we explored its links to established theories. We emphasized the utility of these measures in
formative assessments for instructors and instructional program evaluation. Moving forward, we
stress that definition and measurement of instructor-student interactions and verification of
instructional practices is critical to sustain or improve the quality of any instructional service
system and to transfer any new approach to particular programs of instruction (Chapters 1 and
11). In essence, definition and measurement of OBTE can help turn conventional wisdom into
conventional practice.
We believe that tacit knowledge about good training can be made concrete through programmatic
scientific inquiry. Of necessity, this also reveals connections with domains of training and
education outside the Army. Thus, there is an opportunity to realize the potential for improvement
in virtually everything that influences Army training and education (e.g., see Chapter 15 and
Epilogue) by adapting specific practices within and beyond Army training and education and by
utilizing the more general scientific underpinnings. The research cited in this monograph provides
a theoretical and empirical basis for shared understanding and continuous coordinated
improvement across multiple organizations and organizational levels in training and education
(see Chapter 14). The most immediate result of this is the ability to develop and promulgate
standards for existing values-based requirements in Army doctrine. The longer-term result is that
Army training and education will be aligned more closely and more completely with Army
doctrine and the operational exigencies it represents.
The report will be disseminated by the AWG to all interested stakeholders in training, education,
and leader development including scientists, instructors, instructional designers, course
developers, quality assurance personnel, and commanders in instructional programs. The theory,
method, and findings will guide peer-to-peer sharing of instructional best practices. Informal
presentation of findings to hundreds of such stakeholders has been influencing revisions and
interpretations of Army doctrine relating to training, education, self-development, and leader
development. It also has led to changes in a variety of programs of instruction ranging from
marksmanship training to instructor education at several Army posts. The grounded theory and
associated measures also can facilitate contributions from stakeholders in theater who are the
ultimate customers of the Generating Force. Such opportunities for influence will be possible
because of greater visibility into instructional practices in the Generating Force and a clearer
understanding of their relationship and potential impact on the readiness of Soldiers for Full
Spectrum Operations.
Forward by AWG Commander
viii
Forward
Amid this changing face of warfare, it is very clear that Soldiers must have all the skills necessary
to not only survive on the battlefield but to succeed in their mission. Wars are not fought to
finish, they are fought to win. The nature of todays conflicts call for Soldiers who are more agile
in their application of trained skills and leaders who are adaptive problem solvers. Senior Army
leaders have asked for a new approach to training that promotes a mastery of skills, innovative
thinking, initiative, and accountability. FM 3.0 clearly states the requirement to change how the
Army prepares for Full Spectrum Operations. It also outlines the traits required of both Soldiers
and leaders in order to accomplish this task. The publication of FM 7.0 further reinforces the need
to better prepare our Army for the challenges ahead.
This monograph describes a measurable approach to training called Outcome Based Training and
Education (OBTE). In light of the changing face of war and the increased requirement for units
to prepare for Full Spectrum Operations, OBTE seeks to merge the benefits of training and
education. OBTE helps to create thinking individuals and organizations that can solve problems
under stress based on an understanding of the problem. It stresses the development of intangible
attributes such as initiative, critical thinking, judgment, and responsibility. OBTE uses observable
outcomes to measure the development and effectiveness of training. More importantly, it uses
those outcomes to develop more adaptive Soldiers and leaders who are better prepared for the
rigors of 21st Century combat.
The Asymmetric Warfare Group (AWG) has been working in concert with the Institutional and
Operational Army as well as the scientific and academic communities to develop a method of
demonstrating and quantifying this new approach to training. This monograph provides scientific
evidence supporting the AWGs claims that OBTE can develop Soldiers who are better prepared
for Full Spectrum Operations because it can simultaneously address mastery of a skill,
development of intangible attributes, and the understanding of the relatedness of tasks to different
environments.
The results of the research indicate that OBTE has considerable potential for the Army as an
approach to training and education. The monograph illustrates the measurement of instructional
behavior and its effectiveness as well as the implications for a rigorous way to monitor leader
development across the Army. It is being disseminated to the Institutional and Operational Army
as well as the academic, scientific, leader development, and quality assurance communities. As
quantifiable evidence, it can now be used to support further inclusion of OBTE into policy,
regulations, and doctrine.
More than any other program or project pursued by the AWG to date, I believe that this
transformational effort has been one of the most significant contributors to influencing a positive
change in the culture of our Army. I hope that you all find this research as compelling as I do and
will adopt this approach to training, education, and leader development within your commands
and organizations.
Robert C. Shaw
COL SF
AWG Commander
ix
TABLE OF CONTENTS
.
page
Prologue: A Programmatic View of the Inquiry into Outcomes-Based Training & Education.......1
Historicity of our Research on OBTE ..........................................................................................1
The Approach and Lessons Learned from the Research..............................................................3
Documentation of the Research ...................................................................................................4
Section I. Development of Stakeholder Requirements for OBTE..............................................6
Chapter 1. Preparation for Full Spectrum Operations ......................................................................7
1.1 Requirements of Full Spectrum Operations ...........................................................................8
1.2 Outcomes-Based Training and Education (OBTE)..............................................................10
1.2.1 Exemplar of OBTE: Combat Applications Training Course........................................11
1.2.2 OBTE as a Multifaceted Instructional System .............................................................12
1.3 An Appraisal of Instruction with Respect to OBTE ............................................................13
1.3.1 A Systems Engineering Framework for Integration and Development of OBTE ........13
1.3.2 Preparation for Validation and Verification .................................................................14
1.4 References ............................................................................................................................17
Chapter 2. Formative Measures for Instructors ..............................................................................20
2.1 Development of Formative Measures ..................................................................................20
2.1.1 The COMPASS Methodology ......................................................................................20
2.1.2 Development of Measures for OBTE ...........................................................................21
2.2 Description of Formative Measures .....................................................................................21
2.2.1 Results of the COMPASS Process................................................................................21
2.2.2 Elaboration on the Description of Measures.................................................................23
2.3 OBTE Performance Measures: Planning for Training.........................................................23
2.3.1 Define Outcomes ..........................................................................................................23
2.3.2 Create a Positive Learning Environment ......................................................................25
2.3.3 Create the Parameters of Learning................................................................................27
2.4 OBTE Performance Indicators: Training Execution............................................................28
2.4.1 Communicate the Parameters of Learning....................................................................28
2.4.2 Training Emphasizes Broad Combat or Mission Success ............................................29
2.4.3 Customize Instruction When Possible Based on Constraints/Conditions ....................31
2.4.4 Facilitates Learning of Concepts ..................................................................................32
2.4.5 Creates a positive learning environment.......................................................................34
2.4.6 Instructors Utilize Measures of Effectiveness & Self-Evaluation ................................36
2.4.7 Uses scenarios to facilitate learning..............................................................................38
2.4.8 Instructors exhibit intangible attributes in own actions ................................................40
2.4.9 Hotwashes and Mini-AAR............................................................................................42
2.5 Uses of the Measures ...........................................................................................................43
2.5.1 Formative Measures for Instructors ..............................................................................44
2.5.2 Quality Assurance and Instructor Education ................................................................44
2.5.3 Continuous Improvement of Assessments....................................................................45
2.5.4 Program Evaluation and Organizational Change..........................................................46
2.6 References ............................................................................................................................46
Table of Contents
Chapter 3. Principles and Practices of Outcomes Based Training & Education............................50

3.1 Multifaceted Inquiry.............................................................................................................50
3.1.1 Interaction with Progenitors of OBTE..........................................................................51
3.1.2 AWG Documents on OBTE .........................................................................................52
3.1.3 Collaborative Reflection on Participant Observation in CATC ...................................52
3.1.4 Interaction with Stakeholders .......................................................................................53
3.2 Essential Characteristics of OBTE.......................................................................................53
3.2.1 The Meaning of Developmental is a Critical Difference..............................................53
3.2.2 The Definition of Outcomes is a Critical Difference....................................................56
3.2.3 The Emphasis on Values and Causally Potent Intangibles is a Critical Difference .....58
3.2.4 The Meaning of Experience is a Critical Difference ....................................................61
3.2.5 The Emphasis on Instructor-Student Interactions is a Critical Difference ...................62
3.2.6 The Emphasis on Learning to Learn is a Critical Difference .......................................63
3.2.7 The Emphasis on Collaborative Design and Development is a Critical Difference.....65
3.3 Toward a Grounded Theory for OBTE ................................................................................66
3.3.1 Need for an Integrated Interdisciplinary Framework ...................................................66
3.3.2 Formative Measures of Instructor Behavior as Evolving Best Practices of OBTE......67
3.4 Emerging Best Practices in OBTE for a Community-Centered Environment.....................68
3.4.1 Leadership and Enculturation of Soldiers.....................................................................68
3.4.2 Robust and Adaptable Plan...........................................................................................70
3.4.3 Instructors as Role Models ...........................................................................................70
3.4.4 Collaborative Identification of Outcomes and Measures .............................................71
3.5 Emerging Best Practices in OBTE for a Knowledge-Centered Environment .....................71
3.5.1 Integrated Understanding of Basic Soldier Skills in Full Spectrum Operations ..........72
3.5.2 Task Relevance of Planned Instructional Events..........................................................72
3.5.3 Reveal Operational Relevance of Training...................................................................73
3.5.4 Incorporate Stress into Instructional Events .................................................................73
3.5.5 Identify General Lessons Learned and Extrapolate to New Situations ........................74
3.6 Emerging Best Practices in OBTE for an Assessment-Centered Environment ...................74
3.6.1 Collaborative Reflection and Problem Solving ............................................................75
3.6.2 Communication.............................................................................................................75
3.6.3 Nature and Extent of Guidance.....................................................................................76
3.6.4 Establish a Pervasive Mindset of Collaborative Reflection..........................................76
3.7 Emerging Best Practices in OBTE for a Learner-Centered Environment ...........................77
3.7.1 Soldier Motivation and Development of Intangibles....................................................77
3.7.2 Plan for Development of the Individual .......................................................................78
3.7.3 Get Students to Take Ownership ..................................................................................78
3.7.4 Collaborative Reflection as a Means to Develop Self Efficacy....................................79
3.8 References ............................................................................................................................79
Chapter 4. Grounded Theory for Values-Based Training & Education .........................................86
4.1 Exploration of Holistic and Functionalistic Underpinnings for OBTE ...............................86
4.1.1 Fundamental Units of Analysis.....................................................................................87
4.1.2 Nested Time Scales and Adaptability ...........................................................................88
4.1.3 Adaptability and Ambiguity .........................................................................................90
4.1.4 Mechanistic Analogies and Predominant Experimental Paradigms .............................92
4.2 Three Pillars for the Scientific Foundation of OBTE ..........................................................93
4.2.1 Ecological Psychology..................................................................................................93
4.2.2 Self-Efficacy Theory.....................................................................................................97
4.2.3 Positive psychology ......................................................................................................98
4.3 A More Integrated Scientific Infrastructure .......................................................................101
xi

4.3.1 Self Determination Theory .........................................................................................101
4.3.2 Situated Learning Theory ...........................................................................................103
4.3.3 Existential Psychology................................................................................................105
4.4 Building on the Scientific Infrastructure for OBTE...........................................................109
4.4.1 Triadic Frameworks ....................................................................................................109
4.4.2 Further Development ..................................................................................................112
4.5 References ..........................................................................................................................112
Chapter 5. Passion and Reason in Values-Based Learning & Development ...............................118

5.1 The Nested Self ..................................................................................................................118
5.1.1 An Alternative to Individual versus Collective ..........................................................118
5.1.2 Cognition and Reality .................................................................................................119
5.2 Conscious Experience and the Dynamics of Thinking ......................................................122
5.3 Emotion, Information, and Engagement ............................................................................125
5.3.1 Ecological Perspective on Emotion ............................................................................125
5.3.2 Emotion as Engagement .............................................................................................126
5.3.3 Implications for Training and Education ....................................................................129
5.4 Emotion, Decision-Making, and Inter-Temporal Choice...................................................129
5.4.1 Toward a More Integrated Theory..............................................................................129
5.4.2 Emotion and Decision-Making ...................................................................................130
5.4.3 Emotion and Nested Time Scales ...............................................................................131
5.4.4 Neuroeconomics and Inter-Temporal Reasoning .......................................................132
5.5.5 Inter-Temporal Reasoning and Adaptive Dynamical Systems...................................133
5.5 Beyond Science ..................................................................................................................134
5.5.1 Existentialism..............................................................................................................134
5.5.2 The Soldier-Scholar as an Emergent Property of a Collective Pursuit.......................135
5.6 References ..........................................................................................................................137
Section II. Verification and Validation of OBTE as a Service System ..................................142
Chapter 6. Initial Impressions of Participation in CATC .............................................................143
6.1 Methods..............................................................................................................................143
6.1.1 Participants..................................................................................................................143
6.1.2 Procedure ....................................................................................................................143
6.1.3 Analyses......................................................................................................................144
6.2 Results ................................................................................................................................144
6.3 Implications for Service System Development: Peer Review ...........................................146
6.4 References ..........................................................................................................................147
Chapter 7. Local Development of Measures of Effectiveness .....................................................149
7.1 What do Instructors Believe Soldiers Should Learn in Initial Entry Training? .................149
7.2 Measure Development Process ..........................................................................................150
7.3 What do OBTE-Trained DS Believe is Important to Assess in BRM/ARM? ...................151
7.4 Implications........................................................................................................................156
7.5 Conclusions ........................................................................................................................158
7.6 References ..........................................................................................................................159
Table of Contents
xii
Chapter 8. Observations of Behavior and Communication in Rifle Marksmanship Training .....160

8.1 Methods..............................................................................................................................160
8.1.1 Participants..................................................................................................................160
8.1.2 Procedure ....................................................................................................................160
8.1.3 Analyses......................................................................................................................161
8.2 Results ................................................................................................................................163
8.2.1 Behavior of DS ...........................................................................................................163
8.2.2 Behavior and Performance of Privates .......................................................................165
8.2.3 Patterns of Communication ........................................................................................168
8.2.4 Potential Influence of Instructor Behavior on Performance of Privates .....................170
8.3 Implications for Service System Development..................................................................171
8.3.1 Verification of OBTE .................................................................................................171
8.3.2 Validation of OBTE....................................................................................................172
8.4 References ..........................................................................................................................173
Chapter 9. Impact on Rifle Marksmanship Training....................................................................174
9.1 Behavioral Data Collection During Basic Rifle Marksmanship ........................................174
9.1.1 Method ........................................................................................................................174
9.1.2 Assessment..................................................................................................................175
9.1.3 Results An Overview ...............................................................................................177
9.1.4 Evidence for Influence of OBTE ................................................................................178
9.1.5 Behavior of Drill Sergeants after Exposure to OBTE ................................................180
9.1.6 Behavior of Privates....................................................................................................182
9.1.7 Patterns of Communication ........................................................................................186
9.1.8 Summary .....................................................................................................................186
9.2 Attitudes Toward an OBTE in Basic Training...................................................................187
9.2.1 Method ........................................................................................................................187
9.2.2 Results.........................................................................................................................187
9.4 References ..........................................................................................................................191
Chapter 10. Influence of CATC in an Operational Setting ..........................................................192
10.1 Methods............................................................................................................................192
10.1.1 Participants................................................................................................................192
10.1.2 Procedure ..................................................................................................................192
10.1.3 Analyses....................................................................................................................193
10.2 Results ..............................................................................................................................193
10.2.1 Downstream Impact on Marksmanship ....................................................................193
10.2.2 Downstream Impact on Training in the Units ..........................................................194
10.2.3 Downstream Impact on Self Efficacy .......................................................................195
10.3 Implications for Service System Development: Validation.............................................196
10.4 References ........................................................................................................................197
Chapter 11. Implications for Service System Development.........................................................198
11.1 Lessons Learned about Transfer of OBTE.......................................................................198
11.2 Implications for Service System Development................................................................199
11.2.1 Further Development and Analysis of Stakeholder Requirements for OBTE..........199
11.2.2 Further Development of OBTE as a Service System ...............................................199
11.2.3 Further Verification and Validation of OBTE ..........................................................201
11.3 References ........................................................................................................................203
xiii
Section III. Further Development of OBTE as a Service System ..........................................206

Chapter 12. Development of General Measures for Students ......................................................207
12.1 Intent ................................................................................................................................207
12.2 Performance Measure Development Process...................................................................207
12.2.1 Phase One: Define Performance Indicators (PI).......................................................207
12.2.2 Phase Two: Translate PI into performance measures ...............................................208
12.2.3 Phase Three: Measure refinement.............................................................................208
12.2.4 Phase Four: Retranslation of Measures ....................................................................208
12.3 Product of Measure Development....................................................................................209
12.3.1 Learner Perception of the Instructor and Course ......................................................209
12.3.2 Learner Engagement .................................................................................................211
12.3.3 Student Relationship with Teacher ...........................................................................212
12.3.4 Student Results .........................................................................................................214
12.3.5 Self-Report Measures ...............................................................................................216
12.4 Conclusion........................................................................................................................217
12.5 References ........................................................................................................................217
Chapter 13. Adapting OBTE in a Classroom Environment .........................................................219
13.1 Intent ................................................................................................................................219
13.2 Observing OBTE in the Classroom Environment............................................................219
13.2.1. Participants...............................................................................................................219
13.2.2. Procedure .................................................................................................................220
13.2.3. Measures ..................................................................................................................220
13.3 Utility of OBTE Measures in a Classroom Environment ................................................220
13.3.1 Generality of Measures .............................................................................................220
13.3.2. Implications for Improvement of Measures.............................................................221
13.3.3 Implications for improvement of course design .......................................................222
13.4 Use of 360 Reviews for Collaborative Reflection..........................................................223
13.4.1 The Role of a 360 Review in OBTE .......................................................................223
13.4.2 Narrative of a Participant Observer ..........................................................................225
13.5 Learning, cognitive load and motivation..........................................................................228
13.5.1 The NASA Task Load Index as a subjective measure of student workload.............228
13.5.2 Results.......................................................................................................................229
13.5.3 Implications ..............................................................................................................230
13.6 Conclusions ......................................................................................................................230
13.7 References ........................................................................................................................231
Chapter 14. Organizational Climate and Creation of Durable Change ........................................233
14.1 The Need ..........................................................................................................................233
14.2 Initial Indications of Possible Resistance to Change .......................................................234
14.3 Models and Considerations for Sustainable Change........................................................235
14.3.1 The Change Transition Period ..................................................................................235
14.3.2 Organizational Culture..............................................................................................237
14.3.3 Clarity of Mission and Shared Understanding..........................................................237
14.3.4 Relevant Observations During the Current Investigation.........................................238
14.3.5 Organizational Support and Incentives.....................................................................238
14.4 Conclusions ......................................................................................................................239
14.5 References ........................................................................................................................239
Table of Contents
xiv
Chapter 15. Five ways OBTE can enable the Army Leader Development Strategy....................242
15.1 Background ......................................................................................................................242
15.2 An Emerging Consensus ..................................................................................................244
15.2.1 What Part to Balance?...............................................................................................244
15.2.2 Improving Training, by Design ................................................................................245
15.2.3 Increased Use of dL and Dependence on Self-Development ...................................246
15.2.4 Future Orientation, Unknown Requirements............................................................247
15.2.5 The Quality Instructor Challenge .............................................................................247
15.2.6 Purpose and Design are Key .....................................................................................248
15.2.7 A Natural Advantage ................................................................................................249
15.2.8 Task Specialization or Generalized Competency .....................................................249
15.3 Conclusion........................................................................................................................251
15.4 References ........................................................................................................................252
Epilogue. Integration of Leader Development, Education, Training, and Self-Development .....254
Toward Values-Based Standards for Army Doctrinal Requirements ......................................254
Nested Standards and Quality Assurance.................................................................................256
Needs and Opportunities for Staff & Faculty Development ....................................................259
A Role for Science and Measurement .................................................................................259
Toward Best Practices in Instructor Education....................................................................260
Critical Considerations for Further Scientific Investigation ....................................................263
The Necessity of Long-Term Studies ..................................................................................263
False Dichotomy of Objective-Subjective ...........................................................................264
Clarity About What Is Evaluated.........................................................................................265
Next Steps ............................................................................................................................266
References ................................................................................................................................268
Section IV. Appendices...............................................................................................................270
Appendix A. OBTE Principles & Practices: Instructor Measures................................................271
A.1 Genesis of Formative Measures for Instructors ................................................................271
A.2 Principles of Outcomes-Based Training & Education ......................................................272
A.3 Guide to Using Measures of Instructor Behavior..............................................................276
A.4 Complete Menu of Instructor Measures............................................................................279
Appendix B. OBTE Principles & Practices: Student Measures ...................................................318
B.1 Guide to Using Measures of Student Behavior .................................................................318
B.2 Complete Menu of Student Measures ...............................................................................319
Appendix C: A Commanders View of Outcomes-Based Training and Education .....................340
Summary ..................................................................................................................................340
Definition .............................................................................................................................340
Description...........................................................................................................................340
Elements of OBTE. ..................................................................................................................341
Developing the Outcomes....................................................................................................341
Developing the Training Plan ..............................................................................................341
Conducting Training ............................................................................................................342
How Training is Assessed....................................................................................................344
Conclusion................................................................................................................................344
xv
Appendix D: Warrior Ethos..........................................................................................................345

Analysis of the Concept and Initial Development of Applications..........................................345
Current Understanding of Warrior Ethos.............................................................................345
Purpose.................................................................................................................................348
Approach..............................................................................................................................348
Expansion of the Definition of Warrior Ethos.....................................................................348
The Tenets of Warrior Ethos ...............................................................................................349
Clarifying the Definition of Warrior Ethos..........................................................................351
Warrior Attributes Derived from the Tenets of Warrior Ethos ...........................................353
References ................................................................................................................................355
Supplementary Work Product from Warrior Ethos Project .....................................................355
Appendix E: Indicators of Warrior Ethos.....................................................................................356
Methods....................................................................................................................................356
Participants...........................................................................................................................356
Instruments and Facilities ....................................................................................................356
Procedure .............................................................................................................................356
Results ......................................................................................................................................358
Qualitative Findings.............................................................................................................358
Quantitative Findings...........................................................................................................358
Discussion ................................................................................................................................359
Riccio
Prologue:
A Programmatic View of the Inquiry into Outcomes-Based Training & Education
Gary Riccio
Historicity of our Research on OBTE
This unique investigation resulted from an unusual confluence of events that brought key
personnel from the Asymmetric Warfare Group (AWG) together with key personnel in the
science team that ultimately was engaged by the AWG. In September 2007, Fred Diedrich and
I were involved in a project with the Fort Benning research unit of the Army Research Institute
for the Behavioral and Social Sciences (ARI). One task in that project was directed toward
identification of formative measures for instructors in the Basic Noncommissioned Officer
Course (BNCOC) at Fort Benning. In particular, we were focusing on the Small Arms
Proficiency Training Module (SAPT) of BNCOC. Scott Flanagan was invited to participate in the
measure development process known as COMPASS (described in Chapter 2). Scott was working
on Outcomes Based Training and Education (OBTE) with the AWG at that time.
The most important criteria imposed on the COMPASS process for SAPT were the Pentathlete
Characteristics (i.e., Warrior Leader, an Ambassador, a Critical and Creative Thinker, a Leader
Developer, and a Resource Manager). This introduced an exigency to map the measures of
observable instructional behavior to something more abstract, to something more like cultural
values. While the Pentathlete Characteristics in particular ceased to become a priority over the
course of the project, we consider this kind of mapping to be an important general source of
external validity for a set of instructional measures (Sidman, Riccio, Semmens, et al., 2009).
In a prior project, we addressed a set of values-based concepts, those embodied in Warrior Ethos
(Riccio, Sullivan, Klein, Salter, & Kinnison, 2004; see Appendix D). We found that it is possible
to identify relationships between abstract values and concrete behavior of Soldiers in an
operational context or training context. These relationships led to a deeper understanding of
Warrior Ethos in terms of scientifically traceable concepts and in terms of specific actionable
recommendations for the planning and execution of training. This is important because there is a
natural skepticism about the meaning of values-based terminology that changes from time to
time. In general, we suspect that persistence of a relatively small number of core values can be
identified amid such changes in terminology through their common connections to a scientifically
and philosophically meaningful foundation of enduring concepts (see Chapters 3, 4, and 5).
Given the original intent of the ARI project, it was natural to consider common and convergent
themes across projects in which we had first-hand exposure to various programs of instruction in
the institutional Army. Our involvement with such programs was through research, thus we were
more likely to be exposed to programs in which change was taking place or was being
considered. A significant convergence occurred during this project between the SAPT module of
BNCOC and an initiative of the AWG to introduce a different approach to Army training and
education (Outcomes-Based Training & Education [OBTE]) through its initial application to
marksmanship training (see Chapter 1). The AWG initiative in OBTE became important to
consider because our initial assessment of instruction in BNCOC, using the measures developed
for SAPT module, showed gaps between what BNCOC leadership expected instructors and
students to be doing and what was actually occurring in SAPT. At the same time, OBTE was
Prologue
receiving increasing attention in several programs of instruction (see Chapters 1 and 11). Both
our team and BNCOC leadership began to inquire into the lessons learned by the AWG.
By December of 2007, BNCOC leadership began to inquire into the possibility of BNCOC
instructors attending the AWGs marksmanship course. The intent of this course is to familiarize
trainers with OBTE. At the same time, Scott Flanagan brought information back to the AWG
about the COMPASS process and its potential value in bringing scientific rigorousness to OBTE.
By January, LTC Michael Richardson (Baker Squadron Commander at the AWG) began to
inquire into the possibility of developing measures for OBTE by employing the COMPASS
process and the team that ARI was using to develop measures for BNCOC. After some
deliberation about intent and purpose, in April 2008, the team began a three-month effort with the
AWG to define and measure the practice of OBTE and to begin the development of theoretical
underpinnings for the approach. LTC Richardson monitored the effort with the assistance of CSM
Michael Cortes, with the approval of COL Robert Shaw, and with visibility to many key
personnel in the AWG. Scott Flanagan, Morgan Darwin and Blaise Cornell dEchert (CD) were
key informants about OBTE to the science team on behalf of the AWG. The essential activities
and findings of the initial phase of the investigation are described in Chapters 2, 6, 7, 8, 10, and
11.
At the end of October 2008, the science team was re-engaged for further study of OBTE. In
addition to data collection, COL Shaw wanted the team to produce a monograph that would
reveal the current and potential scholarly breadth and depth of OBTE. By this time, LTC
Richardson had transitioned to a new assignment outside of the AWG and was replaced by LTC
Richard Thewes. CSM Cortes became a critical adviser and technical monitor across this change
of command. CD and Morgan continued in their influential roles as informants about the history,
cultural implications, and ongoing exploration of OBTE in the Institutional Army. The essential
activities and findings of the second phase of the AWGs internal scientific inquiry are reported
in Chapters 9, 12, 13, and 14. The engagement of the science team by the AWG ended in June
2009 as planned, consistent with the AWGs Operations Order for OBTE. Chapters 3, 4, 5, 15,
and the Epilogue were written over several months following the end of the investigation. These
chapters reflect the continuing dialog among a variety of stakeholders about the vision, purpose,
decentralized adaptation, and programmatic implications of OBTE.
The science team was co-led by Fred Diedrich and me. Fred was responsible for all data
collection on the ground and for managing the associated personnel, the most extensive part of
the investigation (see e.g., Chapters 8, 9, and 13). I was responsible for development of the
grounded theory for OBTE (see e.g., Chapter 3, 4, and 5) through continuous interactions with the
progenitors of OBTE about the assumptions, history, and expectations of the approach as well as
about their ongoing dialog with stakeholders. Fred also was primarily responsible for ensuring
that observations of OBTE were as concrete and verifiable as possible by emphasize its
manifestations and effects in the behavior of instructors and students. I was primarily responsible
for conducting due diligence on the conscious experience of instructors and students in OBTE to
achieve a deeper level of understanding of the causes and effects of good instruction. We believe
that the balance between behavior and conscious experience is one of the most noteworthy
achievements in this investigation. It helped us identify and stay focused on behavior that is
meaningful and on conscious experience that is grounded in reality.
Amid the division of labor between Fred and me, there was frequent substantial communication
between us on all aspects of the project. The purpose and outcome of this association was a
reciprocal influence between the evolving theory of OBTE and the findings about the practice of
OBTE to move us systematically toward a theory of practice. In particular, our interactions with
Riccio
progenitors of OBTE directly influenced the development of measures utilized in the

investigation (see Chapters 2, 7, and 12) as well as our understanding of the limits of the
investigation and associated gaps in validation of OBTE (see e.g., Chapter 11 and Epilogue). The
findings from the AWGs multifaceted investigation, in turn, influenced the concurrent dialectic
with OBTE progenitors and stakeholders by providing increasingly clear and relevant questions
centered on concrete issues and opportunities in the ongoing implementation of OBTE. The
current level of understanding of OBTE would not have been possible without the empirical
components of investigation and the theory development that it enabled.
The Approach and Lessons Learned from the Research
The research design is consistent with a state-of-the-art Service System Development (SSD)
process area of the Capability Maturity Model Integration for Services (CMMI-SVC) and also
reflects best practices in scientific support of social and programmatic decision-making (Chapters
1 and 11). Accordingly, the investigation addressed development and analysis of stakeholder
requirements (Specific Goal 1 of SSD), development of OBTE as a service system that meets
such requirements (Specific Goal 2 of SSD), and verification and validation of OBTE as a service
system (Specific Goal 3 of SSD). Specific practices associated with SG 3 include performance of
peer reviews, verification of selected service system components, and validation of the service
system. Peer review was addressed primarily in the context of surveys administered by the AWG.
Verification was conducted primarily through field-based observations in Initial Entry Training
(IET) at Fort Jackson, Fort Benning, and Fort Sill that employed measures specifically developed
to enable formative assessment of instructor and student behavior. The empirical approach to
validation focused on surveys of Soldiers who had been deployed to a combat environment after
being exposed to an early version of a field course in marksmanship that demonstrated
instructional principles of OBTE. Validation also was addressed through development of
grounded theory for OBTE based on established lines of scholarship in the behavioral and social
sciences.
becomes explicitly more directed toward staff and faculty development and quality assurance (see
Epilogue). It is noteworthy that our findings suggest that OBTE can motivate individuals to take
ownership of their own learning and development, and that it can increase self-efficacy for
teaching and developing others. The approach motivates a greater interest in instructor-student
interactions and how these interactions affect progress toward developmental outcomes while
satisfying course-specific learning objectives. While the findings show that there is both an
impact of OBTE and considerable room for improvement in Initial Entry Training, the measures
of OBTE provide actionable information about the gaps and shortfalls. The findings and
associated research have implications for all the factors that influence institutional learning
including student diversity and background, capacity for self development, instructional design
and development, instructor education, reward and recognition, individual assessment and
program evaluation, leadership in an institution of learning, leader development, and the relevant
science of measurement.
Given our approach to development of OBTE as an integrated instructional service system, it is
critical to understand what instructors should do and what they believe they should influence in
particular programs of instruction. Toward that end, we developed a set of measures for instructor
behavior that exemplifies OBTE and that can be applied to any program of instruction (Chapters
2, Appendix A). We also developed measures for student behavior that complement the instructor
measures and that help reveal the efficacy of instructor-student interactions (Chapters 7 and 12,
Appendix B). We thus defined OBTE as a verifiable and replicable instructional service system,
Prologue
and we explored its links to established theories. We emphasized the utility of these measures in
formative assessments for instructors and instructional program evaluation. Moving forward, we
stress that definition and measurement of instructor-student interactions and verification of
instructional practices is critical to sustain or improve the quality of any instructional service
system and to transfer any new approach to particular programs of instruction (Chapters 1 and
11). In essence, definition and measurement of OBTE can help turn conventional wisdom into
conventional practice.
We believe that tacit knowledge about good training can be made concrete through programmatic
scientific inquiry. Of necessity, this also reveals connections with domains of training and
education outside the Army. Thus, there is an opportunity to realize the potential for improvement
in virtually everything that influences Army training and education (e.g., see Epilogue) by
adapting specific practices within and beyond Army training and education and by utilizing the
more general scientific underpinnings. The research cited in this monograph provides a
theoretical and empirical basis for shared understanding and continuous coordinated improvement
across multiple organizations and organizational levels in training and education (see Chapters 14
and 15). The most immediate result of this is the ability to develop and promulgate standards for
existing values-based requirements in Army doctrine. The longer-term result is that Army
training and education will be aligned more closely and more completely with Army doctrine and
the operational exigencies it represents.
Documentation of the Research
The purpose of the investigation was to define and measure OBTE to confirm that it satisfies its
intended requirements and that it will satisfy end-user expectations during actual implementation
of the approach. More specifically, we utilized multiple sources of evidence to understand the
implementation of OBTE and its effects on thought and behavior. The structure of the monograph
reflects these means and ends (Figure 1). As is evident in this structure, there was not a linear
progression through the chapters with respect to development of OBTE as an integrated
instructional service system. Consistent with spiral development, verification and validation both
informs and is informed by the ongoing refinement of stakeholder requirements and continuous
improvement of OBTE as a mature service system. There are reciprocal relationships among
these concurrent systems engineering activities. The AWGs intent is to develop OBTE into a
mature service system that can be transitioned to the institutional Army.
Some of the chapters will have straightforward utility to instructors and their chains of command
(e.g., Chapters 2, 12, and the Appendices); that is, they are actionable without undue
interpretation. It is not, however, a users manual for Soldiers and instructors. The primary
purpose of this document is to support programmatic decision-making (Chapters 1 and 11). In
this respect, the most important contribution of this document is that it reveals the depth and
breadth of evidence necessary to support decisions about instruction pursuant to requirements in
existing Army doctrine. This includes both methods of assessment and grounded theory that are
well established in the scientific literature. We also point to scholarship beyond science that may
be necessary to understand the inevitable interactions with the unknown that presumably are the
reason for emphasis on adaptability and values in preparing Soldiers for Full Spectrum
Operations. With grounding in the science that is relevant to values-based adaptability, and
recognition that science also has its limits in this regard, this monograph lays the foundation for
development of materials specifically for Soldiers and instructors (see e.g., the Epilogue).
Riccio
Figure 1. Evolution of the investigation as reflected in the chapters of this monograph.

References
Bruny, T., Riccio, G., Sidman, J., Darowski, A., & Diedrich, F. (2006). Enhancing warrior ethos
in initial entry training. Proceedings of the 50th Annual Meeting of the Human Factors and
Ergonomics Society, San Francisco, CA.
Freeman, J., Jason, J., Aten, T., Diedrich, F., Cooke, N., Winner, J., Rowe. L., & Riccio, G.
(2008). Shared Interpretation of Commander's Intent (SICI). Final Report to the Army
Research Institute for the Behavior and Social Sciences, contract number W74V8H-06-C0004.
Riccio, G., dEchert, B.C., Lerario, M., Pound, D., Bruny, T., & Diedrich, F. (2006). Enhancing
Joint Task Force Cognitive Leadership Skills. Report to the Army Research Institute for the
Behavioral and Social Sciences, contract number Army contract no. W74V8H-06-P-0186.
Vienna, VA: The Wexford Group International.
Riccio, G., Lerario, M., Cornell dEchert, B., Pound, D., Bruny, T., & Diedrich, F. (2006).
Training a Joint and expeditionary mindset. Report to the Army Research Institute for the
Behavioral and Social Sciences, contract number W74V8H-06-P-0189. Vienna, VA: The
Wexford Group International.
Riccio, G., Sullivan, R., Klein, G., Salter, M., & Kinnison, H. (2004). Warrior ethos: Analysis of
the concept and initial development of applications. ARI Research Report 1827. Arlington,
VA: US Army Research Institute for the Behavioral and Social Sciences.
Sidman, J., Riccio, G., Semmens, R., Geyer, A., Dean, C., & Diedrich, F. (2009). Reshaping
Army institutional training: Current training. Final Report to the Army Research Institute for
the Behavior and Social Sciences, contract number W74V8H-04-D-0047 DO 0010.
Section I. Development of Stakeholder Requirements for OBTE
Riccio et al.
Chapter 1. Preparation for Full Spectrum Operations
Gary Riccio, Scott Flanagan, Morgan Darwin, & Blaise Cornell dEchert
The mission of the Asymmetric Warfare Group (AWG) is to provide operational advisory
assistance in support of Army and Joint Force Commanders to enhance the combat effectiveness
of the operating force and enable the defeat of asymmetric threats. The AWG executes missions,
responsibilities, and functions required to enhance the capabilities of U.S. units by making them
more adept at quickly identifying and attacking enemy vulnerabilities, and by preparing them for
a broader spectrum of threats (Cornell-dEchert, 2009a,b; Shaw & Cortes, 2009). The methods
used by the AWG to promulgate best practices in asymmetric warfare range from seminars to
field-based training. In March of 2008, the AWG initiated a scientific and technical effort to help
define and develop Outcomes-Based Training and Education (OBTE) (Darwin, 2008a,b). The
objectives of this research include:
Identify the unique and essential attributes of OBTE, as an innovation in training, with
sufficient depth and clarity to enable it to be validated, verified, replicated, and
generalized.
Develop measures for instructor and trainee behavior that reflect OBTE principles in the
context of basic and advanced rifle marksmanship in initial entry training.
Develop measures for the intangible effects of OBTE in basic training and beyond to
their potential impact on Full Spectrum Operations.
Demonstrate how evidence can be gathered for the effects of OBTE using measures that
can be traced to scientific, operational, and institutional principles and practices.
Identify organizational factors affecting OBTE implementation (including life-cycle
management) in Army institutions of training and education.
Identify the extent to which initiatives analogous to OBTE can be developed to achieve
similar impact on other programs of training and education.
Definition and measurement of OBTE is timely because approaches related to OBTE are being
introduced into a variety of programs of instruction in the Army (Cavallaro, 2008; Cox, 2008;
Currey, 2008; Ferguson, 2008; Foster, 2009; Markin, 2008; Perry & McEnery, 2009; Sellers,
2008, Tice, 2008). Given this interest, the intent is to develop OBTE into an integrated set of
instructional capabilities that meet the needs of Full Spectrum Operations. Adoption of any such
capabilities in the institutional Army ultimately would require compatibility with the approach of
the United States Army Training and Doctrine Command (TRADOC) to instructional systems
design and associated processes such as analysis, design, development, implementation and
evaluation (TRADOC, 1999).
At present, TRADOC Regulation 350-70 and several key pamphlets associated with this
regulation are being revised to facilitate innovation in training and education. This is important
because there is a need to maintain rigorous systems engineering while expediting adaptation to
the changing needs for instruction in the Army (Swain, 2005). For these reasons, the present work
was influenced by state-of-the-art approaches in the DoD and Industry to the use of systems
engineering in the integration and development of capabilities (Chairman of the Joint Chiefs of
Staff, 2004; Chrissis, Konrad, & Shrum, 2003; Sage & Rouse, 1999; Schaeffer, 2005). In
particular, our adherence to the practices of the Capability Maturity Model Integration for
Services (CMMI Product Team, 2009) increases the likelihood of upward compatibility with
future guidance about Army Training and Education Development (ATED) as well as viability of
the instructional capability over its life cycle.
Preparation for Full Spectrum Operations
This report of our research for the AWG is presented in a format consistent with the Service
System Development (SSD) process area of CMMI for Services. Accordingly, section 1.1 and 1.2
respectively summarize activities of the AWG to develop and analyze stakeholder requirements
(Specific Goal 1 of SSD) and to develop a service system that meets such requirements (Specific
Goal 2 of SSD). Chapters 2 and 3 describe in some detail the practical and scientific
considerations in developing and analyzing stakeholder requirements. Chapters 4 and 5 develop a
unique scientific foundation for OBTE based on our collaborative inquiry with the progenitors
and stakeholders of OBTE. The remainder of the report addresses the scientific and technical
effort to verify and validate the service system (Specific Goal 3 of SSD). Section 1.3 describes
the preparation for verification and validation (Specific Practice 3.1 of SSD). Chapters 2, 7, and
12 also are relevant to Specific Practice 3.1. Chapter 6 describes performance of peer reviews
(Specific Practice 3.2 of SSD). Chapters 7, 8, 9, and 13 describe verification of selected service
system components (Specific Practice 3.3 of SSD). Chapter 10 describes the initial exploration of
methods for validation of the service system (Specific Practice 3.4 of SSD). Chapters 4 and 5 also
are relevant to Specific Practice 3.4.
The interleaving of the chapters in this monograph with respect to service system development
reflects the spiral development approach we utilized in this investigation (see Prologue). This
approach is a best practice in capabilities development and continuous process improvement
(Chairman of the Joint Chiefs of Staff, 2004; Chrissis, Konrad, & Shrum, 2003; Sage & Rouse,
1999; Schaeffer, 2005). Implications for continuous process improvement are summarized in
Chapters 11, 14 and the Epilogue. The implications are as significant for Army training and
education, in general, as they are for OBTE in particular. The reason is that the opportunities for
improvements in the implementation of OBTE reveal precisely where there are shortfalls and
gaps in current training with respect to the Army doctrinal requirements and the associated needs
of Full Spectrum Operations.
1.1 Requirements of Full Spectrum Operations
Full Spectrum Operations (FSO) are characterized by dynamic adjustments in objectives, roles,
responsibilities, tactics, techniques, and procedures (French, 2002; Kilcullen, 2006; Meigs, 2003;
Petreaus, 2006). Such insights about contemporary conflict emphasize that most if not all
individual and collective tasks critically depend on an understanding of context and the
relationships among elements of the operational context. Appreciation of contextual relationships
reveals higher-order invariants across combat and non-combat operations and, thus, facilitates
transition between them. This is important given that, in counterinsurgency operations, transitions
between combat and non-combat situations can occur over time scales that are short relative to
ones physiological reactions to such situations. In such environments, an outward orientation can
help direct attention away from physiological distractions of potential life-and-death situations
(cf., Kolditz, 2007; Lerner & Keltner, 2001; Lerner & Tiedens, 2006; Lukey & Tepe, 2008). In
principle, it can help Soldiers remain grounded and persevere in the trans-extremis environments
of FSO. An outward orientation is fostered by skills of perception and action that involve habitual
orientation to ones capabilities for interaction with the surroundings (cf., J. Gibson, 1977, 1979)
as well as by the associated understanding of ones efficacy (cf., Bandura, 1977, 1997).
There is a consistent thread through Army doctrinal publications that calls for individual cunning
and collective agility in unpredictable and changing environments (Riccio, Sullivan, Klein, Salter,
& Kinnison, 2004; Shaw & Cortes, 2009). The Institutional Army is expected to produce Soldiers
who have the personal attributes and values necessitated by such agility.
Riccio et al.
Soldiers and units at every level must be flexible and adaptive. Often, stability
operations require leaders with the mental and physical agility to shift from
noncombat to combat operations and back again. (Headquarters Department of the
Army [HQDA], 2001, p. 9-5).
Simultaneously, individual Soldiers, NCOs, warrant officers, officers, and the
civilian work force are responsible for training themselves through personal selfdevelopment. Training is a continuous, lifelong endeavor that produces competent,
confident, disciplined, and adaptive Soldiers and leaders with the warrior ethos in our
Army. Commanders have the ultimate responsibility to train Soldiers and develop
leaders who can adjust to change with confidence and exploit new situations,
technology, and developments to their advantage. (HQDA, 2002, p. 1-5).
Commanders ensure that their subordinates know how to think instead of what to
think. They develop their subordinates confidence and empower them to make
independent, situational-based decisions. The goal is to develop subordinates who
have an agile and adaptive mindset. (HQDA, 2003, p. 2-12).
In 2003-2004, it was clear that senior Army leaders recognized training gaps and shortfalls with
respect to the Global War on Terror (dEchert, 2009; Shaw & Cortes, 2009). The AWG training
advisory teams debriefed units returning from Operation Iraqi Freedom (OIF) and Operation
Enduring Freedom (OEF). Without exception, leaders at all levels emphasized that the key to
mission success was proficiency in basic warfighting skills such as moving, shooting,
communicating, and medical skills. Proficiency at these fundamentals allowed units to respond
quickly to new conditions and non-standard missions. Senior leaders were especially critical of
certain aspects of pre-deployment training (Flanagan, 2005; French, 2002).
Based on constant interaction with active duty units in theater, the AWG also recognized a need
for training that incorporates situations in which Soldiers and leaders are encouraged to think, act,
and make decisions as a training event unfolds. The impact of situated training is critically
interdependent with a Soldiers mastery of basic skills (Flanagan, 2005; Darwin, 2008a,b). A key
insight, however, is that the definition of basic Soldier skills implicit in the AWGs use of this
term is broader than its common usage. In the present context, basic Soldier skills should not be
confused with the lowest level of detail, or even perhaps the most basic level of detail, in a typical
task analysis of individual and collective tasks. A different perspective is suggested by the
following observation:
Their honed skills will be validated and sometimes raised to a higher level. Their
thinking skills are challenged; this will prompt review and contemplation from
individual Soldiers and leaders, a comprehensive assessment of teamwork learned,
and new ideas, innovations, and techniques discovered. (Flanagan, 2005, p. 1)
From this perspective, a situation provides a context for linkage of tasks and understanding the
concurrent and downstream consequences of individual actionsthe levels situation awareness
required for adaptability (J. Gibson, 1977; E. Gibson, 1988; cf., Endsley, 1995). Without a basic
level of skill, however, it is difficult to appreciate the opportunities a situation affords and the
implications of ones action in that situation. And, without such an appreciation, it is difficult to
develop a deeper understanding of a skill and reach a higher level of skill. Put simply,
understanding ones own skill includes knowledge about what one can do with it; and ones
understanding of a situation is at least partially dependent on what one can do in that situation (J.
Gibson, 1977; 1979).
10
The implication is that training should address basic skills of perception and action that are
critical to proficiency in moving, shooting, and communicating. This requires a finer grain of
analysis, a level of analysis that also cuts across the distinctions among these tasks. This insight is
a source of innovation in Army training. It is not radical insofar as it is consistent with relatively
neglected implications of a theoretical framework that is familiar in the Army (Gagne, 1962,
1985)training and education for military tasks can benefit from consideration of basic scientific
understanding of purposeful perception and action (Gagne & J. Gibson, 1947) as well as
motivation and emotion. In chapters 3-5, we extend this foundation in meaningful perception and
situated action to philosophically consistent views of social and cultural factors affecting selfdevelopment and, in particular, the development of values.
The AWG analysis of stakeholder requirements for Army training can be summarized in three
conjectures:
Conjecture 1: Instruction can be designed, developed and implemented with respect to basic
skills that lead to better understanding of a situation including relationships among elements
within a relevant situation, that motivate and guide engagement with that situation, and that
afford opportunities to influence the situation instead of merely responding to it. There is a
need to identify this level of analysis for instruction and learning that fosters the development
of basic Soldier skills necessary for success in Full Spectrum Operations.
Conjecture 2: Instruction can be designed and implemented in verifiable ways that apply to
all levels of leadership and leader development for a curriculum or program of instruction. A
general approach to Army training and education can be developed that applies to all
interactions between an instructor and a learner, as well as to self-development, without
sacrificing validity with respect to Full Spectrum Operations. There is a need for a verifiable
and valid approach that can be utilized by instructors and learners together or by learners on
their own across all types of learning events.
Conjecture 3: Soldiers do not rise to the occasion in combat; they fall back to their highest
level of training and education. The highest level of training and education should relate
directly and unambiguously to the most important requirements in Army Doctrine, such as
Army Values and Warrior Ethos. There currently are no doctrinally codified standards for
such values-based requirements. There is a need for values-based standards that can be
applied to any and all learning events.
1.2 Outcomes-Based Training and Education (OBTE)
In response to feedback from active duty commanders and in anticipation of ongoing challenges,
the AWG developed a set of briefings, seminars, workshops, and field-based training courses to
enhance combat skills and to better prepare Soldiers for Full Spectrum Operations (dEchert,
2009; Flanagan, 2005; Shaw & Cortes, 2009). A combat skills course was conceived as a model
for noncommissioned officers (NCO) to set up and conduct training (Flanagan, 2005). The
approach to this course was heavily influenced by decades of lessons learned by special units in
the military for which training is a persistent priority and core value. The intent of such an
approach is "through mentorship and coaching, demonstrate a method of training which is safe,
effective, combat relevant, and encourages a continuous thought process which will demand
accountability" (McNamara, 2008, p. 2). This led to the Combat Applications Training Course
(CATC), a course that demonstrates how to address context (Conjecture 1) and learning to learn
(Conjecture 2) with respect to marksmanship but in ways that are general in principle. The
11
Riccio et al.
specific context of marksmanship allowed the AWG to follow through with instructors and their
chain of command to examine viability of the approach in existing programs of instruction in the
Generating Force (Conjecture 3).
Packaged with broader forums for education of instructors and leaders, and generalized to skills
other than marksmanship, the approach currently is implemented as a set of instructional
capabilities referred to as OBTE (Darwin, 2008b; Cornell-dEchert, 2009a,b). OBTE
systematically seeks to engage an organization to shape instruction and influence the overall
development of an individual, as a Soldier, with respect to broad outcomes that transcend coursespecific objectives. OBTE strives to develop Soldiers who can think and behave adaptively amid
uncertainty, Soldiers who can learn in any situation, and who continuously improve as individuals
and as members of a unit (see e.g., Chapters 2, 3, and 15).
1.2.1 Exemplar of OBTE: Combat Applications Training Course
As a means of demonstrating OBTE and providing context for understanding the principles, the
AWG adapted a field-based course in rifle marksmanship, referred to as the Combat Applications
Training Course (CATC) (Darwin, 2008a). It was originally developed as a course in
marksmanship as provided, for example, to the 82nd Airborne Division (Aug-Dec 2006) and the
101st Airborne Division (Feb-Jun 2007). In CATC, over a period of one week, instruction
progresses from introductory levels (e.g., weapons system function, components and
maintenance, shooting from a prone position) to more advanced and complex situations that
combine moving, shooting, and communicating (e.g., highly mobile engagement of multiple
targets from multiple positions). The instructional content covers elements of both Basic Rifle
Marksmanship (BRM) and Advanced Rifle Marksmanship (ARM). It includes situated exercises
that are complex and challenging due to the introduction of teamwork and activities focused on
problem solving. While the course incorporates some lecture and discussion sessions, the
majority of the course is experiential in nature (cf., Reed, 1996; Riccio, 1993).
CATC has evolved into a course with the primary intent of educating leaders and instructors
about OBTE (Darwin, 2008b). The earliest examples of this train the trainer course were at Fort
Benning and Fort Jackson in 2007-2008. The course attempts to promulgate a developmental
approach to instruction primarily by leveraging the credibility and influence of highly
accomplished instructors as role models for instructional strategies that exemplify OBTE
(Bandura, 1977; Riccio et al., 2004). General features of this developmental approach include
transition from structured command-oriented tasks to less formal self-guided activities, as well as
a gradual increase in task complexity and challenges (Darwin, 2008a,b; cf., Tobias & Duffy,
2009).
An illustrative example of OBTE in marksmanship training is a gradual change in range control
during the course of training, providing less directive communication, and applying less
conventional controls to demonstrate how marksmanship training can be achieved safely while
encouraging individual accountability. Rather than the instructor assuming responsibility for
safety alone, the students are expected to assume that responsibility along with instructors. This is
important because, from the perspective of OBTE, Soldiers ultimately will be accountable in the
field. Such accountability should be promoted as early as possible in the development of Soldiers.
In the application of OBTE to marksmanship, students have accountability for behavior such as
operationally appropriate weapons orientation (down and ready) and clearing (self).
In addition to this focus on individual accountability, any implementation of OBTE should
emphasize understanding the why and developing problem-solving skills. For instance, during
breaks from experiential exercises, After Action Reviews (AAR) are used by instructors to
12
introduce and elaborate on background fundamentals and changes to the training environment. In
marksmanship training, for example, focus is placed on understanding ballistics and why
adaptations are needed (e.g., what happens when firing up or down a hill). Many events are
applied as problems for students to solve (Darwin, 2008a,b; Savory & Duffy, 1995; Vandergriff,
2006). An example is the use of situations in which students effectively manage resources such as
time and ammunition. Critically, in such situations, students are not told solutions that they then
practice. Rather, the students construct solutions under the guidance of an instructor.
1.2.2 OBTE as a Multifaceted Instructional System
An essential attribute to OBTE, as an innovation in instruction, is the critical role for leadership
throughout the organization responsible for instruction. The view of the AWG about this role is
described well in the following passage (quoted in Shaw & Cortes, 2009):
If the transformed Army will require leaders who can operate independently in the
absence of close supervision, the current leader development experience of company
command will have to change. Consequently, the author asks for senior leaders not to
do more, but to do less and thus give subordinates more freedom to innovate. (Wong,
2002, p. v)
There is much implied in this recommendation. One important and difficult implication is that
leaders at all levels should create a climate of trust in their subordinates. This should not be
confused with blind trust. The implied trust is in providing subordinates with opportunities for
initiative in which they are accountable for their decisions. At the same time, a deeper level of
trust is engendered by leaders who demonstrate a willingness to share accountability for any
unintended consequences of the initiative they encourage in their subordinates. Such sharing of
accountability becomes productive and collaborative if there is a serious commitment to AAR.
Such collaborative reflection identifies steps that can be taken to improve training by minimizing
undesirable consequences of a new approach and to sustain the good training that results.
Another important implication of leadership in instructional programs is the demonstration of
common or convergent objectives amid division of labor throughout the command chain. Leaders
can do this, for example, in walkthroughs conducted in the context of peer review (Chrissis et al.,
2003; CMMI Product Team, 2009). Senior leaders who do this provide a model for their
subordinate leaders, both senior and subordinate leaders provide a model for instructors, and
instructors provide a model for their students. It reveals that there is a common stake in learning
and, in the spirit of peer review, an interest in individual adaptation to maximize opportunities for
learning, whatever the situation. Such engagement by leaders is essential from the perspective of
OBTE (Haskins, 2009; Schwitters, 2009; Shaw & Cortes, 2009; Vandergriff, 2007; Appendix C).
The AWGs intent is to develop OBTE into a mature service system that can be transitioned to
the institutional Army. It recognizes that leadership is an essential attribute of the approach. Thus,
a series of workshops, seminars, and short courses were developed for leaders throughout the
command chain for a program of instruction (Darwin, 2008a,b,). These engagements with leaders
are viewed, together with field-based courses for instructors, as an integrated set of instructional
capabilities for an organization (Cornell-dEchert, 2009a,b). Validation of OBTE in a particular
setting will reflect the influence of all such components of the instructional service system
irrespective of the extent to which they reflect OBTE. Thus, leadership influenced by OBTE
should have a constructively additive or multiplicative effect on instruction influenced by OBTE.
If only one of these components of an instructional system is influenced by OBTE, their influence
will tend to cancel or interfere with each other.
13
Riccio et al.
1.3 An Appraisal of Instruction with Respect to OBTE

1.3.1 A Systems Engineering Framework for Integration and Development of OBTE
To date, OBTE has been explored in a variety of programs of instruction in the Army (Cox, 2008;
Cavallaro, 2008; Currey, 2008; Ferguson, 2008; Markin, 2008; Sellers, 2008, Tice, 2008; see also
Vandergriff, 2006, 2007). Growing interest in OBTE recently was exemplified by the attendance
at a workshop organized by the AWG and the Johns Hopkins University. Workshop participants
represented twenty-six locations across HQDA, FORSCOM, Joint, Inter-agency, and allied
organizations (Devens, 2009). Given this widespread interest, it is important to understand the
issues pertaining to implementation of OBTE and how it can be improved as an integrated set of
instructional capabilities (Cornell-dEchert, 2009a,b). This will enable the AWG and others to
help stakeholders translate their interest into action, whether for initial implementation or process
improvement.
A rigorous framework within which to describe and guide the steps toward transition of OBTE is
provided by CMMI (Chrissis, Konrad, & Shrum, 2003; Schaeffer, 2005). Use of CMMI should
not be interpreted as the imposition of onerous processes onto a service system (e.g., OBTE) that
is intended to foster innovation. To the contrary, CMMI enables initiative by ensuring that there
is accountability within a rigorous yet flexible framework for understanding the evolving and
interrelated activities of an organization (Ahern, Clouse, & Turner, 2008; Garcia & Turner,
2006). Rigorous description of intent and execution provides desirable constraints on innovation,
while flexibility of the framework allows an organization to make sense of the results of
innovation (Weick, 1995).
CMMI is used to describe the current research and its implications for continuing integration and
development of OBTE as an instructional service system (CMMI Product Team, 2009). From the
perspective of CMMI, the AWG has been performing and managing OBTE at a level that is
commensurate with capability level 2.
A capability level 2 process is characterized as a managed process. A managed
process is a performed (capability level 1) process that has the basic infrastructure in
place to support the process. It is planned and executed in accordance with policy;
employs skilled people who have adequate resources to produce controlled outputs;
involves relevant stakeholders; is monitored, controlled, and reviewed; and is
evaluated for adherence to its process description (CMMI Product Team, 2009, p. 2324).
The intent of the current research is to enable the implementation of OBTE at a level
commensurate with capability level 3; that is, to enable its institutionalization (CMMI generic
goal, GG 3). The generic practices at this level are to establish a defined process (GP 3.1) and to
collect improvement information (GP 3.2). A premise of the investigation is that the behavioral
and social sciences are indispensable to definition and measurement for a service system at
capability level 3.
A capability level 3 process is characterized as a defined process. A defined process
is a managed (capability level 2) process that is tailored from the organizations set of
standard processes according to the organizations tailoring guidelines and
contributes work products, measures, and other process improvement information to
the organizational process assets A defined process clearly states the purpose,
14
inputs, entry criteria, activities, roles, measures, verification steps, outputs, and exit
criteria (CMMI Product Team, 2009, p. 24).
The appropriate guidance for OBTE at capability level 3 is the Service System Development
process area within CMMI for Services (CMMI Product Team, 2009, pp. 437-462). The specific
goals for this process area are: develop and analyze stakeholder requirements (SG 1), develop
service system (SG 2), and verify and validate service system (SG 3). The activities associated
with SG 2, reviewed above, are treated in more detail elsewhere (Darwin, 2008a,b; CornelldEchert, 2009a,b). The activities associated with SG 1 and SG 3 relate directly to the current
research and, in the context of spiral development, have explicit implications for SG 2.
The specific practices associated with SG1 of the Service System Development process area are:
develop stakeholder requirements (SP 1.1), develop service system requirements (SP 1.2),
analyze and validate requirements (SP 1.3). The specific practices associated with SG 3 are:
preparation for verification and validation (SP 3.1), performance of peer reviews (SP 3.2),
verification of selected service system components (SP 3.3), and validation of the service system
(SP 3.4).
1.3.2 Preparation for Validation and Verification
The purpose of the investigation was to verify and validate OBTE to confirm that it satisfies its
intended requirements and that it will satisfy end-user expectations during actual service delivery.
More specifically, our objective is to understand the implementation of OBTE and its effects on
thought and behavior, and where possible, to get some clues about its impact on instruction and
learning. To accomplish this objective, we utilized multiple sources of evidence from a variety of
techniques. While no single data collection method alone will yield comprehensive results, a
combination of approaches typically yields deeper understanding (e.g., CMMI Product Team,
2009; Jackson, Woods, Durkee, OMalley, Diedrich, Aten, et al., 2008; Kirkpatrick, 1994;
Tashakkori & Teddlie, 1998). In particular, our inquiry involved asking via self-report survey and
watching via observation (Cresswell, 1998; Ellingson, 2009; Miles & Huberman, 1994). The
sources of evidence addressed in Section II of this report are listed in Table 2. Section III
provides additional evidence that builds on this foundation for further development of OBTE.
We began by analyzing results from web-based surveys that were solicited by the AWG from
Soldiers who had participated in CATC. The open-ended responses to these items provided
evidence regarding impressions of participants about the usefulness and potential application of
the course. This form of peer review is consistent with Kirkpatricks Level 1 of program
evaluation in which reactions of participants are collected (Kirkpatrick, 1994). Such reactions are
indicative of the motivation a student might experience in a course, and they can be suggestive
about whether the course achieves its intent, but they do not necessarily indicate why. For this
reason, consistent with peer review in CMMI, attention was focused on the potential use of such
surveys to identify defects or to make recommendations about changes to improve OBTE (CMMI
Product Team, 2009, p. 457).
An essential attribute of CATC as a train-the-trainer course, and of OBTE in general, is its
emphasis on instructor-student interactions. The behavior of instructors as role models, mentors,
facilitators and advisors is assumed to be critical to the development of individuals. For this
reason, an established methodology was employed to identify such behavior, to define its
relationship to OBTE principles and related scientific research, and to develop measures for
verification of OBTE in practice (MacMillan, Entin, Morley, & Bennett, in press; Chapter 2). The
objective of the verification was to provide data for a formative assessment of OBTE and
15
Riccio et al.
feedback to instructors about the efficacy of their instructional behavior (Bransford et al., 2000;
CMMI Product Team, 2009, p. 460).
Table 2. Questions addressed by the various sources of evidence addressed in Section II.
Type of evidence and source
Questions addressed
Peer Review
Was CATC perceived as valuable overall?
Self report data from Soldiers who participated

in CATC
Was CATC perceived as valuable in

increasing broader skills targeted by OBTE?
Source: Web-based survey conducted after

CATC participation
What suggestions did Soldiers have for

improving CATC?
Kirkpatrick Levels 1
Verification (Primary)
Observed behavior for DS who had taken
CATC, acting as instructors in marksmanship
course
To what extent did DS exhibit the OBTE

behavior that was trained in the CATC?
Kirkpatrick Levels 2 (DS)
Source: Observational rating form during

marksmanship course
Verification (Secondary)
Observed behavior for Privates in
marksmanship course taught by DSs who had
taken CATC
To what extent did Privates exhibit high

marksmanship performance?
To what extent did Privates exhibit OBTE
behavior?
Source: Observational rating form during

marksmanship course
Kirkpatrick Level 2 (Privates)
Verification (Secondary)
Were communication patterns consistent

with OBTE principles?
Communication between DS and Privates

Source: Communications rating form during
marksmanship course
Validation (Secondary)
Correlation between OBTE behavior of DS and
performance of Privates
Source Observational rating forms for DS and
Privates during marksmanship course
Validation (Primary)
Survey of deployed Soldiers who had taken the
initial iterations of CATC
Kirkpatrick Level 2 (DS)

Was a higher level of OBTE behavior in DS
associated with higher marksmanship
performance by Privates?
Kirkpatrick Level 3 (DS)
Did deployed Soldiers find their OBTE
training to be valuable upon deployment?
Kirkpatrick Level 3 (toward Level 4)
Source: Survey following return from

deployment
The same methodology was employed to develop measures for student behavior that would be
sensitive to the effects of OBTE. These measures are consistent with Kirkpatricks Level 2 of
program evaluation that attempts to assess learning. This level is complicated in general because
it often addresses nested learning objectives (Kirkpatrick, 1994). In the present context, measures
16
are required that relate both to course-specific learning objectives as well as the longer-term
developmental outcomes that are the reason for OBTE (Cornell-dEchert, 2009b; Darwin, 2009).
Measures to assess learning of both instructors and students are equally important for verification
of OBTE (Bransford et al., 2000; Cornell-dEchert, 2009a,b).
Verification is central to the development of OBTE into a defined process. Thus, the focus and
timing of the investigation was dictated by opportunities to develop and apply OBTE measures in
a program of instruction in which one could expect substantive variation in behavior of
instructors and students. An opportunity emerged from a systematic engagement of the AWG at
Fort Benning and Fort Jackson. The AWG started providing OBTE services at these posts in
August 2007 and continued through the end of this investigation. The engagement at Fort Jackson
included the Center of Excellence for Basic Combat Training, Drill Sergeant School cadre,
Victory University, and Basic Combat Training drill sergeants. The engagement at Fort Benning
included instructors in noncommissioned officer education system, instructors in the officer
education system, drill sergeants in 198th Infantry Training Brigade (one station unit training),
and drill sergeants in the 192nd Basic Combat Training Brigade.
The engagements at Fort Benning and Fort Jackson included briefings, seminars, and workshops
as well as CATC. There was due diligence to ensure that the command chain for instructors
understood the intent and approach of OBTE. This generally included first-hand experience as a
participant in CATC or in an abbreviated (three-day) version of this field course referred to as the
Senior Leader Course. The buy-in of senior leaders was considered critical to establishing a
command climate favorable to OBTE. At Fort Jackson, the Commander of the Basic Training
Center of Excellence and post Command Sergeant Major championed OBTE (Schwitters, 2009).
The Training Brigade Commanders at Fort Benning also made OBTE a priority both in its
implementation and in exercising initiative and leadership its continuing development (Haskins,
2009). Empowering such leaders with information about best practices in OBTE is considered to
be a critical component of OBTE.
By April 2008 the command climate at Fort Jackson and Fort Benning was sufficiently favorable,
and sufficient numbers of Drill Sergeant (DS) had participated in CATC, to conduct an appraisal
of OBTE in the context of Initial Entry Training. Data were collected on DS and students
(Privates) at these posts to verify that instructor behavior consistent with OBTE could be
identified and measured in situ. The relationships between instructor behavior and student
behavior are relevant to Kirkpatricks Level 3, with respect to DS not Privates, insofar as these
data are indicative of DS competency in their job as instructors or as influencers of Privates
(Kirkpatrick, 1994). As such, these data also relate to validation in CMMI-SVC, however, only
secondarily so because the focus of OBTE is on longer-term developmental outcomes about
which one can only get suggestive evidence at the time of instruction.
Our primary method of addressing validation of OBTE focused on outcomes more likely to
reflect long-term developmental impact. In the context of Kirkpatricks level 3 and level 4 of
program evaluation, the most relevant feedback is about performance of Soldiers, as such, in
theater. Level 4 emphasizes the contributions of training to the organizational mission and
objectives (Kirkpatrick, 1994). Toward that end, we developed a post-deployment survey. The
survey was administered to members of the 82nd Airborne Division who, prior to deployment, had
taken an early version of CATC (i.e., before the AWG attempted to implement it more explicitly
as a train-the-trainer course). This particular survey provided further self-report evidence
regarding the influence of the course on preparedness of individuals for combat operations and as
trainers in their own units during deployment. These results therefore highlight the potential of
17
Riccio et al.
OBTE as an approach, as well as the challenges of the approach, for achieving long-term
developmental outcomes in the context of training for proficiency in a particular skill.
Collectively, these various sources of evidence provide a unique look into the potential of OBTE
with respect to assessment of instructor behavior, influence on student behavior, and the
downstream consequences in theater. This multi-faceted appraisal was possible because of the
rigorous definition and methods for measurement of OBTE (e.g., Chapter 2) coupled with clear
statements of intent and purpose (Cornell-dEchert, 2009a, b; Darwin, 2008a, b).
1.4 References
Ahern, D., Clouse, A. & Turner, R. (2008). CMMI distilled: A practical introduction to
integrated process improvement. Upper Salle River, NJ: Addison-Wesley.
Bandura, A. (1977). Social learning theory. New York: General Learning Press.
Bandura, A (1997). Self-efficacy: The exercise of control. New York, NY: Freeman and
Company.
Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000). How people learn: Brain, mind,
experience, & school. Washington, DC: National Academy Press.
Cavallaro, G. (2008). The art of making better drill sergeants. Army Times, Jun 23.
Chairman of the Joint Chiefs of Staff (2004). Joint capabilities integration and development
system. Retrieved November, 2004, from Defense Technical Information Center,
http://www.dtic.mil/cjcs_directives/cdata/unlimit/3170_01.pdf.
Chrissis, M.B., Konrad, M. & Shrum, S. (2003). CMMI: Guidelines for process integration and
product improvement. Boston, MA: Pearson Education.
CMMI Product Team (2009). CMMI for services, version 1.2. (CMU/SEI-TR-2009-001; ESCTR-2009-001). Pittsburgh, PA: Carnegie Mellon University.
Cox, M. (2008). Zeroed in on marksmanship: Army overhauls weapons training and qualification.
Army Times. May 5, 2008, 14-16.
Cornell-dEchert, B. (2009a). An introduction to outcomes-based training and education. Fort
Meade, MD: Asymmetric Warfare Group.
Cornell-dEchert, B. (2009b). Outcomes-based training and education: Implementation guide.
Fort Meade, MD: Asymmetric Warfare Group.
Cresswell, J.W. (1998). Qualitative inquiry and research design: Choosing among five traditions.
Thousand Oaks, CA: Sage.
Currey, C.J. (2008). Outcomes-based training: Whats next? Initial Entry Training Journal, 1, 13.
Darwin, M. (2008a). Asymmetric Warfare Group combat applications training course (CATC)
senior leader discussion (Briefing, January 23, 2008). Fort Meade, MD: Asymmetric Warfare
Group.
Darwin, M. (2008b). Outcomes-based training and education: fostering adaptability in full
spectrum operations (Briefing, December 2008). Fort Meade, MD: Asymmetric Warfare
Group.
Darwin, M. (2009, March). Developing outcomes. Presentation at the US Army Asymmetric
Warfare Group workshop on Outcomes-based Training and Education, Applied Physics
Laboratory, Johns Hopkins University, Laurel, MD.
Devens, M. (2009, March). Introductory remarks. Presentation at the US Army Asymmetric
Ellingson, L. (2009). Engaging crystallization in qualitative research: An introduction. Los
Angeles, CA: Sage.
18
Endsley, M.R. (1995). Toward a theory of situation awareness in dynamic systems. Human
Factors, 37, 32-64.
Ferguson, M.E. (2008). Outcomes based training and education: Targeting the intangibles. The
NCO Journal, October, 2008, 16-21.
Flanagan, S. (2005). Combat skills. Unpublished manuscript (Nov. 16, 2005).
Foster, C. (2009). No approved solutions in asymmetric warfare: Nurturing adaptive leaders in
an outcomes-based training environment. Assembly, July/August, 28-29.
French, D., (2002). Wake up and smell there is something wrong. News From the Front, SeptOct 2002. 26-29. Ft. Leavenworth, KS: Center for Army Lessons Learned.
Gagne, R. (1962). Military training and principles of learning. American Psychologist, 17, 263276.
Gagne, R. (1985). The Conditions of Learning (4th ed.). New York: Holt, Rinehart & Winston.
Gagne, R., & Gibson, J. (1947). Research on the recognition of aircraft. In: J. Gibson (Ed.)
Motion picture training and research. Washington, DC: U.S. Government Printing Office.
Garcia, S, & Turner, R. (2006). CMMI(R) survival guide: Just enough process improvement.
Upper Salle River, NJ: Addison-Wesley.
Gibson, E.J. (1988). Exploratory behavior in the development of perceiving, acting, and the
acquiring of knowledge. Annual Review of Psychology, 39, Pages 1-42.
Gibson, J.J. (1977). The theory of affordances. In R. Shaw & J. Bransford (eds.), Perceiving,
acting and knowing: toward an ecological psychology. Hillsdale, NJ: Erlbaum.
Gibson, J.J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin.
Haskins, C. (2009, March). Development of outcomes based training. Presentation at the US
Army Asymmetric Warfare Group workshop on Outcomes-based Training and Education,
Applied Physics Laboratory, Johns Hopkins University, Laurel MD.
Headquarters Department of the Army (2008). Operations. Field Manual No. 3-0. Washington,
DC: Headquarters Department of the Army.
Headquarters Department of the Army (2002). Training the Force. (Revised 2008, Training for
Full Spectrum Operations). Field Manual No. 7-0. Washington, DC: Headquarters
Department of the Army.
Headquarters Department of the Army (2003). Battle focused training. Field Manual No. 7-1.
Washington, DC: Headquarters Department of the Army.
Jackson, C., Woods, H., Durkee, K., OMalley, T., Diedrich, F., Aten, T., Lawrence, D., & Ayers,
J. (2008). Tools for assessment of operator contribution to system performance. Proceedings
of the Undersea Human Systems Integration Symposium, Bremerton, WA.
Kilcullen, D. (2006). Twenty-eight articles: Fundamentals of company level counterinsurgency.
Retrieved August, 2006, from the Joint Information Operations Center,
http://www.au.af.mil/info-ops/iosphere/iosphere_summer06_kilcullen.pdf.
Kirkpatrick, D., (1994). Evaluating training programs. San Francisco, CA: Berrett-Koehler
Publishers, Inc.
Lerner, J., & Keltner, D., (2001). Fear, anger, and risk. Journal of Personality and Social
Psychology, 81(1), 146-159.
Lerner, J., & Tiedens, L. (2006). Portrait of the angry decision maker: How appraisal tendencies
shape angers influence on cognition. Journal of Behavioral Decision Making (Special Issue
on Emotion and Decision Making), 19, 115-137.
Lukey, B. & Tepe, V. (Eds.) (2008). Biobehavioral resilience to stress. Boca Raton, FL: CRC
Press.
MacMillan, J., Entin, E. B., Morley, R., & Bennett, W. (in press). Measuring team performance in
complex dynamic environments: The SPOTLITE method. Military Psychology.
Marken, W. (2008). Merging doctrine: Outcomes-based training and mission essential task lists
compared. Initial Entry Training Journal, 2, 1-3.
19
Riccio et al.
McNamara, P. (2008). T.A.P.S. Tactical application of practical shooting: Recognize the void in
your tactical training. New York: iUniverse.
Meigs, M. C. (2003). Unorthodox thoughts about asymmetric warfare. Parameters, 33(2), 4-18.
Mutafelija, B., & Stromberg, H. (2003). Systematic process improvement using ISO 9001:2000
and CMMI. Boston, MA: Artech House.
Perry, R. & McEnery, K. (2009). Army reconnaissance course: Defining the aim point for
reconnaissance leader training. Armor, July-August, 14-20.
Petraeus, D.H. (2006). Learning counterinsurgency: Observations from Soldiering in Iraq.
Military Review, Jan-Feb, 2-12.
Reed, E. (1996). The necessity of experience. New Haven: Yale University.
Riccio, G. (1993). Information in movement variability about the qualitative dynamics of posture
and orientation. In: K. Newell (Ed.). Variability and Motor Control (pp. 317-357),
Champaign, IL: Human Kinetics.
Sage, A.P., & Rouse, W.B. (Eds.) (1999). Handbook of systems engineering and management.
New York, NY: Wiley.
Savory, J.R. & Duffy, T.M. (1995). Problem based learning: An instructional model and its
constructivist framework Educational Technology, 35, 31-38.
Schaeffer, M. (2005). DoD and CMMI. 5th Annual CMMI Technology Conference and User
Group. Denver, CO 14-17 November 2005.
http://www.dtic.mil/ndia/2005cmmi/tuesday/schaeffer.pdf
Schwitters, J. (2009, March). Command imperative for change. Presentation at the US Army
Asymmetric Warfare Group workshop on Outcomes-based Training and Education, Applied
Physics Laboratory, Johns Hopkins University, Laurel, MD.
Sellers, T.J. (2008). Basic officer leader course: The so what in junior officer education today.
Infantry, September-October, 2008, 6-8.
Shaw, R. & Cortes, M. (2009, March). Necessity for change to defeat asymmetric threats and
prepare for full-spectrum operations. Presentation at the US Army Asymmetric Warfare
Group workshop on Outcomes-based Training and Education, Applied Physics Laboratory,
Johns Hopkins University, Laurel, MD.
Swain, R. (2005). Changes in instructional system design (ISD): Improving training product
delivery to United States Army Soldiers. USAWC Strategic Research Project. Carlisle
Barracks, PA: U.S. Army War College.
Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and
quantitative approaches. Thousand Oaks, CA: Sage.
Tice, J. (2008). Soldier training is in for a big overhaul. Army Times, April 8, 2008.
Tobias, S. & Duffy, T. (eds.) (2009). Constructivist instruction: Success or failure. New York:
Routledge:
United States Army Training and Doctrine Command (1999). Systems approach to training
management, processes, and products. TRADOC Regulation 350-70 (under revision). Fort
Monroe, VA: Training and Doctrine Command.
Vandergriff, D. (2006). Adaptive leaders course, Part 1: Old dogs teaching new tricks. Army,
November 2006, 59-66.
Vandergriff, D. (2007). From swift to Swiss: Tactical decision games and their place in military
education and performance improvement. Performance Improvement, 45(2), 30-39.
Weick, K.E. (1995). Sensemaking in organizations. London: Sage.
Wong, L. (2002). Stifled innovation: Developing tomorrows leaders today. Carlisle, PA: U.S.
Army War College.
Development of Instructor Measures
20
Chapter 2. Formative Measures for Instructors

Ryan Marceau, Cullen Jackson, Courtney Dean, Fred Diedrich, Sharnnia Artis, Emily Wiese
Aptima, Inc.
Gary E. Riccio
2.1 Development of Formative Measures
2.1.1 The COMPASS Methodology
In order to assess training effectiveness and provide developmental feedback, it is important to
assess instructor performance in relation to the knowledge, skills, and attitudes that are being
developed. A bottom-up process for developing performance measures grounds the assessment in
behavior illustrated by experts and individuals who have attained proficiency in a particular job
(MacMillan, Garrity, and Wiese, 2005). This helps ensure that behavior-based measures are
meaningful and pertain to constructs for which data can be collected reliably.
Our methodology for measure development combines subject matter expertise with established
psychometric practices to produce measures of observable behavior. This methodology, originally
developed by Aptima with the Air Force Research Laboratory, is referred to as COmpetencybased Measures for Performance ASsessment Systems (COMPASSSM) (MacMillan, Entin,
Morley, & Bennett, in press). The COMPASS methodology was used to operationalize the
principles and practices of OBTE into observable behavior that can be measured in evaluation of
a training program and that can provide formative feedback to instructors.
The COMPASS methodology employs an iterative series of three workshops with subject-matter
experts to develop and validate observation-based performance measures (for complete details,
see MacMillan et al., in press). In the first workshop, working as a group with a facilitator,
subject-matter experts identify behavior upon which the performance measures are based. These
performance indicators refer to observable behavior that allows an individual to rate the quality of
individual or team performance. For this workshop, the objective is to identify behavior that can
be observed rather than inferred. The performance indicators allow identification, at a high level,
of behavior for which it is most important to develop specific measures. In addition, a workflow
or a series of tasks typically is addressed to provide context for the performance indicators and
associated measures. This context helps identify behavior that is diagnostic or critical to quality
performance.
While a few performance indicators are readily translated into performance measures, more
detailed information is generally needed to create behaviorally anchored performance measures
that coincide with the performance indicators from the first workshop. For the anchors, it is useful
to consider specific behavior that is related to effective and ineffective performance for each
performance indicator. Therefore, the second COMPASS Workshop consists of a series of oneon-one interviews with subject matter experts to identify explicit behavior that illustrates
superior, average, or poor performance for each of the performance indicators. Conducting this
workshop with individual experts allows for documentation of multiple opinions and, hence,
constant comparison and a more thorough examination of the various assumptions and
considerations for the behaviorally anchored measures.
21
Marceau et al.
The goal of the third workshop is a detailed review and modification of a set of draft performance
measures. The development of an observer-based rating instrument involves concerns regarding
observability (i.e., Will there be an opportunity to observe this behavior?), rating scale (i.e., How
much variability in behavior will be observed?), and wording of the behavioral anchors to reduce
ambiguity and promote inter-rater agreement. In order for performance measures to be
informative, the third workshop also includes a review of relevance and a confirmation that the
performance measures capture the behavior described in the performance indicators derived in
first workshop. Based on the results of the third workshop, the set of performance measures is
further refined, subsequently reviewed, and tested for feasibility of data collection.
2.1.2 Development of Measures for OBTE
The three COMPASS workshops were conducted to develop performance measures for assessing
the extent to which OBTE principles and practices are represented in instructor behavior.
Accordingly, the first COMPASS workshop was conducted in a group setting with the
progenitors of OBTE from the Asymmetric Warfare Group (AWG). These subject matter experts
developed the Combat Applications Training Course (CATC) that the AWG offers to
stakeholders in Army training and education who are interested in OBTE. This train-the-trainer
course presents OBTE in the context of marksmanship training. CATC was used as a point of
departure in the first workshop for most of the discussion about specific behavior and broader
instructional strategies that exemplify OBTE. The performance indicators developed in this
workshop were categories of behavior that reflect OBTE.
The second COMPASS workshop consisted of individual interviews with a selection of the
subject matter experts who were instructors for CATC. The interviews focused on identifying
examples of instructor behavior that would illustrate the novice, average, and expert level of
performance for each performance indicator. The topic of conversation focused on what
instructors should do to be consistent with OBTE. It addressed behavior indicative of effective
performance and behavior indicative of ineffective performance. The information gathered in this
workshop informed the creation of three behavioral anchors to develop a given performance
indicator into a behaviorally anchored measures for instructor performance. Anchors were
developed for most of the performance indicatorsall of the ones for which a rating scale would
be appropriate.
The third and final COMPASS workshop presented the participants from workshop with the final
list of behaviorally anchored measures as they relate to each performance indicator. Performance
measures were reviewed to ensure that the items were seen as meaningful, relevant, and
observable. Subject matter experts reviewed the measures for the performance indicators with
respect to the following criteria: relevance, observability, measurement type (e.g., scale, yes/no,
checkboxes), measure wording, scale type, and scale wording. Consequently, the third workshop
resulted in a complete set of observation-based performance measures for assessing instructors
with respect to OBTE.
2.2 Description of Formative Measures
2.2.1 Results of the COMPASS Process
The product of the COMPASS effort was a set of 65 measures that reflect the principles and
practices of OBTE (see Appendix A for the complete set). Many of the measures also possess
behaviorally anchored rating scales that describe observable instructor behavior at the novice,
average, and expert levels. These measures operationalize OBTE in terms of a formative
22
assessment, that is, assessments that are actionable in the continuing improvement of training and
education. The COMPASS process yielded a set of measures that have meaning within a
community of practice. Disciplined use of the measures provides the opportunity for the user to
become increasingly oriented to the values and best practices of the community. This provides
additional meaning and validity to the endeavors of anyone who uses the measures for formative
feedback. It thus is an important source of confidence for instructors. Given that the measures
refer to observable behavior, they also tend to ground confidence in a sense of competence; that
is, sources of confidence are verifiable by oneself and others.
These formative measures for instructors are intended to be general. For some instructor
measures, a particular focus such as rifle marksmanship training helped reduced ambiguity. In
such cases, it should be relatively straightforward to translate the particulars into those of another
skill or knowledge domain. We believe the measures can be employed to assess instructor
behavior in any domain to provide feedback with respect to the principles and practices of OBTE.
Many of measures developed for planning and execution of instruction are described below. They
are grouped under high-level headings that were instrumental in the development of the measures.
These headings were intermediate products of the initial COMPASS workshop. They resulted
from initial discussions for particular performance measures that eventually became focused and
refined to a level of detail sufficient for identification of specific behavior that exemplifies
OBTE. Figure 1 shows an example of a behaviorally anchored performance measure (Does the
instructor articulate the why to the Soldiers?) in the context of its nested superordinate
headings (in gray). The twelve highest-level headings are used below to group and describe the
measures. In subsequent development of a grounded theory for OBTE, the measures have been
regrouped into practical categories that facilitate their selection and use with respect to broad
outcomes around which OBTE is organized (see Chapter 3 and Epilogue).
4 Communicate the parameters of learning ("why are we out here" / "what, why, and how")
4.1 Communicate the "right" problem (i.e. what is the real problem they're trying to solve)
4.1.1 Combat or mission applications vs. meeting the minimum standard
1. Does the instructor articulate the why to the Soldiers?
Relies only on tasks,

conditions and
standards; focus is on
completing the event
c
c
N/A
N/O
Explains the why but not

in the context of mission
success/problem solving;
states solution in the
context of the problem
Lays the foundation of

why at the beginning of
training; states the
problem, then guides the
Soldiers to discovery of
the tactical relevance
Comments/Notes:
Figure 1. Sample Performance Measure. N/A refers to not applicable and N/O refers to not
observed.
23
Marceau et al.
2.2.2 Elaboration on the Description of Measures

In the description of measures below, we utilized two techniques that are not part of the
COMPASS methodology but that help in utilizing the products of COMPASS. In particular, the
description of each group of measures includes an instructional vignette that characterizes the
thinking and purposeful behavior of notional instructors in planning and executing a learning
event. The vignettes are based on first-hand observations of instructor behavior in CATC and our
discussions with the instructors. Thus they generally refer to marksmanship training. It is
important to reemphasize, however, that the measures are not limited to marksmanship training.
They can and should be adapted for other skills and knowledge sets (see e.g., Chapters 12 and
13). The notions about the thinking and purpose of instructors were influenced by discussions
with CATC instructors and the progenitors of OBTE (see Chapter 3).
For each group of measures described below, we elaborate on the practical and scientific
implications of OBTE based on our interactions with subject matter experts in OBTE and given
our theoretical dispositions and experience as scientists and educators. The descriptions reflect
our constant comparison between themes from our discussions with experts over many months
and well-established lines of thought in relevant scientific disciplines (Camic, Rhodes, &
Yardley, 2003; Denzin & Lincoln, 2003). The citations make the comparisons explicit; they point
from our representations of OBTE to the relevant literature as opposed to being representations of
the literature. The citations represent interrelated bodies of workmost of which evolved over
many decadesthat can provide anchors and verifiable foundations for a grounded theory of
OBTE. In Chapters 4 and 5, we took greater license in exploring the reciprocal relationships
between our inquiry into OBTE and the relevant literature. Together, the connections described in
Chapters 2 through 5 reveal opportunities for continuing improvement of the measures and the
associated practices of OBTE. This is important given the raison dtre for the formative
measures and the empirical investigations that utilize them (see Chapter 1).
As in our development of a grounded theory for OBTE, identifying implications and personal
meaning in the measures is an important part of how they should be used. To be consistent with
OBTE, particular measures should not be imposed on the user, and no particular measure should
be considered mandatory. OBTE is not prescriptive; the measures and associated vignettes are not
a script. The user should take ownership of the continuous quality improvement that the measures
enable and adapt them as appropriate for the situation at hand.
2.3 OBTE Performance Measures: Planning for Training
2.3.1 Define Outcomes
Defining training outcomes is the first and most critical step in building any program of
instruction. OBTE emphasizes that instructional system design should reflect and respond to the
most urgent needs of Full Spectrum Operations. OBTE focuses on developing Soldiers over time
by exposing them to experiences designed with respect to attributes such as confidence,
accountability, and initiative as well as associated capabilities such as awareness, discipline,
judgment, and deliberate thought. OBTE does not emphasize meeting standards alone; it does not
dismiss them either. Rather, it seeks to shape instruction to promote continuing development of
Soldiers who are agile overall and proficient in particular skills under unpredictable and changing
conditions (Cornell-dEchert, 2009b; see also, Chapter 1).
An outcome is different from a learning objective but ultimately should be integrated with
learning objectives associated with a near-term sequence of learning events (see Cornell-dEchert,
24
2009a). Outcomes can be viewed as a broader purpose for training or education. They should
relate to execution of military missions and development of individuals to ensure mission success
(Cornell-dEchert, 2009b). As design goals for learning objectives, longer-term outcomes can
provide a framework for linking of learning events in curriculum level design. If a learning
objective specifies the end state of a learning event, outcomes provide guidance and constraints
on the means to the end. An important implication is that, in principle, learning objectives and
developmental outcomes can be convergent or divergent. Without explicit consideration of
development outcomes that are always being influenced, one way or another in a learning event, a
narrow focus on efficiency and near-term objectives can lead to an unacceptable level of risk of
divergence and intertemporal interference between objectives and outcomes (cf., B. Glaser,
2002).
Designers who understand the developmental role they have with cadre will
refrain from narrowly defining every aspect of training or education and instead
focus on principles. The science of the Designers craft must support the art of
the instructors craft; it must be vigilant in never suffocating the art of
instruction. Designers who desire to create a learning environment for students
will first seek to create a learning environment for cadre. Designers who
recognize the importance of an adaptive and thinking Army will first seek to
create an outline of instruction that fosters adaptation and thinking within the
cadre. [M. Darwin, personal communication, November, 2008]
Example Measures
1. Is the training designed to emphasize the importance of combat applications?
Training focuses on
tasks/events; goal is to
pass training (e.g.
qualify)
Combat applications are

described, but training
focuses on tasks/event
Tasks/Events resemble
combat application and
mission success
2. Do the instructors incorporate development of intangible attributes (Judgment,

Adaptability, Accountability, Problem Solving, Confidence, Initiative, Awareness,
Thinking Skills) into their vision for achieving the Commanders intent?
Vision is focused on
apparently efficient
and correct procedural
accomplishment of the
Vision focuses on correct

Soldier performance of
the task but not
development of the
Vision focuses on
effective development
of the individual and
correct Soldier
25
event/TSP
Marceau et al.
individual
performance of the task
Example Vignette:
An instructors training objective for a certain day is to practice grouping. An expert instructor
might try to schedule a 25-meter range for this day so that the Soldiers can easily walk down
range, see their groups and attempt to self-diagnose before a Drill instructor helps them. An
immediate result is Soldiers who have a better appreciation of factors affecting their shot groups.
More importantly, progress is made toward longer-term developmental outcomes in that Soldiers
come to understand the value of various forms of feedback in learning. A novice instructor, with a
focus on throughput, might plan only to walk down range and then make the changes to the rifle
themselves. The outcome for the novice instructor, whether intended or not, are Soldiers who
believe that practice without feedback is a reasonable approach to learning and who have little or
no awareness that bad habits can be developed as easily as good habits. Expert instructors
understand that there is no such thing as an event in which no learning occurs, they understand
that learning is either good or bad, they understand that time constraints and priorities such as
throughput can lead to instructional shortcuts that have unintended consequences and
counterproductive outcomes. During planning, expert instructors consider these critical influences
on the quality of instruction.
Outcomes and training events should always focus on learning that is relevant to unpredictable
environments rather than the accomplishment of a scheduled event in more or less expedient
instructional settings. The novice instructional behavior in the vignette above might still result in
the Soldiers meeting the grouping standard but very little Soldier development would occur. The
expert instructor allows the Soldiers to view and evaluate the effect they have on their shot
groups. This increases understanding and encourages deliberate thought on behalf of the Soldier.
Though the Soldier may not be able to accurately diagnose the problem at first, guidance from the
instructor will help them to discover the solution to their problem and lead to increased Soldier
confidence and accountability.
2.3.2 Create a Positive Learning Environment
In OBTE, instructors utilize and exemplify leadership. Arguably, the terms instructor and
leader are synonymous in this approach to learning and development. Every instructor is a
facilitator, advisor, and mentor. This means that some mistakes will be allowed, that students will
learn lessons from mistakes instead of being derogated for them, and the instructors role will be
to guide students toward discovering a solution. Time should be built into the schedule for such
guidance and for students to engage in the self-discovery and collaborative reflection that are
especially important when learning events are not completely scripted and controlled.
Instructors should plan to frequently assess how the training is progressing. This assessment does
not have to take any time away from training; it can be as simple as lagging behind the Soldiers
as everyone walks up range. During this short time, instructors can assess if they are achieving
their intent for the day and make changes as necessary. This way, the instructor can adapt the
training to the evolving needs of the students without sacrificing progress toward long-term
developmental objectives.
26

Example Measures
1. Do the instructors plan to scale down from an authoritarian approach to a mentoring
leadership style as appropriate for training events?
Yes
No
2. Do the Drills design the training events to gradually increase in difficulty?
Unaware of stress
management; Plans to
increase difficulty of
training according to a
set schedule/process
Plans to introduce stress,

but not at the appropriate
level or time
Balances stress and

difficulty to the capability
of the Soldiers
Example Vignette:
Drill instructors are planning out how they will have the Soldiers practice shooting from different
positions. Due to the high Soldier to instructor ratio, there will not be enough instructors to ensure
that all the Soldiers are building proper positions. A novice instructor would likely plan to first
demonstrate the positions to all the Soldiers and then spread instructors evenly across the lanes
and run all the Soldiers through in cycles. The instructors would monitor multiple lanes to ensure
safety and provide assistance when necessary. An expert, in this case, might plan to first split the
Soldiers into small groups, one for each instructor. The instructors will facilitate discussions
among their groups on the how to properly build a stable position, guiding the Soldier towards an
understanding of the fundamentals of a stable position.
The experts decision to break the Soldiers into groups will allow instructors to facilitate the
individual development of the Soldiers. By ensuring that each Soldier becomes acquainted with
the fundamentals of how postural configuration and stability relate to marksmanship, they are
increasing the likelihood that Soldier will be successful in shooting not only from the particular
positions experienced but from other positions as well. As a student of combat-relevant
marksmanship, the Soldier will develop perceptual and motor skills of stabilization in addition to
appreciating the implications of stability (cf., Riccio & Stoffregen, 1988; Riccio, 1993). This will
increase a Soldiers confidence in situations where factors of enemy or terrain, for example,
require nonoptimal or unusual postures. Though the initial small group exercises may delay the
start of the shooting portion of the training, the increased understanding and proficiency will
increase the rate of learning in novel or more difficult situations.
The balance between task difficulty and student capability is critical to student development.
Increasing difficulty or applying stress too quickly will lead to failure and be detrimental to the
students confidence. Going too slowly will result in inefficient training and students becoming
disengaged. A proper balance will result in efficient and effective training that also builds
27
Marceau et al.
students confidence as they learn to learn and perform under nonoptimal conditions. Instructors
should employ a building block approach that increases complexity, for example, as students
learn and adapt new skills (see, e.g., Merrienboer & Kirshner, 2007; Tobias & Duffy, 2009). Such
scaffolding can be planned in terms of starting simpler as well as ending more complex.
2.3.3 Create the Parameters of Learning
It cannot be assumed that the students will understand why a training exercise is relevant to their
mission. It is important to design exercises that aid the student in making these connections.
Understanding the why behind training will contextualize students learning and presumably
will lead to better transfer to the operational environment. In OBTE, instructors plan for
opportunities to provide explanations or otherwise to reveal the broader context of a learning
event.
While large blocks of unused time (e.g., hours or days) are generally nonexistent in a training
program, there is a multitude of brief moments (e.g., seconds or minutes) that are wasted in
almost any instructional situation. Much can be communicated during the frequent interstitial
moments between telling and doing. Value can be added in both the content of communication
(e.g., explanation) and the occurrence of communication (e.g., modeling). Instructors should
prepare for such opportunities by having teaching points that can be expressed concisely at
moments that, while not precisely predictable, are reasonably likely to occur. There are
opportunities for education during training.
The inextricable linkage between training and education, in practice, calls for different
approaches in design and development. While training produces a change in behavior, education
produces a change in thinking. Training Soldiers to be agile requires consideration of cognitive
skills even when the focus of a learning event is on fundamental perceptual and motor skills. In
fact, cognition is separable from perception and action only under the most artificial and
contrived situations (cf., Neisser, 1976; Shaw & Bransford, 1977; Winograd, Fivush, & Hirst,
1999). The linkages can be so basic that little or no explanation may be necessary for students to
come to a better understanding of them. The right kind of experience may be sufficient.
Example Measures
1. Do the instructors plan to discuss the tactical relevance of the task with the Soldiers?
Yes
No
2. [Follow-up] As part of the discussion, will the instructors ask the Soldiers to describe the
tactical relevance of the events?
Only plans to lecture to

Soldiers on the task; no
context
Plans to engage the Soldiers

on why the event is
tactically relevant, but
Plans to state the problem,

then guide the Soldiers to
discovery of the tactical
relevance (i.e. problem
solving exercise)
28
Example Vignette:
Soldiers in Army basic training sometimes use sandbags to support their rifles in the pronesupported position. However, it is unlikely that there will be sandbags laid out for them in theater.
The skill that is being taught is how to establish a solid, supported position, not how to shoot
using a sandbag. Expert instructors would plan to ask the students to think about or discuss ways
they could create a stable prone-supported position in theater. What would they use for support?
How would this change their position? Why is this important in combat? This discussion could
happen during the down time in between shooting cycles while Soldiers are rotating or waiting
their turn to shoot.
Creating the parameters of learning helps explain the why the experience and skills acquired in a
learning event relate to a mission application of a current task. To the extent possible, the learning
environment should have operational relevance and realism. Situation-specific resource and
safety constraints often make it difficult for the training environment to look like the field or
operational environment. The tendency in Army training and education is to think that, absent
obvious realism, operational relevance can be addressed through vignettes or other supplementary
materials that tell the student about a mission and set a conceptual context for instruction. To be
sure, such a conceptual context can be useful to the student but, by itself, it has many of the
limitations of telling without doing. The connections are not likely to be understood deeply or
retained by the student.
A learning environment need not appear superficially like an operational environment to have
operational relevance or, more specifically, to develop skills that transfer to an operational
environment (cf., Warren & Riccio, 1985). A learning environment can have fidelity with respect
to perception and action if it replicates constraints on what an individual can achieve and how.
Generally this requires a context that is sufficiently rich to include observable cause-effect
relationships such as linkage among tasks and concurrent or downstream consequences of ones
actions (cf., J. Gibson, 1977, 1979). In the context of marksmanship, for example, the relationship
between constraints imposed by ones momentary biomechanics, the environment, and
perceptual-motor tasks can be manipulated and revealed in operationally relevant ways without
the physical realities of combat. Training can be designed to give individuals experience,
sometimes in unusual settings, that sensitizes them to the interrelationships between postural
control, stabilizing and destabilizing elements of the environment, and visibility in general (cf.,
Riccio, McDonald, & Bloomberg, 1999). The broader implication is that basic research in the
behavioral sciences can inform the design of a training environment (cf., Gagne & Gibson, 1945;
Riccio, 1995; Riccio & McDonald, 1998).
2.4 OBTE Performance Indicators: Training Execution
2.4.1 Communicate the Parameters of Learning
As indicated in section 2.3.3, OBTE instruction should seek to lay the foundation of the why at
the beginning of training. Students should be exposed to the problem and guided to discovering
the tactical relevance of a particular training activity. OBTE instruction emphasizes how a
particular activity applies to the overall mission or desired outcome.
29
Marceau et al.
Example Measures
1. Does the instructor articulate the why to the Soldiers?
Relies only on tasks

conditions and


training; states the
Example Vignette:
Soldiers typically arrive at the range early in the morning following chow. They get into
formation and wait for instruction from the Drill instructor. Often, the Soldiers do not know or
have very little information regarding training content for that day until they are told by a Drill
instructor or the tower to complete a task. Even at this point, they still do not know the reason or
combat relevance of the task, just that they have to complete it. An expert in OBTE would start
the day by bringing everyone in and discussing the goals for that day. The expert does not simply
recite passages from the Training Support Package for the day but presents the goals in the
context of a combat related problem that the Soldiers must overcome in order to achieve mission
success. The instructor takes opportunities to reveal, not simply to lecture about, the critical
interrelationships among barricades, defilade postures, muzzle awareness, location of other unit
members, and characteristics of enemies and noncombatants in terms of the capabilities and
requirements for postural stability and visibility of targets in a task that otherwise is ostensibly
only about marksmanship.
The vignette above describes one way in which an instructor can increase learning by providing
context around a training task. The impact of training presumably is increased to the extent that
Soldiers are sensitized to contextual relationships not limited to the task at hand (cf., J. Gibson &
E. Gibson, 1955; Bransford & Schwartz, 1999). In this case, the context is not just the learning
objectives for the event, but also their relationship to lethality and survivability in combat and
perhaps even to strategic implications of small unit action in the context of FSO. By discussing
why particular activities and props are included in a learning event, the instructor can educate the
attention (cf., J. Gibson, 1979) of the student and help reduce workload due to an unnecessarily
confusing or inexplicably cluttered setting. Through this deeper understanding, Soldiers become
better equipped to apply the same concepts towards solving problems in different situations they
may encounter in the operational environment.
2.4.2 Training Emphasizes Broad Combat or Mission Success
Training should emphasize problem solving irrespective of environment or resources. Many
training environments do not accurately reflect all aspects of the operational environment. This
need not be the case for operational relevance, and it may be desirable for learning (see section
2.3.3.2). In these situations it is important that the instructor ensure the students are making
30

connections in the training task that are similar to the connections on which mission success
depends in an operational environment.
In the case of OBTE, effectiveness wont truly be known until the Soldier performs in combat,
but evidence of learning and development of efficacy (the foundation or capacity for
effectiveness) will manifest itself in ways that allow meaningful assessment of both the Soldier in
training and the trainer in execution. It also provides for explicit or implicit self-assessment and
the development of a sense of self-efficacy (cf., Bandura, 1977, 1997; see also, Ryan & Deci,
1985) in both trainers and their Soldiers (I know when I am performing well and I know that I
can perform well). This is a sense of self-efficacy that is grounded in demonstrable competence
and an appreciation of what ones actions and capabilities for action in a particular environment
afford for good or ill (cf., J. Gibson, 1977; Shaw, Turvey, & Mace, 1982; Turvey, 1992; see also,
Endsley, 1995; Neisser & Jopling, 1997).
Example Measures
1. Does the training emphasize broad combat/mission success?
Train to a specific task

only
Train to examples or
experience only; reflects
specific environments or
resources only
Creates training that

emphasizes problem
solving irrespective of
environment or resources
Example Vignette:
A unit who knew they would be deploying to an urban area in Afghanistan trained extensively in
simulated urban environments. They practiced shooting around buildings, cars and other urban
terrain. They were proficient in room clearing tactics and close quarter marksmanship. Upon their
arrival they were sent to an area of operation that consisted of partial urban and wooded terrain.
The Soldiers did not feel confident in their ability to operate in the wooded terrain because they
had trained in urban terrain only. Consequently, they would be hesitant to follow the enemy into
the woods. Subsequently, in the role of an instructor in a simulated urban environment, one of the
Soldiers made sure that his trainees understood general principles of individual movement,
collective maneuvering, and situation awareness in the context of general characteristics of the
environment such as paths, obstacles, barriers, margins, brinks, footing, partial enclosures,
camouflage, concealment, vistas, and vantage points. Some of this was accomplished by exposing
Soldiers to variation in these constraints amid invariant tasks involving individual perception and
action as well as collective coordination. Soldiers also were asked to consider contingencies in
the way various characteristics of the environment could be utilized.
What the Soldiers described in the initial part of the vignette did not realize was that all the
fundamentals upon which the urban tactics were built could be applied to the wooded terrain. In
contrast, being exposed to variation in the environment and in behavior with respect to it can help
them avoid being distracted by the concrete facts of buildings and roads. They can differentiate
31
Marceau et al.
the essential from the incidental; they can see various aspects of environment in a different way
(cf., J. Gibson & E. Gibson, 1955; E. Gibson, 1991). The implication is that learners can free
themselves from old habits and assumptions carried from different occupations and purposes.
They can come to understand the essential interdependence of task and the way the environment
is perceived (cf., J. Gibson, 1979; Riccio & Stoffregen, 1988). They can become better prepared
for a new occupation and a new purpose in an otherwise familiar environment. Thus they become
more adaptable to novel tasks and conditions. The instructor can facilitate this learning through
judicious and well-timed guidance (cf., Bransford & Schwartz, 1999; Tobias & Duffy, 2009).
This is not to imply that the Soldier understands these things in an abstract way. The implication
is that he simply knows what to do and is not frozen in his tracks by superficial novelty.
2.4.3 Customize Instruction When Possible Based on Constraints/Conditions
OBTE is learner-centered (cf., Bransford, Brown & Cocking, 2000; see Chapter 3). Instructors
should try to balance training difficulty with the students momentary capability to learn. It is
important to recognize the point of diminishing return (i.e. student is exhausted or burnt out) and
adapt the training accordingly. People learn differently and at varying rates. Instructors should
coach individual students or address the individual training predicaments of students who may
have unique needs. This should not be taken to imply that instructors must provide every student
with individual attention all the time or even provide this to most students most of the time.
Sometimes a little individual attention is all a student needs to get back in the game. Other times,
there might be an individual difference that is unnecessarily hindering a students ability to learn
(e.g. a small Soldier with a large rifle). Even if the instructor cannot always provide guidance to
all the students that could benefit from it, it is better to help some students that none of them.
Moreover, the visible act of providing guidance to a student provides a model even for
bystanders. Instructors can be influential as role models for students who are not the object of the
instructors immediate attention (cf., Lave & Wenger, 1991).
In OBTE, it is desirable to provide students with the right amount guidance at the right time. An
implication of this principle is that it is important to know when not to intervene or to instruct
explicitly. Encouraging students to take ownership of their own learning can have valuable effects
on student motivation and engagement in learning (cf., Guay, Ratelle, & Chanal, 2008).
Instructors are not passive during these opportunities for student initiative. They are vigilant in
monitoring the situation and ready to show signs of approval when students demonstrate
discipline and awareness in exercising such initiative. Timely approval shows that instructor was
willing to trust the student and not merely inattentive or uncaring. It reinforces accountability.
Example Measures
1. Does the instructor adapt the training to the audience/environment?
Sticks to set schedule;

unaware of diminished
learning (e.g. ignores
indicators of
exhaustion)
Reactively balances
training difficulty to the
capability of the Soldiers
(e.g. reacts after
performance has
diminished significantly
Proactively balances
training difficulty to the
capability of the Soldiers;
recognizes point of
diminishing return
32
2. Do the instructors successfully address individual training predicaments?
Doesnt recognize
individual problem
Recognizes individual
problem, doesnt help the
Soldier find a suitable
solution
Identifies the issue and

helps the Soldier find a
work around (e.g. shorter
weapon for smaller
person)
Example Vignette:
A Soldier is struggling during a grouping exercise; the Soldiers groups continue to stay large
despite direction from the drill instructor on proper breathing, positioning, and trigger squeezing.
A novice instructor may grow frustrated in this situation and insists that the Soldier is not
listening to instructions. An expert, whose focus is on Soldier development, would look deeper to
examine if something else is going on. Upon further inspection the instructor noticed that this
Soldiers earplugs do not fit correctly. The Soldier is wearing them correctly, but they are not
sealing the ear properly causing to Soldier to flinch every time a shot goes off. The instructor
finds a different style of earplug for the Soldier and the groups start to become remarkably
tighter.
In this situation, an OBTE instructor recognized a Soldiers individual difference that was
preventing him from being successful. The instructor was able to then identify a solution to the
students problem (changed earplugs). Doing so then allowed the student to master the
fundamentals and become more confident in his or her ability as a rifleman. Had this gone
unnoticed, that student might think that he simply was not skilled enough to shoot a solid shot
group, leading to low confidence as a rifleman. Additionally, the instructor acted as a mentor in
this case by demonstrating effective problem solving skills in addressing the Soldiers individual
training predicament.
2.4.4 Facilitates Learning of Concepts
OBTE promises to develop deep understanding and habits of learning that help Soldiers
recognize and exploit similarities between their prior experiences and novel situations. This
competency generally will result in the perception of what a unique situation affords for action
and in the associated readiness for expeditious action that is efficacious and that reveals
additional information that reduces ambiguity (cf., E. Gibson, 1988; J. Gibson, 1979). Such
development of perception and action enables Soldiers and their units to be agile and effective in
Full Spectrum Operations.
Instructors structure the problem and coach the student towards the solution. Not only does this
approach to training increase learning by leading the student to discover the solution, it also
encourages students to take ownership of their own learning. From the perspective of a mentor
and facilitator, the focus of the instructors attention when students exercise initiative is on
whether the students response is adaptive in the sense of being task directed and oriented (i.e.,
33
Marceau et al.
efficacious in principle), not whether it is momentarily effective. Such efficacy is the difference
between adaptability and mere flexibility. Flexibility is better than inflexibility but not much.
Efficacy in general eventually leads to effectiveness in particular situations but not in every
instance. Mistakes will be made. The role of instructors is to ensure that the consequences of
mistakes are bounded (Cornell-dEchert, 2009b). It is to identify a students zone of proximal
development (see Cole, John-Steiner, Scribner, & Souberman, 1978), the margin between their
momentary range of efficacy and their potential range of efficacy. This is the zone within which
students are challenged without being frustrated and within which there is the greatest capacity
for semi-autonomous learning without deleterious consequences.
Example Measures
1. Does the instructor guide Soldiers to self-discovery of how to achieve desired results?
Dictates a specific path

to the solution to a task
Structures the problem; but

still dictates a specific path
to the solution
Structures the problem

and guides Soldier to
efficient self discovery of
solution path
2. Does the instructor use safety as a training enabler?
Focuses on SOP and is

regimented; safety is
disconnected from its
real purpose
Explains safety in the

context of accomplishing
the training events, but not
as a combat and training
enabler (i.e. safety is
restrictive)
Explains safety as a
combat and training
enabler (e.g. weapons
awareness allows for more
independent or complex
scenarios)
Example Vignette:
Marksmanship training is heavily structured and regimented in order to maintain safety.
However, Soldiers do not learning to be safe on their own because the tower is always telling
them exactly what to do. They appear to be afraid of their weapons. A different approach would
be to start off regimented and pull back after Soldiers demonstrate safe behavior. Without
constant direction from the tower, Soldiers can eventually take on the responsibility for being safe
and become more confident in their ability to be safe with a weapon.
34
Transferring the responsibility for safety to the student increases the students awareness and
accountability. The instructor demonstrates trust in the Soldiers ability to be safe.
Soldiers also become more confident in their ability to be safe with their weapons in future
situations when a drill instructor is not present. In Army training, safety and instructional
methodology (i.e., nature, extent, and timing of guidance provided to students) typically are
viewed as the responsibilities of different groups of people if not wholly incommensurate
considerations. The juxtaposition of instructional guidance and safety in OBTE is unusual. As
with most practices in OBTE, this juxtaposition is due to exigencies and realities of the
operational environment, such as the critical interrelationships between lethality and survivability,
especially amid the ambiguities of Full Spectrum Operations. An assumption of OBTE is that,
instead of finding ways to integrate such fundamental capabilities, they should never be separated
in the first place. Accordingly, there should be a persistent coupling between initiative and
accountability from the beginning and at every stage of a Soldiers training, education, and selfdevelopment.
This does mean expanding our own circles of interaction to include, for example,
the range, ammo, and training developers of the Army who rarely see a Soldier
train, but do influence decisions on how, why, and to what level his training will
be supported by the Army Outcomes are in so many ways influenced by the
availability of inputs - resources, including trainers and leaders - and we should
start capturing the inputs required for training more effectively. [K. McEnery,
personal communication, February, 2009]
2.4.5 Creates a positive learning environment
A positive learning environment is foundational in OBTE. A balance of authoritarian and
collaborative styles of instruction is an important part of creating such an environment. One
manifestation of this balance is to avoid being directive about content (what to think) while
being somewhat directive about process (how to think) (HQDA, 2003; cf., Freeman et al.,
2008; Roberto, 2005, 2009). Instructors understand and embrace their role as mentors. They
serve as guides along the Soldiers path of success by providing clear left and right limits, by
noting milestones and decision points, but also by allowing some exploration (cf., Cole, JohnSteiner, Scribner, & Souberman, 1978; Feldbaum, 1965; Filatov & Unbehauen, 2004, E. Gibson,
1988; Henle, 1971; Thelen & Smith, 1994; Safonov, 2001; Tschacher & Dauwalder, 2003). They
facilitate the developmental process by which Soldiers take ownership of their own pursuit of
success. One strategy in this role is for instructors to provide opportunities for Soldiers to think
critically and be problem solvers through training events that require assessment, judgment,
decision-making and execution. As Soldiers try to solve the problem at hand, instructors guide
them through directive questioning and discussion. In the end, the Soldiers solve the problem.
35
Marceau et al.
Example Measures
1. Does the instructor reinforce the importance of problem solving?
Event is scripted;
training is based on
correct performance of
predetermined task
Training provides
opportunities for Soldiers to
be problem solvers but
instructors give the solution
Training provides
opportunities for Soldiers
to be problem solvers;
events require assessment,
judgment, decision
making and execution
2. Does the instructor ask questions and allow/encourage the Soldiers to answer?
Provides answers for

the Soldiers
immediately after
question is asked
Encourages Soldiers to
answer questions, but
provides an answer if one is
not immediately offered
Allows time for Soldiers

to think of an answer and
guides them to appropriate
responses
Example Vignette:
Instructors, during ARM, have set up a range with different stations every five lanes. Soldiers
start at one end and make their way to the other end in pairs, performing different tasks and
exercises at each lane. These exercises include target discrimination, malfunction drills and other
complex tasks. The goal for the day is to develop Soldiers to perform under high stress, combat
like situations. One lane includes a wounded Soldier drill in which the pair of Soldiers has to
react to an injured Soldier, get him out of the line of fire and treat the injury accordingly. Upon
reaching this scenario, two Soldiers are struggling to lift a very large Soldier on stretcher. They
try twice to get him off the ground, but he is simply too heavy. These Soldiers, recognizing that
they have to get the wounded Soldier out of the line of fire, decide to each lift one end of the
stretcher and drag the Soldier out of harms way. Upon seeing this, the drill instructor makes sure
that the Soldiers see that he has noticed their solution while he lets them carry on. Later he
questions the Soldiers with interest about their decision to deviate from standard procedure.
The vignette describes a situation in which the instructor has a choice to intervene or not. He
might have, for example, yelled at the Soldiers, criticizes them for not carrying stretcher the
correct way, and make them start over and do it again. This would have failed to maintain a
positive learning environment. Moreover, overly focused on completing the task correctly
would have punished the Soldiers for not following the prescribed methods for carrying a
wounded Soldier even though what they did was more combat effective given the situation.
Instead, by letting the Soldiers exercise initiative, the instructor allowed them experience
problem-solving under stress. By engaging in collaborative reflection with them after the event,
36
the instructor found out that their goal was to get the wounded Soldier out of harms way. He thus
could acknowledge that they found the most effective way to accomplish that goal given the
constraints of the situation. He would be able to recognize the deliberate thought and sound
judgment of the Soldiers and positively reinforce their behavior.
It is important to understand that, in OBTE, the deliberate thought and judgment required in
mission-relevant problem solving does not necessarily imply cognition that is mentally effortful
or temporally protracted (e.g., on the order of tens of minutes to hours) as we might imagine
based on years of experience with taking tests in learning environments. Deliberate thought is as
much about disciplined awareness, attunement to things that matter in ones surroundings and in
the flow of events, appreciating the consequence of ones behavior for others and vice versa, and
about the attendant purposeful coupling of perception and action. Such awareness can unfold and
inform decisions on time scales as brief as seconds to minutes (cf., Salas & Klein, 2001).
2.4.6 Instructors Utilize Measures of Effectiveness & Self-Evaluation
OBTE is assessment-centered in the sense that assessment is a component of learning not merely
an assessment of learning (cf., Bransford et al., 2000; see Chapter 3). It is important that
instructors are constantly reflecting on the progression of learning. Instructors ensure that
students are getting the training that they need and decide when to advance to the next event or to
revisit a learning objective. OBTE is focused on the student instead of the schedule. Schedules
should not dictate how the training progresses; the students momentary capabilities and potential
should. Instructors should make in-stride adjustments to training as necessary. The informational
basis for these decisions should be explicit to students whenever possible and appropriate. The
point is not to justify changes to the students but to reveal that training is principled, that changes
are event-driven, and that there should be constant vigilance about the conditions in light of
assumptions that may be violated and thus necessitate a change. If this is occasionally made
explicit to students, it will become implicit to them in future decisions of their instructors and
leaders. Student will become more likely to look for conditions that may have motivated a
change; they will become more aware of the situation (cf., Endsley, 1995).
Example Measures
1. Are the instructors discussing the effectiveness of the training?
Instructors
automatically advance
different groups
without considering
training progress
Instructors check on
Soldiers progress but do
not effectively adjust the
training
Check on Soldiers
progress and discusses
when to advance to next
event or when to revisit a
learning objective
37
Marceau et al.
2. Do the instructors use unexpected conditions to provide tactically relevant training

events?
Allow the unexpected

conditions to interfere
with training
Make in-stride adjustments

to continue training but do
not leverage unexpected
conditions as a training
opportunity
Make in-stride
adjustments to incorporate
unexpected conditions
into the training design if
appropriate
Example Vignette:
A schedule calls for several complex ARM exercises to occur on a particular day. However, it is
over 100 degrees out today and the Soldiers are getting burnt out fast. Instructors notice that body
positions and movements of the Soldiers are becoming progressively worse, fundamentally, as the
day goes on. Concentration wanes and shooting accuracy deteriorates. The instructors decide that
the students are reaching the point of diminishing returns. Thus, they make an in-stride
adjustment with the Soldiers by rolling back the complexity of the tasks to ensure the Soldiers do
not fail.
The vignette describes a situation in which the instructors put the emphasis on learning and
development and not on task completion or following the schedule. Though Soldiers will have to
perform under high stress situations in combat, they first need to adequately master the basics
before they can be expected to perform well under more demanding conditions. Had the
instructor decided to continue the training at the higher complexity level, the Soldiers would have
pressed on, but their confidence would have declined as they continued to get poor results and
eventually failed (cf., Merrienboer & Kirshner, 2007). By adjusting the training complexity to the
current ability of the Soldiers to perform, the instructor ensured that the Soldiers confidence in
their ability to perform would remain high. The adjustment would not have been considered if the
instructors were not vigilant about the state and progress of the students.
In OBTE, instructors do not depend unduly on explicit communication with Soldiers to assess
their physical, physiological, cognitive, or emotional states. It can be a useful tool, however, and
one with additional benefits. By occasionally checking with students about their self-perception
and their perception of the situation, instructors reveal to students the importance of situation
assessment in principled approach to training and, more generally, in a principled approach to
decision making in the field (cf., Endsley, 1995). Instructional situations can be dominated by
planned events or facts and still allow students to have opportunities to consider possibilities and
alternatives when assumptions or expectations are violated. These opportunities can be brief or
small but if they are common, they encourage a certain vigilance and anticipation that
automatically engage students more deeply in the situation. Seizing such opportunity doesnt
require instructors or developers to throw out the lesson plan.
38
2.4.7 Uses scenarios to facilitate learning

Scenarios are potentially powerful tools for emphasizing the reasons for a learning event, why the
event is structured the way it is, and what are the essential versus the incidental aspects of the
event with respect to the operational environment. Our use of the term scenario thus is broader
than its connotation in common military usage. In our usage, a scenario is a situation that is
sufficiently rich to reveal the consequences of ones actions beyond the task at hand. If a training
scenario focuses on superficial similarity to its operational analog, it may not be effective and
may even distract students from the key lessons to be learned. There are several contextual
variables that can be manipulated effectively in a scenario-based learning event and that are not
especially dependent on superficial similarity to an operational environment. They include but are
not limited to stressors, linking of basic Soldier tasks, and linking of individual activities with
collective activities (cf., section 2.4.2 above).
Stress can be an excellent facilitator of training through scenarios. Instructors should incorporate
stress (mental and physical) into training in ways that challenge the student but are proportional
to the student capabilities. Stress should not cause the student to fail, but should be challenging.
The goal of stress application is to give the student a sense of accomplishment and confidence in
their ability to perform under adverse conditions. Students should be allowed to discover how
stress affects their performance, and how to mitigate it. Stress can be manipulated easily through
time constraints or by task linkage.
Example Measures
1. Do the instructors group tasks into collective behaviors?
Does not group tasks

into collective
behaviors; results in
incorrect performance
of linked tasks
Does not group tasks into

collective behaviors (no
observable negative
consequences)
Groups task in a way that

simulates the combat
application and reinforces
linked tasks
2. Do the instructors effectively incorporate stress (mental and physical) into training events
to benefit the development of the Soldier?
No stress resulting in
apathy or too much
stress resulting in
chronic failure
Some stress resulting in

some learning; Soldier
unchallenged or overly
challenged
Stress is proportional to
the task and Soldier
capabilities resulting in a
sense of accomplishment
39
Marceau et al.
Example Vignette:
During a training exercise, Soldiers are practicing their fundamentals while shooting from the
prone position. The instructors see that the Soldiers are all doing well so they decide to add stress
by limiting the amount of time the Soldiers have to shoot their ten rounds to one minute. At first
the Soldiers all shoot off their rounds in the first 20 seconds and their shot groups are poor. The
instructors point this out and ask the Soldiers to think about how the time stress affected their
performance. Next cycle, the Soldiers take more time and their shot groups improve. After a
couple cycles the instructors then tell the Soldiers they now have to take ten shots from the prone,
reload their rifle while communicating that they are reloading, and then switch to the kneeling
position and take ten more shots in two minutes total. After a few cycles of this exercise, the
instructors facilitate a discussion with the Soldiers about managing stress in combat.
In the vignette, several simple tasks are gradually combined to create more complex, linked tasks.
Stress is added by incorporating a time constraint. This gradual ramping of stress keeps the
Soldier out of their comfort zone, but not to the point of failure. At the end of the day, the
Soldiers in the vignette will be performing complex, linked tasks that they would have never
thought they were capable of doing at the beginning of the day. More novice instructors may
incorporate the wrong kinds of stress (e.g., verbal intimidation) or may apply too much of one
type of stressor. Consider, for example, an apparently reasonable stressor of having Soldiers run a
mile in full kit or do several strenuous activities prior to participating in a training exercise.
Though this is a kind of stress that is relevant to the operational environment it is applied in a way
that does not allow the Soldier to overcome the effects of stress gradually. Through the gradual
addition of stress and complexity, the Soldiers continuously build up their confidence and become
ready to tackle more complex tasks.
The vignette also shows how the linking of tasks can be relatively simple and straightforward yet
give Soldiers experience with critical interrelationships among basic Soldier skills such as move,
shoot, and communicate. Supplemented with a bit of a backstory provided by the instructor, even
tasks that apparently are purely procedural can be learned in ways that vastly improve a Soldiers
readiness for combat. Consider SPORTS, a procedure for correcting a weapons malfunction.
The mnemonic refers to a process to clear a rifle malfunction by slapping upward on the
magazine, pulling the charging handle back, observing the obstruction, releasing charging handle,
tapping forward assist; and squeezing the trigger. The context in which this procedure would be
performed is not incidental and could potentially have profound implications.
On the range, a weapons malfunction is a benign event. Obviously in a firefight, the situation is
quite different. The speed with which a Soldier can get the weapon back into the fight is a
potentially a life or death matter, and inability to do is not simply a matter of a no-go. Returning
fire is the higher-order task, one that could be accomplished to some degree by others in the unit.
Thus it becomes critically important to communicate to others that the weapons malfunction has
occurred. This could change the priorities of another Soldier or simply just change the direction
of fire or his firing position. At the same time, without an operable weapon, the requirements and
tradeoffs for seeking cover change. The tradeoffs are peculiar to the momentary situation and the
relevant factors of mission, enemy, terrain and weather, troops and support available, time
available, civil considerations. The linkage between move, shoot, communicate, and assessment
of the situation could not be more fundamental and the consequences could not be more
immediate.
40
In a firefight, an outward orientation and the attendant situation awareness is critical but it is
complicated by the distractions of the unusual, if not unique, psychological and physiological
responses to a lethal threat (cf., Kolditz, 2007; Lerner & Keltner, 2001; Lerner & Tiedens, 2006;
Lukey & Tepe, 2008). Habits become important at such times. It is not the time to perseverate on
a mnemonic and its associated procedure if it isnt working. Good habits are necessary. Habits
require frequent repetition. It is never too early to give Soldiers the opportunities to develop good
habits of linking move, shoot, communicate, and situation assessment, and to give them these
opportunities often. A premise of OBTE is that such basic combat skills can be acquired outside
of combat, that they can be acquired in training. These opportunities will be missed if it is
assumed that there are procedures Soldiers can learn in the absence of deliberate thought and
broader awareness.
2.4.8 Instructors exhibit intangible attributes in own actions
OBTE doesnt require that instructors interact with or even be aware of every student all the time.
The instructor is a role model (cf., Bandura, 1977). Instructors should demonstrate behavior and
attitudes their students can model because, more likely than not they will model the instructors
behavior (cf., Lave & Wenger, 1991). It thus is important for the instructor to be aware that his
behavior may be noticed and influential whether or not the behavior was planned or intentional.
One should be aware, for example, that an authoritarian or intimidating style is not necessarily
perceived as flowing from competence. Similarly, evidence that the instructor does not know it
all, or as much as a student knows in some areas, is not necessarily perceived as incompetence.
The expertise with which the instructor should most be concerned is as a leader, mentor, and
facilitator. Their impact is similarly profound whether they accept this responsibility or not, and
whether they are prepared for it or not. The assumption of OBTE is not that instructors should
have a greater impact; it is that they can have a better impact.
Even when there are gaps in an instructors knowledge or experience, there is an opportunity to
demonstrate expertise in learning. The instructor can help a student learn to learn. In the end, this
may be the most important influence an instructor has on a student given that adaptability of
Soldiers in the operational environment implies a capacity to learn. The point here is that students
are always learning to learn in addition to learning what they should learn. We can choose to
teach students to learn one way or another but we cannot pretend that there is not a choice. In
OBTE, initiative and accountability are underpinnings of confidence. Confidence is grounded in
demonstrable competence, and competence is ensured by the capacity to learn. The capacity to
learn is increased by initiative and taking ownership of ones own learning. Responsible initiative
is grounded in accountability and bounded by it. These intangible attributes are not abstract in the
process of learning and teaching. They are observable in the behavior of highly motivated and
engaged individuals.
Example Measures
1. Do the instructors effectively exhibit intangible attributes in their own behaviors as they
conduct their training?
Yes
No
41
Marceau et al.
2. Do the instructors demonstrate openness in changing their training progression?
Training will not

deviate from a set
schedule
Instructors determine when

it is time to move on a new
task
Soldiers have input into

the progression of training
Example Vignette:
A Soldier is shooting well from the prone unsupported position. However, a drill instructor sees
that the Soldier is not resting the magazine on the ground and that doing so would create a more
stable position. The drill instructor tells the Soldier to rest the magazine on the ground even
though this is a controversial practice. After putting the magazine on the ground, the Soldier
begins to shoot poorly and miss his targets. The instructor, realizing that the Soldier is more
effective with the magazine off the ground, tells the Soldier that he can shoot whichever way is
more comfortable and more effective for him. The Soldier then switched back to his original
position and shot more accurately. Later the instructor explained why he was willing to have the
Soldier try a different shooting position.
A novice, in the situation above, might have not been as flexible or confident enough to reverse
his prior guidance to the student (cf., Edmonson, 1999). However, the more expert instructor
recognized that placing the magazine on the ground is only one tactic that can be employed to
create a stable position. He was willing to risk his credibility to show the Soldier that there is no
single right solution in most situations. The instructor created several teaching points through
the guidance he provided and its immediate impact on the Soldiers performance. It created an
opportunity, for example, to talk about the assumptions behind the rule of not resting the
magazine on the ground. This is more than merely interesting trivia; it makes explicit that there
are assumptions associated with any standard procedure and that these assumptions may have
nothing to do with the task at hand (e.g., learning the adaptive skills of marksmanship) and may
not be valid in the contemporary operating environment. Another is that, while there are several
solutions to most problems, the best solution may vary from time to time and from individual to
individual. The instructor also has the responsibility, however, to explain to the Soldier that
neither is the case that anything goes. This provides an opportunity to help the Soldier become
familiar with the notion of left and right limits for most tasks that both allow for and require some
autonomy, and that exploration is a reasonable manifestation of such bounded initiative.
42

2.4.9 Hotwashes and Mini-AAR
After-Action Reviews and Hotwashes are a persistently high priority in OBTE. These discussions
about lessons learned should be conducted in an open and supportive manner regarding. The
purpose is not to summarize what just happened in the training but to facilitate a discussion
among the student that allows them to evaluate their own behavior and performance in relation to
others. The distinction between self-assessment and peer-assessment evaporates in the AAR (S.
Flanagan, personal communication, April 30, 2009). The duality between self and other is
replaced by an extended sense of self (cf., Neisser & Jopling, 1997; Neisser, 1996).
Accountability is developed through a deep and enduring sense of the relatedness within the unit
in the context of shared objectives (cf., Deci & Ryan, 1985, 2008; Lerner & Tetlock, 1999, 2003).
Through these discussions, instructors ensure that students grasp the reasons behind a learning
event and its connection to a mission application. Additionally, this self-critique serves as a
problem solving activity in itself, boosting student confidence as they collaboratively discover
how to perform more effectively.
Example Measures
1. Do the instructors ensure the Soldiers can articulate the consequences of their actions?
Tells Soldiers what

went wrong and why it
is important in combat,
but does not discuss
how to mitigate
mistakes next time
2.
Asks Soldiers to explain

what went wrong, why its
important in
combat/mission, but does
not discuss how to mitigate
mistakes next time
Guides Soldiers through

explaining what went
wrong, why it is
important in
combat/mission, how they
might improve next time
Do the instructors focus the why of training back to the relationship between the
individual and big picture/mission?
Does not go into depth

on why the Soldiers
need the skills that
were trained (e.g. You
need these skills
Facilitates discussion on the

big picture; less focus on
the individuals
contribution
Facilitates discussion on
the big picture; focuses on
the importance of
individuals (i.e. You can
make a difference)
43
Marceau et al.
Example Vignette:
After a long day of training in the hot sun, all the Soldiers in the Platoon are exhausted and
hungry. Their drill instructor asks all the Soldiers to sit down on the shaded bleachers for a few
minutes prior to returning to the barracks. He then begins to tell them a story about a time when
his unit was deployed in Iraq. They were spending 36-hour cycles out in the city as part of an
effort to integrate with the populace. On one occasion, the drill instructors unit received sniper
fire from a neighboring building and one Soldier was wounded. One team broke off to go after
the sniper. They followed the sniper into a neighboring area of operations and eventually detained
him. They decided to secure a building and spend the night there because it was getting late. They
all were all exhausted and did not want to return to their AO in the night without more support.
By the time the Soldiers made it back to the FOB they had spent over 72 hours out in the city on
only 36 hours worth of food and water. The instructor uses this story not only to make
connections with a mission context but also as a point of departure for collaborative reflection
about the distractions of hardship and its effect on individual and collective decision making.
In the vignette, the instructor leverages the circumstances and effects of a long hot day to make
connections between a mission context and momentary conditions of training that otherwise are
distractions and apparently irrelevant to learning. Without his discussion, the Soldiers would
likely have gone to chow and never thought critically about that days training and its lessons for
performance in theater. Instead, the instructor revealed the operational relevance of their shared
experience in training on that day training. He did this not only by providing a narrative, back
story to which tired and hungry students might not even pay attention. He motivated their
collaborative reflection about the effects of such stressors on individuals and the group. He thus
brought them back into the moment as a learning event, even as they rested.
The AAR serves two purposes. It helps guide the activity of remembering in which the meaning
of prior events is actively reexamined with respect ones current capabilities for knowing and
interacting with the world (Neisser & Hyman, 2000). It also helps participants in the discussion
take ownership of their own learning and teaching (Magolda, 1999). In the context of OBTE and
instructor education, these are tightly interwoven cognitive and social activities (Sidman, Riccio,
Semmens, et al., 2009). The principles and practices of OBTE help focus an AAR on interactions
and interrelationships among tasks and among individuals within a task organized unit (see
Chapter 3). The conversation is more likely to be self-referential, actionable, and empowering
than a conversation about the real or imagined external factors that conspire to impede learning.
Changing the conversation is the easiest way to change the culture of training and education
(Cornell-dEchert, 2009b).
2.5 Uses of the Measures
The COMPASS process yielded a set of measures that have meaning within a community of
practice. Disciplined use of the measures provides the opportunity for the user to become
increasingly oriented to the values and best practices of the community. This provides additional
meaning and validity to the endeavors of anyone who uses the measures for formative feedback
(Mislevy, & Riconscente, 2006; Pellegrino, Chudowsky, & R. Glaser, 2001). It thus is an
important source of confidence for instructors. Given that the measures of shared values and best
practices refer to observable behavior, they also tend to ground confidence in a sense of
competence; that is, sources of confidence are verifiable by oneself and others.
44
2.5.1 Formative Measures for Instructors

There are three aspects of a good approach to using formative feedback for instructors: (a) it
should relate to the intent of instruction, (b) it should inform actions that can be taken to improve
instruction with respect to the intent, and (c) it should reflect the instructors span of control. The
relationship of the formative measures to the intent of OBTE is explored in this chapter. This
analysis is developed further in Chapter 3 in the context of categories of measures derived from
the relationship between the measures and the principles of OBTE (see also, Appendix A). For
instructors, the relationship emphasizes that feedback should not be collected willy-nilly. A good
instructor has a plan for what to observe based on the interpretation of intent in terms of the
events of the day and the situation at hand. Measures that support the plan should be given
priority. An experienced instructor may utilize additional measures as needed. In any case, it is as
important to decide what measures not to use, as it is to decide which ones to use, in a specific
learning event. An example of such planning is provided in Appendix A. An implication of
planning for formative assessment is that the measures also can be helpful in planning the
learning event itself. The measures can help an instructor prioritize opportunities and exigencies
of the moment in the context of student learning and development over time. They can help
connect prior and subsequent learning events, and they can help make connections between
learning objectives of an isolated event with longer-term outcomes for a student.
The very nature of the COMPASS methodology ensures that the resulting measures are
actionable because they are based on actual experience and observable behavior of instructors
that is verifiable in principle and in practice (MacMillan, et al., in press; MacMillan, et al., 2005).
Minimal interpretation is required to go from use of the measures to an appreciation of what an
instructor should do differently to improve. In a sense, the measures educate the attention of the
instructor about what is done and what can be done in a learning event with respect to the intent
of OBTE. The COMPASS methodology also tends to isolate actions and outcomes that are within
an instructors span of control. There are nuances in this use of formative feedback, however, in
that the command climate and organizational culture may, explicitly or implicitly, discourage the
behavior emphasized by the measures (Bandura, 1995; Rasmussen, 1997; Rasmussen, Brehmer,
& Leplat; 1991). In such cases, the measures have broader utility (see section 2.5.4 below).
2.5.2 Quality Assurance and Instructor Education
Measures that are informative and actionable to instructors are ideal for an external observer who
would provide feedback to an instructor for the purpose of assuring or improving quality. They
also facilitate the identification and promulgation of best practices among peers and throughout
an organization. One reason for this is that there are many measures that collectively represent
best practices for realizing the principles of OBTE and achieving its intent. An instructor could
improve by behaving better with respect to a particular measure or by behaving differently as
suggested by other measures. In a sense, the complete set of measures is a menu from which
instructors can select in the implementation and systematic improvement of OBTE. In any
situation, it will be easier and more effective to implement OBTE through some types of behavior
than others. Over time, this will become clear in the feedback obtained with various measures by
various instructors. The measures thus facilitate sharing of lessons learned that are of common
interest and relevance.
An interesting use of the complete measures set would be to design an instructor education course
around them (cf., Pellegrino, et al., 2001; Sidman et al. 2009). Categories of measures (see e.g.,
Chapter 3) could be used to design associated teaching modules ranging, for example, from a
half-day to multiple days in length. In a sense, such a course would be a guided version of what
45
Marceau et al.
instructors do in using the measures to plan, execute, and assess of their own instruction. In a
pedagogical context, it would be extremely valuable for instructor to be exposed to the
application of the principles and practices of OBTE to different skills or knowledge sets. This
would be good use of the collaborative tactical decision games promoted by the Adaptive
Learning Model (Vandergriff, 2006, 2007) that is closely allied with OBTE.
2.5.3 Continuous Improvement of Assessments
While the COMPASS process results in measures that are specific enough to provide formative
feedback to instructors, they are pointedly not as specific as they could be. In general, there are
just too many nuances and appropriate variations in how a particular instructor can interact with a
particular student. It would not be feasible to list them all, and it wouldnt be desirable to do so
because they would be too dependent on a particular course context. Moreover, in OBTE, it
would antithetical to tell instructors exactly what to do. This is especially important for
organizations in which instructors historically are told what to do or in which instructors expect to
be told what to do. This does not require individual instructors to be left completely on their own
in applying the measures to a particular course or learning event. A good organizational practice
would be to let instructors engage in collaboration with peers and their chain of command in
adapting the measures for their particular situations (Appendix C; Haskins, 2009; C. Haskins,
personal communication, June 2009; cf., Freeman, et al., 2008; Roberto, 2005, 2009). Such a
practice would be consistent with the approach to measure development described in this chapter
if similar methods of verification and validation were used. The most straightforward extensions
of these methods would be to modify the behavioral anchors for specific measures and to subset
or prioritize the measures with respect to local relevance. Adding new measures should be done
as a last resort and with some prudence.
An interesting use of the complete set of formative measures would be to assess or transform
other methods of assessment. The measures described in this chapter could be used to assess the
extent to which other measures capture some aspect of OBTE principles and practices. If an
external measure cannot differentiate good instructor behavior from poor instructor behavior as
captured in any of the measures described in this chapter, it would reveal a fundamental flaw in
the external measure. Given that assessments often drive or condition behavior within an
organization, it would simply be a matter of time before there is no systematic relationship
between instructional behavior and OBTE. On the other hand, an assessment of the assessment
could lead an organization to abandon irrelevant measures or to try to modify them in some way
to be consistent with OBTE and to differentiate good instructor behavior from poor instructor
behavior.
Consider, for example, adoption of OBTE in a particular program of instruction where other
methods and mindset of assessment are already in place. There may be resistance to change such
assessments given an assumption that they can or should be independent of the approach to
instruction. An example might be the use of Likert scales without any behavioral anchors, let
alone anchors specified at the right level of detail and grounded in best practices of OBTE.
Alternatively, a different set of measures may be developed for OBTE at a particular site. This
could be problematic if such measures are developed from invalid assumptions about OBTE or an
inadequate understanding of it. It is common, for example, for stakeholders with an interest in
OBTE to request a training support package or a description of the techniques that instructors can
be told to execute. Any assessments developed from this mindset might perseverate on situationspecific deviations of OBTE from standard practice such as, in the case of marksmanship
training, placing the magazine on the ground, placing the weapon on safe instead of squeeze the
trigger after SPORTS, shooting 5 round groups, and 200 meter zero. To adhere to such techniques
46
is not necessarily to practice OBTE. It confuses the incidental with the essential. Any measures
based on observation of such techniques, such as merely counting instances, are likely to miss the
mark. Concurrent validation against the measures in this chapter would reveal the flaws in simple
Likert scales or technique counting. Presumably this would stimulate consideration of
improvements in the assessment methodology.
The measures described in this chapter might also be useful in coding narrative accounts or openended comments about instruction and learning to facilitate summary and synthesis across a
variety of comments. This speculative methodology requires further research to understand its
practical viability and theoretical validity. In any case, it could be useful in stimulating and
guiding refinement of the current measures for specific applications.
2.5.4 Program Evaluation and Organizational Change
The application of assessments should be holistic (Pellegrino et al., 2001; Mislevy, 2006). Quality
assurance and improvement can be pursued at multiple levels in an organization (Rasmussen,
1997; Rasmussen et al., 1991). In this respect, it is noteworthy that the most successful
implementations of OBTE to date appear to be at sites in which it was supported by the command
climate (Haskins, 2009; Perry & McEnery, 2009; Schwitters, 2009). A powerful strategy for
creating a positive command climate for OBTE is to include instructors and their chain of
command in decisions about everything from outcomes to the level of specificity about the
structure of courses and associated assessments (see Chapter 14; Appendix C; C. Haskins,
personal communication, June 2009; cf. Roberts, 2009). The measures described in this chapter
also can be used for troubleshooting problems such as unintended or easily remedied resistance to
instructor behavior considered desirable in OBTE. Thus, every improvement does not rest solely
on the shoulders of the instructors. Some improvements can be accomplished at the
organizational level.
Consider, for example, the effect of a training support package for a particular learning event or
of a program of instruction that is specific about the timing and order of events on particular days
over a period of weeks. This level of detail can have a chilling effect on initiative, prioritization,
tradeoffs, judgment, and problem solving of instructors. Perhaps counterintuitively, it also can
have an undesirable effect on accountability and responsibility. While it may seem that such
detail gives an instructor something quite concrete and easily understandable with respect to
which they can be accountable, it makes them accountable for the wrong things. As instructors,
they should primarily be accountable for student learning, not for adherence to a script that may
or may not promote student learning given the situation at hand. The measures in this chapter can
be used to identify the level of detail that is necessary and sufficient to specify in advance for a
course. The result may be a menu of instructional events from which instructors can choose to
achieve the highest priority learning objectives given the conditions, resources, and time that can
be impossible to predict with specificity or certainty.
2.6 References
Group.
Group.
47
Marceau et al.

Bandura, A. (Ed.) (1995). Self efficacy in changing societies. Cambridge, UK: Cambridge
University Press.
Company.
Bransford, J. D., & Schwartz, D. L. (1999). Rethinking transfer: A simple proposal with multiple
implications. In A. Iran-Nejad & P. D. Pearson (Eds.), Review of Research in Education (24,
pp. 61101). Washington, DC: American Educational Research Association.
Camic, P.M., Rhodes, J.E., & Yardley (Eds.) (2003). Qualitative research in psychology:
Expanding perspectives in methodology and design. Washington DC: APA.
Cole, M., John-Steiner, V., Scribner, S., & Souberman, E. (1978). Mind in society: The
development of higher psychological processes (L.S. Vygotsky). Cambridge, MA: Harvard
University Press.
Deci, E.L. & Ryan, R.M. (1985). Intrinsic motivation and self-determination in human behavior.
New York: Plenum.
Deci. E.L. & Ryan, R.M. (2008). Self determination theory: a macrotheory of human motivation,
development, and health. Canadian Psychology, 49, 182-185.
Denzin, N.K, & Lincoln Y.S. (Eds.) (2003). Strategies for qualitative inquiry. Thousand Oaks,
CA: Sage.
Edmondson, A. (1999). Psychological safety and learning behavior in work teams. Administrative
Science Quarterly, 44, 350-383.
Factors, 37, 32-64.
Feldbaum, A. (1965). Optimal control systems. New York: Academic Press.
Filatov, N.M. & Unbehauen, H. (2004). Adaptive dual control: Theory and applications.
Springer.
Gagne, R., & Gibson, J. (1947). Research on the recognition of aircraft. In: J. Gibson (Ed.)
Motion picture training and research. Washington, DC: U.S. Government Printing Office.
Gibson, E. (1988). Exploratory behavior in the development of perceiving, acting, and the
Gibson, E. (1991). An odyssey in learning and perception. Cambridge, MA: MIT Press.
Gibson, J. (1977). The theory of affordances. In R. Shaw & J. Bransford (eds.), Perceiving, acting
and knowing: toward an ecological psychology. Hillsdale, NJ: Erlbaum.
Gibson, J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin.
Gibson, J. & Gibson, E. (1955). Perceptual learning: Differentiation or enrichment. Psychological
Review, 62, 32-51.
Glaser, B. (2002). Efficiency versus sustainability in dynamic decision making: Advances in intertemporal compromising. New York: Springer.
Guay, F., Ratelle, C. F., & Chanal, J. (2008). Optimal learning in optimal contexts: The role of
self-determination in education. Canadian Psychology, 49, 233240.
48

Henle, M. (1971). The selected papers of Wolfgang Kohler. New York, NY: Liveright.
Kolditz, T. (2007). In extremis leadership: Leading as if your life depended on it. New York, NY:
Wiley.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation.
Cambridge, UK: Cambridge University Press.
Psychology, 81(1), 146-159.
Lerner, J., & Tetlock, P. E. (1999). Accounting for the effects of accountability. Psychological
Bulletin, 125(2), 255-275.
Lerner, J., & Tetlock, P.E. (2003). Bridging individual, interpersonal, and institutional approaches
to judgment and choice: The impact of accountability on cognitive bias. In S. Schneider and
J. Shanteau (Eds.), Emerging Perspectives on Judgment and Decision Making (pp.431-457).
Cambridge: Cambridge University Press.
Press.
MacMillan, J., Garrity, M. J., & Wiese, E. W. (2005, November 25). The value of metrics: You
cant train what you cant measure. Mass High Tech. Retrieved June, 2009, from
http://tinyurl.com/km25pw.
Magolda, M.B.B. (1999). Creating contexts for learning and self-authorship: constructivedevelopmental pedagogy. Nashville, TN: Vanderbilt Press.
Mislevy, R. J., & Riconscente, M. M. (2006). Evidence-centered assessment design: Layers,
concepts, and terminology. In S. Downing & T. Haladyna (Eds.), Handbook of Test
Development (pp. 61-90). Mahwah, NJ: Erlbaum.
Neisser, U. (1976). Cognition and reality. San Francisco: W.H. Freeman.
Neisser, U. (Ed.) (1996). The perceived self: ecological and interpersonal sources of self
knowledge. Cambridge, UK: Cambridge University Press.
Neisser, U. & Hyman, I. (2000). Memory observed: Remembering in natural contexts (second
edition). New York, NY: Worth.
Neisser, U., & Jopling, D.A. (Eds.) (1997). The conceptual self in context. New York: Cambridge
University Press.
Pellegrino, J., Chudowsky, N., & Glaser, R. (Eds.) (2001). Knowing what students know: The
science and design of educational assessment. Washington, DC: National Academy Press.
Rasmussen, J. (1997). Risk management in a dynamic society: a modeling problem. Safety
Science, 27, 183-213.
Rasmussen, J., Brehmer, B. & Leplat (Eds.) (1991). Distributed decision making: Cognitive
models for cooperative work. London, UK: John Wiley and Sons
Riccio, G. (1993). Information in movement variability about the qualitative dynamics of posture
and orientation. In: K. Newell (Ed.). Variability and Motor Control (pp. 317-357),
Riccio, G. (1995). Coordination of postural control and vehicular control: Implications for
multimodal perception and simulation. In Hancock, P., Flach, J. Caird, J., and Vicente, K.
49
Marceau et al.
(Eds.), Local applications of the ecological approach to human-machine systems, pp. 122181. Hillsdale, NJ: Erlbaum.
Riccio, G., & McDonald, P. (1998). Characteristics of EVA Mass Handling Skill. Society of
Automotive Engineers Paper No. 981625 (also NASA Technical Paper 3684).
Riccio, G., & McDonald, P. & Bloomberg, J. (1999). Multimodal Perception and Multicriterion
Control of Nested Systems: III. A Functional Visual Assessment Test for Human Health
Maintenance and Countermeasures, NASA/TP-1999-3703c, Johnson Space Center, Houston,
TX.
Riccio G., & Stoffregen T. (1988). Affordances as constraints on the control of stance. Human
Movement Science, 7, 265-300.
Roberto, (2005). Managing for conflict and consensus: Why great leader dont take yes for an
answer. Saddle River, NJ: Wharton School Publishing.
Roberto, M. (2009). Know What You Don't Know: How Great Leaders Prevent Problems Before
They Happen. Saddle River, NJ: Wharton School Publishing.
Safonov, M. (2001). Recent advances in robust control, feedback, and learning. In: S.O. Reza
Moheimani, (Ed.). Perspectives in robust control. London, UK: Springer-Verlag.
Salas, E., & Klein, G. (Eds.) (2001). Linking expertise and naturalistic decision making.
Mahwah, NJ: Erlbaum.
Shaw, R. & J. Bransford, J. (eds.) (1977). Perceiving, acting and knowing: toward an ecological
psychology. Hillsdale, NJ: Erlbaum.
Shaw, R., Turvey, M., & Mace,W. (1982). Ecological psychology: The consequence of a
commitment to realism. In W. Weimer & D. Palermo (Eds.), Cognition and the symbolic
processes (pp. 159226). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Routledge:
Tschacher, W., & Dauwalder, J. (Eds.) (2003). The dynamical systems approach to cognition:
Concepts, and empirical paradigms based on self-organization, embodiment, and
coordination dynamics. River Edge, NJ: World Scientific.
Turvey,M. (1992). Affordances and prospective control: An outline of the ontology. Ecological
Psychology,4, 173187.
Warren, R. & Riccio, G. (1985). Visual cue dominance hierarchies: implications for simulator
design. Transactions of the Society for Automotive Engineering, 6, 937-951.
Winograd, E., Fivush, R., & Hirst, W. (1999). Ecological approaches to cognition: Essays in
honor of Ulric Neisser. Mahwah, NJ: Lawrence Erlbaum.
November 2006, 59-66.
Van Merrienboer, J., & Kirschner, P. (2007). Ten steps to complex learning: A systematic
approach to four-component instructional design. New York: Routledge.
50
Principles and Practices of OBTE

Chapter 3. Principles and Practices of Outcomes Based Training & Education
Gary Riccio and Morgan Darwin
The approach of the U.S. Army Asymmetric Warfare Group (AWG) to Outcomes-Based Training
and Education (OBTE) assumes that teaching in the information age must be different from
predominant approaches in the industrial age (Darwin, 2008a, b). Whether or not education takes
place in an institutional setting, the responsibility for both teaching and learning must become
considerably more decentralized than in the 20th century. OBTE further assumes that educational
practice in the Army must accommodate the fact that teaching and learning already is more
decentralized, and is becoming increasingly so, because of the social and semantic web
(Mikroyannidis, 2007; Paavola & Hakkarainen, 2005; Scardamalia & Bereiter, 2006).
Accordingly, the Institutional Army should be proactive in assessing its preparedness for the
demographic fact that the next generation of Soldiers will be comprised of millennials for
whom the ubiquitous decentralization offered by the World Wide Web is a basic fact of life
(Carlson, 2005; Howe, Strauss, & Matson, 2000). The initiative of the AWG in OBTE is a
response to this need for responsible decentralization of teaching and learning.
Responsible decentralization implies personal autonomy that must be bounded or grounded in
some way. In Army training and education, the appropriate grounding is in Army Values and
other cultural imperatives. OBTE helps identify how Army training and education can be
grounded in this organizational culture. Partially for this reason, OBTE is what many would
consider to be good training; however, such a characterization begs all the important questions.
Thus, the intent of this chapter is to elucidate specifically what good means in the context of
Army training and education, and how it relates to the emerging understanding of Full Spectrum
Operations. The result of this review is a better understanding of the established and emerging
instructional practices that should be sustained and those that should be improved to meet the
needs of the future. It provides the basic facts for a deeper scientific inquiry into the social and
psychological foundations of OBTE as an approach to teaching and learning.
3.1 Multifaceted Inquiry
Over the course of the investigation, a variety of methodologies were employed iteratively and
concurrently to help achieve a deeper and multifaceted understanding of the principles and
practices of OBTE (CMMI Product Team, 2009; Cresswell, 1998; D. Kirkpatrick, 1994; Miles &
Huberman, 1994; Msse, Moser, Stokols, et al., 2008; Tashakkori & Teddlie, 1998; Trochim,
2004; Trochim, Marcus, Msse, Moser, Weld, 2008). In addition to the COMPASS process
(Chapter 2), we analyzed surveys of participants in the AWGs Combat Applications Training
Course (CATC) (Chapter 6), observed CATC-trained instructors and their students in rifle
marksmanship training (Chapters 8 and 9), surveyed alumni of CATC about its downstream
impact in theater (Chapter 10); observed potential applications of OBTE in a classroom-based
instructor education course (Chapter 13), and identified critical issues pertaining to organizational
change (Chapter 14) and leader development (Chapter 15).
Our development of a grounded theory for OBTE also has been informed by several other
concurrent activities. In particular, we interviewed progenitors and practitioners of OBTE and
discussed relevant scientific research with them, attended AWG workshops and seminars on
OBTE, reviewed documents from the AWG, observed AWG instructors in CATC, participated in
CATC, interviewed stakeholders in the Army Training and Doctrine Command (TRADOC), and
51
Riccio & Darwin
participated in various non-AWG conferences and workshops in which OBTE was addressed.
These sources of evidence are briefly summarized below. The COMPASS process and its
products guided and facilitated qualitative crystallization across these multiple methodologies
(Camic et al., 2003; Denzin & Lincoln, 2003; Ellingson, 2009; Wise, 1980). The crystallization
involved more than constant comparison among different kinds of evidence. There also was a
balance and tension among the perspectives of a variety of experts with substantially different
backgrounds in a dialectic that extended over more than a year (Figure 1). The descriptions of
OBTE provided in this chapter reflect the claims, insights, and clarifications that emerged from
this multifaceted inquiry.
Scientific
Perspective
Officer
Perspective
SOF
Perspective
Perspective of
Conventional
Forces
NCO
Perspective
Engineering
Perspective
Figure 1. Balance and tension of different perspectives in the collaborative inquiry.

3.1.1 Interaction with Progenitors of OBTE
The military experts involved in our collaborative inquiry included both active duty and retired
Army noncommissioned officers (NCO) and commissioned officers associated with the AWG
who had been involved in the development of their initiative in OBTE. The most striking aspect
of this collaboration was that it did not involve a common division of labor across topics based on
assumed relevance of background. Instead, all perspectives were brought to every aspect of the
inquiry on a level playing field in which there was no place for assumptions of hierarchy or
dominance relations among different types of experience or expertise. The role of the technical
experts was not merely to transcribe the opinions of the operational experts but, instead, to
interpret their perspectives in terms of familiar scientific assumptions and paradigms. Similarly,
the operational experts interpreted the putatively relevant science in terms of exigencies of the
Full Spectrum Operations and preparation of Soldiers for it.
In additional to seeking common ground, the intent was to influence the sense making within the
multidisciplinary team in ways that were visible and traceable over an extended period of time.
The elaboration of the principles, practices and measures of OBTE summarized in this chapter
52
(see also Chapters 2, 4, and 5) are the result of these extended and iterative deliberations. This
process of crystallization is typical of the approach we have used in other research on Army
training and education (e.g., Riccio, Sullivan, Klein, Salter, & Kinnison, 2004; Sidman, Riccio,
Semmens, et al., 2009). Moreover, this kind of collaboration is representative of the
organizational environment of the AWG from which the initiative in OBTE emerged.
3.1.2 AWG Documents on OBTE
Various members of the AWG had prepared documents for internal coordination and for
presentation to stakeholders outside the AWG. Four documents were particularly influential in
our identification of assumptions and claims about OBTE (Darwin, 2008a, b; Cornell-dEchert,
2009a, b). These documents have been widely distributed among stakeholders in Army training
and education. Over a period of two years, one or more of these documents has been presented in
numerous forums in which there was opportunity for feedback and discussion of the meaning and
implications of OBTE at particular sites and in particular programs. While there were many
contributors to these documents, there was a clear first author and main contributor for each set of
two documents. Both of these authors are progenitors of OBTE.
One of the authors of this chapter, a retired Command Sergeant Major from the U.S. Army
Special Operations community, took the lead in preparing and maintaining a working document
comprised of approximately 100 PowerPoint slides. Darwin (2008a) is an early version, and
Darwin (2008b) is a more recent version of this document (the document is updated periodically).
The intended audience is primarily senior leaders associated with specific TRADOC units and
programs of instruction. The early version (Darwin, 2008a) laid out the purpose, vision, and
philosophical underpinnings of CATC. The later version (Darwin, 2008b) extended the concepts
more explicitly beyond marksmanship training to OBTE in general and revealed the relationship
between OBTE and doctrinal requirements for adaptability in Full Spectrum Operations. A retired
U.S. Army Lieutenant Colonel took the lead in preparing documents intended primarily for
personnel who are responsible for design, development, and implementation of specific Army
courses. One of these documents (Cornell-dEchert, 2009a) presents twenty-six articles for
OBTE that describe its intent and attempts to clarify misconceptions about the place of OBTE in
broader Army culture and practices. The other document (Cornell-dEchert, 2009b) is an
implementation guide that begins to translate the twenty-six articles into practices and
considerations for adapting the approach for existing programs of instruction. It is intended as a
living document to guide peer-to-peer sharing of best practices and lessons learned.
3.1.3 Collaborative Reflection on Participant Observation in CATC
Four members of the research team participated in the one-week CATC as students. One pair of
researchers took the course one week, and another pair took the course another week. Each pair
included one scientist and one person with military experience. The researchers in each pair were
encouraged to discuss their experiences with each other as the week unfolded and to document
key experiences and insights to the extent possible after each day of instruction. After completion
of the course, another member of the research team interviewed the participant-observers. The
interviews were conducted with two participant observers at a time, one from each of the prior
pairings; the two scientists were interviewed together, and the two retired military officers
together (one had been an Army Infantry Captain, and the other was a retired U.S. Marine Major).
The primary role of the interviewer was to stimulate dialog between the interviewees in which
they would compare and contrast their respective experiences and the meanings they made of it.
This dialog facilitated subsequent discussions with progenitors of OBTE, instructors in CATC,
and participants in CATC.
53
Riccio & Darwin
3.1.4 Interaction with Stakeholders

There were two main contexts in which there were opportunities for substantial and extended
interaction with OBTE stakeholders in Army training and education (See Chapter 1, section
1.3.2). One context involved the AWG seminars and briefings to individuals at sites associated
with particular programs of instruction. Members of the research team attended several of these
briefings. Another context was our preparation for field-based observations of the implementation
of OBTE at Fort Benning, Fort Jackson, and Fort Sill (see, e.g., Chapters 8, 9, and 13). We also
have interacted with many other stakeholders through various AWG events (e.g., AWG, 2009;
see Chapter 11, section 7.2.3).
3.2 Essential Characteristics of OBTE
The most important contributions of the AWGs initiative in Outcomes-Based Training &
Education (OBTE) and the associated research are the claims and evidence that (a) the gap
between Army instruction and Full Spectrum Operations can be viewed in terms of the
relationship between a singular focus on tangible, near-term objectives and the resulting effect on
enabling intangible attributes valued by Army leaders, (b) everything that is done in Army
training and education has an impact on this relationship whether one addresses it or not, (c) this
impact can be understood and described thus allowing leadership to make informed decisions
based on verifiable observations and valid scientific reasoning, (d) OBTE is an approach that can
be utilized to inform leaders at all levels on the direction their unit is moving in the development
of leadership traits, problem solving skills and intangible attributes, (e) the approach can be
verified and validated, and (f) it can influence without necessarily replacing any extant
instructional methodology (M. Darwin, personal communication, June 30, 2009).
3.2.1 The Meaning of Developmental is a Critical Difference
OBTE is an approach to training and education that focuses on development of the individual in
relation to basic needs of the individual as well as associated cultural values and objectives of the
individuals occupation (Darwin, 2008a, b; Cornell-dEchert, 2009a, b). The meaning of
developmental thus is fundamentally different from its typical connotations in training and
education, which emphasize individual differences and, in many cases, refer to remedial learning
in specific subject matter. This is not to say that OBTE is unconcerned with Soldiers mastery of
specific knowledge and skills. An assumption of OBTE is that longer-term developmental
outcomes can be pursued while Soldiers are being prepared for specific tasks (Cornell-dEchert,
2009b). In OBTE, immediate and long-term objectives are not considered fundamentally
inconsistent. There may be tradeoffs about nested instructional objectives to be made on the scale
of hours in particular learning events to the scale of months or years in curriculum design, but this
more likely would be a question of return on investment rather than incompatibility (see Chapter
5 and Epilogue). More generally, OBTE is an approach in which persistent awareness of longterm outcomes, including but not limited to the explicit learning objectives for an event,
influences the planning and execution of instruction in important ways.
Why consider Soldier development in training and education when it seems to be the purview of
other programs, the responsibility of other people, or a focus for time periods outside of formal
learning events? The reason is simple and inescapable. Every interaction with a student influences
long-term development whether or not it is specifically considered in planning or execution of
instruction. The profundity of this human factor in training and education has been recognized
from antiquity to modern times (e.g., Aristotle, trans. 1925; Augustine, trans. 1953; Bandura,
54
1977; Dewey, 1915/2008; Guay, Ratelle, & Chanal, 2008; James, 1890/1907; Keatinge, 1896;
Lave & Wenger, 1991). OBTE takes into consideration that an individual can be changed by the
learning experience in ways that transcend the particular knowledge or skills acquired during a
learning event.
On this philosophical foundation in learning and development, OBTE stridently rejects any
separation of learning and doing. This is not merely a superficial claim about experiential
learning (i.e., that one learns by doing). OBTE claims that, while applying knowledge or skill in
practice, individuals also can learn about other things that can be done or that can be known.
Individuals weave performatory behavior with exploratory behavior to achieve a more
differentiated and more refined understanding of their capabilities and of the environment in
which they are situated (J. Gibson & E. Gibson, 1955; E. Gibson, 1988, 1991). This weave of
performance and discovery essentially is an inter-temporal compromise between immediate and
long-term goals of behavior (cf., Glaser, 2002; Riccio, 1993a). In general, inter-temporally
directed behavior often involves minimal tradeoffs to the extent that exploratory behavior can be
nested within performatory behavior with little or no interference (cf., Feldbaum, 1965; Filatov &
Unbehauen, 2004; Riccio, 1993a, b; Riccio & McDonald, 1998). This is addressed in Chapters 4
and 5 in the context of individual adaptability and collective agility.
OBTE claims that there are ubiquitous opportunities for instructors and students to engage in
thinking or activity beyond the task at hand, amid the activities required by the task at hand.
These opportunities exist on the scale of seconds to minutes. Moreover, OBTE claims that
instructors and students often are engaged in such apparently incidental behavior, that is, behavior
not prescribed by the task at hand. The less esoteric of version of this claim is that human beings
are not inanimate objects or machines even when they are implicitly or explicitly treated as such
in the development or execution of instruction. Our informal observations in the field revealed
that, when students are engaged motivationally in a learning event, their thinking and behavior
outside the task at hand relates to the task at hand. For example, under favorable conditions,
students will spontaneously talk with each other about performance on a task even if they are not
required to do so. Such ad hoc engagement in learning does not introduce any delays in the
learning event. Under less favorable conditions, something unrelated to the learning event will
capture the attention of students in the time periods between behavioral scripts. In both cases,
habits of learning and engagement are nurtured, for good or ill.
Rarely if ever, is each student involved in scripted behavior at every second of every learning
event, especially when the student-to-instructor ratio is high. An assumption of OBTE is that,
during waiting time (e.g., in the ubiquitous queues), students are making sense of their experience
in a learning event. This does not necessarily involve overt or even explicit intentional analysis of
the event. In the extreme, it may involve nothing more elaborate than the conclusion that there is
little sense to be made of the event but the consequences for human development are no less
profound. OBTE seeks to engage such metacognitive activity to have a more purposeful influence
on development. As discussed below, this can be achieved in the approach to planning and
execution of instruction without necessarily adding tasks for the instructor or the student. At a
high level, OBTE influences metacognitive activity by addressing the reasons behind the
structure of a learning event and the constraints on its validity. It also does this by providing
models (i.e., real-time examples) of collaborative reflection and reinforcing it wherever
appropriate. This does not require an instructor to be everywhere at all times or to help every
student exploit every opportunity for useful reflection. It merely requires some examples of
collaborative reflection that students can model whether they participated in the examples or
experienced them vicariously.
55
Riccio & Darwin
Even occasional engagement of students on a metacognitive level is powerful because this is

where ownership of their own learning and development resides. This should not be interpreted in
ways that imply an infinite regress in analysis of ones own thinking and behavior.
Fundamentally, it is about the deeply personal struggle between the known and the unknown,
between the controllable and the uncontrollable. It is where passion and faith intermingle with
reason and belief (Figure 2) (Kierkegaard, 1843). Through development, it is where individuals
internalize previously external sources of motivation and where they make meaning of the
balance between what they can control and what they are either unwilling or unable to control,
that is, with respect to which they yield or cede power (cf., Aristotle, trans. 1925; Augustine,
trans. 1991; Hoffman, Yang, Kaklauskas, & Chan, 2009; Idel & McGinn, 1999; James,
1902/1982; Maslow, 1964; Plato, trans. 1892a; Solomon, 2004). Engaging Soldiers on this level
helps them find a balance between choice and responsibility that is personally sustainable and
more likely to help them be successful in the ambiguous and changing conditions of Full
Spectrum Operations. Students will find their own levels but it is in the interest of future
commanders, peers, and subordinates for Soldiers as students to understand and continually
develop their relationships with various sources of power that influence their decisions and
actions, beyond the formal or explicit lines of authority within their units. For this reason, as
discussed below, OBTE motivates the development and use of formative measures that engage
students and instructors on this level (see Chapter 2).
Figure 2. Intangible metacognitive and metaphysical influences on capabilities for thinking,

decision-making, action, and their development.
With respect to development and assessment, OBTE is related to approaches in education that
focus on progress maps, especially those that consider multidimensional constructs including
latent variables (Pellegrino, Chudowsky, & R. Glaser, 2001). The latent variables of OBTE,
however, are not mathematical constructs or other abstractions derived computationally from
quantitative data. They are based on assumptions about shared values and qualitative evidence
about an individuals values that manifest in both mundane and extraordinary situations (cf.,
Aspinwall, & Staudinger, 2003; Keyes & Haidt, 2003; Peterson & Seligman, 2003; Riccio et al,
2004; Webster, 2001). OBTE considers Soldier development to be the most powerful focus for
any approach to Army training and education insofar as it addresses attitudes and values that
guide life-long learning and self-development. More than anything else, this focus engages
instructors as leaders and leader developers. In this sense, OBTE essentially is both values-based
and leadership-based training and education. It thus helps bring what the Army does best into the
56
realm of training and education (United States Army Training and Doctrine Command
(TRADOC], 2009). It seeks to give training and education a higher status and sense of calling
(Riccio et al., 2004).
3.2.2 The Definition of Outcomes is a Critical Difference
The current revision of TRADOC Regulation 350-70 tries to provide guidance on the selection
and use of a variety of instructional methodologies and associated theories of education. In this
respect, it is important to note that OBTE is more general and inclusive than a particular method
of instruction or a prescriptive set of instructional techniques. At the same time, from the
perspective of OBTE, not all methods are equal. OBTE is not, for example, consistent with
outcomes based methods of instruction that focus exclusively on course-specific objectives.
OBTE should not be confused with constructs that are based on a definition of outcomes that is
more like the definition of common learning objectives and that implies a different developmental
approach. While the outcomes based education movement of the 1990s (see e.g., Spady &
Marshall, 1991; Spady, 1994, 2009), for example, stressed more of a focus on the objective
output than on an a priori script or input focus for an instructional event (Figure 3), it is
inconsistent with OBTE on a fundamental level in that it rejects values, attitudes, internal mental
processes or psychological states of mind as outcomes (Spady, 1994, 2009).
While reverse planning is a common aspect of different approaches to outcomes-based
instruction, OBTE starts with different kinds of outcomes, ones that relate to attitudes and
attributes developed in any learning event whether or not one explicitly recognizes it. Another
fundamental difference is that OBTE is developed for situations, as in the Army, in which there
are practical constraints on time and resources. In the Army, there is not the luxury to allow
students to achieve a pre-specified level of performance at their own pace. OBTE requires an
inter-temporal focus that balances immediate learning objectives of the task at hand with longerterm habits of thinking and action that transfer to different situations. In this respect, it is not wise
to give high performers short shrift so that relatively low performers can be given more assistance
with their performance. All Soldiers can benefit from being engaged on the level of
metacognitive development (section 3.2.1) irrespective of their level of knowledge or skill.
Currently there is considerable confusion in the use of the terms objectives and outcomes.
Sometimes they are used synonymously, sometimes not, and there is a conspicuous lack of
consistency in the use of each of these terms. An outcome in OBTE is not the same as a terminal
learning objective that typically is the focus of Army training and education. We equate
objectives with intended results of a well-defined component of training and education, whether
short-term or long-term. We refer to outcomes as all results of a well-defined component of
training and education, whether intended or not. In general, intentions are specified explicitly
only with respect to short-term results, at least in any way that is verifiable or measurable; thus
use of the term objectives often connotes short-term results. When verifiability of objectives is
considered, the focus is almost exclusively on tangible results. The use of the term outcomes in
OBTE emphasizes long-term results that, of necessity, are mostly intangible but that also can
become verifiable and measurable, in conjunction with traditional short-term results.
One way to clarify the use of the terms, objectives and outcomes, in OBTE is to begin a rigorous
and traceable inquiry into the relationship between verifiable intangible long-term results
(outcomes) and verifiable tangible short-term results (a priori objectives and ensuing outcomes).
The first step in this inquiry is to identify ways to talk about and measure both tangible and
intangible results in which they are commensurable. Then one can begin to identify the
antecedents of long-term results that can be observed at the same time short-term results are
57
Riccio & Darwin
observed (e.g., Deci & Ryan, 2008; Peterson & Seligman, 2004; Rick & Loewenstein, 2008;
Riccio et al., 2004). This would enable identification of conflicts and synergies in the means to
achieving both short-term and long-term results. We believe that a promising line of inquiry for
OBTE is the role of motivation and emotion in tradeoffs involving intangible long-term outcomes
that influence perception and action (Camber, Loewenstein, & Prelec, 2005; Loewenstein &
Lerner, 2002; Loewenstein, Weber, Hsee, & Welch, 2001; cf., Deci & Ryan, 2008; Ryan & Deci,
2008; Peterson & Seligman, 2004). As demonstrated by the measures of Chapter 2, intangibles
should not necessarily be interpreted as fuzzy concepts. They are only intangible in terms of their
manifestations under the uncertain conditions of the future (cf., Rick & Loewenstein, 2008).
Figure 3. Highly stylized depiction of three approaches to training and education. Dotted
horizontal line represents target performance associated with the terminal learning
objective for a learning event. Green lines represent relatively fast learners; red lines
represent relatively slow learners. Gray band indicates uncertainty about retention of
skills or knowledge or transfer of learning to a new situation.
It may be helpful to consider the antecedents of long-term outcomes as analogous to enabling
learning objectives typically associated with terminal learning objectives in Army training and
education. This suggests a potential synthesis by recognizing that objectives are nested (cf.,
Feldbaum, 1965; Filatov & Unbehauen, 2004; Glaser, 2002; Riccio, 1993a,b; Riccio &
McDonald, 1998; Riccio & Stoffregen, 1991), perhaps in ways that are directly analogous to
58
nested intent statements in mission command (Cornell-dEchert, 2009b; cf., Appendix C;

Freeman et al., 2008). Longer-term attitudes and values can be addressed, along with immediate
learning objectives, in the context of the same learning event. This approach seems timely given
the exigency in the most recent version of Army leader development strategy to integrate leader
development with training and education (TRADOC, 2009). Toward this end, we believe it is
essential to make more progress in identifying scientific underpinnings for OBTE. The
development of measures for instructors helped us recognize some of these connections (see
Chapter 2). In this chapter, we begin to describe the research that followed the development of
measures, that helped us refine and organize them, and thus that helped us realize a theoretical
commitment for a rigorous line of inquiry (see Chapters 4 and 5).
3.2.3 The Emphasis on Values and Causally Potent Intangibles is a Critical Difference
The principles of OBTE (e.g., Darwin, 2008a, b) reveal the kinds of outcomes that can be
addressed during any learning event. The principles also suggest why these outcomes are
important with respect to retention and transfer of learning to operational environments (i.e., subbullets in list of principles below).
Training to Grow Problem Solving

o teach Soldiers to learn for themselves the skills necessary to the success of
their mission, within an established framework of knowledge
Training to Increase Intangibles
o develop intangible attributes like confidence, accountability, initiative, judgment
and awareness; reinforce positive character traits
Training to Increase Understanding and Awareness
o teach through contextual understanding of the task and its mission application
Training to Increase Deliberate Thought
o condition Soldiers to exercise always a deliberate thought process (evaluation,
judgment and decision) while under stress
Training to Improve Combat Performance
o condition Soldiers to overcome the psychological and physiological effects, and
the physical requirements of combat
A focus for our interactions with the progenitors of OBTE was to develop a better understanding
of the causal potency of these principles. The most significant discussions revolved around the
so-called intangibles of OBTE. This collaborative inquiry involved constant comparison with
diverse lines of research in psychology and related disciplines that have a long history of
scholarship with respect to intangible attributes and latent causal variables. While such
interdisciplinary connections can be as complex as one would like them to be, we chose to
identify a simple framework for sustained trans-disciplinary communication and collaboration
that is well grounded in a variety of disciplines across the physical, biological, behavioral, and
social sciences. In particular, we chose a triadic framework within which a balance and
equilibrium of two causal factors is manifested with respect to a third factor (see e.g., Burke,
1985; Lorentz, Einstein, Minkowski, & Weyl, 1923/1952; Glass & Mackey, 1988; Goldberg,
1986; Heisenberg, 1958; Lanczos, 1970; Wolfram, 2002). The point here is not to equate our
current scientific understanding of OBTE with these well-developed branches of science. Instead,
the point is that considerable explanatory power with respect to complex phenomena can be
achieved from relatively simple concatenation operations if one starts with primitives and
59
Riccio & Darwin
principles that are trustworthy. Such confidence typically derives from a history of careful
observation of phenomena from which primitives and principles are abstracted.
We believe a triadic framework is the simplest possible expression of the essential features of
OBTE and thus is most useful in translating theory into practice. The primitives and principles in
this framework are based on decades of experience of a variety of people who have approached
Army training and education from different albeit converging perspectives (Figure 4). The
nascent concatenation operations of balance and tension between primitives are based on this
experience as well as on themes from a variety of philosophical perspectives that have stimulated
thought and dialogue for millennia. Some of the important implications of a triadic framework are
discussed further in Chapters 4 and 5 and in the Epilogue.
Figure 4. Triadic balance and equilibrium among intangible personal attributes

and the associated scaffolding for capabilities of the human dimension in Full
Spectrum Operations and competence in trans-extremis environments (e.g.,
transitions between strategically significant combat and noncombat operations).
Early on in the collaborative inquiry with progenitors of OBTE, we identified three intangible
attributes as foundational for OBTE. Confidence, initiative, and accountability became the
organizing framework for outcomes that are given special attention in OBTE. We believe that any
two of these attributes has an impact on the third attribute but, for power and simplicity, we
emphasize the balance between initiative and accountability. The power of this dyad can be
understood in terms of the interplay between choice and responsibility (Aristotle, trans. 1925;
Augustine, trans. 1991; Hickman, L.A. & Alexander, 1998; James, 1890; Solomon, 2004) or
between agility and the variable contextual constraints of Full Spectrum Operations (HQDA,
2003; 2005). This triad emphasizes the urgency of choice within contextual understanding of
military situations, with particular attention to the powerful aspect of initiative coupled to moral
accountability, as compared to ethical responsibility (cf., Lvi-Valensi, 2006; Clausewitz, 1976
trans.; Kierkegaard, 1843). There are any number of motivational contrivances and tricks that can
be employed to make individuals more flexible or to display more initiative but this is not what
the Army or the Nation needs of its Soldiers. Soldiers need to be adaptable in initially ambiguous
60
and in changing conditions. Adaptability implies that their decisions and actions are appropriate
given the context. At the same time, the Army needs Soldiers to exhibit cunning and not merely
to be driven by context. There is a need for units to be agile, proactive rather than reactive, and to
recognize the functional potency of human agency in social structures.
The balance of agility and context is not merely about recognition of the limits of power. It also is
about an appreciation of the consequences of exercising power (cf., Clausewitz, 1976 trans.).
Consequences include longer-term effects, including unintended effects, as well as impact on the
task at hand. OBTE assumes that habits of such inter-temporal awareness and thinking can be
developed in any and all learning events, in formal learning events or informal life-long learning.
This is an important determinant of the confidence the Army needs in its Soldiers, not a false
confidence bred from ignorance, invalid assumptions, or unexamined misinterpretations. This is
not to imply that Soldiers must know everything at all times. To the contrary, OBTE assumes that
habitual experience with the interplay between initiative and accountability helps Soldiers come
to terms with uncertainty and the application of knowledge and skill amid uncertainty. While this
is a deeply personal experience, in OBTE, it is not a solitary endeavor. OBTE builds habits of
collective struggling with uncertainty and collaborative experience even in the context of
ostensibly individual tasks.
There are other intangibles emphasized in OBTE. To maintain simplicity, such intangibles are
envisioned in terms of a triadic infrastructure (Figure 4). This conceptual nesting should be
viewed as analogous to the grouping of causal factors in a factor analysis. The nesting is not
intended to represent all the relationships or to explain all the variance. It is intended to be the
least arbitrary grouping that achieves the simplest possible description of a myriad of concepts.
By simplest, we mean the easiest to put into practice. In this sense, every attribute can be
addressed through the interplay of two other attributes.
We explored a triadic infrastructure in another project to simplify inquiry into the inculcation of
Warrior Ethos in Initial Entry Training (Riccio et al., 2004; Bruny, Riccio, Sidman, Darowski, &
Diedrich, 2006; Appendices D and E). We found that this framework also helped achieve a useful
mapping among a variety of values-based concepts in current Army doctrine. Again, while such
relationships are not intended to suggest one-to-one mappings or a full understanding of each
concept, they do alleviate unnecessary complexity in important sets of Army requirements and
thus suggest ways in which the requirements have actionable implications for Army training and
education. Along those lines, the mapping has recently been extended to yet another set of valuesbased requirements in Army training and education (Sidman, Riccio, Semmens, et al., 2009). The
composite triadic mapping is indicated in Table 1.
Table 1. Triadic mapping among plethora of concepts in Army training and education.
OBTE
Intangibles
Confidence
Initiative
Judgment
Deliberate thought
Accountability
Discipline
Awareness
Attributes of
Warrior Ethos
Perseverance
Adaptability
Prioritization
Tradeoffs
Sense of Calling
Responsibility
Dependence
Tenets of
Warrior Ethos
Mission First
Never Accept Defeat
Never Leave Fallen
Never Quit
Army
Values
Courage
Respect
Duty
Honor
Integrity
Selfless Service
Loyalty
Pentathlete
Characteristics
Warrior Leader
Critical Thinker
Creative Thinker
Leader Developer
Resource Manager
Ambassador
61
Riccio & Darwin
The triadic mapping between the intangibles of OBTE and the attributes of Warrior Ethos is most
striking. This is important because the focus of the project that generated the attributes of Warrior
Ethos was to provide valid and verifiable ways to assess progress in inculcating Warrior Ethos in
Soldiers during Initial Entry Training. A promising line of inquiry is the adaptation and utilization
of the attributes of Warrior Ethos as measures of long-term development, perhaps even standards,
that relate to values-based requirements in current Army doctrine (see Epilogue).
3.2.4 The Meaning of Experience is a Critical Difference
In OBTE, experience refers to more than just a personal history of participation in learning; that
is learning by doing. To be sure, participation in ones own learning is central to OBTE; however,
participation also is the best way to experience learning and to find meaning in it. Experiencing
the conditions, process, and consequences of learning is important. In other words, consistent
with the philosophy of John Dewey, it matters what the student experiences during learning
(Hickman & Alexander, 1998). In OBTE, it is particularly valuable if the learning experience
includes reflection on the dynamic interplay of attributes such as initiative and accountability, the
confidence one derives from the balancing of these attributes, the resulting capacity to navigate
the unknown with what is known, and negotiation between internal and external sources of
power. While this may seem like an existentially significant or cognitively intense undertaking, it
need not be. That it can be more mundane is, of course, critically important in practice. That it
should become somewhat routine or ordinary is the point of OBTE (cf. Riccio et al., 2004;
Appendices D and E).
Figure 5. Development of thinking in action and thinking about action in

outcomes-based training and education [from Darwin, 2008a].
An essential feature of experience in OBTE is that it is inter-temporal (see e.g., Figure 5). There
is the students experience during participation in the designed learning event, and there is the
students experience outside the context of this required participation. The latter typically is
neglected. It can be neglected either unintentionally or even by design based on priorities or
expectations that do not associate intangible attributes with the performance or learning of a task
or subject. One challenge is that instructors typically are not involved with every student during a
scripted learning event, thus it may appear that instructors cannot have an influence on it. Another
62
challenge is that discipline may preclude overt activity of students when they are not involved in
a scripted learning event (e.g., during waiting time), or the layout of the range or the classroom
may make the informal activity of students difficult to observe. In neither case would it be
accurate to assume that no thinking or communication is occurring. If neglected, the thinking that
occurs during such time periods may not be favorable to learning and retention of the knowledge
or skills addressed in the formal learning event. In any case, there is an opportunity to stimulate
reflection during these frequent periods of time, however brief, to help students make valuesbased connections with experience in the learning event, and thus to influence longer-term
outcomes. OBTE aggressively and systematically seeks to exploit such micro-experiences to
build habits of inter-temporal thinking. We believe that micro-experiences are a significant
untapped or misused resource in most learning events (M. Darwin, personal communication, May
5, 2009).
3.2.5 The Emphasis on Instructor-Student Interactions is a Critical Difference
The most concrete findings from our inquiry into OBTE are best practices associated with
interactions between instructors and students (Chapter 2, Appendix A). These practices are not
expressed as immutable scripts but as guidelines for the ongoing pursuit of mastery in the art of
teaching (cf., Augustine, trans. 1953; James, 1890/1907; Keatinge, 1896). The essential aspects of
this art in OBTE are depicted in Figure 6.
Figure 6. Three critical roles of an instructor as a leader, leader developer, and

influencer of human development [From Darwin, 2008a].
Instructor-student interactions seem to be relatively neglected in the science of education,
presumably because it is considered to be an art and something that can only be addressed in the
field, or perhaps because it is deeply and broadly embedded in apparently esoteric research of
scientific psychology. We agree that acumen in working with students is something that can only
be learned in the field but we reject any notion of a strict separation between the art of teaching
and the science of psychology (cf., James, 1890/1907). Part of the intent of OBTE, and of
chapters 3 through 5 in particular, is to reorient Army training and education to theory and data
from relevant subdisciplines in scientific psychology. Our position is that the design,
63
Riccio & Darwin
development, and implementation of instruction can get considerable practical guidance from a
deeper understanding of recent research and historical inquiry in scientific psychology that
addresses the reciprocal influence between individuals and their environment and the impact
these interactions have on the development of motivation, personal identity, and values.
3.2.6 The Emphasis on Learning to Learn is a Critical Difference
The emphasis of OBTE on instructor-student interactions begs questions about how to enable this
through staff and faculty development (see Epilogue). We dont believe that this implies a
necessity for additional instructor education in established theories of education with which
OBTE seems to have most in common, such as experiential education (e.g., Hickman &
Alexander, 1998; Kolb, 1984), constructivist education (e.g., Hmelo-Silver, 2004; Tobias &
Duffy, 2009), or adult education (e.g., Jarvis, 2004; Knowles, Holton, Swanson, 2005). We
believe it is important to return to the foundations for educational theory in psychological science
both historically and in terms of significant trends in psychological science over the last thirty
years (see Chapters 4 and 5). Partly for this reason, and to avoid premature conflation of OBTE
with extant theories of education, the progenitors of OBTE have chosen to orient their principles
and practices with respect to a simple framework for education that has endured for millennia
(Figure 7). Interestingly, this framework suggests an approach to Army training and education
that is reminiscent of a liberal arts education rather than a pre-professional education.
Figure 7. Conceptualization of outcomes-based training and education in terms of

the Trivium and the spiral development of individual competence based on
continuing experiences [from Darwin, 2008a].
It could be argued that one of the most important objectives of a liberal arts education is to help
students acquire learning skills inside the classroom that will help them learn outside the
classroom. OBTE assumes that, in the Army, such skills center around learning collaboratively
with others (cf., Lave & Wenger, 1991; Wenger, 1998). Collaborative experience and intersubjectivity thus is important in OBTE, not just in transactions between instructor and student but
also in interactions between peers. Peer-to-peer dialog is valuable for several reasons. It provides
students with opportunities to learn with, through, and from another person who is not necessarily
64
in a position of authority. Thus the student can learn in a formal learning event how to learn
outside the context of formal learning events, that is, in situations in which an authority figure is
not available to tell individuals what to do. It also provides students with the experience of
transitioning between momentary context-specific roles of leader and subordinate. This skill is
invaluable in task organization of groups in ambiguous or changing conditions.
Learning to learn is directly relevant to notions of adaptability (see Chapter 4). There seems to be
different meanings of adaptability in the institutional and operational Army. In an attempt to
bridge this gap, we offered a more axiomatic formulation of the problem of adaptability in a
workshop held by the University of Southern California and Institute for Creative Technologies,
in April 2009.
1) The vast majority of learning in the Army occurs on the job or in ad hoc instructional events.
a) Instructional events designed and delivered by leaders at the company level and below.
b) Instructional objectives driven by exigencies of the day and influenced by priorities
implicit in intent of higher commanders.
c) The implication is that most learning happens outside the context of curricula and courses
that result from the Army's formal instruction system design (ISD) process.
d) Need: What can be done in the institutional Army to develop curricula and courses that
improve the quality of instruction and learning outside this formal context?
2) The amount of operationally relevant learning occurring in formal curricula and courses in
the Army has been increasing.
a) This is partly because of greater and more prevalent experience of instructors in the full
spectrum of operations (i.e., offense, defense, stability, support)
b) This is partly because of the intent of leadership in Army schools to utilize experience of
instructors and close the gap with the operational Army.
c) It is accepted that lessons learned from experience in Full Spectrum Operations can be
valuable in the classroom.
d) Need: How can we determine what kind of experience is useful in the classroom and how
it should influence curriculum development and instructional activities?
3) The most fundamental unit of analysis in the Army is the group not the individual.
a) Adaptability and the associated agility of groups in the Army is exemplified and fostered
by task direction and organization (i.e., ad hoc task-specific division of labor that takes
into account what is known about group members).
b) Adaptability and the associated agility of groups in the Army is exemplified and fostered
by ad hoc prioritization and tradeoffs (e.g., typically there is more to be done than can be
done given time and resource limitations) that take into account understanding of one's
responsibility to others in the group and one's dependence on them.
c) Need: What are the implications for curriculum development (e.g., nature and extent of
instructional guidance) if we assume that Soldiers must become better at utilizing and
eliciting guidance outside the classroom to achieve collective agility rather that assuming
that they need to become less dependent on guidance in moving from the classroom to
the job?
In OBTE, micro-experiences typically are inter-subjective as well as inter-temporal. The
experiences create a habitual orientation to shared values and long-term outcomes. This has
implications about the consideration of individual differences in personality or learning style. In
particular, it suggests that there is some value to acquainting students with the effects of
individual differences in collaborative work (see e.g., Allik & McCrae, 2004; de Raad &
65
Riccio & Darwin
Perugini, 2002; Heine & Noranzayan, 2006; Hofstede, Pedersen, & Hofstede 2002; Matsumoto
&Yoo, 2006) insofar as it could help them with task organization and the collaborative agility it
enables. Interestingly, the assumptions about adaptability in OBTE also suggest that too much
accommodation for individual differences during a learning event may be counterproductive. The
actors and trans-extremis conditions in Full Spectrum Operations are not likely to accommodate
individual differences in ways that instructors can, in principle. Thus, careful thought about
downstream consequences should be given to the treatment of individual differences in Army
training and education.
3.2.7 The Emphasis on Collaborative Design and Development is a Critical Difference
In Army training and education, historically, there has been a rather strict division of labor
between analysis, design, development, implementation, and evaluation (ADDIE). OBTE argues
for less separation and more collaboration across these curricular activities and responsibilities
(see e.g., Chapters 14 and 15; Cornell-dEchert, 2009a, b). In particular, we believe that instructor
cadre should be included in local decisions involving prioritization and tradeoffs about the extent
to which various skills and knowledge can be addressed in a program of instruction as well as
when and how they should be addressed. This helps establish a command climate in which
mutual trust is both highly valued and salient (cf., Bandura, 1995; Edmonson, 2003; Vaughan,
1997; Rasmussen, 1997). It helps an instructor take ownership of the instructional environment;
and it helps them take responsibility for instructional effectiveness rather than merely nominal
efficiency. The instructor is able to prioritize on the task set forth by the Army, to the standard set
forth by the Army, but in a manner that permits adaptation of the instructional environment to the
uniqueness of the instructors situation. OBTE maintains that this results in a positive, long-term
outcome for instructor and student.
Figure 8 depicts a process that has emerged in the adoption of OBTE in various programs and at
various sites in the Army (e.g., Appendix C; Perry & McEnery, 2009). Commanders in
instructional programs have established a command climate and communicated an intent that
fosters collaborative discussion and decision-making by instructional cadre and their chain of
command to achieve a balance of near-term learning objectives and the long-term developmental
outcomes emphasized by OBTE. These commanders are influenced heavily and self-consciously
by a belief that there is a gap between the historical priorities of institutional training and
emerging operational needs such as preparation for the ambiguous and changing trans-extremis
conditions of Full Spectrum Operations. With the participation of their instructional cadre, they
establish priorities based on emerging needs of Full Spectrum Operations as defined by the Army,
and make tradeoffs based on limited time and resources while still achieving the intent of higher
commanders.
An important aspect of the decentralized programmatic decision-making occurring in the
adoption of OBTE is the identification of outcomes that are observable during training and that
have a plausible causal relationship with downstream outcomes in the operational environment
(see section 3.2.2). The principles and intangibles of OBTE have been helpful to commanders and
their instructional cadre in their collaborative identification of outcomes. The intangibles guide
thinking about plausible relationships between near-term outcomes and long-term development
outcomes (see section 3.2.3). The principles of OBTE have been helpful in identifying outcomes
that are most important in Full Spectrum Operations and in identification of instructional
practices that help develop attributes in Soldiers on which such outcomes depend. Verification
and validation is given special attention in such programs. Commanders and their instructional
cadre are diligent in observing and reflecting on the developing instructional practices and
outcomes in their programs (see sections 3.2.4-3.2.6). In this context, there is considerable
66
interest in formative measures such as those discussed in this chapter (see also Chapter 2), and
there is increasing interest in measures that more directly relate to outcomes of interest in Full
Spectrum Operations (see Epilogue).
Figure 8. Collaborative design and development for OBTE in a local

command. Solid lines represent direct programmatic influences. Dotted lines
connote informal or indirect programmatic influences. Heavy solid lines
connote direct psychological causes and effects of instruction on students. For
simplicity of depiction, Command Intent & Collaborative Decision-Making
includes all activities involved in planning, executing, and assessing
instruction. Compare with Figure 1 in the Epilogue.
3.3 Toward a Grounded Theory for OBTE
3.3.1 Need for an Integrated Interdisciplinary Framework
As an approach, OBTE is multifaceted and multidisciplinary because it addresses (a) the various
ways that an organization invests in or otherwise influences the development of individuals and
(b) the long-term return on this investment for the individual and the organization. It recognizes
that instruction requires careful consideration of the student, the instructor, and the nested
structure of the instructional institution and the culture it serves (Bransford, et al., 2000; Dewey,
1915/2008; Hmelo-Silver, 2004; James, 1890/1907; Keatinge, 1896; Lewin & Grabbe, 1945;
Magolda, 1999; Riccio, et al., 2004; Wenger, 1998). Effective instruction should have
components that have an impact on all these elements and levels of an instructional system. For
these reasons, OBTE includes interventions that address individual motivation and well-being,
interpersonal interaction and influence, the ecology and development of perception and action,
and reciprocal means-end relations within adaptable organizations.
We use Bransford et al. (2000) as a framework within which to organize and describe the
multidisciplinary considerations implicit in the set of formative measures for OBTE (Chapter 2;
67
Riccio & Darwin
Appendix A). This framework addresses learner-centered, knowledge-centered, and assessmentcentered needs of a learning environment that are grounded in a deep understanding of
community-centered needs (Figure 9). Bransford et al. (2000) argue that learning systems will be
more effective to the extent that they meet these needs.
Figure 9. Exigencies for design of a learning environment that is effective with respect to
human development and far transfer [after Bransford et al., 2000].
We believe that this framework facilitates understanding of the capability gaps to which OBTE is
a response as well as the way in which it addresses those gaps. It is a broad and integrated view of
educational programs that also has implications for program evaluation (see Chapters 1 and 11).
In particular, we believe Bransfords four essential characteristics of a good learning environment
ground the development of measures in a way that helps achieve an integration of Kirkpatricks
four levels of evaluation (D. Kirkpatrick, 1994), integration envisioned by Kirkpatrick but that
has proved to be elusive in most implementations of his system (J. Kirkpatrick & W. Kirkpatrick,
2009). At the same time, a concrete integrated approach to program evaluation helps those
involved in various phases of the ADDIE process understand specifically what they can do to
address the needs of a learning environment as articulated by Bransford et al.
One of the most important results of the methodological crystallization in our inquiry about
OBTE was an appreciation of the actionable implications of principles and practices of OBTE
(Darwin, 2008a, b; Cornell-dEchert, 2009a, b) for a learning environment consistent with the
framework of Bransford et al. The following sections address these opportunities in the learning
environment in OBTE.
3.3.2 Formative Measures of Instructor Behavior as Evolving Best Practices of OBTE
The product of our COMPASS effort with instructional designers and cadre of OBTE yielded a
set of 65 operationally relevant and observable measures (see Chapter 2; Appendix A). These
measures define OBTE at a level sufficient for rigorous verification of instructional practices of
OBTE and, thus, they are useful for both program evaluation and formative feedback to
instructors. It is important to note that the set of 65 measures are intended as a menu from which
68
one can chose a manageably small subset for a particular learning event. The measures codify the
best practices of OBTE that, in essence, are perspectives on how to translate theory into practice.
The measures are not specific to one application, such as marksmanship training, and they can be
employed to assess instructor behavior in virtually any training domain for quality assurance or
improvement. In this respect, there is a value to measures such as these beyond the particular
relevance they have to OBTE. It is critically important in quality assurance to have some method
of verifying what instructors actually are doing. Absent that, it would be difficult if not
impossible to justify conclusions about what to sustain or what to improve based merely on
measures of learning or student performance. Validation is useless without verification (CMMI
Product Team, 2009); dependent variables are problematic without independent variables. A
fundamental premise of OBTE is that instructor-student interactions are the most important factor
affecting the learning and development of Soldiers. An enhanced capability for verification of this
aspect of instruction is the most important contribution of the formative measures developed in
our research on OBTE.
The essential characteristics of a good learning environment for OBTE were used to identify
categories for the measures of instructor behavior that facilitate selection and use of the measures
(see Appendix A). This was accomplished by ranking the principles in terms of their relevance to
each of the 65 measures. The measures were then grouped into categories of five to nine items
based on similarity (cf., Tversky, 1977) with respect to the four characteristics of a good learning
environment. Ten high-level categories emerged including three for planning, six for execution,
and one for collaborative reflection. Collaborative reflection currently has three small
subcategories that we expect to expand as experience is gained with OBTE in a wider range of
applications.
3.4 Emerging Best Practices in OBTE for a Community-Centered Environment
To be community-centered, Bransford, et al. (2000) argue that learning systems must encourage
exploration in an environment that is relevant to and reflects the community in which learning
will be applied. Consistent with this, OBTE strives to emphasize the relationship between what is
learned early in training and education and what will be necessary in order to function
appropriately in Full Spectrum Operations. Also consistent with Bransfords framework, OBTE
strives to create an open and supportive learning environment that encourages the manifestation
and development of leadership. This is important for the community that OBTE serves, given that
these are characteristics of a learning environment that reflects Army Values. We believe that the
characteristics of a community-centered environment relate directly to leadership and
enculturation of Soldiers (see Table 2).
3.4.1 Leadership and Enculturation of Soldiers
What can be done to develop curricula and courses that improve the quality of life-long learning
and development such as in an operational context where significant and critical learning occurs?
One answer to this question follows from the realization that a collective, not an individual, is the
most fundamental unit of analysis in the Army. Task organization and division of labor in the
Army both requires and provides opportunities for momentary leadership that is not necessarily
formal as designated by rank, position or specialty. Leadership and guidance that individuals
provide to each other is not limited to formal learning events. The goal should not necessarily be
to wean Soldier-students from guidance so that they are competent on their own when they leave
formal instruction. It should be to make them more competent in eliciting, utilizing, and
69
Riccio & Darwin
providing guidance within a task-organized unit. Learning events can be designed to develop this
competency in individuals and thus to prepare Soldiers for success in Full Spectrum Operations.
Table 2: Exigencies of the learning environment in OBTE
Leadership and enculturation of Soldiers
Inculcation of Warrior Ethos and Army Values
Instructor is a leader, facilitator, advisor, mentor, and role model
Instructor gives purpose & vision, allows passion & risk, fosters growth
What is taught and how it is taught reflects necessity of Full Spectrum Operations
Community-centered learning environment
Integrated understanding of basic Soldier skills in Full Spectrum Operations
Condition Soldiers to overcome the psychological and physiological effects of combat
Condition Soldiers always to exercise deliberate thought under stress
Demonstrate the linking of tasks in a military situation
Understand relationships between what is taught and why, when, and how it is taught
Knowledge-centered learning environment
Collaborative reflection and problem solving
Training to grow problem solving
Teach Soldiers to learn for themselves within an established framework of knowledge
Teach through contextual understanding of the task such as its mission application
Draw out of the Soldier a critique of performance during the process
Assessment-centered learning environment
Soldier motivation and development of intangibles
Training to develop intangibles such as confidence, initiative, and accountability.
Assist the Soldier to understand the situation and desired result
Assist the Soldier in identifying obstacles to the desired result
Allow the Soldier to work towards a solution within defined principles
Learner-centered learning environment
Given that there is an abundance of Soldiers with recent and relevant experience in Full Spectrum
Operations, how can we determine what kind of experience is useful in the classroom? OBTE
assumes that the most valuable experience is in the development of individuals. The outcomes
emphasized by OBTE are long-term developmental trajectories and milestones (B. Glaser, 2002;
Rick & Loewenstein, 2008; Peterson & Seligman, 2004; Deci & Ryan, 1985, 2008) more
commonly associated with leader development (TRADOC, 2009). In this sense, OBTE
essentially is a leadership-based approach to training, education, and self-development. OBTE is
based on the recognition that Soldiers, who are both confident and competent, are developed
when leaders allow subordinates reasonable autonomy to exercise individual and small-unit
initiative. This requires a climate of accountability that is fostered by mutual trust among
superiors and subordinates (Riccio et al., 2004). It also requires that leaders at every level
recognize their influence as role models (Bandura, 1977, 1997; Lave & Wenger, 1991) by
thinking and acting flexibly based on constant awareness and adjustment to deviations from
optimal or expected conditions in any situation.
An instructional strategy associated with this principle is to create opportunities for Soldiers to
assume responsibility for their actions, and hence, to develop a sense of accountability (Darwin,
2008a, b; Cornell-dEchert, 2009a, b). The priority is to encourage mutual awareness among
peers. Awareness of the consequences of ones behavior with respect to ones peers also leads to
70
a heightened awareness of the behavior of those peers (Bandura, 1977; Lewin, 1951). This can
lead to ad hoc peer coaching that can be a valuable instructional multiplier. For example, consider
what happens when a Soldier is taught the importance of pointing his weapon in a safe direction
at all times. As Soldiers move around during training, they become aware of where their weapon
is pointing, but also where other Soldiers weapons are pointing, so that they do not intentionally
walk into a potential line of fire, and call out other Soldiers for a lack of such awareness. Contrast
this with Soldiers who are told to always point their weapon up and down range when on the
firing line. They follow those instructions to the letter, and ignore the purpose of pointing the
weapon up and down range, for safety, and may perseverate on this rule even in situations when it
results in their weapon pointing at someone who may be standing on an elevated terrain. This
example illustrates that reciprocal influence and accountability need not be a complex
undertaking. The foundations for collective agility and success can be developed early in a
Soldiers training and education.
3.4.2 Robust and Adaptable Plan
There are eight measures in this category (Appendix A). A representative measure is Do the
instructors adapt outcome expectations to account for Soldiers who learn slower/faster than
anticipated? This measure includes the following anchors for novice, intermediate, and expert
behavior respectively: (a) cancels training or maintains focus on specific outcomes for original
task; (b) chooses different outcomes but does not achieve effective training; or (c) chooses
outcomes to achieve training effectiveness, leverages condition as another training opportunity.
Outcomes and training events should focus on the mission relevance of knowledge, skills, and
attitudes rather than strict adherence to prescribed (e.g., expository) details of a certain task. In
OBTE, instructors plan to emphasize effectiveness over efficiency; that is, they focus on progress
with respect to development of the Soldier as a student and development of the student as a
Soldier, not merely task completion or throughput. In OBTE, an instructor is prepared with a
frameworka mindset, intent, and nested objectivesthat helps prioritize an inexhaustible array
of prior considerations and opportunities that may arise during a learning event (Cornell-dEchert,
2009b; Haskins, 2009a, b; see Appendix C).
To ensure that training is effective, instructors should plan to frequently assess how the training is
progressing. During this assessment, instructors can determine if they are achieving their intent
for the day and making changes as necessary. The selection of a manageable number of formative
measures for a particular learning event can facilitate assessments that can be conducted during or
after a learning event (see Appendix A). They provide a theme for a learning event that is robust
to uncontrolled variation in conditions. Planning flexibility is important in dealing with variables
that may be out of the instructors control. A mature instructor should have a contingency plan or
expectation to facilitate in-stride adjustments, for example, to inclement weather or limited
resources. In every situation, the instructor should anticipate leveraging unforeseen conditions as
a training opportunity rather than a constraint. For example, a rainy day can be leveraged to build
Soldier confidence in the capabilities of their weapon to operate in inclement weather.
3.4.3 Instructors as Role Models
There are six measures in this category (Appendix A). A representative measure is Do the
instructors demonstrate openness in changing their training progression? This measure includes
the following anchors for novice, intermediate, and expert behavior respectively: (a) training will
not deviate from a set schedule; (b) instructors determine when it is time to move on to a new
task; or (c) Soldiers have input into the progression of training.
71
Riccio & Darwin
Instructors can monitor whether students are getting the training they need and decide when
students should advance to the next event or revisit a learning objective within the parameters of
established time resources. Instructors, as mentors, exhibit behavior associated with the same
attributes that they are trying to grow in their students throughout training (e.g., confidence,
initiative, accountability). In OBTE, this means that instructors demonstrate that they are open to
deviations from the plan and that they are open to insights and guidance from others such as their
peers in the cadre. A central premise of OBTE is that all instructors are influential as role models,
all the time, whether or not they intend to be role models and irrespective of the extent to which
they are prepared to act as role models (Bandura, 1977, 1997; Lave & Wenger, 1991). Instructors
may have admired status, or otherwise may have high credibility, whether or not they deserve it.
The vastly important implication of this assumption is that instructors may actually do more harm
than good if they are not educated about the potential sources and manifestations of such
influence. The measures for this category are a simple mechanism to help instructors achieve
awareness of their influence and a framework for instructor education.
3.4.4 Collaborative Identification of Outcomes and Measures
There currently are no measures in this category. The need for this category is emphasized by the
importance of collaborative discussion among instructional cadre and their chain of command to
achieve a balance of near-term learning objectives and the long-term developmental outcomes
emphasized by OBTE. Such collaborative decision-making is verifiable and best practices can be
identified (Appendix C; Haskins, 2009a, b; Perry & McEnery, 2009). Development of formative
measures for this activity of an instructional organization is one way to focus attention on a
change initiative and to track its progress within the organization (cf., Chapter 14). Measures of
this kind have not yet been developed for OBTE because addressing organizational change and
associated frictions was not part of the original intent of the investigation. Nevertheless,
organizational change has emerged as a critical element of OBTE.
Interestingly, the process used to develop measures for OBTE (e.g., Chapters 2, 7, and 12) is a
model for how outcomes and measures can be developed collaboratively by an instructional cadre
and their chain of command in a way that is scientifically rigorous and traceable (see section
3.2.7). In principle, one also can observe and measure implementation of this model as a critical
element of organizational behavior and come to a better understanding of the factors affecting its
implementation (cf., Bandura, 1995; Edmonson, 2003; Vaughan, 1997; Rasmussen, 1997).
Formative measures should facilitate post hoc internal or external review of the collaborative
conduct and performance of instructional cadre and their chain of command in identifying: (a)
the psychological and physiological effects and the physical requirements of combat; (b)
outcomes associated with adaptability amid the trans-extremis conditions of Full Spectrum
operations; (c) outcomes associated with inculcation of Warrior Ethos and Army Values; (d)
observable instructor behavior associated with being a leader, facilitator, advisor, mentor, and
role model; and (e) instructional strategies that introduce and control passion and risk.
3.5 Emerging Best Practices in OBTE for a Knowledge-Centered Environment
Knowledge-centered environments foster learning in ways that lead to an integrated
understanding of a discipline and to subsequent transfer (Bransford et al., 2000). There is an
emphasis on a network of connections within learning events, between learning events, and with
both prior and subsequent learning. A student becomes oriented within a curriculum-level
landscape instead of being lost in a disconnected complex of knowledge and skills acquired over
an extended period of time. We believe that the characteristics of a knowledge-centered
72
environment relate most directly to integrated understanding of basic Soldier skills in Full
Spectrum Operations (see Table 2).
3.5.1 Integrated Understanding of Basic Soldier Skills in Full Spectrum Operations
OBTE emphasizes that instruction is most effective when it reflects and responds to the most
urgent requirements of Full Spectrum Operations. These requirements are considered in terms of
basic Soldier skills such as move, shoot, and communicate but at a layer or two deeper than what
those words commonly connote (Flanagan, 2005). These operational requirements can be
addressed at a level that relates to individual adaptability and collective agility and that addresses
the why, what, when, where, and how of actions that can have irrevocable consequences in
operational settings and missions (Darwin, 2008a, b). Thus, they can be addressed at a level that
benefits all Soldiers and their units.
Many of these requirements relate to stress, psychological and physiological effects of stress,
performance given these effects, and strategies that are resistant to stress (Lerner & Keltner,
2001; Lerner & Tiedens, 2006; Lukey & Tepe, 2008). Experience with these relationships and
effects across disparate learning events helps a Soldier overcome stress by avoiding distraction,
maintaining an outward orientation, and always exercising deliberate thought (Darwin, 2008a,b;
cf., Kolditz, 2007; Lukey & Tepe, 2008). It helps students make sense of the relationships
between what is taught and why, when, and how it is taught. It provides them with a path toward
ever deepening understanding of their capabilities in context (Hmelo-Silver, 2004; Magolda,
1999; Riccio et al., 2004).
An instructional strategy within the framework of OBTE is for instructors to communicate with
Soldiers and guide them to discover the purpose and intent of the training. Purpose generally
illuminates a broader context within which tasks are interrelated and the role of an individual is
related to the mission of the unit (J. Gibson, 1977; Lewin, 1951). A focus on purpose reveals the
relevance of the instructional task to an operational mission. More deeply, it reveals the aspects of
the surroundings about which a Soldier should be aware in a collective task, whether in training
or in an operational mission. This lays the foundation for education of the attention, habitual
orientation to the surroundings in any and all situations, and an integrated understanding of a
situation (Endsley, 1995; E. Gibson, 1988; J. Gibson, 1977).
3.5.2 Task Relevance of Planned Instructional Events
There are five measures in this category (Appendix A). A representative measure is Do the
instructors build time for this discussion [about tactical relevance of the task] into the schedule?
This measure includes the following anchors for novice, intermediate, and expert behavior
respectively: (a) does not build in time for discussion into the schedule; (b) builds in time for
short discussions after the day is complete; or (c) builds in time for short discussions between
tasks.
It cannot be assumed that the students will understand why a training exercise is relevant to their
mission. To ensure a knowledge-centered environment, it is important to design exercises that aid
the student in making these connections. For example, understanding the design of a learning
event and the reasons behind it will contextualize students learning, and presumably lead to
better skill retention and transfer to the operational environment. A learning environment that is
both community-centered and knowledge-centered should reflect and develop values of the Army
culture (see Chapters 1-5 and the Epilogue; Riccio et al., 2004). The training environment does
not have to look like the operational environment to develop values, basic Soldier skills, and
73
Riccio & Darwin
resilience to stress. Operational fidelity of a training event can be considered, instead, in the
context of the conditions that foster development of intangible attributes that are critically
important in an operational context or mission.
In OBTE, instructors focus on setting up conditions that foster fidelity of interpersonal
interactions and task linkage that, in turn, provide students with an appreciation of the context in
which knowledge, skills, and attitudes make a difference (cf., Neisser, 1976; Shaw & Bransford,
1977; Winograd, Fivush, & Hirst, 1999). In this approach, variation in conditions is an enabler for
the development of competencies that can transfer to ambiguous or changing environments. The
most robust competencies are those defined in terms of the relations or interactions between the
individual and the environment (Bandura, 1977; E. Gibson, 1988; J. Gibson, 1977; Lave &
Wenger, 1991; Lewin, 1951).
3.5.3 Reveal Operational Relevance of Training
There are seven measures in this category (Appendix A). A representative measure is Do the
instructors group tasks into collective behavior? This measure includes the following anchors for
novice, intermediate, and expert behavior respectively: (a) does not group tasks into collective
behavior, results in incorrect performance of linked tasks; (b) does not group tasks into collective
behavior (no observable negative consequences); or (c) groups task in a way that simulates the
combat application and reinforces correct performance of linked tasks.
OBTE seeks to lay the foundation for training, education, and self-development (Darwin, 2008a,
b). Soldiers, for example, should come to understand the reasons behind a training event,
especially if its operational relevance is not readily apparent. The purpose is not for the instructor
to seek the approval or buy-in of students. Instead, it is to help students understand that the
operational relevance of learning events is not necessarily manifested in tangible ways or through
superficial similarities to an operational scenario. In addition to the interpersonal interactions and
task linkages mentioned above in the context of planning for operationally relevant instruction,
students should gain experience in collaborative problem solving and the collective agility it
enables. They should gain experience dealing with the unknown and the learning in real time that
it necessitates. They should be given a rich variety of experiences that gives them practice in
making connections among disparate events and in extrapolating to new situations (Tobias &
Duffy, 2009; Van Merrienboer & Kirschner, 2007). Finally, they should understand that, in these
ways, training is like the operational environment.
3.5.4 Incorporate Stress into Instructional Events
There are six measures in this category (Appendix A). A representative measure is Do the
instructors encourage deliberate thought in stressful situations? This measure includes the
following anchors for novice, intermediate, and expert behavior respectively: (a) rushes or
encourages Soldier to complete the task without deliberate thought, focuses on speed; (b)
encourages Soldier to think but fails to establish effective training conditions to reinforce
thinking; or (c) focuses Soldier on thinking through situation regardless of the stress level,
focuses on accountability.
As indicated above, stress is not inconsistent with a positive learning environment if the source
and manifestations of stress are representative of demanding operational environments such as
combat (see, e.g., Lukey & Tepe, 2008). Complexity introduced through scenarios is one way to
create stress that is relevant and that potentially allows Soldiers to develop strategies that reduce
stress or mitigate the effects of stress. Training scenarios need not look exactly like operational
74
scenarios to be useful in this way. In general, scenarios can reveal complexity in linkage among
tasks, for example, and the related implications for interdependence of individuals in a taskorganized unit. Systematic manipulation of such complexity has instructional utility because it is
relevant to the understanding and capabilities for action that Soldiers acquire during training.
Experience and development of self-efficacy in such situations helps Soldiers build habits of
outward orientation amid a growing repertoire of personal capabilities. OBTE assumes that an
understanding of what one can do, ones self-efficacy, in various situations can help one avoid
distractions from the physiological or psychological effects of stress (cf., Kolditz, 2007; Lerner &
Keltner, 2001; Lerner & Tiedens, 2006; Lukey & Tepe, 2008).
3.5.5 Identify General Lessons Learned and Extrapolate to New Situations
There currently are two measures in this evolving category (Appendix A). A representative
measure is Does the instructor ensure Soldiers can articulate how to apply concepts to new
situations? This measure includes the following anchors for novice, intermediate, and expert
behavior respectively: (a) discussion focuses on the specific task only, does not discuss combat or
mission application; (b) facilitates a discussion of the specific tasks and how they apply to
missions; or (c) facilitates a discussion of the intangible attributes underling the task and how it
applies to ambiguous combat/mission situations.
This category of measures, generally utilized after a learning event, addresses the role of intersubjectivity and collaboration in learning and development. It is relevant to acquisition of
knowledge and skill but at multiple levels. Thus it is involves refection on more than the
knowledge or skill per se. Some of the measures also should reveal the development of
metacognitive understanding about how the knowledge or skill is utilized in particular situations
and how this is influenced by the situation. An important consideration in further development of
these measures should be to help Soldiers reflect on the role of emotion in thinking and
performance (Lerner & Keltner, 2001; Lerner & Tiedens, 2006; Lukey & Tepe, 2008). The
assumption is that such reflection will develop metacognitive awareness can help Soldiers (a)
exercise deliberate thought under stress; (b) be better prepared for transitions into, out of, and
between stressful situations characteristic of Full Spectrum Operations; and (c) learn lessons from
their experience in particular stressful situations and trans-extremis conditions that can be applied
to different situations.
3.6 Emerging Best Practices in OBTE for an Assessment-Centered Environment
Assessment-centered environments emphasize formative feedback that promotes understanding
(Bransford et al., 2000). Opportunities for feedback should be ubiquitous and should be driven by
noteworthy events in individual or group activities. Feedback should help a student make
connections between current learning events and other parts of the curriculum and the students
life. It should help students appreciate and become more skillful at self-assessment and peer
assessment (Bransford et al., 2000). Consistent with this characteristic of a good learning
environment, OBTE extensively utilizes after-action reviews and ad hoc instructor-student
interactions to draw out of the Soldiers a critique of their own performance throughout the
learning cycle. We believe that the characteristics of an assessment-centered environment relate
most directly to collaborative reflection and problem solving (see Table 2).
75
Riccio & Darwin
3.6.1 Collaborative Reflection and Problem Solving

OBTE seeks to establish a mindset of collaborative reflection and problem solving in which there
is continuous vigilance for noteworthy events and lessons learned worth discussing among
instructors, students, and peers. Development of mastery in assessing oneself and ones unit is a
key outcome for OBTE. It is important to understand that collaborative reflection is a ubiquitous
opportunity that exists in any program of instruction. It can be done on an as-needed basis when
the lessons learned justify the investment of time and resources. It can exploit naturally occurring
events and utilize whatever method is appropriate at the time, such as after action reviews, hot
washes, and informal conversation. This does not mean that constant discussion is a goal of
OBTE or that discussion itself is a reflection that OBTE is occurring. It is as important to decide
not to engage in collaborative reflection if there has not been a noteworthy event or if there is a
more valuable use of time for learning. This mindset, thus, also continually reinforces the
importance of prioritization and tradeoffs required in most military scenarios (Riccio et al., 2004).
If a decision is made to engage in collaborative reflection (e.g., to execute a planned after-action
review), the discussion should be conducted in an open and supportive manner regarding lessons
learned. The purpose is not to summarize merely what happened in the learning event but to
facilitate a discussion that allows students to evaluate their own performance in terms of the
consequences of their actions for the rest of the team and with respect to other linked tasks
(Neisser, 1996; Neisser & Jopling, 1997). Collective self-examination leads to an understanding
in which the whole is greater than the sum of the parts (cf., Barnes, 1959; Cotkin, 2003; Solomon,
2004).
A core strategy of OBTE is to guide Soldiers to the discovery of solutions, rather than merely
telling them what to do (e.g., telling the solution prior to introduction of the problem) (Darwin,
2008a, b; Savory & Duffy, 1995; Vandergriff, 2006, 2007; see also Tobias & Duffy, 2009). This
prepares Soldiers to take initiative in addressing unique problems that may be faced in a mission
environment. OBTE recognizes that training design cannot predict all of the problems that
Soldiers will face. Therefore, rather than try to condition a repertoire of scripted responses, OBTE
attempts to develop skills that can support adaptive behavior even in unforeseen or novel
environments. One of these skill sets is the ability to make decisions about collaborative
reflection and to engage effectively in it. This problem solving approach helps to develop a sense
of initiative that is grounded in awareness of contextual relationships and vigilance about the
attendant consequences of ones own behavior (J. Gibson, 1977; Shaw & Kinsella-Shaw, 2007).
Guidance provided by instructors includes stimulating and facilitating awareness of this dynamic
context rather than simply telling the Soldier about the best course of action given these
considerations.
3.6.2 Communication
There currently are six measures in this evolving category (Appendix A). A representative
measure is Does the instructor encourage inter-trainee communication and discussion? This
measure includes the following anchors for novice, intermediate, and expert behavior
respectively: (a) Soldiers do not have the opportunity to discuss; (b) training design provides
opportunities for unguided inter-trainee communication; or (c) instructor facilitates discussions
and encourages follow-on inter-trainee discussion, training design provides opportunities.
This category includes measures that initially were developed outside of the COMPASS process
using an approach employed in previous research on adaptive communications analysis (e.g.,
Entin, Diedrich, & Rubineau, 2003). The purpose of these measures was to conduct a quantitative
76
assessment of communication between instructors, students, and peer coaches that is emphasized
in OBTE (see Chapter 2). Some of the measures (e.g., guided learning or self-discovery and
constructive or diagnostic communication) also can be useful in formative assessment; thus, we
expect these measures to be a priority in continuing development of the comprehensive set of
measures for OBTE.
3.6.3 Nature and Extent of Guidance
There are seven measures in this category (Appendix A). A representative measure of how an
instructor might gauge the need for guidance to a Soldier is Does the instructor recognize when a
Soldier is too withdrawn/distracted to effectively participate in training? This measure includes
the following anchors for novice, intermediate, and expert behavior respectively: (a) does not
recognize a deeper individual problem when Soldier is struggling with a task; (b) sees that there is
a deeper problem, but doesnt do anything (doesnt realize he can do something); or (c)
recognizes there is a deeper problem, takes appropriate action to help Soldier get back in the
game.
OBTE occurs when instructors make principled decisions about when and how to adapt the
instructional environment to achieve a positive outcome. Such adaptation includes utilization of
whatever instructional methods are appropriate for the problem at hand, whether that is direct
instruction, problem-centered instruction, or some other method (see e.g., Hmelo-Silver et al.,
2007; Tobias & Duffy, 2009; Van Merrienboer & Kirschner, 2007). The goal is to provide
learners with the right amount of guidance at the right time to achieve both the learning objective
for the event and longer-term developmental outcomes such as the growth of confidence,
initiative and accountability.
3.6.4 Establish a Pervasive Mindset of Collaborative Reflection
There currently are two measures in this evolving category (Appendix A). A representative
measure is Does instructor conduct AAR or hot-washes as needed? This measure includes the
following anchors for novice, intermediate, and expert behavior respectively: (a) does not conduct
AAR or hot-wash, or conducts regardless of need; (b) Tells or lectures Soldiers the lessons
learned in a timely manner; or (c) facilitates discussion on lessons learned amongst the Soldiers in
a timely manner.
This category of measures, generally utilized after a learning event, addresses the practices of
inter-subjectivity and collaboration in learning and development. Further development of these
measures will parallel the development of measures outlined in section 3.6.2 that generally are
utilized during a learning event. The difference is that these measures should foster collaborative
metacognitive reflection. Engaging others in reflection, after joint participation in an event, on the
characteristics and effectiveness on communication and collaboration that had occurred during
the event can help metacognitive reflection from becoming an infinite regress into the abstract. It
keeps it inter-subjectively valid and grounded in a shared reality. The prior intent and
interpretations of other participants can be made explicit, thus, the interpersonal causes for
miscommunication and misinterpretation are more readily revealed. It is important to know
whether problems were due to differences in prior experience, prior knowledge, and associated
assumptions or whether they were due to personality characteristics or even momentary
conditions (e.g., stressors). Differentiating these potential causes, and identifying the critical ones,
can improve collaboration and performance of the group but it also potentially can improve an
individuals understanding and awareness of interpersonal factors in different groups.
77
Riccio & Darwin

3.7 Emerging Best Practices in OBTE for a Learner-Centered Environment
To be learner-centered, an educational system must address what learners know, what they do not
know, how they learn, and what they are motivated to learn (Bransford et al., 2000). In OBTE,
the strategy is to provide learners with right amount of guidance at the right time to achieve both
learning and development. As a mentor and facilitator, the instructor assists the Soldier to
understand the situation, the desired result, and the obstacles to the desired result. The instructor
creates an environment that encourages initiative, a broad appreciation of accountability, and the
development of confidence that is grounded in demonstrable competence. We believe that the
characteristics of a learner-centered environment relate most directly to Soldier motivation and
development of intangibles (see Table 2).
3.7.1 Soldier Motivation and Development of Intangibles
OBTE strives to provide learners with guidance that helps them achieve both the learning
objective for the event and longer-term developmental outcomes such as the growth of
confidence, initiative and accountability (Deci & Ryan, 1985; Guay, Ratelle, & Chanal, 2008;
Peterson & Seligman, 2004). The intent is to leverage internal motivation and to internalize
sources of motivation by balancing initiative with a sense of accountability based on discipline
and awareness of ones relationship to others, and by building confidence based on enhanced selfefficacy and commitment to the pursuit of mastery (Bandura, 1997; Deci & Ryan, 2008; Ryan &
Deci, 2008).
instructional environment to achieve a positive outcome for particular students in particular
situations, that is, when they exemplify leadership. Such adaptation includes utilization of
whatever instructional methods are appropriate for the problem at hand (Tobias & Duffy, 2009).
The ways in which OBTE can benefit from methods motivated by direct instruction (e.g.,
Kirschner, Sweller, & Clark, 2006), constructivist instruction (e.g., Hmelo-Silver, Duncan, &
Chinn, 2007), and experiential instruction (e.g., Hickman & Alexander, 1998) remain to be seen
but the exigencies of OBTE can inform decisions about the nature and extent of guidance
provided to students or instructors in any of these methods. For example, even in situations in
which students do a "worked example" or solve a well-defined problem given by the instructor, it
is beneficial for the student to understand that the assumptions of the given problem are valid and
why the assumptions are necessary, even if they are not ready to be thrown into a situation in
which assumptions are violated. This allows them to become ready, to become prepared to deal
with such a situation. The foundation for collective agility is thus reinforced in all training and
educational situations, even ones that ostensibly do not require individual adaptability.
A key strategy for the continuing development of intangibles is an emphasis on mastery (Darwin,
2008a, b). From OBTE perspective, pursuit of mastery is essential for confidence that is grounded
in the development of competence. In addition, mastery enables Soldiers to see beyond the
instructional situation and beyond the associated standards as end states. Competence as a pursuit
should motivate an interest in further learning and in obtaining enhanced depth or breadth of
competence. An instructional consideration in this strategy is setting up conditions in which
Soldiers are allowed to experience improvement amid awareness that they can do better (CornelldEchert, 2009a, b; Bandura, 1997; Cole, John-Steiner, Scribner, & Souberman, 1978). For each
of these strategies, the cornerstone of OBTE is for the instructor to set the conditions that create
opportunities for learning (e.g., Dewey, 1915/2008; Bransford et al., 2000; Tobias & Duffy,
2009).
78
3.7.2 Plan for Development of the Individual

There are nine measures in this category (Appendix A). A representative measure is Is the
training designed to develop intangible attributes (Judgment, Adaptability, Accountability,
Problem Solving, Confidence, Initiative, Awareness, Thinking Skills)? This measure includes
the following anchors for novice, intermediate, and expert behavior respectively: (a) focus of
training is solely to accomplish the task; (b) instructor discusses intangibles directly related to
task being trained, does not focus plan on development; or (c) instructor discusses intangibles
directly related to the task being trained, plan focuses on development.
Instructors should prepare training that develops Soldier understanding of a concept, and supports
adaptive thinking and problem solving that is robust to changes in conditions. Instructors should
not be the proximate source of stress through techniques such as intimidation and belittlement
that have a counterproductive effect on Soldier confidence, initiative, and accountability. This
does not imply that instructors should coddle Soldiers. There are times for a strident authoritarian
style but it should be used judiciously. To utilize stress in training, instructors should prepare
challenges that are more likely to carry over to realities of the operating environment. This frees
the instructor to be a source of guidance and insight about how to manage stress (Darwin, 2008a,
b; Cornell-dEchert, 2009a, b).
Students learn best in a relatively comfortable, low stress environment, but this does not ensure
that they will learn to deal with stress. To address this tradeoff, in OBTE, instructors plan for
systematic increases in stress across learning events (Darwin, 2008a, b; cf., Hmelo-Silver et al.,
2007; Van Merrienboer & Kirschner, 2007). Stress does not necessarily have negative
connotations in this context. One source of increasing stress or challenge is to provide students
with opportunities to take ownership of their own learning. This is a powerful motivator and
source of initiative (Deci & Ryan, 2008; Ryan & Deci, 2008) but it also requires increased
discipline, awareness, and accountability that bounds ones own initiative and ensures that it is
suitably robust to the inevitable mistakes that are critical in learning to learn.
Time should be built into the schedule for students to engage in self-discovery and collaborative
reflection. Though it may be quicker to simply tell a student what the answer is, it is operationally
more relevant if they are provided the tools to arrive at the solution themselves (cf., HmeloSilver, 2004; Tobias & Duffy, 2009). This helps students acquire the skills and internal sources of
motivation that enable self-development in and outside of formal learning environments (Darwin,
2008a, b; dEchert, 2009a, b; Guay, Ratelle, & Chanal, 2008).
3.7.3 Get Students to Take Ownership
There are eight measures in this category (Appendix A). A representative measure is Does the
instructor foster self-development amongst the Soldiers? This measure includes the following
anchors for novice, intermediate, and expert behavior respectively: (a) tells them how they did
without giving Soldiers a chance to reflect; (b) asks the Soldiers how they think they did, tells
them what they did wrong; or (c) asks the Soldiers how they think they did, facilitates selfdiscovery of mistakes through targeted questioning.
A balance of power and leadership is critical in OBTE. Achieving this balance is complicated for
many reasons, not the least of which is that students may be unfamiliar with this instructional
style. The hurdle can be overcome more easily if the instructor acts as an advisor and mentor with
a clear interest in the students development. In OBTE, instructors allow passion and measured
risk (Darwin, 2008a, b). They encourage the development of individual initiative that is
79
Riccio & Darwin
constrained by the collective accountability implicit in interdependence and by confidence

grounded in demonstrable competence. Thus, they foster self-determination in Soldiers that is
consistent with military order.
3.7.4 Collaborative Reflection as a Means to Develop Self Efficacy
There currently are five measures in this evolving category (Appendix A). A representative
measure is Does the instructor ask Soldiers to assess their own performance? This measure
includes the following anchors for novice, intermediate, and expert behavior respectively: (a)
does not ask Soldiers to assess their own performance or is overly negative in assessment; (b)
asks Soldiers to assess their own performance but does not recognize accurate self-assessments;
or (c) asks Soldiers to assess their own performance but monitors to ensure they are realistic.
This category of measures, generally utilized after a learning event, addresses the effects of intersubjectivity and collaboration in learning and development. The important concept of selfefficacy, ones perception of ones own capabilities, is most noteworthy when there is an apparent
discrepancy (Bandura, 1977, 1997). Inter-subjective influences can either increase or decrease
discrepancies. Collaborative reflection is useful in this regard to the extent that others are willing
and able to help inform one about ones own capabilities, that is, to foster the veridical perception
that is critical in engagements with the real world. Such guidance from more astute observers,
from less biased observers, or simply from a different perspective, is even more critical amid the
uncertainty of inferences about ones zone of proximal development (cf., Cole et al., 1978). The
acumen or perspective of others can help one identify paths in the pursuit of mastery that may not
be obvious or that may be obscured by habits, assumptions, or unknowns about strategies for
cognition and action. Collaborative reflection in groups fosters intersubjective convergence on
reality (cf., Chapters 4 and 5).
Collaborative reflection and inter-subjectivity also are central in the development of values
implicit in the balance of choice and responsibility (cf., Aristotle, trans. 1925; Augustine, trans.
1991; Hoffman et al., 2009; Maslow, 1964; Neisser & Jopling, 1997; Solomon, 2004). Others can
help reveal constraints of shared values on ones cognition and action, they can help reveal nonobvious ways in which ones cognition and action is situated in a shared context, and the impact
of the mere existence of others in a shared context on ones cognition and action. It both directs
and educates the attention. In the context of the group, which is the fundamental unit of analysis
in the Army, inter-subjectivity is a source of coupling among individuals (i.e., their respective
cognition and action) that renders the notion of individual adaptability almost irrelevant.
Individual adaptability is of consequence and is admissible only in the context of collective
agility, in a context that bounds unplanned activity and that gives it purpose.
The principles and practices of OBTE, summarized in this chapter, provide basic facts from
which to build an appropriate scientific foundation for OBTE. Such scientific foundations are
explored in the Chapters 4 and 5.
3.8 References
Allik, J., & McCrae, R. R. (2004). Towards a geography of personality traits: Patterns of profiles
across 36 cultures. Journal of Cross-Cultural Psychology, 35, 13-28.
Augustine (1953). The teacher. In: J. Burleigh (Trans. & Ed). Augustine: earlier writings.
Philadelphia, PA: The Westminster Press. (Original work written ca. 389)
Augustine (1991). Confessions. H. Chadwick (Trans). New York: Oxford University Press.
(Original work written ca. 400)
80
Aristotle (1977). Nicomachean ethics. In: R.M. Hutchins et al. (Eds.). Aristotle II. Chicago, IL:
Encyclopedia Britannica. (Reprinted from The Works of Aristotle, by W.D. Ross Ed. &
Trans., 1925, Oxford, UK: Oxford University.) (Original work written ca. 350 BCE)
Aspinwall, L. & Staudinger, U. (Eds.) (2003). A psychology of human strengths: Fundamental
questions and future directions for a positive psychology. Washington, DC: American
Psychological Association.
University Press.
Company.
Barnes, H. (1959). Humanistic existentialism: The literature of possibility. Lincoln, NE:
University of Nebraska.
Burke, W. (1985). Applied differential geometry. Cambridge, UK: Cambridge University.
Camber, C., Loewenstein, G., & Prelec, D. (2005). Neuroeconomics: How neuroscience can
inform economics. Journal of Economic Literature, 43(1), 9-64.
Expanding perspectives in methodology and design. Washington DC: APA.
Carlson, S. (2005). The net generation goes to college. The Chronicle of Higher Education,
Volume 52 (7), A34.
Clausewitz, C. von (1976). On war. In: M. Howard. & P. Paret (Trans. and Eds.). On war.
Princeton: NJ: Princeton University. (Original work written 1832)
development f higher psychological processes (L.S. Vygotsky). Cambridge, MA: Harvard
University Press.
Cotkin, G. (2003). American existentialism. Baltimore, MD: Johns Hopkins University.
Cresswell, J. (1998). Qualitative inquiry and research design: Choosing among five traditions.
Group.
Group.
Deci, E. & Ryan, R. (1985). Intrinsic motivation and self-determination in human behavior. New
York: Plenum.
Deci. E. & Ryan, R. (2008). Self determination theory: a macrotheory of human motivation,
81
Riccio & Darwin
CA: Sage.
De Raad, B., & Perugini, M. (2002). Big five assessment. Gottingen, NL: Hogrefe & Huber.
Dewey, J. (2008). The school and society, the child and the curriculum. Chicago, IL: Centennial
Publications of The University of Chicago Press. (Original work published 1915)
Ellingson, L. (2009). Engaging crystallization in qualitative research: An introduction. Los
Angeles, CA: Sage.
Factors, 37, 32-64.
Entin, E. E., Diedrich, F. J. & Rubineau, B. (2003). Adaptive communication patterns in different
organizational structures. Proceedings of the Human Factors and Ergonomics Society Annual
Meeting, Denver, CO.
Filatov, N.M. & Unbehauen, H. (2004). Adaptive dual control: Theory and applications. New
York: Springer.
Flanagan, S. (2005). Combat skills. Unpublished manuscript (Nov. 16, 2005).
Review, 62, 32-51.
Glass, L. & Mackey, M. (1988). From clocks to chaos: The rhythms of life. Princeton, NJ:
Princeton University.
Goldberg, S. (1986). Introduction to difference equations: With illustrative examples from
economics, psychology, and sociology.
Haskins, C. (2009a, March). Development of outcomes based training. Presentation at the US
Haskins, C. (2009b). Outcomes-based training and education. White Paper solicited by
TRADOC Headquarters, August, 2009.
Headquarters Department of the Army (2003). Mission command: Command and control of
Army forces. FM 6-0. Washington, DC: Department of the Army.
Headquarters Department of the Army (2005). The Army. FM 1. Washington, DC: Department of
the Army.
Heine, S. J. & Norenzayan, A. (2006). Toward a psychological science for a cultural species.
Perspectives on Psychological Science, 1(3), 251-269.
Heisenberg, W. (1958). Physics and philosophy: The revolution in modern science. New York:
Harper.
Hickman, L.A. & Alexander, T.M. (1998). The essential Dewey, Volume 1: pragmatism,
education, and democracy. Bloomington, IN: University of Indiana Press.
82
Hmelo-Silver, C. (2004). Problem-based learning: What and how do students learn? Educational
Psychology Review, Vol. 16(3), 235-266.
Hmelo-Silver, C. Duncan, R.G., & Chinn, C. (2007). Scaffolding and Achievement in ProblemBased and Inquiry Learning: A Response to Kirschner, Sweller, and Clark (2006).
Educational Psychologist, 42(2), 99107.
Hoffman, L., Yang, M., Kaklauskas, F., Chan. (2009). Existential psychology East-West.
University of. Rockies Press. Colorado Spring.
Hofstede, G. J., Pedersen, P., & Hofstede, G. (2002). Exploring culture: exercises, stories, and
synthetic cultures. Boston, MA: Intercultural Press.
Howe, N., Strauss, W., & Matson. R.J. (2000). Millennials rising: the next great generation. New
York, NY: Vintage.
Idel, M., & McGinn, B. (Eds.) (1999). Mystical union in Judaism, Christianity, and Islam: An
ecumenical dialogue. New York: Continuum.
James, W. (1890). Principles of psychology. New York, NY: Henry Holt and Company.
James, W. (1907). Talks to teachers on psychology: and to students on some of life's ideals. New
York, NY: Henry Holt and Company. (Original work published 1899)
James, W. (1982). The varieties of religious experience: A study in human nature. New York:
Penguin. (Original work published 1902)
Jarvis, P. (2004). Adult education and lifelong learning: Theory and practice. London, UK:
Routledge-Falmer.
Keatinge, M. (1896). The great didactic of John Amos Comenius. Whitefish, MT: Kessinger
Publishing.
Keyes, C. & Haidt, J. (2003). Flourishing: Positive psychology and the life well-lived.
Washington, DC: American Psychological Association.
Kierkegaard, Soren (1843) under the pseudonym Silentio, Johannes. Fear and Trembling, edited
by Alastair Hannay. (1985): Penguin Classics
Publishers, Inc.
Kirkpatrick, J., & Kirkpatrick, W. (2009). The Kirkpatrick four levels: A fresh look after 50 years,
1959 2009. Retrieved May, 2009, from http://kirkpatrickpartners.com.
Kirschner, Sweller, and Clark (2006). Why minimal guidance during instruction does not work:
An analysis of the failure of constructivist, discovery, problem-based, experiential, and
inquiry based teaching. Educational Psychologist, 41, 75-86.
Knowles, M., Holton, E., Swanson, R. (2005). The adult learner. Burlington, MA: Elsevier.
Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development.
New Jersey: Prentice-Hall.
Kolditz, T. (2007). In extremis leadership: Leading as if your life depended on it. New York:
Wiley.
Lanczos, C. (1970). The variational principles of mechanics. New York: Dover.
Psychology, 81(1), 146-159.
Lvi-Valensi, J. (Ed.) (2006). Camus at combat: Writing 1944-1947. Princeton, NJ: Princeton
University. (Original work written 1944-1947).
Lewin, K. (1951). Field theory in the social sciences. New York, NY: Harper.
Lewin, K & Grabbe, P. (1945). Conduct, knowledge, and acceptance of new values. The Journal
of Social Issues, 1 (3), 53-64.
83
Riccio & Darwin
Loewenstein, G., & Lerner, J.S. (2002). The role of affect in decision making. In: R. Davidson,
K. Scherer, & H. Goldsmith (Eds.), Handbook of affective science (pp. 619642). New York:
Oxford University Press.
Loewenstein, G., Weber, E., Hsee, C., & Welch, N. (2001). Risk as feelings. Psychological
Bulletin, 127 (2), 267286.
Lorentz, H., Einstein, A., Minkowski, H., & Weyl, H. (1952). The principle of relativity: A
collection of papers on the special and general theory of relativity. W. Perrett & G. Jefferey
(Trans.). New York: Dover. (Original work published 1923)
Press.
Magolda, M.B.B. (1999). Creating contexts for learning and self-authorship: constructivedevelopmental pedagogy. Nashville, TN: Vanderbilt Press.
Maslow, A. (1964). Religions, values, and peak-experiences. New York: Penguin.
Msse, L. C., Moser, R. P., Stokols, D., Taylor, B. K., Marcus, S. E., Morgan, G. D., Hall, K.L.,
Croyle, R.T., Trochim, W. (2008). Measuring Collaboration and Transdisciplinary
Integration in Team Science. American Journal of Preventive Medicine, 35(2, Supplement 1),
S151-S160.
Matsumoto, D., & Yoo, S. H. (2006). Toward a new generation of cross-cultural research.
Mikroyannidis, A. (2007). Toward a social semantic web. Computer, November 2007, 113-115.
Los Alamitos, CA: IEEE Computer Society.
Miles, M.B. & Huberman, A.M. (1994). Qualitative data analysis. Thousand Oaks, CA: Sage.
Neisser, U. (1976). Cognition and reality. San Francisco: W.H. Freeman.
University Press.
Paavola, S. & Hakkarainen, K. (2005). The knowledge creation metaphor An emergent
epistemological approach to learning. Science and Education, 14(6), 535-557.
Peterson, C., & Seligman, E. P. (2004). Character, strength, and virtues. Oxford, UK: Oxford
University Press.
Plato (1977a). Phaedo. In: R.M. Hutchins et al. (Eds.). Plato. Chicago, IL: Encyclopedia
Britannica. (Reprinted from The Dialogues of Plato, by B. Jowett, Trans., 1892a, Oxford,
UK: Oxford University.) (Original work written ca. 360 BCE).
Science, 27, 183-213.
Riccio, G. (1993a). Information in movement variability about the qualitative dynamics of
posture and orientation. In: K. Newell (Ed.). Variability and Motor Control (pp. 317-357),
Riccio, G. (1993b). Multimodal perception and multicriterion control of nested systems: Self
motion in real and virtual environments. (UIUC-BIHPP-93-02). University of Illinois at
Urbana-Champaign: Beckman Institute for Advanced Science & Technology (also in NASA
Technical Paper series 3703).
Riccio, G., & McDonald, P. (1998). Characteristics of EVA mass handling skill. Society of
Riccio, G., & Stoffregen, T. (1991). An ecological theory of motion sickness and postural
instability. Ecological Psychology, 3, 195-240.
84
Rick, S. and Loewenstein, G. (2008). Intangibility in intertemporal choice. Philosophical
Transactions of the Royal Society B, 363, 38133824.
Ryan, R. & Deci. E. (2008). From ego depletion to vitality: Theory and findings concerning the
facilitation of energy available to the self. Social and Personality Psychology Compass 2(2),
702717.
Savory, J.R. & Duffy, T.M. (1995). Problem based learning: An instructional model and its
constructivist framework Educational Technology, 35, 31-38.
Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology.
In K. Sawyer (Ed.), Cambridge Handbook of the Learning Sciences (pp. 97-118). New York:
Cambridge University Press.
Shaw, R., & Kinsella-Shaw, J. (2007). The survival value of informed awareness. Journal of
Consciousness Studies, 14(1), 137-154.
Sidman, J., Riccio, G., Semmens, R., Geyer, A., Dean, C., & Frederick Diedrich, F. (2009).
Reshaping Army institutional training: Current training. Final Report to the Army Research
Institute for the Behavior and Social Sciences, contract number W74V8H-04-D-0047 DO
0010.
Solomon, R. (Ed.) (2004). Existentialism. Oxford, UK: Oxford University Press.
Spady, W. (1994). Outcome based education: Critical issues and answers. Arlington, VA:
American Association of School Administrators.
Spady, W. (2009). Outcome based education. Florence, KY: Cengage Learning.
Spady, W., & Marshall, K. (1991). Beyond traditional outcome-based education. Educational
Leadership 49 (2):6772.
Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and
quantitative approaches. Thousand Oaks, CA: Sage.
Routledge.
Trochim, W. (2004). Concept mapping. In Matheson, S. (Ed.) Encyclopedia of Evaluation.
Trochim. W., Marcus, S.E., Msse, L.C., Moser, R.P., Weld, P. (2008). The evaluation of large
research initiatives: A participatory integrative mixed-methods approach, American Journal
of Evaluation, 29, 1, 8-28.
Tversky, A. (1977). Features of similarity. Psychological Review, 84(4), 327-352.
United States Army Training and Doctrine Command (2009). Army Leader Development Strategy
(Draft, May 21, 2009), Ft. Monroe, VA: Training and Doctrine Command.
November 2006, 59-66.
Vaughan, D. (1997). The Challenger Launch Decision: Risky Technology, Culture, and Deviance
at NASA. Chicago, IL: University Of Chicago Press.
Webster, D. S. (2001). In defense of the 'Natural Attitude': charting a return from the madness of
interpretation. Assemblage 6, Available at:
http://www.shef.ac.uk/~assem/issue6/webster_web.html
85
Riccio & Darwin
Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. New York :
Cambridge University.
Wise, G. (1980). American historical explanations: A strategy for grounded inquiry.
Minneapolis, MN: University of Minnesota.
Wolfram, S. (2002). A new kind of science. Champaign, IL: Wolfram Media.
Values-Based Training & Education
86
Chapter 4. Grounded Theory for Values-Based Training & Education

Gary Riccio
In the Operational Army, the relevant driving factors for OBTE are information abundance,
rapidly changing environments, individual adaptability and collective agility demanded by
changing roles in trans-extremis conditions, task organization with individuals and units from
unfamiliar organizations and cultures, decentralized decision-making and action, and the potential
strategic importance of individual action irrespective of rank (French, 2002; Kilcullen, 2006;
Meigs, 2003; Petreaus, 2006). We have concluded that this kind of complexity demands a more
integrated approach to training and education with reworked scientific underpinnings (Lakkala,
Muukkonen, Paavola, & Hakkarainen, 2008; Scardamali & Bereiter, 2005). The development of
formative measures for instructors (Chapter 2) began a dialectic involving many different
scientific and operational perspectives that extended over more than a year (see Section 3.1). As
principles and practices were identified and refined (Chapter 3), connections to the scientific
literature became clearer (see Chapters 2 and 3). Moving forward, there is a need to articulate
theoretical and empirical foundations that are conceptually consistent, or at least complementary,
and are well established in the scientific literature. This is necessary to ensure a systematic
approach to the continuing development of OBTE and its sustainability as an instructional service
system (see Chapters 1 and 11). The principles and practices of OBTE grounded our search for
the relevant science. They provide a bridge between theory and praxis.
4.1 Exploration of Holistic and Functionalistic Underpinnings for OBTE
The definition of OBTE through this investigation has been the result of broad and deep
collaborative inquiry with individuals who could represent the collective wisdom about an
integrated approach to leadership, education, training, and service in the Army. Among other
things, the definition of OBTE emphasizes individual adaptability and a balance of choice and
responsibility in a collective context. This has two important implications for continuing
development of OBTE as a service system. First, best practices will emerge and evolve from the
initiative of individuals and the decentralized decision-making in a variety of programs at a
variety of sites in the institutional Army. In fact, we have witnessed such decentralized innovation
over the course of this investigation. Secondly, and equally important, promulgation of best
practices and increasingly clarity (even simplicity) of the approach requires a framework that can
be used collectively to interpret and adapt such innovation. While it may be counter-intuitive, a
rigorous framework is more important for an adaptable system than it is for a prescribed and
momentarily optimized system.
In Chapter 1, we claimed that a rigorous approach to service system development enables
initiative by ensuring that there is accountability within a rigorous yet flexible framework for
understanding the evolving and interrelated activities of an organization and that rigorous
description of intent and execution provides desirable constraints on innovation, while flexibility
of the framework allows an organization to make sense of the results of innovation. Most
importantly, in the context of this investigation, we claimed that the behavioral and social
sciences are indispensable to definition and measurement for a service system that is tailored
from the organizations set of standard processes according to the organizations tailoring
guidelines and contributes work products, measures, and other process improvement information
to the organizational process assets (p. 11). The science that appears to be most consistent with
the assumptions of OBTE draws predominantly on a philosophy that is traceable, at least within
87
Riccio
scientific psychology, to William James (1890, 1899/1907) and John Dewey (1915/2008;
Hickman & Alexander, 1998) (see also Boring, 1927/1950; Menard, 2001; Reed, 1997). We were
guided by these philosophical sensibilities in identifying potential scientific foundations for
OBTE in well-established lines of research.
4.1.1 Fundamental Units of Analysis
A first principle of Army training and education is that cognition and action must survive contact
with reality even when thinking and behavior is disrupted or threatened by first contact. The
foundations for training and education in psychological science first and foremost must include a
path to understanding how individuals come to know reality and how their thinking and behavior
is situated in reality. We thus believe an appropriate place to begin developing scientific
foundations for OBTE is from a realist perspective in which the primitive entities and operations
are at the level addressed in Army operations that relate to individual perception and action. One
should be able to observe such primitives directly. We focus on observable attributes of
individuals and their interactions with the surroundings. We consider these interactions as tight
couplings that create emergent properties or functions that dont exist in their component parts.
The couplings can be physical and governed by contact forces or they can be informational as in a
control system (Figure 1). This approach allows us to identify lessons learned about difficult
concepts such as adaptability and stability from scientific study of concrete systems and models
that plausibly can be applied to human beings and human systems (Riccio, 1993a; Riccio &
McDonald, 1988; Riccio & Stoffregen,
1988).
Figure 1. Human-environment coupling is an ontological primitive

The couplings also are nested, that is, there can be couplings within couplings. This is critically
important because it emphasizes the need to understand function not just in terms of proximate
causes but also in terms of second-order and third-order effects. This helps us manage complexity
in the study of human and sociotechnical systems (Riccio, 1993b, 2003; Riccio & Vicente, 2001).
More importantly, consideration of second-order and third-order effects provides a framework for
understanding intangible concepts such as values and culture that is commensurate with
observation of tangible concepts such human-environment and interpersonal interactions. This
88
will be critical to our development of an integrated framework for training, education, and leader
development. The notional framework is represented in Figure 2.
Figure 2. Notional framework for scientific underpinnings of OBTE.

4.1.2 Nested Time Scales and Adaptability
OBTE is an approach that reflects a kind of multicriterion decision making that takes into account
the dynamic interplay among experiences in nested time scales over which the consequences of
decisions and actions unfold. We assume that awareness of such near-far relationships, and the
ability to discover them, imbues individuals with adaptability in uncertain or changing conditions
in a manner analogous to adaptive dual control in dynamical systems (Feldbaum, 1965; Filatov &
Unbehauen, 2004; Grass, Caulkins, Feichtinger, Tragler, & Behrens, 2008; Riccio, 1993b; Riccio
& Vicente, 2001; Safonov, 2001; cf., Chemero & Turvey, 2007; Sheridan, 1999). Research on
adaptive control systems in uncertain or changing conditions reveals a formal problem in
assessing whether change in such systems is stable (truly adaptive) or unstable (maladaptive). A
resolution to this formal problem is the recognition that stability and robustness of a broad class
of such systems is characterized by asymptotic convergence on veridical observation of the
dynamical relationship between a system and its surroundings. Asymptotic convergence is
fostered by persistent excitation (i.e., activity useful for observation rather than control) that is
sufficiently rich (e.g., goes beyond the immediate needs for control) and does not interfere with
performance on the task at hand (cf., Canudas de Wit, 1988; Chalam, 1987; Narendra, 1986). The
relevant lesson from this body of work is that exploratory experience is important in successful
adaptation and that such activity can be concurrent with performance and non-interfering.
Asymptotic convergence in adaptive dual control is related to the concept of trajectorial
objectives in dynamical systems insofar as the objectives by which adaptive performance can be
evaluated reflect the journey as well as the destination (B. Glaser, 2002; cf., Pellegrino,
Chudowsky, & R. Glaser, 2001). The correspondence is even tighter in light of the bi-criterial
synthesis in adaptive dual control that employs two cost functions, one for control per se with
respect to the task at hand, and one for reduction in uncertainty about the dynamical context for
control (see e.g., Filatov & Unbehauen, 2004). The latter cost function (uncertainty reduction)
places learning, about the ways in which tasks are situated, on equal epistemological footing with
momentary capabilities for performance on a task (E. Gibson, Riccio, Schmuckler, Stoffregen,
Rosenberg, & Taormina, 1987; Riccio & Stoffregen, 1988, 1991; Riccio, 1993a; Riccio &
McDonald, 1988; Van Wegen, van Emmerik, & Riccio, 2002). The relevant lesson is that
89
Riccio
learning and doing are not necessarily separable activities. This is not to say simply that one
learns by doing as is often implied by the term experiential learning. The more important point
is that, by doing, one can learn about the context that is relevant to what one might do next (see
Chapter 3, section 3.2.1).
One way in which exploration can be concurrent and non-interfering with performance is for it to
occur on time scales that are smaller than and nested within the decisions and actions that
determine the nature and level of performance on a task. This principle holds at all time scales
from milliseconds up to and beyond the periods of time over which formal learning events are
executed. It also holds across all kinds of interactions with the environment from those involving
physical contact forces to those involving action based on the pick up of information about the
surroundings (Riccio, 1993a,b). Figure 3 shows an example of nested exploration and
performance in the context of a novel postural control task (specifically, an astronaut learning
how to control posture in a space suit so that manual tasks can be performed during
extravehicular activity). In this case, an individual may have to execute a movement or make a
decision about how to move once or twice a second but the individual can experience interactions
with the surroundings more frequently than this.
Figure 3. Information in nested time scales (from Riccio & McDonald, 1998).
90
The point is that one can experience interactions with the surroundings on time scales within and
around those required for performance on a task. This is important because it means that one can
identify the surroundings or changes in the surroundings (i.e., context for a task) that can inform
changes (adaptation) in how one interacts with the surroundings. In the postural control example,
this may be as simple as tracking changes in the asymmetry of fluctuations that indicate a
departure from equilibrium. Experience with such information and the adaptation it affords
provides one with a strategy for postural adaptation in general. This esoteric example is
interesting because expert instructors in OBTE employ strategies that enable this very kind
experience and adaptation in marksmanship training (see e.g., Chapter 2, section 2.3.2). The more
general point, however, is that adaptability may depend critically on micro-experiences that can
help one achieve veridical understanding of the consequences of ones actions. We believe that
micro-experiences are directly analogous to persistent excitation in adaptive dual control. Instride decisions (adaptation) thus, in principle, can be grounded in reality.
Figure 4. Information-driven adaptation (from Riccio & Stoffregen, 1991). Stable

adaptation indicated by asymptotic improvement in performance that implies asymptotic
convergence on a veridical observation of interactions with the surroundings (left panel).
Adaptation also can manifest as in-stride changes to the task at hand (right panel).
4.1.3 Adaptability and Ambiguity
Another important homology between adaptability in dynamical systems and OBTE is the focus
on tangible practical implications about the dynamic interplay between the known and the
unknown and the downstream consequences of this interplay (see section 3.2.1-3.2.2). Adaptive
dual control, and the bi-criterial synthesis in particular, address the role of exploration in the
reduction of uncertainty given what is known. Micro-experiences, for example, need not be
arbitrary or serendipitous. Exploration can focus on aspects of the persistent context that are most
ambiguous or on ambiguities that are having the biggest impact on performance. One can be more
confident in venturing into the unknown if one has some sense of boundaries that cannot be
crossed without irrevocable consequences (Figure 4). This is consistent with approaches to
control of dynamical systems that blend robust control and adaptive control in ways that take into
91
Riccio
account that there almost always is a continuum between complete certainly and complete
uncertainty (e.g., Gutman, 2001; Riccio, 1993a; Riccio & McDonald, 1998). In such approaches,
the range of behavior can be partitioned into subsets within which there is freedom to explore and
innovate but between which there are clear boundaries (Figure 4). Another approach is a form of
supervisory control in which some experimentation is allowed, in which non-catastrophic failures
become informative, and where lack of failure becomes an indicator of success (Safonov, 2001).
The approach of unfalsified control essentially is a survival of the fittest. In principle, this
approach also requires a sense of some boundaries lest an unfit strategy destroy the context in
which another potential strategy could survive. This is analogous to the focus on exploration of
boundaries in our approach to stability and control of dynamical systems (Riccio, 1993a; van
Wegen, et al., 2002).
The importance of boundaries in adaptive dual control is analogous to the role of constraints,
restrictions, and limitations in adaptive operations pursuant to a higher commanders intent in
mission command (HQDA, 2003). In our work on communication of commanders intent, we
concluded that a sense of shared values is important in interpretation of an intent statement with
respect to details of an operational situation that are different from or at a finer grain than
addressed in the intent (Freeman, Sidman, Aten, Diedrich, Cooke, Winner, Rowe, & Riccio,
2008). The implications for OBTE are that values can provide omnipresent boundary conditions
on adaptability can become fundamental to everything a Soldier does. We believe much of this
relationship can be understood in terms of the interplay between choice and responsibility (see
Chapter 3). Furthermore, we believe that exploration of such values-based relationships is why
OBTE emphasizes micro-experiences in learning and the development of adaptability in
individuals (cf., Riccio et al., 2004). Micro-experiences that reveal values-based implications of
learning are an important theme in our exploration of scientific foundations for OBTE.
There are empirical reasons to believe that exploratory micro-experiences can be invaluable in
adaptability of human systems and collective tasks in military operations. One source of evidence
for this is from the nuclear power industry (see e.g., Kaufman, Lanik, Spense, & Trager, 1992;
Mumaw, Roth, Vicente, & Burns, 2000; Roth, Mumaw, & Lewis, 1994). There are a number of
key similarities between this industry and the military, such as: (a) operation of exceedingly
complex systems, (b) coordination among teams of teams of operators, (c) years of training for
operators and especially for leadership positions, (d) standards and procedures developed by a
regulating organization, (e) range and combinations of potential events that are impossible to
address comprehensively by standards and procedures, (f) control and decision-making that is
somewhat distributed and decentralized, (g) formal capture of lessons learned, and (h) innovation
at the ground level that may not be captured in lessons learned.
Specifically with regard to micro-experiences, in the nuclear power industry, the phenomenon of
extra-procedural activity refers to decision making and behavior that is not prescribed by policy
and that appears to be essential in the collective agility of teams in dealing with unforeseen events
(Roth, et al., 1993). This activity comes about because of the complexity of a nuclear power plant
and the uncountably large number of possible combinations of events, however rare, that can
occur and to which some response is required that goes beyond what previously had been
anticipated by extant procedures. Riccio & Vicente (2001) recommended that the nuclear power
industry set up formal processes to track the development of such procedures and codify the
innovations that the industry comes to understand and approve after the fact. They also
recommended that the industry develop training programs to familiarize operators with
boundaries on such activity that allow some license to engage in this activity (Figure 5). In
essence, our recommendations about decentralized design in OBTE (e.g., Chapter 14 and
Appendix C) are the same as these recommendations to the nuclear power industry.
92
Figure 5. Micro-experiences as extra-procedural activity (adapted from Riccio & Vicente, 2001).
4.1.4 Mechanistic Analogies and Predominant Experimental Paradigms
The point of considering analogy and homology between OBTE and dynamical systems is not
that we should view Soldiers or students as a different kind of machine, a more sophisticated
machine. One reason for addressing adaptability in dynamical systems is that it can be viewed as
93
Riccio
a particularly concrete and rigorous manifestation of an analytic philosophy of relevance to

human adaptability and the design of environments that foster such adaptability. Another reason
for mentioning this line of research in a human context is to minimize the tendency to consider
physical analogies for human cognition, and the associated analytical philosophies, only with
respect to relatively familiar computer architectures (Anderson, Bothell, Byrne, Douglass,
Lebiere, & Qin, 2004; Newell, 1990; Newell, Simon, & Shaw, 1958). As suggested above, there
are a wide variety of ways to model human cognition by way of analogy to physics or machines
(see also, Gray, 2007; Richardson, Shockley, Riley, Fajen, & Turvey, 2008; Tschacher &
Dauwalder, 2003; Wolfram, 2002). Ultimately the validity of any such attempt has to take into
account an understanding of human cognition that, on the face of it, also captures essential
aspects of what it means to be human. Our purpose is to ground our inquiry into OBTE in a transdisciplinary context that is sufficiently rich to appreciate such essential attributes.
In OBTE, uniquely human qualities of interest are the development of values-based behavior and
the associated inter-temporal decision making that guides the balance and tension of choice and
responsibility. The issue of nested time scales is relatively neglected in scientific psychology
because it is problematic for the predominant experimental paradigms that are limited to
investigation of human capabilities over time scales commensurate with the attention span
assumed by most entertainment in popular culture (i.e., fractions of a second to a few hours). The
tyranny of these times scales is evident even in longitudinal research or times-series
methodologies in which the interactions among brief intermittent events (e.g., transformations)
typically are not addressed directly. We believe that education, on the other hand, requires
consideration of nested time scales as well as hypotheses about causality sufficient for inquiry
into cognition and behavior that is meaningfully situated, grounded, adaptable, and sustainable.
As the scientific foundation for OBTE develops, there will be considerable value in the
development of paradigms for empirical research that respond to the needs of programmatic
decision making in education (see, e.g., Bailar, 1997; Denzin & Lincoln, 2003; Glasziou &
Haynes, 2005; Godfrey-Smith, 2003; Mislevy & Riconscente, 2006; Pelligrino et al., 2001,
Swales, 2000) and that, while grounded in psychological theory, are not limited to predominant
experimental paradigms of scientific psychology (consider, e.g., Barnes, 1959; Cotkin, 2003;
Solomon, 2004a; Wise, 1980).
4.2 Three Pillars for the Scientific Foundation of OBTE
The citations in Chapters 2 through 5 reflect the constant comparison between our collaborative
inquiry with the progenitors of OBTE and relevant scientific literature. Our comparisons with the
scientific literature obviously were influenced by our prior histories but certainly were not limited
to these biases. Frankly, the inquiry led us to lines of research we did not anticipate and that, in
some cases, were surprising. Two criteria have been critical in our consideration of relevant
scientific research. One is that there must be a strong theoretical underpinning for the research
characterized by scholarly debate and refinement over the time span of decades. The second is
that there should be a strong empirical component that helps build a bridge between theory and
praxis. In other words, the theoretical connection should be actionable for individuals involved in
some phase of the ADDIE process, preferably in all phases. The research should help individuals
understand OBTE, and it should help them communicate and collaborate about it in planning,
execution, and assessment.
4.2.1 Ecological Psychology
We recommend that ecological psychology and related scholarship on dynamical systems be
considered as one pillar in the scientific foundations for OBTE because the interaction between
94
an individual and the environment is an ontological primitive, because the interplay of

exploratory and performatory behavior is an epistemological primitive, and because of a
commitment to realism (see section 4.1). We also are strongly influenced by one of most
important yet relatively neglected tenets of this broad and deep line of scholarshipthat
individuals are in a continual process of learning and development, of coming to know (e.g., E.
Gibson, 1991; J. Gibson, 1979; J. Gibson & E. Gibson, 1955; Reed, 1996). In a sense, this
commitment brings the arrow of time into psychology as in the physics of dynamical processes,
but not limited by the physics of inert matter (e.g., Shaw, Turvey, & Mace, 1982; Kugler &
Turvey, 1987; Thelen & Smith, 1994). Consideration of nested time scales is a powerful approach
to understanding such learning and development and thus adaptability. While consistent with
extensive theory and experience for adaptive dual control in dynamical systems, the relevant
work in ecological psychology has a strong empirical component that crosses the Rubicon from
inanimate systems to animate systems (E. Gibson, 1988; E. Gibson, Riccio, et al., 1987; Riccio,
1993a; Riccio & McDonald, 1998, van Wegen, et al., 2002). Empirical work along these lines in
human experimental psychology and allied fields will be invaluable in development of measures
that help make the intangible more tangible or at least verifiable.
The value of ecological psychology is partly in the broad theoretical foundations for
understanding human engagement with the world. This will be helpful in coming to a better
understanding of the ways in which any and all learning events influence outcomes other than
those that are intended. Empirical research in ecological psychology will be particularly helpful in
the development and use of measures in training and education that relate to situated perception,
coordination, control, and skill (see e.g., Bernstein, Latash, & Turvey, 1996; Chemero & Turvey,
2007; Davids, Bennett, & Newell, 2005; Fajen, Riley, & Turvey, 2009; Frank, Michelbrink,
Beckmann, Schollhorn, 2008; Hancock, Flach, Caird, & Vicente, 1995; Sternad, 2009; Turvey,
2007; Warren, & Wertheim, 1990). A relatively neglected principle of ecological psychology is
that one must understand the nested structure of human-environment interactions to understand
what it means for perception and action to be situated (J. Gibson, 1979). This does not necessarily
imply intractable complexity in studying perception and action. Typically, the approach redirects
analytical reduction to an intermediate level of analysis that is pluralistic and naturalistic (cf.,
Bunge, 1977; Treisman, 1988).
Consideration of the nested structure of an operational situation helps with bookkeeping for the
influences of higher and lower levels on the most basic level of a situation (Rasmussen, 1997;
Riccio, 1993b; Riccio & Stoffregen, 1988; Riccio & Vicente, 2001). Such subordinate and
superordinate constraints on interactions with the surroundings often reveal ways to simplify the
understanding of such interactions. The simplifying schemes characteristically emphasize the
importance of 2nd-order and 3rd-order effects of such interactions (cf., Gibson, 1977). Examples of
this kind of analytical reduction are show in Figure 6 for piloting an aircraft and in Figure 7 for
collective operation of a nuclear power plant. We believe such an approach will be valuable in
characterizing and coming to a deeper understanding of the structure of a good learning
environment and, in particular, the influence of a community-centered perspective on learnercentered, knowledge-centered, and assessment-centered perspectives on the structure of a good
learning environment (Bransford et al., 2000). In the context of OBTE, a deeper understanding of
the nested structure of a learning environment provides clues about outcomes that are influenced
in every learning event whether or not they are intended. This working assumption will guide our
integration of ecological psychology with other perspectives that are necessary for a more
complete understanding of a values-based approach to training and education.
95
Riccio
Figure 6. Consideration of basic levels and the nested structure of perception and action
in piloting an aircraft (from Riccio, 1993b).
Figure 7. Consideration of basic levels and the nested structure of perception and action
in the collective operation of a nuclear power plant (from Riccio & Vicente, 2001).
96
Ecological psychology is not sufficient for our purposes in that it has not yet provided much
insight into the social and cultural issues that are critical in OBTE. Nevertheless, its commitment
to realism will be quite helpful when we complement this line of research with other disciplines
that can drift uncomfortably into constructs having questionable validity and verifiability with
respect to the realities of OBTE. We believe that self-efficacy theory and positive psychology are
the most promising complementary perspectives to ecological psychology. The principles and
themes of most relevance to OBTE are highlighted below.
It should be noted that there has been very little engagement to date, either positive or negative,
among these three lines of thought. Presumably this is due to apparent philosophical dissonance
among them and trans-disciplinary obstacles within the culture of academe. The study of social
and cultural phenomena, for example, is preoccupied with the implications of inductive inference
in interpretations of ones experience or in extrapolation to hypothetical situations, especially
when there are nonintuitive or otherwise interesting inter-subjective disparities between cognition
and reality. While there is little doubt about the causal potency of induction in ones experience,
we do not focus on terminal induction, that is, on premature conclusions based on inductive
inference. We are more interest in deductive inference over time and on the role of induction, en
passant, in directing the focus and in identifying the limits of deductive inference (cf., Chapter 3,
section 3.2.1). OBTE is concerned with the arrow of time and coming to know a reality that can
be shared experientially, that can verified, and that is valid with respect to engagements based on
such knowledge. The interest, of necessity, is in adaptable and sustainable interactions with
reality. We thus emphasize a commitment to realism (cf., J. Gibson, 1979; Reed, 1996; Reed &
Jones, 1979, 1982) to the extent that it is possible and reasonable given the nascent confluence of
these lines of thought. Initially, we believe that the philosophically tentative juxtaposition of
ecological psychology with other lines of thought in social psychology can benefit greatly from a
focus on the ways in which concrete and verifiable experience inside and outside of formal
learning events influences self knowledge, motivation and emotion, and the development of
values.
A tenet of ecological realism that fits comfortably with self-efficacy theory and positive
psychology is that perception, cognition, and action must be understood as fundamentally situated
in a broader context (J. Gibson, 1979; see also, Neisser, 1996; Reed & Jones, 1982; Shaw &
Bransford, 1977; Shaw et al., 1982; Stoffregen, & Bardy, 2001; cf., Georgas, van der Vijver, &
Berry, 2004; Heine, & Norenzayan, 2006; Heft, 2001; Henle, 1971; Hofstede, 2001; Lewin, 1951;
Matsumoto, & Yoo, 2006; Wang, 2006; Winograd, Fivush, & Hirst, 1999). This axiom is much
less controversial than when it was introduced three decades ago as a departure from the
reductionism that was dominant in the behavioral and social sciences at that time. Consideration
of context is common today, albeit often implicitly as an additive factor influencing thinking and
behavior. It should be noted that the influence of context is not additive in ecological realism. The
individual can have as much of an effect on the surroundings as the surroundings can have on the
individual. The dynamics of the coupled system are emergent. Thus a realist commitment in
OBTE avoids both structural-functionalism and cognitive constructivism in the social sciences
(cf., Bandura, 1997; Denzin & Lincoln, 2003; Scott, 2000; Wenger, 1998). Meaning is
constructed to the extent that individuals change their surroundings and situations, by coupling
with it, and thus change what it affords for them. Meaning, whether or not constructed in this
way, is observable and verifiable in the behavior and performance of individuals with respect to
each other and their surroundings.
97
Riccio
4.2.2 Self-Efficacy Theory

We recommend self-efficacy theory (e.g., Bandura, 1977, 1995, 1997) as a second pillar
primarily because of the centrality of interpersonal influence, role models and social support
structures in OBTE as well as the relationship between the pursuit of mastery and perception of
ones abilities over time. We are interested in exploring connections with self-efficacy theory
apart from aspects of social cognitive theory that take a loan on intelligence that science can
never repay (Turvey, Shaw, Reed, & Mace, 1981, p. 248; cf., Dennett, 1971, p. 96). We
recognize that selecting one component of a line of research from a broader theoretical amalgam
is precarious from the perspective of scholarship, so we do so with some caution. It is worth
noting, in this respect, that even Bandura points out that self-efficacy is separable, conceptually,
from social cognitive theory (Bandura, 1997, p. 34; although presumably he would prefer them
not to be separated in a holistic explanation of human cognition and action). As in theories of
perception, in general, one can debate the relative epistemological importance of inductive and
deductive inference as a basis for beliefs or about the ontological status of beliefs. We believe it is
more productive to focus on identifying the information and engagements through which
perception of ones own capabilities becomes grounded in reality and to consider the nature of
that reality (e.g., Riccio, 1993a,b; Riccio & McDonald, 1998; Riccio & Stoffregen, 1988; Shaw et
al., 1982).
We are deeply indebted to self-efficacy theory for an extensive empirical foundation that reveals
the variety of causally potent aspects of the social environment for human learning and
development. We seek explanations of these causal influences in terms of augmentation to an
individuals perceptual systems, critical influences on the education of attention, and essential
coordinative structures in the pickup of information about reality. We believe this will have
concrete implications about how to improve the accuracy of an individuals self perception. In a
sense, we view elements of the social environment with which an individual shares reciprocal
causal relations as informative dynamic components in nested interpersonal systems (cf.,
Riccio, 1993a,b). This means that we can view others as augmentation to our own perceptual
systems rather than merely as a separate source of information. While the latter suggests a source
of information imposed on a passive knower, the former emphasizes an active knower who can
obtain additional sources of information that complement ones own individual capabilities to
know. This is one reason why OBTE stresses collaborative reflection even in the context of
learning ostensibly by individuals in learning events designed for an individual (see Chapter 3).
The importance of our ecological perspective on self-efficacy is a focus on information, on
coordinated action, and collaborative information pickup in task-organized groups. It changes the
emphasis from the existence of social support structures to the nature of the transactions within a
social group. It focuses specifically on what instructors or other members of a group do to
facilitate, advise, and mentor a learner inside or outside formal learning activities. These
considerations are entirely consistent with self-efficacy theory even if the causal explanations are
different. We seek a multifaceted empirical program for OBTE that can yield depth in the
understanding of instructor-student interactions and collaborative learning that can complement
the breadth of considerations inherited from self-efficacy theory and related work in social
psychology. We seek a deeper understanding of the nature of the arduous work that is critical to
the pursuit of mastery (Bandura, 1997).
Ubiquitous exploration, micro-experiences, inter-temporal perception and action, and adaptability
are key concepts in OBTE. We agree with Bandura (1997) that one need not postulate an inherent
motivation to explore. We also agree that exploratory behavior is best understood in the context
of outcomes it promotes, as in adaptive dual control theory (see Section 4.1). At the same time,
98
we recognize that a certain amount of exploratory potential is inherent in the variability of life,
especially human perception and action; thus, all exploratory behavior need not be motivated by
intended outcomes (Riccio, 1993a,b; Riccio & McDonald, 1998; van Wegen, et al., 2002).
Furthermore, we allow for the possibility that exploratory behavior can lead to information-based
inter-temporal decisions that change ones goals or desired outcomes (E. Gibson, Riccio, et al.,
1987; Riccio, 1993; Riccio & Stroffregen, 1991; cf., Safonov, 2001). Thus, exploratory behavior
can be directed and serendipitous. This has important implications for instructor behavior and for
the structure of learning environments that promote learning to learn (see e.g., section 3.2.1).
The centrality of outcomes is an important commonality across the three pillars of the foundation
we propose for OBTE. Importantly, self-efficacy theory emphasizes a comprehensive
functionalist view of outcomes (Bandura, 1997, p. 22) that accommodates the immediate
tangible effects of learning as well as the inter-subjective implications and meaning of learning.
Self-efficacy theory thus recognizes the importance of personal and shared values in the
regulation of behavior. It begs for a more extensive and nuanced treatment of these positive
psychological influences on learning and development, one sufficient to explain the power they
have on human thinking and action. This exigency of self-efficacy theory guides us in the
direction of complementary lines of scholarship that reflect persistent themes across the millennia
of philosophical reflection on the meaning of existence (e.g., Aristotle, trans. 1925; Augustine,
trans. 1991; James, 1902/1982; Maslow, 1968; Plato, trans. 1892a; Solomon, 2004a).
4.2.3 Positive psychology
A third pillar in the scientific foundations for OBTE is positive psychology (e.g., Aspinwall &
Staudinager, 2003; Keyes & Haidt, 2003; Lopez & Snyder, 2009; Peterson & Seligman, 2004;
Seligman, Steen, Park, & Peterson, 2005; cf., Maslow, 1968) because the defining focus for
OBTE is development of the individual with respect to persistent attributes embodied in Army
Values and Warrior Ethos, specifically the assumption that such attributes can be addressed in
concrete and verifiable ways during Army training (cf., Riccio et al., 2004). Positive psychology,
while largely atheoretical, has pulled together a body of work on character strengths and values
across cultures and over significant periods of history. Initially, the progenitors of this approach
surveyed significant canonical texts associated with Confucianism and Taoism in China,
Buddhism and Hinduism primarily in India, ancient Greece, Judeo-Christianity, and Islam after
developing ten criteria for identifying strength of character (Peterson & Seligman, 2004). The
authors argue for six core virtues that generalize across the traditions they surveyed: wisdom,
courage, humanity, justice, temperance, and transcendence. They further defined character
strengths as mediating psychological constructs through which virtues are displayed. Table 1
shows the initial list of twenty-four character strengths organized by the associated virtues.
The indicated virtues reflect many of the same aspirations expressed in Army Values and Warrior
Ethos (see Chapter 3; Appendices D, E). Peterson & Seligman (2004, p. 14) also distinguish
virtues and character strengths from situational themes which they refer to as behavior habits
tied to particular situations and settings (e.g., work, family). Presumably these themes are
analogous to performance indicators developed for OBTE (see e.g., Chapter 2). The examples
they provide from the Gallup organizations situation themes include (a) anticipating the needs of
others, (b) making others feel part of a group, and (c) seeing what is good in situations and in
people. This indicates an important difference between the predominant methods of Positive
Psychology and the methods developed for our investigation of OBTE. Positive psychology
focuses on retrospective self-reports or reports of others that are not specific to any particular
event. Our methods emphasize measures that can be applied to the observation of specific
behavior in a specific event (concurrently or retrospectively). We believe this behavioral
99
Riccio
grounding is essential to the scientific development of OBTE and, arguably, any intervention that
seeks to influence character strengths and virtues (see e.g., Seligman et al., 2005). Such measures
provide increasing clarity on what individuals actually do in sufficient detail and context to make
it possible to identify consequences of such actions. At the same time, we do not diminish the
value of surveys. In fact, we believe they complement behavioral measures (see e.g., Chapters 1011) and that they will be necessary to determine the full impact of OBTE (see Epilogue). We
dont believe that surveys are sufficient, however, to assess the impact of an intervention on
character strengths and values (see e.g., Chapter 1).
Table 1: The twenty-four strengths and virtues of positive psychology (from Peterson &
Seligman, 2004).

1. Wisdom & knowledge
Creativity
Curiosity
Open-mindedness
Love of learning
Perspective

2. Courage
Authenticity
Bravery
Persistence
Zest

3. Humanity
Kindness
Love
Social intelligence

4. Justice
Fairness
Leadership
Teamwork

5. Temperance
Forgiveness
Modesty
Prudence

Cognitive strengths that entail the acquisition and use of knowledge
Thinking of novel and productive ways to do things
Taking an interest in all of ongoing experience
Thinking things through and examining them from all sides
Mastering new skills, topics, and bodies of knowledge
Being able to provide wise counsel to others
Emotional strengths that involve the exercise of will to accomplish goals in
the face of opposition, external or internal
Speaking the truth and presenting oneself in a genuine way
Not shrinking from threat, challenge, difficulty, or pain
Finishing what one starts
Approaching life with excitement and energy

Interpersonal strengths that involve tending and befriending others
Doing favors and good deeds for others
Valuing close relations with others
Being aware of the motives and feelings of self and others

Civic strengths that underlie healthy community life
Treating all people the same according to notions of fairness and justice
Organizing group activities and seeing that they happen
Working well as member of a group or team

Strengths that protect against excess
Forgiving those who have done wrong
Letting ones accomplishments speak for themselves
Being careful about ones choices; not saying or doing things that might
later be regretted
Self-regulation
Regulating what one feels and does

6. Transcendence
Strengths that forge connections to the larger universe and provide
meaning
Appreciation of
Noticing and appreciating beauty, excellence, and/or skilled performance
beauty and excellence in all domains of life
Gratitude
Being aware of and thankful for the good things that happen
Hope
Expecting the best and working to achieve it
Humor
Liking to laugh and tease; bringing smiles to other people
Religiousness
Having coherent beliefs about the higher purpose and meaning of life
100
Surveys are not sufficient for the purposes of explanation, prediction, or assessment of
interventions because they generally do not provide enough detail to identify critical situational
factors that influence the behavior of individuals (cf., Bandura, 1997, p. 39-42). We are more
sanguine than Bandura about the utility of surveys if they are grounded in behavioral measures in
specific situations. We are in agreement with Bandura that characterization of personality traits
from such surveys is, at best, a distraction in any attempt to explain or predict human behavior or
to assess interventions. The interest of positive psychology in individual differences and traits
seems to be a legacy of its evolution from clinical psychology. We expect such interests to
become incidental to the growing body of research in this relatively recent movement in
psychology (see e.g., Lopez & Snyder, 2009). The important contribution of positive psychology
is that it is helping researchers understand outcomes of relevance to any intervention that claims
to have an impact on values. It helps focus on development of methods and measures for
acquiring evidence that can, in principle, support such claims.
Positive psychology and self-efficacy theory superficially can be misinterpreted as narcissistic,
self-indulgent, or focused on simple cheerfulness. This is definitively not the intent (see e.g.,
Bandura, 1997; Peterson & Seligman, 2004). The intent of these independent lines of scholarship
is to enable scientifically rigorous inquiry into the some of the most serious aspects of the human
condition, self-identity and the meaning of existence, although it seems that they will need some
help living up to this potential. Ecological psychology enters into this realm by emphasizing that
the ontological primitives for human cognition and behavior are not defined in terms of an
isolated individual, yet neither are they subordinated to a mindless collective. It helps get us
closer to a scientific understanding of values-based behavior that is observable and verifiable by
emphasizing nested reciprocities between the individual and the environment as well as intertemporal information in immediate circumstances that is relevant to future engagements with the
environment. All these lines of scholarship, in essence, pointedly address something of
significance that is both deeply personal and larger than the individual, something that is relevant
beyond a moment in time.
Figure 8. In principle, one can address reciprocal influences and their emergent properties in
the same ways for interactions with social environment as with the physical environment. 1storder others share the same physical context. 2nd-order others are engaged largely with a
different context. Presumably coordination with such individuals is guided by more abstract
commonalities such as shared values. Such values exemplify a broader community that
includes 3rd-order others with whom one momentarily is not interacting.
101
Riccio
4.3 A More Integrated Scientific Infrastructure
In the further development of a scientific infrastructure for OBTE, we will continue to focus on
observable behavior of individuals, generally of individuals interacting with each other, such as in
the transactions between an instructor and a student. We believe such ecologically grounded
measures will provide opportunities to close the gap between the causally potent constructs of
self-efficacy theory and the aspirational outcomes emphasized by positive psychology. The key to
establishing these conceptual connections will be to identify behavior that is inter-subjectively
valid and that has potentially verifiable relationships with values-based outcomes. In the
following sections, we summarize several lines of research that can help connect the three pillars
of the scientific foundation for OBTE (Figure 2).
4.3.1 Self Determination Theory
We have found self-determination theory (e.g., Deci & Ryan, 1985, 2008; Ryan & Deci, 2008) to
be a valuable bridge between self-efficacy theory and positive psychology. To be sure, selfdetermination theory connects with positive psychology more comfortably (see e.g., Brown,
Ryan, & Cresswell, 2007; Seligman et al., 2005) than with self-efficacy theory (see e.g., Deci &
Ryan, 2000). While there are important differences between self-efficacy theory and selfdetermination theory, they are more complementary than one might suspect based on the
opposing positions of a historically important polemic in scientific psychology about sources of
motivation and personal agency. Both theories, which have matured over decades, have a
nuanced appreciation of reciprocal influences of internal and external sources of motivation. The
difference that matters in the present context is the centrality in self-determination theory of the
triadic interplay among inherent aspects of motivation: competence, autonomy, and relatedness
(see Chapter 3, section 3.2.3). This seems to capture more of the existential subtleties assumed by
the intangible attributes emphasized in OBTE (e.g., confidence, initiative, and accountability)
than does the equally important but somewhat vague effects of shared values in the
comprehensive functionalism of self-efficacy theory.
The central thesis of self-determination theory is that there are three innate psychological needs
that always influence motivation, vitality, and development (i.e., well being). Specifically, the
theory assumes that the need for competence, autonomy, and relatedness has a positive effect on
well being to the extent that external conditions allow them to be satisfied, and they have a
negative effect to the extent that they are thwarted by external conditions (see e.g., Deci & Ryan,
2000; Ryan & Deci, 2008). In self-determination theory, competence is the need to feel effective,
autonomy is the need for volition rather being controlled, and relatedness is the need to feel
significant and connected. Within a commitment to realism, however, these innate endowments
are inherent in the human condition of being in the perceivable world, and there is no need to
suggest purely psychological qualities that exist apart from the perceivable world. The needs
articulated by self-determination theory and external conditions need not be considered as
independent ontological primitives to be compared in terms of concordance or dissonance.
Individuals can influence conditions in which they are situated through purposeful engagement
with their surroundings on multiple levels. In principle, individuals can influence the conditions
that influence their own behavior and performance (cf., section 4.2.1). While there are limits to
such nested engagement, the critically important theoretical implication is that the causal potency
of external conditions need not connote that the individual is a pawn of the system, for good or
ill, and without responsibility. The practical implication is that inquiry into the determinants of
competence, autonomy, and relatedness can focus on observable and verifiable manifestations of
the reciprocity between the individual and environment (i.e., the social environment and the
physical environment).
102
The emphasis on autonomy in self-determination theory should not be confused with an internal
locus of control. More fundamentally, it connotes the experience of freedom and an integrated
sense of self (Deci & Ryan, 2008). The distinction is important because it implies that one can
choose to yield or cede power to someone or something else and still be self-determined. Building
on this, the theory thus allows for the possibility that a self-determined individual can engage in a
complex undertaking in which there is a spectrum of various combinations of internal control and
external control. We believe that this is precisely what occurs in a task-organized group in which
individuals (e.g., Soldiers) complement each other, and even teach each other irrespective of rank
or duty position. We believe that training and education can prepare Soldiers for the skills of task
organization. There is a significant body of work on the pedagogical implications of such
collaboration (Hmelo-Silver, 2004; Tobias & Duffy, 2009) for which the research in selfdetermination theory and self-efficacy theory provides a broad scientific foundation.
Giving up some measure of control is more consistent with freedom and autonomy to the extent
that one shares values or significant relationships with others to whom one cedes power (cf.,
Bandura, 1997; Lewin & Grabbe, 1945). For such reasons, self-determination theory emphasizes
the importance of the process by which an individual internalizes external sources of motivation,
that is, the inculcation of shared values (Deci & Ryan, 2000; cf., Riccio et al., 2004; Peterson &
Seligman, 2004). It thus is difficult, if not impossible, to separate the influential roles of
autonomy and relatedness in self-determination theory. Self-determination theory seems to limit
consideration of relatedness to interpersonal relationships and understanding (Deci & Ryan, 2000;
cf., Hume, 1740/2009; Peterson & Seligman, 2004; Plato, trans. 1892b; Smith, 1759/1976).
While the evidence about such interpersonal influence can be an important contribution to the
future development of OBTE, we believe that relatedness should also take into account the
reciprocal relationships with the physical environment; that is, it should include all aspects of
being in the world (see also, section 4.2.1). Purposeful behavior is constrained by awareness of
the consequences and implications it has on both the social and physical environment in which it
is situated. In OBTE, education of the attention for better situation awareness is critical in
fostering adaptability and learning to learn. Behavioral scripts, even inadvertent ones, can
undermine the development of skills needed for situation awareness and the pursuit of mastery.
Competence in self-determination theory is closely related to self-efficacy and to the meaning of
confidence in OBTE. As one of the inseparable intangibles of OBTE, confidence refers to a sense
of self that is grounded in competence but, importantly, it also includes a sense of ones zone of
proximal development (Cole, John-Steiner, Scribner, & Souberman, 1978). In OBTE, confidence
is dependent on ones ability to perceive ones capabilities for developing enhanced efficacy.
Given a commitment to realism, the emphasis is on ones metacognitive awareness of the
strategies of attention and action that are efficacious with respect to the pursuit of mastery (see
e.g., Chapter 3, section 3.2.6). Initiative at the boundaries of the known and the unknowable is
essential in picking up information that grounds adaptation in better understanding of reality and,
thus, that gives it a developmental character (section 3.2.1). To be robust, however, initiative
must be balanced by a sense of consequences, especially consequences that are catastrophic or
otherwise irrecoverable (cf., Riccio & Stoffregen, 1988, 1991; Riccio 1993a,b). This is the triadic
balance that is central to OBTE (see Chapter 3, section 3.2.3), and it maps usefully into the triad
of competence, autonomy, and accountability of self-determination theory. This mapping is
exceeding important for the further development of OBTE given that the findings and
applications of this body of work have direct relevance to training and education (Deci & Ryan,
2008; Guay et al., 2008).
103
Riccio
4.3.2 Situated Learning Theory

Our broad intent has been to identify the relevance and lessons of OBTE for formal programs of
training and education in the Army. We chose not to begin with theories and best practices in
formal programs of training and education in general even though there is an extensive literature
behind it (e.g., in educational psychology). Instead, our strategy has been to begin with the
identification of principles and practices of OBTE and the associated wisdom of Soldiers as
professionals, leaders, and members of a community with well-defined values. We then allowed
this to take us wherever we needed to go to identify the most relevant scientific underpinnings. In
a sense, we have peeled back the layers of pedagogical theory and practice to reveal that there is a
core with clear and uncluttered relevance to the needs of the Army. Returning to foundations of
pedagogy in psychological science minimizes the distractions of incidental details in particular
applications of psychological theory to pedagogical practice. It also avoids problems that derive
from replicate fading in the translation of psychological theory to pedagogical theory as is the
case, for example, in adaptations such as adult learning theory.
It is noteworthy that the common foundation we have identified is not at the most abstract level,
that is, not at the level of scientific or philosophical conjecture about properties of the mind of an
individual learner. Nor did we seek a common foundation in the considerable bodies of
reductionistic scientific research in learning theory that such abstracta have spawned over the
last one hundred years or so. Instead, our aim has been to identify foundations for teaching and
learning that cut across the myriad of settings in which learning occurs. A key objective is to
inform curriculum-level or career-level programmatic reflection and decisions about teaching and
learning in the context of its impact on development of individuals in a community with common
values. The basic level of inquiry to which we thus were led concerns learning that is explicitly
and irreducibly situated in a physical, social, and cultural environment (see Figures 2, 7 and 8).
To this point, we have highlighted significant bodies of scientific research in the behavioral and
social sciences that offer theoretically well-established methods and measures with direct
implications and utility for verification, validation, and further development of a values-based
approach to training and education (see Chapter 1). The theoretical and empirical foundations are
relevant to outcomes that have meaning in a common community and have relevance to the
environment within which the practices of a community have purpose (i.e., situated learning).
Adding to this foundation, it is important to point out relevant applications of situated learning
theory outside the Army. In particular, we briefly highlight the work on communities of practice
(e.g., Lave & Wenger, 1991; Wenger, 1998). This is a different kind of evidence with respect to
the claims of OBTE. Unlike the research cited in previous sections, it is not a source of evidence
that directly addresses claims about the causal potency of the principles and practices of OBTE,
nor is it evidence for the trustworthiness of methods and measures with which the impact of
OBTE can be evaluated. The work on communities of practice is important, however, insofar as it
promulgates lessons learned about problems and opportunities in applying situated learning
theory in various organizational settings and its implications for transfer of learning. Free from
unnecessarily limiting dependence on cognitive constructivism or information-processing
metaphors, this work helps establish a tighter connection between other social learning theories
(e.g., self-efficacy theory) and ecological psychology; that is, it helps unify the social and
physical elements of reciprocal influence between an individual and the environment.
Wenger and Lave are much more deferential to postmodern sociopolitical sensibilities than we
are inclined to be. We certainly do not share many of the motivations and conclusions of
postmodern thought, especially in its occasional dalliances with science (Hacking, 1999; Latour,
2004). Nevertheless, the convergence of our thinking with that of Wenger and Lave does reflect
104
some common concerns across these diverse perspectives, such as identifying a proper role for
science in social and programmatic decision-making (see Flyvbjerg, 2001; Schrim, & Caterino,
2006; see also Chapter 11). In this respect, the work of Wenger and Lave is noteworthy because it
represents an anthropological style of inquiry to which we aspired in our investigation of OBTE.
It looks at what people actually do and experience in particular settings irrespective of how those
settings are designed or independently characterized. This is critically important in situations
characterized by change, innovation, decentralized adaptation, or what we refer to as task
organization. It is not surprising that, in our own inquiry, we have come to conclusions similar to
that of Wenger and Lave (e.g., Wenger, 1998).
The work on communities of practice reveals that most people spend much more time learning,
and in a wider variety of settings, than is suggested by the body of research on educational theory
and practice. Individuals engage in significant learning outside of formally designed curricula and
fabricated learning environments. They make connections between situations, between the
learning that occurs in different situations, and they integrate that knowledge. They do so whether
or not the situations are designed as part of same curriculum. Learning and knowledge in one
situation (e.g., outside a formal educational setting) certainly can influence learning in a different
situation (e.g., inside a formal educational setting) whether or not the different situations are even
considered by instructional designers (cf., Bransford et al., 2000). For us, the implication is that
anyone with a programmatic stake in understanding and influencing such integrated learning has
an even greater need for methods of verification, validation, and continuous improvement (e.g.,
formative assessment) than someone who is committed more narrowly to programmatic control
of a learning environment. This is the only way to identify and accommodate factors affecting the
transfer of learning over which students have more control than curriculum designers. In OBTE,
rigorousness will be associated with formative assessment that is organic and habitual rather than
with maximally detailed scripts for instruction and equally detailed specifications for learning
environments.
Formative assessments should focus on the most important factors that influence learning.
Importantly, our inquiry converges with the work on communities of practice in concluding that
(a) social factors are the most important determinants of learning outside of formal learning
environments and perhaps even inside such formal settings, and (b) social factors have their
greatest causal potency in a shared environment within which there is a common purpose
(Wenger, 1998). We join Wenger in expressing some surprise that such a conclusion is
noteworthy. That it is noteworthy emphasizes how a hegemonic focus on the structure of
knowledge, and debate over the various methods of implanting it in the mind of a student, have
allowed educational research to drift from exigencies associated with the obvious fact that a
student is a human being and, as such, has a natural proclivity to find meaning and establish an
identity through engagement with the real world, most notably with others from whom the
student can learn (e.g., Augustine, trans. 1953; Keatinge, 1896). For this reason, from the outset,
our inquiry has focused on identification of instructor-student interactions that are explicitly
relevant to the conditions of learning and the task at hand. This focus enabled the development of
formative assessments that capture and promulgate instructional best practices.
Another important connection with communities of practice is the realization that teaching and
learning can be roles that shift within and between situations, that there is not necessarily an
immutable power asymmetry between individuals within a community. There may be particular
settings in which one person has a formal role as a teacher and others have formal roles as
students but, beyond this context, individuals generally need to move into and out of such roles
from moment to moment (see e.g., section 3.2.6). For us, this is an important consideration in far
transfer of knowledge and skills from a formal learning environment. Equally important, and
105
Riccio
implicit in this assumption, is that there is transfer of learning from prior experience to formal
learning environments. This becomes a more important consideration for instructional design
during times of broad cultural change. While many of the claims about the information age are
debatable, a strong case certainly can be made that the current generation of adolescents have
much more experience in the obtaining of stimulation, the active pick up of information, and the
generation of content through web-based engagement because of the increasing capabilities of
emerging electronic media in entertainment and education (Lakkala, et al. 2008; Paavola &
Hakkarainen, 2005; Scardamali & Bereiter, 2006).
Learning theory that is based on the ubiquity of communities of practice gives us a broader
appreciation of the various ways that individuals engage with their surroundings and helps us
understand the influence of these engagements on continuous learning and development. An
individuals identity is an important focus of this work. The emphasis is on the developmental
nature of identity, that it is a dynamic weave of trajectories over time within and across the
various communities with which one concurrently and sequentially has membership (Wenger,
1998). Every individual has a multiplicity of developmental trajectories because of communityspecific outcomes and associated modes of engagement and belonging. At the same time, there is
a coherence to these trajectories that both drives and reflects the development of an individuals
identity. An important implication of this observation is that learning can be influenced strongly
by such coherence, that is, by the personal and contextual factors that influence transfer of
learning among concurrent and sequential engagements with reality. This, in turn, implies that
instructors can have a powerful effect on learning by leveraging such factors. A fundamental
tenet of OBTE is that intangible attributes such as confidence, initiative, and accountability are
enabling outcomes that can be influenced in every learning event and, in fact, that are influenced
in any meaningful engagement with reality. We believe that these are an important set of factors
that influence the trajectorial coherence that is critical to the development of identity and to both
persistence and change in ones identity (see section 4.1.2).
The development of identity within the context of multiple concurrent and sequential
communities of practice also emphasizes the importance of cultural values as a coherent context
within and across communities of practice (see section 4.2.3). Values are a transcendent aspect of
the coherence in ones developmental trajectories and in ones engagement with reality (cf.,
Lopez & Snyder, 2009; Maslow, 1968; May, 1983). The scientific foundations for OBTE require
that we avoid viewing students as passive receptacles for information about values or an
indifferent lump of clay to be molded through the craft of applying pressure here and there over a
period of time. Individuals engage with their surroundings and situations in ways that may or may
not be obvious. They find meaning and create meaning in their surroundings through their
engagements and through their experiences that have both a history and a future. An instructor
can create a momentary set of conditions in which a student has no apparent control but that does
not mean the instructor has total control of the meaning that students make of the situation. The
students experience will be influenced by the history and values that influence their
interpretation of the situation, in the moment, as well as what they take away from the situation
and employ in future engagements within which they have more control (Wenger, 1998).
4.3.3 Existential Psychology
There is an obvious convergence of the elegant prose and accessible philosophy of Wenger and
Lave with the influence of existential thought on psychology, such as reflected in the equally
cogent writing of Rollo May (e.g., May, 1983; May, Angel, & Ellenberger, 1958). Existentialism
may be a direct influence on situated learning theory, or the convergence may be an intellectual
homology due to reflection on personal agency and development of identity in situations
106
characterized by existentially significant events and change. In any case our own qualitative
inquiry into the principles and practices of OBTE, the situations in the Army to which they are a
response, and our constant comparison with relevant scholarship, has led us inexorably to
existentialism. We believe it is important to make this explicit because existentialism provides a
powerful grounding for further development of OBTE given the historical depth and breadth of
the scholarship in this tradition (see also Chapter 5). It would be difficult to find a more poignant
and insightful body of thought about an individuals engagement with reality with clear
implications for the behavioral and social sciences. We thus look to existential psychology to
facilitate identification of these implications (e.g., Greenberg, Koole, & Pyszczynski, 2004;
Hoffman, et al., 2009; May, 1983; May et al., 1958).
In addition to the direct relevance of existential psychology to OBTE, it also is useful in
establishing an integrated scientific foundation for OBTE. In particular, it dovetails with
ecological psychology and positive psychology in deep and meaningful ways. With respect to the
sociology of science, both existential psychology and ecological psychology grew out of
dissatisfaction with the pointedly meaningless behaviorism and the ontologically reductionistic
cognitivism that dominated most of the 20th century in psychology (see e.g., Maslow, 1968; May,
1983; Reed & Jones, 1979, 1982; Shaw et al., 1982). More fundamentally, with respect to the
philosophy of science, both ecological psychology and existential psychology take the radical
turn of rejecting Cartesian dualism in favor of a monistic ontology that avoids problems created
by the false dichotomy of subject and object and by confusing an epistemological commitment
with an ontological commitment (see e.g., May, 1983; J. Gibson, 1979).
The fundamental tenet of ecological psychology and existential psychology is that individuals are
perceptually in developmental trajectories that inform and derive from their engagement with
reality (see e.g., May 1983; E. Gibson, 1991). As an empirical tradition, ecological psychology
has concentrated more on experimental psychology and the rigorous description of behavioral
couplings between an individual and the environment. The focus of existential psychology has
been on the clinical psychology that addresses causes and experiential consequences of an
individuals lack of acumen in engagements with reality or in frank disengagement with reality.
In spite of the deep philosophical commonality between these two lines of scholarship, the
sociological obstacles between clinical and experimental psychology have kept them from a
realizing their potential complementarity. The most promising nexus of ecological psychology
and existential psychology is in weaving together rich descriptions of behavior and experience in
a coherent philosophical framework that captures more fully what it means to be human, and thus
that facilitates crystallization of scientific perspectives and other perspectives on engagements of
individuals with the physical and social aspects of the world around them. This is both critical
and practically relevant to any endeavor that seeks to describe the wisdom of Soldiers or any
other group of experts involved in preparing individuals to engage meaningfully and responsibly
with their surroundings and situations.
There also is an interesting juxtaposition of existential psychology and positive psychology given
that the historicity of both lines of scholarship is in clinical psychology of what goes wrong with
people. The powerful contribution in the paradigmatic shift of positive psychology is that there is
an untapped wealth of information to be uncovered about what goes right with people (Peterson
& Seligman, 2004; Seligman et al., 2005). In doing so, positive psychology explicitly and
strategically avoids an assumption that one can improve the health and well-being of others with
the same interventions that make people less unhealthy or less miserable. It assumes that
interventions can be designed on a solid foundation of millienia of scholarship on values and
strength of character (e.g., Aristotle, trans. 1925). In this assumption, and in the research it
motivates, we believe positive psychology wanders into the realm of education and leadership.
107
Riccio
The intent of positive psychology is to avoid focusing merely on cheerfulness or narrow

interpretations of pleasantness that the word positive implies. As yet, however, it has not
produced a body of work that reveals a gravitas commensurate with the context and impact of
Army training and education, although it is making some progress (see e.g., Lopez & Snyder,
2009). This may be because of sociological pressures in the scientific community for positive
psychology to differentiate itself from its clinical psychological foundations and the attendant
focus on the negative. We believe existential psychology can help positive psychology emerge
from this dilemma and, more specifically, become more relevant to Army training and education.
The shortfalls and opportunities in the early stages of positive psychology are reminiscent of the
existential assessments in the psychology of Carl Rogers (May, 1983; Maslow, 1968).
Essentially, Rogerian theory appeared to be too optimistic or not sufficiently deep existentially to
illuminate the most profound influences on the growth and development of an individuals values
and identity. In a manner of speaking, existential psychology views the negative as a positive. To
be a bit more precise, the assumption is that the occurrences in which one experiences the
strongest and most immediate challenges to ones identity and existence have the greatest
influence on ones ability to transcend the moment while remaining grounded in the moment, to
grasp ones potential while understanding ones limits, to make choices with a sense of
responsibility for the outcomes, and to make a commitment to a particular kind of engaged
existence.
May (1983) distinguishes between three modes of existence or being in the world. One mode is
engagement with the surroundings reminiscent of the mutuality between the animal and
environment that is central to ecological psychology (J. Gibson, 1979). This is the domain in
which natural law applies (e.g., Kugler & Turvey, 1987; Shaw et al., 1982) and is accepted as
valid in existentialism (May, 1993; Solomon, 2004a). Like ecological psychology, existential
psychology considers an individuals relationship with the surroundings to be fundamental
ontologically. Reductionistic inquiry that separates subject from object is considered to be
marginal at best. Empirical inquiry must have ecological validity to yield an understanding of
human cognition and behavior that is existentially authentic and trustworthy. The predominance
of natural law in this mode of existence might suggest to some that it has a lower status relative to
modes that more directly address the social and cultural concerns of human beings. It is important
to note, however, that the modes of existence are simultaneous and constitute an existential
gestalt. Even the virtues are enhanced by ones engagement with the physical environment, most
notably through the development of coordination, control, and skill and the attendant pursuit of
mastery (cf., Aristotle, trans. 1925). Thus, in this mode of existence, there is an interesting
connection between the contributions of ecological psychology and related approaches to human
movement science (see e.g., Bernstein, Latash, & Turvey, 1996; Hancock, Flach, Caird, &
Vicente, 1995; Sternad, 2008) and positive psychologys homage to nicomachean ethics
(Peterson & Seligman, 2004).
A second and simultaneous mode of existence is in relationships with other individuals. Here too,
the whole is greater (and different) from the sum of the parts but not in the sense that one
surrenders ones identity to a collective (cf., Lewin, 1951). The sense of choice and responsibility
that is the hallmark of existentialism requires that this social field is a ubiquitous context with
respect to which one experiences the world and comes to understand the potential impact of ones
personal agency. To date, there has been relatively little work in ecological psychology on
interpersonal engagements but such modes of existence fit comfortably with ecological notions of
information-based coupling with the surroundings (e.g., J. Gibson, 1979; Riccio, 1993b;
Stoffregen & Bardi, 2001; Stoffregen & Riccio, 1988; Turvey et al., 1981). Neisser (1996)
summarizes some of the early research into the implications of ecological psychology for
108
interpersonal interactions. More generally, this mode of existence is addressed with great depth
and breadth in the theories related to social learning that we suggest for the scientific
infrastructure of OBTE (e.g., situated learning theory, self-efficacy theory, and self-determination
theory).
The third simultaneous mode of existence presumably is uniquely human in the sense that it
refers to ones ability to transcend ones own experience, to reflect on it, and to find meaning in
it. It is the mode in which, though engagement with the reality, individuals observe
manifestations of values and exemplify values in ones own behavior. It also is the mode in which
individuals appreciate the causal potency of the past, of the consequences of ones own actions
and the actions of others; and it is the mode in which one comes to appreciate the cascading
implications of ones own actions which inform choices and illuminate ones responsibility. It is
the mode of existence that ostensibly is the raison dtre for positive psychology. The difference
is that existential psychology has utilized a large and diverse body of work outside the sciences
that, in essence, provides a natural history of existentially significant engagements of human
beings with their surroundings and situations (see Solomon, 2004a). While it philosophically is
not limited to engagements that seem negative, it has such a focus because of the significance it
gives to ones capability to reflect on ones own mortality and morbidity as well as that of others.
In practice, is has the opposite problem of positive psychology. It seems that there would be
mutual benefit to a dialog between positive psychology and existential psychology in which, for
example, there is collaborative inquiry into the effects of existentially challenging events on the
development of virtues (cf., Greenberg, Koole, & Pyszczynski, 2004; Hoffman, et al., 2009;
Lopez & Snyder, 2009) and the behavioral manifestations of virtues in communities of practice
(cf., Riccio et al., 2004).
The application of existential psychology to education can benefit from lessons learned in its
application to clinical practice. In particular, existential therapy is not a set of techniques. It is a
framework for understanding human existence. Techniques logically follow this approach to
understanding; understanding does not follow merely from the application of techniques that
define how to do existential therapy. Thus, an existential approach allows for variability in the
techniques of therapy but the choice and disciplined use of a technique must make sense for the
situation at hand, it must find meaning in the engagement with the concrete reality of the moment.
Any exploration of the meaning of some historical or future occurrence must be grounded in the
concrete reality of the present, not the other way around. One cannot find meaning in the present
by attempting to ground it by remembering (i.e., a process as opposed to an occurrence) or by
imagining (i.e., mentally constructing or knowing about some hypothetical future occurrence).
Details matter, and one can engage with the details of reality only in the present. The present
always influences ones relentless march into the future whether one chooses to engage or not;
ones choices in the present are what one carries into the future.
A technique for interaction between a therapist and a patient is appropriate to the extent that it
helps ground ones modes of existence in the reality of the present without unnaturally limiting
ones experience to the present. If it doesnt, if it looks for meaning in the wrong places, it is
inappropriate and potentially counterproductive. The same principle applies to the instructorstudent interactions that are central to the principles and practices of OBTE. It is important to
note again that this is not simply an argument for experiential learning as commonly conceived. A
patient or a student is affected by experience of the situation more than merely by the amount of
time in the situation. In fact, these are existentially inseparable. Nevertheless they can be
separated pedagogically in a misguided implementation and misinterpretation of experiential
learning. One of the reasons that OBTE focuses on the instructor as mentor, advisor, and
facilitator is that students (as with patients) can benefit from inter-subjective guidance in
109
Riccio
collaborative engagements with the reality of the present. In existential psychology, this requires
that instructors become participants in the students reality. While this creates all sorts of tricky
problems for therapists and their patients, it is much less problematic in an instructional setting.
The link between existential therapy and education is striking in the following passage by Rollo
May: The therapist is what Socrates named the midwifecompletely real in being there, but
being there with the specific purpose of helping the other person to bring forth something from
within himself (May, 1983, p. 161). Existential psychology thus requires authentic participation
of the instructor, not as a character in a scripted play nor, worse, as a narrator. Collaborative
reflection with the student avoids the hollowness of a subject-object distinction and a subjectiveobjective dichotomy when there is true engagement of the instructor. The content of collaborative
reflection is not, however, a reliable indicator of collaborative engagement. Mere descriptions of
the principles and concepts of existentialism are so resonant, for example, especially in an
existentially poignant context such as Army training and education, that one can delude oneself
about the authenticity of ones engagement (May, 1983). The interpersonal mode of existence
implies that all participants in a relationship are changed by the relationship. Awareness of this
proposition will help instructors notice an otherwise subtle drift into detachment.
To the extent that the student is aware that the instructor is a participant in the struggle, they will
be less inclined to assume that they must find meaning in their engagements though a process of
introjection in which they merely take on the values of the instructor. Existential commitment
requires that one choose how to engage with the world and find meaning in ones actions, to take
responsibility for ones choices (May, 1983; Solomon, 2004a). Instructors can exemplify valuesbased behavior but it is for the student to be moved and to choose to internalize such values or to
come to understand how to translate shared values into action. This need not and should not be
the result of an almost paralyzing analysis or protracted deliberation. It can have the immediacy
of direct perception if presented in a salient context or in a significant emotional event. It can be
attended by a sense of engaged vitality rather than the hangover of an event with high mental
workload (cf., Ryan & Deci, 2008; see also, Chapter 11).
The directness of the connections between ones choices, ones actions, and shared values is
enabled by ones understanding of ones place in a community of practice within which meaning,
in a sense, is negotiated (Wenger, 1998). Such negotiation should not be view as a compromise
among different perceptions of a shared reality and certainly not as accommodation of an
indiscriminant cultural relativity. Instead, it is more akin to a crystallization of different
perspectives; complementary perspectives that help one negotiate the boundaries between the
known and unknown and that provide principled reasons to momentarily and partially cede power
to someone or something else (see section 3.2.1). Without such negotiation and the responsibility
that this implies, ones choices simply connote marginality and, even without ceding power, ones
powerlessness. This subtle yet profound realization is the core of the Armys established practice
of task organization.
4.4 Building on the Scientific Infrastructure for OBTE
4.4.1 Triadic Frameworks
Table 2 summarizes the relationship between the various lines of scholarship in the scientific
infrastructure for OBTE in terms of the triadic causal framework of OBTE (e.g., the intangibles
of confidence, initiative, and accountability). The correspondence among these triadic
frameworks is more like a projective transformation than a one-to-one mapping (cf., Wyszecki &
Stiles, 1982). The relationships thus are a useful guide to the ways in which the lines of
110
scholarship complement one another as opposed to an attempt to reduce them to a lowest

common denominator. It also should be noted that there is not a one-to-one relationship between
the triadic causal framework of OBTE and the three pillars of the scientific infrastructure for
OBTE. In fact, they are categories that are different in kind. The triadic causal framework is
about personal epistemology while the three pillars are about scientific epistemology. This is why
a table is used to describe the relationship.
Table 2: Triadic relationships among pillars of a scientific foundation for OBTE
OBTE Intangibles
Positive Psychology
Self Determination Theory
Self-Efficacy Theory
Situated Learning Theory
Ecological Psychology
Existential Psychology
Confidence
Pleasantness
Competence
Person
Identity
Nested Self
Existence
Initiative
Engagement
Autonomy
Behavior
Participation
Perception-Action
Choice
Accountability
Meaning
Relatedness
Environment
Community
Affordances
Responsibility
One might ask why we describe three pillars to the scientific foundation for OBTE instead of six
or more. The reason is that we believe there are only three fundamentally independent
components to the object of an integrated scientific inquiry with relevance to OBTE. They
correspond roughly to the three simultaneous modes of existence (ontological categories) in
existential psychology: (a) reciprocal relationships with the surroundings dominated by natural
law, (b) reciprocal interpersonal relationships that are directly perceivable, (c) and transcendent
meaning. In principle, an experimental existential psychology ultimately could address all the
scientific pillars of OBTE. In practice, we must rely on the historical body of theory and
observations from existing lines of the scholarship that collectively have addressed the three basic
ontological categories and that heretofore have not been integrated.
As pillars, we chose lines of scholarship with the most extensive bodies of theoretical and
empirical evidence that are relevant to the claims of OBTE. Much more important than the
converging evidence that this provides for the claims of OBTE, however, is the guidance it
provides for the further development of OBTE and an integrated approach to leadership,
education, training, and service (see Epilogue). The scientific infrastructure for OBTE suggests
concrete topics of conversation about the integration of leadership, education, training, and
service. It provides guidance on what should be observed and what can be observed in rigorous
ways and that have actionable implications that are both valid and verifiable. In this respect, it is
noteworthy that the three pillars also relate to the triad of situated behavior, situated experience,
and situated meaning (epistemological categories that are inseparable ontologically), although
again, not in one-to-one mapping. These categories help us identify how to observe or measure
engagements that matter in learning and development.
Positive psychology focuses on self-reports but with an interest in the deep meaning of an
individuals circumstances or state of mind. It does not address cause-effect relationships with a
depth and breadth sufficient to craft effective settings for training and education. In this respect,
self-determination theory and self-efficacy theory complement positive psychology and they also
complement each other. Self-efficacy theory has more to say about the broad range of
environmental characteristics that are conducive to learning and development, especially the
collaborative learning and development that is central to OBTE. Self-determination theory
provides more of an appreciation of intervening variables, the proximate motivational influences
on learning and development, and their potential link to values (see e.g., Lopez & Snyder, 2009).
Self-efficacy theory, self-determination theory, and positive psychology together can help in
111
Riccio
coming to a better understanding of best practices in training and education that foster
development of the intangibles, their employment as sources of motivation, and achievement of
the nested outcomes that they enable (see section 3.2).
Self-determination theory and self-efficacy theory also focus on experience that can be described
in self-reports. Fundamental to our multifaceted approach to inquiry into OBTE is the proposition
that scientific methods must balance experience and behavior. Experience underscores the
meaning of situated behavior, that is, engagements with the environment. Situated behavior
grounds the description of experience in reality. Reliable observation of behavior is important for
a scientific appraisal of the credibility and trustworthiness of self-reports but it also is valuable for
collaborative reflection and inter-subjective engagements with the environment. Selfdetermination theory and self-efficacy theory thus are balanced by behavioral observations
conducted with an ecological sensibility, that is, with an eye toward reciprocal influence in an
individuals engagement with the physical and social environment. Behavioral observations can
come to play a similarly important role in positive psychology because, existentially, values or
virtues have meaning only to the extent that they translate into action (cf., Riccio, et al., 2004).
Methodologically, the whole is greater than the sum of the parts if there is a coherent theoretical
framework to guide the selection, employment, and interpretation of observations and
measurements.
Figure 9. Three pillars of a strong theoretical and empirical foundation for continuing
development of outcomes-based training and education. Links in the superstructure
(triangle) represent other important lines of research that can help make connections
among the pillars that heretofore have evolved independently of each other.
112

4.4.2 Further Development
Our initial identification of scientific foundations for OBTE in this chapter has focused on
observable interactions with the physical and social surroundings and on the meaning they have
for an individual. The three pillars and connecting lines of research begin to establish a balance
among the lines of scholarship that address behavior and those that address conscious experience.
To come to a better understanding of micro-experiences that are so important in OBTE, we need
to delve more deeply into conscious experience, and we need to do so in a way that is grounded in
behavior and interactions with the surroundings. We need to push forward into lines of
scholarship that address the details of emotion, thinking, and decision-making. More specifically,
we need to reveal scientific underpinnings for the interplay of passion and reason (see Chapter 3,
section 3.2). Relevant lines of scholarship are discussed in Chapter 5.
4.5 References
Anderson, J., Bothell, D., Byrne, M., Douglass, S., Lebiere, C., & Qin, Y. (2004) An integrated
theory of Mind. Psychological Review, 111, 1036-1060.
Augustine (1953). The teacher. In: J. Burleigh (Trans. & Ed). Augustine: earlier writings.
Philadelphia, PA: The Westminster Press. (Original work written ca. 389)
Aspinwall, L. & Staudinger, U. (Eds.) (2003). A psychology of human strengths: Fundamental
questions and future directions for a positive psychology. Washington, DC: American
Psychological Association.
Bailar, J. (1997). The promise and problems of meta-analysis. New England Journal of Medicine,
337(8), 559-561.
University Press.
Company.
Bernstein, N., Latash, M., & Turvey, M. (1996). Dexterity and Its Development (Resources for
Ecological Psychology). Mahwah, NJ: Lawrence Erlbaum.
Boring, E. G. (1950). A history of experimental psychology. New York: Appleton, Century,
Crofts. (Original work published 1927)
Brown, K., Ryan, R., & Cresswell, J. (2007). Mindfulness: Theoretical foundations and evidence
for its salutary effects. Psychological Inquiry, 18(4), 211237.
Bunge, (1977). Levels of reduction. American Journal of Physiology: Regulatory, Integrative and
Comparative Physiology, 233(3), R75-R82
Canudas de Wit, C.A. (1988). Adaptive control for partially known systems: theory and
applications. Amsterdam: Elsevier.
Chalam, V.V. (1987). Adaptive control systems: Techniques and applications. New York: Marcel
Dekker.
113
Riccio
Chemero, A., & Turvey, M. (2007). Gibsonian affordances for roboticists. Adaptive Behavior,
15(4), 473-480.
development f higher psychological processes (L.S. Vygotsky). Cambridge, MA: Harvard
University Press.
Davids, K., Bennett, S., & Newell, K. (2005). Movement system variability. Champaign, IL:
Human Kinetics.
Deci, E. & Ryan, R. (1985). Intrinsic motivation and self-determination in human behavior. New
York: Plenum.
Deci, E. & Ryan, R. (2000). The what and why of goal pursuits: Human needs and the selfdetermination of behavior. Psychological Inquiry, 11 (4), 227268.
Deci. E. & Ryan, R. (2008). Self determination theory: a macrotheory of human motivation,
Dennett, D. (1971). Intentional systems. Journal of Philosophy, 68, 87-106.
CA: Sage.
Fajen, B., Riley, M. R., & Turvey, M. T. (2009). Information, affordances and control of action in
sports. International Journal of Sports Psychology, 40, 79-107.
Filatov, N.M. & Unbehauen, H. (2004). Adaptive dual control: Theory and applications. New
York: Springer.
Flyvbjerg, B. (2001). Making social science matter: Why social inquiry fails and how it can
succeed again. Cambridge, UK: Cambridge University.
Frank, T.D., Michelbrink, M., Beckmann, H., Schollhorn, W.I. (2008). A quantitative dynamical
systems approach to differential learning: Self-organization principle and order parameter
equations. Biological Cybernetics, 98(1), 19-31.
Freeman, J., Sidman, J., Aten, T., Diedrich, F., Cooke, N., Winner, J., Rowe. L., & Riccio, G.
French, D., (2002). Wake up and smell there is something wrong. News From the Front, SeptOct 2002. 26-29. Ft. Leavenworth, KS: Center for Army Lessons Learned.
Georgas, J., van der Vijver, F., & Berry, J. (2004). The ecoculture framework, ecosocial indices,
and psychological variables in cross-cultural research. Journal of Cross-Cultural Psychology,
35, 74-86.
Gibson, E., Riccio, G., Schmuckler, M., Stoffregen, T., Rosenberg, D., & Taormina, J. (1987).
Detection of the traversability of surfaces by walking and crawling infants. Journal of
Experimental Psychology:Human Perception and Performance., 13, 533-544.
Review, 62, 32-51.
Glasziou, P., & Haynes, B. (2005). The paths from research to improved health outcomes. ACP
Journal Club, 142(2), A8-10.
114
Godfrey-Smith, P. (2003). Theory and reality. Chicago, IL: University of Chicago Press.
Gray, w. (Ed.) (2007). Integrated models of cognitive systems. Oxford, UK: Oxford University.
Greenberg, J., Koole, S., & Pyszczynski, T. (2004). Handbook of experimental existential
psychology. New York: Guilford Press.
Gutman, P. (2001). Robust and adaptive control: Fidelity or a free relationship? In: S.O. Reza
Hacking, I. (1999). The Social Construction of What? Cambridge, MA: Harvard University.
Hancock, P., Flach, J. Caird, J., & Vicente, K. (1995), Local applications of the ecological
approach to human-machine systems. Hillsdale, NJ: Erlbaum.
Headquarters Department of the Army (2003). Mission command: Command and control of Army
forces. FM 6-0. Washington, DC: Department of the Army.
Heft, H. (2001). Ecological psychology in context: James Gibson, Roger Barker, and the legacy
of William Jamess radical empiricism. Mahwah, NJ: Lawrence Erlbaum.
Hickman, L.A. & Alexander, T.M. (1998). The essential Dewey, Volume 1: pragmatism,
education, and democracy. Bloomington, IN: University of Indiana Press.
Hoffman, L., Yang, M., Kaklauskas, F., & Chan, A. (Eds.) (2009). Existential psychology eastwest. Colorado Springs: University of the Rockies.
Hume, D. (2009). A treatise of human nature. Breibigsville, PA: Merchant Books. (Original work
published 1740).
Kaufman, J., Lanik, G., Spense, R., & Trager, E. (1992). Operating experience feedback report
human performance in operating events: commercial power reactors. NUREG-1275, 8.
Washington, DC: U.S. Nuclear Regulatory Commission.
Keatinge, M. (1896). The great didactic of John Amos Comenius. Whitefish, MT: Kessinger
Publishing.
Keyes, C. & Haidt, J. (2003). Flourishing: Positive psychology and the life well-lived.
Washington, DC: American Psychological Association.
Kilcullen, D. (2006). Twenty-eight articles: Fundamentals of company level counterinsurgency.
Retrieved August, 2006, from the Joint Information Operations Center Web site:
http://www.au.af.mil/info-ops/iosphere/iosphere_summer06_kilcullen.pdf
Kugler, P., & Turvey, M. (1987). Information, natural law, and the self-assembly of rhythmic
movement. Mahwah, NJ: Lawrence Erlbaum.
Lakkala, M., Muukkonen, H., Paavola, S., & Hakkarainen, K. (2008). Designing pedagogical
infrastructures in university courses for technology-enhanced collaborative inquiry. Research
and Practice in Technology Enhanced Learning, 3(1), 33-64.
Latour, B. (2004), Why has critique run out of steam? From matters of fact to matters of concern,
Critical Inquiry, 30, 225-248.
115
Riccio
Lopez, S. & Snyder, C. (2009). Oxford Handbook of Positive Psychology. Oxford, UK: Oxford
University.
Maslow, A. (1968). Toward a psychology of being. New York: Wiley.
Meigs, M. C. (2003). Unorthodox thoughts about asymmetric warfare. Parameters, 33(2), 4-18
May, R. (1983). The discovery of being. New York: W.W. Norton & Co.
May, R., Angel, E., & Ellenberger, H. (1958). Existence: A new dimension in psychiatry and
psychology. New York: Clarion.
Menand, P. (2001). The metaphysical club: A story of ideas in America. New York: Farrar,
Strauss, & Giroux.
Mumaw, R., Roth, E., Vicente, K., & Burns, C. (2000). There is more to monitoring a nuclear
power plant than meets the eye. Human Factors, 42(1), 36-55.
Narendra, (1986). Adaptive and learning systems: Theory and applications. New York: Plenum
Press.
Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University.
Newell, A., Simon, H., & Shaw, J. (1958). Elements of a theory of human problem solving.
Psychological review, 65, 151-166.
Paavola, S. & Hakkarainen, K. (2005). The knowledge creation metaphor An emergent
epistemological approach to learning. Science and Education, 14(6), 535-557.
Petraeus, D. H. (2006). Learning counterinsurgency: Observations from soldiering in Iraq.
Military Review, Jan-Feb, 2-12.
Peterson, C., & Seligman, E. P. (2004). Character, strength, and virtues. Oxford, UK: Oxford
University Press.
Plato (1977a). Phaedo. In: R.M. Hutchins et al. (Eds.). Plato. Chicago, IL: Encyclopedia
Britannica. (Reprinted from The Dialogues of Plato, by B. Jowett, Trans., 1892a, Oxford,
Plato (1977b). Symposium. In: R.M. Hutchins et al. (Eds.). Plato. Chicago, IL: Encyclopedia
Britannica. (Reprinted from The Dialogues of Plato, by B. Jowett, Trans., 1892b, Oxford,
Science, 27, 183-213.
Reed, E. (1996). Encountering the world: Toward and ecological psychology. New York: Oxford
University.
Reed, E. (1997). From Soul to mind: The emergence of psychology, from Erasmus Darwin to
William James. New Haven: Yale University.
Reed, E., & Jones, R. (1979). J.J. Gibsons ecological revolution. Psychology of the Social
Sciences, 8, 189-204.
116
Reed, E., & Jones, R. (1982). Reasons for realism: selected essays of James J. Gibson. Hillsdale,
NJ: Lawrence Erlbaum.
Riccio, G. (2003). Program summary report. Report to Natick Soldier Center for Objective Force
Warrior program (Agreement DAAD16-02-9-0002). Menlo Park, CA: Exponent
Inc/Wolfpack Enterprise.
Riccio, G.E. & Stoffregen, T.A. (1988) Affordances as constraints on the control of stance.
Human Movement Science, 7, 265-300.
Riccio, G. & Vicente, K. (2001). Coping with change and novelty in energy facility operations:
Recommendations for knowledge-based, extra-procedural problem solving. In: M. Gross &
T. Ayes (Eds.), Electrical Power Research Institute Report 1004666. Palo Alto, CA: EPRI.
Richardson, M. J., Shockley, K., Riley, M. R., Fajen, B. R., & Turvey, M. T. (2008). Ecological
psychology: Six principles for an embodied-embedded approach to behavior. In P. Calvo &
T. Gomila (Eds.), Handbook of cognitive science: An embodied approach (pp. 161-190).
Oxford, UK: Elsevier.
Roth, E., Mumaw, R., & Lewis, P. (1994). An empirical investigation of operator performance in
cognitively demanding simulated emergencies. NUREG/CR-6208. Washington, DC: U.S.
Nuclear Regulatory Commission.
702717.
Safonov, M. (2001). Recent advances in robust control, feedback, and learning. In: S.O. Reza
Scardamalia, M., & Bereiter, C. (2005). Does education for the new knowledge age need a new
kind of science? European Journal of School Psychology, 3(1), 2140.
Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology.
In K. Sawyer (Ed.), Cambridge Handbook of the Learning Sciences (pp. 97-118). New York:
Cambridge University Press.
Schrim, S. & Caterino, B. (2006). Making political science matter: debating knowledge,
research, and method. New York: New York University.
Scott, J. (2000). Social network analysis: A handbook. London, UK: Sage.
Seligman, M., Steen, T., Park, N., & Peterson, C. (2005). Positive psychology progress:
Empirical validation of interventions. American Psychologist, 60(5), 410421.
Shaw, R., Turvey, M., & Mace, W. (1982). Ecological psychology: The consequence of a
117
Riccio
Sheridan, T. (1999). Descartes, Heidegger, Gibson, and God: Toward an eclectic ontology of
presence. Presence, Vol. 8(5), 551559.
Smith, A. (1976). A theory of the moral sentiments. New York: Oxford University Press.
(Original work published 1759).
Solomon, R. (Ed.) (2004a). Existentialism. Oxford, UK: Oxford University Press.
Sternad, D. (2008). Progress in motor control: A multidisciplinary perspective (Advances in
Experimental Medicine and Biology, No. 5). New York: Springer.
Stoffregen, T., & Bardy, B. (2001). On specification and the senses. Behavioral and Brain
Sciences, 24, 195213.
Stoffregen, T., & Riccio, G. (1988). An ecological theory of orientation and the vestibular
system. Psychological Review, 95, 3-14.
Swales, J. (2000). The troublesome search for evidence: Three cultures in need of integration.
Journal of the Royal Society of Medicine, 93, 402-407.
Thelen, E., & Smith, L. B. (1994). A dynamic systems approach to the development of cognition
and action. Cambridge, MA: MIT Press.
Routledge.
Treisman, A. (1988). Properties, parts, and objects. In: Boff, K., Kaufman, L., & Thomas, J.
(Eds). Handbook of Perception and Human Performance, Volume II: Cognitive Processes
and Performance (35-1 35-70). New York: John Wiley and Sons.
Tschacher, W., & Dauwalder, J. (Eds.) (2003). The dynamical systems approach to cognition:
Concepts, and empirical paradigms based on self-organization, embodiment, and
coordination dynamics. River Edge, NJ: World Scientific.
Turvey, M. T. (2007). Action and perception at the level of synergies. Human Movement
Science, 26, 657-697.
Turvey, M., Shaw, R., Reed, E. & Mace, W. (1981). Ecological laws of perceiving and acting: In
reply to Fsdor and Pylyshyn (1981). Cognition, 9, 237-304.
Wegen, E., van Emmerik, R., & Riccio, G. (2002). Postural orientation: Age-related changes in
variability and time-to-boundary, Human Movement Science, 21(1), pp. 6184.
Wang, Q. (2006). Culture and the development of self knowledge. Current Directions in
Psychological Science, 15(4), 182-187.
Warren, R. & Wertheim, A. H. (Eds.) (1990). Perception and control of self-motion. Hillsdale,
NJ: Lawrence Erlbaum.
Wise, G. (1980). American historical explanations: A strategy for grounded inquiry.
Minneapolis, MN: University of Minnesota.
Wolfram, S. (2002). A new kind of science. Champaign, IL: Wolfram Media.
Wyszecki, G., & Stiles, W. (1982). Color science: Concepts and methods, quantitative data and
formulae. New York: Wiley.
118
Passion & Reason

Chapter 5. Passion and Reason in Values-Based Learning & Development
Gary Riccio
The scientific infrastructure for OBTE provides an integrated view of situated behavior, situated
experience, and situated meaning in an individuals engagement with the world. The methods of
the recommended lines of scholarship will be helpful in assessing both the practices of OBTE, its
immediate influence, and its long-term influence. There is, however, a gap in that conscious
experience and values generally are assessed after an event while behavior can be assessed during
an event. It may be difficult to assess experience while it is unfolding but it should nevertheless
be taken into consideration so that we might come to a deeper appreciation of the uniquely human
ways in which conscious experience and concurrent behavior are meaningfully intertwined. This
chapter addresses these important issues by reviewing research that is relevant to conscious
experience and meaning and that, at least in principle, can be grounded in behavior. In doing so,
we delve more deeply into what it means for an individual to be situated in a meaningful context,
we explore the reciprocal influences between passion and reason, and we re-examine classical
and cutting-edge scholarship on inter-temporal decision making. We conclude with a brief
glimpse beyond science into sources of scholarship that can be considered in a more
comprehensive approach to the outcomes that are influenced by all teaching and learning.
5.1 The Nested Self
5.1.1 An Alternative to Individual versus Collective
There is a consistent theme throughout our discussion of the grounded theory for OBTE. We
emphasize that the fundamental unit of analysis is not the individual but rather the reciprocal
causal relationship, a coupling, between the individual and the surroundings. This could lead to
the misinterpretation that we are arguing that the individual is subordinate to a collective or that
one should surrender ones identify to a group or a thing. This is definitively not the case. Choice
and responsibility ultimately resides with the individual, existence has a deeply personal meaning,
and such concepts are at the core of OBTE. At the same time, the intent of OBTE is not to foster
egocentrism or narcissism. Nor does OBTE strive for some balance between the individual and
the collective, nor some midpoint within a simplistic notion of an individual-collective
continuum.
The distinction between individual and collective has become a popular construct in the social
sciences (Hofstede, 2001; Hofstede, Pedersen, & Hofstede, 2002). OBTE is unconcerned with
classification of personality types, especially decontextualized classification schemes. Individuals
are always coupled with their surroundings through information or exchanges of energy (J.
Gibson, 1979; Kugler & Turvey, 1987; Shaw et al., 1982), multiple couplings exist at the same
time (Riccio, 1993b), and they change from moment to moment primarily as a function of the
tasks in which one is involved (Riccio 1993a). This perspective on individual engagements with a
broader culture is more consistent with recent research that addresses variability in behavior and
experience that is driven by the particulars of a situation and the surroundings in which an
individual is embedded (Matsumoto &Yoo, 2006; Heine & Noranzayan, 2006; cf., Idel &
McGinn, 1999). Thus, to the extent that it makes sense to conceptualize a continuum from
individual to collective, we assume that individuals exist as a changing spectrum along this
continuum. The richness of ones experience then would be related directly to ones ability to
appreciate the variety in this spectrum of engagement (Figure 1).
119
Riccio
Figure 1. Reciprocal influences in an individuals interactions with social environment as

with the physical environment (see Chapter 4). 1st-order others share the same physical
context. 2nd-order others are engaged largely with a different context. Presumably
coordination with such individuals is guided by more abstract commonalities such as
shared values. Such values exemplify a broader community that includes 3rd-order others
with whom one momentarily is not interacting.
The implication for a situated pedagogy is that an instructor should consider the multiplicity of
connections between a student and the context of a learning event above and beyond the elements
of the learning environment over which the instructor has direct control (Lave & Wenger, 1991;
Wenger, 1998; see Chapter 4, section 4.3.2; see also Bransford et al., 2000). Instructors should
consider this context of broad engagement because, whether or not they do, the students will. This
need not imply that instructors allow themselves to be overwhelmed with the reality of the
potentially dizzying freedom of a students will. At the very least, awareness of this reality can
sensitize an instructor to incidental occurrences that provide clues about the extent to which a
student is engaged in the learning event and the extent to which the students behavior
exemplifies choice and responsibility. In special circumstances such as basic training in the
Army, these considerations help bring together a mentors roles as an instructor and as a
counselor. More generally, it brings the considerations of leadership and leader development
more directly into the realm of training and education (United States Army Training and Doctrine
Command [TRADOC], 2009).
5.1.2 Cognition and Reality
The preceding discussion implies that there is a multiplicity to the self, not in the sense of the
neurotic condition of multiple personalities but in the sense of a persistence and change in a
coherent nesting of reciprocal relationships with the surroundings. We use the concept of nesting
in the sense of J. Gibson: they would constitute a hierarchy except that this hierarchy is not
categorical but full of transitions and overlaps. (J. Gibson, 1979, p. 9). Neisser and his
colleagues have systematically treated the coherent multiplicity of self from a wide variety of
scholarly perspectives including but extending beyond ecological psychology (e.g., Neisser,
1993, Neisser & Fivush, 1994; Neisser & Jopling, 1997). At the outset of this program, Neisser
described five kinds of self-knowledge. The first two are the ecological self and the
Passion & Reason
120
interpersonal self, modes of existence that correspond to two of three modes held to be
fundamental in existential psychology. The temporally extended self enriches the ecological
and the interpersonal modes of existence by addressing persistence and change over periods of
time that are greater than what colloquially we refer to as the span of attention. It is an aspect of
the transcendent mode of existence in existential psychology but just one aspect of it (see Chapter
4, section 4.3.3).
The conceptual self gets at the essence of the transcendent mode in that it reflects ones beliefs
about oneself based on interactions with the physical and social surroundings and on shared or
negotiated meaning within nested communities in which one has membership (cf., Bandura,
1997; Wenger, 1998; see Chapter 4, section 4.3.2). Finally, the private self relates to the
inescapable fact that ones existence is ones own. While this does not relate to the fundamental
modes of existence as such, it emphasizes the causal potency of situations that lead one to
consider ones own death or catastrophic change in ones identity. The reason for mentioning
Neissers program is not the mere correspondence with concepts we have already addressed in
Chapters 3 and 4, such as related triadic frameworks and the simultaneous modes of existence.
His work fosters deeper understanding and further development of OBTE because it is a
multifaceted inquiry that points to significant bodies of empirical work in the social and
behavioral sciences that are directly relevant to largely neglected issues in Army training and
education and to which OBTE is a self-conscious and organized response. Beyond this, Neissers
program provides an inspiring example of crystallization of a wide variety of scientific
perspectives and much more in that it is not limited to science per se. One is left with the
impression that this is a necessity rather than an academically interesting diversion. This
reinforces our own conclusion that science is not sufficient to understand the meaning of OBTE
for instructor and students, and certainly not to appreciate the various perspectives of the
progenitors and current champions of OBTE. We thus follow Neissers lead by suggesting a
similar undertaking for OBTE (see section 5.5).
While Neissers program is useful insofar as it touches all three pillars of the scientific foundation
for OBTE, its most unique contribution is the body of work on the temporally extended self.
Most people have a tendency to think about memory, for example, by way of analogy to storage
of information in daily life, such as audio or video recordings and electronic storage media
associated with computers. The central issues in this analogical thinking are how much can be
stored, how difficult it is to access such stored information, and the extent to which it is accurate
or reliable. In ecological psychology, the emphasis is on the existentially richer and more
significant activity of remembering. Unlike the analogy to inanimate storage media, remembering
is a constructive activity in which one revisits an experience from a new and more current
perspective, seeing with new eyes as it were (Neisser, 1993; Neisser & Hyman, 2000). While this
invokes the concept of cognitive constructivism, it is important to note that we refer to a
constrained construction that is grounded in reality in two ways. First, it is based on ones actual
experience with the real world. It may lead to a remembered experience that is quite different
from a prior experience because the relationship is different; the person involved in the
experience is different. So what good is a remembered experience that is not an exact replica of a
prior experience? One possibility is that it educates the attention; it prepares one to engage the
world in a different way. This leads to the second way in which constructive remembering is
grounded in reality. It influences how we engage the real world before us, in the present.
Constructive remembering that has little validity in the present will quickly be vitiated by direct
perception of ones engagement with the physical and social surroundings (cf., May, 1983).
The profound difference between the natural remembering of human beings and the memory
storage and retrieval of inanimate objects and systems has direct relevance to OBTE. Recall that
121
Riccio
the best practices of OBTE are organized in terms of planning, execution, and review (see e.g.,
Chapter 2; Appendix A). After-action reviews (AAR) are important in OBTE as they are in the
Army in general. This activity can be trivialized by a mechanistic view of memory, but it can
become developmentally significant by an understanding that is existentially more valid. One can
reiterate a sequence of occurrences from an artificially detached perspective or one can find
meaning after the fact as a result of being changed by the experience or otherwise by revisiting
the experience from a new perspective. Collaborative reflection is the easiest way to come to
appreciate a different perspective on a shared experience. Collaborative reflection is the intent of
an AAR but the potential rarely is realized. The trivializing notion of recalling memories from
storage often leads instructors to de-prioritize the AAR, to give it short shrift when there is time
pressure, and to lapse into a script that neglects any discovery learning that may have occurred.
Figure 2. Stylized depiction of the notions of self in the context of reciprocal influences
between an individual and both momentary and temporally extended situations. The
vertical axis essentially is a level of abstraction (cf., Rasmussen, 1997) with increasingly
more pervasive communities. The horizontal axis is time that has no independent
meaning psychologically, not linear physical time, in that it recognizes the reciprocal
influence between ongoing experience and prior experience. The relationships
represented in this figure have direct implications for design and assessment of
instruction (see Epilogue; cf., Bransford et al., 2000; Wenger, 1998).
There are two potential benefits of collaborative reflection that are existentially significant. The
first is that the inter-subjectivity of co-participants in an event is a powerful constraint on the
potential errors or potential meanderings of inductive inference (cf., Neisser & Fivush, 1994).
This is not to say that the experience of every participant must converge onto a common
experience. Instead we assume that multiple perspectives on the shared experience must fit in the
sense or crystallization. At its best, this activity of crystallization during collaborative reflection is
122
Passion & Reason
not merely passive learning of someone elses experience or introjection of the meaning that the
shared event had for another person. It can be, in a sense, a re-experiencing of the event from a
different perspective but grounded in ones own reality. It can be another opportunity for new
learning, not just reinforcement of whatever learning had already occurred. Our observations in
many different settings over many years have underscored the power that AAR can have with
respect to motivation and learning when collaborative reflection is taken seriously and
approached within a framework of constructive remembering and inter-subjective crystallization
(cf., Bruny, Riccio, Sidman, Darowski, & Diedrich, 2006; Neisser & Fivush, 1994; Riccio,
Sullivan, Klein, Salter, & Kinnison, 2004). It can have a dramatic effect on what students and
instructors take away from a learning event.
5.2 Conscious Experience and the Dynamics of Thinking
Typically the relationship between behavior and conscious experience is viewed in terms of the
effects of the former on the latter (see e.g., Riccio, 1993a,b). From a less dualistic perspective,
behavior and conscious experience can be viewed as different facets of ones engagement with
the world, each with a characteristic kind of causal potency. The power of conscious experience
certainly is not limited to positive or negative reinforcement for particular behavior or to an
associated capability for reactive adaptability. We are interested in experience as a reflection of
the dynamics of choice, as a window into the factors affecting the directedness of thinking and
personal agency. We believe this is important if instructors are to develop a better understanding
of how to foster the development of confidence, initiative, and accountability in their students.
Our interest is more aligned with research on naturalistic decision-making than with consciously
protracted or laborious algorithmic approaches to decision making (Klein, 1989; Klein, 2008;
Klein, Orasanu, Calderwood, & Zsambok, 1993; Klein, Ross, Moon, et al., 2003) but our focus is
different. We are more interested in the nature of experience during intentional behavior than in a
commitment to particular models for the ways in which the circumstances of contemporaneous
and prior experience influence decision-making. These considerations are potentially convergent,
however, to the extent that ones experience of a situation reflects the circumstances of ones
experience and helps explain the commitment one makes to action in the situation (cf., Klein,
2008; May, 1983).
The subtlety of experience to which we would like to orient some scientific and pedagogical
attention concerns the dynamics of thinking. We use the word dynamics here, as in our other
references to dynamical systems. We are referring not merely to change and transition. Our use of
the term is deeper in that it refers to the directedness of a process and the objectives of
engagement along with the associated attractions of equilibrium and the exigencies of stability
(cf., Henle, 1971; Lewin, 1951; Riccio, 1993a,b; Riccio & Stoffregen, 1988, 1991). A radical
implication of this perspective is that it is worthwhile to consider what can be observed and
communicated about the product-oriented processes of thinking. It is a point of entry for a
pedagogy that is relevant to productive thinking (cf., Wertheimer, 1945/1959). The conjecture
of scholarly reflection on the dynamics of thinking is that awareness of such dynamics can be
useful to the thinker and that it can be a consideration in inter-subjective coordination even
though ones awareness of the dynamics of thinking may seem to be difficult to put into words or
difficult to reduce to tidy algorithms. Wertheimer presented cogent arguments for taking this
seriously, for example:
123
Riccio
Many are of the opinion that men do not like to think; that they will do much to
avoid it; that they prefer to repeat instead. But in spite of many factors that are
inimical to real thinking, that suffocate it, here and there it emerges and
flourishes. And often one gets the strong impression that men, even children,
long for it. [Wertheimer, 1945/1959; pp. 1-2]
It is important to emphasize that we are neither arguing for practices that would lead individuals
to become lost in thought, nor are we interested in inquiry into cognitively laborious operations.
Instead, we are concerned with heightened awareness of ones engagements with the real world
and ones agency in it. We are interesting in the clarity of mind that expedites action (as opposed
to a paralyzing ambiguity) by illuminating personal choice, responsibility, and the balance
between them that gives meaning to ones existence (cf., Camus, 1942/1988, 1956). Before we
can embark on such an existentially significant enterprise, we must begin with an appreciation of
the aspects of productive thinking with respect to which individuals plausibly can develop better
awareness. The most important early work in this area was conducted by Max Wertheimer (e.g.,
Wertheimer, 1945/1959) and by the Wurzburg school early in the 20th century (see Boring, 1950;
Humphrey, 1951; Wertheimer, 1970). This work was a counterpoise to theories of logic and
association that dominated most of the 20th century in psychology and philosophy. Observable
characteristics of thinking emphasized by this research include.
Inklings (imageless thought or determining tendencies): Feeling that one is on the verge
of a solution or a productive line of thinking. We can set aside for a moment the question
about the generative power of such mental phenomena (something that is difficult if not
impossible to study experimentally). The point here is that, at the very least, it has an
emotional and motivational quality that is worth considering.
Perspectival insight (paradigm shift, phase change, self organization): A change in

perspective that is traceable to some stimulating event (e.g., interaction with another
person) and that makes one aware of characteristics of a situation or relationships that one
previously did not appreciate, often attended by inklings or a sense that the new approach
is pregnant with possibilities.
Gaps (incompleteness, shortfalls, antithesis, conflict): Awareness of limits or problems to

one's current approach to a problem or situation that has the power to influence or redirect one's approach (thought or action).
Ambiguity (confusion, lack of clarity, disorientation): A conspicuous lack of

understanding in situations that cannot be ignored. A persistently unpleasant feeling that
begs to be resolved. Unlike gaps, this may not provide a clear re-direction of one's
thought or action but it may motivate exploratory thought and action (e.g., queries of
others).
Seductive simplification (superficial or presumptive conclusions): The tendency to

check the box or otherwise just get beyond the problem at hand because it is unpleasant,
frustrating, or unsatisfying. This is a feeling often apparent in one's attempt to resist it
because of a sense that the simple conclusion is inadequate.
Passion & Reason
124
Clarity of end state (reverse planning, criteria for completion, goodness of a solution or
a practice): This is noteworthy when the means to the end is not immediately obvious.
Epiphany (surprise, sudden insight or clarity). A satisfying sense of qualitative

developmental progress. This also could be a counterpoint to any of the above.
These kinds of observables, for what Neisser refers to as the private self, have shared relevance
in the context of coordinated action in a task-organized group, and especially in the context of
collaborative reflection. Hunches and intuitions, or other feelings that are difficult to put
conclusively into words, can be fodder for collaborative decision-making. Others may be able to
bring to conclusion what one cannot. Others may be able to pick up circumstantial or nonverbal
information about ones dispositions and experience of a situation (see e.g., C. Darwin,
1899/2009; Ekman, 2007; Marsh, Richardson, Baron, & Schmidt, 2006). In any case, personal or
interpersonal awareness of aspects of experiencesuch as confusion, distraction, uncertainty,
interest, attentiveness, vitality, and emotionalitycan provide a nonarbitrary basis for individual
or collective intentionality (e.g., what to do next). At the very least, such inconclusive
considerations or dynamical constraints on thinking provide a better sense of the confidence one
should have in a particular pursuit or in a particular conclusion for the situation at hand. This is
critically important in negotiating the boundaries between the known and the knowable, and
between the knowable and unknowable (see Chapter 3, section 3.2.1), the ubiquitous
characteristic of individual or collective engagements with the world under ambiguous or
changing conditions (see Chapter 3, section 3.2.6).
There are fairly direct implications of the dynamics of thinking for AAR. AAR should be
instrumental in producing or reinforcing learning. Learning how to do an AAR and giving it
suitable priority could be an enabling learning objective in most learning events. One should not
assume, for example, that AAR have to be a simple and direct recitation of what was supposed to
happen or even necessarily an articulate account of what actually happened. It is okay if it
includes an inconclusive concern about the ineffable. Neither should participants in an AAR be
overly concerned with being pleasant. There may be nothing wrong with attributional comments
as long as it is not presented or perceived as an ad hominum attack. It is well recognized in the
Army that bad news doesnt age well. Moreover, a certain amount of digression can be
valuable if it is a way to articulate what otherwise may be difficult in the context of the learning
event at hand. Participants may need to draw on richer experiences to make more explicit the
meaning, or lack thereof, that they find in the learning event. In any case, AAR can address things
done and not done in the learning event and, more importantly, the reasons why. In other words,
AAR can address intentionality and decision-making.
The productive role of immediate experience in intentionality, most notably with respect to
changes in intention, have received very little attention in the scientific psychology that explicitly
addresses thinking and adaptability. There are many reasons for this theoretically, empirically,
culturally, and historically. In general, scholars are inclined to conceptualize thought as
something that is deliberative and either momentarily correct or incorrect. Less analytical aspects
of private experience typically have been considered to be in the realm of the emotions, clinical
practice, or otherwise entirely outside the reach of science. The next section highlights a relevant
line of scholarship concerning the role of emotion on thinking and decision-making.
125
Riccio
5.3 Emotion, Information, and Engagement
5.3.1 Ecological Perspective on Emotion

A fundamental tenet of OBTE is that motivation and emotion are influenced directly and
powerfully by any approach to instruction and, in particular, by interactions between instructors
and students. The principles and practices of OBTE tackle this issue head on to ensure that the
influence has a positive effect on learning (see Chapters 2 and 3). It is critically important to note
that, in addressing emotion, we are not simply referring to the pleasantness or unpleasantness of a
learning experience. Nor are we referring to feelings or the physiological stimulation and
sensory qualities commonly associated with one emotion or another such as anger, fear, disgust,
sadness, happiness, and surprise (Colleta, Vernet-Maurya, Delhommeb, & Dittmarb, 1997; cf.,
Ekman & Friesen 1975; James, 1890). While we dont deny these sensory experiences, we
believe their particular quality is not fundamentally important. Within a commitment to realism,
the major concern is that a focus on the sensory qualities of emotions is analogous to a focus on
sensations in the scientific inquiry into human perception and action in general. The problem with
such a focus is the fact that all engagements with the real world involve the stimulation of
multiple sensory systems (J. Gibson, 1966). While the study of single isolated sensory systems or
qualities may be useful in understanding physiological systems, it can be exceedingly misleading
with respect to understanding perception and purposeful engagement with the surroundings (J.
Gibson, 1979; Stoffregen & Riccio, 1988, 1991; Stoffregen & Bardy, 2001; Figure 3).
Figure 3. Intermodal invariants in an individuals engagements with the surroundings.

The focus of ecological psychology, with its commitment to realism, is on information about
couplings between an individual (human or nonhuman) and the environment and what they afford
for purposeful engagements with the environment. So the question about emotion becomes: What
information is available in emotion stimulation, combined with other forms of stimulation (e.g.,
optical, mechanical, acoustical) about affordances for action in the environment? A clue is
provided by another tenet of ecological psychology, specifically that it is difficult if not
impossible to understand perception by considering a passive observer or imagining that the
observer is passive just because the activity of the perceptual systems are not obvious (see e.g., E.
Gibson, 1988, 1991; J. Gibson, 1966, 1979; Kugler & Turvey, 1987; Shaw et al., 1982; Turvey et
al., 1981). Multimodal perception enables attunement to intermodal invariants that provide direct
information about cause-effect relationships in ones interaction with the surroundings
(Stoffregen & Riccio, 1988, 1991; Stoffregen & Bardy, 2001). One class of invariants is the
proximate effects of ones own actions (E. Gibson, et al., 1987; Riccio & Stoffregen, 1988, 1991).
This corresponds to inner loop feedback in control-theoretic analysis of dynamical systems
Passion & Reason
126
(Riccio, 1993b). It reflects ones ability to exert an influence on the surroundings. Together with
outer-loop feedback about the consequences of this influence, these invariants provide
information about ones capabilities for action (affordances). In ecological psychology,
information about ones capabilities for action is highly nuanced and typically includes many
nested perception and action systems as well as inter-temporal relationships that informs ones
preparation for action (Riccio, 1993a,b; Riccio & McDonald, 1998; Riccio, et al., 2001; van
Wegen et al., 2002).
A conjecture that follows directly from these control-theoretic aspects of ecological psychology is
that perception and control of ones interactions with the surroundings can become more refined
and more elaborate through attention to information that specifies ones preparation for action.
Our current elaboration of this claim is that such information is available in multimodal
stimulation that includes but is not limited to emotion stimulation. The implication is that the
experience of emotion generally refers to choices about ones engagement with the surroundings.
This is not especially controversial in the sense that this is a recapitulation of common claims
such as those about the value of physiological activity associated with preparation for fight or
flight. Beyond this, however, we posit that such observable preparation for action is
considerably more nuanced and multifaceted than suggested by a binary interpretation of
physiological activity in provocative situations. Information about ones capabilities for action,
and that supports control of such action, is uniquely available in the juxtaposition of emotion
stimulation with external stimulation of various perceptual systems obtained through interaction
with the surroundings. Emotion stimulation due to a potential lethal confrontation may be
attended by optical stimulation that reveals either potential defilade or potential enfilade locations
and postures. This difference matters. Our claim is that information in this multimodal
stimulation supports ones choices about engagement with the surroundings.
As limited as the various qualities of emotion stimulation might be, considered in isolation, there
is an uncountably large variety of affordances that one can come to know based on the equally
large variety of possible couplings that one can establish with the environment. And, at any
moment, the perception of such affordances can be quite elaborate because of the multiplicity of
nested interactions that concurrently are involved in even the simplest interactions with the
surroundings (Riccio & Stoffregen, 1988; Riccio, 1993a,b). On this view, there is a virtually
limitless capacity for perceptual learning and development with respect to emotion experience
and its role in engagements with the surroundings (cf., Higgins, 2006; Solomon, 2003; 2004b).
5.3.2 Emotion as Engagement
An ecological view of emotion converges nicely with lines of research that view emotion as
engagement or, more specifically, as about preparation for action (see e.g., Frijda, 1986, 2007,
2009; Frijda & Sundararajan, 2007; cf., Griffiths & Scarantino, 2008; Lambie, 2009; Ping,
Dhillon, & Beilock, 2009; Solomon, 2003). Nico Frijda describes three aspects to a scientific
epistemology that can guide inquiry into emotion: control precedence, wholeness, and valuation.
Control precedence reflects the experience of volition, that is, the aspects of experience that take
priority in the moment by directing attention and action and that also have a sort of tenacity that
transcends the moment by resisting distraction and caprice. Wholeness refers to the emotion
experience that reflects engagement of a person as a whole with the surroundings and does so in a
way that reveals the persons relationships with the surroundings. Valuation refers to the value
one finds in an experience or ones commitment to the attendant engagement with the world. We
find a direct correspondence between Frijdas scientific epistemology and our own (i.e., the three
scientific pillars for OBTE) as well as existential psychology. Control precedence emphasizes
issues that are the focus of ecological psychology such as reciprocal relationships with the
127
Riccio
surroundings that are dominated by natural law (situated behavior). Wholeness invokes a breadth
of issues commensurate with the considerations of self-efficacy theory such as reciprocal
interpersonal relationships that are directly perceivable (situated experience). Valuation clearly
addresses the same kinds of issues as positive psychology such as transcendent meaning (situated
meaning).
Interestingly, Frijda (2009) also points out that the aspects of emotion experience that correspond
to the three components of his scientific epistemology are not always present at the same time and
to the same degree. The strength of control precedence, for example, is directly related to the
extent of ones engagement. This suggests that this aspect of emotion experience could be a good
indicator of the extent to which an event is engaging, and it should relate to the motivational
exigency of autonomy that has centrality in self-determination theory. Wholeness presumably is
weak when, and to the extent that, emotion experience refers to others with whom one has an
indeterminate relationship, such as in reading a compelling story about a character who is
otherwise unknown to the reader. We believe Neissers notions about the varieties of self
presumably will be instructive for further scientific inquiry into wholeness. Valuation would be
limited if one were thrown into a situation so novel that it would be difficult to identify principled
behavior amid a plethora of unfamiliar details. It would be difficult to appreciate the personal
meaning or relevance of such situations (e.g., ones characterized by surprise or amazement).
Similarly valuation might be short-circuited in situations that force one to focus only on shortterm objectives and not also on longer-term outcomes or downstream consequences of ones
decisions and actions. Frijda argues thatwhile there is independent variation in the experience
of control precedence, wholeness, and valuationparadigmatic cases for the role of emotion are
characterized by prominence of all three of these aspects of emotion experience.
The work of Frijda, and others who view emotion as engagement with the world (e.g., Solomon,
2003; 2007), is consistent with our ecological view of emotion as one aspect of the multimodal
perception that informs and guides an individuals engagement with the world. Consider an
analogy to the effect of the intensity of light on visual acuity. Visual acuity, detection, and
recognition generally improve with increased light intensity (Boff & Lincoln, 1988). We suggest
that emotion is like the intensity of light to the visual system or, for that matter, like the intensity
of stimulation for any sensory system. By way of analogy to the colourfulness of light, emotion
also is analogous to various sensory qualities but those qualities are not finally the point.
Variations in intensity or color within and across objects in our field of regard enable us to pick
up information about those objects. Given that the relationship between an individual and the
surroundings is ontologically primary in our approach, the information in stimulation is about the
coupling, the fit, or the affordances for engagement with the world. We agree with Frijda (2009)
that emotion experience typically consists of the perception of emotionally meaningful objects,
events, or states (p. 266) or affordances (p. 267). The analogy deepens by considering the
control-theoretic aspects of ecological psychology (Riccio, 1993b). In this respect, the inner-loop
information about ones momentary capabilities for action (i.e., the basis in reality for ones selfefficacy), and ones preparation for action (i.e., utilizing ones capabilities), reveals a more direct
link between perception of affordances, emotion stimulation, and engagement with the world.
On this view, enhancing the emotion experience is like increasing the intensity of light or, in a
control-theoretic sense, increasing the gain on the system. Emotion, however, also increases our
attunement to the meaning of our engagement with the world. The learning and adaptation that
this attunement enables thus may benefit from increased emotional intensity. The progenitors of
OBTE have repeatedly emphasized the importance of significant emotional events in Army
training and education. At the same time, we understand that this relationship is not monotonic,
that too much stimulation can impair perception and performance. Carrying our analogy further,
Passion & Reason
128
consider a situation in which one has to track an object to a point where one is looking directly
into the sun. One would close or cover ones eyes, and perhaps employ some intermittent looking
strategy to keep track of the object without saturating ones visual system. This is a relatively rare
situation in which one actually attends to the sensory qualities of ones experience. While this is a
limiting case in perception and action, it is possible and thus important to consider in designing or
assessing a situation. Similarly, any situation that leads one to focus on the sensory qualities of
emotion experience will be problematic for learning and performance to the extent that it distracts
attention away from the information in emotion experience about ones preparation for action and
ones momentary capabilities for action (cf., Lambie & Marcel, 2002).
We expect further development of OBTE to be guided by a more sophisticated appreciation of
emotion that considers it as an essential aspect of an individuals engagement with the world. We
believe the control-theoretic aspects of ecological psychology can be helpful in this pursuit.
Continuing with our analogy to light and the eye, consider the multicriterion control system that
controls even something as simple as pupil diameter (see e.g., Stark, 1968). Optically, pupil
diameter influences both the intensity of retinal illumination and blur of the retinal image that is
due to imperfections in the lens of the eye. Visually, it influences both brightness and depth of
field. It thus is appropriate that control of pupil diameter is influenced by the intensity of light and
the proximity of an object of regard. Behavior in the environment resolves the attendant
ambiguities by utilizing other actions systems (from the hands to the feet) that differentially
modify retinal illumination or proximity to an object of regard. The point is that such many-tomany mappings also exist for emotion stimulation and objects in ones field of regard while
engaging with the world. It may be no more difficult to resolve these ambiguities than it is for
brightness and clarity of vision.
Lest the reader think that the analogy to pupillary control is a bit arbitrary, it is worth noting that
pupil diameter also is influenced by emotional states in ways that are similar to the reciprocal
influence between emotional states and facial expressions. Pupil diameter may even complement
facial expressions in the perception of emotional states of others (see e.g., Harrison & Critchley,
2007). Moreover, there is evidence that pupillary dynamics can become coupled between two
individuals and that the nature and extent of this coupling influences emotion experience
(Harrison, Gray, Critchley, 2008). This coupling is reminiscent of coordinative structures that
have been observed in other nested action systems and that have been modelled in ecology
psychology (see e.g., Kugler & Turvey, 1987). One reason why such an ecological approach to
dynamical systems is powerful is that it can easily be extended to interpersonal interactions
(Marsh et al., 2006). This is because informational coupling can be as strong as exchanges of
energy, they can result in (i.e., enable) the same patterns of behavior, and they can be modelled in
the same way. Control-theoretic aspects of ecological psychology, in particular, can be used to
understand the nuances of the nested couplings that characterize typical engagements with the
surroundings (Riccio et al., 1993b). This allows us to find deep agreement with Frijda on the
architecture of engaged experience (e.g., control precedence and wholeness) even though we
differ with the epistemological dualism inherent in his adaptation of Sartres phenomenology of
emotion (Frijda, 2009; see also Solomon, 2004). We differ with him because we believe he makes
unnecessary assumptions about the role of induction in self-awareness and the perception of
affordances (see Chapter 4, sections 4.2.1, 4.2.2). Nevertheless, this difference is not problematic
relative to what we agree on; it is inconsequential with respect to the implications of our views of
emotion for training and education.
129
Riccio
5.3.3 Implications for Training and Education

The most important theme of the scholarship we have highlighted is that emotion is at least as
proactive as it is reactive insofar as it guides ones attention and provides information about
preparation for action. The implication for training and education is that the learning environment
should include situations in which individuals can learn how to pick up information in emotion
stimulation that is most relevant to the task at hand. Toward this end, it would be beneficial if
some learning events require interactions that have some emotional significance. This could be as
simple as targets on a shooting range that include representations of friend, foe, and other (S.
Flanagan, personal communication, March, 2008). It also would be useful to design some
situations that create emotional distraction (i.e., attention to sensory physiology). Together such
experiences foster perceptual learning though which one can become increasingly more
sophisticated in differentiating between events that compel attention to sensory physiology and
events that require attention to information about existentially significant affordances. If emotion
is not considered in the design and implementation of learning events, instructors will miss
opportunities to educate the attention of students for performance in emotionally evocative
situations, and they may inadvertently mis-educate the attention of students for such situations.
How can a learning event be counterproductive with respect to perceptual skill in emotionally
evocative situations? Consider, for example, a learning environment in which the apparent values
of ones community of practice suggest that one should control the expression of ones emotion
and, implicitly, the experienced qualities of emotion. In other words, what are the consequences
of assuming that emotions are irrational and dangerous, especially in lethal or hazardous
situations? Undoubtedly, one would be inclined to minimize the stimulation of sensory systems
associated with emotion experience, to divert attention away from this experience, or to avoid the
pick up of information in emotion stimulation. In control-theoretic terms, this essentially results
in decoupling an individual from the surroundings, to some extent, by eliminating a feedback
loop or reducing observability of an individuals states relative to the surroundings. In any case,
the result is that ones capabilities to make informed decisions or execute informed actions with
respect to the surroundings are reduced. Moreover, with respect to coordinated action within a
unit, individuals would be less likely to benefit from the information in the emotion experience of
others. We are not encouraging the unfettered expression or the uncontrolled experience of
emotion in learning events and certainly not in operational settings. Instead, we emphasize the
importance of sophistication in the control and experience of emotion. On the view that emotion
is engagement with the world, emotion can and should be controlled by controlling ones
engagement with the world, not by ignoring experience. Sophistication in the experience of
emotion is fostered by learning events that explicitly address (a) the information in emotion about
ones momentary preparations for action, (b) the affordances of such preparations for action with
respect to individual and collective tasks, and (c) unintended consequences of ones actions.
5.4 Emotion, Decision-Making, and Inter-Temporal Choice
5.4.1 Toward a More Integrated Theory
Frijdas notions of control precedence and wholeness in emotion experience each have relevance
to Neissers notions of the ecological self and the interpersonal self. That is, it is important to
perceive ones proclivities to act with respect to others as well as the physical surroundings and to
perceive the social and physical consequences of ones actions. This helps one understand ones
relationships to others and to the physical surroundings, it helps one understand that the whole
that is greater than the sum of the parts and, in a more nuanced way, it helps one understand the
nested self as a spectrum of concurrently available capabilities ranging from the purely individual
Passion & Reason
130
to the purely collective. Frijdas notion of valuation in emotion experience has relevance to
Neissers notions of the temporally extended self and the conceptual self. While control
precedence and wholeness have immediate relevance and salience in the moment, valuation
comes into play to a greater extent over larger times scales within which the task at hand is
nested. This is reminiscent of the reciprocal influences between nested time scales in adaptive
dual control (see Chapter 4, section 4.1.2, 4.1.3). The accumulation of micro-experiences from
moment to moment provides information about causes and consequences in ones engagement
with the world over larger time scales. The information about such affordances also can influence
the choices one makes in the moment.
The interplay of control precedence and wholeness presumably plays an important role in the
weaving of nested time scales that reveals value and meaning in ones engagement with the
world. We believe there is a scientifically useful correspondence between this triad of emotion
experience and other psychologically potent triads we have discussed. In particular, control
precedence relates to initiative and autonomy, wholeness relates to accountability and relatedness,
and valuation relates to confidence and competence. On this view, Frijdas three aspects of
emotion experience are important indicators of the extent to which an individual is engaged in a
situation in ways that motivate a deeper and more sustained engagement (cf., Deci & Ryan, 2000,
2008; Guay, et al. 2008; Ryan & Deci, 2008). Viewing emotion as engagement thus can lead to a
deeper scientific and philosophical understanding of motivation within a commitment to realism.
It more completely situates theories of motivation in the verifiable coupling of individuals with
the physical and social surroundings and in the reality of nested time scales and the nested self
(cf., Higgins, 2006).
5.4.2 Emotion and Decision-Making
Our perspective on valuation in emotion experience is that it both drives and reflects the choices
one makes. Thus it is useful to consider research that looks at the role of emotion in decisionmaking. Research on decision-making in recent decades has been dominated by the bounded
rationality theories of behavioral economics (Kahneman, 2003; Kahneman & Tversky, 1979,
1984; cf., Simon, 1955/1957) and an associated preoccupation with cognitive biases or errors
with respect to models of rational choice. In fact, studies that require individuals to utilize
terminal inductive inference in making decisions show consistent effects of emotional bias
(Lerner, & Keltner, 2001; Lerner & Tiedens, 2006; Loewenstein & Lerner, 2002). Anger, fear,
sadness, and happiness can bias ones assessments of past, present, and future situations; and
different emotions do so in different ways. Angry people tend to attribute blame to others for
negative events while sad people tend to attribute causes to situational factors. Angry people have
been shown to make more optimistic risk assessments for the same conditions in which fearful
people make pessimistic risk assessments. In the moment, angry people and happy people both
tend to think in more superficially, presumptively, or heuristically than do sad people who are
more inclined to engage in deliberate thought. These effects are complex, however, in that they
depend on various situational factors. For example, the outcomes of decision-making in
emotionally evocative situations may or may not be appropriate. The takeaway point is that
emotion matters with respect to inferential bias, thus it is wise to consider such effects in coming
to a better understanding of decision-making (Lerner & Keltner, 2001; Lerner & Tiedens, 2006).
Research on the role of emotion in decision-making is in a relatively early stage of development.
There have been some interesting trends though. In particular, with progress in the understanding
of emotion, the biasing effects also have become more differentiated. The biasing effects at a
more refined level of abstraction frequently are counter to what would have been predicted at a
higher level of abstraction. For example, biases of negative emotions are not necessarily more
131
Riccio
similar to each other than to positive emotions. The experience and effects of any particular
emotion (e.g., anger) are influenced by factors such as certainty, control, and responsibility
(Lerner & Keltner, 2001; Lerner & Tiedens, 2006). Presumably, the effects of emotion on
decision making also will be influenced by even more nuanced situational factors such as the
momentary nested couplings between an individual and the surroundings. In the context of these
couplings, concepts such as certainty, control, and responsibility have greater specificity and
clarity with respect to affordances for action and the downstream consequences of ones
decisions.
5.4.3 Emotion and Nested Time Scales
The methods used in decision-making research generally dont involve prolonged engagement of
an individual (the so-called research participant) with the surroundings. The individual
typically has little or no personal agency to influence the surroundings and unusually has little
time or opportunity to come to know reality. The situations typically are inferentially truncated
(i.e., biased toward terminal induction) and dont provide reality-testing feedback to individuals.
As the research on the role of emotion in decision-making matures, and more naturalistic
decision-making is observed (see Klein, 2008), we expect that the effects of various emotions
may not necessarily be problematic in the sense of forcing departures from rationality.
Momentary bias in attention or presumption may be outweighed by an increased salience of
disconfirming feedback, or more elaborate feedback, especially if others typically present in
existentially significant situations provide that feedback. Differences in emotions may even
benefit collaborative reflection in a task-organized unit to the extent that it fosters productive
diversity within the group (cf., Druskat, 2005; Jehn, Northcraft, & Neale, 1999; Bowers, Pharmer,
& Salas, 2000). This benefit may be realized through the collaborative reflection among
individuals who collectively provide diversity of perspectives or in the additional information
picked up as a result of attention to more facets of a situation. This suggests that group learning
may be facilitated by intentionally creating different emotion experiences for different individuals
in the same learning event. Different emotion experiences thus are not viewed as biases to be
cancelled out. Collectively, they reflect a set of complementary perceptual skills that enable
attunement to a variety of affordances in collective engagement with the world. They enable a
richer understanding of the world (cf., Lambie, 2009).
Emotion can influence the meaning one makes of prior events (Lerner & Tiedens, 2006; Neisser
& Fivush, 1994; Neisser & Hyman, 2000). Changes in emotion thus can provide a powerful set of
lenses through which one can appreciate multiple facets of a prior experience (cf., May, 1986) as
long as one has come to understand such effects of emotion (Lambie, 2009). Emotion also can
influence expectation and decisions about future events (Camber, Loewenstein, & Prelec, 2005;
Lerner & Tiedens, 2006; Lowenstein & Lerner, 2002; Loewenstein, Weber, Hsee, & Welch,
2001). The effects of emotion on experience are not simply a linear sequence of causal influences
that progress from past through the present to the future. In essence, such cognitive biases
momentarily establish a temporally extended self. The temporarily extended self is a momentarily
coherent influence on perceiving, remembering, and imagining that is causally symmetric
forward and backward over a sequence of events (cf., Neisser, 1996; Neisser & Fivush, 1994;
Neisser & Hyman, 2000; Neisser & Jopling, 1997). Exposure to different perspectives, to
different emotion experiences, could establish a new temporally extended self and a new coherent
influence on the meaning one makes of the past, present, and future (cf., May, 1983). Emotion
experience that is well grounded in reality, such as through reflection on current behavior and
awareness of the surroundings in which it is situated, presumably can help reduce biases away
from reality. If so, inter-subjectivity and collaborative reflection can enrich this grounding in
reality.
Passion & Reason
132
5.4.4 Neuroeconomics and Inter-Temporal Reasoning

The reciprocal influence between current events and future outcomes is of particular interest in
the context of outcomes-based training and education. The most relevant research on these
reciprocal effects is in behavioral economics and the recent interdisciplinary niche called
neuroeconomics (Camber et al., 2005; Lowenstein, Rick, & Cohen, 2008). From the perspective
of OBTE, one of the most interesting developments in this area is the conjecture that intangibility
may be essential to explain patterns observed in the most common inter-temporal choices that
people make (Rick and Loewenstein, 2008).
The discounted utility model has dominated research in behavioral economics (see e.g.,
Kahneman, 2003; Kahneman & Tversky, 1984). In its various forms, this model assumes that
people have a bias toward options that are available sooner and tend to discount options that
would be available later. More specifically, delaying a desired objective by a day, for example,
has greater importance today than the same delay one hundred days from now (i.e., hyperbolic
discounting). This is generally considered to be one of the most common if not canonical ways
that human decision-making is irrational (given a shared assumption that all descriptions of the
future are reliable and accurate). Rick and Loewenstein replace discounted utility with the face
valid assumption that future outcomes are always less tangible or less certain than near-term
outcomes. Given this new assumption, their innovation is the conjecture that emotions associated
with anticipated outcomes make near-term outcomes and future outcomes commensurable, that
is, they allow one to make comparisons in terms of emotions instead of using other frameworks
that confound time delay with tangibility or certainty. They suggest that emotions are the
common currency for inter-temporal decision-making. This helps explain choices in which
value is not discounted hyperbolically over time (Rick & Loewenstein, 2008).
In a logical extension of their thesis, Rick and Loewenstein suggest that emotions may be the
common currency for decisions involving cognitive comparisons among any outcomes that differ
in tangibility whether or not the outcomes differ with respect to time delay. In particular, they
offer an example in which outcomes differ along what we have referred to above as the spectrum
of individual-collective. The idea is that attention to emotions evoked by collective objectives
may be commensurate with those evoked by individual objectives amid otherwise problematic
differences in tangibility or certainty of collective objectives relative to individual objectives. In
this manifestation, we believe the conjecture of Rick and Loewenstein in behavioral economics
dovetails in an elegant way with Nico Frijdas triadic relationship among control precedence,
wholeness, and valuation in emotion experience as well as with other triads we have considered
(e.g., Rollo Mays modes of existence). Decisions influenced by emotion in the way described by
Rick and Loewenstein are not necessarily more rational than decisions based on discounted
utility; however, our various discussions of collaborative reflection in this chapter suggest a way
that they might become more rational over time. Emotions and their effects on behavior (and, by
implication, their effects on thinking and decision-making) are relatively easy to talk about
because they are concrete and, to some extent, because they are directly perceivable. Thus they
may be a much easier way for a group to compare and contrast perspectives than by using a more
abstract framework (e.g., one that is not closely tied to the unpredictable unfolding of a shared
event). A small group of individuals engaged in a shared experience may provide the most
rational unit for decision-making if information in emotion experience is considered along with
other concurrently available information about individual and collective engagement with the
world.
133
Riccio
The new perspective in behavioral economics offered by Rick and Loewenstein is pregnant with
possibilities for cross-fertilization across scientific subdisciplines. An equally important
consideration is the admirable caution by Frederick & Loewenstein (2008) about the preference
literature that many of the widely cited, stylized facts remain facts only by virtue of an
unwitting convergence in research methodologies (p. 232). We would add that, in addition to a
diversity of methodologies, a research community should ensure that the collective research
portfolio includes paradigms that are as close as possible to the natural situations that the research
seeks to understand (see e.g., Flyvbjerg, 2001; Riccio, 1993a; Schrim, & Caterino, 2006). In the
case of OBTE, we are compelled to consider research that does justice to the nature and extent of
collective engagement of small units in Full Spectrum Operations FSO. Our claim is that such
research on learning and development in formal programs of instruction will be more likely to be
relevant to FSO. Furthermore, we claim that any program of research that is guided by such
research also is likely to result in learning and development that is more likely to transfer to FSO.
5.4.5 Inter-Temporal Reasoning and Adaptive Dynamical Systems
The scientific foundation described in the last three chapters (i.e., the three pillars and connecting
lines of thought) reflects a commitment to a more naturalistic approach to learning and
development, one that closes the gap between FSO and formal programs of instruction. In
particular, the commitment emphasizes that (a) the interaction between an individual and the
physical and social environment is the fundamental unit of analysis on an individuals
engagement with the real world, (b) the interplay of exploratory and performatory behavior is
critical in coming to know reality, (c) perceptual learning is ubiquitous and relentless over time
scales that are greater than a typical designed learning event, (d) inter-subjective collaboration
facilitates learning and development with respect to valued outcomes within a community of
practice, and (e) collaborative learning is characterized by crystallization of multiple perspectives
about the manifestation of shared values in the behavior of individuals and its consequences with
respect to common tasks and intent.
The methods we have utilized given this commitment are convergent with independent lines of
research in adaptive dual control theory (see Chapter 4, sections 4.1.2 and 4.1.3). In this respect,
it is noteworthy that inter-temporal reasoning recently has been viewed from the perspective of
dynamical systems (Scherbaum et al., 2008). The critical importance of the dynamical systems
approach is that it directly addresses both coupling within a system (e.g., a subsystem coupled
with its surroundings) and trajectories in system performance (Riccio, 1993b). A contribution of
this perspective is insight into potential causal explanations about what drives the trajectories. In
dynamical systems, the influence can be described as a layout of systems states that vary more or
less continuously in terms of being relatively attractive and relatively repelling with respect to
some objective for system performance (Riccio & Stoffregen, 1988; 1991; Riccio, 1993a,b;
Riccio & McDonald, 1998; van Wegen et al., 2002; Sherbaum et al., 2008). Typically, this
objective function derives from a causal relationship between important dimensions of variation
in a subsystem (i.e., its capabilities for action) and changes in its surroundings; that is, it reflects
the potency of a subsystem in effecting change in its surroundings, most notably in its
relationship with its surroundings (Riccio, 1993a,b). Nonlinear control-theoretic aspects of
ecological psychology emphasize the detectability of these dimensions of variation and the
implied stabilizability of the system (Riccio & McDonald, 1998). Consistent with adaptive dual
control theory, in particular, this approach also reveals that detectability and stabilizability are
fostered by information in variability of system behavior over time scales that are short relative to
the demands on system performance (Riccio, 1993a; Riccio & McDonald, 1988; van Wegen, et
al., 2002; see Chapter 4, sections 4.1.2 and 4.1.3).
134
Passion & Reason
Two questions emerge in the analysis of dynamical systems. First, what is the relevant state
space, the relevant capabilities for action that it parameterizes? Second, what are the objectives
for system performance with respect to which such states are evaluated? In adaptive dual control,
there is a further implication that observable effects on short-time scales are commensurate with
observable effects on longer time scales. Observation of the former enables the system to adapt to
varying conditions to achieve desired system performance irrespective of the variation in
conditions. How would we extend this to inter-temporal reasoning? First, consistent with Rick &
Loewenstein (2008), we believe that consideration of emotion associated with long-term
outcomes will be essential in making them more tangible. Consistent with our ecological
approach to emotion and intermodal invariants, emotion stimulation must be part of the state
space that describes ones capabilities for action. Second, it will be essential to address long-term
outcomes explicitly in the moment, in the task at hand, however small or mundane the task seems
to be when considered in isolation (cf., Riccio et al., 2004; see also Appendix D). In other words,
the micro-experiences we emphasize in OBTE are experiences that can be explicitly considered
and discussed in the context of long-term outcomes, and emotional engagement in these microexperiences is critical to make intangible outcomes more concrete and immediate.
5.5 Beyond Science
5.5.1 Existentialism
In our attempt to engage the practical wisdom of Soldiers as instructors and leaders, and to shine
a light on it, we were taken to bodies of work that we had not fully anticipated. In some cases, we
were taken to lines of thought that were surprising. The most important example of this is the path
we followed from the science associated with motivation, emotion, and values to the philosophy
and literature on existentialism (see e.g., Barnes, 1959; Cotkin, 2003; Solomon, 2004a). As it
turned out, this was a natural path because of our commitment to experience as emphasized by
James, Dewey, and the Gestalt Psychologists (see e.g., Heft, 2001; Maslow, 1968; Reed, 1996a,b,
1997; Riccio, 1993a); that is, meanings one makes of conscious experience of the world rather
than the mere fact of experience in the world. Our inquiry into the meaning Soldiers, and
especially leaders, make of training and education led us to question common assumptions about
the dichotomy of passion and reason (M. Darwin, 2008a,b; Hume, 1740/2009; Smith, 1759/1976;
Solomon, 2003, 2004a,b, 2007). Consideration of existential thought was relevant because of
consistent themes in the practical wisdom of Soldiers that suggest the need to foster life-long
development of attributes that prepare Soldiers for emotionally evocative ambiguity, not to
develop programs intended to help Soldiers (think they can) necessarily resolve ambiguity and do
so dispassionately. Thus, there is value in exploring ideas about sources of dauntlessness that
have emerged in changing or oppressive societies in various historical periods (see e.g.,
Augustine, trans. 1991; Heidegger, 1927/1962; Kierkegaard, 1846/1992; Lvi-Valensi, 2006;
Nietzsche, 1888/1967; Sartre, 1943/1956; Solomon, 2004a).
Consider the conceptual triads listed in Table 1 that suggests where engagements with the
humanities can be helpful (this is not intended as an exhaustive or representative list of relevant
concepts). The use of triadic frameworks and relationships is not unique in science (see Chapter
3, section 3.2.3) or the humanities (see e.g., Hernade, 1995). Based on innumerable conversations
with many stakeholders in OBTE at a variety of sites and programs of instruction, we believe the
various personal meanings that will be found in OBTE are likely to dovetail with the philosophies
embodied in one or another of these triads. The value of such triads is that they reveal concepts
that are not separable and that, in combination, have emergent properties (see also, Chapters 3
and 4). Juxtaposing the triads is not to imply a one-to-one mapping between concepts in each
column. Rather, frameworks such as this are intended to reveal ways in which the concepts can be
135
Riccio
broadened, especially beyond the common connotations, and to reveal where the boundaries of
the concepts are. In other words, there will be complementarities and inconsistencies that sharpen
the use of each concept in capturing the meaning that OBTE has for various individuals. Such
frameworks also will have implications for further development in the thinking of OBTE
stakeholders with respect to its local implementation and value. They will be important because
OBTE does not dictate to stakeholders precisely what meaning they should make of it. It simply
asks that one become aware of the meaning one makes of OBTE and that one can trace it to the
principles and practices of OBTE.
Table 1. Examples of conceptual triads that are relevant to experience and adaptability in Full
Spectrum Operations in the context of values-based requirements in Army doctrine (cf., Chapter
4, section 4.4.1, table 2).
OBTE Intangibles
Aristotle
Aristotle
Augustine
Augustine
Soren Kierkegaard
Soren Kierkegaard
Friedrich Nietzsche
Friedrich Nietzsche
Martin Heidegger
Martin Heidegger
Jean-Paul Sartre
Jean-Paul Sartre
Albert Camus
Albert Camus
Confidence
Eudaimonia
Phronesis
Teaching
Love
Aesthetic Existence
Subjective Truth
Ubermensch
Will to Power
Being-in-the-world
Existence
Self
Being in Itself
Self Identify
Emotions
Initiative
Reason
Episteme
Signs
Grace
Religious Existence
Existential Dialectic
Slave Morality
Self Expression
Being-unto-death
Fallenness
Transcendence
Being for Itself
Reflection
Rationality
Accountability
Virtue
Techne
Significance
Charity
Ethical Existence
Ethical Individual
Master Morality
Self Mastery
Authenticity
Facticity
Facticity
Being for Other
Self Awareness
Lived Experience
5.5.2 The Soldier-Scholar as an Emergent Property of a Collective Pursuit

The AWGs scientific engagement in OBTE seems to be a departure from the predominant use
and view of science in the U.S. Army. Typically, the intent of scientific engagements in the Army
is to retain scientists or obtain scientific subject matter expertise that is related to an operational
problem. As useful as this might be, it does not capture the most important attribute of science,
that it is the emergent activity of a community over time, a collaborative coming to know.
Science has both historicity and a social dimension (cf., Godfrey-Smith, 2003). We do not take
the position that scientific knowledge is socially constructed. We do, however, emphasize that it
involves dialectical inter-subjectivity in a collective search for converging evidence, for
meaningful evidence from an authentic multi-faceted inquiry. This is reflected in the iterative
activity and mutually influential tasks documented in this monograph (see Prologue). Practical
wisdom was not simply identified and translated into scientifically valid and verifiable concepts.
Practical wisdom also was elaborated and refined in this investigation. The understanding of both
Soldiers and scientists about OBTE evolved. Both communities benefited from the reciprocal
influence, and both were left with enhanced capabilities for continued development. Science
should not be thought of in terms of nouns; it embodies verbs and most if not all verb tenses.
The present investigation was even more unusual in that it was interdisciplinary or transdisciplinary. It was not merely a compartmentalized multidisciplinary enterprise. It was selfconsciously holistic and replete with internal reciprocal influence. Drawing from various
Passion & Reason
136
disciplines and stances, a plurality of methods was employed to inquire into the practical wisdom
of Soldiers as instructors and leaders (cf., Flyvbjerg, 2001; Schram & Caterino, 2006). It is
important to note, however, that the intent was not an indiscriminate pluralism or the cultural
relativism it implies. Cultural relativism is fundamentally incompatible with the vision and
purpose of a military as well as the political conditions that give rise to its employment (cf.,
Clausewitz, 1976 trans.). Indiscriminant relativism is unlikely to foster a clear sense of virtue that
is essential to decisions based on practical wisdom when life and death are the consequences (cf.,
Aristotle, trans. 1925).
Although we alluded to some of the connections to existentialism made in the constant
comparison documented in this chapter as well as in Chapters 3 and 4, this exploration largely
was not documented in the manuscript because it was beyond our charge. Our conclusion is that,
even with the extensive grounding in science we have suggested for the practical wisdom of
Soldiers, scholarship in the humanities also is necessary to understand the variety of assumptions
and commitments we encountered. In a sense, the realization that the humanities should
accompany the sciences in the collaborative reflection on practical wisdom is a corollary of our
claim that science is a collective pursuit considerably richer than the typical scientific
engagements in the U.S. Army. Most science is conducted in academic communities that also are
populated by scholars in the humanities. While faculty in the sciences and humanities rarely
receive grants to conduct collaborative research together, and while formal scholarly interactions
are the exception, there are some reciprocal influences simply by virtue of the fact that these
individuals live and work together as part of a common community. In any case, the point is that
there are opportunities for reciprocal influence even amid some apparently insurmountable
obstacles. It can be argued that social and programmatic decision-making can benefit greatly from
both the sciences and the humanities (Flyvbjerg, 2001; Schram & Caterino, 2006).
We highly recommend that the humanities be included alongside the sciences in a collective
pursuit that informs continuous adaptation of Army training and education to a changing world.
Our expectation is that this journey through increasing levels of apparent complexity, perhaps
even some esoterica, will lead us to a simple core that we can communicate in a way that is
immediately meaningful and uncontroversial to Soldiers. This expectation is based on the
assumption that only the core truths, or at least inescapable realities, persist over millennia. Thus
we considered threads of wisdom (not superficial positions) that can be traced from antiquity to
the present. We also expect that this pursuit will be characterized by theme and variation. This is
important because of the pluralism of perspectives that will be encountered in the increasingly
broad consideration and interpretation of OBTE. There will be immediate practical value to being
able to identify themes amid variation and to recognize when variations are inconsistent with the
principles and practices of OBTE.
We believe that Soldiers who engage in this kind of collective pursuit, and an academic
community that engages with Soldiers, are what it means to pursue the ideal of the Soldierscholar. The point is not for all Soldiers to become students of the great ideas any more than it is
for scientists to feign understanding of in extremis conditions faced by Soldiers. The point is to
strive for a dialectic that includes both groups and in which deep questions are posed and pitfalls
are avoided because of collaborative consideration of knowledge and experience that is not easily
accessible to each group alone. In this vision, the Soldier-scholar is an emergent property of task
organization among Soldiers, scientists and scholars in the humanities.
137
Riccio
5.6 References
Boring, E. G. (1950). A history of experimental psychology. New York: Appleton, Century,
Crofts. (Original work published 1927)
Camber, C., Loewenstein, G., & Prelec, D. (2005). Neuroeconomics: How neuroscience can
inform economics. Journal of Economic Literature, 43(1), 9-64.
Camus, A. (1956). The Fall, trans. by Justin OBrien. New York: Knopf, (Translated by Justin
OBrien)
Camus, A. (1988). The Stranger. New York: Knopf, (Originally published in French in 1942;
translated by Matthew Ward)
Colleta, C., Vernet-Maurya, E., Delhommeb, G., & Dittmarb, A. (1997). Autonomic nervous
system response patterns specificity to basic emotions. Journal of the Autonomic Nervous
System, 62 (1-2), 45-57.
Darwin, C. (2009). The expression of emotions in man and animals. New York: Penguin Classics.
(Original work published 1899)
Group.
Group.
Deci, E. & Ryan, R. (2000). The what and why of goal pursuits: Human needs and the selfdetermination of behavior. Psychological Inquiry, 11 (4), 227268.
Druskat, V. (2005). Linking emotional intelligence and performance at work: Current research
evidence with individuals and groups. Mahwah, NJ: Lawrence Erlbaum.
Ekman, P. (2007). Emotions revealed: Recognizing faces and feelings to improve communication
and emotional life. New York: Holt.
Ekman, P. & Friesen, W. (1975) Unmasking the Face: A Guide to Recognizing Emotions from
Facial Expressions. Englewood Cliffs, New Jersey: Prentice Hall.
Frederick, S., & Loewenstein, G. (2008). Conflicting motives in evaluations of sequences.
Journal of Risk and Uncertainty, 37, 221235.
Frijda, N. (1986). The Emotions. Cambridge: Cambridge University Press.
Frijda, N. H. (2007). The laws of emotion. Mahwah, NJ: Lawrence Erlbaum.
Frijda, N. H., & Sundararajan, L. (2007). Emotion refinement. Perspectives on Psychological
Science, 2, 227241.
Passion & Reason
138
Frijda, N. (2009). Emotion experience and its varieties. Emotion Review, 1(3), 264271.
Gibson, E., Riccio, G., Schmuckler, M., Stoffregen, T., Rosenberg, D., & Taormina, J. (1987).
Detection of the traversability of surfaces by walking and crawling infants. Journal of
Experimental Psychology:Human Perception and Performance., 13, 533-544.
Gibson. J. (1966). The Senses Considered as Perceptual Systems. Boston. MA: Houghton
Mifflin.
Griffiths, P. & Scarantino, A. (2008). Emotions in the wild: The situated perspective on emotion.
In: P. Robbins & A. Murat (Eds.), Cambridge handbook of situated cognition (437-453).
Cambridge, UK: Cambridge University.
Harrison, N., & Critchley, H. (2007). Affective neuroscience and psychiatry. British Journal of
Psychiatry, 191, 192-194.
Harrison, N., Gray, M., Critchley, H. (2008). Dynamic pupillary exchange engages brain regions
encoding social salience. Society for Neuroscience, 4(3), 233-43.
Heft, H. (2001). Ecological psychology in context: James Gibson, Roger Barker, and the legacy
of William Jamess radical empiricism. Mahwah, NJ: Lawrence Erlbaum
Heidegger, M. (1962). Being and Time. New York: Harper & Row. (Original work published
1927; trans. by John Macquarrie & Edward Robinson).
Henle, M., J. Jaynes, & Sullivan, J. (Eds.) (1973), Historical conceptions of psychology (pp. 257266). New York, NY: Springer.
Hernade, P. (1995). Cultural transactions: Nature, self, society. Ithaca, NY: Cornell University.
Higgins, E. T. (2006). Value from hedonic experience and engagement. Psychological Review,
113, 439460.
Hofstede, G. (2001). Cultures consequences: Comparing values, behaviors, institutions, and
organizations across nations (Second Edition). London, UK: Sage Publications.
Hume, D. (2009). A treatise of human nature. Breinigsville, PA: Merchant Books. (Original work
published 1740).
Humphrey, G. (1951): Thinking, An Introduction to its experimental psychology, Methuen,
London.
Idel, M., & McGinn, B. (Eds.) (1999). Mystical union in Judaism, Christianity, and Islam: An
ecumenical dialogue. New York: Continuum.
Kahneman, D. (2003). Maps of bounded rationality: Psychology for behavioral economics.
American Economic Review, 93, 1449-1475.
Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decisions under risk.
Econometrica, 47, 313-327.
139
Riccio
Kahneman, D., & Tversky, A. (1984). Choices, values and frames. American Psychologist, 39,
341-350.
Kierkegaard, S. (1992). Concluding unscientific postscript to philosophical fragments. Princeton:
Princeton University (Original work published 1846; translated by Howard Hong & Edna
Hong).
Klein, G. (1989). Recognition-primed decisions. In W. B. Rouse (Ed.), Advances in man-machine
systems research (Vol. 5, pp. 4792). Greenwich, CT: JAI Press.
Klein, G. (2008). Naturalistic decision making. Human Factors, 50(3), 456460.
Klein, G., Orasanu, J., Calderwood, R., & Zsambok, C. (Eds.) (1993). Decision making in action:
Models and methods. Norwood, NJ: Ablex Publishing Corporation.
Klein, G., Ross, K., Moon, B., Klein, D., Hoffman, R., & Hollnagel, E. (2003). Macrocognition.
IEEE Intelligent Systems, 18(3), 8185.
Kugler, P., & Turvey, M. (1987). Information, natural law, and the self-assembly of rhythmic
movement. Mahwah, NJ: Lawrence Erlbaum.
Lambie, J. (2009). Emotion experience, rational action, and self-knowledge. Emotion Review,
1(3), 272-280.
Lambie, J., & Marcel, A. (2002). Consciousness and the varieties of emotion experience: A
theoretical framework. Psychological Review, 109, 219259.
Psychology, 81(1), 146-159.
Lvi-Valensi, J. (Ed.) (2006). Camus at combat: Writing 1944-1947. Princeton, NJ: Princeton
University. (Original work written 1944-1947).
Loewenstein, G., & Lerner, J.S. (2002). The role of affect in decision making. In: R. Davidson,
K. Scherer, & H. Goldsmith (Eds.), Handbook of affective science (pp. 619642). New York:
Oxford University Press.
Loewenstein, G. F., Rick, S., & Cohen, J. D. (2008). Neuroeconomics. Annual Review of
Psychology, 59, 647-672.
Loewenstein, G., Weber, E., Hsee, C., & Welch, N. (2001). Risk as feelings. Psychological
Bulletin, 127 (2), 267286.
Marsh, K., Richardson, M., Baron, & Schmidt, R. (2006). Contrasting approaches to perceiving
and acting with others. Ecological Psychology, 18(1), 138.
Maslow, A. (1964). Religions, values, and peak-experiences. New York: Penguin.
Maslow, A. (1968). Toward a psychology of being. New York: Van Nostrand.
May, R. (1983). The discovery of being. New York: W.W. Norton & Co.
Neisser, U., & Fivush, R. (Ed.) (1994). The remembering self: construction and accuracy in the
self narrative. Cambridge, UK: Cambridge University Press.
Neisser, U. & Hyman, I. (2000). Memory observed: Remembering in natural contexts. New York,
NY: Worth.
University Press.
Nietzsche, F. (1967). The Will to Power. New York: Vintage. (Original work published ca. 1888;
Passion & Reason
140
translated by Walter Kaufmann and R.J. Hollingdale)

Ping, R., Dhillon, S., Beilock, S. (2009). Reach for what you like: The bodys role in shaping
preferences. Emotion Review, 1(2), 140-150.
Science, 27, 183-213.
Reed, E. (1996a). Encountering the world: Toward and ecological psychology. New York:
Oxford University.
Reed, E. (1996b). The necessity of experience. New Haven: Yale University.
Reed, E. (1997). From Soul to mind: The emergence of psychology, from Erasmus Darwin to
William James. New Haven: Yale University.
Riccio, G.E. & Stoffregen, T.A. (1988) Affordances as constraints on the control of stance.
Human Movement Science, 7, 265-300.
Riccio, G.E., Van Emmerik, R.E.A. & Peters, B.T. (2001). Movement dynamics and the
environment to be perceived. Behavioral and Brain Sciences, 24, 237-238.
Rick, S. and Loewenstein, G. (2008). Intangibility in intertemporal choice. Philosophical
Transactions of the Royal Society B, 363, 38133824.
702717.
Sartre, J.-P. (1956). Being and nothingness: A phenomenological essay of ontology. New York:
Philosophical Library. (Original work published 1943; translated by Hazel Barnes)
Scherbaum, S., Dshemuchadse, M., & Kalis, A. (2008). Making decisions with a continuous
mind. Cognitive, Affective, & Behavioral Neuroscience, 8(4), 454-474.
Shaw, R., Turvey, M., & Mace,W. (1982). Ecological psychology: The consequence of a
Simon, H. (1957). A behavioral model of rational choice. In: H. Simon (Ed.), Models of Man.
New York: Wiley. (originally published in the Quarterly Journal of Economics, Volume 69,
1955)
Smith, A. (1976). A theory of the moral sentiments. New York: Oxford University Press.
(Original work published 1759).
Solomon, R. (2003). Not Passions Slave: Emotions and Choice. New York: Oxford University
Press.
Solomon, R. (Ed.) (2004a). Existentialism. Oxford, UK: Oxford University Press.
141
Riccio
Solomon, R. (Ed.) (2004b). Thinking about feeling: Contemporary philosophers on emotions.

Oxford, UK: Oxford University.
Solomon, R. (2007). True to Our Feelings. New York: Oxford University Press.
Stark, L. (1968). Neurological control systems: Studies in bioengineering. New York: Plenum.
Stoffregen, T., & Bardy, B. (2001). On specification and the senses. Behavioral and Brain
Sciences, 24, 195213.
Stoffregen, T., & Riccio, G. (1988). An ecological theory of orientation and the vestibular
system. Psychological Review, 95, 3-14.
Stoffregen, T., & Riccio, G. (1991). A critique of the sensory conflict theory of motion sickness.
Ecological Psychology, 3, 159-194.
Turvey, M., Shaw, R., Reed, E. & Mace, W. (1981). Ecological laws of perceiving and acting: In
reply to Fsdor and Pylyshyn (1981). Cognition, 9, 237-304.
United States Army Training and Doctrine Command (2009). Army Leader Development Strategy
(Draft, May 21, 2009), Ft. Monroe, VA: Training and Doctrine Command.
Van Wegen, E., van Emmerik, R., & Riccio, G. (2002). Postural orientation: Age-related changes
in variability and time-to-boundary, Human Movement Science, 21(1), pp. 6184.
Wertheimer, M. (1959). Productive thinking. New York: Harper & Brothers. (Original work
published 1945).
Werthiemer, M. (1970). A brief history of psychology. New York: Holt, Rinehart and Winston.
Section II. Verification and Validation of OBTE as a Service System
143
Jefferson et al.
Chapter 6. Initial Impressions of Participation in CATC
Tyrone Jefferson, Zachary N. J. Horn, and Kevin T. Durkee
Aptima, Inc.
The first source of evidence about OBTE as a service system is self-report data obtained from
Soldiers who had participated in CATC. The timing and open-ended format of this survey
provides an understanding of Soldier impressions of the course although not necessarily impact
on the behavior. In particular, the goal was to investigate the degree to which Soldiers made
reference to the principles (Table 1) and to the general approach of OBTE (e.g., a focus on
building confidence, deliberate thought, and understanding in students) above and beyond
learning solely about marksmanship. These data were obtained from a survey of CATC
participants that had been administered by the AWG prior to this investigation.
6.1 Methods
6.1.1 Participants
The AWG distributed the voluntary survey to individuals who participated in CATC between
April 2007 and March 2008 at Fort Benning, GA, and Fort Jackson, SC. Most respondents
completed the survey online and submitted their responses intermittently throughout the weeks
following their course participation. The response rate for the AWG survey indicates a volunteer
response rate of 14.3%. More specifically, the archival data used in the current analysis consisted
of 1,170 open-ended item responses provided by 234 Soldiers over a period of April 2007 to
March 2008. Responses were randomly selected and content-coded to identify the affective and
learning reactions of Soldiers to the OBTE training approach implemented in CATC. These
responses were gathered from DS, First Sergeants, Platoon Sergeants, Company Commanders,
Squad Leaders, Section Sergeants, Battalion Commanders, Section Sergeants, and Medics. There
were no requirements or incentives provided for survey completion. The response rate and results
should be considered in this context.
6.1.2 Procedure
The Survey was designed by the AWG as a way to document Soldiers thoughts and opinions
about OBTE, whether positive or negative in nature. The survey contained five open-ended
questions that were used for this analysis:
Did the course meet your expectations? How? Why?

What impact (positive or negative) will CATC have on the current culture of our Army?
Briefly describe your understanding of the CATC philosophy.
As a leader, were there any additional training benefits (other than marksmanship
development skills) that were gained by attending the CATC program?
Do you have any additional comments?
144
Initial Impressions of CATC

6.1.3 Analyses
Our analysis utilized the AWG surveys as archival data. For the purpose of the post hoc analysis,
the 1,170 open-ended responses were reviewed to understand Soldiers impressions after
participating in the course. Given the open-ended nature of responses, a coding scheme was
developed to capture and analyze the Soldiers responses. Specifically, multiple categories were
developed in which to classify Soldiers impressions about the course, including comments that
focus on (a) OBTE training principles (i.e., problem solving, intangibles,
understanding/awareness, deliberate thought, and combat performance); (b) weapons operation;
(c) general quality of OBTE (i.e., positive comments, perceived constraints toward implementing
OBTE in the Army, and suggestions to improve the course); and (d) resulting training transfer of
OBTE (i.e., positive vs. negative training results in oneself or others).
Three raters were assigned to identify the frequency with which responses contained references to
each of the above categories. Many responses made reference to more than one facet of the
course. Each response was eligible to be content coded into one or more of the categories
specified above. Agreement among raters for response coding was achieved by independently
analyzing a subset of responses. This analysis revealed 88% inter-rater agreement.
6.2 Results
Three key findings emerged in the AWG survey results that merit attention. First, the respondents
reported a strong overall belief that OBTE, as demonstrated in CATC, resulted in positive
training outcomes for themselves and for others (Figure 1). The frequency of these comments
suggests that a large proportion of respondents valued the instruction that was based on OBTE
principles, particularly when it had a beneficial effect on their development as Soldiers. For
example, multiple comments reported how this class definitely improved on my shooting
ability, and I have learned more about marksmanship in that one week than in my 10 years of
military service. Furthermore, there was not a single response indicating that the AWG course
had a negative effect on Soldiers, further suggesting the tangible benefits of the approach are
noticed.
A logical question, however, is whether OBTE distinguished the AWG course from other
marksmanship training in the Army? This second key finding of a positive attitude toward CATC
was by far the most commonly cited theme of all response categories. In total, there were 585
comments positively mentioning the OBTE approach, which more than tripled any other response
type. These data illustrate consensus that the Soldiers who completed the survey regard OBTE as
largely beneficial. The number of positive responses certainly suggests the potential of OBTE to
engage Soldiers in training with a degree of motivation and investment that would be welcome in
any program of instruction (Kirkpatrick, 1994).
145
Jefferson et al.
Figure 1: Frequency of responses from 234 participants in CATC.

These data suggest that OBTE, applied in the context of rifle marksmanship training, is the most
plausible reason for the positive training outcomes reported by respondents. The implications of
the responses are that OBTE concepts such as mentoring Soldiers, using problem solving, and
allowing Soldiers to assume responsibility were enablers for training rather than limitations.
Secondly, the overwhelming frequency of positive responses to the AWG course implies that
OBTE offers a more effective training approach than other training to which Soldiers have been
exposed. For example, respondents often reported statements such as: (a) Now that I am
deployed to a combat zone, I realized this training should have been available to every Soldier all
along, and (b) [I was able to see] a different perspective on training versus what is used in the
big Army. This is definitely better.
The third key finding from the AWG survey addressed the question of whether they fully
understood OBTE as a general approach to instruction and Soldier development, or if they
viewed the AWG course solely in terms of marksmanship. The survey results indicated that
respondents indeed focused their comments on OBTE principles more than on weapons operation
(see Figure 2). There were 407 references to the five combined OBTE principles throughout the
responses, compared to 165 references to weapon operation as the primary learning outcomes.
This finding suggests the majority of Soldiers understood the intent of OBTE as not simply a
marksmanship course. While many comments cited their enhanced skills as marksmen, the
distinction shown in Figure 2 suggests that most Soldiers who participate in the AWG course tend
to develop marksmanship skills as a byproduct of building problem solving skills, understanding
and awareness. Sample comments praise the course for: (a) thinking outside the box, how to
use resources available wisely, andletting us be accountable and responsible for our actions;
(b) fosters confidence through almost immediate results; and (c) makes the individual
Soldier more accountable for his actions.
146
Figure 2: Frequency of references to OBTE principles in comparison to weapons operation.

Despite the largely positive reception to OBTE by Soldiers, responses to the survey revealed
concerns about potential organizational friction associated with the implementation of OBTE in
the Army. Various roadblocks were mentioned among 63 responses to this particular question.
One particularly common response is that the institution of the Army will have difficulty with any
change of approach to training: Old school Soldiers and trying to get them to train with this
method. It is working, but slowly, [as] change is hard in the Army.
Respondents also provided suggestions for improving the AWG course. There were six
suggestions in particular that were raised by multiple respondents: (a) it should last longer than its
current one week format; (b) it should be available to more Soldiers, as this will more quickly
integrate the approach into the mainstream Army culture; (c) it should provide handouts, such as
PowerPoint slides, to help instructors begin training others with OBTE; (d) it should include
combat pistols and side arms given the growing number of pistols coming into units; (e) it should
include certification for Soldiers who complete the course; and (f) it is best suited for teaching
advanced marksmanship to more seasoned shooters but not necessarily for novice shooters. These
comments were offered in conjunction with praise for the course; the focus of these comments
was intended to enhance what they considered to be an already successful approach. The overall
conclusion from the AWG survey is that CATC, as an exemplar of OBTE, was perceived as
efficacious with respect to developmental outcomes involving improved cognition, emotion, and
motivation in the context of skill training such as marksmanship and weapons operation.
6.3 Implications for Service System Development: Peer Review
The AWG surveys are a valuable source of data on reactions of participants in OBTE
(Kirkpatrick, 1994). They should be continued as one method of peer review in OBTE (CMMI
Product Team, 2009). Post-processing of the open-ended questions helped reveal ways in which
CATC satisfied the intent of OBTE. Questions should be added to the survey that relate directly
to the principles of OBTE as in the analysis described above. There should be periodic revision of
the questions based on lessons learned, over a suitable interval and sample, from the surveys as
well as from other sources. Another source of information that should be considered is the
evolving set of measures of instructor behavior and student behavior that define the
147
Jefferson et al.
implementation of OBTE. For example, survey questions could be derived from the behaviorally
anchored rating scales used to assess OBTE implementation (cf., Klein, Salter, Gates, Sullivan,
Kinnison, Riccio, Lappin, 2005; Knapp, Heffner, & Campbell, 2003; Chapter 2). It also is the
case that, over time, lessons learned from the surveys could help refine the measures of instructor
behavior and student behavior.
We believe that it should be possible to obtain a response rate in the range of 25-30%. Methods
that could help achieve a higher response rate include: (a) a notice sent prior to survey
distribution; (b) reminder notices sent a few weeks after participation in OBTE; (c) multiple
modes of notification; (d) enhance the appearance of the surveys to connote that their feedback is
valued; (e) incentives for participation; (f) assure anonymity and be clear about the intentions; (f)
keep surveys brief and make this intent clear but allow for more extensive feedback; (g) persuade
respondents that their feedback will be influential by indicating that it will influence preplanned
improvements to OBTE; and (h) consider nested surveys in which a smaller sample of potential
respondents address more detailed issues.
Several steps can be taken to increase the likelihood of getting feedback about ways in which an
implementation of OBTE can be improved: (a) one-on-one or small group interviews made up of
trusted peers; (b) solicit feedback on neutral ground such as in a neutral conference room or
virtual meeting place; (c) let them know you are serious about receiving negative feedback; (d)
listen to what participants feel are the most important concerns before interjecting your points or
other issues; (e) avoid the perception of jumping to judgments, criticisms or rebuttals of the
information respondents are sharing; (f) limit questions that begin with "Why", and instead use
phrases like "Tell me more about..., and What suggestions do you have for me?"; (g) emphasize
learning such as whether there things we can do to better support learning; and (h) ask what they
themselves can change to support their own learning.
Peer review such as represented in the AWG surveys is a good method to address Kirkpatricks
level 1 of program evaluation (Kirkpatrick, 1994) that provides actionable feedback about the
interest and motivation of participants. Level 1 evaluation corresponds to the considerations of a
learner-centered environment. We believe that level 1 evaluation can be improved by considering
the confluence of learner-centered and community-centered characteristics of a learning
environment as emphasized by Bransford et al. (2000) (See Chapter 3). In the context of OBTE, a
community-centric orientation is provided by grounded theory (Camic et al., 2003; Denzin &
Lincoln, 2003), in particular, the definition and measurement of OBTE in terms of values and
best practices of leaders with extensive and recent combat experience. Such values and best
practices should be manifested in a community-centered learning environment. The communitycentered orientation on learner-centric issues is provided by further development of methods for
peer review that incorporate the measures of OBTE that were developed with combat experienced
leaders. This also helps address Kirkpatricks level 4 of program evaluation that attempts to
bridge the gap between impact on the individual and relevance to the collective mission and
objectives.
6.4 References
Expanding perspectives in methodology and design. Washington, DC: APA.
148
CA: Sage.
Publishers, Inc.
Klein, G., Salter, M., Gates, J. W., Sullivan, R., Kinnison, H., Riccio, G., & Lappin, M. (2005).
Sergeants as drill sergeants: returning sergeants to drill sergeant duty (Final report to the
Army Research Institute for the Behavioral and Social Sciences, contract number W74V8H04-F-0008). Columbus, GA: The Wexford Group International Inc.
Knapp, D. J., Heffner, T. S., & Campbell, R. C. (2003). Recommendations for an Army NCO
semi-centralized promotion system for the 21st Century (ARI Research Report 1807).
Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.
149
Dean et al.
Chapter 7. Local Development of Measures of Effectiveness
Courtney Dean, Fred Diedrich, Sharnnia Artis, Zachary Horn, Tyrone Jefferson
Aptima, Inc.
Gary E. Riccio
7.1 What do Instructors Believe Soldiers Should Learn in Initial Entry Training?
The U.S. Army Asymmetric Warfare Group (AWG) developed the Combat Applications
Training Course (CATC) to, among other things, familiarize instructors with the principles and
practices of Outcomes Based Training and Education (OBTE). Given this objective, it is
informative to inquire about the extent to which instructors are able to apply lessons learned in
CATC to their understanding of their own instructional activities in rifle marksmanship training.
This is important because OBTE emphasizes enduring outcomes that supplement the learning
objectives associated with Tasks, Conditions, and Standards that are explicitly specified in
training support packages. Accordingly, in rifle marksmanship training, OBTE should enable
Soldiers to master the skills being trained, develop the intangible attributes required for success in
Full Spectrum Operations, and understand the skills being trained as they relate to combat
effectiveness. However, given the obstacles typically associated with implementation of new or
modified training, it is unclear whether Drill Sergeants (DS), for example, can fully transfer the
lessons learned in CATC to Initial Entry Training (IET).
Do DS understand the changes in training associated with OBTE, and do they believe they can
integrate these changes into the Basic Rifle Marksmanship (BRM) and Advanced Rifle
Marksmanship (ARM)? Do they focus solely on Task, Conditions, and the associated Standards,
or do they also incorporate elements of OBTE for which associated values-based standards have
not yet been codified? We argue that one answer to these questions lies in observing what DS do
while instructing (see Chapters 2, 8, and 9). We also believe it is useful to ask DS what they think
their influence on students (Privates) should be. In other words, what are they trying to achieve,
and what effects on Soldiers do they think they should be able to observe in BRM and ARM? We
believe that this is important because it can shed light on gaps in specific interventions (e.g.,
CATC) designed to familiarize instructors with OBTE and it can reveal potential obstacles to
implementation of OBTE.
More specifically, we inquired into these issues by working with in the development of a
comprehensive set of metrics for describing and assessing Soldier behavior. The development of
measures provides a systematic way of describing observable and verifiable effects of instruction
on Soldier behavior with respect to expectations of instructors and learning objectives for a
course. In addition, such measures can be used as dependent variables or covariates in assessing
the effects of instruction to the extent that they are comprehensive or at least representative of the
intent of instruction. They also can be used as measures of effectiveness of OBTE to the extent
that DS are able to transfer lessons learned about OBTE to their own instruction (see Chapters 8
and 9). An aim of the work reported in this chapter is to come to understand whether and to what
extent measures of effectiveness for OBTE can be developed with instructors who have been
familiarized with OBTE as in the attempted implementations at Fort Jackson and Fort Benning.
Tentative Measures of Effectiveness
150
We worked with CATC-trained instructors to identify desired Soldier behaviors in the form of
measures that capture areas of relative strength and weakness in terms of the abilities of Soldiers
from the DS perspective. During the development of the performance measures, we worked with
the DS to leverage their knowledge regarding critical skills that are manifested in BRM and
ARM. As described below, the results of this process yielded BRM and ARM performance
measures that are:
Behaviorally anchored. Behavioral anchors provide raters with observable features of

performance that observers can link to ratings on a scale.
Designed to be taken at specific intervals in the training program. Measures taken at
specific intervals address performance at critical phases in the training, rather than as an
average across the entire course of instruction, and allow the rating to be tied to specific
phases in the training.
Focused on aspects of performance that are not captured in the qualification
standard alone. Items assess those aspects of performance that require observation
beyond qualification scores (for example, following safety rules and moving and
positioning correctly).
Useful for assessment of knowledge and skills that are exercised in the training
environment. Collectively, the items are designed to reflect critical training objectives
from the perspective of the DS.
7.2 Measure Development Process
Our objective was to develop a comprehensive set of competency-based measures for the
assessment of marksmanship performance of individual Soldiers (Privates) attending BRM and
ARM. Competency based measures are systematic descriptions of what constitutes good
performance in a particular job or task, and the knowledge and skills needed for that job
(MacMillan, Garrity, and Wiese, 2005). More specifically, to ensure that these behavior-based
measures are meaningful and pertain to constructs for which data can be reliably collected, we
utilized a measurement development approach that combined subject matter expertise with
psychometric practices to produce measures of observable behavior. This approach, called the
COmpetency-based Measures for Performance ASsessment Systems (COMPASSSM)
methodology (MacMillan, Entin, Morley, & Bennett, in press; see Chapter 2), strives to arrive at
a set of performance measures that are both feasible in application and meaningful in
interpretation. Here, the COMPASS methodology was used to operationalize the DS assumptions
and expectations about the desired effects of instruction in IET.
At the heart of the COMPASS methodology is the development of measures using a combination
of subject matter expertise and psychometric theory. Accordingly, the COMPASS process
consists of a series of three workshops that leverage insights from both subject matter experts and
scientists, operating on relatively equal footing. In this case, the experts were DS who had the
knowledge needed to create measures that were relevant to the domain of IET. In the present
context, performance measures were developed to assess the behavior of Soldiers, typically
Privates, engaged in BRM and ARM. Accordingly, the first COMPASS workshop was conducted
in a group setting with experts from the 198th Infantry Training Brigade at Fort Benning, GA, in
April of 2008. DS (ranging from Staff Sergeant to Sergeant First Class) from the 198th who had
already taken CATC were the primary participants and supplied the knowledge necessary
regarding the objectives of BRM and ARM. Participant DS were instructed to discuss training in
the context of their understanding of the philosophy of the CATC course. They were provided
with reminders of the principles of CATC. Although in this workshop the full range of IET was
briefly discussed (e.g., BRM, ARM, MOUT, First Aid, etc.), BRM and ARM were selected as the
151
Dean et al.
domains for assessment given the emphasis of CATC. The output of the first workshop was a set
of nested categories of performance indicators for rifle marksmanship training from which
measures could be developed.
The second workshop consisted of one-on-one interviews (two hours each) with the same DS
from the first workshop to obtain detailed information with which to develop the behaviorally
anchored measures and scales. The interviews were also conducted in April of 2008, and focused
on identifying examples of Soldier behavior that would illustrate novice, average, and expert
behavior associated with each performance indicator. The DS were asked a variety of questions in
order to (a) obtain information on the behavior demonstrated by Soldiers during each period of
training; (b) elicit behavioral anchors relevant to each performance indicator; and (c) determine
the appropriate type of measures for each performance indicator (i.e., simple yes/no distinctions
vs. gradations in performance best collected via rating scales). The behavior identified in the
interviews of the second workshop were then analyzed along with notes from the interviewers to
develop a set of candidate behaviorally-anchored measures that describe performance for each
performance indicator. Single performance indicators could be represented by one or more
measures that describe various aspects of the observable behavior for either individuals or teams
of Privates.
The third and final workshop, conducted in May of 2008, presented a group of DS from the 198th
Infantry Training Brigade with a draft list of behaviorally anchored performance measures.
Performance measures were reviewed with the group to ensure that the items were seen as
practical and useful. DS reviewed the performance measures with respect to their associated
performance indicators according to the following criteria: relevance, observability, measurement
type (e.g., scale, yes/no, checklists), measure wording, scale type, and scale wording. Each
performance measure was discussed in detail with real-time edits and additions based on the input
from the DS participants. The result of the third COMPASS workshop was a finalized set of
measures for the assessment of Soldier performance that are applicable to both BRM and ARM.
7.3 What do OBTE-Trained DS Believe is Important to Assess in BRM/ARM?
Overall, assessment of the measures developed with DS trained in OBTE indicated a general
attitude towards BRM and ARM reflecting the explicit doctrine and common practices associated
with Tasks, Conditions, and Standards that focus on knowledge and skill (Dean, Diedrich, Artis,
Horn, Jefferson, & Riccio, 2008). Nevertheless, some measures did reveal aspects of
understanding consistent with OBTE (compare with survey results discussed in Chapter 9). For
instance, consider the example measure shown in Figure 1. This sample set of metrics is focused
on the ability of the Soldier to correct malfunctions. The first example focuses on a procedure
known by the acronym SPORTS, specifically completion of SPORTS in a timely manner. The
second example addresses knowledge of the rifles cycles of operation. In particular, the anchors
for the measure reflect a difference between blindly following the standard sequence of actions
and solving a problem using knowledge of cycles of operation. Hence, in this measure developed
with DS, there is an emphasis on assessment of the Soldier in terms of understanding and
problem solving skills. This is consistent with OBTE principles as opposed to addressing
SPORTS as a conditioned response.
Another example is the measure shown in Figure 2. In this example, the focus is on zeroing of
the weapon, with particular emphasis on the behavior of the Soldier being observed. The
difference between the anchors suggests failure to react on the one extreme and self-correction of
errors on the other extreme. The anchor for a 5 suggests an emphasis on self-correction and
152
making adjustment without direct DS guidance. Hence, this measure demonstrates DS emphasis
on Soldier accountability and initiative, important attributes emphasized in OBTE.
Does the Soldier properly apply SPORTS when faced with a malfunction?
N/A
N/O
Fails to perform
SPORTS correctly
Performs SPORTS with

one or two errors; selfcorrects
Correctly performs
SPORTS in a timely
manner
Does the Soldier demonstrate knowledge of interruption of cycles of operation?

N/A
N/O
Repeatedly performs
SPORTS but does not
recognize the need for
remedial action
Performs SPORTS twice then

initiates remedial action with
limited success or with minor
errors that prevents correction
(i.e., fails to drop magazine)
Terminates SPORTS
when malfunction is
diagnosed/resolved; or
initiates remedial action
and successfully corrects
malfunction
Figure 1. Example measures related to correction of malfunctions.
Does the Soldier adjust sights using feedback from the target?
N/A
N/O
Does not adjust sights

based on feedback
Adjusts sights based on

feedback with some
assistance
Adjusts sights
effectively without
assistance
Figure 2. Example measures related to zeroing of the weapon.
153
Dean et al.
Figure 3 shows a measure related to the use of barriers for supported firing. Significantly, the use
of such barriers in ARM did not exist at Ft. Benning and Jackson prior to introduction of CATC.
From the perspective of OBTE, the use of such barriers in training, and in particular the
employment of such barriers for cover and support, generally reflects a commitment to training to
improve combat performance. Relative to firing without such barriers, the barriers begin to
introduce elements that provide the opportunities (and potentially costs) associated with more
realistic combat situations.
Does the Soldier utilize barriers?
N/A
N/O
Exposes too much of

their body; will not use
barrier to support
themselves
May not use best position

for each barrier; Does not
use barrier as support all
the time
Consistently shields
body using barrier; uses
barrier as support
Figure 3. Example measures related to use of barriers.

As these three examples illustrate, many of the measures of Soldier performance contained
elements consistent with OBTE and practices utilized in CATC. It is important to note, however,
it is possible that many of these elements could simply reflect beliefs that were in place long
before the experience of DS in CATC. Hence, although suggestive, these findings are not
conclusive. In any case, they are noteworthy given the focus of the DS on items such as
understanding and problem solving as, for example, in the introduction and use of barriers to
create a learning environment conducive to such cognitive engagement of the learner.
Does the peer coach assess shot group to identify errors in fundamentals?
N/A
N/O
Unable to spot errors;

provides bad feedback
Identifies errors in a
specific fundamental and
provides some feedback
Identifies errors in
fundamentals and
provides feedback
Figure 4. Example measures reflecting interpretation of shots.
154
Discussion with the DS during the COMPASS workshops indicated that many aspects of
instruction had changed since the experience with CATC. DS noted such changes as the use of
barriers, changes to standard practices for weapons orientation on the range (down at the low
ready vs. pointed in the air), and the elicitation of shot group diagnosis from students or their peer
coach (see Figures 3 and 4). DS recognized and could articulate the value of some of these
changes.
More generally, there is ample evidence in the measures and associated comments of the DS of a
lack of full integration of lessons learned from CATC. For instance, the DS generally expressed
that OBTE applied well to ARM, but not to BRM, and these comments centered around surface
features such as a focus on movement in ARM as in CATC. DS often recognized some of the
more concrete and particular aspects of CATC such as the employment of barriers and the use of
scenarios involving position changes and rapid reloads. These scenarios simulate the combat
experience much more so than BRM activities where the shooter remains in a single position for
an entire cycle or only changes positions or reloads on command. Hence, BRM was often deemed
to be too basic for OBTE. This indicated a lack of understanding of general principles and
strategies of the approach that can be applied to almost any instructional situation (see e.g.,
Chapter 3).
It is illuminating to consider some aspects of CATC that DS could, in principle, apply in BRM as
well as ARM. Arguably, some of the most compelling learning occurs during the first days of
CATC when the cadre strategically exercises gradually less range control during progressive
cycles. After several strings during grouping and zeroing exercises, DS students find themselves
policing themselves, ensuring the range is no longer hot, and heading down to the targets without
specific instruction from the cadre. This practice is an exceptional demonstration of several of the
principles of OBTE (see Chapter 3). Though it may not be possible to recreate this identical
environment in BRM, aspects of it could be employed and Privates could be induced to model
and demonstrate reciprocal initiative and accountability. Moreover, many of the most critical
aspects of marksmanshipthose that may provide critical contextual understandingpertain to
BRM more than to ARM because of the importance of establishing fundamentals as early as
possible in training and thus minimizing the likelihood of having to undo bad habits in later
training. Specifically, a deeper understanding of fundamentals is reflected in proper interpretation
of shot patterns and zeroing of the rifle which can be facilitated by an appreciation of minutes of
angle and ballistics. A conceptual appreciation of such factors does not require an understanding
of the physics or trigonometry as such as long as it promotes awareness of sources of feedback
that have actionable implications. These aspects of marksmanship are heavily emphasized in
CATC but may be overlooked in the application of CATC to other training programs because of
their more academic connotations or because of an assumption that they are adequately taught
somewhere else at some other time.
Many of the measures reflected a heavy reliance on the existing formal program of instruction
and on a related narrow view of Tasks, Conditions, and Standards. While OBTE does not dismiss
Tasks, Conditions, and Standards, it nevertheless does focus on intangible influences on behavior
that are not readily apparent in many of the measures developed with the DS. For instance, Figure
5 shows several measures that focus explicitly on meeting the standard, following the safety rules,
and qualification. Although OBTE does not suggest that such considerations are inappropriate, it
is notable is that 78 of the approximately 88 measures developed with the DS (89% of the
measures) did not mention behavior consistent with the intangibles emphasized in OBTE. Thus,
while some measures and comments suggest that CATC influenced the thinking of DS about
instruction, the majority of these measures showed no such orientation.
155
Dean et al.
Does the Soldier fire a group to standard (e.g., 4 cm radius)?

N/A
N/O
Shots do not group
Shot group meets

standard
Shot group
exceeds standard
Does the Soldier adhere to the four safety rules?

N/A
N/O
Repeatedly violates one

or more safety rule
a.
Adheres to safety rules

after being reminded
or self-corrects
Always adheres to
safety rules
What was missed?

Weapon orientation
Keep your finger off trigger
Treat weapon as loaded
Be aware of target
Did the Soldier qualify?

Yes
No
N/A
N/O
Figure 5. Example measures focused on standards and qualification. .
156

7.4 Implications
We developed the measures for two primary purposes. The first was to develop measures of what
Soldiers actually do as students in BRM and ARM in a study of what DS actually do (see
measures in Chapter 2) within the framework of OBTE (see Chapters 8 and 9). Second, we
recognized the measures developed with DS who participated in CATC would provide
indications about the DS understanding of OBTE as a result of their exposure to it in CATC as
well as in briefings about OBTE that they may have received in their current or prior
assignments. The measures reflect the extent to which DS understand how they can apply OBTE,
and hence, the measures shed light on the extent to which DS may transfer ideas from CATC to
their own training environments.
There was a conspicuous lack of measures developed by the DS that were specifically directed at
the tracking development with respect to attributes such as confidence, initiative, and
accountability and associated capabilities such as judgment, deliberate thought, discipline, and
awareness. This point is illustrated in Figure 6 that shows some measures constructed by the
progenitors of OBTE as examples of formative assessments that, in principle, could be applied to
Soldiers to shed light on development of such attributes (see Chapter 2; Appendix A). It is
noteworthy that the measures shown in Figure 6 attempt to assess individual Soldier development
beyond the training task in isolation. They illustrate, in stark contrast to the majority of measures
created by the DS, how Soldiers could be assessed in ways that go far beyond skill-based or
knowledge-based assessments to include values-based assessments as well (see Epilogue).
The survey findings described in Chapter 6 suggest that Soldiers can come to understand the
deeper mission intent of OBTE through their participation in CATC. This is reinforced by survey
findings described in Chapters 9 and 10. Our interactions with the DS during the COMPASS
workshop suggest, however, that they had some difficulty applying lessons learned from CATC
to the conditions of their own instruction. The question becomes what might account for this
apparent relative lack of transfer, and what can be done to promote it in the future? In considering
these questions, it is useful to start by considering general influences on transfer. For instance, in
their review of the training transfer literature, Burke and Hutchins (2007) note three broad classes
of issues that can mitigate transfer, including learner characteristics (e.g., cognitive ability, selfefficacy, motivation), intervention design and delivery (e.g., learning goals, content relevance,
practice and feedback), and work environment (e.g., supervisory support, peer support,
opportunity to perform). As the authors note, each of these factors has the potential to positively
or negatively influence transfer. Given our focus, we concentrate on issues concerning the OBTE
intervention and the nature of the work environment (see also Chapter 14).
First, with respect to the nature of the intervention, it is important to note that CATC, as an
example of OBTE, is ostensibly focused on marksmanship. Indeed, during the week long CATC
to which the DS studied here were exposed, the focus was almost solely on marksmanship, with
only some discussion on application of the methods to a different domain (e.g., land navigation).
Unfortunately, it appears that the attention of DS, as instructors, is drawn to the most concrete
and immediately relevant aspects of CATC such as the use of barriers, scenarios, and steel
targets. We believe the novelty and validity of the techniques utilized in CATC tended to divert
attention of DS away from the personal attention paid to the individual, the discussions, the
thinking, and self diagnosis that are broader intent of OBTE. From the perspective of OBTE, it is
not the use of barriers and scenarios that make Soldiers better, but rather the barriers and
scenarios are simply there to provide a context for development, to set the conditions for learning,
(see e.g., Dewey, 1915/2008; see also Bransford, Brown, and Cocking, 2000).
157
Dean et al.
Does the Soldier demonstrate problem-solving skills when facing a unique situation?
N/A
N/O
Soldier does not

demonstrate recognition
of the problem; no
change in behavior
Soldier recognizes problem;

applies training to the
problem
Soldier recognizes problem;

properly applies training to
the problem with positive
results
Is the Soldier able to integrate task and skills into future training (e.g., understands the context of
the skill, task)?
N/A
N/O
Does not integrate

skills
Integrates some skills

into other activities
Integrates skills and

lessons learned across
the training program
Does the Soldier maintain deliberate thought and action as stress is added to training?
N/A
N/O
The Soldier responds

reflexively and in an
ineffective manner;
performs task faster than
personal capability; does
not think through the
issue
The Soldier attempts to

assess the situation, but
struggles to respond in a
deliberate and purposeful
manner; performs task
slightly faster than
personal capability
The Soldier assesses the

situation and responds in
a deliberate, purposeful
and effective manner;
performs task at a speed
proportional to personal
capability
Figure 6. Example measures focused on problem-solving and deliberate thought.
158
It is possible for participants in CATC to miss these larger lessons about instructional principles
and strategies because of their inherent interest in marksmanship (see Chapters 2 and 3).
Moreover, exposure to only one exemplar generally is not effective in promoting generalization
and abstraction. As indicated above, the DS had an easier time applying lessons learned to ARM
than to BRM because ARM shared more superficial similarities to CATC (e.g., an emphasis on
movement and communication). To the extent transfer occurred, it tended to be near rather than
far in nature (cf., Leibrecht, Wampler, & Pleban, 2009).
Second, also with respect to the nature of the intervention, there was relatively little support
provided to CATC students following the marksmanship-specific class to support transfer. For
instance, elements such as take-home materials that explore application to different domains
could be provided, perhaps employing multimedia-based vignettes that challenge the learner to
apply lessons. Similarly, such materials and exercises could be employed in workshops that
promote student application to domains outside of marksmanship through the creation of
problems that require application of the OBTE approach (Sidman, Riccio, Semmens, Dean, &
Diedrich, 2009). The critical point is to provide support that challenges students to apply OBTE
to a variety of applications. Without such additional support, it is likely that students will be
unable to fully abstract the larger lesson on instructional strategy.
Third, with respect to work environment, an essential result that emerged was that the DS
indicated that they felt somewhat constrained by the nature of the programs of instruction and
associated resources. Lack of conspicuous transfer may be rooted, at least in part, to the
constraints associated with the BRM and ARM curriculum and the extent to which the DS
believed they could deviate from the curriculum (see Chapter 9). The programs of instruction
mandate particular sequences of learning events with associated logistical constraints on
schedules and activities (i.e., range time can be limited, as can instructor-student ratios). It is not
surprising, then, that DS stayed very close to common historical assumptions about the programs
of instruction in their definition of measures for student outcomes. In some instances, they even
expressed that they had to address certain items or perform certain actions because it was
mandated. As an example, during BRM and ARM, Soldiers and ammunition are typically kept
separate as much as possible. Privates handling weapons are typically not authorized to carry
loaded magazines down the line. Other Soldiers, whose rifles are safely stowed away, distribute
these magazines. Therefore, despite the fact that Soldiers are expected to carry hundreds of
rounds and loaded weapons on a daily basis in theater, in BRM and ARM, they not permitted
access to live ammunition until the last possible moment (e.g., immediately before being
instructed to fire down range). This artifact of marksmanship training, designed to promote
safety, works against the development of accountability and confidence in young Soldiers,
outcomes given priority in OBTE.
7.5 Conclusions
A key challenge, given the implications describe above, is the extent to which training curricula
can be made to be more flexible (see e.g., Chapters 14, 15, and Appendix C), and DS can be
empowered to believe that they can take initiative in order to more fully implement OBTE. Such
an organizational climate is viable because OBTE emphasizes consideration of values-based
objectives consistent with requirement in existing Army doctrine (see Chapter 1 and Epilogue)
but not at the expense of Task, Conditions, and Standards and associated safety initiatives.
Rather, OBTE suggests that there are other ways to achieve existing objectives that are more fully
compliant with the expectations of Army training doctrine in general and preparation of Soldiers
for Full Spectrum Operations in particular. It implies that a new emphasis should be given to the
159
Dean et al.
balance between short-term learning objectives with longer-term developmental outcomes (see
Chapter 3).
It has become clear that the best way to achieve a balance of learning objectives and
developmental outcomes is to engage the instructional cadre and their chain of command in the
development of outcomes and associated observables that can help instructors plan for and track
progress during implementation of OBTE (Appendix C). While this was done to some extent at
Fort Jackson and Fort Benning, the collaborative process and the associated staff and faculty
development with respect to OBTE are still in their infancy (see Epilogue). In fact, they are the
most rapidly developing aspects of OBTE. As such processes are improved, instructors will gain
a deeper and more general understanding of OBTE, and better measures of the effectiveness of
OBTE will follow (see Epilogue). Recent experience with such organizational change initiatives
in the Army Reconnaissance Course at Fort Knox (Perry & McEnery, 2009) and the Department
of Military Instruction (Haskins, 2009) at West Point indicates tends to support these predictions.
7.6 References
Burke, L.A., & Hutchins, H.M. (2007). Training transfer: An integrative literature review. Human
Resources Development Review, 6, 263-296.
Dean, C., Diedrich, F., Artis, S., Horn, Z., Jefferson, T., & Riccio, G. (2008). Outcomes-based
training and education: Student measures for rifle marksmanship. Report to the Asymmetric
Warfare Group (Contract W9113M-06-D-0005). Vienna, VA: Wexford-CACI, Inc.
Haskins, C. (2009). Outcomes-based training and education. White Paper solicited by TRADOC
Headquarters, August, 2009.
Leibrecht, B., Wampler, R., & Pleban, R. (2009). Methodology for evaluating transfer of learning
from the U.S. Armys advanced leaders course. Research Product 2009-05. Washington, DC:
U.S. Army Research Institute for the Behavioral and Social Sciences.
MacMillan, J., Garrity, M. J., & Wiese, E. W. (2005, November 25). The value of metrics: You
cant train what you cant measure. Mass High Tech. Retrieved June, 2009, from
http://tinyurl.com/km25pw.
Implementation in Rifle Marksmanship Training
160
Chapter 8. Observations of Behavior and Communication in Rifle Marksmanship Training

Zachary N. J. Horn, Fred Diedrich, Courtney Dean,
Sharnnia Artis, Tyrone Jefferson, Ryan Marceau
Aptima, Inc.
The second source of evidence about OBTE as a service system is a series of observations about
the extent to which participants subsequently employed the approach as instructors. Specifically,
this evaluation focused on DS when instructing Privates in rifle marksmanship as part of Initial
Entry Training. Such observations indicate whether OBTE is verifiable in instructional practice.
Verifiable implementation of OBTE is central to understanding the potential effectiveness of
courses such as CATC as a means to promulgate OBTE.
8.1 Methods
8.1.1 Participants
DS who had participated in the AWG course were observed while instructing Soldiers (typically
Privates) in rifle marksmanship. Observations were conducted at Fort Jackson (Columbia, SC)
and Fort Benning (Columbus, GA), in May and June of 2008. Seven Companies were observed
across both locations, such that the observations were made on 236 participants (46 DS and 190
Privates, randomly selected), although three of the original 46 DS observations were removed
from the current analysis due to incomplete data. Five Companies (29 DS and 118 Privates)
conducting Basic Rifle Marksmanship (BRM) training were observed at Fort Jackson, while two
Companies (17 DS and 72 Privates) conducting Advanced Rifle Marksmanship (ARM) training
were observed at Fort Benning.
8.1.2 Procedure
The investigation was conducted in BRM exercises two through 11, which were focused on
grouping, zeroing, Location of Miss And Hit (LOMAH), field fire pop-up target shooting, and
pre-qualification exercises. Observations of ARM training covered exercises two through six,
which focus on shooting with barriers, weapons malfunctions, reflexive fire, ready up drills,
target selection and discrimination, night fire, and moving targets.
For the current study, all observations were conducted in three-person observer teams in order to
capture the dynamics between an instructor (DS), the student being trained (a Private), and a peer
coach when applicable (i.e., other students who help the shooter by commenting on technique and
shot results). The CATC instructors (cadre) and OBTE designers from the AWG captured data on
instructor behavior; members of the research team who had taken CATC captured data on student
behavior, as they were able to recognize and code behavior associated with rifle marksmanship. A
member of the research team who had not taken CATC captured data on communication between
the DS instructor, Privates as shooters, and Privates as peer coaches.
For BRM, three three-person observer teams recorded the behavior of 29 DS instructors and 118
Privates. These observations were made over the course of five days and approximately 40 hours
of observations. Each observation in BRM (i.e., interaction between the DS and Private as well
as peer coach when applicable) lasted approximately 30-60 minutes. For ARM, three three-person
observer teams recorded the behavior of 17 DS instructors and 72 Privates. These observations
161
Horn et al.
were made over the course of five days and approximately 40 hours of observations. Each
observation in ARM lasted approximately 30-60 minutes. For every observation session, the same
DS instructed multiple Privates and peer coaches; thus, observations were conducted for multiple
Privates and peer coaches per DS.
8.1.3 Analyses
8.1.3.1 DS Behavior. Measures of DS instructional behavior were developed with AWG
personnel and OBTE experts to identify observable behavior associated with the instructional
approach (Chapter 2). Thus, the behavior rated were those expected of DS who effectively apply
OBTE to the way they instruct Privates in rifle marksmanship (Darwin, 2008a,b). The measures
employed in this assessment consisted of approximately 45 Likert scale, yes/no, and checkbox
items. The measures enabled the execution of training to be assessed. These observer-based
measures of DS instructor behavior have been used in a series of data collection efforts for the
AWG with an inter-rater agreement of 94%, in which agreement is defined as identical responses
on yes/no items or scale ratings within a single scale value. An example of a DS performance
rating scale is shown in Figure 3. This particular measure captures the extent to which a DS
articulates the purpose of training to Soldiers in training.
Figure 3. Example measure for assessing OBTE behavior of DS.

8.1.3.2 Behavior and Performance of Privates. Measures for the behavior of Privates were
developed with input from DS who participated in the AWG course (Dean et al., 2009). The
intent was to capture the set of behavior that Privates are expected to learn in rifle marksmanship
(Darwin, 2008a,b). The measures consisted of approximately 235 items that covered all stages of
rifle marksmanship. The measures reflected the DS perspectives on the objectives of rifle
marksmanship training in light of what they learned in CATC.
The measures consisted of Likert-scale, yes/no, and checkbox items. An example of a rating
scale is shown in Figure 4. This sample measure captures the extent to which the Private
demonstrates deliberate thought by considering the specific steps necessary to get the rifle back in
operation when required to reload. These observer-based measures of Private behavior (Dean et
al., 2009) have been used in a series of data collection efforts for the AWG with an inter-rater
agreement of 77%, in which agreement is defined as identical responses on yes/no items or scale
ratings within a single scale value.
162
Figure 4. Example measure for assessing behavior of Privates.

8.1.3.3 Communication. In order to capture the communication patterns between DS, Privates,
and peer coaches (when applicable), measures were developed to conduct a quantitative
assessment of the various types of communications, communicative elements, and utterances.
Communication between the DS, Privates, and peer coaches were directly observed by the
observer teams and were categorized according to a predetermined set of criteria that reflect key
communication patterns among DS instructors, Privates, and the peer coaches.
Members of the observer teams who were responsible for assessing communications observed the
frequency of communication for the following seven types of communication patterns:
Constructive/Diagnostic, Performance, Guided learning/Self-discovery, Directive, Non-verbal
Correction, Acknowledgment, and Non-task related questions. The categories of
Constructive/Diagnostic and Guided learning/Self-discovery were derived through a qualitative
examination of the types of communications emphasized in OBTE. In contrast, the Performance,
Directive, Non-verbal Correction, Acknowledgment, and Non-task related categories are
indicative of the directive communication typically given more weight in traditional Army
training. This categorization technique was initially developed and used in previous research on
the topic of adaptive communications analysis (Entin, Diedrich, & Rubineau, 2003).
The Constructive/Diagnostic communication category was defined as thoughtful feedback (e.g.,
from instructor) designed to correct a perceived weakness or inconsistency in the approach used
by the student. Exchanges that were categorized as Performance communication were those that
referred to the quality of performance on which one is being evaluated. The Guided learning/Selfdiscovery category represented those comments in which, for example, the instructor would ask
one or more questions that prompt the trainee to discover the causes and consequences of
particular performance outcomes. Directive communications were in the form of a command,
with no accompanying explanation or rationale for such an action. The Non-verbal Correction
category captured instances in which someone physically moved the student, or the weapon, into
proper placement. This category also captured hand signals that were used in place of verbal
interaction to indicate corrections or adjustments. The Acknowledgment category denotes
communication in which the student indicates understanding of the communicated message.
Lastly, the Non-task related questions category captured any conversations or statements that
were not relevant to the exercise. These observer-based measures of communication have been
used in a series of data collection efforts for the AWG with an inter-rater agreement of 93%, in
which agreement is obtained when each rater indicates the same degree of communication
frequency (i.e., high, medium, and low) within a single communication category per observation.
163
Horn et al.
8.2 Results
8.2.1 Behavior of DS
For the DS behavioral measures, the anchors of the five-point Likert rating scales typically
ranged from behavior associated with current training methods (rating of 1) to behavior
associated with application of OBTE to training (rating of 5). For this reason, it is important to
note that any ratings above a score of 1 indicate the use of instructor behavior that reflects some
elements consistent with OBTE. Overall ratings of DS as instructors in rifle marksmanship
suggest that these DS applied some aspects of OBTE to their training approach. The findings
from these observationswith a mean score of 2.56 (N = 43, SE = 0.16) on a five-point rating
scaleindicate that DS did not fully implementing OBTE in the rifle marksmanship course or
perhaps didnt fully understand how they could do so.
8.2.1.1 Overall Pattern. The DS behavioral measures were grouped by the overarching
instructional categories that were identified by OBTE designers as indicators of their approach to
training (Chapter 2). These categories included the following instructor behavior:
Customizing instruction to individual learning needs (i.e., increase training difficulty with
students readiness level)
Creating a positive learning environment (i.e., balancing instructor power and leadership
by taking a mentorship role and helping students)
Exhibiting intangible attributes (i.e., serve as a behavioral role model for intangibles such
as confidence, accountability, and initiative)
Communicating parameters of learning (i.e., explain why a certain exercise is important,
and guide students to discovering its relevance)
Facilitating learning of concepts (i.e., structure the problem and coach the student toward
a solution)
Using scenarios to facilitate learning (i.e., explain connections between elements of
scenarios that appear different)
Emphasizing combat or mission success (i.e., explain how each task relates to mission
success)
Using measures of effectiveness and self-evaluation (i.e., constantly evaluate when to
incorporate the next level of training difficulty)
Conducting After Action Reviews (AARs; i.e., facilitate discussion of lessons learned to
promote student self-development)
The average behavioral ratings of DS (Table 3) indicated that they exhibited some behavior
consistent with OBTE (i.e., average ratings were greater than 1). Relatively higher mean scores in
the Customize instruction and Create a positive learning environment behavioral categories
suggest that the DS exhibited some behavior consistent with a learner-centered environment
(Bransford et al., 2000). For example, a Soldier was struggling to tighten his shot group during a
BRM exercise, despite direction from the DS on proper breathing, positioning, and trigger
squeezing. The instructor began to grow frustrated in this situation but maintained his focus on
Soldier development. Upon further inspection the instructor noticed that this Soldiers earplugs,
while worn correctly, did not fit appropriately and the sound of the rifle caused the Soldier to
flinch every time his weapon fired. The instructor, recognizing this as a problem unique to that
Soldier, provided the Soldier with an alternative style of earplug. In this situation, the instructor
recognized a problem unique to a student and provided a customized solution that allowed the
student to focus on the fundamentals and become more confident as a rifleman. Had this gone
unnoticed, the student and instructor could have perceived the inability to group shots as a lack of
skill, leading to decreased confidence in the student as a rifleman. Additionally, the instructor
acted as a mentor in this case by demonstrating effective problem solving skills in addressing the
164
students individual predicament. However, the relatively low mean scores across all instructional
categories relative to ideal OBTE instruction (score of 5) illustrate that DS did not display the
consistency of OBTE behavior that would indicate a more complete transfer of training from
CATC.
Table 3. Descriptive statistics for the presence of OBTE instructional behavior
OBTE Instructional Category
Customize instruction when possible based on constraints
n
39
Mean
2.82
SE
0.17
IQR
2.00
Creates a positive learning environment
42
2.68
0.17
2.00
Instructors exhibit intangibles in own actions
42
2.59
0.17
2.00
Communicate the parameters of learning; the "why"
44
2.54
0.16
2.00
Facilitates learning of concepts
45
2.51
0.15
1.00
Uses scenarios to facilitate learning
28
2.42
0.22
1.00
Training emphasizes broad combat or mission success
42
2.35
0.14
1.00
Instructors utilize measures of effectiveness & self-evaluation
30
2.31
0.20
1.00
Hot washes and Mini-AARs
31
2.30
0.23
2.00
2.49
0.19
1.56
Overall Average
Note: n represents the number of DS for whom this instructional behavior was observed; SE
represents the standard error, pooled across all DS observations; IQR represents the interquartile
range between the first and third quartiles of DS scores (i.e., middle 50% of scores).
8.2.1.2 Analysis of the extremes. The five highest-rated and lowest-rated types of behavior from
the ungrouped data are listed in Table 4. Overall, the patterns of ratings indicate that DS
demonstrate some ability to assume a mentor-like relationship with their Privates. Instructors took
steps to create a less confrontational environment than what is typical during the first few weeks
of basic training. This means that small mistakes become more acceptable. The intent of this
mentorship is to transfer the ownership of the training to the students and build students
confidence as they learn to perform under stress (Lave & Wenger, 1991).
In contrast, the lowest-rated types of behavior indicate that DS were not helping the Privates
discover and understand the purpose of training and its intended combat application. For
example, although the DS incorporated escalation of complexity and stress into training, these
data suggest that they did not utilize stress effectively (i.e., helping Privates understand why stress
is being added, how it affects them, and how to manage it). If stress is escalated by emphasizing
throughput (movement to the next activity) regardless of learner state, DS may be restricting
Privates from learning the basics, and their confidence is likely to suffer. In addition, failure to
address the why of training adequately may have detrimental effects on motivation and sense
making (Deci & Ryan, 2008; Guay, Ratelle, & Chanal, 2008). Thus, these data suggest that while
DS did exhibit some behavior that was consistent with OBTE, it is important for DS to develop a
better understanding of the importance and potential role of communication in making progress
with respect to developmental outcomes during rifle marksmanship training.
165
Horn et al.
Table 4. DS highest-rated and lowest-rated behavior

Top 5 DS Measures
Mean
SE
Does the instructor balance perception of power vs. leadership?
46
3.21
0.18
Do the instructors adapt coaching to individual Soldiers?

Do the instructors effectively exhibit intangible attributes in their own
behavior as they conduct their training?
46
3.15
0.18
46
3.08
0.22
Do the instructors conduct AAR/Hot washes as needed?

Does the instructor assume the role of helper in Soldiers pursuit of
success?
24
3.05
0.30
45
2.91
0.19
Bottom 5 DS Measures
Do the instructors ensure the Soldiers can articulate how to apply
concepts to new situations?
Do the instructors ensure that the Soldiers recognize the effects of their
actions on their teams?
Do the instructors introduce the Soldiers to a broad range of stressors
and cultivate stress management?
Do the instructors encourage the Soldiers to think about how the stress
affects their capabilities?
Do the instructors focus the why of training back to the relationship
between the individual and the big picture/mission?
Mean
SE
28
2.02
0.19
11
2.00
0.43
22
1.98
0.24
32
1.86
0.17
28
1.75
0.17
Note: Lower numbers of observations reflect particular DS behavior that was at times not
observable.
8.2.2 Behavior and Performance of Privates
In order to observe the performance of Privates in rifle marksmanship, behavioral anchors for the
5-point Likert rating scales were developed with input from DS previously trained in CATC
(Darwin, 2008b; Chapter 2). These anchors ranged from behavior failing to meet expectations
(rating of 1), average standards for acceptable behavior (rating of 3), to behavior exceeding
standards of acceptable behavior (rating of 5). Consequently, scores above 3.0 reflect behavior
that exceeds average expectations for Privates. Overall, ratings of Privates had a mean value of
2.96 (N = 190; SE = 0.26).
8.2.2.1 Opportunities for Improvement. Measures relating to a specific behavioral category
(e.g., safety, firing positions, marksmanship fundamentals) were combined for further analysis of
the extent to which Privates performed effectively when presented with varying opportunities.
Overall means were obtained for each behavioral category, and the percentages of opportunities
for which Privates failed to implement the appropriate behavior were calculated for any measures
rated on a dichotomous (e.g., yes/no) scale (see Table 5). This latter analysis was conducted
specifically for the behavioral categories of Safety and Marksmanship Fundamentals, which were
assessed multiple times across the various observation groups (N in Table 5 represents the
number of observations).
166

Table 5. Measures of Privates by behavioral category.
Measures of Private Performance
% Failed to
implement
Mean
SE
269
162
2.41
0.08
Steady Position
--
--
54.64%
Breathing
162
--
--
57.47%
Trigger Squeeze
162
--
--
79.67%
Follow-through
162
--
--
91.55%
155
3.42
0.10
Treat Weapon as if it's loaded
155
--
--
29.31%
Finger off trigger
155
--
--
54.17%
Weapon Orientation
155
--
--
42.59%
Target Awareness
155
--
--
8.33%
Correct Malfunctions
136
2.79
0.13
89
2.29
0.09
Apply/Maintain Fundamentals
Adherence to Safety Rules
SPORTS
19
3.00
0.38
349
3.04
0.06
Effectively Shoot from all positions
75
2.78
0.09
Prone
48
3.68
0.12
Kneeling
41
3.43
0.15
Choice of Position
25
3.04
0.10
46
3.27
0.11
Remedial Correction
Positions
Movement
Note: SE represents the standard error of scores across all Private observations.
The resulting composite scores provide an understanding of performance of Privates on specific
marksmanship behavior across behavioral categories. For instance, the composite score for Safety
gives us a view of how well Privates are following the four safety rules. This score of 3.42
indicates that Privates are generally learning safety techniques to a degree that is better than
merely acceptable, but the percentage of missed instances suggests that these Privates are
commonly forgetting to engage in each of the individual safety behavior. One frequent
observation involved Privates who attempted SPORTS in response to weapon malfunctions.
SPORTS is a process to clear a rifle malfunction by slapping upward on the magazine, pulling the
charging handle back, observing the obstruction, releasing charging handle, tapping forward
assist; and squeezing the trigger. Privates often skipped steps or failed to return the rifle to
working order quickly. In addition, Privates often fall short with fundamentals. Over 50% of the
Privates did not adequately apply the fundamentals, and many Privates failed to implement
multiple fundamentals.
While safe weapons handling and acceptable marksmanship scores reflect the standards-based
criteria typically evaluated in rifle marksmanship, OBTE is intended to promote additional
growth in areas such as problem solving, deliberate thought, intangibles, and understanding and
awareness (Darwin, 2008b; Cornell-dEchert, 2009b). There are many opportunities in rifle
marksmanship for Privates to develop and demonstrate the behavior consistent with OBTE.
Evidence from these observations suggests that the CATC-trained DS helped Privates exploit
these opportunities. For instance, developmental outcomes are reflected in the measures for
Privates that address peer coaching (see Table 6). While these measures do not explicitly identify
whether a Private demonstrated problem solving, deliberate thought or intangibles, the behavioral
167
Horn et al.
anchors do reflect these priorities (Dean et al., 2009). Privates performing at average levels (i.e., a
score of three on the five-point scale below) are potentially demonstrating some desirable OBTE
characteristics such as problem solving, judgment and initiative, but they are not doing so
consistently or regularly. There is room for further improvement. The measures provide clarity
about specifically where improvements can be made, and they suggest how to achieve such
improvement.
Table 6: Measures of Privates demonstrating OBTE developmental characteristics
Measures & Anchors for Private OBTE Behavior
Item: Does Soldier demonstrate knowledge of interruption of cycles of operation?
(5): Terminates SPORTS when malfunction is resolved; successfully corrects malfunction
(3): Performs SPORTS twice then initiates remedial action with limited success
(1): Repeatedly performs SPORTS but does not recognize the need for remedial action
Item: If the rifle is not zeroed, does Soldier differentiate human vs. mechanical error?
(5): Properly distinguishes between human and mechanical error
(3): Attempts to diagnose problem as human or mechanical error
(1): Adjusts point of aim rather than sights
Item: Does Soldier choose the best firing position regarding stability/cover/speed?
(5): Chooses an optimal firing position for the situation
(3): Chooses an acceptable firing position for the situation
(1): Chooses an inappropriate firing position for the situation
Item: Does Soldier shoot the group in sequence based on scenario instructions?
(5): Engages targets quickly in correct sequence with correct number of shots
(3): Engages targets in correct sequence but misses occasionally
(1): Engages targets out of sequence, hits wrong targets, may use too many rounds
Item: Does the Soldier perform a rapid magazine change?
(5): Maintains SA, tracks target movement, makes smooth reloads
(3): Loses SA looking for next magazine, misses target movement
(1): Forgets multiple steps; loses SA removing magazine; and reloads slowly
Item: Does the Soldier utilize barriers?
(5): Consistently shields body using barrier; uses barrier as support
(3): May not use best position for each barrier; does not use barrier as support all the time
(1): Exposes too much body; will not use barrier for support
Item: Does the peer coach provide feedback on zeroing the rifle?
(5): Offers detailed feedback specific to shooters fundamentals and rifle sight settings
(3): Offers some useful feedback
(1): Misdiagnoses errors; cannot offer useful feedback
Item: Does the peer coach provide effective feedback?
(5): Provides quality feedback on shots to correct errors; DS direction not needed
(3): Provides quality feedback on shots, requiring direction from DS
(1): Provides too much feedback on each shot; DS intervention needed
Mean
SE
19
3.00
0.38
3.50
0.50
25
3.04
0.10
10
2.30
0.25
54
2.60
0.14
42
3.39
0.52
17
3.50
0.27
3.50
0.39
Note: Values in parentheses represent the score associated with each behavioral anchor.
8.2.2.2 Analysis of the extremes. Similar to the DS measures, the highest-rated and lowest-rated
behavior of Privates was identified in order to illustrate the relative ease or difficulty of
instruction with respect to the intent of OBTE (see Table 7). Overall, the five highest-rated
measures suggest that the Privates learned some key skills with the potential for downstream
operational impact while behaving safely in the training environment. The vastly important
implication of this finding is that operational relevance with a potential for operational impact of
training does not require a tradeoff with safety (Darwin, 2008b; Cornell-dEchert, 2009b).
The lowest-rated measures, however, demonstrate that the Privates did not grasp certain aspects
of marksmanship fundamentals. Additionally, these data suggest a decrease in performance of
Privates as complexity and stress were added to the activity. The three lowest-rated types of
behavior in Table 5 each represent an activity that combines thinking and motor behavior to
168
complete the task, a level of complexity that seemed to give Privates some difficulty. One
possible explanation for this result is that the increased complexity and stress was introduced
before Privates had learned the fundamentals. Specifically, with regard to shooting while
walking, Privates are required to apply proper fundamentals while continuing to move. Other
activities allow for movement followed by shooting. This combination of activities may have
surpassed the capabilities of the Privates at the time the exercise was introduced into training and
consequently may explain the difficulty experienced by Privates in managing increasing stressors.
Note that these measures are also consistent with observations of DS indicating that their
escalation of stress was not tailored to learner state and may have happened prior to sufficient
learning of more basic skills.
Table 7. Highest-rated and lowest-rated Private behavior.
Top 5 Private Measures
In Field Fire 2 (when presented with two timed targets) does the Private
engage the closest threat first?
Mean
SE
9
28
24
13
6
4.56
0.10
0.19
0.20
0.20
0.24
Bottom 5 Private Measures
Mean
SE
Does the Private maintain fundamentals throughout the engagement?
63
31
12
2.79
2.58
0.12
0.17
0.10
10
2.30
0.25
11
2.06
0.37
Does the Private successfully get the rifle back into the fight?
Does the Private adhere to the four safety rules?
Does the Private shoot from a stable prone unsupported position?
Does the Private walk, turn, and shoot effectively?
Does the Private fire a group to standard (i.e., 4cm radius)?

Does the Private effectively shoot while walking?
Does the private shoot the group in the proper sequence according to
scenario instructions (e.g., shape color number sequence)?
Does the private properly apply SPORTS when faced with a
malfunction?
4.21
3.92
3.92
3.92
2.71
It should be noted that, with respect to the proper application of SPORTS when faced with a
malfunction (the lowest-rated Private behavior), an additional measure addressed whether the
Private successfully got the rifle back into the fight. This measure was the second-highest
measure (4.214), which is a noteworthy disparity from the ineffective display of SPORTS by
Privates (2.060). Such a finding indicates that although Privates eventually resolved the
malfunction, the Privates did not necessarily properly perform SPORTS, and they would not have
gotten the rifle back into the fight efficiently or effectively. SPORTS is a series of actions
designed to return an inoperable rifle to functionality quickly. It is trained to a standard of five
seconds. Five seconds reflects an urgency to return to lethality and protect the unarmed
combatant. Employing an incomplete knowledge of the rifle to resolve a malfunction in a time
period greater than five seconds puts the unarmed combatant at greater risk of being exposed to
the enemy. Therefore, even though Privates did successfully resolve malfunctions, they seldom
managed to complete this task to standard.
8.2.3 Patterns of Communication
As noted previously, the communication between the instructor and student that clarifies the
purpose of training is central to the effectiveness of the OBTE training approach. An analysis of
the communications data uncovered a limited degree of communication occurring in the observed
training exercises. More specifically, DS seldom communicated with Privates during shooting
169
Horn et al.
sessions in BRM. The majority of statements produced by DS in BRM were categorized as

Constructive/Diagnostic or Directive (see Table 8). Similar types of communication by Peer
Coaches with trainees occurred in BRM. Although Peer Coaches provided feedback less often
than DS, the nature of the communication was similarly classified as being either
Constructive/Diagnostic or Directive. Very few observations of communication among the
Privates, as trainees instead of as Peer Coaches, occurred during BRM exercises.
Table 8. Frequency of communication in Basic Rifle Marksmanship exercises.
Communication Categories
DS-PVT
PVT-DS
DS-PC
PC-DS
PC-PVT
PVT-PC
PVT-PVT
Total
Constructive/Diagnostic
1.50
0.01
0.11
0.05
0.75
0.01
0.05
2.48
Performance
0.36
0.00
0.00
0.01
0.16
0.02
0.07
0.62
Guided Learning/Self-discovery
0.20
0.10
0.00
0.01
0.05
0.13
0.03
0.52
Directive
0.63
0.00
0.08
0.05
0.55
0.02
0.14
1.47
Non-verbal Correction
0.15
0.00
0.00
0.10
0.20
0.00
0.00
0.45
Acknowledgement
0.20
0.19
0.00
0.05
0.01
0.02
0.00
0.47
General Questions
Communication Across
Categories
0.00
0.00
0.00
0.01
0.01
0.01
0.00
0.03
3.04
0.30
0.19
0.28
1.73
0.21
0.29
Note: DS = DS; PVT = Private (Trainee); PC = Peer Coach

These communication results from BRM indicate that when DS and Peer Coaches did
communicate with the Privates, typically during a shooting task, the feedback provided was
typically either a unidirectional statement intended to impart knowledge for improvement, or
unidirectional orders to take action with no explanation for how the required action is beneficial
to the shooter. In addition, the relative lack of Guided Learning/Self-Discovery clearly illustrates
that DS are not thoroughly engaging Privates in BRM in such a way that promotes understanding
and awarenesskey components of OBTE (e.g., asking questions to the Privates). This finding is
further supported by the absence of inquiry by Privates.
In contrast, the communication results between DS and Privates in ARM illustrated a greater
frequency of interactions consistent with OBTE. Specifically, DS had more instances of
communication and feedback with their Privates. Increased frequencies of communications were
obtained for Guided Learning/Self-Discovery and Directive and, to a lesser extent,
Constructive/Diagnostic and Acknowledgements (see Table 9). The increase in communication
consistent with OBTE in ARM as compared to BRM is likely due to the interactive nature of
ARM exercises, which contain greater complexity and typically require more detailed
explanations by the DS instructors in order to teach the Private as shooters.
During ARM, DS showed improvement in the quality, depth and breadth of the feedback given to
their Privates. However, it should be noted that frequency of communications between DS and
Privates did not adequately reflect the amount of communication necessary to convey the purpose
of training. While Peer Coaches offer the potential to provide timely feedback to the shooter,
these individuals were either underutilized or not permitted to converse with the shooter. The
trend indicated by these analyses suggests a potential misuse or non-use of the Peer Coach
position. Future application of OBTE to rifle marksmanship may well serve to promote the use of
guided learning, self-discovery, and other developmental communications from both the DS and
the Peer Coach (Cornell-dEchert, 2009b).
170

Table 9. Frequency of communication in Advanced Rifle Marksmanship exercises.
Communication Categories
DS-PVT
PVT-DS
DS-PC
PC-DS
PC-PVT
PVT-PC
PVT-PVT
Total
Constructive/Diagnostic
0.80
0.03
0.00
0.00
0.00
0.00
0.08
0.91
Performance
0.48
0.00
0.00
0.00
0.00
0.00
0.00
0.48
Guided Learning/Self-discovery
3.01
0.36
0.00
0.00
0.00
0.00
0.02
3.39
Directive
3.02
0.10
0.00
0.00
0.00
0.00
0.34
3.46
Non-verbal Correction
0.42
0.01
0.07
0.00
0.00
0.00
0.00
0.5
Acknowledgement
0.96
0.52
0.00
0.00
0.00
0.00
0.00
1.48
General Questions
Communication Across
Categories
0.05
0.09
0.00
0.00
0.00
0.00
0.00
0.14
8.74
1.11
0.07
0.00
0.00
0.00
0.44
Note: DS = DS; PVT = Private (Trainee); PC = Peer Coach

8.2.4 Potential Influence of Instructor Behavior on Performance of Privates
Based on the ratings of DS and Privates, correlations were conducted to assess whether highly
rated DS (i.e., DS who were using an OBTE approach) were associated with Privates who were
also rated highly. Although conclusive causal inferences are not possible, a significant positive
correlation indicates that an approach to training consistent with OBTE is associated with higherperforming Privates. To conduct this analysis, mean scores of DS were compared with the mean
scores of the Privates for whom they were responsible during either ARM or BRM.
Figure 5. Correlation between DS and Private behavior.

The results of this analysis reveal a significant positive relationship (r = 0.32, p < 0.05, N = 40 DS
for whom data on corresponding Privates were obtained; see Figure 5) between highly rated DS
and highly rated Privates. The significant correlation suggests that DS whose behavior exhibited
171
Horn et al.
greater application of OBTE were more successful as instructors. This correlation does not
indicate causation, and further empirical evidence is needed to understand this relationship. The
practical implication is that any intervention resulting in enhanced instructor behavior consistent
with OBTE may result in better training and thus may improve performance in Privates.
8.3 Implications for Service System Development
8.3.1 Verification of OBTE
In the present context, systems engineering verification (Chrissis, Konrad, & Shrum, 2003;
CMMI Product Team, 2009), addresses whether and to what extent an organization can verify
that OBTE actually is being implemented in instruction. Methods of verification are necessary to
ensure continual adaptation and process improvement. Any capability that is designed and
developed from a life-cycle perspective will undergo product or process improvement as
experience is gained and adjustments are made either because of defects in the capability or
because of changes in the environment of use (Chairman of the Joint Chiefs of Staff, 2004; Sage
& Rouse, 1999). The exigency for bringing science and engineering to OBTE is to ensure that the
service system includes practices for verifiability over the life cycle. A rigorous approach to
verification begins with the definition and measurement of OBTE (Chapter 2). This foundation
enables scientific investigation of any implementation of OBTE. The theoretical foundations and
replicability inherent in a scientific investigation reveals specific activities that can be used and
adapted by others to verify the capability. This level of maturity differentiates CMMI and ISO
9001:2000 from earlier or less advanced instantiations of systems engineering (see also,
Mutafelija & Stromberg, 2003; Swain, 2005).
The findings for our observation of DS verify that instructor behavior is, in fact, observable in
both the nature and extent to which it reflects principles, strategies and best practices of OBTE. In
other words, the implementation of OBTE is verifiable using our definition and measurement of
OBTE (Chapter 2) and replicable methods of data collection and analysis such as those employed
in the current investigation. This does not preclude further development of the measures or in the
methods of data collection and analysis (Chrissis, Konrad, & Shrum, 2003; CMMI Product Team,
2009; Sage & Rouse, 1999); in fact, such developments are desirable given the early stage of the
life cycle for the capabilities supporting OBTE. An important milestone has been achieved in
OBTE in that this instructional service system now has the capacity for development, including
improvement or adaptation, in response to lessons that will be learned in its employment and
downstream as an outcome of its employment.
One of the most salient areas for further process improvement is in the measures of student
behavior. The measures developed for the current investigation are limited in their use for
verification of OBTE. The reason is that these measures reflected the level of understanding of
OBTE by DS who participated in development of the measures. In fact, the measures developed
with DS suggest gaps in their understanding of OBTE (Dean et al., 2009). Irrespective of data
collected using the measures, the measures themselves can inform further development of OBTE
as a service system. As the measures of student behavior improve to reflect the intent of OBTE,
they can become more useful in educating instructors about OBTE. On our view, progress toward
holistic formative assessments should involve the development of student measures that, like the
instructor measures, have broader applicability beyond marksmanship.
Along with the instructor measures, improved student measures will provide a more holistic
formative assessment that can be used by instructors to plan, execute, and reflect on their own
behavior and performance in their job. This addresses level 3 of program evaluation (Kirkpatrick,
172
1994) applied to instructors (Bransford et al., 2000). Verification thus has implications for
instructor training and education that improves the instructional service system (CMMI Product
Team, 2009, p. 460).
Verification such as represented in data collected with measures of instructor and student
behavior is a good method to address Kirkpatricks level 2 of program evaluation (Kirkpatrick,
1994) that provides actionable feedback about learning. Level 2 evaluation corresponds to the
considerations of an assessment-centered environment because this is the most timely and
actionable source of feedback for improvement of an instructional program. We believe that level
2 evaluation can be improved by considering the confluence of assessment-centered and
community-centered aspects of a learning environment, as emphasized by Bransford et al. (2000)
(see Chapter 3). In the context of OBTE, a community-centric orientation on assessment-centric
issues is provided by the further development of measures for student and instructor behavior that
can become part of the learning environment, measures that can be used during learning, for
learning. An example is the use of formative measures in decisions about when and how to
engage in collaborative reflection. Such measure are valid to the extent they are expressed in
terms of values and best practices of leaders with extensive and recent combat experience. Thus,
they also help develop skills more likely to transfer, because of their direct and concrete
relevance, to the operational environment and reinforce values that are of great consequence in
Full Spectrum Operations. This also helps address Kirkpatricks level 4 of program evaluation
that attempts to bridge the gap between impact on the individual and relevance to the collective
mission and objectives.
8.3.2 Validation of OBTE
In the present context, systems engineering validation (Chrissis, Konrad, & Shrum, 2003; CMMI
Product Team, 2009) addresses whether and to what extent OBTE fulfills its intended use. While
distinctly different considerations, verification and validation are always tightly interrelated
practices. One reason for this is that overlapping methodologies can be used for verification and
validation. Another is that the results of verification often have implications for validation, and
vice versa. For example, the measures of instructor behavior that reflect OBTE (Chapter 2),
together with methods of data collection and analysis reported herein, enable an organization to
verify implementation of OBTE. In addition, the clear statements of intent and purpose of OBTE
(Cornell-dEchert, 2009a,b), as well as extensive collaborative reflection with progenitors and
practitioners of OBTE (Chapter 2), led to a set of measures that have face validity to instructors in
general. In this sense, the OBTE measures of instructor behavior may demystify the concept of
good training. In other words, definition and measurement of OBTE in terms of instructor
behavior and instructor-student interactions suggest measures that may be applicable to any
training to determine its adequacy and potential for improvement.
As with verification, validation is a matter of degree. The capability to validate is the key
(Chrissis, Konrad, & Shrum, 2003; Mutafelija & Stromberg, 2003; Sage & Rouse, 1999).
Correlations between instructor behavior and student behavior are one method for validation that
emerges from our investigation. With the development of better measures of student behavior,
such correlations will become correspondingly more meaningful. The correlation results show
considerable room for improvement in the provision of good training in IET as defined by OBTE.
This is an opportunity in the context of a mature capability that includes rigorous processes for
verification and validation. The description of training capability gaps is rich with cause-effect
relationships; that is, it is actionable. There are implications in the data, for example, that IET is
not developing basic combat skills to the extent that it could, but measures derived from OBTE
173
Horn et al.
reveal ways to address these shortcomings. These implications will be discussed after further data
on validity of OBTE from the post-deployment survey are examined.
8.4 References
Chairman of the Joint Chiefs of Staff (2004). Joint capabilities integration and development
system. Retrieved November, 2004, from Defense Technical Information Center,
http://www.dtic.mil/cjcs_directives/cdata/unlimit/3170_01.pdf.
Group.
Group.
Deci. E.L. & Ryan, R.M. (2008). Self determination theory: A macrotheory of human motivation,
Publishers, Inc.
Mutafelija, B., & Stromberg, H. (2003). Systematic process improvement using ISO 9001:2000
and CMMI. Boston, MA: Artech House.
Sage, A.P., & Rouse, W.B. (Eds.) (1999). Handbook of systems engineering and management.
New York, NY: Wiley.
Barracks, PA: U.S. Army War Col
Impact on Rifle Marksmanship Training
174
Chapter 9. Impact on Rifle Marksmanship Training

Courtney Dean, Tyrone Jefferson, Zachary Horn,
Eric Jones, Brandon Beltz, Matt Puglisi, Fred Diedrich
Aptima, Inc.
The purpose of this investigation was to collect evidence of change following the intent at Fort
Sill to implement the approach of the Asymmetric Warfare Group (AWG) to Outcomes Based
Training and Education (OBTE) (see Chapters 1, 3). We conducted two related data collection
efforts, each conducted before and after instructors participated in a one-week train-the-trainer
course in which OBTE was demonstrated in the context of marksmanship (see, e.g., Chapter 1).
The first effort involved collecting data with measures of observable behavior to assess the extent
to which instructors exhibit behavior consistent with the principles and practices of OBTE and to
collect data that reveal whether those behaviors positively affect trainee performance. The second
effort involved surveying personnel exposed to OBTE in units at all levels of the organization
(e.g., from Commander to Drill Sergeant) to gain an understanding of their familiarity with and
attitudes toward OBTE. Based on this field-based investigation, we addressed the following
questions:
Are there differences in the way Drill Sergeants (DS) instruct Privates in Basic Rifle
Marksmanship (BRM) following exposure of DS to OBTE?
Are there differences in the behavior of Soldiers following exposure of their DS to
OBTE?
Are there changes in attitudes toward and understanding of OBTE following exposure to
OBTE?
9.1 Behavioral Data Collection During Basic Rifle Marksmanship
9.1.1 Method
9.1.1.1 Participants. Participants for this study included Drill Sergeants (DS) and Privates from
units within the 434th Field Artillery Brigade at Fort Sill, Oklahoma, who were conducting
training in Basic Rifle Marksmanship (BRM). Two main observations were conducted, one in
October of 2008 (prior to familiarization with OBTE) and one in April-May of 2009 (after
familiarization with OBTE). In total, five Batteries were observed. Observations were made of
122 participants (26 DS and 96 Privates, randomly selected). In particular, two Batteries (15 DS
and 55 Privates) conducting BRM training were observed in 2008, and two different Batteries (11
DS and 41 Privates) conducting BRM training were observed in 2009. One additional Battery
was observed conducting ARM, but these data are not reported here. The Soldiers in the units
observed were associated with a variety of specialties (Artillery, Combat Services, and Combat
Services Support) and components (Active Duty, National Guard, and Reserves).
The DS observed in this effort drawn in the 434th Brigade. Surveys of the DS indicated that they
were exposed to OBTE in several ways (See Section 7.2). Of the 91 respondents to a survey
conducted during the investigation in 2009, 71 participated in the AWGs Combat Applications
Training Course (CATC) (see Chapter 1), 31 participated in AWG workshops, 40 participated in
AWG briefings, 38 participated in non-AWG briefings on OBTE, 53 were exposed to training in
OBTE at other locations (e.g., DS School at Fort Jackson, South Carolina), and 42 were exposed
to other applications of OBTE beyond BRM. In addition, Brigade and Battalion Commanders
175
Dean et al.
participated in a briefing by the research team in January of 2009 about lessons learned from the
data collected in October 2008 (i.e., data collected prior to the events listed above).
9.1.2.1 Procedure. The initial data collection occurred in October 2008. At that time, the DS we
observed had no confirmed exposure to CATC. The second phase of data collection occurred in
April and May 2009. At that time, with one exception, all observed DS had completed CATC.
The investigation was conducted in BRM exercises two through four, six, nine, and ten, which
were focused on grouping, zeroing, location of miss and hit (LOMAH), and automated field fire
pop-up target shooting. During the period between the two data collections, OBTE was
implemented across the 434th Brigade using multiple methods. It was not possible to have control
conditions in which no changes were made. Therefore, while changes in behavior between the fall
and spring may be broadly associated with OBTE, it is not possible to isolate the effects of
particular elements of the intervention because it was not possible to fully control for various
influences.
In addition, we provided formative feedback in the form of actionable recommendations to units
at Fort Sill in order to guide adoption of OBTE. In other words, we did not withhold the October
2008 results from leaders in the 434th Brigade. While this aspect of the design influences the
interpretation of results, it was necessary for the following reasons. Assessment with formative
measures, and the continuous process improvement that they enable, has become central to OBTE
(see Chapters 1, 11, and Epilogue). Also consistent with OBTE, these commitments should be
fully decentralized as well as part of the command climate. Engaging in continuous data-driven
improvement is an essential part of what it means to implement OBTE. The measures that have
become formalized in the definition and institutionalization of OBTE are the same measures used
to collect data on instruction in the present investigation (see, e.g., Chapter 2). This is critical
insofar as one cannot assess the impact of OBTE if one cannot verify that OBTE is actually being
implemented and to what extent (Chapter 1, 2).
For the current study, we utilized a methodology similar to that used in prior investigations at
Fort Jackson and Fort Benning (see Chapter 6). All observations were conducted in three-person
observer teams in order to capture the dynamics between an instructor (DS), the student being
trained (a Private), and a peer coach when applicable (i.e., other Privates who help the shooter by
commenting on technique and shot results). CATC instructors (cadre) and OBTE designers from
the AWG captured data on instructor behavior. Members of the research team who had
participated as students in CATC captured data on student behavior because they were able to
recognize and evaluate behavior associated with rifle marksmanship. A member of the research
team who had not taken CATC captured data on communication between the DS instructor,
Privates as shooters, and Privates as peer coaches. Observers were not blind to condition (pre- vs.
post-implementation of OBTE).
Observations in October 2008were made over the course of five days and approximately 40 hours
of observation. Observations in April-May 2009 were made over the course of seven days and
approximately 56 hours of observation. Each observation (i.e., interaction between the DS and
Private, as well as peer coach when applicable) lasted approximately 60 minutes depending on
the time spent by Privates on the firing line.
9.1.2 Assessment
9.1.2.1 DS behavior. Measures of DS instructional behavior were developed with AWG
personnel, OBTE designers, and CATC Cadre to identify observable behaviors associated with
the instructional approach (see Chapter 2). The behavior observed and rated in this study reflects
176
expectations about DS who exemplify OBTE in the way they behave as instructors and role
models (cf., Chapter 8). The measures employed in this assessment consisted of approximately 45
Likert scale, yes/no, and checkbox items to assess the execution of training. These observer-based
measures of DS instructor behavior have been used in a series of data collection efforts for the
AWG with an inter-rater agreement of 94%, in which agreement is defined as identical responses
on yes/no items or scale ratings within a single scale value.
9.1.2.2 Behavior and Performance of Privates. Measures for the behavior of Privates were
developed with input from DS who participated in CATC (Chapter 2). The intent was to capture
the behavior that Privates are expected to learn in rifle marksmanship. The measures consisted of
approximately 115 items that covered all stages of basic rifle marksmanship. The measures
reflected the DS perspectives on rifle marksmanship training in light of what they learned in
CATC, and addressed elements such as marksmanship fundamentals and safety. Several
additional measures were developed to assess Privates demonstration of behavior consistent with
OBTE. The measures assessed whether or not, or to what degree, Soldiers were engaging in the
types of behavior representative of OBTE. Behavioral anchors support each measure at the 1, 3,
and 5 levels of a five-point scale. Supporting narrative information is also provided to help the
observer recognize the expected behaviors. An example measure is featured below in Figure 1.
Twenty measures assessing characteristics such as problem solving, confidence, judgment,
accountability and deliberate thought were developed with AWG personnel to identify relevant
and observable behaviors. This additional set of measures was also used during the assessment
process.
Does the Private demonstrate problem solving skills when facing a unique situation?
Private does not

demonstrate
recognition of the
problem; no change in
behavior
For example, the
Soldier does not grasp
the tactical application
of SPORTS or
applying a tourniquet,
and focuses on
memorizing the task to
pass evaluation
Private recognizes
problem; applies training
to the problem
For example, the Soldier
grasps the tactical
application of SPORTS
or applying a tourniquet,
but remains focused on
memorizing the task as a
generic, scripted solution
Private recognizes
problem; properly
applies training to the
problem with positive
results
For example, the
Soldier grasps the
tactical application of
SPORTS or applying a
tourniquet, and adapts
their understanding of
the task to meet the
challenge of a new
situation
Figure 1. Sample measure from OBTE Principle Measures list.

The measures for Privates consisted of Likert-scale, yes/no, and checkbox items. These observerbased measures of Private behavior (Dean et al., 2009) have been used in a series of data
collection efforts for the AWG with an inter-rater agreement of 77%, in which agreement is
defined as identical responses on yes/no items or scale ratings within a single scale value.
177
Dean et al.
9.1.2.3 Communication. Measures were developed to conduct a quantitative assessment of

communication between DS, Privates, and (where possible) Peer Coaches in BRM. Observations
were categorized according to a predetermined set of criteria that reflect variations in patterns and
styles of communication that are considered to be important in OBTE.
Frequency of communication was observed for the following seven types of communication
patterns: Constructive/Diagnostic, Performance, Guided learning/Self-discovery, Directive, Nonverbal Correction, Acknowledgment, and Non-task related questions. The categories of
Constructive/Diagnostic (e.g., feedback about behavior) and Guided learning/Self-discovery (e.g.,
asking questions) were derived through a qualitative examination of the types of communications
that were heavily emphasized in OBTE (Cornell-dEchert, 2009a,b; see also Chapters 2 and 3).
The Performance (e.g., miss/hit), Directive (e.g., stand there), Non-verbal Correction (e.g.,
reposition of Soldier), Acknowledgment (e.g., yes Drill Sergeant), and Non-task related
categories (e.g., off topic conversation) were included to capture the range of utterances typically
observed. This type of categorization technique was initially developed and used in previous
research involving adaptive communications analysis (e.g., Entin, Diedrich, & Rubineau, 2003).
These observer-based measures of communication have been used in a series of data collection
efforts for the AWG with an inter-rater agreement of 93%, in which agreement is obtained when
each rater indicates the same degree of communication frequency (i.e., high, medium, and low)
within a single communication category per observation.
9.1.3 Results An Overview
Results fall into three major categories, corresponding to the analyses described above: behavior
of DS during BRM, behavior and performance of Privates during BRM, and communication
during BRM. Before presenting the detailed results from these analyses, we summarize the major
findings concerning the effects of OBTE on the BRM training.
Overall mean comparisons were conducted between participant scores from 2008 and 2009 to
identify any changes in the behavior of DS or Privates. Behavior for Privates were further
divided into marksmanship behavior, and behavior consistent with OBTE principles.
Comparisons were made using independent one-tailed t-tests and taking into account unequal
sample sizes. In each case, ratings on Likert scales ranged from 1 (DS behaviors not in
accordance with OBTE, or Private behaviors below expectations) to 5 (DS behavior consistent
with OBTE, or Private behaviors above expectations). We observed significant improvements in
performance across all three sets of measurements (Figure 2).
More specifically, DS demonstration of instructional behaviors consistent with OBTE increased
from fall 2008 to spring 2009. Mean scores for each DS were calculated across all behavior that
could be observed and measured. Using a one-tailed t-test and taking into account the unequal
sample sizes, the means for Drill Sergeants were determined to be significantly different (t(24) =
5.435, p < 0.0001) from fall to spring (Levenes test of equality of error variance was used to
verify homogeneity of variance). As can be seen in Figure 2, the change in scores shows
substantial improvement in behavior with respect to OBTE. Overall, the difference in means
suggests a substantial improvement in the instruction, but the scores do not yet reach moderate
levels (e.g., 3.00) signifying that additional improvement is still possible.
178
Figure 2: Overall mean comparisons between 2008 and 2009 data collection
efforts. Error bars represent the standard error.
For Privates, scores for both the BRM Performance (marksmanship) and the OBTE Principles
(e.g., problem-solving, confidence, accountability, etc.) also increased from fall to spring. We
calculated mean scores for each Private across all behavior that could be measured for each
category. Independent t-test of unequal variance comparisons yielded significant differences for
both sets of measures (BRM Performance scores, t(94) = 3.790, p < 0.0001. Levenes test for
equality of variance was significant (0.013) for the BRM Performance scores; thus, the t and p
values reported for BRM Performance Scores reflect correction for unequal variance.
Demonstration of OBTE Principles likewise yielded significant differences, t(53) = 2.589, p <
0.05), variances were not significantly different. As for the DS, the difference in means suggests
an improvement, but the scores remain close to moderate levels (e.g., 3.00), signifying that there
is additional room for improvement. Overall, however, these data indicate that in the spring, as
compared to the fall, DS behavior was more consistent OBTE, and these changes were
accompanied by positive changes in the behaviors of Privates. DS seem to have changed the way
they taught, and this was accompanied by positive changes in performance of Privates.
9.1.4 Evidence for Influence of OBTE
Based on the ratings of DS and Privates, correlations were conducted to assess whether highly
rated DS (i.e., DS who were using an OBTE approach) were associated with Privates who were
also rated highly. Although conclusive causal inferences are not possible, a significant positive
correlation indicates that an approach to training that is more consistent with OBTE is associated
with higher-performing Privates. To conduct this analysis, mean scores of DS were compared
with the mean scores of the Privates for whom they were responsible, combining data from
October 2008 and April-May 2009.
The results of this analysis revealed a significant, but relatively modest positive relationship (r =
0.462, p < 0.05, N = 26) between highly rated DS and highly rated Privates (DS for whom data on
corresponding Privates were obtained; see Figure 3). The significant correlation suggests that DS
whose behavior exhibited characteristics associated more strongly with OBTE were more
179
Dean et al.
successful as instructors, in general (Figure 3 includes data from both the spring and fall
collections). This correlation does not indicate causation. However, the practical implication is
that any intervention resulting in enhanced instructor behavior consistent with OBTE may result
in better training and thus may improve performance in Privates.
Figure 3: Relationship between DS Performance and Private Performance.

The correlation was computed between DS behavior consistent with OBTE behavior (i.e.,
average DS behavior across all DS-Private observations) and performance of Private (i.e., the
average performance across all observed Privates instructed by each DS). It shows a significant
positive relationship between DS behavior and observed performance of Privates (r = .46; p <
.01). However, caution should be taken when interpreting this result, the correlation does not
account for the considerable difference among Drill Sergeants with respect to the number of
observed DS-Private interactions (Mean = 3.72, SD = 3.17). To remove any ill effects due to the
variance in instructional observations, a more robust hierarchical regression was conducted (see
Table 1). This analysis determines whether DS behavior consistent with OBTE has a direct
influence on average performance scores of Privates after removing the influence of variance due
to number of observed DS-Private interactions. Results from this hierarchical regression ( = .46;
R2 = .21, p < .05) suggest that after controlling for the number of instructional observations, DS
engagement in OBTE behavior is responsible for a significant amount of variance (21%) in
performance of Privates. In other words, greater use of behavior consistent with OBTE is
associated with an improvement in performance of Privates.
180

Table 1. Hierarchical regression results for the influence of Drill Sergeant OBTE
Behaviors on Private Performance, controlling for Number of Privates Instructed by
each Drill Sergeant.
DV=Average Private
Performance per DS
B
S.E.
B
t
p
R2
F
(N=25)
Model 1:
.01
0.12
Number of observed
.02
.04
.07
0.35
.732
interactions
Model 2:
.21*
2.99+
Number of observed
.00
.04
.02
0.11
.914
interactions
DS OBTE Behaviors
.46
.19
.46
2.42
.024
+
* p = .02; p = .07; Number of observed interactions refers to the number of observed
Privates instructed by each Drill Sergeant.
9.1.5 Behavior of Drill Sergeants after Exposure to OBTE
Given our emphasis on instructional process improvement, the anchors of the five-point Likert
rating scales for DS behavior typically ranged from behavior associated with current training
methods (rating of 1) to behavior ideally associated with application of OBTE to training (rating
of 5). For this reason, it is important to note that any ratings above a score of 1 indicate
instructor behavior that reflects some elements consistent with OBTE. Overall ratings of DS as
instructors in rifle marksmanship suggest that these DS exhibited some aspects of OBTE in their
approach to training. However, the findings from these observations also indicate that DS did not
fully utilize OBTE in the rifle marksmanship training after being exposed to OBTE. In particular,
their mean score was 2.61 (N = 26, SE = 0.09, min = 1.91, median = 2.70, max = 3.40, range =
1.49) on a five-point rating scale. The results do not completely reveal why this is the case, but
they do suggest a high degree of variance in instructor behavior, room for improvement and,
more importantly, specific ways to improve instruction with respect to OBTE.
Table 2. Descriptive Statistics for the Presence of OBTE Instructional Behavior
OBTE Instructional Category
Customize instruction when possible based on constraints / conditions learnercentered
Creates a positive learning environment
Facilitates learning of concepts
Instructors utilize measures of effectiveness & self-evaluation
Hotwashes and Mini-AARs
Uses scenarios to facilitate learning
Communicate the parameters of learning; the "why"
Instructors exhibit intangibles in own actions
Training emphasizes broad combat or mission success
Mean
28
28
28
19
24
27
28
27
22
3.21
3.01
2.90
2.58
2.33
2.31
2.21
1.96
1.80
Note: n represents the number of independent observations of DS during the course of the data collection effort; SE
represents the standard error, pooled across all DS observations.
SE
0.17
0.18
0.19
0.18
0.21
0.18
0.16
0.17
0.19
181
Dean et al.
9.1.5.1 Patterns of instructor behavior. The DS behavioral measures were grouped by the
overarching instructional categories that were identified by working with OBTE designers in the
measure development process (Chapter 2). The average behavioral ratings of DS (Table 2)
indicated that they exhibited some behavior consistent with OBTE (i.e., average ratings were
greater than 1). Mean scores appearing at or above the midpoint (e.g., 3.0) in the Customize
instruction and Create a positive learning environment behavioral categories suggest that the
DS fostered a more learner-centered environment (Bransford, Brown, & Cocking, 2000). While
these values are substantially higher than the baseline value of 1.0, they leave room for additional
change. In addition, measures for Training emphasizes broad combat or mission success and
instructors exhibit intangibles in own actions indicate substantial room for change. In
particular, this suggests a need for DS to increasingly emphasize the relationship of training to
mission success and to have additional opportunities to demonstrate behaviors consistent with the
principles of OBTE (e.g., problem-solving, initiative).
9.1.5.2 Analysis of extremes. The five highest-rated and lowest-rated types of behavior from the
ungrouped data are listed in Table 3. The patterns of ratings indicate that DS demonstrated some
ability to act in the role of a mentor with their Privates and construct a more learner-centered
environment (e.g., Bransford et al., 2000). Instructors took steps to create a less authoritarian
environment than is typical during the first few weeks of basic training, and they attempted to
address individual learning needs. One manifestation of this strategy is that small mistakes are
corrected with developmental guidance rather than confrontation. The intent of this mentorship is
to transfer the ownership of the training to the students (Guay, Ratelle, & Chanal, 2008; Lave &
Wenger, 1991) and build students confidence as they learn to perform under stressstress
engendered by the task and conditions, not artificially generated by an intimidating instructor.
Table 3. DS Highest-Rated and Lowest-Rated Behavior
Top 5 DS Measures
Do the instructors successfully address individual training predicaments?
Does the instructor balance perception of power vs. leadership?
Do the instructors adapt coaching to individual Soldiers?
Does the instructor assume the role of helper in Soldiers pursuit of success?
Does the instructor recognize when a Soldier is too withdrawn/distracted to
effectively participate in training?
Bottom 5 DS Measures
Do the instructors ensure that Soldiers understand the combat/mission
application of a task?
Do the instructors ensure the Soldiers can articulate how to apply concepts to
new situations?
Do the instructors introduce the Soldiers to a broad range of stressors and
cultivate stress management?
Do the instructors ensure that the Soldiers recognize the effects of their actions
on their teams?
Do the instructors focus the why of training back to the relationship between
the individual and big picture/mission?
n
23
28
27
28
Mean
3.76
3.75
3.52
3.43
SE
0.12
0.13
0.14
0.17
12
3.33
0.19
Mean
19
1.71
0.21
23
1.54
0.12
17
1.50
0.18
16
1.47
0.14
19
1.45
0.13
Note: n again represents number of independent observations. Lower numbers of observations reflect particular DS
behavior that was at times not observable.
In contrast, the lowest-rated types of behavior indicate that DS did not always help the Privates
discover and understand the purpose of training and its intended combat application. Tasks are
SE
182
still largely taught in isolation with the Soldiers gaining little contextual understanding and sense
of greater mission application. Specifically, soldiers are not discovering how their actions have an
impact others in their unit. In addition, although the DS incorporated escalation of complexity and
stress into training, these data suggest that they did not utilize stress effectively; for example, they
were not helping Privates understand why stress is being added, how it affects them, and how to
manage it. When stress was applied by emphasizing throughput (movement to the next activity)
regardless of learner state, DS actions compromised Privates ability to learn the basics, and we
hypothesize that their confidence may have suffered. Failure to properly communicate the why
of training to students may have detrimental effects on motivation and sense-making (Bransford
et al, 2000; Deci & Ryan, 2008; Guay, Ratelle, & Chanal, 2008). Thus, these data suggest that
while DS did exhibit some behavior consistent with OBTE, it is important for DS to develop a
better understanding of the importance and potential role of communication and sense-making
with respect to developmental outcomes during rifle marksmanship training.
9.1.6 Behavior of Privates
In order to observe the performance of Privates in rifle marksmanship, we developed behavioral
anchors for the 5-point Likert rating scales with input from DS previously trained in CATC (Dean
et al., 2009). These anchors ranged from behavior failing to meet performance expectations
(rating of 1), average expectations for acceptable behavior (rating of 3), to behavior exceeding
expectations (rating of 5). Note that these anchors are different from those developed for Drill
Sergeants with respect to OBTE. While both are performance related, the BRM measures reflect
performance against a standard of competence consistent with Army marksmanship standards
(e.g., firing a group within a 4 cm circle). BRM performance ranges from failing to meet the
standard to exceeding the standard. Performance scores for Privates in BRM had a mean = 3.55
(N = 41, SE = 0.10, min = 1.87, median = 3.75, max = 4.58, range = 2.71) on a five-point rating
scale in April-May, after DS exposure to OBTE. This indicates that Privates are showing some
evidence of exceeding the standards for marksmanship performance. The results demonstrate
several areas for improvement that are discussed in the next section.
Additional measures were developed to assess Privates demonstration of behavior consistent
with OBTE. In April-May, after DS were exposed to OBTE, behavior of Privates that reflected
principles of OBTE had a mean = 3.11 (N=22, SE = 0.18, min = 1.31, median = 3.50, max = 4.17,
range = 2.86) on a five-point rating scale. While these results are slightly lower than the results
for BRM performance, they are still above the average rating (e.g., 3.00). The results of the
analysis of these measures help to illustrate the impact of the attempt to implement OBTE. The
opportunity to observe the desired behavior among Soldiers is often contingent upon the training
environment provided by the instructors. As instruction becomes more interactive and
communication becomes less directive, Soldiers are given greater opportunity to demonstrate
characteristics such as problem solving, initiative, accountability, and confidence.
Note that the difference in sample size between these two sets of measures reflects the
opportunities there were for observation. Assessment of marksmanship performance was easily
observed for all selected participants during the course of an evaluation. However, due to the
limited time in which to observe each participant, many participants were not rated with respect
to our additional measures. Hence, the number of observations was smaller for Privates behavior
consistent with principles of OBTE.
9.1.6.1 Opportunities for improvement. Measures relating to a specific behavioral category (e.g.,
safety, firing positions, and marksmanship fundamentals) were combined for further analysis.
Means were obtained for each behavioral category, and the percentages of opportunities during
183
Dean et al.
which Privates failed to demonstrate the appropriate behavior were calculated for all measures
rated on a dichotomous (e.g., yes/no) scale (see Table 4).
Table 4. Performance of Privates by Behavioral Category
Measures for Performance of Privates
n
Mean
Apply/Maintain Fundamentals
63
3.54
Sight Picture
Steady Position
Breathing
Trigger Pull
Adherence to Safety Rules
63
4.29
Treat Weapon as if it is loaded
Finger off trigger
Weapon Orientation
Target Awareness
Positions
Effectively Shoot from all positions
70
3.55
Prone Supported
54
3.85
Prone Unsupported
8
3.25
Kneeling
8
2.13
Note: SE represents the standard error of scores across all Private
observations, n indicates number of observations.
SE
0.16
%
Failed
to do
16.4%
29.1%
36.4%
56.4%
0.06
6.3%
3.2%
4.8%
1.6%
0.12
0.09
0.33
0.36
This analysis was conducted specifically for the behavioral categories of Safety and
Marksmanship Fundamentals, including three categories: (a) applying and maintaining these
fundamentals, (b) adhering to safety rules, and (c) performance in several firing positions. The
behavior was assessed multiple times across the various observation groups (n in Table 3
represents the number of observations). The resulting composite scores provide an understanding
of Private performance on specific marksmanship behavior across behavioral categories. For
instance, the composite score for Safety indicates how well the Privates followed the four
safety rules. This score of 4.15 indicates that, in general, Privates learned safety techniques to a
degree better than expected (ratings of 4 and 5 on the Likert scale were associated with exceeding
expectations); however, the percentage of missed instances for individual rules (e.g., treating the
weapon as if it is loaded) suggests that some Privates tended to forget to engage in some of the
individual safety behaviors. That being said, it is important to note that these percentages were
very similar to those observed, in October 2008, before DS were exposure to OBTE. This
suggests that safety has been maintained despite increased handling of weapons resulting from
the emphasis of OBTE in April-May 2009. Chi square analyses were run to compare instances of
safety violations before and after DS exposure to OBTE. Only one significant change was
observed (2(2) = 8.343 (p < .005). Privates observed in April-May were significantly less likely
(e.g., 19.8% in October vs. 3.2% in April-May) to violate the safety rule of marksmanship: keep
your finger off the trigger until you are prepared to fire. There were no significant differences
for the other three rules: treat the weapon as if its loaded weapon orientation, and target
awareness.
184
Referring to the other categories, the composite score for applying and maintaining the
fundamentals is slightly below acceptable (M = 2.68). When examining the specific behaviors,
trigger pull, followed by breathing, are the most neglected. Again, a comparison between pre
and post-intervention data collection was conducted using chi square. Only one fundamental,
Sight Picture and Alignment, changed (improved) significantly (2(2) = 5.27, p < .03). Of the 3
positions examined, the Privates were most effective in the prone supported position, followed by
prone unsupported, and then kneeling. The average of all the observations across all the
measures indicated slightly higher than acceptable performance, but not performance that exceeds
expectations.
9.1.6.2 Analysis of the extremes BRM measures. Similar to the DS measures, the highest-rated
and lowest-rated behavior of Privates was identified in order to illustrate the relative ease or
difficulty of instruction with respect to the intent of OBTE (see Table 5). Overall, the five
highest-rated measures suggest that the Privates were able to learn and perform basic
marksmanship skills, in addition to some key skills with the potential for downstream operational
impact while behaving safely in the training environment. The most important take-away
implication of this finding is that operational relevance with a potential for operational impact
does not require a tradeoff with safety (Cornell-dEchert, 2009a,b).
Table 5. Highest-Rated and Lowest-Rated BRM Performance Metrics
Top 5 Measures
In Field Fire 2 (when presented with two timed targets), does the Private
engage the closest threat first?
Does the Private adhere to the four safety rules?
Does the Private properly load the weapon?
Does the Private shoot for a stable prone supported position?
Does the Private properly apply the fundamentals for marksmanship?
Mean
7
63*
16
56
63
4.43
4.29
3.83
3.81
3.54
Bottom 5 Measures
Does the Peer Coach provide feedback on fundamentals?
Does the Private shoot from a stable kneeling position?
Does the Private adjust point of aim using feedback from the Peer Coach?
Does the Private effectively transition between targets?
Does the Private maintain fundamental when transitioning between targets?
N
4
8
13
8
8
Mean
1.75
2.13
2.25
2.38
2.38
Note: n again represents number of independent observations. Lower numbers of observations reflect particular
The lowest-rated measures, however, demonstrate several points for continuing development.
For instance, these measures denote that Privates did not fully grasp certain aspects of
marksmanship fundamentals. These data suggest a decrease in performance of Privates as
complexity and stress were added to the activity. Specifically, two of the lowest-rated types of
behavior in Table 6 each represent an activity that combines thinking and motor behavior to
complete the task, a level of complexity that seemed to give Privates some difficulty.
Specifically, with regard to transitioning between targets, Privates are required to apply proper
fundamentals while scanning their shooting lane and acquiring their next target. Other activities
allow for movement, including target acquisition, then followed immediately by shooting. The
combination of these activities (target acquisition and the application of fundamentals) may
SE
.057
.06
.29
.09
.38
SE
.48
.36
.33
.63
.50
185
Dean et al.
consequently explain the difficulty experienced by Privates in managing increasing stressors.

These data suggest that due to throughput considerations, Privates may be advancing before they
are ready to do so. This has a potential unintended consequence of lowering proficiency and
confidence.
Table 6. Highest-Rated and Lowest-Rated OBTE Principles
Top 5 Measures
Does the Private demonstrate the ability to overcome the psychological
challenge of killing an enemy human when legally required during combat
operations?
Is the Private able to integrate task and skills into future training (e.g.,
understands the context of the skill, task)?
Does the Private demonstrate the ability to perform under highly stressful
situations?
Do the Privates discuss their performance with each other?
Does the Private demonstrate initiative?
Bottom 5 Measures
Does the Private discuss how his/her actions can impact the rest of the team?
Do the Privates demonstrate an understanding of their own performance?
Do the privates work together to solve the problem?
Does the Private demonstrate problem solving skills when facing a unique
situation?
Does the Private demonstrate confidence based on valid skill level?
Mean
SE
17
4.71
.11
16
3.66
.16
10
3.55
.25
28
24
3.38
3.25
.21
.25
N
7
11
28
Mean
1.29
2.45
2.68
SE
.29
.31
.17
28
2.73
.21
25
2.88
.19
Note: n again represents number of independent observations. Lower numbers of observations reflect particular
In addition, these measures indicate that Privates need further guidance on the importance of
staying engaged and providing feedback to others when serving in the role of Peer Coach. While
it is not critical that they have all of the answers (e.g., they may not fully understand
fundamentals), it is important that they stay engaged as active collaborators in solving problems
with their peers. DS can and must provide corrections as needed. Peer coaches need to remain
engaged and may need additional direction regarding what to coach and how to coach. This in
turn will benefit them as shooters. Likewise, it may be that Peer Coaches are not in the proper
observational position to assess the practice of fundamentals by their peers. It is important that
Peer Coaches are able to see and hear feedback from DS at the targets as well as watching the
shooter while shooting.
9.1.6.3. Analysis of the extremes measures of OBTE principles. The highest-rated and lowestrated behavior reflecting OBTE principles were identified (see Table 6). Overall, the five
highest-rated measures suggest that the Soldiers are developing the capabilities to engage enemy
combatants and are integrating new lessons into their existing knowledge. This shows promise for
Soldier mastery of complex skills and combat survival given the proper training environment.
The lowest-rated measures, however, indicate that Soldiers are still lacking full comprehension of
the greater mission context. The Soldiers are not recognizing their individual role within the
greater team or how their performance impacts the welfare of others. Further evidence is found in
186
the lack of teamwork in problem solving. The Soldiers rarely worked together to solve problems.
This is consistent with results observed above regarding the engagement and participation of peer
coaches. There is little mutual support being offered between the shooter and the peer coach at
times when skill development could substantially benefit from teamwork.
9.1.7 Patterns of Communication
In order to identify differences in the amount of communication observed, we conducted a series
of independent samples t-tests on the observed and categorized utterances. Given the multiple
tests on these types of data, a Bonferronis correction was employed to set alpha-level for
significance at 0.0027.
Communication between the instructor and student that clarifies the purpose of training is
important in OBTE. Accordingly, we investigated changes in communication patterns by
comparing observations before and after DS were exposed to OBTE. Specifically, DS rated in
April-May 2009 were generally found to communicate significant more often with the Privates
under their direction (M = 10.01; SE = 2.82) compared to the DS rated in October 2008 collection
(M = 4.87; SE = 0.49; t(24) = 2.142, p < .0027).
In addition to an increase in the overall quantity of communications, we uncovered several
qualitative differences for both DS and Privates, between the October 2008 and April-May 2009.
More specifically, we found that DS asked more Guided Learning/Self-Discovery questions in
after being exposed to OBTE (M = 2.20; SE = 0.46) compared to the DS observed earlier (M =
0.34; SE = 0.10). An independent samples t-test showed that these means were significantly
different (t(24) = 3.894, p < .0027). Data collected for the Constructive/Diagnostic and Directive
categories displayed differences as well with trends toward more communication after DS were
exposed to OBTE, but these differences were not statistically significant.
Communication with DS initiated by Privates, and for Peer Coaches in general, were also coded
and analyzed. None of these categories coded for the observed Privates were found to be
significantly different between the October 2008 and April-May 2009, although trends were
toward more communication after DS were familiarized with OBTE. However, there were two
noteworthy differences in communication between peers. Specifically, we found that Privates
spoke more often to their Peer Coaches in April-May 2009 (M = 5.47; SE = 1.09) compared to
October 2008 (M = 2.45; SE = 0.63), t(24) = 2.556, p < .0027*. Also, we found that Peer
Coaches spoke more often to their assigned peers in April-May (M = 11.34; SE = 3.72) compared
to October (M = 3.39; SE = 1.58), t(24) = 2.522, p < .0027*. None of the other categories
referring to Peer Coaches contained significant differences of note.
9.1.8 Summary
The findings on the behavior of DS and Privates in BRM indicate that behavior associated with
OBTE can be increased by familiarizing instructors with the principle and practices of OBTE
through a combination of participant observation in a train-the-trainer course supplemented with
briefings and seminars. These findings, thus, build on the findings from earlier investigations
demonstrating that implementation of OBTE can be verified and validated (see, e.g., Chapter 8).
As in previous investigations, however, several items indicated room for additional
improvements. In particular, the current findings DS can improve with respect to management of
increasing stress and throughput as well as communicating the relevance of current learning
events to future events and mission success.
187
Dean et al.
Collectively, the findings from the communications data are important in the assessment and
improvement of OBTE in BRM. DS asked Privates more questions, after being familiarized with
OBTE, thereby providing opportunities for Privates to engage problem-solving skills and develop
the habit of thinking deliberately. As DS improved in their communication with Private, Privates
also engaged their peer coaches more frequently. Peer collaboration on key instructional
challenges allows peer coaches to serve as a force-multiplier to help counteract large studentinstructor ratios.
9.2 Attitudes Toward an OBTE in Basic Training
We conducted a survey to evaluate attitudes and beliefs held by personnel across the 434th Field
Artillery Brigade regarding their understanding of OBTE and its utility.
9.2.1 Method
9.2.1.1 Participants. A survey regarding attitudes toward OBTE was completed on a volunteer
basis in Fall 2008 and Spring 2009 by many of the Officers and Non-Commissioned Officers
(NCOs) in the 434th Brigade at Fort Sill. The Fall 2008 survey received 131 total volunteer
respondents, including 81 Drill Sergeants, 40 Battery Command/Staff, and 10 Battalion or
Brigade Command and Staff. From this sample, 98 participants had not yet received training in
the Asymmetric Warfare Groups (AWG) Combat Application Training Course (CATC), and
thus serve as a pre-training sample. The survey data collection in Spring 2009 resulted in 91 total
volunteer respondents, including 70 Drill Sergeants, 20 Battery Command/Staff, and one
Battalion Staff member. From this sample, 71 participants had completed the CATC course, and
thus serve as a post-training sample.
9.2.1.2 Survey items. Surveys were administered online and were available for completion from
one to two weeks following dissemination. Responses to the surveys remained anonymous, as
direct identifiers (e.g., names) were not collected. The respondents in both samples were
administered identical sets of items that pertain to various perceptions of an OBTE approach to
training. A set of 28 items asked respondents to report their agreement with statements regarding
the importance of teaching marksmanship skills and OBTE principles in Basic and Advanced
Rifle Marksmanship. Items were worded such as: BRM [ARM] should teach privates to shoot,
and BRM [ARM] should teach privates to problem solve. A set of five open-response items
inquired about attitudes toward implementing an OBTE approach in Army training. Items
included the following: Do you think Basic Training puts Soldiers in situations that develop
their confidence, accountability, initiative, and judgment? Explain why. In addition, both
samples were asked nine demographic items including rank, Military Occupation Specialty
(MOS), current duty position, years in the military, duration and number of times of deployment,
and participation in the CATC course. One additional demographic item that inquired about
different sources of exposure to OBTE (e.g., AWG workshops) was present only in the spring
2009 survey.
9.2.2 Results
9.2.2.1 OBTE in rifle marksmanship training. We attempted to determine the extent to which
respondents believe that an OBTE approach to training should be adopted in BRM and ARM.
Respondents were asked to indicate their level of agreement as to whether general OBTE
principles (e.g., problem solving) and more typical Army-trained skills (i.e., moving, shooting,
thinking, and communicating) should be taught in BRM and ARM. A series of between-group
comparisons on average scores (independent samples t-tests shown in Figure 4) indicated that the
188
principles of OBTE in both BRM and ARM are viewed with similar importance as training
Soldiers to move, shoot, think and communicate.
Figure 4. Importance of teaching typical Army-trained skills vs. OBTE principles in (a)
BRM and (b) ARM.
Responses from individuals before and after taking CATC did not differ significantly with regard
to training Soldiers to move, shoot, think, and communicate in either BRM (t(167) = 0.33, ns) or
ARM (t(167) = 0.00, ns). Similarly, responses did not change from before to after CATC
instruction regarding the importance of OBTE principles in either BRM (t(167) = 0.61, ns) or
ARM (t(167) = 0.73, ns). As is evident in Figure 4, the lack of significant differences is cannot be
attributed to high variance in the responses. It appears that the OBTE principles of problem
solving, intangibles, understanding and awareness, deliberate thought, and overcoming the
demands of combat were seen as important as typical Army-trained skills both before and after
receiving CATC instruction. We cannot rule the possibility that the pattern of responses simply
reflects indifference or ambiguity. Differences found on other items in the surveys indicate,
however, that DS did attempt to answer the questions thoughtfully.
9.2.2.2 Perceptions of training objectives in rifle marksmanship. A second objective of this
survey study was to examine whether familiarization with OBTE influenced perceptions of the
formal training objectives that underlie current training in BRM and ARM. Respondents were
asked to articulate via open-ended responses their current perception of the objectives in both
BRM and ARM at the time of data collection. The responses were coded into three primary
categories of course objectives: (1) qualification standards (e.g., To qualify with assigned
weapon), (2) marksmanship fundamentals (e.g., To teach Soldiers to understand the
fundamentals of BRM), and (3) OBTE principles (e.g., Confidence, competence, and decision
making process).
The frequencies of responses for each category in Fall 2008 and Spring 2009 is shown for both
BRM and ARM in Figure 5. Results of a chi-square test indicates that there is a significant
difference between the pattern of responses in Fall 2008 and Spring 2009 regarding course
objectives for both BRM (2(2) = 41.89, p < .001) and ARM (2(2) = 60.08, p < .001). More
specifically, after six months of implementation of OBTE, there is a statistically significant
increase in the percentage of respondents who indicate that qualification standards are the formal
objectives of both BRM and ARM. This finding is somewhat surprisingly given the emphasis of
OBTE on more general development along with marksmanship. One possibility is that these
results indicate a key take-away from CATC is a strong emphasis on marksmanship qualification,
possibly due to the intense marksmanship focus of CATC (however, see next section).
189
Dean et al.
Figure 5. Perceptions of (a) BRM and (b) ARM course objectives at the time of data
collection.
A second possibility is that respondents are more aware of the limitations in the ways BRM and
ARM are currently designed because of their experience in CATC and familiarization with OBTE
more generally. Perhaps, with a better understanding of OBTE in the spring, they are indicating
an understanding of the limitations and focus of what is currently taught. This notion is reinforced
by the results about OBTE Principles. With a better understanding of OBTE, DS perceive current
training objectives to be less relevant to the principles of OBTE (Figure 5) at the same time that
they recognize that such principles are equally important as current training objectives (Figure 4).
9.2.2.3 Influence of basic training on Soldier intangibles. A central tenet of OBTE is the
emphasis on development of intangibles such as confidence, accountability, and initiative.
Thus, a third objective of this survey study was to examine changes in perceptions between Fall
2008 and Spring 2009 about whether Basic Training, in general, puts Soldiers in situations that
develop the intangibles. Responses to this open-ended item (shown in Figure 6) were coded into
three categories: (1) Yes, (2) Sometimes, and (3) No. Most responses were worded similarly to
the following: Yes, we give them opportunities to make their own decisions, On some cases
yes but majority of the time they are just getting pushed through the training whether they grasp
the concept or not, and No. Training does not seem realistic.
Figure 6. Perceptions of whether Basic Training develops intangibles in Soldiers.
190
To determine whether responses differ between Fall 2008 and Spring 2009, a chi-square
distribution test was conducted using the response frequencies shown in Figure 6. Results indicate
that respondents in the Spring believed to a greater degree that Basic Training puts Soldiers in
situations that develop intangibles (2(2) = 10.37, p < .01). These results suggest that the adoption
of OBTE throughout the 434th Brigade has successfully increased the extent to which respondents
believe that outcomes such as the intangibles can be and are addressed in Basic Training. This
result is striking given the interpretation of related results shown in Figures 4 and 5. The
implication is that DS are taking the initiative to create situations favorable to OBTE because
they believe the principles of OBTE are important (Figure 4) and in spite of the fact they are
perceived to not be relevant to current training objectives (Figure 5).
9.3 Conclusions
Overall, our findings provide evidence for change in behavior of DS and Privates in the 434th
Field Artillery Brigade at Fort Sill. In particular, evidence suggests that DS changed their
instructional behavior to be more consistent with OBTE after being familiarized with it through
their experience as participant observers in CATC and in various seminars and briefings. DS
altered elements of their instruction to be more learner-centered, and they asked Privates more
questions to promote deliberate thought and problem solving. They also attempted to involve Peer
Coaches as collaborative problem solvers, for example, in working to group and zero during
marksmanship instruction. Critically, these changes were accompanied by changes in the
behavior and performance of the Privates. Their marksmanship and performance on items central
to OBTE principles improved. These changes occurred with no apparent decreases in safety
despite increased handling of weapons by Privates. Moreover, although data on attitudes suggest
that qualification standards are seen as the objective of BRM and ARM, responses indicate
increased awareness of the potential for OBTE in such training.
Taken together, these data constitute strong and converging lines evidence for the potential
impact of OBTE, at least as in basic training, consistent with evidence from a prior investigation
at Fort Jackson and Fort Benning (see Chapter 8). It is noteworthy that there was evidence of an
impact of familiarization with OBTE, in both investigations, even though the evidence for
implementation of OBTE in DS behavior was rudimentary. Our overall interpretation, thus, is that
there is considerable potential for improvement in implementation and for greater impact on
Soldiers. While compelling, however, it must be noted that these data are not fully conclusive.
Due to the nature of the work reported here (observation of naturally occurring events), many
elements changed between the pre- and post-implementation collection events, and we were
unable to employ control conditions or to utilize completely nave observers. Thus we cannot
draw precise conclusions about the relative causal potency of the various changes in the
command climate at Fort Sill and the associated interventions intended to familiar DS with
OBTE. These caveats suggest future efforts that could target specific elements of the
interventions in laboratory or other highly controlled circumstances.
Implementation of OBTE was neither as complete nor as effective as it might have been. Scores
for both DS and Privates reflected moderate ratings rather than those viewed as ideal in the
assessment framework that is available for verification of OBTE (see Chapters 1 and 2). The
challenges thus identified by this study constitute themes to address in continued implementation
and refinement. For instance, when presented with more complex tasks, Privates adherence to
marksmanship fundamentals began to wane and became sloppy. This suggests that many Privates
were not yet fully ready to move forward in the exercises, and yet they were moved forward due
to throughput constraints. Due to this constraint, much training remains focused too strongly on
efficiency as opposed to effectiveness, and throughput remains the greatest threat to effective
191
Dean et al.
training. In addition, DS needed to draw more explicit connections between isolated tasks and the
larger context as well as giving explanations of why things matter. These connections should
facilitate sense making (see Chapters 2 and 3). Finally, while DS are employing peer coaches
more effectively, they can mentor these coaches more on how best to help their peers (e.g., what
to look for, and how to provide feedback); they can involve them more fully in feedback sessions
so that they have maximal opportunities to engage in collaborative problem solving while
providing them with timely and appropriate guidance. More generally, such opportunities to
develop problem-solving skills and to think and act as part of team will greatly benefit progress
toward key outcomes of OBTE.
9.4 References
Meade, VA: Asymmetric Warfare Group.
Fort Meade, VA: Asymmetric Warfare Group.
192
Operational Influence of CATC

Chapter 10. Influence of CATC in an Operational Setting
Rob Semmens
Imprimis, Inc
Zachary N. J. Horn, Kevin Durkee, Tyrone Jefferson
Aptima, Inc
Gary E. Riccio
The third source of evidence about OBTE as a service system is self-report data obtained from
Soldiers who had an opportunity to implement OBTE with their own units upon deployment in
the contemporary operating environment. Such data provide an understanding of the operational
value of OBTE to the Army. In particular, they provide indications about the influence of OBTE
on the competency of Soldiers in their job as Soldiers in lethal engagements with the enemy
(Kirkpatrick, 1994, level 3), and they suggest the corresponding contributions of OBTE to the
organizational mission and objectives of the U.S. Army (Kirkpatrick, 1994, level 4). Similar to
the AWG survey, these data attempted to determine whether Soldiers acquired a thorough
understanding of OBTE, rather than learning solely about marksmanship skills, despite the focus
on marksmanship in early versions of CATC. In particular, our goal was to assess whether
Soldiers implement OBTE principles in training to prepare their own units for combat, and if so,
to understand the quality of that outcome.
10.1 Methods
10.1.1 Participants
This survey was administered to a small sample of individuals from three battalions in the
Armys 82nd Airborne Division. Participants were told that participation was voluntary and
Soldiers were under no obligation to complete the surveys. All respondents participated in CATC
in November or December of 2006 prior to deployment. Hence, participation was with respect to
the initial version of CATC, prior to its development into a more explicit train-the-trainer course
at Fort Benning and Fort Jackson. Responses were obtained via paper surveys from 17 Soldiers
consisting of seven Sergeants (SGT), nine Staff Sergeants (SSG), and one Sergeant First Class
(SFC). All participated in the initial AWG course and were identified by their Commanders for
participation in this survey. All respondents had been in positions of leadership within Platoons
during their most recent deployment.
10.1.2 Procedure
This survey was developed specifically for Soldiers in the Armys 82nd Airborne Division who
had recently returned from a tour in either Iraq or Afghanistan. Surveys included 30 numerically
scaled multiple-choice questions along with five open-response questions. These questions
targeted four distinct topics:
Operational impact of CATC on Soldier behavior and performance in theater

The degree to which CATC increased the willingness and ability to train others
according to OBTE philosophy
193
Semmens et al.
Their level of improved understanding of intangibles and core principles

following CATC participation
Potential impediments to implementing OBTE throughout the Army
10.1.3 Analyses
Responses to the survey were summarized and reviewed with OBTE experts at the AWG. The
results parallel the key findings in the AWG survey. However, those findings were revisited from
in the post-deployment surveys from the perspective of active-duty Soldiers who had the
opportunity to implement OBTE in their own units and to experience its impact and outcomes in
Full Spectrum Operations.
10.2 Results
10.2.1 Downstream Impact on Marksmanship
The first key finding from the post-deployment survey addresses the operational impact of OBTE
on Soldier behavior and performance as marksmen in a firefight or in a situation where a firefight
was likely to occur. Respondents reported that OBTE had a notable influence on behavior in
combat situations. There are multiple responses that emphasize this assertion. First, all 17
respondents reported that techniques learned in the AWG course influenced something they did in
the contemporary operating environment on two or more occasions, such as rigging of equipment,
reloading, or alternating firing positions (76% of respondents indicated more than five occasions).
Second, 16 of the 17 respondents reported that OBTE had some level of influence on how they
moved, communicated, and returned fire while in fire fights (63% of these responses indicated
significant influence). Third, most respondents managed to reload their weapon when they
wanted to more often than when they had to (47% reported much more often, 24% reported
slightly more often), a point emphasized in CATC. Finally, most respondents reported making
use of objects in the environment to increase the stability of their firing position (65% reported
always trying to do this, 24% reported doing this only when additional control was needed),
another point emphasized in CATC.
These self-report results suggest that the AWG course was particularly effective in transferring
the training to situations the Soldiers faced in theater. It is important to note that this transfer not
only consisted of tactical skills but also general thinking skills. Respondents indicated that they
reload their rifles when they wanted to and made use of objects in their environment to increase
stability. These results demonstrate a direct transfer of OBTE principles (e.g., deliberate thought,
understanding and awareness). It also appears that Soldiers found success in implementing these
techniques during combat given that most respondents applied OBTE in multiple combat
situations. This is consistent with the AWG survey findings in which Soldiers impressions of the
AWG course were overwhelmingly positive. The self-report data suggest that behavior learned
through OBTE are retained, readily adapted and applied to novel situations faced by Soldiers in
theater.
194

Table 10: Influence of OBTE on Soldier willingness and ability to perform training actions.
Relative to other training, to what
extent did CATC increase your
willingness & ability to do the
following?
Same/Worse*
Slightly/Much Better
Frequency
Response %
Frequency
Response %
Define training outcomes
23.53%
13
76.47%
Match outcomes to situations
23.53%
13
76.47%
Mentoring Soldiers
11.76%
15
88.24%
Soldiers take own responsibility
11.76%
15
88.24%
Provide challenging conditions
11.76%
15
88.24%
Increase difficulty over time
17.65%
14
82.35%
Get the most out of training
11.76%
15
88.24%
Interdependence
11.76%
15
88.24%
Problem solving
11.76%
15
88.24%
Soldiers provide feedback to peers*
11.76%
15
88.24%
Want to improve training
11.76%
15
88.24%
Total (Across Principles)
27
14.44%
160
85.56%
Note: *Only one response (regarding providing feedback to peers) indicated that CATC was
worse than other Army training; all other ratings in this category indicated the same influence
as other training.
10.2.2 Downstream Impact on Training in the Units
The second key finding from this post-deployment survey addresses whether Soldiers train others
in their own units by making use of OBTE. The vast majority of respondents expressed
willingness to train other Soldiers in theater with the OBTE approach. Table 10 illustrates this
point by showing how Soldiers responded to a list of OBTE points of emphasis in the context of
their experience with other training in the Army. They were highly positive in responses about
training others using an approach like OBTE. For all 11 items, respondents generally indicated
that the AWG course does a better job than other approaches at increasing willingness and ability
to perform the trained behavior (52% of all responses indicated much better, 34% of all
responses indicated slightly better). Only one response indicated that attempts to teach with
OBTE were worse than typical Army training, specifically with regard to providing feedback to
peers during training (this rating is combined with the Same category in Table 10). There were
no ratings of much worse with respect to OBTE training.
These results strongly suggest that, based on their CATC experience, Soldiers held OBTE in high
regard compared to other explicit or implicit approaches to training the Army. While the observed
frequencies in Table 10 did not depart significantly from the expected frequency distribution
[Likelihood Ratio () = 2.94; df = 10; ns], respondents indicated either a slight or much
better improvement of OBTE over other Army training approaches to a significant degree across
all principles (z = 9.72, p < .01). OBTE also appears to motivate Soldiers to employ the
approach while training others even if that was not the explicit intent of the course in which they
were exposed to the approach. This finding is consistent with the AWG survey findings in which
positive affect toward the OBTE methods was by far the most commonly cited theme among
Soldier responses. Soldiers tend to experience positive consequences of OBTE (e.g., a focus on
mentoring Soldiers, interdependence, problem solving) as well as the willingness to apply the
195
Semmens et al.
lessons learned to training other Soldiers with the same approach. This suggests significant
potential for using the OBTE training approach throughout the Army.
10.2.3 Downstream Impact on Self Efficacy
The third key finding from the survey addresses the degree to which Soldiers feel they personally
improved in ways that reflect developmental outcomes of OBTE. We found that all 17
respondents felt they experienced improvement with respect to each of eight outcomes as a result
of participating in OBTE. As shown in Table 11, the most common responses about these
outcomes were notable or great improvement (81% of all responses). The two outcomes that
exhibited the most improvement were situation awareness (100% of respondents indicated
notable or great improvement) as well as judgment and thinking skills (94% of respondents
indicated notable or great improvement). There were zero responses of no improvement
across all outcomes.
Further analyses of the responses shown in Table 11 indicate that respondents were significantly
more likely to report notable or great improvement for these principles [Likelihood Ratio ()
= 20.93; df = 7; p < .01]. The difference in the overall percentages between reports of
notable/great improvement and slight/moderate improvement was also significant (z =7.20; p <
.01). Overall, these tests indicate that all respondents found value in OBTE with regard to
outcomes for individual development as a Soldier. Perhaps the most noteworthy finding is that
every respondent reported some degree of improvement across every outcome. These results are
consistent with the AWG survey findings, in which the majority of Soldiers understood that the
OBTE training approach applies to more than just marksmanship. In an attempt to explain the
variance in improvement across the outcomes shown in Table 11, we propose two explanations:
(a) the OBTE approach used in CATC influences development with respect to some outcomes
more effectively than others; or (b) the outcomes with respect to which improvement was greatest
in CATC may not be addressed in standard Army training programs.
Table 11: Degree of self-reported improvement in eight developmental outcomes of OBTE.
To what extent has CATC helped
you improve in the following?
Slight / Moderate
Notable / Great
Frequency
2
Response %
11.76%
Frequency
15
Response %
88.24%
General Confidence
23.53%
13
76.47%
Accountability
41.18%
10
58.82%
Initiative
29.41%
12
70.59%
Discipline
35.29%
11
64.71%
Situation Awareness
0.00%
17
100.00%
Judgment
5.88%
16
94.12%
Thinking Skills
5.88%
16
94.12%
Total
26
19.12%
110
80.88%
Confidence in Marksmanship
196
10.3 Implications for Service System Development: Validation

The results from the 82nd Airborne post-deployment survey provide further evidence that: (a)
OBTE can influence development of cognition and behavior, beyond tactical marksmanship
skills, that transfer to combat situations; (b) compared to other training within the Army, some
Soldiers perceive OBTE to be an approach they prefer and are willing to employ in training
others; and (c) the focus on OBTE principles during training helps Soldiers develop with respect
to outcomes typically not addressed in marksmanship training (e.g., confidence in marksmanship,
situation awareness, judgment, and thinking skills). The reported improvement in these areas also
suggests that, more often than not, Soldiers can come to understand that the intent of OBTE is to
develop personal attributes and competencies beyond marksmanship and weapons operation.
However, while these findings illustrate positive effects of OBTE on performance and teaching in
theater, the small nonrandom sample of respondents cannot be generalized to the population of all
participants in OBTE. The findings nevertheless show the potential of OBTE to influence
Soldiers. Therefore, it is recommended that the link between OBTE and performance in theater
should be a focus of future investigations and the associated development of supporting
capabilities for validation of OBTE as a service system (CMMI Product Team, 2009).
In our view, the most conspicuous gap in Army training and education is the gap between the
Generating Force and the Operating Force (Riccio, dEchert, Lerario, Pound, Bruny, & Diedrich,
2006; Riccio, Lerario, dEchert, Pound, Bruny, & Diedrich, 2006). It is not surprising, then, that
the mission of the AWG came to include demonstrations such as OBTE. Attention to the
downstream impact of instruction should be an essential component of any instruction service
system in the Army. In this respect, there are lessons to be learned from the post-deployment
survey of the 82nd Airborne. First, the development of this survey attempted to reflect essential
attributes of OBTE such as represented in the measures of instructor behavior (Chapter 2). This is
critically important insofar as collaboration between instructional institutions and units in theater
should address what the attendant conversations and exchanges of information should be about.
Otherwise, this reach back is likely to dissipate into yet another time-inefficient process or
overwhelming knowledge repository (Riccio, Diedrich, Lerario, dEchert, B., Sidman, Geyer, &
Bruny, 2007; Riccio, Lerario, et al., 2006). Measures of instructor behavior, or the performance
indicators from which they were derived, can provide a framework of common relevance that can
help collaboration be efficient and actionable (Riccio, dEchert, et al., 2006).
The survey of the 82nd Airborne also suggests improvements that should be considered for
methods of validation that provide actionable feedback. The most obvious limitation of that
survey was the small sample size. The sample was sufficient, however, to confirm the utility and
usability of the survey. In particular, it showed the value of questions derived from the principles
of OBTE and the associated measures of instructor behavior. It was not sufficient, however, to
generalize to the population of Soldiers who have been exposed to OBTE or to speculate about
the potential impact of OBTE on Soldiers and other personnel who have not taken OBTE. Larger
surveys with wider distribution may or may not be the solution. In any case, other methods should
be considered for obtaining feedback about downstream operational impact of OBTE and
associated developmental outcomes. The Web is one obvious source of alternatives for closing
the gap with Full Spectrum Operations (Riccio et al., 2007) and providing a more communitycentered environment for instruction (Bransford et al., 2000).
Validation, such as represented in the survey of the 82nd Airborne, is a good method to address
Kirkpatricks level 3 of program evaluation (Kirkpatrick, 1994) that provides actionable feedback
about job performance. Level 3 evaluation corresponds to the considerations of a knowledgecentered learning environment because the requirements of the job are the most basic level of
197
Semmens et al.
knowledge around which an instructional program should be organized. We believe that level 3
evaluation can be improved by considering the confluence of knowledge-centered and
community-centered aspects of a learning environment, as emphasized by Bransford et al. (2000)
(see Chapter 3). In the context of OBTE, a community-centered orientation on knowledge-centric
issues is provided by the further development of methods, for interacting with Soldiers during or
after deployment, that incorporate operationally relevant measures of OBTE. Measures that can
provide valuable feedback about performance on the job as well as learning in a formal program
of instruction not only have greater validity but their co-evolution can help sustain and improve
the operational relevance of learning environments. This also helps address Kirkpatricks level 4
of program evaluation that attempts to bridge the gap between impact on the individual and
relevance to the collective mission and objectives.
10.4 References
Publishers, Inc.
Riccio, G., Diedrich, F., Lerario, M., dEchert, B., Sidman, J., Geyer, A., & Bruny, T. (2007).
Tools to help prepare Soldiers for the Contemporary Operating Environment. Procedings of
the Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC),
Orlando, FL.
Implications for Service System Development
198
Chapter 11. Implications for Service System Development

Gary E. Riccio
Fred Diedrich
Aptima, Inc.
11.1 Lessons Learned about Transfer of OBTE
The overall goal of OBTE is to foster the development of Soldiers with respect to personal
attributes and competencies that are critical for success in Full Spectrum Operations (Table 1). It
offers a variety of strategies to enable progress toward such outcomes (Darwin, 2008; CornelldEchert, 2009a,b). More specifically, CATC focuses on the role of instructors in the
development of Soldiers through a field-based marksmanship course. Results from the surveys
indicate that participants are generally positive when asked about their experiences in CATC and
that they hold the instructional approach in high regard. Moreover, their comments tend to focus
on outcomes around which the approach is organized (problem solving, intangibles,
understanding and awareness, and combat performance). This suggests that they understand the
intent of OBTE. In addition, when asked about longer-term impact, respondents indicated that the
AWG course had an influence on their behavior when in theater, that it influenced their training
approaches with their own units, and that they experienced growth with respect to core outcomes.
The evidence from observations of actual instruction following CATC reveals both opportunities
and challenges. In particular, our findings suggest that DS do not consistently apply OBTE to the
extent possible in Initial Entry Training. In the particular application of OBTE at Fort Benning
and Fort Jackson, transfer of training from CATC is not complete (see also, Dean et al. 2009).
CATC provides exemplary marksmanship training while developing outcomes consistent with
OBTE. While marksmanship is a valuable motivator, it may distract participants in CATC from
learning as much as they otherwise might learn about OBTE as instructors. Perhaps more
importantly, as the AWG surveys suggest, a single one-week exposure to OBTE probably is not
sufficient to educate instructors about the varied opportunities for detailed implementation. This
has implications for further integration and development of instructor education about OBTE into
the Institutional Army (Cornell-dEchert, 2009a,b). In particular, there is a need for capabilities
that can provide or guide opportunities for continuing education about OBTE beyond field-based
courses like CATC or occasional workshops and seminars.
From the perspective of OBTE for process improvement, our data provided actionable feedback
about gaps in DS application of OBTE in Initial Entry Training. For instance, stress often was
added too quickly due to throughput considerations, such that increasing complexity was not used
as a training enabler. Privates did not come to understand why stress is added and how to manage
it. Instead, as training increased in complexity, Privates failed to execute fundamentals adequately
in the face of increasing complexity. This worked against development of confidence, initiative,
and combat performance. In addition, the level of one-on-one verbal interaction between DS and
Privates was very low. Privates rarely asked questions, and DS rarely asked Privates questions.
This limited the opportunities for growth in problem solving skills and understanding. Privates
often were told what to do and told solutions, rather than being guided to develop those solutions
themselves. Similarly, DS rarely stressed explanations of why. A consequence of this seemed to
be that Privates likely did not understand the deeper meaning behind their tasks and had limited
chances for real sense-making (e.g., Bransford et al., 2000). All these gaps and shortfalls should
199
Riccio & Diedrich
be relatively easy to address without fundamental changes to doctrinally prescribed Programs of

Instruction or Training Support Packages (Cornell-dEchert, 2009b).
becomes explicitly more directed toward instructor education (Cornell-dEchert, 2009a,b). It is
noteworthy that, even without explicitly addressing instructor education, our findings suggest that
OBTE can motivate individuals to take ownership of their own learning and development and that
it can increase self-efficacy for teaching and developing others. The approach motivates a greater
interest in instructor-student interactions and how these interactions affect progress toward
developmental outcomes while satisfying course-specific learning objectives (Cornell-dEchert,
2009b). This has clear implications for the kind of future research that could have a practical
impact on training in the Army (Bandura, 1995; Deci & Ryan, 1985, 2008; Ryan & Deci, 2008;
Schwartz, Lin, Brophy, & Bransford, 1999; Sidman, Riccio, Semmens, Dean, & Diedrich, 2009).
11.2 Implications for Service System Development
In the context of CMMI for Services (CMMI Product Team, 2009), our findings lead to
recommendations for both instructors and their units. It emphasizes aspects of instruction to
sustain or to improve with respect to the ideals of OBTE (Cornell-dEchert, 2009a,b).
11.2.1 Further Development and Analysis of Stakeholder Requirements for OBTE
The practices associated with requirements in the Service System Development process area of
CMMI are not simply a front-end activity. The needs of end-users and stakeholders can change
over the life cycle of a service system. It also is the case than the understanding of requirements
becomes more refined and elaborated over experience with service system development,
transition and delivery (CMMI Product Team, 2009, p. 440; Swain, 2005). Development and
analysis of requirements should be continuous, for example, in due diligence on service incidents
and institutional constraints, restrictions, and limitations on the delivery of service. Some problem
areas observed in the current investigation warrant further collaboration with end-users and
stakeholders, such as:
Emphasis on throughput, or efficiency over effectiveness, puts understanding and

mastery at risk. While it is certainly the case that logistical demands necessitate
efficiency, this criterion is meaningless if knowledge or skills are not retained or not
readily adaptable to Full Spectrum Operations.
To the greatest extent possible, advancement across events and application of increasing
complexity should be based on leaner needs, implying that the pace of training needs to
be more fully differentiated between learners. From the perspective of OBTE, this
emphasizes mastery as a pursuit that builds confidence and effective combat application.
Where possible, instructors should allow Soldiers to assume responsibility for safety and
their learning. From the perspective of OBTE, such self-reliance builds confidence,
initiative, and accountability. Moreover, to the extent that Soldiers become accountable,
DS may be able to concentrate more on instruction and less on management.
11.2.2 Further Development of OBTE as a Service System

While CATC is focused on marksmanship, it is not about marksmanship per se, and this
distinction can be difficult to grasp given the participants intense interest in marksmanship. The
weeklong course, to which the present sample of DS was exposed, focused almost solely on
marksmanship. There was only a small amount of discussion on application of the methods to a
200
different domain (e.g., land navigation). DS had difficulty understanding the instructional
techniques they witnessed (e.g., discussing minutes of angle in ballistics) as exemplars of a more
general instructional strategy (e.g., explaining the why, encouraging deliberate thought, and
awareness of context dependence of skills). This is not surprising given that exposure to only one
exemplar is not likely to promote generalization and abstraction. Hence, we recommend that
instructor education include applications of OBTE across multiple domains of skill and learning.
It also is clear that additional support can be provided to participants in a field-based train-thetrainer course soon after they complete the course. For instance, elements such as take-home
materials that explore application to different domains could be provided, perhaps employing
multimedia-based vignettes that challenge the instructors to apply lessons learned (Bruny,
Riccio, Sidman, Darowski, & Diedrich, 2006). In addition, such materials and exercises could be
employed in workshops, following marksmanship training for example, that promote student
application to domains outside of marksmanship (Sidman et al., 2009). The critical point is to
provide support that gives instructors opportunities to apply what they have learned about OBTE
to a variety of domains. Consistent with OBTE, the approach to further instructor education
would be to introduce additional domains that provide instructors with general strategies and
considerations rather than an apparent script for instruction in the new domain. Some general
strategies are reiterated below:
Where possible, instructors should allow Soldiers to solve problems. The emphasis
should be on asking leading questions instead of telling Soldiers the solution. From the
perspective of OBTE, this strategy gradually builds skills of thinking and problem
solving, and reinforces expectations of accountability rather than dependence on their
superiors.
The easiest change with a potential for significant impact is to create an environment that
fosters communication with Soldiers, rather than communication directed at Soldiers or
the absence of communication. From the perspective of OBTE, to the extent that
Soldiers feel that they can ask questions, and make some mistakes, they will come to
better understand combat application, master skills, and grow in confidence.
Where possible, peer coach should be utilized to overcome the limitations of instructorstudent ratio. Although peer coaches will no doubt provide inappropriate guidance on
occasion, this may be outweighed by the benefit of Soldiers coming to see themselves as
active participants in problem solving and discovery. This is especially true with the high
student-to-instructor ratios that are likely to be the biggest perceived obstacle to full
implementation of OBTE. To the extent that instruction is viewed not only as the transfer
of knowledge, but as an opportunity for collaborative problem solving, then the role of
the peer coach becomes critical.

It is important to note that inculcation of a mindset consistent with OBTE requires that the work
environment should be open to and supportive of change (e.g., Bandura, 1995; Burke & Hutchins,
2007; Rasmussen, 1997). As noted by Dean et al. (2009), DS indicated that they felt somewhat
constrained by the nature of the programs of instruction and associated resources. It is likely that
gaps in transfer of OBTE were rooted, at least in part, in perceived constraints on rifle
marksmanship training and the extent to which the DS believed they could deviate from the
common practices. A key challenge, therefore, is the extent to which programs of instruction can
be made to be more flexible and DS can be empowered to believe that they can take initiative to
more fully implement OBTE. Hence, there is a critical need to educate commanders and
supporting units about the value of OBTE as a service system. Without a command climate that
201
Riccio & Diedrich
fosters agility in instructional units, an approach like OBTE is not likely to be successful
(Appendix C; Haskins, 2009; Schwitters, 2009).
11.2.3 Further Verification and Validation of OBTE
There is a broad range of best practices for peer review, verification, and validation in Service
System Development (CMMI Product Team, 2009, pp. 454-462). As in the current investigation,
it is important to employ a multifaceted and multidisciplinary approach to these activities to avoid
sub-optimized solutions and unintended consequences. Consideration of a portfolio of
methodologies, and even some exploration with alternative methods, also helps an organization
find the right level of effort and detail for its process improvement (Garcia & Turner, 2006). In
the present context, a specific aim was to identify ways in which the behavioral and social
sciences can provide guidance to verification and validation of an instructional service system.
There are two general characteristics of our approach in this respect: (a) develop grounded theory
for OBTE; and (b) promulgate scientific inquiry into OBTE as a process over time conducted
within a diverse community of practice.
On our view, identification and development of theoretical foundations is critically important in
service system development because of the diversity of methodologies that are available and that
may have to be employed for verification and validation. Theoretical coherence arguably is the
only way to identify convergence among otherwise incommensurate sources of evidence. From
the outset, we intentionally included in the research team, a diversity of subdisciplines and
theoretical commitments, mostly within the discipline of scientific psychology. The grounded
theory that emerged in this investigation came to be dominated by theoretical commitments allied
with the interrelated lineages of social learning theory, situated cognition, motivation and
emotion, ecological psychology, and dynamical systems theory in the social and behavioral
sciences. We are explicit about this bias wherever appropriate. More generally, we believe that
reflection and candor about theoretical biases should be a characteristic of systems engineering
applied to the integration and development of capabilities that have an impact on behavioral and
social phenomena (Flyvbjerg, 2001; Godfrey-Smith, 2003; Schrim & Caterino, 2006; see also,
Epilogue in this monograph).
We believe that the value of a theoretical commitment (e.g., coherence and directedness)
outweighs the potential disadvantage of narrowed vigilance in the development of grounded
theory. At the same time, we recognize that involvement of a broader scientific community can
mitigate the potential problem of narrowness. This is important beyond the value of debate,
skepticism, and alternative sources of evidence. It emphasizes an aspect of science that is
consistent with the intent of peer review, verification, and validation in systems engineering.
Scientific and technical understanding becomes more refined and elaborate over time with the
accumulation of evidence. One should avoid presumptive judgments based on point estimates
and isolated comparisons with questionable generalizability in domains as broad as training and
education or in specific applications that are replete with uncontrollable sources of variance.
In this regard, there is much to be learned from the other evidence-based services systems and
associated scientific disciplines about how to use and how not to use scientific evidence in social
decision-making (American Association for the Advancement of Science, 1993/2009; Bailar,
1997; Best, Trochim, Haggerty, Moor, & Norman, 2008; Foster & Huber, 1998; Glasziou &
Haynes, 2005; Kohn, Corrigan, & Donaldson, 1999; Mislevy & Riconscente, 2006; National
Research Council, 2009; Pellegrino, Chudowsky, & Glaser, 2001; Sackett, Rosenberg, Gray,
Haynes, & Richardson, 1996; Swales, 2000). A socially aware and scientifically based approach
to verification and validation should strive for theoretical coherence, juxtaposition of
202
complementary and opposing perspectives, traceability to programs of research that span decades,
empirical evidence from multiple methods that are replicable, and consideration of sources of
variability and uniqueness in empirical findings to ensure that conclusions are credible,
transferable, dependable, and confirmable (cf., Denzin & Lincoln, 2003; Foucault, 1966/2002;
Godfrey-Smith, 2003; Kuhn, 1962/1970; Popper, 1959). It should not be limited by narrow
conceptions of hypothesis testing (cf., Flyvbjerg, 2001; Henkel & Morrison, 1970/2006; Kline,
2004; A. Ryan, 1959; Schrim & Caterino, 2006).
With respect to verification and validation, as well as scientific influence on these endeavors, an
important contribution of this investigation is to stress the importance of productive dialog about
OBTE within a diverse community of stakeholders. The intent of this dialog should not be to
prescribe the use of particular instructional methods or techniques but to provide some scientific
guidance about the most fruitful topics of conversation and innovation by instructors (cf., James,
1899/1907, pp. 7-11). In particular, we believe this investigation can stimulate productive dialog
because of the definition of OBTE in terms of instructor behavior and instructor-student
interactions and, more specifically, because of the measures of instructor and student behavior
that enable OBTE to be verifiable. They provide one topic of conversation for forums in which
there is peer-to-peer sharing of information about best practice in training and education (e.g.,
Costanza, Leibrecht, Cooper, & Sanders, 2009). In the context of continuous verification and
validation, we believe such conversations should include scientists as well as instructors,
instructional designers, course developers, quality assurance personnel, and commanders in
instructional programs. The most radical departure would be to facilitate contributions from
stakeholders in theater (Riccio, dEchert, et al., 2006; Riccio, Lerario, et al., 2006).
There is dialog and debate about OBTE in a diverse community of stakeholders, and it appears to
be growing (AWG, 2009). Figure 6 suggests a challenge in achieving efficient sharing of issues
and lessons learned about OBTE. Stakeholders are widely dispersed. Decentralized collaboration
and the resulting lessons learned thus are not readily apparent in timely fashion to decision
makers. Potential solutions to this problem are emerging in the nascent Army Training Network
(ATN) that builds on the recently revised Army Field Manual, FM 7-0 Training for Full
Spectrum Operations, and transforms FM 7-1 into a Virtual Field Manual (Davis, 2009). One
use of this forum would be for continuous peer review that is central to verification and validation
in CMMI Service System Development (CMMI Product Team, 2009). A scientific approach to
verification and validation can help ATN establish a topic of peer-to-peer discussion that is more
likely to be on point, efficient, systematic, and actionable for OBTE in particular and for good
training in general.
Figure 6: Needs for distributed peer-to-peer collaboration about OBTE (after Devens, 2009)
203
Riccio & Diedrich
11.3 References
American Association for the Advancement of Science (2009). Benchmarks for scientific literacy.
New York: Oxford University Press. Retrieved April, 2009, from American Association for
the Advancement of Science, Project 2061, http://tinyurl.com/c9mrcr. (Original work
published 1993)
Asymmetric Warfare Group (2009). U.S. Army Asymmetric Warfare Group workshop on
Outcomes-based Training and Education. Applied Physics Laboratory, Johns Hopkins
University, Laurel, MD.
Bailar, J. (1997). The promise and problems of meta-analysis. New England Journal of Medicine,
337(8), 559-561.
University Press.
Best, A., Trochim, W., Haggerty, J., Moor, G., & Norman, C. (2008). Systems Thinking for
Knowledge Integration: New Models for Policy-Research Collaboration. In L. McKee, E.
Ferlie & P. Hyde (Eds.), Organizing and Reorganizing: Power and Change in Health Care
Organizations (pp. 154-166). New York, NY: Palgrave Macmillan.
Burke, L.A., & Hutchins, H.M. (2007). Training transfer: An integrative literature review. Human
Resources Development Review, 6, 263-296.
Costanza, M., Leibrecht, B., Cooper, W., & Sanders, W. (2009). Peer-To-Peer Training
Facilitators Guide (ARI Research Product, in press). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences.
Darwin, M. (2008). Outcomes-based training and education: fostering adaptability in full
Group.
Davis, J. (2008, January 21). Call for Army Training Network (ATN) training products. Retrieved
March, 2009 from US Army Combined Arms Center, Blog Network, Army Training
Network, http://tinyurl.com/alxknw.
Devens, M. (2009, March). Introductory remarks. Presentation at the US Army Asymmetric
Deci, E.L. & Ryan, R.M. (1985). Intrinsic motivation and self-determination in human behavior.
New York: Plenum.
204
Foster, K, & Huber, P. (1998). Judging science: Scientific knowledge and the federal courts.
Cambridge, MA: MIT.
Foucault, M. (2002). Order of things: An archaeology of the human sciences New York:
Routledge Classics. (Original work published 1966).
Garcia, S, & Turner, R. (2006). CMMI(R) survival guide: Just enough process improvement.
Upper Salle River, NJ: Addison-Wesley.
Glasziou, P., & Haynes, B. (2005). The paths from research to improved health outcomes. ACP
Journal Club, 142(2), A8-10.
Henkel, R. & Morrison, D. (Eds.) (2006). The significance test controversy: A reader.
Piscataway, NJ: Aldine Transaction. (Original work published 1970)
James, W. (1899/1907). Talks to teachers on psychology: and to students on some of life's ideals.
New York, NY: Henry Holt and Company.
Kline, R.B. (2004). Beyond significance testing: Reforming data analysis methods in behavioral
research. Washington, DC: APA.
Kohn, L., Corrigan, J., & Donaldson, M. (Eds.) (1999). To err is human: Building a safer health
system. Washington, DC: National Academy Press.
Kuhn, Thomas (1970). The structure of scientific revolutions. Chicago: University of Chicago
Press. (Original work published 1962)
National Research Council (2009). Strengthening forensic science in the United States: A path
forward. Retrieved February, 2009 from National Academies Press,
http://www.nap.edu/catalog/12589.html.
Popper, Karl (1959). The logic of scientific discovery. New York: Basic Books.
Science, 27, 183-213.
Ryan, T. (1959). Multiple comparisons in psychological research, Psychological Bulletin, 56, 2647.
Sackett, D.L., Rosenberg, W.M., Gray, J.A., Haynes, R.B., & Richardson, W.S. (1996). Evidence
based medicine: What it is and what it isn't. British Journal of Medicine, 312(7023), 71-72.
Schwartz, D.L., Lin, X., Brophy, S., & Bransford, J.D. (1999). Toward the development of
flexibly adaptive instructional designs. In: C.M. Reigilut (Ed.). Instructional Design Theories
and Models: Volume II. Hillsdale, NJ: Erlbaum.
205
Riccio & Diedrich
Sidman, J., Riccio, G., Semmens, R., Geyer, A., Dean, C., & Frederick Diedrich, F. (2009).
Reshaping Army institutional training: Current training. Final Report to the Army Research
Institute for the Behavior and Social Sciences, contract number W74V8H-04-D-0047 DO
0010.
Barracks, PA: U.S. Army War College.
Swales, J. (2000). The troublesome search for evidence: Three cultures in need of integration.
Journal of the Royal Society of Medicine, 93, 402-407.
Routledge:
Section III. Further Development of OBTE as a Service System
206
207
Marceau et al.
Chapter 12. Development of General Measures for Students
Ryan Marceau, Cullen Jackson, & Fred Diedrich
Aptima, Inc.
Gary E. Riccio
12.1 Intent
In March of 2008, the Asymmetric Warfare Group (AWG) initiated a scientific and technical
effort to help define and develop Outcomes-Based Training and Education (OBTE) (Darwin,
2008; Cornell-dEchert, 2009). With respect to measures development, this initial effort focused
primarily on instructor application of OBTE within field training (see chapter 2). To further this
research, the AWG initiated a follow-on effort aimed at defining observable performance
measures for OBTE based learning that are focused on the student in either a field-based or
classroom setting. The intent was to develop measures that can be used to provide feedback to
students and their instructors about student performance in the context of important
characteristics of the learning environment (Bransford, Brown & Cocking, 2000). Such measures
provide a means for instructors, and students, to reflect on the behavior of students and what they
mean for instruction that realizes (or fails to realize) OBTE. Hence, they are intended to support
formative feedback to students and instructors that enables reflection on student behavior, the
actions that instructors can take to influence those behavior, and the meaning of those behavior in
terms of student progress toward key learning objectives and developmental outcomes in the
context of OBTE (see chapter 3; Pellegrino, Chudowsky, & Glaser, 2001).
12.2 Performance Measure Development Process
We employed a method that focuses on behavior that relates to the acquisition of knowledge,
skills, and competencies in the context of OBTE. The process for developing performance
measures grounds assessment in the behavior of experts and individuals who have attained
proficiency. This helps ensure that behavior-based measures are meaningful and pertain to
constructs for which data can be reliably collected. More specifically, our methodology for
measure development combined subject matter expertise with established psychometric practices
to produce measures of observable behavior. Our approach was based loosely on Aptimas
COMPASS design process (e.g., MacMillan et al., in press). In this case, we employed a fourphase approach to operationalize the principles and practices of OBTE in terms of student
behavior that can be observed and measured. This approach relied heavily on collaborative
discussions with subject matter experts that were conducted in four sequential steps.
12.2.1 Phase One: Define Performance Indicators (PI)
The first phase was to conduct a workshop with the experts to define high level performance
indicators for students in OBTE. This first workshop was conducted in a group setting with the
progenitors of OBTE from the AWG. The first workshop involved an intensive discussion of the
principles and practices of OBTE and how they are manifested in the behavior of learners. More
specifically, in this workshop, subject-matter experts identified behavior on which the
performance measures would be based. These performance indicators (PI) are types of behavior
that allow an observer to rate the quality of a Soldiers performance in individual or collective
tasks. For this workshop, it is critical to identify behavior that can be observed rather than
Development of Student Measures
208
inferred from little or no observable evidence. The PI provide a solid basis on which to develop
measures that are not sensitive to individual rater differences and therefore can be rated reliably
by multiple expert observers. Identifying the PI is a key milestone from which to develop specific
measures of performance within a learning environment.
12.2.2 Phase Two: Translate PI into performance measures
While some PI are readily translated into performance measures, more detailed information is
generally needed to create behaviorally-anchored performance measures from the PI identified in
the first workshop. To develop these behavioral anchors, it is necessary to identify specific
behavior that is relatively effective and behavior that is relatively ineffective for each PI.
Therefore, a second workshop involved one-on-one interviews with subject matter experts to
deliberate about behavior that could illustrate expert, average, and novice performance for each of
the PI.
The second workshop consisted of individual interviews with some of the AWG OBTE
progenitors. The interviews were focused on identifying the novice, average, and expert behavior
for each PI. The topic of conversation focused on the actions of a student that demonstrate
development of intangibles (e.g., confidence, initiative, and accountability), as well as effective
learning. The information gathered in this workshop served to inform the creation of behavioral
anchors appropriate for the measures.
12.2.3 Phase Three: Measure refinement
The goal of the third workshop was a detailed review and modification of a set of draft
performance measures. Subject-matter experts were asked to guide the refinement of measures to
reduce ambiguity and promote inter-rater agreement. This phase relied on expert input to review
and revise all of the performance measures according to the following criteria:
Relevance: Does the measure assess an aspect of performance that is important for
mission readiness?
Observability: Does the measure assess a behavior that is truly observable?
Question wording: Does the measure make sense to other subject matter experts?
Scale type: Is the scale used appropriate for differentiating behavior?
Scale wording: Do the behavioral anchors make sense to other subject matter experts?
The third workshop presented the experts from the first workshop with a draft set behaviorally
anchored performance measures for each PI. Performance measures were reviewed to ensure that
they were seen as meaningful, relevant, and reflective of the community in which training will be
embedded. Upon completion of the third workshop, we produced a comprehensive set of
performance measures for the assessment of student behavior with respect to the principles and
practices of OBTE.
12.2.4 Phase Four: Retranslation of Measures
During this final phase, subject matter experts were asked to read each measure and come up with
key words that describe what the particular question aims to measure. This workshop ensured that
the final measures are consistent with the PI developed in Phase one. If a measure was assigned
key words that were not relevant to its associated PI, that measure was regrouped under a
different PI or dropped. In so doing, we sought to understand and document how individual
209
Marceau et al.
measures lined up with and support the principles of OBTE. In this case, this workshop was
again conducted in a group setting with some of the progenitors of OBTE from the AWG.
12.3 Product of Measure Development
The product of our measure development effort with the OBTE instructional designers yielded a
set of 31 observable measures that are focused on learner behavior. Learner behavior is not,
however, independent of the behavior of the instructor. In most cases, learner behavior is viewed
as being couple to the environment created by an instructor. Thus, we believe that instructors
should utilize these measures as an aide to establish and maintain an effective learning
environment. Overall, these measures define OBTE at a level sufficient for rigorous verification
of student learning in OBTE environments and, thus, they are useful both for program evaluation
and formative feedback to instructors and students.
More specifically, the observable performance measures cluster around four performance
indicators:
1. Learner Perception of the Teacher and Course
2. Learner Engagement in the Course
3. Learner Relationship with the Teacher
4. Learner Results and Understanding
The following sections provide detailed descriptions, sample measures, and example applications
of each PI presented by subject matter experts during discussions. A complete set of measures is
available in Appendix B.
12.3.1 Learner Perception of the Instructor and Course
The learners perception of the instructor and the course plays a critical role in the value that the
individual places on the experience. It is important that instructors exhibit a reasonable level of
expertise in the subject matter, but they do not need to know everything. Rather, instructors need
to know how best to leverage their expertise and how to find additional resources as needed for
addressing new challenges. It is critical that the learner has trust and confidence in the instructor
to be in engaged in learning and to be prepared for ongoing self-development (Darwin, 2008).
This trust and confidence is not born out of an assumption by the learner that the instructor knows
all there is to know about a topic. Rather, we believe it comes from a perception that the
instructor truly cares about the student and wants the student to understand the material. The
instructor only needs to know enough about the domain to serve as an effective agent who can set
the conditions to facilitate and guide the students learning.
Students should perceive their instructors as mentors. This perception comes from the
reinforcement of initiative and awareness in the students. Students should feel comfortable
challenging the instructors assertions, when appropriate. Students who perceive the course as
valuable and trust the instructor are more likely to take the initiative to engage the instructor
during and outside of formal periods of instruction (e.g. before class, after class, or during
breaks). This active seeking of additional interactions regarding course content with the teacher
and peers demonstrates that the student has confidence in the meaningfulness of the course and
trust in the teacher.
210

Example Measures
2. Do the students behave as if they have confidence in, and respect, for the teacher?
Students display negative

attitude towards teacher, do
not pay attention; comment
negatively to other
students; verbally challenge
teachers expertise in an
inappropriate way
Students quietly takes

notes, but do not fully
engage in discussion or
dialogue (enforced respect)
Students challenge
teachers assertions
appropriately; engage
teacher during breaks;
ask appropriate, thoughtprovoking questions
(nurtured respect)
3. Do the students engage with the teacher outside formal periods of instruction (e.g.
before/after class, during breaks)?
None of the students

engage with the teacher
Some of the students

Most or all of the student

Example Vignette:
A new teacher had been successfully instructing a course for a few months. His instructional
techniques resulted in conversations between himself and the students. Upon consistent requests
from students, who had been conditioned by past learning experiences, he decided to begin
handing out his teaching notes as study guides. Upon doing so, the teacher noticed a sharp decline
in student retention and test performance. He had not changed any of the content, yet the
performance dropped.
The teacher in the hypothetical vignette may have changed the instructional environment from
one of collaborative learning to one of memorization by handing out his notes. Once the students
had the notes, they did not feel motivated to take any further initiative toward understanding the
material. They already had the answers and they had been previously conditioned to memorize
and recall specific information in order to pass tests. This perceived norm was reinforced as soon
as they received the list of notes. Previously, when the cue was not present, the students behaved
differently. Through collaborative interaction with the instructor they were forced to think more
critically about the content in order to derive the key points. The instructor served as a guide in
this process of discovery (Dewey, 1915/2008; Hmelo-Silver, 2004). This concept of norms in
learning is important to note. Changes to the norm may induce a negative reaction in some
students who become frustrated when the teacher does not tell them what to know. However, it
211
Marceau et al.
is expected that after they begin to see that the teacher truly wants them to understand the
material, the students will become more enthusiastic about the course and instruction.
12.3.2 Learner Engagement
We believe that engagement is largely the observable behavioral manifestation of motivation in
students. Behavior such as responsiveness, alertness, attentiveness, and demonstration of
initiative are all indicative of high levels of student engagement. It is critical that the instructor
fosters engagement (Gagne, 1985). The instructor should provide an environment that encourages
and inspires students to think and participate. For example, practical applications and exercises
should require the learners to discover solutions to problems. Behavior demonstrating student
engagement should be reinforced continuously in order to ensure students understand that their
actions will make a difference. High engagement can be seen in students who try to take the next
step or advance an idea further. Such behavior shows that they are interested and desire to learn
more. One key indicator that students are engaged is when they constructively challenge an
assertion made by the instructor or another student. This shows that a student is not only attentive
but also is actively trying to understand and contrast the instruction with his or her past beliefs.
This behavior can also be seen in spontaneous side conversations in which students
collaboratively reflect on an assertion or idea posed by the instructor. These discussions should be
encouraged and even anticipated at critical points during the program of instruction.
Example Measures
1. Do the students demonstrate initiative?
Students wait for prompting

before responding with needed
action or avoid action
Students responds with

action without prompting,
but do not further dialog
with additional ideas
Students correctly respond

as needed without
prompting, and extend
concepts/actions one step
further
2. Do the students ask and answer questions?
Students asks questions on

procedures and processes to
meet course requirements, or
fail to ask questions even
when not understanding
Students ask questions for

clarification; focus on
current topic and
application
Students ask questions to

advance the idea to other
situations; apply
discussion to applications
outside of test items
212
Example Vignette:
A new instructor in a Basic Officer Leadership Course is lecturing his students. After his
introduction, he is aware that many are not paying full attention. As he continues, he realizes that
more and more of the students are losing interest. His instruction is rather dry, and he has not had
a lot of practice teaching this particular block of instruction. Students are beginning to fidget and
their body language is indicating that they are becoming disengaged (e.g. little eye contact,
slouching, etc.).
Recognizing this, the instructor chooses to use a Tactical Decision Exercise (TDE) as a way to
bring the students back into the discussion. This particular exercise requires the students to react
to a difficult and ethically challenging problem. At the end of the exercise students are asked to
discuss the course of action they would have chosen and why. Students slowly begin to volunteer
their choices. Eventually the students are all participating and highly engaged in this controversial
issue. Students are debating their approaches and rationale for their decisions.
During this time, the instructor is listening to the conversation, but he is not participating due to
his lack of experience in the chosen scenario. He is unable to reconcile the different points of
view that have emerged in the classroom. As a result, the instructor decides to stop the
conversation after a few minutes in order to return to the PowerPoint presentation that covers the
terminal and enabling learning objectives. The students halt their debate and do not engage each
other for the rest of the day.
The instructor in the above hypothetical vignette demonstrated a high level of awareness
regarding his students level of engagement during the instruction. His choice to leverage a TDE
to increase engagement suggests that he understood learner state, and wanted the students to
understand the content and learn. However, the instructor made a critical mistake by using a TDE
that was outside of his own domain of expertise and understanding. As a result he was unable to
participate effectively in the discussion. He could have played an active role in guiding the
students through the discussion had he chosen a topic for his TDE that he understood more
thoroughly or chosen one in which he could have recruited outside expertise or pointed students
toward critical resource. The instructor could have acted as a mentor and guide in the students
pursuit of the solution. By doing so, the instructor would have created an environment in which
collaborative discussion and debate was seen as an acceptable and encouraged method of coming
to know. Instead, the instructor created the opposite effect by halting the discussion and setting
the implicit precedent that this type of discussion is not appropriate.
12.3.3 Student Relationship with Teacher
It is important that students feel they can have an effect on the direction and progression of the
course. Students should not have full control of the course, but instructors should be flexible, as
appropriate, to allow students to take ownership of their learning. Students should feel
comfortable asking questions, especially when they are feeling lost or overwhelmed. Questions
posed by students, whether they serve to advance the current idea, or are seeking clarification,
should cause a noticeable effect in the classroom. This perceived change on the learners part will
create an understanding that they have some control over the classroom environment. This
perception of control is known as an internal locus of control. Individuals with internal loci of
213
Marceau et al.
control are more likely to exhibit high levels of motivation to learn because they more likely to
accept feedback and take action to correct performance problems (e.g., Goldstein & Ford, 2002;
Guay, Ratelle, & Chanal, 2008).
Additionally, instructors should act as mentors as the students mentally struggle toward an
understanding of the training content or problem. It is important to note that struggling, in this
sense, describes a willing drive to overcome an obstacle or problem on the behalf of the student.
This drive may be manifested as motivation and engagement in the learner. Instructors should
support students on their paths toward discovering the solution by asking key questions designed
to stimulate further thought, providing reminders of learning points when needed, and giving
specific suggestions on paths toward solutions if needed. Instructors should also encourage
students to relate training to their past experiences and ask questions when they are struggling.
Questions like What if we did it this way or What about this situation show that students
are aware and are relating training content to their past experiences and assumptions. Once
students have discovered the answer or solution, instructors should ensure that students are able
to articulate their thought processes. Understanding where they were momentarily wrong or right
will further increase their understanding.
Example Measures
1. Do the students take initiative to catch up when feeling overwhelmed or left behind?
Students do not speak up and

ask for help; do not respond to
teacher queries
Students do not speak up,

but respond to teacher
when queried
Students proactively speak

up and ask questions when
falling behind or confused
2. Do the students struggle to discover a solution and solve unique problems?
Students execute standard

procedures even when not
appropriate for problem; fail
to solve problem
Students make some

progress on unique
problem, but fail to
effectively solve the
problem
Students adapt training to

discover an effective
solution to a unique
problem
214
Example Vignette:
An experienced instructor posed a unique problem to his students. The students then began to
discuss and propose potential solutions to each other. The instructor, who had facilitated this
exercise several times before, monitored the discussion and participated as needed to keep
students from going too far down the wrong road. The students struggled mentally with the
problem but enjoyed the challenge and were highly engaged. As expected, the students eventually
discovered the solution through trial, error, and collaboration.
Next, the instructor facilitated a discussion to elicit explanations about how the students figured
out the answer. The students began to describe a decision-making process and method of
investigation which was consistent with the instructors expectations. However, one student, who
had come to the same solution, proposed a completely different path to discovery. The instructor,
after listening to his explanation, had to pause for a moment. He had never thought about the
problem in this particular way before. After thinking about it from the students unique
perspective, the instructor told the student that she also was correct. At this point the instructor
changed the focus of the discussion to a conversation about the two different, but equally correct
approaches.
Engaged students are continuously relating course content to past experiences. The hypothetical
vignette above describes a case in which one students past experiences allowed her to approach a
problem in a very unique way. That student felt confident and comfortable enough in his learning
environment to challenge what his instructor and peers were proposing as the one method for
arriving at the solution. The instructor, having recognized that the student was also correct,
reinforced the students initiative and validated her unique approach. The students perspective on
the problem served to enrich the discussion and shape the direction of the course. The other
students benefited from this discussion through exposure to the diversity of thought.
The instructors desired outcome for the exercise was to allow his students to build confidence in
their ability to solve difficult problems. He achieved this outcome and also leveraged the
unexpected answer to increase awareness regarding problem solving and to reinforce initiative.
12.3.4 Student Results
Student retention of factual information is important but this alone does not create knowledge.
Scoring 100% on a multiple choice test only shows that the students remembered enough
information long enough to recall it and use it on the test. It is possible that much of this inert
information will not be retained after training.
Outcomes for training should include student learning, awareness, understanding and
development. Students best demonstrate knowledge and awareness through the application of
learned information to unique problems or contexts, rather than through mere reproduction of
isolated facts on tests. Hence, instructors should provide opportunities for students to apply
information in situations that are different from those used as exemplars during instruction
(Tobias & Duffy, 2009). Through these opportunities, students will have the chance to integrate
lessons learned with their own unique knowledge to achieve individual solutions. The presence of
215
Marceau et al.
these opportunities should have a positive effect on transfer by allowing learners to relate training
to past knowledge as well as to additional contexts, ideas, and environments.
Example Measures
1. Do the students retain and apply factual information (demonstrate a technical and contextual
understanding)?
Students do not retain

information (e.g., cannot
answer question)
Students retain information

(e.g., give textbook
answer)
Students retain and apply

information to create
knowledge (e.g., show
application to mission,
technical and contextual
understanding)
2. Do the students apply lessons learned to improve performance?
Students do not apply lessons

learned to change behavior;
continue to make same errors
Students change behavior

in accordance with lessons
learned
Students integrate lessons

learned with existing
behavioral strengths to
yield optimized individual
approach
Example Vignette:
A Private, while learning how to assemble and disassemble his weapon, remembers a slightly
different method his father had taught him when he was younger. The instructor allows the
Private to try this method because he knows it is a safe and acceptable alternative. The Private
integrates the new method with his past knowledge to come up with an alternative successful
procedure. The Private is then able to safely assemble and disassemble his weapon faster than the
other Privates.
Further along in training the Privates are learning how to assemble and disassemble another
weapon. A second Private, while trying to take a shortcut, unknowingly puts himself into a very
dangerous situation by performing steps out of order. The Drill Instructor recognizes the danger
and immediately stops the Private from injuring himself and explains what could have happened
and why it was unsafe. Recognizing his error, the Private follows the standard procedure next
time to successfully disassemble and assemble the weapon.
216
The hypothetical vignette above describes two situations in which a soldier acquired knowledge.
In both cases, the end result was a soldier who is able to complete the task to standard (e.g.
assemble and disassemble his weapon). The first private became more knowledgeable about his
weapon by applying past knowledge resulting in a positive result (e.g. performing the task more
efficiently). The second private experienced a negative result (e.g. put himself in an unsafe
situation) but still acquired knowledge. As a result of the experience, the second Private became
more aware of why the process exists.
Instructors should always allow students to take ownership of their learning and, with some
guidance and constraints, discover solutions to problems. Small failures can lead to success and
overall positive learning results. Conversely, too many failures can be discouraging, hurt
confidence, and have an overall negative effect on learning. It is important that instructors find an
appropriate balance between allowing students to struggle and providing guidance when
necessary.
12.3.5 Self-Report Measures
In addition to the observer-based measures described above, much information can be gained by
asking students about their impressions across numerous areas ranging from what they believe
they learned to impressions regarding instructors. Hence, although in the sections above we focus
on observable measures, our process also developed a set of self report measures as a tool for
assessing students perceptions. The measures are related to items such student perceptions of the
instructor and course, perceived value of the course, and level of perceived individual
development. These measures are useful to the instructor in determining if the desired learning
environment is being created or if in-stride adjustments are needed. Additionally, they can help
the student become more aware of the learning objectives by making the teachers goals
transparent to the learner.
Example Self-report measures:
1. Do you have trust in your teacher to develop your individual skill set?
I dont think the teacher

cares if I do well or not
I feel that the teacher is

interested in my success in
this course

genuinely interested in
my success in my
mission
2. Do you feel that you are a better problem solver as a result of your last practice exercise?
Not at all; no different than

before I participated in the
exercise.
Yes, but only within the

context of this course
I feel I am a better
problem solver overall as
an individual
217
Marceau et al.
It is important to note that an instructor should utilize self-report measures at multiple times
throughout her courses. While such feedback at the end can be helpful, feedback during the
course can help guide real-time adaptations that enable the course to be more student-centered. In
addition, these measures do not necessarily need to be administered in paper survey form; they
also could be used to frame a discussion. Of course, these self-report measures can be tailored to
the learning objectives of the course or particular block of instruction to provide the instructor
with detailed information regarding student learning. The overall point of these measures,
however, is to complement observer-based measures of student behavior with self-report
measures that can help instructors triangulate on student attitudes. Multiple kinds and sources of
feedback can minimize the risk of misinterpretation and help determine the urgency of an instride modification or deviation from a plan.
12.4 Conclusion
The student measures presented in Appendix B, and summarized in this chapter, include both
self-report and observer-based items that relate to key themes such as learner perception of the
teacher and the course, learner engagement, learner relationship with the teacher, and learner
understanding. Taken together, they capture key elements of student behavior that are useful for
guiding instruction. However, not all of the observable performance measures and self-report
measures are ideally suited for every program of instruction. Instead, instructors and observers
are encouraged to refine or adapt these measures to construct their own assessment tools that are
focused on their desired outcomes and observable behavior within their training environments.
Such adaptations will be consistent with OBTE if (a) they take into account long-term results and
developmental outcomes, and (b) there is careful documentation of the changes and the reason for
them. In doing so, instructors and observers will have demonstrated both initiative and
accountability in taking ownership of quality assurance for the course.
As models for key elements of assessment, the measures described in this chapter serve essential
objectives. First, along with measures of instructors (chapter 2), they define OBTE based on a
verifiable set of expected actions and observable behavior. These measures can therefore be
useful for course designers and instructors who have the desire to adopt an outcomes-based
approach. Second, they can provide feedback to course designers and instructors based on
observable learning and behavior of students. Hence, an instructor or observer can use these
measures to gauge the level of learning taking place in a field or classroom environment. This
pre-planned feedback can guide course refinement or real-time adjustments. Finally, they can be
useful for providing feedback to students related to course-specific learning objectives and
longer-term developmental outcomes as in a progress map (Pellegrino, et al., 2001). The
measures provide observable behavior that can be used to gauge student state, not relative to other
students but relative to the students further development. Accordingly, they represent a key tool
in creation of an assessment-centered environment in which assessment is an integral part of
learning.
12.5 References
Bransford, J., Brown, A., & Cocking, R. (2000). How people learn: Brain, mind, experience, &
school. Washington, DC: National Academy Press.
Cornell-dEchert, B. (2009). Outcomes-based training and education: Implementation guide. Ft.
218
Darwin, M. (2008). Outcomes-based training and education: fostering adaptability in full

spectrum operations (Briefing, December 2008). Ft. Meade, VA: Asymmetric Warfare
Group.
Gagne, R. (1985). The Conditions of Learning (4th ed.). New York: Holt, Rinehart & Winston.
Goldstein, L., & Ford, K. (2002). Training in organizations: Needs assessment, development, and
evaluation. Belmont, CA: Wadsworth Thomson Learning.
Routledge
219
Artis et al.
Chapter 13. Adapting OBTE in a Classroom Environment
Sharnnia Artis, Camilla Knott, Kristy Reynolds, Fred Diedrich
Aptima, Inc.
Rob Semmens
Imprimis, Inc.
13.1 Intent
Outcomes-based Training and Education (OBTE) systematically seeks to engage an organization
to shape instruction and influence the overall development of an individual, as a Soldier, with
respect to broad outcomes that transcend course-specific objectives. Given the challenges of fullspectrum warfare, OBTE strives to develop Soldiers who can think and behave adaptively amid
uncertainty, Soldiers who can learn in any situation, and who continuously improve as individuals
and as members of a unit (Cornell-dEchert, 2009; see Chapter 1). Accordingly, OBTE seeks to
enable mission effectiveness through application of a variety of core principles related to training
and education that are designed to facilitate the growth of key abilities (see Chapters 2 and 3): (a)
leadership exemplified by instructors as role models and mentors that, among other things,
promotes enculturation of Soldiers with respect to Warrior Ethos and Army Values; (b) an
integrated understanding of basic Soldier skills that underpin the ability to move, shoot, and
communicate, that are necessary for individual adaptability and collective agility, and that relate
to the why, what, when, where, and how of actions that can have irrevocable consequences; (c) a
mindset of collaborative reflection and problem solving that develops skills and habits of self
assessment, peer assessment, and situation assessment in task-organized teams; and (d) Soldier
motivation and structured self-determination consistent with military order that fosters the longterm development of attributes such as confidence, initiative, and accountability.
In this chapter, we explore the application of OBTE to a classroom environment, in order to
complement and extend prior results focused on field-based instructional settings (see Chapters 8
and 9). In particular, we focus on the role of assessments that employ self-report measures as
well as measures of observable behavior (see, e.g., Chapter 2 and 12). Our objectives were (a) to
demonstrate that best practices of OBTE could be verified through measures that can be applied
in the classroom, and (b) to assess behavior of instructors and students and provide actionable
feedback to support student learning and improvement of instruction with respect to the principles
and practices of OBTE (see Chapters 1 and 11; see also, Pellegrino, Chudowsky, & Glaser,
2001). Although we focus on the measures developed and used to measure instructor and student
behavior in the current application, we believe that our methods and findings have implications
for Army training and education in general.
13.2 Observing OBTE in the Classroom Environment
13.2.1. Participants
A team of two researchers observed an initial pilot application of OBTE in a classroom
environment at Fort Sill (Lawton, OK). This course focused on familiarizing students with
several teaching methodologies including but not limited to OBTE. The research team observed
five instructors and four students. Of the four students, two were active-duty Army personnel and
two were members of the research team. The embedded students were selected as participants in
Application to Classroom Environment
220
the course to provide the researchers a first-hand perspective on what it is like being a student in
classroom environment in which there is an attempt to apply OBTE.
13.2.2. Procedure
The researchers made all observations in two-person teams in order to capture the dynamics
between the instructor and the students. Both instructor behavior and student behavior was
captured by members of the research team. Two researchers observed each instructor. Each
researcher also observed two of the students, throughout the two-week period. In addition to the
observations, all of the students completed the self-report and NASA Task Load Index (TLX)
questionnaires (Hart & Staveland, 1988). The embedded students from the research team
completed a third questionnaire, the 360 assessment.
13.2.3. Measures
During the two-week course, the research team assessed the effectiveness of OBTE by utilizing a
combination of observer-based measures and self-report measures. Observer-based measures
were employed to assess instructor behavior and student behavior (see Chapters 2 and 12). These
measures refer to behavior that can be observed by an external observer. Only the students used
self-report measures. We assumed that, in most cases, student and structure measures are related
to the extent that student behavior is coupled to the environment created by an instructor. Thus,
we believe that instructors can utilize both the student and instructor measures as an aide to
establish and maintain a positive learning environment. They also can be used for program
evaluation and formative feedback to instructors and students (Kirkpatrick, 1994). We did not
conduct statistical tests due to small the sample size observed. Our intent was to explore
application of measures in a classroom environment to determine what might be observable, what
might be missing, and how the measures might be improved to prepare for more rigorous
verification and validation of such applications in the future (see, e.g., Chapter 1, section 1.3.2).
13.3 Utility of OBTE Measures in a Classroom Environment
13.3.1 Generality of Measures
The application of OBTE in the classroom can benefit from an environment that is learnercentered, knowledge-centered, assessment-centered, and community-centered (Bransford, Brown,
& Cocking, 2000). A learner-centered environment takes into account an individual students
momentary capabilities for knowing, for coming to know, and for being motivated to participate
in learning. To be knowledge-centered, effective classroom environments provide opportunities
for sense-making rather than mere memorization. Such opportunities are created through
consideration of prior knowledge and future with which students will try to integrate with the
current learning experience. Learning should be considered in the context of a broader curriculum
and even the continuing self-development of students. To be assessment-centered, the learning
environment includes formative feedback to support learning. Assessment is part of learning not a
conclusion about how much learned. A community-centered learning environment encourages
exploration in an open environment that is relevant to, and reflects, the community in which it is
embedded.
A learning environment with all these characteristics also should be sufficiently adaptable and
flexible to provide learners with the right amount of guidance at the right time to achieve both the
learning objective for the event and to have a positive effect on longer-term developmental
outcomes such as the growth of confidence, initiative and accountability. All learning
221
Artis et al.
environments have an influence on such outcomes whether or not they are explicitly considered
and whether or not they are part of the explicit intent of a learning event (Chapter 3). The
framework of Bransford et al. guided us in the COMPASS process for developing measures of
instructor behavior and student behavior (see Chapters 2, 7, and 12). This helped us develop
measures that, while grounded in the verifiable experience of the subject matter experts, were
more general than the specific learning events explicitly addressed in the COMPASS process.
During the two-week observation, we verified that many of the observer-based measures of
students and instructors were applicable to the classroom environment at Fort Sill and could be
observed easily. Out of 75 student and instructor observable measures, 54 measures (72%) were
verified in the sense of being relevant and observable in the classroom environment; that is, we
were able to collect data with using these measures. For the student measures, 21 of 31 measures
(83.9%) were verified. For the instructor measures, 33 of 44 measures (75%) were verified.
Twenty-one of the measures were not employed because they were either not applicable to the
classroom environment or not observable during particular blocks of instruction. Six of the
measures were not applicable, and fifteen were not observable.
13.3.2. Implications for Improvement of Measures
Observer-based measures that were not directly relevant to classroom instruction tended to relate
to safety (e.g., like that on a rifle range) or introducing stress in the learning environment. In other
cases, the measure or anchors were too specific or otherwise did not leave any leeway for
observations outside of the specific conditions. Examples of non-applicable student and instructor
measures are shown below.
Example of student behavior that was not applicable:
Do the students demonstrate the ability to perform under highly stressful situations?
Students fail to
perform in stressful or
dynamic situations
Students function with

marginal performance in
stressful or dynamic
situations (arrive at
solution, but in non-optimal
way, or very slowly)
Example of instructor behavior that was not applicable:

action in stressful or
dynamic situations to
arrive at solution
Does the instructor use safety as a training enabler?

real purpose

the training events, but not
as a combat and training
enabler (i.e. less restrictive)
combat and training
awareness allows for more
complex scenarios)
222
The generality of measures, as a whole, can be improved by including course instructors and
designers in a refinement process similar to that used in development in the original development
of the measures (MacMillan et al., in press; Chapter 2). The collaboration with subject matter
experts, consistent with this methodology, results in the identification of specific behavior that
can be observed in a particular setting and that is diagnostic of performance. The collaborative
selection, refinement, and exclusions of measures for a particular application also can be a
powerful technique within an organization to assist instructors and stakeholders in taking
ownership of quality assurance and continual improvement (Appendix C).
13.3.3 Implications for improvement of course design
The measures that were not easily observable in the classroom setting tended to relate to training
to increase understanding and awareness or training to increase deliberate thought. A few of these
non-observable measures were developed to assess whether or not the students understood the
task as it relates to other tasks and its mission application. Examples of these non-observable
instructor and students measures are below.
Example of student behavior that was not observable:
Do the instructors focus the why of training back to the relationship between
the individual and big picture/mission?

on why the Soldiers
were trained
Facilitates discussion on the

big picture; less focus on
the individuals
contribution
the big picture; focuses on
the importance of
individuals
Example of instructor behavior that was not observable:

Does the instructor recognize when a Soldier is too withdrawn/distracted to
effectively participate in training?
Does not recognize a

deeper individual
problem when Soldier
is struggling with a
task
Sees that there is a deeper

problem, but doesnt do
anything (doesnt realize he
CAN do something)
Recognizes there is a
deeper problem, takes
appropriate action to help
Soldier get back in the
game
These measures were not readily applicable to the course because the learning environments did
not consist of exercises that required the students to understand the instruction and its mission
application. As a result, the students were not able to exercise a deliberate thought process to
223
Artis et al.
evaluate the instruction and how the skills are necessary for the success of their missions. An
implication of this is that a change in the course might be more influential than a change in the
measure (Appendix C). For example, course designers might consider including learning events
that further challenge students, thereby stimulating more complex thought and problem solving.
Mission context and the associated linkage of tasks is a nonarbitrary way to introduce such
complexity.
The instructor would have to take care in planning and be vigilant in execution not to introduce
too much complexity too fast. Thus, course designers and instructors would have to utilize the
practice of scaffolding in proceeding from relatively simple learning events to relatively complex
events. Arguably this should be a consideration in any course (see e.g., Merrienboer & Kirshner,
2007; Tobias & Duffy, 2009). In any case, the measures cited above tend to focus the attention of
course designers and instructors on issues pertaining to the introduction of complexity into a
course, the ways that it challenges students, and the potential it has for revealing their zone of
proximal development (Cole, John-Steiner, Scribner, & Souberman, 1978).
The introduction of mission context and task linkage into a course requires that instructors have
sufficient contextual expertise to use this context to challenge and observe students and to adjust
instruction accordingly. Without this level of expertise, it can be difficult for instructors to foster
an environment is sufficiently rich to engage students and to motivate them and to avoid
overwhelming or frustrating them. Instructors bring varied experiences to course. Their
knowledge, skills, and abilities relative to any particular group of students may be difficult to
predict, especially in a course that is more about teaching how to teach than about teaching
particular content. Thus, it may not be prudent to specify a particular mission context or
operational problem for learning events in designing a course. A better approach, consistent with
OBTE, is to provide guidelines to instructors about using context to manipulate complexity and to
provide license to them in identifying the context. This way, instructors can leverage their
particular expertise to make instruction more effective. As suggested above, the inability to use a
measure can reveal an opportunity rather than a problem. The opportunities will be important to
the extent the measures are relevant to long-term outcomes and preparing Soldiers for FSO
(Appendix C).
13.4 Use of 360 Reviews for Collaborative Reflection
13.4.1 The Role of a 360 Review in OBTE
Members of the research team were embedded as participant observers in the course. The focus
for their observations was collaboratively to identify and reflect on opportunities to incorporate
OBTE within and across learning events. As it has been defined elsewhere (see Chapter 3),
OBTE is more general than a particular method of instruction or a particular set of instructional
techniques. It can guide the selection and adaptation of instructional methods to foster
development of the individual in relation to basic needs as well as the associated cultural values
and objectives of the individuals occupation. In particular, immediate learning and long-term
development are considered to be complementary objectives when leaders allow subordinates
reasonable autonomy to exercise initiative. This requires a climate of accountability that both
guides and constrains initiative (see Chapter 3; Appendix C). Accountability is fostered by mutual
trust between superiors and subordinates, but also between peers. Consequently, in the classroom,
OBTE has implications for instructors and for students as learners and peers.
It is incumbent on the instructor to engage the student in learning and to be accountable for it.
Addressing motivation is an essential part of any strategy to engage students (Colquitt, LePine &
224
Noe, 2000). Motivation can be increased by creating a learner-centered environment in which

there is opportunity to take initiative, demonstrate accountability, and develop confidence (Guay,
Ratelle, & Chanal, 2008). Confidence is gained by providing context that helps the student
integrate knowledge across learning events as in a knowledge-centered environment (Bransford et
al., 2000), that is, by revealing relevant prior knowledge, future applications, and thus broader
competence. In the context of teacher education it is important for the cadre of instructors can
point out how individual events motivate students and relate to a broader context. This can be
done explicitly by telling and by drawing it out of students. A 360 review is a method of
integrating these different sources of insight.
In the 360 review, individuals received feedback from superiors and peers (also from anyone
acting in a subordinate role, if available). In organizations, it is commonly the case that an
individual is both a reviewer and receiver of reviews (Wohlers & Hallagher, 1990).
Understanding the perspective of a reviewer can help the receiver understand a review more
deeply. Similarly, understanding the perspective of a receiver can help one do a better review.
The purpose of these multiple perspectives is to increase everyones awareness of areas for
personal growth. Properly designed questionnaires can help leaders compare their selfperceptions with the observations of colleagues or others who know them well. However, there is
some controversy as to whether this feedback actually improves performance (Pfau & Kay,
2002). In one study, Bailey & Austin, (2006) unexpectedly found that the focal individual's
initial self-assessment predicted changes in self-efficacy better than the favorability of ratings
received. London, et al. (1990) proposed that the best use of 360 feedback was to identify gaps
between what the receiver feels and what the audience observed. If done poorly, 360 reviews can
negatively impact on the development of intangible attributes related to self-efficacy. But what
does it mean to do a 360 review well?
In OBTE, a 360 review is viewed in the context of a formative assessment. The structure of
questionnaires or rating scales should provide clear indications about what to do differently and,
longitudinally, should reveal progress toward such milestones (cf., Pellegrino et al., 2001). Thus,
the assessment is an integral part of learning, as in an assessment-centered learning environment,
not merely a conclusion about learning. The specific items should be clearly related to the
developmental outcomes for the course. In other words, the feedback should indicate the ways in
which learning will have an impact on downstream performance outside of the learning
environment. This makes the learning relevant to the student as in a community-centered learning
environment (Bransford et al., 2000). While open-ended comments have some value, most of the
360 review should utilize a carefully structured format developed in a rigorous process that
includes subject matter experts and stakeholders (see e.g., Chapter 2). Absent this, verification
and validation of the review and its impact would be complex and unacceptably ad hoc, at best.
The typical intent of 360 reviews is entirely consistent with a fundamental principle of OBTE
a mindset of collaborative reflection and problem solving (see Chapters 2 and 3). In particular,
both recognize the importance of multiple perspectives on a shared experience, including selfassessment and peer-assessment as well assessment by superiors and subordinates. The point of
such assessments is to learn. At their best, they are not isolated events. They are a longitudinal
process of coming to know. AAR are more formal instantiations of collaborative reflection but
such learning occurs whenever individuals pause to reflect on lessons learned in a consequential
event. Whether planned or not, whether formal or not, collaborative reflection can benefit from a
framework for observations that are actionable and that reflect the priorities of Full Spectrum
Operations. The measures developed for instructors (Chapter 2) and for students (Chapter 12)
provide a rigorous framework for such assessment that can be a valuable tool in 360 reviews.
225
Artis et al.
13.4.2 Narrative of a Participant Observer

As mentioned in the introduction, the research team was able to embed two participant-observers
as students in the course. The following are the observations one student:
From discussions with the course designers prior to the course, we interpreted the purpose
of the 360 feedback to be multifaceted. The 360 feedback was intended to provide the
reviewer with multiple perspectives, in accordance with the traditional intent of this
method, as well as provide an opportunity for the student instructors to practice counseling
skills with each other. Further, the instructor would be able to be a role model for proper
counseling behavior in his provision of feedback.
During the course, however, the purpose of the 360 feedback was not clearly communicated to
the students. While the instructor made the case for multiple sources of feedback in general (e.g.
corporations use this to help them get better), it was not clear how it was immediately relevant to
the students. The primary method of implementing the 360 feedback was a form referred to as a
blue card. The blue cards were intended to allow other students to provide feedback quickly
and easily. The instructors wanted to balance the need to provide feedback against requiring the
students to complete lengthy assessments that would be viewed as tedious by the students.
The original blue card had four spaces for narrative comments. Those sections included
strengths, weaknesses, improvements, and general comments. The other side of the form
contained Likert scales for organization, content knowledge, visuals and media, interaction with
students, delivery, method of instruction, and instructional outcomes. The behavioral anchors
associated with this form included phrases such as:
Instructor did a poor job presenting information in a logical sequence.
Instructor presents information in a somewhat logical sequence.
Instructor presents information in a logical interesting sequence which the audience can
follow.
One student raised an objection to the form: the blue card is ok, but it doesnt tell you how to fix
the problem. For that, the student-instructor would have to rely on the narrative comments.
This turned out to be the case as I (the participant observer) personally experienced later in the
course. On one feedback form, I received a rating that fell between the second and third box. It
was clear based on the comments made in the narrative part of the form that the observer did not
understand my presentation; however, the observer had been prompted to fill out this form
independent of his ability to assess my performance. The fact that the observer did not understand
my presentation was a data point for me to gauge my effectiveness as an instructor but the utility
of this observation was severely limited by the absence of additional feedback to guide
improvement.
After a round of observations with the blue cards, the instructor team switched to a different form
that had a 1-10 scale system instead of a scale with behavioral anchors. This shifted the feedback
from a formative evaluation to a summative evaluation, which was further emphasized by the
requirement to provide an Overall Assessment. Moreover, the space for narrative comments
was divided in a manner that promoted short bullet points instead of longer, more detailed
comments, and the categories were changed.
Indeed, the new form caused some confusion. The students had become familiar with the blue
card, even though the scale of the behavioral anchors did not closely reflect what was occurring
226
in the student-led instruction. For example, none of the students presented information in an
entirely illogical sequence. Students ranged from somewhat logical to logical. They might
have had one or two slides out of order, but it was difficult to indicate that on the form.
Nonetheless, the students were beginning to become familiar with the different categories. On the
new form, there was a distinction between Instructor Presentation Skills, and Facilitation
Skills, yet that distinction was not made explicit to the student. It was difficult to know how to
change. From a formative perspective, the effect of an assessment is only as good as its ability to
convey how, what, and why to change.
Moreover, the impact of this now summative evaluation was immediate. Some of the feedback
received from other students was poor and reflected little effort on their part. Below is an
example of the vague feedback form I received:
Instructor Presentation Skills
Strengths
Weaknesses
good slides
know the slides
Confidence / Eye Contact /

Voice / Positive / Motivating /
Mannerisms / Posture /
Distracters / Control of Students
/ Adaptable to Changes /
Problems / Knows the lesson
plan
This offered little in terms of how I could improve as an instructor. I developed all of the slides
for the class that I taught, so I felt comfortable with the content of each slide. Thus, I thought that
perhaps my transitions between slides had been poor. In light of the feedback that I received
from other students, however, the path to improvement grew murkier. Two other students rated
me as a 10 in this area, with no mention of the slides at all. It was becoming clear that the scales
developed for rating the different areas were of marginal value. Students did not know which
behavior resulted in which score. The results of two rounds of student feedback are displayed
below (Figure 1).
Figure 1. Results of a summative evaluation. Peer ratings plotted as a function of categories on a

summative evaluation form for the first presentation (30 min) and the second presentation (40
min).
As the student instructor and recipient of this feedback, looking across the ratings provided by my
fellow students, I am not sure of how to improve or if I need to improve. A 10.2 on a 10-point
227
Artis et al.
scale is pretty good, which suggests that I might not need to change anything. Further, my first
presentation (the 30 minute presentation) was ranked higher than my second presentation. I was
attempting a new technique in the second presentation that did not work as well as I had hoped,
so I somewhat expected the lower ratings. Nevertheless, based on the resulting ratings from the
summative feedback, I felt discouraged from experimenting with new methods that could
potentially be an improvement to current methods. The nonspecific feedback from lower scores
led me to use only the techniques with which I was familiar.
As an observer of other students, I did not want to give them low scores, as it could have caused
the recipient to become defensive and discount my comments (Argyris, 1991). Further, because I
did not know what distinguished a 6 from a 7, I keep my comments around 8, with really good
behavior scoring a 9 or a 10, and behavior that needed improvement scoring a 7. This technique
allowed the receiver to gain some indication of his or her level of performance, that is, feedback
about what he or she did well or what should improve based on comparing across the other scores
that I provided. I was trying to make the scale into something that was more useful than it was.
The point here is that a scale intended to be simple and expedient will require a significant
investment of time and thought by a reviewer who is genuinely interested in giving useful
feedback to the student and, without such an investment, a simple and expedient scale is likely to
be useless.
In discussion with the other participant observer, we concluded that the summative form had little
value for directing improvement. The short narrative comments did not offer enough specificity.
Because we both wanted real feedback for improvement, we determined that a real-time freeform commentary of the students performance would be more appropriate. In addition, it was
easier and less time consuming to write down comments as the student instructor presented his or
her topic. If he did something well, we could note it, if he did something poorly, we could note
specifically what confused us. With either approach, however, the quality and utility of such
comments is entirely dependent on ability of an individual reviewer to innovate in the moment
and without guidance. One cannot expect much even from the most experience reviewers under
such constraints.
13.4.3 Recommendations for Further Use of the 360 Review
As mentioned in Chapter 2 (section 2.5.1), there are three aspects of a good approach to using
formative feedback for instructors: (a) it should relate to the intent of instruction, (b) it should
inform actions that can be taken to improve instruction with respect to the intent, and (c) it should
reflect the instructors span of control. Further, formative assessments can help inform students
who are observing a student-instructor about desired behavior.
From the perspective of the participant observers, one of the greatest benefits of this course was
that students in the course were able to participate in various forums for discussion and feedback
that allowed them to gain insight into their own strengths and weaknesses as instructors. In
addition, the format enabled the students to apply what they learned in the first week of
instruction during student presentations in the second week of the course. The practical
experience gained by the students in applying OBTE, for example, during their presentations will
help them in applying the principles of OBTE as instructors. Students seemed to benefit more
from the practical experience gained during the course than through the 360 review which, in
essence, was a summative assessment. Instructors should use formative assessments to raise the
students awareness of what good instruction looks like. The assessments could be used to
provide students with a model of effective instruction through appropriately targeted assessment
questions and behavioral anchors.
228
It is important to note that the behavioral anchors are essential to illustrate what right looks like.
Although a community of experts typically develops the behavioral anchors, students should not
accept them uncritically or without understanding. Knowles (1975) has argued that motivation to
learn becomes internal as a person matures. This suggests that motivation to learn via formative
assessment may increase when the expectations are explained to students, put into context, or
otherwise integrated with prior knowledge. Collaborative reflection that utilizes formative
assessments can help achieve this deeper understanding because the assessment items and
associated anchors are connected with aspects of a learning event, to help learn lessons from the
event. Formative assessments thus gain from collaborative reflection through integration of
knowledge as collaborative reflection gains from formative assessments.
The measures fundamentally change the conversation in an AAR or any other method of
collaborative reflection. Instead of simple platitudes such as "you did a great job" or "I like what
you did," or their derogatory opposites, the conversation would be more likely include actionable
observations such as "this is what you did well" or "it was valuable that you did..." The former
might lead to a momentary motivation, while the latter can lead to a more persistent motivation
by either validating one's own observations or redirecting one's attention to a relatively neglected
area of personal competence. Choices made about the use of measures can provide insight into
why the user feels confident or not. Changes in these choices over time can provide insight into
the multifaceted development of confidence (i.e., beyond simply more or less).
13.5 Learning, cognitive load and motivation
Learning depends on a variety of interrelated factors such as cognitive workload, motivation, and
goal selection. Workload, which can be characterized by mental effort, has been shown to affect
performance in various environments (Kahneman, 1973; Wickens, 1991) and can affect learning
and motivation. Previous work has shown that increased workload can hinder the successful
completion of tasks or activities involved in learning (Gaillard, 2001; 2008). On the other hand,
excessively low levels of workload can lead to boredom and fatigue and thereby also hinder
learning. Motivation can increase or decrease depending on a number of factors including the
learners goals and approach in pursuing those goals (Dweck & Legget, 1988). As a consequence
of goal definition by the learner, the level of engagement can vary along with motivation and
effort. During a learning event, a certain amount of mental effort must be voluntarily mobilized
in order to acquire a new set of skills (Gaillard, 2008), and this is influenced by various factors
including motivation (e.g., Veltman & Gaillard, 1997).
During our observation of a classroom application of OBTE, the research team measured student
workload using a questionnaire. Although this is somewhat limited given the complex
interactions among variables that affect learning and motivation, the subjective measure we
applied allows for the analysis of multiple dimensions affecting workload including frustration
and perceived performance. Measures of student and instructor behavior, along with the workload
measures, can provide insight into learning, level of engagement, motivation and perhaps even
learner goals.
13.5.1 The NASA Task Load Index as a subjective measure of student workload
The NASA Task Load Index (NASA TLX; Hart & Staveland, 1988) is a subjective measure of
workload commonly used in applied science settings such as studies of air traffic management,
aviation automation and human computer interaction. It has been shown to reliably capture
variations in workload based on task demands. The questionnaire is designed to measure how
229
Artis et al.
well cognitive resources are matched with demands of a task. It is a multidimensional scale
comprised of six subscales each designed to capture various dimensions of workload related to
cognitive resources and demands of a task. The dimensions include: level of effort, frustration,
perceived performance, mental demand, physical demand, temporal demand. The former three
subscales are associated with the cognitive resources and the latter three with task demands.
Participants are asked to provide a rating for each of the subscales based on their perception of
task demands (mental, physical, and temporal), the level of effort they exerted to meet those
demands, frustration they experienced during the task, and their performance of the task. The six
subscales are not linearly related. Most findings reflect that perception of performance declines as
task demands increase (i.e., temporal, mental, and physical demands of the task). In cases where
task demands are excessively low, performance can decline. In contrast, frustration and level of
effort is typically shown to increase with task demands and also when task demands are
excessively low (discussed in Wickens, Gordon, & Liu, 1998). Performance is likely to be
sustained at ideal levels workload is optimal, that is, when task demands are neither excessively
high nor low so that cognitive resources can appropriately match those demands (Wickens, 1992).
In the case of our observation of OBTE in the classroom, the demands of the students task were
related to learning, and our hypothesis was that optimal workload would support student
motivation and maximize learning. The NASA TLX was used to capture the subjective overall
effect of each days lessons on student workload. At the end of the day, the students were asked
to think about the class that day and provide a rating between 1 and 20 (low to high) for each of
the six dimensions or subscales. Students were asked to provide an average rating on each
dimension because workload was assumed to vary throughout the day.
13.5.2 Results
Analysis of the TLX data focused on the perceived mental demands of the days lesson, level of
effort, frustration experienced, and perceived performance during the day (Figure 2). The
Temporal and Physical demands subscales did not vary across instruction days and were
determined to minimally reflect the effects of instruction on workload and were excluded from
the analysis of the data. Analysis of the remaining four scales showed that mental demands,
effort, and performance generally were rated lower during the first week of instruction compared
to the second week, but frustration was rated higher during the first week compared to the second
week of instruction.
Figure 2. NASA TLX subscale ratings plotted as a function of instruction day.

230
Visual inspection of Figure 2 shows that students perceived their performance to vary directly
with effort and mental demands, such that performance was perceived to decline with lower effort
and mental demands, and improve with higher effort and mental demands. In contrast, students
reported their level of frustration to be directly related with mental demands and effort only
during the first week of instruction (Day 1 to Day 5), but inversely related to effort and mental
demands in the second week (Day 6 to Day 9), such that frustration decreased as effort and
mental demands increased
13.5.3 Implications
During our observation of the application of OBTE in the classroom, we hypothesized that
optimal levels of cognitive workload would support student motivation and maximize learning.
Specifically, we aimed to use student ratings on the NASA TLX to (a) measure workload through
ratings of mental demands of a task and (b) infer engagement and motivation through the
interactions among frustration, mental demands, and effort exerted to meet those demands. To
assess learning, however, observer-based and self-report measures must be assessed in
conjunction with data from the NASA TLX.
Cognitive workload is modulated by various factors related to task demands (measured by
temporal, mental, and physical demands subscales) and cognitive resources (measured by effort,
frustration, and performance subscales). The Mental Demands subscale was the most relevant of
the task demands subscales and most consistent with the putative effects of workload on learning.
Therefore, the Mental Demands subscale was used as the metric for modulation of workload by
task demands. Although the Physical and Temporal Demands subscales were excluded in the
analysis of the NASA TLX data, based on the research teams observations, both subscales
appropriately captured the classroom environment by showing that students did not feel time
pressure in learning the days lesson, but that lessons required physical activity (associated with
TDE and group exercises). This was confirmed by at least one student who indicated that he did
not feel that time pressure or physical demands of the class contributed to his workload.
An important distinction should be made between the first and second week of instruction. The
first week, including Day 6, consisted mainly of instructor led activities and was consistent with
typical Army course instruction. In contrast, the second week consisted mainly of student lead
instruction exercises in which students were required to select a topic that they would teach to
other students. This enabled students to practice classroom instruction, which was one of the
goals of the course. The relevance of the workload data to the application of OBTE in the
classroom becomes more apparent in light of this distinction. In short, a key observation is that
self-reported frustration went down as effort and mental demand went up. This change was
correlated with a change in instructional activities.
13.6 Conclusions
With the increasing interest in OBTE and its potential contributions to military instruction, it is
possible to lose sight of the fact that the formative measures for instructors and students have
been an important contribution to OBTE. They are the reason the research team came to the
attention of the Asymmetric Warfare Group. As the result of the experience with OBTE in the
classroom, we have come to believe that the use of formative measures couched in an iterative
longitudinal 360 review can be one of the most valuable tools for instructor education.
231
Artis et al.
Our findings also indicate that workload assessments can provide valuable additional information
about engagement of students in the learning environment. The workload data show that
cognitive load is a multidimensional construct that influences learning and motivation in complex
ways. For example, instructors should be as concerned about excessively low workload as about
excessively high workload. Higher workload is not necessarily a problem if it is a reflection that
students are engaged. Workload assessments could be a valuable diagnostic to guide in-stride
adjustments given the complex effects of cognitive load that are difficult to predict in advance.
13.7 References
Argyris, C, (1991). Teaching Smart People How to Learn. Harvard Business Review, 4(2), 4-15.
Bailey, C. & Austin, M. (2006). 360 Degree Feedback and Developmental Outcomes: The Role
of Feedback Characteristics, Self-Efficacy and Importance of Feedback Dimensions to Focal
Managers' Current Role. International Journal of Selection and Assessment, 14(1), 51-66.
Bransford, J.D., Brown, A.L. & Cocking, R.R. (2000). How people learn: Brain, mind,
experience, and school. Washington, DC: National Academy Press.
development of higher psychological processes (L.S. Vygotsky). Cambridge, MA: Harvard
University Press.
Colquitt, Jason A., LePine, J.A.; Noe, R.A. (2000). Toward an integrative theory of training
motivation: A meta-analytic path analysis of 20 years of research. Journal of Applied
Psychology, 85(5), 678-707.
Cornell-dEchert, B. (2009). Outcomes-based training and education: Implementation guide. Ft.
Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and
personality. Psychological Review, 95(2), 256-273.
Gaillard, A.W.K. (2001). Stress, workload, and fatigue as three behavioral states: A general
overview. In P.A. Hancock & P.A. Desmond (Eds.). Stress, Workload, and Fatigue.
Mahwah, NJ. Erlbaum, 623-639.
Gaillard, A.W.K. (2008). Concentration, stress and performance. In P.A Hancock & J.L. Szalma.
(Eds.). Performance Under Stress. Ashgate, Chichester, England (pp. 59-75).
Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (task load index): Results
of empirical and theoretical research. In P. A. Hancock & N. Meshkati (Eds.) Human
MentalWorkload. North-Holland: Elsevier Science Publishers (pp. 139-183).
Kahneman, D. (1973). Attention and Effort. Englewood Cliffs, NJ: Prentice-Hall.London.
Publishers, Inc
Knowles, M. (1975). Self-directed learning. Chicago, IL: Follet.
London, M. & Beatty, R. (1990). 360 degree feedback as a competitive advantage. Human
Resource Management, 32 (3), 353 372.
Pfau, B. & Kay, I. (2002). Does 360-degree feedback negatively affect company performance?
Studies show that 360-degree feedback may do more harm than good. What's the problem?
HRMagazine, Jun 2002. 47(6),54-60.
Routledge.
232
Veltman, J.A. & Gaillard, A.W.K. (1997). Dissociation between Task Demands and Mental
Effort. Proceedings of the International Ergonomics Association. Tampere, Findland..
Wickens, C. D. (1991). Processing resources and attention. In D. Damos (Ed.), Multiple-task
Performance. London: Taylor & Francis (pp. 3-34).
Wickens, C. D. (1992). Engineering psychology and Human Performance (2nd edition). New
York, NY: HarperCollins.
Wickens, C.D., Gordon, S., and Liu, Y. (1998). Introduction to Human Factors Engineering (1st
edition). New York: Addison-Wesley-Longman.
Wohlers, A.J. & Hallagher, P. (1990). 360-Degree feedback surveys: A source of feedback to
guide management development. Journal of Management Development, 9, 1731.
233
Semmens et al.
Chapter 14. Organizational Climate and Creation of Durable Change
Rob Semmens, C. Megan Austin
Imprimis, Inc.
Gary E. Riccio
14.1 The Need
The Asymmetric Warfare Group (AWG) sought an independent scientific study of their initiative
in Outcomes Based Training and Education (OBTE). Two of the objectives of that effort (see
Chapter 1) are addressed briefly in this chapter:
Identify organizational factors affecting OBTE implementation (including life-cycle

management) in Army institutions of training and education.
Identify the extent to which initiatives analogous to OBTE can be developed to achieve
similar impact on other programs of training and education.
The Army has mandated change in Field Manual 1, The Army (Headquarters Department of the
Army [HQDA], 2005). It states that the Army culture needs to change to support the
transformation of the force to counter both conventional and irregular threats. While there have
been considerable gains in the implementation of OBTE over the last two years (see Chapters 1
and 11), organizational factors are likely to affect its durability (see Appendix C). Further, the
individuals and activities required to sustain OBTE are not the same as those instrumental in
initial implementation.
Early in the investigation it became apparent that there might be resistance to changing the
current approach to Army training. We thus looked for opportunities to acquire evidence for local
situational factors influencing the implementation of OBTE that were not directly under the
control of the AWG. From the outset, we were interested in answering, Why have things always
been this way? The individual officers and non-commissioned officers with whom we interacted
on the ranges and in training areas generally wanted to do the right thing. There seemed to be
some systemic constraints that did not set them up for success, however, and the apparent result
was training that had considerable room for improvement (see Chapter 11, sections 9.2-9.3).
In interviews with the AWG train-the-trainer cadre, it appeared that many common practices or
procedures had to be disregarded to demonstrate OBTE, and that organizational barriers were
prohibiting or curtailing the implementation of this approach to training (anonymous, personal
communication, April 16, 2008). Trainers in the Army have often had difficulties with Range
Control, and the demonstration of OBTE proved to be no exception. In one case, a Brigade
Commander told Range Control to go away, because he was not going to stop training to
accommodate Range Controls preferences. On another occasion, an instructor, when questioned
by Range Control about his risk assessment, had to call the post commander on his cell phone and
put Range Control personnel on the phone with the General who proceeded to approve the
training (anonymous, personal communication, May 18, 2009).
Organizational Change
234
Sustainment of OBTE may require involvement of a wider range of personnel to reduce current
organizational frictions associated with training. Involvement of doctrine writers in the U.S.
Army Training and Doctrine Command (TRADOC), such as the working group for Regulation
350-70, and their coordination with instructor educators for OBTE is an example of the dialog
and diversity in a social network that can make a difference. Based on the organizational change
literature, even broader coordination may be required (e.g., Cooperrider, Whitney, & Stavros,
2003; Ludema, Whitney, Mohr, & Griffin, 2003; Schein, 1992).
Resistance to change is well described in the behavioral science, quality improvement, and
business literature. In the early days of social psychology, Kurt Lewin developed a model of the
change process in human systems (e.g., Lewin & Grabbe, 1945; see also, Schein, 2009). He
described resistance to change as forces in any system that hold the status quo in placeon one
side there are forces that push for change and on the other side there are forces pushing against
the change (Lewin, 1951; see Chapter 3; see also, Henle, Jaynes, & Sullivan, 1973). He
conceptualized reciprocal influences within a complex social network analogous to feedback in
electrical networks and dynamical systems (cf., Henle, 1971; Turvey Shaw, & Mace, 1982;
Riccio & Vicente, 2001). Interpersonal interactions and dialog are key manifestations of the
opportunities for such reciprocal influence (Schein, 1968; 1993).
One should distinguish the concept of resistance to change from any meaning associated with a
subordinate as a management problem or difficulties in getting a subordinate to do what the
superior wants. It demeans the concerns subordinates may have about the issues they face on the
front lines of an organization (Edmonson, 1999; Roberto, 2005, 2009). The primary locus of
friction may even be at the management level (Vaughan, 1997). In any case, friction cannot be
attributed to a single individual, subgroup, or service function. The issues surrounding a change
initiative are embedded within the complexity of an organization. Implementations of OBTE that
seem to be the most successful, at least in terms of minimal complaints about friction, seem to
address this complexity directly and through collaborative dialog (see Appendix C; Haskins,
2009; cf., Cooperrider et al., 2003; Ludema et al., 2003; Schein, 1993).
14.2 Initial Indications of Possible Resistance to Change
As reported in Chapter 10, we administered a post-deployment survey to Soldiers who had
participated in CATC to get an indication of whether the approach would be sustained in the
Operating Force. In the short survey, one question attempted to reveal indications of potential
resistance to OBTE:
Once your unit enters the next training cycle, what might you see as obstacles to
executing a CATC [OBTE] approach to some or all of your training?
While the number of respondents to the survey was small (N=17), all had been deployed in
leadership positions within a platoon. These individuals had the most to gain by this new training,
but they had little influence in planning or resourcing the training. Nine Soldiers mentioned
concerns with the larger organization being open to changing the way the Army trains. They
mentioned doctrine, safety regulations, and that some more senior echelons were resistant to
adopting a new approach or were stuck in old training standards. Five Soldiers mentioned a
possible lack of resources as an impediment, which could be interpreted as a lack of acceptance
by higher echelons in that platoons have little influence on the control the direction of resources.
235
Semmens et al.
Another interesting indication of resistance to change was provided by a survey we administered

prior to familiarization with OBTE at Fort Sill and, again, six months after the beginning of
familiarization with OBTE (see Chapter 9).
Figure 1. Attitudes toward change at Fort Sill (see Chapter 9 for details).
The figure above indiciates that, among two different samples of officers and non-commissioned
officers, there was little change in the number of skills they thought OBTE could help them train
during basic training. All reported positive feelings about the training, however. In fact, the
Commander of the 434th BCT, indicated that their implementation of OBTE has been effective at
increasing the number of first time qualifiers (personal communication, June 30, 2009.) It is
likely that these Soldiers value the approach as useful, yet for a variety of reasons, they find it
difficult to implement formally or systemically during Basic Training. This could be due to real
or perceived organizational resistance to change (see also Chapter 9, section 9.2.3).
We continued to ask questions of those involved with implementing OBTE on an ad hoc and
informal basis. The lessons learned from this informal inquiry are used to identify leads that can
be pursued in the organizational change literature and subsequent research on OBTE.
14.3 Models and Considerations for Sustainable Change
14.3.1 The Change Transition Period
A literature review was conducted to determine similarities across several change models as an
effort to provide insight to further change implementation. While a full-scale, coordinated effort
of continuous change management cannot be outlined fully, identification of common obstacles
to change suggest activities that can be undertaken, in principle, to improve the durability of any
implementation of OBTE. The review showed that while there are many change models varying
in complexity and scope, several shared one common theme. They all describe cases where an
initial innovation found success and demonstrated a marked improvement over the standard
practice but, over time, this successful change eroded under the pressures of larger organizational
236
influences (cf., Rasmussen, 1997; Snook, 2000). Figure 2 illustrates the transition period where
change is the least durable and vulnerable to loss.
Figure 2. Influences and opportunities during the change transition period.

The change transition period has been defined in several models, yet all of the descriptions are
remarkably similar. In Kotters model (Kotter, 1995) this stage is called Consolidate
Improvements and Produce More Change. It is the seventh of eight of his stages and builds on
the sixth stage in which short-term, attainable, demonstrative improvements are made. In the
seventh stage, it is necessary to use the increased credibility from the initial success to change
systems, structures, and policies that are incongruent with the vision as well as to create an influx
of new personnel who can implement the vision and reinvigorate the change process. In the
eighth stage, the change leaders must articulate the connections between the new behavior and
success, and create new social norms and shared values consistent with changes (cf., Chrissis,
Konrad, & Shrum, 2003; CMMI Product Team, 2009). Based on findings reported in previous
chapters, it is likely that OBTE is currently in the sixth stage (although, see Appendix C for an
exception and a model for a path forward).
Several other models differentiate between the initial and sustained change. Berquist (1993)
differentiated between first-order change and second-order change. First-order change is doing
more of something or doing it better, and it is reversible. It is characterized by adjustments within
the existing structure that are not transformational; the old story can still be told. Second-order
change is considered fundamentally different in that, once the new way is in place, it becomes
impossible to return to the way things are done before. Juran (1999) agrees with this, indicating
that any change initiative actually has two change components: the intended change and the social
consequence. He defines this social consequence as the impact of the intended change on the
237
Semmens et al.
cultural pattern of the human beings involvedon their pattern of beliefs, habits, traditions,
practices, and status symbols. This indicates a lasting and deeper implementation.
Its important to note that different kinds of influencers and implementers are needed at different
stages in a change process. Hishborn (2000) described two personalities that manifest in change
initiatives. In organizational settings, zealots provide the best leadership when the campaign's
strategic theme has yet to take root. Then, consensus builders provide the best leadership when
corporate policies need to be changed. The Diffusion of Innovations model distinguishes between
those who adopt a change early and those who adopt a change later (Rogers, 2005). The early
adopters are more open to change and more willing to take a risk. The late adopters and laggards
are likely to have fewer resources, are more skeptical of the change, and demand more evidence
before adopting the change. These different types of personalities suggest that different activities
are needed to convince different people to adopt a change (cf., Bowling, Beehr, Wagner, &
Libkuman, 2005).
14.3.2 Organizational Culture
A theme in the literature on organizational change is that an understanding of organizational
culture is of paramount importance in implementing change. Vicente (2006) describes culture as
an emergent property, the invisible hand that guides behavior.
Culture is manifested both in what people expect from one another and what people
expect from their dealings with the external environment of customers, competitors,
supplies and stakeholders. Culture is unspoken, implicit, taken for granted. You feel
the effects of culture when what you do feels appropriate or inappropriate. Culture is
largely invisible, especially to those that live within it. One of the reasons it is
difficult to change is that it is almost impossible to see the norms and expectations
that hold it in place. [p. 10]
Project failure can be closely linked to poor understanding of the targeted culture as well as the
misguided assumptions of project designers (Berger et al, 2007, pp. 121-122; Juran, 1999).
Culture must be understood and addressed if the change is to be durable (Schein, 2009). The use
of climate surveys often helps (Zohar, 2007). An understanding of culture helps disambiguate
questions that are inherently vague, such as What aspects of the Army culture are the biggest
barriers to change? Where assessment of the culture of an organization reveals friction, there
will be implications that some aspects of culture may have to change for a new initiative to be
viable and sustainable. Considering such consequences of culture, and the consequences for it,
thus can lead to the identification of new requirements. These derived requirements appropriately
supplement the requirements that originally motivated the innovation (cf., Chrissis, et al., 2003).
Rigorously traceable change proposals become contextualized, and the influence of context
becomes traceable and verifiable.
14.3.3 Clarity of Mission and Shared Understanding
A focused mission that is clearly stated and linked to a broader vision can both reflect and
influence the culture of a place (Juran, 1999). The need for change must be understood across the
organization, and that vision for the change must be maintained as newcomers join the ranks of
the change agents. Lack of understanding about the vision is the most often cited barrier to
change. Participants who understand the vision should be able to answer the question Whats in
it for me? If employees cannot explain the risks of not participating in the change, it is unlikely
238
they truly understand the mission (Prosci Research, 2007). Employees are more likely to
understand a mission if they are included as active participants in refining the definition of the
mission to ensure its viability in the local culture, consistency with a broader shared vision, and in
revealing potentially unintentional consequences (Cooperrider et al., 2003; Freeman et al., 2008;
Ludema et al., 2003; Schein, 1992, 1993; see also Appendix C).
Collaboration and communication can be utilized within an organization to achieve both buy in
and a deeper understanding of mission, objectives, and practices. This is not to abdicate
leadership. Good leaders can be directive about the process of decision making without being
directive about the solution or content of a decision; leaders can be assertive while leading with
restraint (Roberto, 2005, 2009). They can inspire leadership in others rather than thinking for
them and getting them to do what they are told.
14.3.4 Relevant Observations During the Current Investigation
Broad collaboration and communication is being used to achieve shared understanding of the
organizational climate for change in the Department of Military Instruction (DMI) at the United
States Military Academy (C. Haskins, personal communication, June 9, 2009). The Commander
of DMI has used this approach in prior assignments to build consensus around new initiatives
related to OBTE. He includes instructors and their command chain in the definition of outcomes
and in the continual adaptation of courses. He requires that the outcomes for his training be
written in plain English so that his subordinates dont apply preconceived notions to
understanding the change he is attempting to implement (Appendix C). Thus they are not merely
included. They are participants in thinking and solving problems, which is precisely what they
will try to get their students to do in the courses in DMI. Our informal observations of this
command climate suggest that it is energizing for the participants and that it motivates deeper and
broader engagement.
At Fort Sill, we were able to talk with one battery commander who had been involved with the
implementation of OBTE in his Basic Combat Training Battery. He had generally positive
feelings about OBTE as applied to marksmanship and medical skills. However, when asked if his
feedback went beyond the Battalion, he was unable to say. This may indicate a risk to the change,
as the proponents of the innovation do not have clear lines of communication with the new
adopters. More formal supports could be provided for shared understanding in the context of
organizational change (Cooperrider et al., 2003; Ludema et al., 2003; Schein, 1993; see also,
Appendix C).
14.3.5 Organizational Support and Incentives
Ultimately, even the most optimally primed environment for change must provide adequate
support. Support includes sustainment training, education, material resources, as well as time to
assimilate the new methods (Jorgensen, Owen, & Neus, 2009; Juran, 1999; National Academy for
Academic Leadership, 2007). There is a temptation for upper echelons to assume support is
adequate. Thus it is important to continue to probe lower echelons to determine if the necessary
operational and logistical resources are available to all levels in order to promote success (Berger
& Benhow, 2001; National Academy for Academic Leadership, 2007). This gives subordinates
the opportunity to comment on whether or not it is good and sufficient.
Incentives play a large role in accelerating change (Juran, 1999; National Academy for Academic
Leadership, 2007). If participants dont feel any incentive to participate they may not do so, either
actively or passively, or they may participate inconsistently. A critical consideration is that all the
239
Semmens et al.
factors influencing an individuals behavior and performance within an organization should be

aligned. This can be complex in that implicit and indirect influences may not be well understood,
they may be different or inconsistent with explicit criteria, and they may combine with explicit
criteria to create emergent influences that may not be desirable (Rasmussen, 1997). With respect
to OBTE, all the right intentions and preparation for teaching students how to be adaptable and to
develop long-term intangible attributes easily can be undermined by an implicit or explicit
criterion based on throughput or a narrow conception of risk. For such reasons, in OBTE, it is
important to assess what instructors are doing in addition to what they are achieving, and it is
important to query them about why they are doing what they are doing.
14.4 Conclusions
The intent of this chapter was to demonstrate fruitful lines of inquiry into sources of resistance to
changes necessitated or implied by OBTE. The forces described above are the barriers all
organizations must anticipate and understand if they are to succeed in their quality improvement
initiatives. Given the common themes underlying resistance to change, we designed questions to
help identify patterns and key areas that are blocking progress at all levels involved in planning
and executing OBTE (see, e.g., Chapters 9 and 13). With these questions, we began to assess
attitudes and depth of understanding as well as to identify patterns of resistance and the highest
yield areas for improving the rate of adoption. We began to see the art of the possible in surveys
and interviews with individuals who were responsible for instruction.
Moving forward, it is clear that a thoroughgoing assessment of an instructional service system
should include assessments of organizational climate and resistance to change at all levels of an
organization that influences instruction and learning. Moreover, the intent for these assessments
should be similar to the assessments we developed for instructors (see Chapter 2) and students
(see Chapters 7 and 12). They should be replicable and actionable (Chrissis et al., 2003; CMMI
Product Team, 2009). They should provide insights that are credible, transferable, dependable,
and confirmable (Denzin & Lincoln, 2003; see Chapter 11). The multidisciplinary research
relevant to organizational change, exemplified by citations in this chapter, can be leveraged to
develop such methods of assessing organizational factors affecting change. Finally,
organizational assessments should be commensurate with the measures of instruction and
learning so that there can be a synthesis of the lessons learned. Ultimately, programmatic
decision-making requires an integrated understanding of the instructional service system with
respect to overarching goals and multifaceted ways of making coordinated progress toward those
goals (cf., CMMI Product Team, 2009; Rasmussen, 1997; Riccio & Vicente, 2001; Pellegrino et
al., 2001; see Chapters 1 and 11).
14.5 References
Berger R. & Benhow, D. (2001). The certified quality engineer handbook. ASQ Quality Press.
Bergquist, W, (1993). The Modern Organization: Mastering the Art of Irreversible Change. San
Francisco: Jossey-Bass.
Bowling, N., Beehr, T., Wagner, S., & Libkuman, T. (2005). Adaptation-level theory, opponent
process theory, and dispositions: An integrated approach to the stability of job satisfaction.
Journal of Applied Psychology, 90, 6, 10441053.
CMMI Product Team (2009). CMMI for services, version 1.2. (CMU/SEI-TR-2009-001; ESCTR2009-001). Pittsburgh, PA: Carnegie Mellon University.
240
Cooperrider, D., Whitney, D. & Stavros, J. (2003). Appreciative inquiry handbook: The first in a
series of AI worksbooks for leaders of change. Bedford Heights, OH: Lakeshore
Communications.
Edmondson, A. (1999). Psychological safety and learning behavior in work teams. Administrative
Science Quarterly, 44, 350-383.
Headquarters Department of the Army (2005). The Army. Field Manual No. 1-0. Washington,
DC: Headquarters Department of the Army.
Henle, M., J. Jaynes, & Sullivan, J. (Eds.) (1973), Historical conceptions of psychology (pp. 257266). New York, NY: Springer.
Hirschhorn, L. & May, L. (2000). The campaign approach to change. Change, 32(3). 30-37.
Jorgensen, H., Owen, L., & Neus, A. (2009). Stop improvising change management! Strategy and
Leadership, 37(2), 38-44.
Juran, J. M. (1999). The quality improvement process. In J.M. Juran & A.B. Godfrey (Eds.).
Jurans Quality Handbook (5th edition). McGraw Hill, New York.
Kotter, J. P. (1995). Leading change why transformation efforts fail. Harvard Business Review,
March-April, 59-67.
Ludema, J., Whitney, d., Mohr, B., & Griffin, T. (2003). The appreciative inquiry summit: A
practitioners guide to leading large-group change. San Francisco, CA: Barrett-Koehler.
National Academy for Academic Leadership (2007). Leadership & institutional change.
Retrieved April 1, 2009 from http://www.thenationalacademy.org/ready/change.html.
Prosci Research (2007). Best practices in change management: Benchmarking report. Loveland,
CO: Prosci Research. Retrieved March 30, 2009 from http://www.prosci.com/tutorialresistance.htm.
Science, 27, 183-213.
Riccio, G. & Vicente, K. (2001). Coping with change and novelty in energy facility operations:
Recommendations for knowledge-based, extra-procedural problem solving. In: M. Gross &
T. Ayes (Eds.), Electrical Power Research Institute Report 1004666. Palo Alto, CA: EPRI.
Roberto, (2005). Managing for conflict and consensus: Why great leader dont take yes for an
answer. Saddle River, NJ: Wharton School Publishing.
Roberto, M. (2009). Know What You Don't Know: How Great Leaders Prevent Problems Before
They Happen. Saddle River, NJ: Wharton School Publishing.
Rogers, E. M. (2005). Diffusion of Innovations. Glencoe: Free Press.
Snook, S. (2000). Friendly Fire: The Accidental Shootdown of U.S. Black Hawks over Northern
Iraq. Princeton, NJ: Princeton University Press.
Salmoni, B. (2007). Recommendations to plenum from panel E: Military as culture. Pedagogy
for the Long WarTeaching Irregular Warfare. Quantico, VA: United States Naval
Academy and Marine Corps.
Schein, E. (1968). Personal change through interpersonal relationships. In W. Bennis, W., E.
Schein, F. Steele, & D. Berlew, D. (Eds.). Interpersonal Dynamics. Homewood, NJ: Dosey
Press.
241
Semmens et al.
Schein, E. (1992) Organizational culture and leadership. 2d. Ed. San Francisco, CA.: Jossey Bass.
Schein, E. (1993) On dialogue, culture, and organizational learning. Organizational Dynamics,
Winter, 40-51.
Schein, E (2009). Kurt Lewin's change theory in the field and in the classroom: Notes toward a
model of managed learning. MIT Sloan School of Management. Retrieved April 16, 2009,
from http://www.solonline.org/res/wp/10006.html.
Vicente, K. J. (2006). The Human Factor: Revolutionizing the Way People Live with
Technology. Routledge, New York.
Zohar, D. (2000). A group-level model of safety climate: Testing the effect of group climate on
microaccidents in manufacturing jobs. Journal of Applied Psychology, 85(4), pp. 587596.
242
Leader Development
Chapter 15. Five ways OBTE can enable the Army Leader Development Strategy
Blaise Cornell dEchert
15.1 Background
Since the release of the Adapt or Die TRADOC White Paper subsequently published in Army
magazine (Fastabend & Simpson, 2004), there has been extensive discussion and other activity
related to training (and educating) Soldiers differently for challenges in the contemporary
operational environment (COE). Senior leader calls to accelerate change and transform
notwithstanding, the emergent changes have not, so far, been as effective, durable, or as
appropriate as they could have been. The delays and many draft revisions of FM 7-0 are
symptomatic of the divergence of opinions about what needs to be changed in Army training. The
same is true of both the Army Training & Leader Development Strategy (ATLDS) and the Army
Leader Development Strategy (ALDS). There are, within all of those documents and expressions
of a new vision, common and recurring themes but they are over-shadowed by retaining too much
legacy content from prior expressions of how the Army trains (United States Army Training and
Doctrine Command [TRADOC], 2009).
The seeming slow pace of change has less to do with the quality of leader vision than it has to do
with the details of implementation. It is not for any lack of understanding that the COE has
imposed a requirement for change in both how and what we train. The COE has changed the
operational Army, making it adapt and evolve in different, sometimes difficult and unexpected
ways. Even as this forced adaptation occurs, everyone continues to urge the Army to be more
adaptive, agile, and innovative. Even a shallow understanding of the Army operational concept of
Full Spectrum Operations (FSO) makes it clear that the way we used to train will no longer
suffice. Army senior leadership insists that there be a fundamental difference of thought about
how the Army trains. It seems clear that everyone understands this. The awkward remaining
obstacle is how can it be done? A call for new thinking may be insufficient to bring about the
kinds of change in training and education that is required to support the Armys operational
concept.
How does the Army change the thinking about the way it trains when the people who will be
doing it are emotionally attached to what made them successful? Where is the incentive to
change an entire system if the Army still is accomplishing its missions?
The Army fundamentally has a task training culture that both permeates and resonates throughout
the activities that prepare Soldiers, units and formations for the kinds of missions envisioned in
FSO. This culture and the associated behavior, attitudes, and beliefs are deeply rooted and cannot
be undone quickly. Yet the Army has an opportunity to turn in a new direction if they offer an
alternative that more closely aligns with experiences of operating in the COE.
The battalion commanders of today were formed and normed as lieutenants in the 1990s. This
followed the phenomenal successes of Operations Just Cause and Desert Storm, a period of
constrained resources, of non-traditional stability operations, of prescriptive training strategies,
pre-deployment training checklists and other mandatory training requirements. Even for company
commanders, little imagination was required or expected. Training management skill was valued
more highly than any evidence of innovation in designing or executing training. Leaders became
243
Cornell d-Echert
if not comfortable with, then conditioned to, someone else deciding what training they should do.
The rapidly evolving conditions in Iraq and Afghanistan from 2003-2007 reinforced this
expectation of being told what to train. Clearly, there was justifiable need to provide rapid
training solutions to forces deploying to fight in a new way with new equipment and new threats.
However, because of their experiences, this generation of leaders may not view all training
challenges as their own to solve. They may depend upon others to provide the approved solution.
These same commanders have also experienced the power that our doctrine of battle and mission
command envisions. The requirement for greatly decentralized operations, mission-type orders
that still embody ambiguity with the latitude to do what is necessary, has become the new norm,
if not the reality. Commanders increasingly comment on the disparity observed between the
qualities needed in their Soldiers and leaders and what they received from the training base. As
some of these same commanders subsequently returned to CONUS to fill positions in TRADOC
schools and centers they had two choices. They could make the changes they felt were necessary
and were empowered to do, or they could become defenders of the organizations they were now
part of. For those that pursue change, the systems resilience and resistance to change proves
daunting and frustrating unless the most senior leaders intervene.
The duration of our current persistent engagement, and the constancy of senior leader urgings
to change, to adapt, to think differently is making a difference. A growing population of leaders
in the training base is no longer content to find a way to make the system work, but is instead
seeking to change the system altogether. The revised FM 7-0 (Headquarters Department of the
Army [HQDA], 2002), a new AR 350-1 (HQDA, 2009), new thinking in the ATLDS and ALDS,
even the revision of TR 350-70 (United States Army Training and Doctrine Command
[TRADOC], 1999) and its associated pamphlets will assist, albeit slowly. Other associated
components of the system must be compelled to change as well, such as the accreditation focus
on compliance that could be changed to focus on identifying impediments to change. Local
bureaucracies and their associated over-reaching regulations need restraint from a controlling
relationship over unit training desires to a more supporting relationship. Overall, the training
system needs change there is much inertia to overcome. This will require extensive leader
involvement that may be hard to achieve. Transformation challenges, solving ARFORGEN
puzzles, contending with BRAC issues, and the constant balancing act of operating below
authorizations with no diminution of requirements consume much of available leader time.
Trainers, course managers and leaders need a comprehensive and holistic solution that can help
change their mind-set, yet does so in a way that does not discard the mechanics of the system the
Army depends on for readiness reporting and forecasting training resources. The solution can
account for the current operational experiences, and the impressions they made, as well as
recognizing the significance of the requirements of FSO. The solution can be both new and old at
the same time. It can focus most of its attention on influencing the thinking of those that will plan,
support, execute, and evaluate training and education.
For many of the reasons cited above, a solution known as Outcomes Based Training & Education
(OBTE) continues to attract attention and interest. Originally conceived as a solution to
commanders concerns about training effectiveness, especially in weapons and small unit tactics,
more practice and experience implementing the OBTE solution led to a deeper exploration of
where, why and how effectiveness in OBTE differed from the traditional training approaches.
Under the aegis of the Army Asymmetric Warfare Group (AWG), the program known as the
Combat Applications Training Course (CATC) migrated from assisting pre-deploying units to
assisting Basic Combat Training at Fort Jackson, then to Fort Benning and elsewhere (see
Chapter 1). From early 2006 until today, Soldiers, leaders and training specialists all describe
244
Leader Development
how familiar the experience in the CATC is to other good training yet just how different the
results prove to be. Most of the difference and the change that they sense and experience is the
result of a change of focus that forces changes in how they think about what they are doing.
This paper will try to describe briefly that change of focus and the different thinking that
emerges. It will also show OBTE as a way to provide the new thinking desired about how we
train and educate Soldiers and develop leaders. Accordingly, implementing OBTE can also
potentially enable the Army Leader Development Strategy through the broad coincidence of
purpose.
15.2 An Emerging Consensus
Sometimes key studies, working groups, and panels produce interesting work revealing possible
futures; they do not receive much attention, but can be persuasive years later in unplanned or
unexpected ways. In 2003, the Defense Science Board (DSB) released the second portion of a
study of military training titled Training for Future Conflicts (DSB, 2003). The DSB expressed
a concept they called training surprise, evidence that we have seen affecting two allies: the 2006
Hezbollah Israeli Defense Force clash and the 2008 Russian incursion in Georgia. In 2006, the
Army Research Institute hosted for TRADOC a science of learning workshop (Quinkert,
Morrison, Fletcher, Moses, & Roberts, 2007) that validated such TRADOC initiatives as dL
(distance learning), leveraging social networking and the use of competencies in development and
life-long learning. Many of these ideas found their way into the latest iteration of FM 7-0
describing what training should do to Soldiers, leaders and units in preparing them for the FSO
expected to be the new norm.
FM 7-0 also continued to make a distinction between trainings effect on doing and educations
affect on thinking even though it can be argued that such distinctions are neither accurate nor
useful. That idea was a feature in the USMC Training and Education Command and USNA 2007
conference on teaching irregular warfare (Salmoni, 2008). One recommendation was to achieve a
better balance between training and education, rather than considering them as distinctly different
activities, because thinking (what we really mean is judgment) is an essential requirement at all
echelons in almost every conceivable military environment (see Chapters 3, 4, 5).
There is a reason for citing these particular conference proceedings. While there may be others,
these in particular are future oriented but framed from a context of analyzing what is and is not
working within the COE. Our military is not alone in finding the traditional mode of training
soldiers and armies to perform tasks inadequate. Throughout the world, the need for thinking
soldiers demanding new methods of preparation and development are underway. We should
expend the effort necessary now when we can, rather than find ourselves in a position where we
must.
Adaptability is therefore one of the goals of our training policies. More than
ever before we will be stressing versatility in individuals and units.
Your Training Job, The Officer Journal, June 1950
15.2.1 What Part to Balance?
Operational experience reinforced with a balanced portion of training and education enhances
military versatility in action and agility of thought. Such appears to be the thrust of thinking in
both FM 7-0 and the ALDS. For the most part, operational experience is what it is, and
245
Cornell d-Echert
opportunity to influence what occurs (that shapes, forms or develops) is minimal. Purposefully
balancing the blend of training and education should be easier to achieve and should be a key
component of any development strategy.
Army leaders, training developers and educators struggle to find any balance when considering
the differing requirements of training (for certainty) and educating (for uncertainty). While newer
definitions of training and education may minimize the debate between trainers and educators,
they may be insufficient to achieve the new thinking (and new action) that is required. It might
prove more useful to bring the two ideas together. Changing the argument from either training or
education to one that is both, and could make the difference. The description of paradigm
shifts in the ALDS seems to offer this opportunity. Given that every Soldier is a potential future
leader, the Army Leader Development Strategy essentially describes the ends and ways for
preparing all Soldiers for full spectrum operations by providing training and education leavened
with operational (or some kind of relevant) experience. This also offers a way to re-direct
thinking away from training for task performance and toward development for broader
competence.
The training and education system that most Army leaders experienced in the past two decades
filled a particular function to prepare them for an offense-oriented doctrine. Over the past eight
years, what has become more evident with this new operational environment of persistent
engagement is that former distinctions of relevance are much less clear. Traditional distinctions
between conventional and unconventional, regular and irregular, war and peace, even between
winning and losing all have different meanings. Expectations about traditional threats, how an
enemy will fight and how our Army needs to be organized, trained and equipped to counter
possible threats have become much less certain. Our Army, instead of training for a war, is
learning that it must prepare for not just for war, but also for anything.
Any viable solution to help change thinking or execution of Army training should recognize that,
over time, the current training system evolved extensive controlling systems. To move beyond
that system requires a loosening of controls. Such loosening depends upon two key concepts -Trust and Confidence (see Appendix C). The current generation of commanders does not have
this experience in training settings, but they do in combat. Leaders need to return to the business
of being trainers themselves, rather than just training managers. Combat conditions of this decade
have made true many of our beliefs about mission and battle command. Counter Insurgency
(COIN) success begins with a controlling idea but demands a vastly decentralized execution.
Preparing the Army for success in FSO will require the same kind of commitment.
15.2.2 Improving Training, by Design
OBTE is a comprehensive approach that conjoins training and education practices within the
context of operational requirements that either exploit past experience or prepare for experiences
still to come (see Chapter 3). The balance leaders are asking for should not be about how much
training or what to educate because both are required. The balance is to ensure that education is
about solving problems without describing solutions, about teaching skills instead of just training
tasks, and to do these things at the same time.
Achieving this balance does not happen without some thoughtful effort. Design is a critical
element. Just as we have experienced with operations over extended time, preparing Soldiers and
leaders for FSO success will take time and requires a design perspective. Such a perspective
offers opportunity to consider purpose, to prioritize and link activities and to leverage
opportunities on tangents or in the margins. Most importantly, it makes clear that every action (or
246
Leader Development
inaction) has consequence, either positive or negative, so the designer will seek ways to maximize
the positive (see Chapter 3).
The manner by which such design occurs depends upon certain elements of information or
guidance. The ALDS contains some descriptions of what the Army wants of or from various
leader cohorts. Without further elaboration, schools and their course managers will generate lists
and attempt to crosswalk desired attributes (or outcomes) to the training and education tasks that
already exist. While that is efficient, it will do very little to achieve the qualitative improvement
the new thinking about training envisions. More importantly, lacking sufficient design guidance
and continued dependence upon the uncertain efficacy of critical task analysis training developers
will be unable to leverage the potential of the new instructional media the Army continues to
make investments in (see Epilogue).
15.2.3 Increased Use of dL and Dependence on Self-Development
In the past decade, the Army has invested millions building the infrastructure to support dL.
Recent adoption of knowledge management and social networking increases the availability of
information and learning portals. As dL technology and the content offered continue to improve,
it is equally important that future users be, in fact, ready to take full advantage of what will be
available. Making Soldiers ready to learn by dL and self-development does not mean just
exposing them to those mediums. Developing their awareness of what they need (as an
individual) to learn is, and is a second way that OBTE can enable the ALDS and improve changes
in training and education (see Chapter 5 and Epilogue).
Some of that readiness begins in a Soldiers earliest exposures to Army training and education.
Formative experiences in initial military training establish foundations that sustain and support
future enlargement, or prove unsound and require extensive re-building. The effectiveness of dL,
self-development, or lifelong learning as a development strategy requires that Soldiers have
mostly positive experiences. Leaders and trainers should promote the value of self-awareness,
increase opportunities to practice self-development, and teach how to respond to or act as a coach
and mentor.
Self-awareness, increased efficacy and protg behavior emerge or are evident when training and
education are designed, developed and implemented using OBTE principles (see Chapter 3).
When Soldiers have the opportunity to learn by solving problems, to recognize the relatedness of
various tasks in the context of a mission setting, to continue to strive for their personal best
without an over reliance on contrived metrics, the result is that they begin to understand how they
themselves learn. This is significant and most leaders quickly realize that it is not something that
routinely occurs in the current training and education environment. However, it is something that
can and does occur in units where leaders and their subordinates enjoy a more natural coachplayer relationship.
OBTE Principles (Broad & Programmatic)
Principle 1. Leadership and enculturation of Soldiers
Principle 2. Integrated understanding of basic Soldier skills in Full Spectrum Operations
Principle 3. Collaborative reflection and problem solving
Principle 4. Soldier motivation and development of intangibles
247
Cornell d-Echert
15.2.4 Future Orientation, Unknown Requirements

There is another reason to favor an approach that encourages self-development and learning how
to learn. Training in the Army today has to account for unknown work performed in the future
whereas in the past Army training was mostly focused on a specific job or function, much as
training in industry does. FSO argue strongly for more broadly gauged generalists to handle the
ambiguities and uncertainties of a future operating environment. However, core competencies
still call for if not expertise, at least abilities beyond proficiency in the critical skills of a military
specialty (or unit type). Soldiers who understand how they learn (even with predominantly handson kinds of tasks) can more rapidly improve their abilities to perform in either environment (see
Chapter 3).
Yet another benefit of self-aware Soldiers is that they can develop multifunctional skills. The
increasing complexity of tasks, and the equipment or concepts to accomplish them, portends far
more essential training occurring in units because there will not be enough time in school courses.
Soldiers empowered by awareness of how to learn will be able to develop a degree of mastery in
a specialized task as and when needed when using new equipment or implementing new
concepts. Finally, Soldiers that understand how to develop their own skill mastery can
dramatically improve the collective performance of the unit when operating in a command
climate that supports innovation. It can be argued that this is characteristic of high performing
units. Such inherent flexibility and versatility can potentially immunize expeditionary formations
from the effects of training surprise.
15.2.5 The Quality Instructor Challenge
The relative importance of education and the instructors that provide it is a feature of the ALDS
as a major component of improving leader development. High quality instructors are necessary to
provide the kinds of educational outcomes needed for high performing units. Evidence of
selecting high quality instructors is most apparent in some Captain Career Courses but the high
quality population is small and in high demand throughout the Army. Arguably, all schools and
courses should have the most qualified, the most experienced and the highest quality leaders to
serve as instructors. After all, these are the people that build our bench. Fixing quality education
by making instructor duty attractive, desirable or beneficial in building a career profile, while
useful, seldom remains a durable solution. There will always be a new, more pressing situation
demanding the highest quality to implement some new program or initiative.
OBTE offers a third way to enable the ALDS by making qualitative improvements in military
instruction with whatever instructors are available. One way to improve military instruction is to
stop believing that Army instructors are some separate species that require special preparation.
Clearly, there are activities that will help prepare leaders in their transition from leadership in
units to their new role as instructors. What should not happen, however, is for these instructors to
discard everything they have learned about developing Soldiers. The same skills and many of the
same techniques apply as equally in the classroom or on the training field as they do in the squad
bay, motor pool, or Forward Operating Base. Too many great leaders, after indoctrination in the
system, believe their requirement is merely to present instruction, or impart information for
knowledge transfer. Leaders that attend the mandatory Army Basic Instructor Course rarely
emerge with a sense that their mission is to teach. We should not compel new instructors to
forego all that they have learned (through experience) about teaching, coaching and mentoring
Soldiers just because they are now in a school setting (see Chapter 4).
Leader Development
248
All the same, exceptional operational experience and practical leadership does not guarantee a
leader will be a good teacher. Part of the rationale supporting the production of training products
and their method of presentation is meant to negate this possible shortcoming. The training
developers goal is to provide a product that is useful for an inexperienced instructor the first time
in front of students. The problem, as with all such universal solutions, is that the bar is often set
too low. More options, better tools, and a range of learning activities will serve a broader range of
instructors and allow achieving more, rather than fewer learning objectives per session. Arguably,
such an approach will also ensure training products have broader utility to different learning
locales such as in schools, mobile training teams, or in unit-based training (see Epilogue).
Training and education using the principles of OBTE encourages trainers and instructors to take
ownership of their instructional content to the extent that they have a responsibility to ensure
students going to the operating force are immediately useful rather than an added training burden.
When instructors believe the idea, when their leaders endorse this approach, when measures of
effectiveness account for what matters rather than quantitative and statistical measures, then
instruction invariably shows significant quality differences. A sign that such practice is occurring
is the extent and quality of the instructor-student interaction and communication (see Chapters 2
and 3). In traditional methods of presenting instruction such interaction is very limited. An
OBTE aware instructor assumes a role similar to a coach or mentor, and instructor-student
interactions increase (see Chapters 8 and 9). Clearly this is only proportionally effective; the
greater the student-to-instructor ratio, the less opportunity for such interactions to occur. We all
know this intuitively and this is why most professional development courses stipulate small group
instruction.
15.2.6 Purpose and Design are Key
OBTE principles aid course managers and training developers in their analysis, design and
development activities as they prepare material for instructor use (see Chapter 3). The traditional
application of Instructional Systems Design (ISD) views learning as a sequential and progressive
execution of learning objectives linked to critical tasks associated with particular occupational
specialties. Evaluation tends to focus on the ability to accomplish the task (or action) consistent
with a pre-determined standard of performance. The inevitable result of such a system is that
success in evaluation is more important than what is taught (and learned); this is training to the
test. It is why Drill Sergeants may think that the purpose for training basic rifle marksmanship is
for the Soldier to qualify, not to be confident using the weapon. This is why cadre may believe
the purpose for teaching troop leading procedures (TLP) is to ensure leaders know the steps and
proper sequence of TLP, but new officers cannot plan or execute a mission without extensive
coaching. This is why NCO can describe terrain features and plot grid coordinates but many
cannot pass a land navigation course on the first attempt.
If course managers and training developers instead focused their analysis on how to achieve
outcomes aligned with a particular course, they could use this as a guidepost to check the validity
of learning activities within a current or planned POI. Further, use of outcomes guides design and
development by viewing all learning activities as opportunities to develop the student toward
achieving the desired course outcomes. Outcomes and objectives are not synonymous. Outcomes
broadly describe a design goal, objectives shape the means to achieve those goals, and the
training developer can then choose learning activities that purposefully achieve those objectives.
The developer can view every activity from the context of relatedness to other activities and all
guided by the holistic perspective of the outcome desired. With this design approach, learning
activities selected will use appropriate learning strategies to meet objectives, goals and outcome.
If a particular objective requires a particular training technique, educational strategy, or some
249
Cornell d-Echert
blending of the two, the training developer is better able to provide the instructor an appropriate
training support package (TSP). The one-size fits all TSP template, while efficient for rapid
training development, does not always lead to positive outcomes or always assure meeting the
objective (see Chapter 3).
There is no intent to discard the elaborate architecture that has grown to make training
management and training development efficient, nor to denigrate the role of standards in
measuring skills and abilities. Tasks and their associated standards are essential components to
allow the Army to make readiness and resource forecasting decisions. Their use to develop
training has become less than optimal given the new requirements and challenges Soldiers face
today (see Epilogue).
New technologies inevitably displace and make what they replace obsolete. The new
characteristics of warfare and the capabilities required for success in FSO are making traditional
notions of training and education, if not obsolete, at least unproductive. The dependence on tasks
and drills as the basis of designing, executing and evaluating training is seriously at odds with the
requirements of preparing Soldiers and units for FSO. The drills and standards that have stood our
Army in such good stead since Valley Forge, through the mass mobilizations of two world wars,
and eventually creating the irresistible force of Desert Storm have to be re-examined in light of
their influence on the thinking of their practitioners.
15.2.7 A Natural Advantage
American military forces already enjoy an unprecedented advantage over allies and competitors
without doing any training or education. The cultural implication of living free in the United
States and all that it entails brings tremendous benefit to an environment where adaptability,
creativity and versatility are required (see Chapter 4). It is unfortunate then that much of the
Armys training culture unintentionally negates this unique American characteristic. In
operational environments, some leaders that acquired their skills in a more stable place and time
cannot abide friction, organized chaos, or questioning attitudes and impose requirements that
diminish the qualities that provide our Soldiers and units a qualitative advantage over any
adversary. Desires to impose artificial controls, minimize the incidence of chance, and eliminate
the threat of risk serve only to create a climate where doing nothing is the safest course of action.
On the contrary, leaders get the best performance from American Soldiers when they give
mission orders and allow themselves to be surprised with the results. Such a climate can only
survive when nourished by trust and confidence. The conditions of creating that climate must
begin early and be reinforced often.
Quite simply, the biggest problem getting in the way of creating this climate is how we train.
Baldly stated, task and standards-based training is insufficient to develop Soldiers or to encourage
their natural adaptability. If anything, unthinking and rigid adherence to training to tasks and
standards exclusively seems based upon unfounded assumptions about the role of discipline and
standards on the modern battlefield. It also assumes that the tasks are the right tasks and the
standards are applicable to battlefield conditions, right now. Our training should seek to exploit
the inherent versatility and adaptability of our Soldiers. Confident leaders can do this.
15.2.8 Task Specialization or Generalized Competency
Task-centric and standards based training and education does a superb job of presenting fact
based and procedural information. It has served our Army well for almost 100 years. It even
assures a highly reproducible occurrence of and evidence of learning through performance
Leader Development
250
measure validation of task execution. Without additional intervention, such training and
education fails to provide the Soldier opportunity to organize newly acquired information or to
link a new understanding to subsequent action. Trainers frequently observe this with newly
trained (and technically task proficient) Soldiers confronted with typical problems in mission (or
simulated but highly realistic) environments. Their learning was superficial. While the Soldier
had amassed information, lack of action and reflection with that information rendered it inert.
Many Army schools and courses suffer the same defect, and the generational evolution of the
systems approach to training made it that way.
Integrating OBTE principles in training and education is a way to meet the Armys need for
broadly gauged generalists able to acquire rapid mastery of specific skills. From an OBTE
perspective, tasks are always taught with a contextual nature that helps explain why tasks should
be learned rather than practicing the task in isolation (see Chapter 3). Designing and executing
learning activities use standards as the baseline performance measure and when executed
appropriately, student performance usually exceeds the standard, often by a significant margin.
More importantly, because their learning includes action, reflection, feedback, and practice,
Soldier knowledge is activated rather than passive or inert. Such knowledge enables Soldiers to
leverage what they know so that they respond positively to the conditions of an altered situation
(see Chapter 4). This is the essence of adaptability in a military context.
The realization that it is impossible to train Soldiers for every task they might need to perform is
perhaps the most powerful inducement to transform Army training and education. The training
system as it exists today will prove resistant to this reality (see Chapter 14). The idea of training
only the most important tasks is commendable but probably unmanageable. The nature of critical
task lists and how they are compiled almost guarantees that the lists will grow rather than remain
small and focused. An unintended consequence of task lists is that accounting for the tasks (with
their associated standards) becomes the objective rather than achieving the more desirable
intangible attributes that will prove valuable to the Soldier and their unit.
Current efforts with automating the Combined Army Training Strategy (CATS), the Digital
Training Management System (DTMS) and the Training Development Capability (TDC) as tools
designed to improve training management and development can potentially make the problem
worse and harder to circumvent. Training tasks derived from a DA approved unit-type Mission
Essential Task List (METL) depend first upon the tasks chosen being right enough. In terms of
missions for units as designed, the chances are good that missions and the task groups will be
correct. Where it becomes murky is in the selection and specification of collective and individual
tasks as determining elements of preparing units and Soldiers for FSO. Rather than starting by
asking what tasks must my Soldiers be able to perform if might be better to ask what
capabilities do I want my Soldiers to have at the end of training (see Appendix C; Perry &
McEnery, 2009).
The key perhaps is using capabilities to determine what is important. Describing desired
capabilities in terms of outcomes allows commanders the flexibility and latitude to make
judicious selection of the tasks that will, aggregated, generate capabilities. Commanders, in dialog
with their employing commander, can defend their training decisions. Training managers then use
TDC and CATS to determine the resource requirements and the means available to design a
training program to meet the commanders intent. This approach effectively reverses the way
most people view training management. It also places responsibility and accountability for the
preparation of Soldiers and units where it belongs, with the commander (see Chapter 3; Appendix
C).
251
Cornell d-Echert
Trying to manage training by accounting for all the individual tasks that need to be trained to
enable the collective tasks that are essential to mission tasks is too often a frustrating exercise.
Combat experienced leaders do not want to design training by task selection because missions are
not a series of tasks executed in some sequence and success does not always depend upon exact
execution of performance measures. Soldiers do not want to train to perform tasks, they want to
be capable and accomplish the mission. As an Army, we train Soldiers to fight and win the
nations wars or to be successful in any activity charged to complete. Training and education that
prepares Soldiers who are confident and will take initiative and not solely conditioned to respond
to particular situations will certainly prove more resilient in more occasions. They will be
responsible to others, be accountable for their actions, and not overwhelmed when faced with a
new or unexpected situation (see Chapter 3). Perhaps, instead of trying to develop an allencompassing listing of tasks designed to prepare forces for anything there is a better way to
state what results training should produce. The METL can be used to assure a level of proficiency
in core competency tasks because they may be required. Without exception, though, training
guidance should stipulate that Soldiers have the capacity to adapt to any circumstance without
loss of capability. Then leave it to commanders to develop a plan to accomplish this based on
their own units conditions and situation.
15.3 Conclusion
The Army will always go to war, or support operations, with what it has, as it is. That is why
flexibility, adaptability and mental agility are highly valued traits. Many of the recent demands
that the Army increase its ability to adapt are somewhat short sighted. From a broader
perspective, seeking only to promote adaptability will establish unnecessary limits for what could
and should be a much more ambitious goal. Adaptability alone suggests reaction to events that
have already occurred, useful to a point. It would be far better to develop Soldiers and leaders
with enough training, education and experience to be able to anticipate, and be ready for what
might come next. This seems to be the real essence of our operational imperative to seize, retain
and exploit initiative in any military operation.
As the Army seeks ways to achieve balance, especially in sustainment of core competencies
while maintaining current operational requirements, there will be arguments for and against
objectives at each end of the balance beam. The ALDS, enabled by implementing OBTE
principles, can serve as a counterpoise that makes the arguments about training versus education
lose power.
The training system needs a radical course correction without delay. Elements of radical change
thread throughout both the ALDS and the observations of OBTE. Fundamentally, the Army can
no longer think in terms of training the force. In the past we could train the force to fight and
prevail in a war. Today we must prepare for anything. That preparation demands that we develop
Soldiers of character, imbued with the warrior ethos and empowered with a service ethic (see
Chapter 3 and Epilogue). Such development is not the sole province of schools and centers, nor is
it left to unit leadership. It is an all-encompassing task. The time for arguments about what should
be training and what should be education is behind us. Similarly, the notion that we train Soldiers
and develop leaders is short sighted. All Soldiers are potential future leaders and they are equally
deserving of development.
An inescapable reality is that any new solution must be unequivocal on training resources and
time. We cannot ask for more. Leaders will be compelled to find ways to do more with less, an
unfortunate and impractical oxymoron. Rather than doing more with less, it will be better for the
Army when leaders with the moral certitude are encouraged, supported and rewarded to do less,
252
Leader Development
better. Examples of such leadership will diminish systemic resistance and encourage support for
such a change (see Chapter 14).
Soldiers and leaders require training, education and experiences that prepare them to be
successful in the uncertainties and ambiguities of operating in and across the full spectrum of
operations. The training should provide opportunities for mastery of fundamental and essential
skills. The education should provide opportunities to relate those skills to myriad possibilities of
threats and challenges. Experience should provide opportunities to translate information into
knowledge of human dynamics, materiel shortcomings, and the vagaries of translating decision to
action in time and space (see Chapter 5). When using the principles of OBTE to analyze, design,
develop and execute development of Soldiers, all these things occur.
Whether applied to unit pre-deployment training, or training in the institutional domain, there is
enough evidence of the results of OBTE to state that the benefits are real. Leaders gain
confidence in their units, Soldiers gain confidence in themselves. Soldiers are more confident
because:
Having done something with what they learn, they know what they can do
They are practiced in solving relevant problems, tactical and otherwise
They know what they must learn
They understand the value and utility of initiative
They become accountable for their own performance and as part of a team
Ultimately, this is why we are compelled to transform Army training to the broader requirements
of preparation and development. Our Soldiers, trained, educated and experienced, can be ready
for anything. They provide America an unmatchable asymmetric advantage. Even though our
technology continues to outpace both allies and potential adversaries, technology alone will never
be enough to assure operational dominance. Outcomes Based Training and Education, linked with
the Army Leader Development Strategy is the new thinking needed to produce those highly
competent individuals and teams.
15.4 References
Defense Science Board Task Force (2003). Training for future conflicts. Washington, DC: Office
of the Undersecretary of Defense for Acquisition, Technology and Logistics.
Fastabend, D., & Simpson, R. (2004). The imperative for a culture of innovation in the U.S.
Army: Adapt or die. Army Magazine, 54(2). Retrieved January 2010 from
http://www3.ausa.org/webpub/DeptArmyMagazine.nsf/byid/CCRN-6CCSBU
Headquarters Department of the Army (2002). Training the Force. (Revised 2008, Training for
Full Spectrum Operations). Field Manual No. 7-0. Washington, DC: Headquarters
Department of the Army.
Headquarters Department of the Army (2009). Army training and leader development. Army
Regulation No. 350-1. Washington, DC: Headquarters Department of the Army.
Quinkert, K., Morrison, J., Fletcher, J., Moses, F. & Roberts, E. (2007). The Army Science of
Learning Workshop (RN 2007-02). Washington, DC: Army Research Institute for Behavioral
and Social Sciences.
Salmoni, B. (2008). Pedagogy for the long war: Teaching irregular warfare. Quantico, VA: U.S.
Marine Corps Training and Education Command.
253
Cornell d-Echert
United States Army Training and Doctrine Command (1999). Systems approach to training
management, processes, and products. TRADOC Regulation 350-70 (under revision). Fort
Monroe, VA: Training and Doctrine Command.
United States Army Training and Doctrine Command (2009). Army Leader Development
Strategy. Ft. Monroe, VA: Training and Doctrine Command.
254
Epilogue
Epilogue. Integration of Leadership, Education, Training, and Self-Development
Gary Riccio
The purpose of this final chapter is to summarize the topics of discussion that, by the end of the
AWGs scientific engagement in OBTE, were most urgent and vibrant with respect to
implications for social and programmatic decision-making about Army training, education, and
leader development.
Toward Values-Based Standards for Army Doctrinal Requirements
To date, the proponents and participants in OBTE have stressed that we should consider all the
outcomes of specific learning events that are relevant to preparation of Soldiers for Full Spectrum
Operations (FSO), not just the outcomes that are intended or explicit in a Program of Instruction
or Training Support Package. This has led us to the conclusion that OBTE provides an
overarching vision for training, education, and leader development that makes all these endeavors
relevant to FSO and to operational commanders in theater (see Chapters 3, 4, and 5; see Figure 1).
The principles and practices of OBTE follow, arguably in a deductive fashion, from this vision of
relevance. While not especially surprising theoretically, at least in hindsight, this conclusion has
important practical implications. Most importantly, it suggests that OBTE is both a manifestation
and response to existing Army doctrine. The cascading implications are outlined below.
Figure 1. Triadic balance and equilibrium among intangible personal attributes

and the associated scaffolding for capabilities of the human dimension in Full
Spectrum Operations and competence in trans-extremis environments (e.g.,
transitions between strategically significant combat and noncombat operations).
See Chapter 3, section 3.2.3.
255
Riccio
The current notion of OBTE as vision for learning and development suggests that we can
approach the relationship between immediate learning objectives and long-term developmental
outcomes from the opposite direction, in a sort of reverse planning from what the Army expects
of its Soldiers coming out of programs of training and education. The intangibles such as
confidence, initiative, and accountability help bridge the gap between behavior that is
immediately observable and behavior that is meaningful and efficacious in FSO. But what are the
ultimate outcomes? Our current belief is that OBTE does not suggest a fundamentally new set of
concepts to be added to doctrine. Given the vision and the purpose of preparing for FSO,
outcomes should be derived from doctrinally accepted concepts such as Army Values and
Warrior Ethos or at least they should be traced to them.
One of the strengths of OBTE is that its implementation can be verified. OBTE thus can help
translate existing doctrinal requirements such as Warrior Ethos and Army Values into practices
that actually are observable and around which programs of instruction can be designed and
developed. By showing that intangibles can be observed through their behavioral manifestations
(a common proposition in the physical, biological, and behavioral sciences) and that measures
can be developed for observable behavior, we also can claim that standards can be identified for
such concepts. The logical conclusion of this line of argument is that a rigorous scientific
approach to development, verification, and validation of OBTE provides a way to identify
standards for value-based requirements in existing doctrine for which there currently are no
measures or standards.
A more axiomatic formulation of this line of argument is:
(1) The most fundamental aspects of doctrine are the requirements that are explicit or implicit in
Army Manuals, Regulations and associated documents. Concepts like Army Values and
Warrior Ethos are among the most common requirements stated explicitly in Army doctrine.
(2) The Army is a standards-based organization but there currently are no doctrinally codified
standards for some of the most important requirements in Army doctrine (e.g., as Army
Values, Warrior Ethos).
(3) Verifiability and validity are two universally accepted criteria in capabilities development and
quality assurance that all standards should address directly. Verifiability means that one can
observe something tangible that relates directly to the standard (e.g., one can verify that one
is doing what one says one is doing or should be doing). Validity refers to the relevance of
the standard to the requirement (e.g., does training to the standard help one meet the
requirement).
(4) The validity of all standards (e.g., current marksmanship qualification standards) is intangible
with respect to some of the most important requirements in the Army (i.e., Army Values,
Warrior Ethos). The relationship is inductive not deductive. Nevertheless, not all standards
and inductive relationships are created equal. One should develop, refine, and utilize
standards for which confidence in validity is greatest (i.e., for which evidence about validity
has the most depth and breadth).
(5) To date, OBTE is the only approach to training, education, and leader development that has
generated measurespotential standardsfor some of the most important requirements in
the Army (e.g., Army Values, Warrior Ethos). While there is evidence for the validity of
these potential standards, the more important point is that OBTE has shown that such
standards can be developed in principle.
256
Epilogue
(6) Tasks, conditions, and standards are particular solutions to particular requirements. The most
desirable solutions are ones that can evolve though refinement, upgrades, or other adaptations
to a changing context for the associated requirements over the life cycle of the solution.
Verification and validation fosters continued viability and relevance of a solution by
providing feedback about what to sustain and what to improve.
(7) OBTE arguably is the only approach to training, education, and leader development that has
generated a methodology for integrated verification and validation.
The application of OBTE to doctrine embodies a broader conceptualization of standards, one that
connotes a quality of something that is attainable, not just standardization and evaluation with
respect to norms. The implications about standards and their employment reveal another way that
OBTE transcends particular instructional methods. Irrespective of the methods utilized in training
and education, there is a need for further development of standards for values-based requirements
in current Army doctrine. The validity of such standards cannot depend critically on any
particular methodology even though, in particular situations or under particular conditions, some
instructional methodologies may be more effective than others in achieving values-based
standards.
Nested Standards and Quality Assurance
The elaboration and refinement of values-based standards provides critical guidance to instructors
and their chain of command but it is not sufficient. More guidance is needed with respect to how
such standards can be achieved. Without clarity about the means to the ends, recognition of gaps
or shortfalls in achieving standards will not be actionable; there will not be a way to assure that
achievement of standards will be sustained let alone to identify ways to improve instruction with
respect to the achievement of standards. In this respect, the most important contribution of the
scientific engagement in OBTE to date has been the development of formative measures for
instructors (e.g., Chapter 2, Appendix A) and the scientifically based approach to verification and
validation with which the measures are associated. The measures are the only way to verify that
OBTE is being practiced as it has been defined and thus to validate that it can be effective. They
provide a rigorous framework for the development of values-based standards that are achievable.
More generally, the most important realizations about OBTE are that there are requirements in
Army doctrine for which solutions are undefined or unverifiable, that measurable standards can
be developed for them, that values-based standards need not displace existing knowledge-based
or skill-based standards, that the achievability of values-based standards can be verified with
formative measures for instructors, and that achievement of values-based standards need not
require additional resources for training and education. The logical implication of these
realizations is that multiple standards of different kinds should be addressed in every learning
event. Note that the implication is not that values-based standards are merely a higher level
quantitatively on measures associated with existing standards. They are standards of a
fundamentally different kind.
The formative measures developed for OBTE (Appendices A and B) provide a menu from which
instructors, and other stakeholders with an interest in quality assurance, can select a manageable
number of measures for planning, executing, and learning lessons from a particular learning
event. None of these measures precludes assessment with respect to knowledge-based or skillbased standards, although they certainly can inform priorities and tradeoffs with respect to an
exhaustive list of tasks, conditions and standards that are not practicable (Appendix C). The
257
Riccio
evidence from this investigation supports these claims, in essence, that multiple measures of
different kind can be used in the same learning event and that values-based measures in particular
can inform decision-making in planning and executing instruction (e.g., Chapters 8, 9, 13). This
realization offers great promise for packaging training and education in more efficient ways (e.g.,
Perry & McEnery, 2009). OBTE, and specifically the associated formative measures, provide
clues about how to link tasks to achieve multiple goals at the same time. Thus, in addition to
showing that values-based standards dont impose an additional resource load, formative
measures for OBTE may even show a path to achieving more with fewer resources.
It also is clear that there is much to be learned about the efficient and effective use of multiple
measures. Importantly, given the foundation provided by this investigation, increasing clarity on
such critical issues now is largely a matter of paying attention to the most important outcomes of
training and education whether intended or not, focusing collective priorities in a program of
instruction within a command intent that addresses the most important outcomes, and
documenting progress (Chapter 11; Appendix C). It is noteworthy that the value of these actions,
and the clarity they bring, is not limited to programs that adhere to the principles and practices of
OBTE. The principles and practices of OBTE will, however, bring greater value insofar as they
were developed pointedly to address values-based outcomes (Chapters 1-5).
Key constructs in OBTE are intangibles such as confidence, initiative and accountability. They
are assumed to have causal potency with respect to the developmental outcomes that motivated
OBTE. The trustworthiness of this assumption is revealed by the extensive dialectical reflection
and analysis we conducted during measure development, field-based observation, and the
development of grounded theory for the practical wisdom of Soldiers as leaders and instructors
(e.g., Chapters 2-5). The intangibles emphasized by OBTE relate directly to values-based
doctrinal requirements such Army Values and Warrior Ethos (see e.g., Chapter 3; Appendix D).
The identification and promulgation of values-based standards will require additional attention to
outcomes that are influenced by these intangibles (cf., Bandura, 1997; see Figure 1). There is a
need to identify and refine the understanding of such outcomes as effects on students that can
transfer to different situations including performance in Full Spectrum Operations (cf., Leibrecht,
Wampler, & Pleban, 2009) if not to other aspects of a Soldiers life (see Chapter 4, section 4.3.2;
Chapter 5, section 5.2).
We recommend that development of values-based standards consider the outcomes defined or
refined locally in collaborative decision-making among the instructor cadre within the local
commanders intent (Chapter 3, section 3.2.7; Chapter 14; Appendix C). Such outcomes and
associated measures currently are in the early stages of decentralized development across various
programs (Haskins, 2009; Perry & McEnery, 2009). Outcomes can be addressed systematically
across programs and within a time scale that is short relative to the refinement of doctrine.
Doctrine can assimilate this localized innovation and translate it into guidelines that stimulate and
focus innovation rather than discourage it. Within such guidelines, instructors and their chains of
command are in the best position to identify observable and achievable outcomes with the most
relevance to stakeholders for their programs of instruction. Collaborative identification of
outcomes is a means for instructors to develop and direct internal sources of motivation for good
instruction. It is a means for the instructional organization to operate coherently with respect to
the commanders intent and to achieve a command climate within which unnecessary unintended
friction is readily identified and eliminated.
Epilogue
258
Figure 2. Programmatic view of the development and employment of values-based

standards as well as associated instructional principles, practices, and measures (i.e.,
OBTE). Solid lines represent direct programmatic influences. Dotted lines connote
informal or indirect programmatic influences. Heavy solid lines connote direct
psychological causes and effects of instruction on students. Compare with Figure 8 in
Chapter 3.
To summarize key points about multiple standards:
Standards for values or ethos can be addressed in the same learning event as existing
standards.
Tasks, conditions, and standards can be designed to allow multiple standards to be

addressed in the same learning event, conditions permitting.
Addressing multiple standards in the same learning event suggests the potential to
accomplish more with the same resources or less.
Standards should reflect outcomes that are developed in decentralized innovation across
programs of instruction.
259
Riccio
Needs and Opportunities for Staff & Faculty Development
A Role for Science and Measurement

Several promising directions for instructor education in OBTE are outlined below. While the
recommendations are based on established practices, they are nuanced in the context of OBTE.
The nuances derive from the intent to be directive about process while not being directive about
content (see e.g., Chapter 3, section 3.2.7; Chapter 14; Appendix C), to foster flexibility and
adaptability in instructional planning and execution, and to strike a balance between the science
of teaching and the art of teaching. The subtlety we seek was captured well by William James in
his talks to teachers over a century ago:
The teachers of this country, one must say, have its future in their hands. The
earnestness which they at present show in striving to enlighten and strengthen
themselves is an index of the nations probabilities of advance in all ideal
directions [p. 3] You make a great, a very great mistake, if you think that
psychology, being the science of the mind's laws, is something from which you
can deduce definite programs and schemes and methods of instruction for
immediate school-room use. Psychology is a science, and teaching is an art; and
sciences never generate arts directly out of themselves. An intermediate inventive
mind must make that application, by using its originality [p. 7] A science only
lays down lines within which the rules of the art must fall, laws which the
follower of the art must not transgress; but what particular thing he
shall positively do within those lines is left exclusively to his own genius [p. 8]...
many diverse methods of teaching may well agree with psychological laws [p.
9] To know psychology, therefore, is absolutely no guarantee that we shall be
good teachers. To advance that result we must have an additional endowment
altogether, a happy tact and ingenuity to tell us what definite things to say and do
when that pupil is before us. That ingenuity in meeting and pursuing the pupil,
that tact for the concrete situation, though they are the alpha and omega of the
teachers art, are things to which psychology cannot help us in the least [p. 9]
We know in advance, if we are psychologists, that certain methods will be
wrong, so our psychology saves us from mistakes. It makes us, moreover, more
clear as to what we are about. We gain confidence in respect to any method
which we are using as soon as we believe it has theory as well as practice at its
back. [p. 11] (James, 1899/1907).
A vast amount of pedagogically relevant research has been conducted in the century since
Jamess remarks but his advice is still relevant. Our advantage today is that psychology and other
academic disciplines are in a better position to help instructors become more keenly aware of
where certain methods will be wrong, to save us from mistakes, and to give us confidence in
respect to any method which we are using as soon as we believe it has theory as well as practice
at its back. We were mindful of Jamess advice from the beginning of the investigation, and it
thoroughly infused our program of research. In general, through our interactions with the
progenitors and stakeholders of OBTE, we diligently sought to identify the tact and ingenuity to
tell us what definite things to say and do when that pupil is before us. Concurrently, we
identified relevant science and scholarship that lays down lines within which the rules of the art
must fall, laws which the follower of the art must not transgress (see e.g., Chapters 2 and 3).
Without something like the formative measures for instructors developed in this investigation, for
example, there is no principled basis for instructor education with respect to values-based
Epilogue
260
requirements in current Army doctrine. The student measures that were added toward the end of
the scientific engagement (Chapter 12; Appendix B) also can provide formative feedback to
instructors and to students about the efficacy of instructor behavior. They are measures that show
whether the instructor is having some effect on the student and what some of those immediate
effects are. However important, they are merely indicators (i.e., efficient causes) that outcomes
are likely to be influenced by instructors, especially if high scores are obtained on the instructor
measures (see Chapters 8 and 9). These can be dependent variables in a study of OBTE but they
should not be confused with outcomes. With regard to outcomes emphasized by OBTE, they are
intervening variables. The grounded theory outlined in Chapters 3-5 provides a foundation for
further research into measures that provide a more complete picture of the impact instructors and
their programs of instruction can have on individual learning and development with respect to
values-based outcomes.
Toward Best Practices in Instructor Education
OBTE illuminates both needs and opportunities for development of staff and faculty associated
with Army training, education, and leader development. It is important to note that none of these
actionable implications are uniquely required by OBTE. They apply to any approach to training,
education, and leader development, in principle, and they apply to current practices in these areas.
The need for increased resources often is associated with OBTE merely because practitioners
have a keener awareness about the best use of resources. They have greater motivation to achieve
more or at least they have a more coherently directed motivation. The insights of OBTE may
simply stimulate reconsideration about the distribution of resources. It may also reveal that an
increase in resources devoted to training and education, such as instructor education, can have a
disproportionately high increase in return on the investment. Again, this is not to be confused
with a requirement of OBTE for more resources.
Field-based training. The AWGs Combat Applications Training Course (CATC) is an effective
way to familiarize instructors with OBTE (see Chapter 6). Such opportunities for experiential
learning are an important element of instructor education but CATC is not sufficient to ensure
that instructors will understand the general principles of the new approach. Instructors should be
familiarized with the application of OBTE to different skill sets or knowledge sets (see Chapter
11). We recommend use of formative measures (see e.g., Chapter 2) in such courses to assure
quality and for collaborative reflection with participants in the course. The measures would help
student-instructors appreciate the ways in which the general principles and practices of OBTE are
manifested in particular instructional strategies and events. They would help see beyond the
particulars of the learning experience to reveal the deep structure of learning and instruction.
Seminars and workshops. The AWG has provided seminars on a regular basis to stakeholders in
OBTE (AWG, 2008a,b, 2009). These events have been very useful in helping instructors and
their chain of command come to understand the vision, purpose, principles, and intended
outcomes of OBTE even though the events are only several hours in duration. We prefer a
workshop format with two half-day periods on consecutive days given that discussion about
OBTE generally simulates subsequent reflection with noteworthy personal implications. We
expect that workshops will be more successful when they are coupled with experiential learning
(e.g., CATC). This assumption is consistent with our informal observations during this
investigation.
We have recommended that workshops include multiple sessions in which the topic of discussion
is the experience of instructors (Sidman, Riccio, Semmens, et al., 2009). Toward that end, it
would be valuable to have a corpus of narrative vignettes that provide concrete examples of an
261
Riccio
instructors thinking and behavior. The vignettes could describe instructional events at a level of
detail that provides enough substance and enough left unstated to stimulate meaningful
collaborative reflection on a time scale commensurate with the time scale of the events described
in the vignettes. The formative measures for OBTE (e.g., Chapter 2) provide an excellent
framework both for generating such a corpus and for discussing the vignettes. In a sense,
instructional vignettes can provide a way for the student-instructor to get inside the head of a
more experienced instructor as conditions are observed, as in-stride decisions are made, and as
consequences unfold. We believe this would help demystify good instruction. It also would have
the added value of showing student-instructors how to use formative measures for quality
assurance and self-development.
Collaborative decision-making. We have stressed the importance of participation of instructional
cadre and their chain of command in making decisions about outcomes and associated measures
of interest in their own programs (Chapter 3; section 3.2.7; Chapters 7 and 14; Appendix C).
These collaborative discussions have the added value of helping individual instructors develop a
broader and deeper understanding of OBTE. It is another way of learning from peers and
contributing to that learning. It provides more direct connections between general theory and
immediate practice.
Multimedia products. An advantage of a field-based course is that a student-instructor can see
examples of good instruction. One of the limitations of this experiential learning is that it can be
difficult to differentiate the essential from the incidental, to appreciate the general principles and
practices in the particulars in which they are manifested. Exposure to a variety of instructors and
applications of OBTE can be a powerful and effective method to help students generalize and to
learn more deeply. Video records would be a simple, low-cost, and highly accessible method for
student-instructors to be exposed and re-exposed, on demand, to a variety of instantiations of
OBTE. Moreover, established methods in multimedia instruction could blend aspects of fieldbased training and seminars. We have developed multimedia materials for instructor education
that utilized formative measures for values-based requirements to guide viewers through video
segments of instructional events (Bruny, Riccio, Sidman, Darowski, & Diedrich, 2006; cf.,
Straus, Shanley, Burns, Waite, & Crowley, 2009).
There is added value to multimedia methods of instructor education that address a recognized
need. Student-instructors and their leadership sometimes express skepticism that they are capable
of the kind or level of instruction demonstrated by master instructors or by individuals with
extraordinary credentials as Soldiers. While OBTE or any instruction in the Army can benefit
from extraordinary experience and expertise in Soldiering, we believe that OBTE does not require
such levels of mastery. Common instructors have the potential to improve their instruction with
respect to the principles and practices of OBTE (e.g., Chapters 8 and 9). Nevertheless, it would be
valuable for student-instructors to be exposed to instructors like me in situations like mine. This
need not be limited to instructors. Multimedia materials also can highlight the role of an effective
Brigade or Battalion commander, a tipping-point First Sergeant or Company commander, or other
influential leaders.
Distributed collaboration. Collaborative decision-making and peer-to-peer sharing of lessons
learned need not be limited to forums in which all participants are physically co-located (Chapter
11). There will be increasing opportunities for web-based social networking for the foreseeable
future. Such capabilities should be leveraged for formal or ad hoc instructor education. One of the
current needs in this context is finding outside experts who can help relative novices find the right
information in the right amount and in the right amount of time. More generally, there is a need to
help individuals share their respective expertise in the context of common or convergent interests.
Epilogue
262
Internet portals that facilitate such decentralized ad hoc task organization are a promising
potential solution to this need (Mikroyannidis, 2007; Riccio, Lerario, et al., 2006). Web-based
utilities that enable user generated content and its distribution are transforming use of the web
as well as the perception of what the web affords. This can lower barriers to entry into
communities of practice and thus extending the opportunities to engage in and benefit from
volunteerism (McKee, 2007; Putnam, 2001; Thoits, & Hewitt, 2001).
\
Textbooks, pamphlets, and pocket guides. Textbooks are how novice instructors outside the
military teach with little or no background in teaching and even sometimes with little background
in the subject matter. One generally can use a text however one wants but at least it provides a
framework and reference for one's local and momentary initiative. If the textbooks dont help
instructors do this, they are not chosen or used. In any case, the authors get useful feedback about
the value of their contributions. This market-based solution does not necessarily require monetary
gain by the authors and may not even require monetary transactions. Other motives and
transactions can connect volunteers with people who value their special expertise. Recognition,
inclusion, and opportunities for impact can be powerful motivators (Maslow, 1943, 1968).
Content generated by volunteers would require review and perhaps editing by TRADOC but this
demands fewer resources than generating content. With regard to content for instructor education,
reviewing and editing would be simplified significantly by the formative measures developed in
this investigation (see Chapters 2, 7, 12).
In the context of OBTE, textbooks need not and should not tell instructors what to do. They
should provide guidance about how to become a more effective instructor. We believe that an
excellent source of guidance would be a corpus of narrative vignettes that provide concrete
examples of an instructors thinking and behavior as suggested above in the context of workshops
for instructor education. In any case, formative measures for instructor behavior are a valuable
framework within which to seek, provide, and interpret guidance about instruction. Given that
such guidance is not a script for everything an instructor does in a course, it can be concise.
Mnemonics also can be produced on pocket cards for instructors who have been familiarized with
the guidance represented in the cards. Pocket cards reflect the value of guidance that can be
accessed in situ. This is especially important for novices who may not have fully assimilated
lessons learned or retained knowledge of best practices or who may be overloaded during an
instructional event (Freeman, Jason, Aten, et al., 2008).
Portable electronic aids. Elaborate frameworks and context-dependent guidance can be
extremely valuable in situ. Hand-held electronic devices are becoming increasingly common aids
to performance in everyday tasks as well as on the job. As such, they are a natural extension to
pocket cards that are at once ubiquitous and excessive in the Institutional Army and the
Operational Army. Electronic aids are interactive, thus, they allow the presentation of information
to be as simple or as complex as needed. Clever design of menus and the underlying structure
also can help a user find information without knowing exactly what they are looking for. More
generally, use of an electronic aid and the information it provides can be context dependent.
Moreover, read-write capabilities enable hand-held electronic devices also to replace the green
book that is a common tool for collecting notes and observations in the field, especially during
instructional events. The capability to record noteworthy events within a framework of formative
measures can facilitate both instruction and learning. It can greatly facilitate collaborative
reflection either in formal After-Action Reviews or in ad hoc groups after a noteworthy event (see
Chapter 3). Such devices also offer the added value of facilitating the capture and sharing of
lessons learned with peers and stakeholders (see e.g., Riccio, dEchert, Lerario, et al., 2006).
263
Riccio
Critical Considerations for Further Scientific Investigation
The Necessity of Long-Term Studies

In our view, all the debate about OBTE reduces to lack of agreement about the impact of training
and education, in general, irrespective of approach. By way of analogy, any competition that
would pit OBTE against some other approach would be impossible to initiate until all parties
agree on the rules of the game. The various parties in the debate about OBTE are not even close
to being on the same page about what outcomes matter. In fact, the use of the term outcomes
across the Army, and more broadly in education, has become so muddled it may be wise to
consider dropping the term altogether (see Chapter 3). In any case, in the terminology of
empirical science, the most important and difficult questions about OBTE involve the choice of
dependent variables. If we can achieve some agreement on what effects to measure, then we can
tackle the tough decisions about independent variables (i.e., interventions, treatments) or
comparisons (e.g., demographics and characteristics of samples, groups, situations).
OBTE emphasizes long-term developmental outcomes consistent with values-based requirements
in existing Army doctrine. Scientific inquiry can play an important role in identifying behavioral
manifestations (i.e., outcomes) of the values developed in Soldiers as well as antecedents that
reflect progress in the development of such values. Short-term investigations such the ones
reported in this manuscript are valuable but they are not sufficient for overarching programmatic
decisions about OBTE. The value of the empirical work summarized herein is in revealing
proximate effects of any approach to training and education and, more importantly, in verifying
the implementation of any approach (Chapter 1). It cannot, by definition, reveal long-term effects
such as inculcation of values described in Army doctrine. Long-term studies are warranted if not
required if science is to help the Army meet its existing requirements. Moreover, theory
development is necessary to explore connections between short-term and long-term effects (see
e.g., Chapters 3-5). Without theory, it is difficult to take any empirical investigation seriously,
and it certainly is impossible to draw any actionable conclusions based on the evidence. The
broader and deeper the theory, the more confidence one can be about interpretations of the
evidence, and the more responsible one can be in making recommendations based on the
evidence. The seriousness of the tasks currently faced by Soldiers in FSO deserves theory that is
commensurate in depth and breadth.
OBTE claims that, when there is singular focus on a task and prescribed conditions associated
exclusively with a knowledge-based or skill-based standard, there is an unintended reliance on
short-term memorization and conditioning. This may result in a reasonable level of performance
in the learning event but our claim is that it also results in an unintended negative impact on longterm outcomes. In particular, it can be counterproductive with respect to attributes and values
necessary to perform in ambiguous situations or across the variations implicit in Full Spectrum
Operations (FSO).
OBTE claims that, when leadership at all levels seeks to create a command climate that strongly
encourages the development of values and attributes emphasized by OBTE, the instructors will
naturally begin to identify opportunities to achieve the knowledge-based or skill-based standard
established for performance while also developing a Soldier with respect to long-term outcomes.
Further, we argue that this will lead to the discovery of new and better ways to instruct with
respect to short-term objectives and long-term outcomes within current resource constraints; that
is, it will lead to a more efficient use of resources. Such adaptability and ingenuity is exactly what
is needed in FSO.
Epilogue
264
In essence, OBTE claims that the Army is already conducting outcomes based training and
education, in that there is always long-term outcomes from every event or encounter with a
student. A singular focus on knowledge-based or skill-based standards, however, can lead to
negative outcomes. These are empirical questions, and this is where science can help. These
issues suggest that further research would do well to include social psychological and even
econometric perspectives (see Chapters 4 and 5) more deeply than we were able to address in the
current investigation which initially drew mostly on industrial-organizational psychology and the
experimental psychology associated with human perception and performance, cognition and
learning, and human development (see Chapter 3). Further research should be re-conceptualized
at a higher level in terms of inter-temporal risk-benefit or cost-benefit relationships, for example:
Benefit: What is the impact of any program of training and education on outcomes that
matter for mission effectiveness in Full Spectrum Operations?
Risk: Does OBTE have a positive or negative effect with respect to near-term learning
objectives in a particular program of instruction?
Risk-Benefit Analysis: Does impact of OBTE on outcomes justify additional risk, if any,
to near-term learning objectives?
Cost: Does OBTE require additional costs for materiel or non-materiel resources or
redistribution of such resources to maintain overall cost?
Cost-Benefit Analysis: Does impact of OBTE on outcome justify additional cost or

redistribution of costs, if any, of its adoption?
Note that risk-benefit and cost-benefit analyses are relevant only if there is a tradeoff, that is, if
outcomes come at a risk relative to near-term learning objectives or at cost in additional resources
used. With improved instructional design informed by experience with OBTE, we believe that
there will be no tradeoff. This seems to be the case at sites where instructor cadre and their
leadership have collaboratively re-designed their courses around OBTE (Haskins, 2009; Perry &
McEnery, 2009; Appendix C).
False Dichotomy of Objective-Subjective
Measures and standards that relate to long-term developmental outcomes will look different from
measures typically used to assess performance with respect to knowledge-based and skill-based
standards. There is some concern about subjectivity in measures associated with values-based
standards and, in particular, there are some assumptions about unreliability relative to objective
measures associated with more common standards. As a synonym for reliability and validity,
objectivity is a worthy pursuit and one that is convergent with rigorous subjectivity, especially
inter-subjectivity (Bridge, 2000; Dienes & Perner, 1999; Neisser & Fivush, 1994; Reed, 1996;
Searle (1997). The dichotomy is largely an illusion, however, for a variety of reasons.
Consider standards being applied in training today that are assumed to be objective. For any
standard, there are numerous ways to get to that standard that are not specified or precluded in the
associated regulations. Some of the unspecified ways of approaching a standard would be
interpreted as cheating, others are not so clear because of judgment on the spot when conditions
are not as assumed in the standard. In either case, the result depends critically on subjectivity. In
addition to this practical constraint on objectivity, there may be legitimate questions about the
putative validity of the standard with respect to the operational task to which it ostensibly is
265
Riccio
relevant. Potential violations of the assumption about validity generally are an even more serious
challenge to the common connotations of objectivity. The only way out of this conundrum is to
trust instructors and evaluators in using judgment (subjectivity), to establish a command climate
that motivates and fosters trustworthiness (shared values), to provide instructors and evaluators
with instruments and guidelines that are trustworthy (rigorous) without precluding judgment, and
to engage continually in meaningful collaborative reflection (inter-subjective validity).
When focused on nave objectivity (assessments that could be done by a novice), measures of
performance can pervert the purpose of training and education. Without the currency and
relevance (that novices are incapable of considering) mere verifiability and repeatability are
insufficient to assure development of the Soldier. Novice assessments tend to focus on specific
aspects of the nominal training and education process rather than on the various outcomes,
especially intangible outcomes, that the process influences. Novices can be expected to depend
upon these factors and to operate so as to maximize the arbitrarily observable aspects of their
instruction and minimize the role of their own expertise as a Soldier and instructor. On the other
hand, if instructors focus on intangible outcomes, their experience as a Soldier and instructor is a
critical foundation for reliable assessment of good training. Leaders are familiar with assessing
intangible attributes. The systems for evaluating Officers and Non-Commissioned Officers
depend on the ability of experienced raters to make judgments about competencies, skills, values
and other qualitative characteristics of an individual (AWG, 2009). This is no less valid for
assessing training and education.
The behavioral and social sciences have developed a plethora of methods that attempt to strike a
balance between objectivity and subjectivity, that reflect both considerations rather than an
assumption that one has to choose between them, and thus that provide sophisticated and
trustworthy perspectives on verifiability and validity (Chapters 4, 5, and 11). There is no reason
for programmatic decision makers to ruminate over these problems of measurement or, worse, to
make suboptimal and even counterproductive decisions because of an assumption that the
problems are insolvable. Science provides plenty of options for moving forward, especially when
it is embedded in a broader scholarly context (Chapters 3-5; see also, Flyvbjerg, 2001; GodfreySmith, 2003; Schrim & Caterino, 2006).
Clarity About What Is Evaluated
We have developed measures that reflect best practices of OBTE and, most importantly, are
verifiable in practice. Without this, there is no way of knowing whether, or the extent to which,
OBTE actually is implemented; and without such verifiability, investigating the results in a
program of instruction is meaningless. No scientific conclusions can be drawn nor can any
improvements be suggested without verification of instructional practices. Documents describing
a program of instruction, a training support package, or any other kind of script for a learning
event do not satisfy this criterion. Measures are required so that scientists and evaluators can
verify what is occurring on the ground at the same time measures of effectiveness are being
applied and data on learning are being collected.
The focus of OBTE is on the instructor because instructors are the means through which the
organization has influence on the students. Measurement of instructor behavior is necessary for
this reason and, while it may be sufficient for scientific conclusions, it may not be sufficient to
come to conclusions that are actionable operationally. The reason is that there are organizational
factors that influence what an instructor can do (see Chapters 14 and 15; Appendix C). There is a
need to understand best practices within an instructional organization, including but not limited to
the command climate, with a level of theoretical grounding and practical detail commensurate
Epilogue
266
with the formative measures we have developed for instructor behavior. The thinking that
scientists would do to figure out what to measure within an organization is the very same thinking
that a programmatic stakeholder would have to do to reexamine and reprioritize activities and
objectives within a program. A provocative implication is that it would be useful to create a
forum in which scientists, leaders in programs of instruction, and active duty commanders or their
representatives could engage in collaborative reflection about the principles, practices, and
desired outcomes of training and education. The collaboration in such a forum could look very
much like the COMPASS process used to develop formative measures for OBTE (see e.g.,
Chapter 2).
The return on the investment in science to support programmatic decision-making is greatest if
the scientific assessment yields information that either can improve the capability or refine the
requirements that motivated the capability. One of the most important contributions of the
scientific inquiry into OBTE to date is to argue for a reorientation toward the common
competencies and attributes of Soldiers that matter to active duty units involved in Full Spectrum
Operations, to strive for a fuller and actionable interpretation of requirements for training and
education, and to close any associated gap between the Institutional Army and Operational Army.
Another important contribution is to show how best practices of instruction can become more
readily identifiable based on this reorientation. Together, this allows for systematic scientific
inquiry and a broader scholarly dialog among scientists and nonscientists that can have the
following results:
Stimulate debate and facilitate decision making about instructional methodologies

Identify and facilitate a coherent approach to training, education, and leader development
Provide framework for curriculum-level design and development
Provide framework for continuous and rigorous quality improvement
Next Steps
Further research should vigorously pursue development of (a) standards for the values-based
outcomes of interest to OBTE and Army doctrine; (b) measures that relate to various aspects and
stages of individual progress with respect to values-based outcomes; (c) theory and empirical
methodologies that reveal inter-temporal relationships between immediate observable effects and
long-term outcomes; and (d) an approach to verification and validation that brings quality
assurance into a tight correspondence with staff and faculty development within a comprehensive
approach to program evaluation.
This broad program of research would not be possible without the grounded theory we developed
over the course of this investigation (see e.g., Chapters 3, 4 and 5). These scientific underpinnings
are just a beginning for serious scholarship into OBTE and, more generally, into the critical
interrelationships among training, education, self-development, and leader development. At the
same time, this foundation is sufficient to initiate empirical research and to being to weave
together otherwise disparate sources of empirical evidence needed to support programmatic
decision-making in the institutional Army (cf., Flyvbjerg, 2001; Godfrey-Smith, 2003; Schrim &
Caterino, 2006; see also Chapter 11, section 11.2.3).
Figure 3 shows ways in which the scientific underpinnings for OBTE can guide the development
of values-based standards that can be verified and validated. This should be viewed as a road
map. The specific measures have yet to be identified and applied. It is important to note, however,
that these scientific underpinnings for OBTE are sources of measures that have been scientifically
validated and employed to come to a deeper understanding of situated behavior, situated
267
Riccio
experience, and situated meaning. That is, they preclude having to start from scratch and develop
new measures for all the important activities, decisions, and processes that are critical in OBTE
and, more generally, in the institutional Army. Our collaborative inquiry into the practical
wisdom as instructors and leaders has given us a good start. The integrative work should be
continued so that programmatic decision-makers can have the information they need to have the
greatest impact on preparation of Soldiers for FSO (see Chapters 11 and 15).
Figure 3. Sources of potential measures to support analysis, design, development,

implementation and evaluation of instruction that reflects a programmatic view of the
development and employment of values-based standards. Compare with Figure 2 in this
Epilogue and with Figure 8 in Chapter 3.
268
Epilogue
References
Asymmetric Warfare Group (2009, March). US Army Asymmetric Warfare Group workshop on
Outcomes-based Training and Education, Applied Physics Laboratory, Johns Hopkins
University, Laurel MD.
Company.
Bridge, G. (2000). Rationality, ethics, and space: on situated universalism and the self-interested
acknowledgement of difference. Environment and Planning D: Society and Space 2000, 18,
pages 519-535.
Group.
Group.
Dienes, Z. & Perner, J. (1999) A theory of implicit and explicit knowledge. Behavioral and Brain
Sciences 22(5), 735-808
Leibrecht, B., Wampler, R., & Pleban, R. (2009). Methodology for evaluating transfer of learning
from the U.S. Armys advanced leaders course. Research Product 2009-05. Washington, DC:
U.S. Army Research Institute for the Behavioral and Social Sciences.
Maslow, A. (1943). A theory of human motivation. Psychological Review, 50, 370-396.
Maslow, A. (1968). Toward a psychology of being. New York: Van Nostrand.
McKee, J. (2007). The new breed: Understanding and equipping the 21st century volunteer.
Loveland, CO: Group Publishing.Mikroyannidis, A. (2007). Toward a social semantic web.
Computer, November 2007, 113-115. Los Alamitos, CA: IEEE Computer Society.
Neisser, U., & Fivush, R. (Ed.) (1994). The remembering self: construction and accuracy in the
self narrative. Cambridge, UK: Cambridge University Press.
Putnam, R. (2001). Bowling alone: The collapse and revival of American community. New York:
Simon & Schuster.
Reed, E. (1996). The necessity of experience. New Haven: Yale University.
269
Riccio
Searle, J. (1997). Construction of social reality. New York: Free Press.
Straus, S., Shanley, M., Burns, R., Waite, A., & Crowley, J. (2009). Improving the Armys
assessment of interactive multimedia instruction courseware. Santa Monica, CA: RAND.
Thoits, P.A., & Hewitt, L.N. (2001). Volunteer work and well-being. Journal of Health and
Social Behavior, 42, 115131.
Section IV. Appendices
270
271
Riccio & Diedrich (Eds.)

Appendix A. OBTE Principles & Practices: Instructor Measures
A.1 Genesis of Formative Measures for Instructors
Our objective was to develop an observer-based measurement instrument comprised of items that
could be rated on a scale ranging from poor to excellent and, thus, that could provide formative
feedback about instructional effectiveness with respect to how well instructor behavior reflects
the principles of OBTE. Our goals in the development of the specific measurement items
included several criteria (see Chapter 2 for more detail about the methodology used in measure
development):
Behaviorally anchored items. Behavioral anchors provide raters with observable features of
performance that observers can link to ratings on a scale. To the extent possible, behavioral
anchors avoid the use of relative terms such as little or low, somewhat or moderate, and good
or high that leave judgment as to what constitutes poor or good performance up to the
observer.
Measures taken at specific intervals in a program of instruction. Measures taken at

specific intervals address performance at critical phases in the instruction, rather than as an
average across the entire course of instruction, and allow the rating to be tied to specific
phases in the instruction.
Focus on aspects of performance that are not captured in attainting terminal learning
objectives alone. The items should assess those aspects of performance that require human
observers who have experience and expertise commensurate with expectations of the domain
of application. Typically this level of expertise is the minimum an organization would require
for leader development.
Capture performance during hot spots. In order to differentiate low from high
performance, it is critical to assess performance at the most vulnerable points in learning,
rather than at times when there is little variation across individuals.
Assess the knowledge and skills that are exercised in both the learning and application
environment. Collectively, the items in the questionnaire should cut across particular skills
and knowledge. As a supplement to standard practice in assessment of such particulars, these
measures relate to the potential for continuing learning and to the context within which
particular skills and knowledge are used; that is, they relate to far transfer and impact of
learning.
The measures can be used as rubrics for instructors and instructional developers or
designers. Instructors can use data from these measures for continual improvement of
instruction. Developers and designers can use such data, especially from different instructors
and applications, to continually improve the structure of learning events and
recommendations about instructional methodologies that are most appropriate for particular
learning events.
Appendix A: Instructor Measures
272
A.2 Principles of Outcomes-Based Training & Education

We use Bransford, Brown, & Cocking (2000) as a framework within which to organize and
describe the multidisciplinary considerations implicit in the set of formative measures for OBTE
(Chapters 2 and 3). This framework addresses learner-centered, knowledge-centered, and
assessment-centered needs of a learning environment that are grounded in a deep understanding
of community-centered needs (Figure 1). Bransford et al. argue that learning systems will be
more effective to the extent that they meet these needs. We believe that this framework facilitates
understanding of the capability gaps to which OBTE is a response as well as the way in which it
addresses those gaps.
Figure 1. Needs for design of a learning environment that is effective with respect to
human development and far transfer [after Bransford et al., 2000].
273
Table 1: Needs of the learning environment in OBTE

Leadership and enculturation of Soldiers
Inculcation of Warrior Ethos and Army Values
Instructor is a leader, facilitator, advisor, mentor, and role model
Instructor gives purpose & vision, allows passion & risk, fosters growth
What is taught and how it is taught reflects necessity of Full Spectrum Operations
Community-centered learning environment
Integrated understanding of basic Soldier skills in Full Spectrum Operations
Condition Soldiers to overcome the psychological and physiological effects of combat
Condition Soldiers always to exercise deliberate thought under stress
Demonstrate the linking of tasks in a military situation
Understand relationships between what is taught and why, when, and how it is taught
Knowledge-centered learning environment
Collaborative reflection and problem solving
Training to grow problem solving
Teach Soldiers to learn for themselves within an established framework of knowledge
Teach through contextual understanding of the task such as its mission application
Draw out of the Soldier a critique of performance during the process
Assessment-centered learning environment
Soldier motivation and development of intangibles
Training to develop intangibles such as confidence, initiative, and accountability.
Assist the Soldier to understand the situation and desired result
Assist the Soldier in identifying obstacles to the desired result
Allow the Soldier to work towards a solution within defined principles
Learner-centered learning environment
A.2.1 Leadership and enculturation of Soldiers

What can be done to develop curricula and courses that improve the quality of life-long learning
and development such as in an operational context where significant and critical learning occurs?
One answer to this question follows from the realization that a collective, not an individual, is the
most fundamental unit of analysis in the Army. The goal should not necessarily be to wean
Soldier-students from guidance so that they are competent on their own when they leave formal
instruction. It should be to make them more competent at eliciting, utilizing, and providing
guidance within a task-organized unit. Task organization and division of labor in the Army both
requires and provides opportunities for momentary leadership that is not necessarily formal as
designated by rank, position or specialty. Leadership and guidance that individuals provide to
each other is not limited to formal learning events. Learning events can be designed to develop
this competency in individuals and thus to prepare Soldiers for success in Full Spectrum
Operations.
Given that there is an abundance of Soldiers with recent and relevant experience in Full Spectrum
Operations, how can we determine what kind of experience is useful in the classroom? OBTE
assumes that the most valuable experience is in the development of individuals. The outcomes
emphasized by OBTE are long-term developmental trajectories and milestones more commonly
associated with leader development. In this sense, OBTE essentially is a leadership-based
approach to training, education, and self-development. OBTE is based on the recognition that
274
competent confident Soldiers are developed when leaders allow subordinates reasonable
autonomy to exercise individual and small-unit initiative. This requires a climate of
accountability that is fostered by mutual trust among superiors and subordinates. It also requires
that leaders at every level recognize their influence as role models by thinking and acting flexibly
based on constant awareness and adjustment to deviations from optimal or expected conditions in
any situation. We tend to interpret this need in terms of its implications for establishing a
community-centered learning environment.
A.2.2 Integrated understanding of basic Soldier skills in Full Spectrum Operations
OBTE emphasizes that instruction is most effective when it reflects and responds to the most
urgent requirements of Full Spectrum Operations. These requirements are considered in terms of
basic Soldier skills such as move, shoot, and communicate but at a layer or two deeper than what
those words commonly connote. These operational requirements can be addressed at a level that
relates to individual adaptability and collective agility and that addresses the why, what, when,
where, and how of actions that can have irrevocable consequences. Thus, they can be addressed at
a level that benefits all Soldiers and their units.
Many of these requirements address stress, psychological and physiological effects of stress,
performance given these effects, and strategies that are resistant to stress. Experience with these
relationships and effects across disparate learning events helps a Soldier overcome stress by
avoiding distraction, maintaining an outward orientation, and always exercising deliberate
thought. It helps students make sense of the relationships between what is taught and why, when,
and how it is taught. It provides them with a path toward ever deepening understanding of their
capabilities in context. We tend to interpret this need in terms of its implications for establishing a
knowledge-centered learning environment.
A.2.3 Collaborative reflection and problem solving
OBTE seeks to establish a mindset of collaborative reflection and problem solving in which there
is continuous vigilance for noteworthy events and lessons learned worth discussing among
instructors, students, and peers. Development of mastery in assessing oneself and ones unit is a
key outcome for OBTE. It is important to understand that collaborative reflection is a ubiquitous
opportunity that exists in any program of instruction. It can be done on an as-needed basis when
the lessons learned justify the investment of time and resources. It can exploit naturally occurring
events and utilize whatever method is appropriate at the time, such as after action reviews, hot
washes, and informal conversation. This does not mean that constant discussion is a goal of
OBTE or that the mere occurrence of discussion is a reflection that OBTE is occurring. It is as
important to decide not to engage in collaborative reflection if there has not been a noteworthy
event or if there is a more valuable use of time for learning. This mindset, then, also continually
reinforces the importance of prioritization and tradeoffs required in most military scenarios.
If a decision is made to engage in collaborative reflection (e.g., to execute a planned after-action
review), the discussion should be conducted in an open and supportive manner regarding lessons
learned. The purpose is not to summarize merely what happened in the learning event but to
facilitate a discussion that allows students to evaluate their own performance in terms of the
consequences of their actions for the rest of the team and with respect to other linked tasks.
Collective self-examination leads to an understanding in which the whole is greater than the sum
of the parts. We tend to interpret this need in terms of its implications for establishing an
assessment-centered learning environment.
275
A.2.4 Soldier motivation and development of intangibles

instructional environment to achieve a positive outcome; that is when they exemplify leadership.
Such adaptation includes utilization of whatever instructional methods are appropriate for the
problem at hand, whether that is direct instruction, problem-centered instruction, or experiential
learning for example. The strategy is to provide learners with right amount of guidance at the
right time to achieve both the learning objective for the event and longer-term developmental
outcomes such as the growth of confidence, initiative and accountability. Even in situations in
which students do a "worked example" or solve a well-defined problem given by the instructor, it
is beneficial to the student to understand that the assumptions of the given problem are valid and
why the assumptions are necessary, even if they aren't ready to be thrown into a situation in
which assumptions are violated. This allows them to become ready, to become prepared, to be
thrown into such a situation. The foundation for collective agility is thus reinforced in all training
and educational situations, even ones that ostensibly do not require individual adaptability.
The intent is to leverage internal motivation and to internalize sources of motivation by balancing
initiative with a sense of accountability based on discipline and awareness of ones relationship to
others, and by building confidence based on enhanced self-efficacy and commitment to the
pursuit of mastery. We tend to interpret this need in terms of its implications for establishing a
learner-centered environment for training and education.
276

A.3 Guide to Using Measures of Instructor Behavior
Table 2. Mapping between the needs of the learning environment

in OBTE and the categories of observable instructor behavior.
Usage is as follows: (a) start with planning, execution, or AAR; (b)
select a need or a behavioral category; (c) choose a row that maps
to one of the four columns or a column that maps to one of the
behavioral categories; (d) if undecided, use cells with higher
number for guidance to go between rows and columns. Each cell
generally will lead to between five and nine measures from which to
choose. Note that, while the priorities below are helpful for novice
users, each measure is relevant to all columns to some degree.
1. Leadership
2. Prepared for FSO
3. Problem Solving
4. Develop Intangibles
Table 2 is a guide intended primarily for novices in the use of instructor measures for OBTE. The
example after the table demonstrates how the table can be used to select measures from Table 3.
Before Learning Event (Planning)

(1) Robust and adaptable plan for instructional events
(2) Consider how to reveal task relevance of instructional events
(3) Focus on development of the individual
3
1
1
3
1
1
1
2
1
3
During Learning Event (Execution)

(4) Instructors are role models
(5) Reveal multifaceted operational-relevance of events
(6) Incorporate stress into instructional events
(7) Facilitate communication and collaborative problem-solving
(8) Nature and extent of guidance provided by instructor
(9) Get students to take ownership of their own learning
After Learning Event (AAR)
(10) Identify general lessons and extrapolate to new situations
(11) Establish a pervasive mindset of collaborative reflection
(12) Collaborative reflection as a means to develop self efficacy
3
1
3
2
1
1
3
2
2
2
1
2
3
1
2
3
2
8 measures
5 measures
9 measures
1
1
2
3
6 measures
7 measures
6 measures
6 measures
7 measures
8 measures
2 measures
2 measures
5 measures
277
Example: Choose "problem solving" (column 3) for instructional focus in an upcoming learning
event
1. Before learning event (planning)
a. Not sure what to do, so choose cell in the focus on development of individual row
over low ranked rows in column 3
b. Go to table of individual measures for "focus on development of the individual"
c. Nine measures to choose from, some of which seem better than others for problem
solving
d. Choose measure 3.7 because of interest in getting students to begin learning on their
own
e. Consider a few additional measures so instructional focus is not too narrow
f. Choose another column such as "develop intangibles" (column 4) because of
relevance to current interest
g. Could look into another behavior category but stay with focus on development of
individuals because many measures seem relevant
h. Choose measure 3.6 because transfer of ownership seems to be another important
determinant of student beginning to learn on their own
i. Go to another behavior category to keep instructional focus from being too narrow
j. Choose Consider how to reveal task relevance of instructional events row to keep
focus on individual development
k. Five measures to choose from but can't make good pedagogical connection, so scan
nearby cells
l. Measure 2.2 catches the eye because of inkling that task organization may be a good
way to transfer ownership
m. The two behavior categories and two columns provide some useful constraints to
focus planning for upcoming event
n. Planning can be assessed with respect to the three measures chosen: 2.2, 3.6, and 3.7
o. That wasn't so hard, it didn't take much time, and probably saved time that would
have been wasted amid uncertainty
p. There is greater confidence in the value of the planned learning event and outcomes it
can influence
2. During learning event (execution)
a. Some but not all measures should be chosen before execution to focus attention of
observers or participants
b. Now that there is a focus for the learning event, nature and extent of guidance
provided by instructor looks like a logical behavior category to use
c. There are seven measures to choose from
d. Both measure 8.1 and 8.6 look pretty interesting and pedagogically relevant
e. Choose 8.1 because explaining the why of task organization looks like a good way to
draw students into participation
f. There is a gist of adapting to the students in several other measures so needs for
adaptation would be good to look for
g. Instructor can self assess or be assessed by others with respect to measures such as
8.2 to 8.5
h. Collaborative reflection can occur in stride, one need not wait until a formal AAR to
use measures
3. After learning event
a. Some but not all measures should be chosen before the learning event to ensure that
there is a clear plan for the AAR
278
b. Measure 11.2 foster an environment that allows Soldiers to discuss their mistakes
and consequences might be a good one to discuss immediately after the attempt to
utilize task organization
c. Measure 11.2 can be revisited in the formal AAR perhaps by way of elaboration
using measure 10.2 relationship between the individual and big picture/mission
d. There was a lot that could have been attended to and discussed but the focus made
the situation manageable
e. The issues that could have been addressed during or after the learning event can help
in designing the next event
f. There is a growing sense of confidence in a systematic approach to teaching and
learning across learning events
g. Accountability within respect to a system of measures reveals opportunities and
purpose in variable conditions
h. There is growing understanding that initiative can build on lessons learned and
progress in prior learning events
i. There is growing understanding and confidence that describable initiative can lead
one into the knowable unknown
279
2. Prepared for FSO
3. Problem Solving
1.1
Does instructor plan collective decision/evaluation points during learning?
1.2
Does instructor design/plan the learning event to be flexible depending on conditions of instruction?
1.3
Does instructor adapt outcome expectations to account for environmental conditions?
1.4
Does instructor adapt outcome expectations to account for Soldiers who learn slower or faster?
1.5
Does instructor adapt outcome expectations to account for limited resources?
1.6
Does instructor adapt outcome expectations to account for unsupportive command climate?
1.7
Does instructor plan for maximizing learning opportunities under less than perfect conditions?
1.8
Does instructor identify and provision information resources necessary to support learning objectives?
1. Leadership
A.4 Complete Menu of Instructor Measures
Table 3. Menu of formative measures for instructors used before a learning event (planning). Measures
are listed in the order in which they are presented in the following pages. The number in the leftmost columns
indicates the reference code for each measure. The numbers in the cells corresponding to the needs of the
learning environment (rightmost four columns) indicate the relative extent to which each measure reflects the
needs of the learning environment. Higher numbers indicate a stronger relationship.
Robust and adaptable plan for instructional events
2
2
Consider how to reveal task relevance of instructional events
2.1
Is learning event designed to emphasize the importance of combat applications?
2.2
Does instructor plan to task organize the Soldiers and themselves to facilitate learning?
2.3
Does instructor plan to discuss the tactical relevance of the task with the Soldiers?
2.4
Does instructor build time for discussion of tactical relevance into the schedule?
2.5
Will instructor ask the Soldiers to describe the tactical relevance of the events?
2
1
Focus on development of the individual
3.1
Is learning event designed to develop intangibles?
3.2
Do measures of effectiveness relate learning event to the development of intangibles?
3.3
Does Commander's intent focus on effective learning of task and development of the individual?
3.4
Does instructor incorporate development of intangibles into vision for achieving CDR intent?
3.5
Does instructor plan to scale down from authoritarian style to mentoring leadership style?
3.6
Does instructor plan to transfer ownership of learning to the Soldiers?
3.7
Does instructor structure the learning events to allow Soldiers to learn for themselves?
3.8
Does instructor actively participate in creating a positive learning environment?
3.9
Does instructor design the learning events to gradually increase in difficulty?
1
3
1
2. Prepared for FSO
1. Leadership
Table 4a. Menu of formative measures for instructors used during a learning event (execution). Measures
are listed in the order in which they are presented in the following pages. The number in the leftmost columns
indicates the reference code for each measure. The numbers in the cells corresponding to the exigencies of the
learning environment (rightmost four columns) indicate the relative extent to which each measure reflects the
exigencies of the learning environment. Higher numbers indicate a stronger relationship.
3. Problem Solving
280
Instructors are role models
4.1
Are instructors communicating with each other regarding the progression of instruction?
4.2
Are instructors discussing the effectiveness of the instruction?
4.3
Are instructors evaluating how effectively they are delivering the instruction?
4.4
Does instructor effectively exhibit intangible attributes in his/her own behaviors during instruction?
4.5
Which intangible attributes were exhibited by the instructors?
4.6
Does instructor demonstrate openness in changing the progression of instruction?
Reveal multifaceted operational-relevance of instructional events
5.1
Does the learning event emphasize broad combat/mission success?
5.2
Does learning event contain problem solving events that require Soldiers to understand the why?
5.3
Does instructor encourage Soldiers to discuss the why beyond the current context?
5.4
Does instructor use unexpected conditions to provide tactically relevant learning events?
5.5
Does instructor put tasks into context and relate them to other tasks?
5.6
Does instructor group tasks into collective behaviors?
5.7
Does instructor ensure that Soldiers understand the combat/mission application of a task?
Incorporate stress into instructional events
6.1
Does instructor effectively incorporate stress into learning events to benefit development of Soldier?
6.2
Does instructor encourage deliberate thought in stressful situations?
6.3
Does instructor encourage the Soldiers to think about how the stress affects their capabilities?
6.4
Does instructor introduce Soldiers to broad range of stressors and cultivate stress management?
6.5
Does instructor effectively manipulate stress levels to achieve learning objectives?
6.6
Does instructor effectively manipulate task complexity to achieve learning objectives?
Facilitate communication and collaborative problem-solving
7.1
Does instructor encourage inter-trainee communication and discussion?
7.2
Does instructor encourage Soldiers to discuss the why with each other?
7.3
Does instructor provide feedback to student in a constructive/diagnostic (non-directive) manner?
7.5
Does instructor communicate an interest in the quality of performance?
7.5
Does instructor communicate a willingness to guide learning through self-discovery?
7.6
Does instructor positively influence motivation though non-task related comments?
281
3. Problem Solving
8.1
Does instructor articulate the why to the Soldiers?
8.2
Does instructor adapt the learning event to the audience/environment?
8.3
Does instructor adapt coaching to individual Soldiers?
8.4
Does instructor successfully address individual learning predicaments?
8.5
Does instructor recognize when a Soldier is too withdrawn/distracted to participate in learning?
8.6
Does instructor guide Soldiers to self discovery of how to achieve a desired outcome?
8.7
Does instructor use safety as an instructional enabler?
Table 4b. Menu of formative measures for instructors used during a learning event (execution).
Measures are listed in the order in which they are presented in the following pages. The number in the
leftmost columns indicates the reference code for each measure. The numbers in the cells corresponding
to the exigencies of the learning environment (rightmost four columns) indicate the relative extent to
which each measure reflects the exigencies of the learning environment. Higher numbers indicate a
stronger relationship.
1. Leadership
2. Prepared for FSO
Nature and extent of guidance provided by instructor
Get students to take ownership of their own learning
9.1
Does instructor balance perception of power vs. leadership?
9.2
Does learning event foster enthusiasm in the Soldiers?
9.3
Does instructor foster self-development amongst the Soldiers?
9.4
Does instructor assume the role of helper in Soldiers pursuit of success?
9.5
Does instructor reinforce the importance of problem solving?
9.6
Does instructor incorporate resource constraints as a problem solving challenge for Soldiers?
9.7
Does instructor ask questions and allow/encourage the Soldiers to answer?
9.8
Does instructor encourage Soldiers to ask questions to discover the why?
10.1
Does instructor ensure the Soldiers can articulate how to apply concepts to new situations?
10.2
Does instructor focus the why on the relationship between the individual and big picture/mission?
Table 5. Menu of formative measures for instructors used after a learning event (AAR). Measures
are listed in the order in which they are presented in the following pages. The number in the leftmost
columns indicates the reference code for each measure. The numbers in the cells corresponding to the
exigencies of the learning environment (rightmost four columns) indicate the relative extent to which
each measure reflects the exigencies of the learning environment. Higher numbers indicate a stronger
relationship.
10
11
11.1
11.2
12
1. Leadership
3. Problem Solving
282
2. Prepared for FSO
Identify general lessons learned and extrapolate to new situations
Establish a pervasive mindset of collaborative reflection

Does instructor conduct AAR and Hotwashes as needed?
Does instructor foster an environment that allows Soldiers to discuss their mistakes and
consequences?
1
3
Collaborative reflection as a means to develop self efficacy
12.1
Does instructor ask Soldiers to assess their own performance?
12.2
Does instructor ensure the Soldiers can articulate the why regarding the learning event?
12.3
Does instructor ensure the Soldiers can articulate the consequences of their actions?
12.4
Does instructor ensure that the Soldiers recognize the effects of their actions on their teams?
12.5
Do the instructors identify any needs for remedial instruction?
283

Exigency: Leadership
1.1 Does instructor plan collective decision/evaluation points during learning event?
c Yes
c N/A
c N/O
c No
Comments/Notes:
1.2 Does instructor design/plan learning event to be flexible depending on conditions (e.g.
Soldier skill level, environment, etc.)?
c Yes
c N/A
c N/O
c No
Comments/Notes:
284

1.3 Does instructor adapt outcome expectations to account for environmental conditions (e.g.
inclement weather)?
Cancels instruction or
maintains focus on
specific outcomes for
original task
c
c
N/A
N/O
Chooses different
outcomes but does not
achieve effective
instruction
Chooses outcomes to
achieve instruction
effectiveness; leverages
condition as another
opportunity
Comments/Notes:
1.4 Does instructor adapt outcome expectations to account for Soldiers who learn slower/faster
than anticipated?
maintains focus on
original task
c
c
N/A
N/O
Comments/Notes:
Chooses different
achieve effective
instruction
Chooses outcomes to
achieve instruction
opportunity
285

1.5 Does instructor adapt outcome expectations to account for limited resources (e.g. types of
ranges, ammo, time, number of trainees, transport)?
maintains focus on
original task
c
c
N/A
N/O
Chooses different
achieve effective
instruction
Chooses outcomes to
achieve instruction
opportunity
Comments/Notes:
1.6 Does instructor adapt outcome expectations to account for unsupportive command
environment (if applicable)?
Maintains focus on
original task
c
c
N/A
N/O
Comments/Notes:
Chooses different
achieve effective
instruction
Chooses outcomes to
achieve instruction
opportunity
286

1.7 Does instructor plan for maximizing instructional opportunities under less than perfect
conditions?
Does not develop a

backup plan
c
c
N/A
N/O
Develops a backup plan

that will not achieve
learning objectives
Turns less than perfect

opportunities into
opportunities for Soldier
problem solving
Comments/Notes:
1.8 Do the Drills identify and provision the information resources necessary to support learning
objectives?
Does not recognize the

need for information
resources
c
c
N/A
N/O
Comments/Notes:
Recognizes need for

information resources;
focuses only on specific
event/task (e.g. reliance on
TSP)
Recognizes need for

information resources;
anticipates and prepares
for broader questions
from Soldiers
287

Exigency: Prepared for Full-Spectrum Operations
2.1 Is learning event designed to emphasize the importance of combat applications?
Instruction focuses on
tasks/events; goal is to
pass (e.g. qualify)
c
c
N/A
N/O
Combat applications are

described, but instruction
focuses on tasks/event
Tasks/Events resemble
combat application and
mission success
Comments/Notes:
2.2 Does instructor plan to task organize the Soldiers and themselves to facilitate learning?
Does not task organize
c
c
N/A
N/O
Comments/Notes:
Task organizes to achieve

efficiency
Task organizes groups

and Instructors to
facilitate learning
(capabilities, group size,
Soldier/cadre ratio)
288

2.3 Does instructor plan to discuss the tactical relevance of the task with the Soldiers?
c Yes
c N/A
c N/O
c No
Comments/Notes:
2.4 [Follow-up] Does instructor build time for this discussion into the schedule?
Does not build in time

for discussion into the
schedule
c
c
N/A
N/O
Builds in time for short

discussions after the day is
complete
Builds in time for short

discussions between tasks
Comments/Notes:
2.5 [Follow-up] As part of the discussion, will instructor ask the Soldiers to describe the tactical
relevance of the events?
Only plans to lecture to

Soldiers on the task; no
context
c
c
N/A
N/O
Comments/Notes:
Plans to engage the

Soldiers on why the event
is tactically relevant, but
Plans to state the

problem, then guide the
the tactical relevance (i.e.
problem solving exercise)
289

Exigency: Develop Intangibles
3.1 Is the learning event designed to develop intangible attributes (Confidence, Initiative,
Accountability, Awareness, Discipline, Judgment, Deliberate Thought)?
Focus of instruction is
solely to accomplish
the task
c
c
N/A
N/O
Instructor discusses
intangibles directly related to
task being trained; does not
focus plan on development
Instructor discusses
intangibles directly
related to the task being
trained; plan focuses on
development
Comments/Notes:
3.2 Do the measures of effectiveness relate the learning event to the development of the
intangible attributes (Confidence, Initiative, Accountability, Awareness, Discipline, Judgment,
Deliberate Thought)?
c Yes
c N/A
c N/O
c No
Comments/Notes:
290

3.3 Does the Commanders intent focus on effective learning of the task and the development of
the individual?
CDRs intent is focused

on apparently efficient
event/TSP
CDRs intent focuses on

correct Soldier
but not development of
the individual
CDRs intent focuses on

correct Soldier
3.4 [follow-up] Does instructor incorporate development of intangibles into the vision for
achieving the Commanders intent?
Vision is focused on
apparently efficient
event/TSP
c
c
N/A
N/O
Comments/Notes:
Vision focuses on correct

Soldier performance of
the task but not
development of the
individual
Vision focuses on
correct Soldier
performance of the
task
291

3.5 Does instructor plan to scale down from an authoritarian approach to a mentoring leadership
style as appropriate?
c Yes
c N/A
c N/O
c No
Comments/Notes:
3.6 Does instructor plan to transfer ownership of instruction to the Soldiers?

c Yes
c N/A
c N/O
c No
Comments/Notes:
3.7 Does instructor structure learning event to allow Soldiers to learn for themselves?
Focus on efficiency,
regimented instruction,
learning outcomes not
specified
c
c
N/A
N/O
Comments/Notes:
Builds in time to explain

the learning outcomes to
the Soldiers
Builds in time for Soldier

self-discovery of learning
outcomes
292

3.8 Does instructor actively participate in creating a positive learning environment?
Not involved in
planning
c
c
N/A
N/O
Involved in planning from

a functional standpoint;
task/resource focused
Demonstrates creativity
and enthusiasm; wants to
make instruction better;
planning is an
opportunity
Comments/Notes:
3.9 Does instructor design the learning events to gradually increase in difficulty?
Unaware of stress
management; Plans to
increase difficulty of
instruction according to a
set schedule/process
c
c
N/A
N/O
Comments/Notes:
Plans to introduce stress,

but not at the appropriate
level or time
Balances stress and

difficulty to the capability
of the Soldiers
293

4.1 Are instructors communicating with each other about the progression of instruction?
c Yes
c
c
N/A
N/O
c No
Comments/Notes:
4.2 Are instructors discussing the effectiveness of the instruction?
Instructors
different groups
without considering
progress
c
c
N/A
N/O
Instructors check on
Soldiers progress but do
not effectively adjust the
instruction
Check on Soldiers
progress and discusses
when to advance to next
event or revisit a learning
objective
Comments/Notes:
4.3 Are instructors evaluating how effectively they are delivering instruction?
Instructors
without evaluating
delivery
c
c
N/A
N/O
Comments/Notes:
Instructors monitor the

effectiveness of delivery
but do not adjust
accordingly
Instructors collectively
monitor effectiveness of
delivery and adjust
delivery accordingly, or
revisit learning objectives
if necessary
294

4.4 Does instructor effectively exhibit intangible attributes in his/her own behavior during
instruction?
c Yes
c N/A
c N/O
c No
Comments/Notes:
4.5 [Follow-up] Which intangible attributes were exhibited by the instructors?
Judgment
Adaptability
Accountability
Problem Solving
Confidence
Initiative
Awareness
Thinking Skills
c
c
N/A
N/O
Comments/Notes:
N/A
N/O
Yes
No
295

4.6 Does instructor demonstrate openness in changing the progression of instruction?
Instruction will not

deviate from a set
schedule
c
c
N/A
N/O
Comments/Notes:
Instructors determine when

it is time to move on a new
task
Soldiers have input into

the progression of
instruction
296
(5) Reveal multifaceted operational-relevance of instruction

5.1 Does learning event emphasize broad combat/mission success?
Train to a specific task

only
c
c
N/A
N/O
Comments/Notes:
Train to examples or
experience only; reflects
specific conditions or
resources only
Creates instruction that

emphasizes problem
solving irrespective of
conditions or resources
297

5.2 Does learning event contain problem solving events that require Soldiers to understand the
why in order to succeed?
Problem solution is
based on memorization
or repetitive action
c
c
N/A
N/O
Problem solution may not

require an understanding of
the why; rooted in either
technical or tactical
application
Successfully solving the

problem requires
investigation of the why;
rooted in both tactical and
technical application
Comments/Notes:
5.3 [Follow-up] Does instructor encourage Soldiers to continue to discuss the why beyond the
context of the current learning event?
c Yes
c
c
N/A
N/O
c No
Comments/Notes:
298

5.4 Does instructor use unexpected conditions to provide tactically relevant learning events?
Allow the unexpected

conditions to interfere
with instruction
c
c
N/A
N/O
Make in-stride adjustments

to continue instruction but
do not leverage unexpected
conditions as an
opportunity
Make in-stride
adjustments to
incorporate unexpected
conditions into the
instruction design if
appropriate
Comments/Notes:
5.5 Does instructor put tasks into context and relate them to other tasks?
Trains task in isolation;

task taken out of
context; detached from
application
c
c
N/A
N/O
Comments/Notes:
Trains task within context

but does not relate to other
tasks
Trains task within context

and explains relationship
to other tasks
299

5.6 Do the instructors group tasks into collective behavior?
Does not group tasks

into collective
behavior; results in
incorrect performance
of linked tasks
c
c
N/A
N/O
Does not group tasks into

collective behavior (no
observable negative
consequences)
Groups task in a way that

simulates the combat
application and reinforces
linked tasks
Comments/Notes:
5.7 Do the instructors ensure that Soldiers understand the combat/mission application of a task?
Does not explain

context/combat
relevance of the task
c
c
N/A
N/O
Comments/Notes:
Tells Soldiers the combat

Has Soldiers mentally

simulate the combat
situation to discover
300

6.1 Does instructor effectively incorporate stress (mental and physical) into learning events to
benefit the development of the Soldier?
No stress resulting in
apathy or too much
stress resulting in
chronic failure
c
c
N/A
N/O
Some stress resulting in

some learning; Soldier
unchallenged or overly
challenged
Stress is proportional to
the task and Soldier
capabilities resulting in a
sense of accomplishment
Comments/Notes:
6.2 Does instructor encourage deliberate thought in stressful situations?
Rushes or encourages
Soldier to complete the
task without deliberate
thought; focuses on
speed
c
c
N/A
N/O
Comments/Notes:
Encourages Soldier to
think but fails to establish
effective conditions to
reinforce thinking
Focuses Soldier on
thinking through situation
regardless of the stress
level; focuses on
accountability
301

6.3 Does instructor encourage the Soldiers to think about how the stress affects their capabilities?
Marginalizes the true

effects of stress; does
not discuss
c
c
N/A
N/O
Tells Soldiers how the

stress affected their
performance
Allows Soldiers to
discover how stress
affects their performance
and how it is mitigated
Comments/Notes:
6.4 Does instructor introduce the Soldiers to a broad range of stressors and cultivate stress
management?
No strategy for
effective introduction
of stressors and stress
management as a tool
for positive Soldier
development
c
c
N/A
N/O
Comments/Notes:
Strategy for stress

introduction and
management exists but
does not effectively
promote positive Soldier
development
Strategy for stress

introduction and
management promotes
positive Soldier
development
302

6.5 Do the instructors effectively manipulate stress levels to achieve learning objectives?
Trains the
fundamentals of a task
in a stressful
environment; increases
stress level before
Soldiers are ready
c
c
N/A
N/O
Fails to effectively
manipulate stress
proportional to Soldier
skill level
Increases stress level as

Soldiers begin to master
the current task in a no
stress environment
Comments/Notes:
6.6 Do the instructors effectively manipulate task complexity to achieve learning objectives?
Trains the
fundamentals of a task
in a complex
environment; increases
complexity before
Soldiers are ready
c
c
N/A
N/O
Comments/Notes:
Fails to effectively
manipulate task
complexity proportional to
Soldier skill level
Increases complexity as
Soldiers begin to master
the current task in a
minimally complex
environment
303

Exigency: Problem Solving
7.1 Does instructor encourage inter-trainee communication and discussion?
Soldiers do not have

the opportunity to
discuss
c
c
N/A
N/O
Instruction design provides

opportunities for unguided
inter-trainee
communication
Instructor facilitates
discussions and
encourages follow-on
inter-trainee discussion;
instruction design
provides opportunities
Comments/Notes:
7.2 [Follow-up] Does instructor encourage Soldiers to discuss the why with each other?
c Yes
c
c
N/A
N/O
c No
Comments/Notes:
304

7.3 Does instructor provide feedback to student in a constructive/diagnostic (non-directive)
manner?
c Yes
c
c
N/A
N/O
c No
_______ Number of Instances
Comments/Notes:
7.4 Does instructor communicate an interest in the quality of performance?

c Yes
c
c
N/A
N/O
c No
Comments/Notes:
7.5 Does instructor communicate a willingness to guide learning through self-discovery?

c Yes
c
c
N/A
N/O
c No
Comments/Notes:
7.6 Does instructor positively influence motivation though non-task related comments?
c Yes
c
c
N/A
N/O
c No
Comments/Notes:
305

8.1 Does instructor articulate the why to the Soldiers?
Relies only on tasks

conditions and
c
c
N/A
N/O


instruction; states the
Comments/Notes:
8.2 Does instructor adapt learning event to the audience/environment?
Sticks to set schedule;

unaware of diminished
learning (e.g. ignores
indicators of
exhaustion)
c
c
N/A
N/O
Comments/Notes:
Reactively balances
learning difficulty to the
capability of the Soldiers
(e.g. reacts after
performance has
diminished significantly
and then makes a change)
Proactively balances
learning difficulty to the
capability of the Soldiers;
recognizes point of
diminishing return (e.g.
exhaustion/ getting burnt
out)
306

8.3 Does instructor adapt coaching to individual Soldiers?
Doesnt connect with

individual Soldiers;
only at group level
c
c
N/A
N/O
Tells the Soldier how to fix

the problem
Provides individual
coaching and helps the
Soldier discover the
solution
Comments/Notes:
8.4 Does instructor successfully address individual learning predicaments?
Doesnt recognize
individual problem
c
c
N/A
N/O
Comments/Notes:
Recognizes individual
problem, doesnt help the
Soldier find a suitable
solution
Identifies the issue and

helps the Soldier find a
work around (e.g. shorter
weapon for smaller
person)
307

8.5 Does instructor recognize when a Soldier is too withdrawn/distracted to effectively
participate in instruction?
Does not recognize a

deeper individual
problem when Soldier
is struggling with a
task
c
c
N/A
N/O
Sees that there is a deeper

problem, but doesnt do
anything (doesnt realize
he CAN do something)
Recognizes there is a
deeper problem, takes
appropriate action to help
Soldier get back in the
game
Comments/Notes:
8.6 Does instructor guide Soldiers to self discovery of how to achieve a desired outcome?
Dictates a specific path

to the solution to a task
c
c
N/A
N/O
Comments/Notes:
Structures the problem; but

still dictates a specific path
to the solution
Structures the problem

and guides Soldier to
efficient self discovery of
solution path
308

8.7 Does instructor use safety as a learning enabler?

real purpose
c
c
N/A
N/O
Comments/Notes:

the instruction events, but
not as a combat and
learning enabler (i.e. safety
is restrictive)
combat and learning
awareness allows for
more independent or
complex scenarios)
309

9.1 Does instructor balance perception of power vs. leadership?
Overly authoritarian
and maintains symbols
of authority; Soldiers
do not approach
instructors
c
c
N/A
N/O
Removes symbols of
authority but remains
authoritarian; Soldiers still
reluctant to approach
Removes symbols of
authority and exhibits
approachability, assumes
role of mentor; guides
Soldiers through solving
the problem
Comments/Notes:
9.2 Does learning event foster enthusiasm in the Soldiers?
Purely procedural
c
c
N/A
N/O
Comments/Notes:
Enthusiasm is focused on
winning a competition or
event
Enthusiasm is focused on
improvement and combat
application; instructor
reinforces improvements
310

9.3 Does instructor foster self-development amongst the Soldiers?
Tells them how they

did without giving
Soldiers a chance to
reflect
c
c
N/A
N/O
Asks the Soldiers how they

think they did, tells them
what they did wrong
Asks the Soldiers how

they think they did;
facilitates self-discovery
of mistakes through
targeted questioning
Comments/Notes:
9.4 Does instructor assume the role of helper in Soldiers pursuit of success?
Dictates instructions;
threatening; (instructor
focus is
event)
c
c
N/A
N/O
Comments/Notes:
Acts as a mentor but

occasionally reverts back
to role of dictating
Acts as a mentor; wants

Soldier to do well;
(instructor focus is
success of the Soldier)
311

9.5 Does instructor reinforce the importance of problem solving?
Event is scripted;
instruction is based on
predetermined task
c
c
N/A
N/O
Instruction provides
to be problem solvers but
instructors give the
solution
Instruction provides
to be problem solvers;
events require
assessment, judgment,
decision making and
execution
Comments/Notes:
9.6 [Follow-up] Does instructor incorporate resource constraints as a problem solving challenge
for the Soldiers?
c Yes
c N/A
c N/O
c No
Comments/Notes:
312

9.7 Does instructor ask questions and allow/encourage the Soldiers to answer?
Provides answers for

the Soldiers
immediately after
question is asked
c
c
N/A
N/O
Encourages Soldiers to
answer questions, but
provides an answer if one
is not immediately offered
Allows time for Soldiers

to think of an answer and
guides them to
appropriate responses
Comments/Notes:
9.8 Does instructor encourage Soldiers to ask questions to discover the why?
Discourages Soldiers
from asking questions
c
c
N/A
N/O
Comments/Notes:
Questions are answered

but very little time exists
for discussion
Reinforces the Soldiers

for asking questions that
focus on understanding
the why
313
(10) Identify general lessons and extrapolate to new situations

10.1 Does instructor ensure Soldiers can articulate how to apply concepts to new situations?
Discussion focuses on
the specific task only;
does not discuss
combat/mission
application
c
c
N/A
N/O
Facilitates a discussion of
the specific tasks and how
they apply to
combat/missions
the intangible attributes
underling the task and
how it applies to
ambiguous
combat/mission situations
Comments/Notes:
10.2 Do the instructors focus the why of instruction back to the relationship between the
individual and big picture/mission?

on why the Soldiers
were trained (e.g. You
need these skills
because we are at
war.)
c
c
N/A
N/O
Comments/Notes:
the big picture; less focus
on the individuals
contribution
the big picture; focuses
on the importance of
individuals (i.e. You can
make a difference)
314
(11) Establish a pervasive mindset of collaborative reflection

11.1 Does instructor conduct AAR/Hotwashes as needed?
Does not conduct

AAR/Hotwash or
conducts regardless of
need
c
c
N/A
N/O
Tells/lectures Soldiers the

lessons learned in a timely
manner
lessons learned amongst
the Soldiers in a timely
manner
Comments/Notes:
11.2 Does instructor foster an environment that allows Soldiers to discuss their mistakes and
consequences?
Overly authoritarian;
does not discuss
consequences
c
c
N/A
N/O
Comments/Notes:
Establishes a mentor like

rapport but too little
discussion of real life
consequences
Establishes a mentor like

rapport and reinforces the
real life consequences
315

12.1 Does instructor ask Soldiers to assess their own performance?
Does not ask Soldiers

to assess their own
performance or is
overly negative in
assessment
c
c
N/A
N/O
Asks Soldiers to assess

their own performance but
does not recognize
accurate self assessments
Asks Soldiers to assess

their own performance
but monitors to ensure
they are realistic
Comments/Notes:
12.2 Does instructor ensure Soldiers can articulate the why for the learning event?
Instructor is only
directive
c
c
N/A
N/O
Comments/Notes:
Instructor delays shifting

from telling to asking
about the why
Instructor shifts from

telling to asking about the
why in a timely manner
316

12.3 Do instructors ensure Soldiers can articulate the consequences of their actions?
Tells Soldiers what

went wrong and why it
is important in combat,
but does not discuss
how to mitigate
mistakes next time
c
c
N/A
N/O
Asks Soldiers to explain

what went wrong, why its
important in
combat/mission, but does
not discuss how to mitigate
mistakes next time
Guides Soldiers through

explaining what went
wrong, why it is
important in
combat/mission, how they
might improve next time
Comments/Notes:
12.4 Does instructor ensure that the Soldiers recognize the effects of their actions on their teams?
Focuses discussion on
individual mistakes;
ignores team dynamics
c
c
N/A
N/O
Comments/Notes:
how the team worked
together during the
learning event but does not
discuss strategic
consequences
Facilitates a discussion on
of how the team worked
together during the
learning event and
discusses the strategic
consequences
317

12.5 Do instructors identify any needs for remedial instruction?
Glosses over the

problems, focuses on
throughput; does not
put together resources
to solve problems
c
c
N/A
N/O
Comments/Notes:
Identifies those who need

additional instruction but
does not fully assess why;
puts together resources to
give them additional
practice
Identifies those who need

additional instruction and
assesses why; puts
together resources to
solve their specific
problem
Appendix B: Student Measures
318
Appendix B. OBTE Principles & Practices: Student Measures

B.1 Guide to Using Measures of Student Behavior
Table 1. Menu of student measures for external observers. Measures are listed under the exigency of an
OBTE environment to which they most closely correspond. The numbers in the leftmost columns indicates the
reference code for each measure. The grouping below can be used to help select student measures that
correspond to instructor measures when the exigencies of an OBTE environment are also used to select those
measures (e.g., Appendix A).
Leadership and Enculturation of Soldiers
1.1.1
Does teacher exhibit reasonable expertise in the subject area being taught?
4.3.2.1
Were the course learning objectives met?
1.3
Do the students view the teacher as a mentor?
1.1.2
Do the students behave as if they have confidence in, and respect, the teacher?
1.1.2
Do the students engage with the teacher outside formal periods of instruction?
2.3
Are there third party factors impacting student engagement in training?
2.3.1
Does the teacher/student role blur as the course continues?
4.3.1
Do the students demonstrate the ability to perform under highly stressful situations?
Integrated Understanding of Basic Soldier Skills (Prepared for FSO)
2.2.1
Do the students have the necessary materials?
2.1.3
10
How effectively do the students relate the discussion to their own experiences?
4.2.1
11
Do the students retain and apply factual knowledge (demonstrate a technical and contextual understanding)?
4.2.2.2
12
Can the students execute the correct standard procedure?
4.3.1
13
Do the students apply lessons learned to improve performance?
4.3.1
14
Do the students integrate task and skills into future training (e.g., understands the context of the skill, task)?
A mindset of collaborative reflection and problem solving
1.2
15
Do the students understand what is going on in the classroom?
1.4
16
When called for, how well do the students collaborate to solve problems?
2.1.4
17
Do the students engage in peripheral conversation?
2.1.4
18
Do the students effectively discuss their performance with each other?
2.1.1
19
Do the students engage the teacher regarding an idea or to challenge an assertion?
2.1.2
20
Do the students ask and answer questions?
3.2.1.1
21
Do the students struggle to discover a solution and solve novel problems?
4.1.4
22
Do the students pursue alternate paths to discovering the solution?
1.1.2
23
Is the teachers method or style of instruction conducive to learning?
3.1.2.1
24
Do the students questions drive the focus of discussion?
3.2.1.3
25
Do the students demonstrate a sense of accountability for their actions?
2.1.2
26
Do the students demonstrate initiative?
2.1.5
27
How effectively do the students discuss how their actions can impact the rest of the group?
2.2.2
28
Do the students exhibit body language that reflects engagement in the course?
2.3.1
29
Do the students maintain a high level of engagement or increase engagement over time?
3.1.2
30
Do the students take initiative to catch up when feeling overwhelmed or left behind?
3.2.1.3
31
How do the students react to teacher feedback to an incorrect solution?
Soldier motivation and self determination (development of intangibles)
B.2 Complete Menu of Student Measures

1.1 Confidence
1.1.1 Students demonstrate confidence in teachers subject matter expertise
4. **Does the teacher exhibit reasonable expertise in the subject area being taught so as to
create a positive learning environment?
c
c
N/A
N/O
c
c
Yes
No
Comments:
1.1.2 Students demonstrate confidence in teachers teaching

5. **Is the teachers method or style of instruction conducive to learning?
c
c
N/A
N/O
Not conducive to learning;

students cannot perform to
standard
Comments:
Conducive to procedural
learning; students can
repeat/follow a standard or
process
Conducive to contextual
learning; students can
adapt standard procedure
to new situations; teacher
articulates the why
320
6. Do the students behave as if they have confidence in, and respect, the teacher?
c
c
N/A
N/O
Students display negative

attitude towards teacher, do
not pay attention in class;
comment negatively to
other students; verbally
challenge teachers
expertise in an
inappropriate way
Students quietly takes

notes, but do not fully
engage in discussion or
dialogue (enforced respect)
Students challenge
teachers assertions
appropriately; engage
teacher during breaks;
ask appropriate, thoughtprovoking questions
(nurtured respect)
Comments:
7. Do the students engage with the teacher outside formal periods of instruction (e.g.
before/after class, during breaks)?
c
c
N/A
N/O
None of the students

Comments:
Some of the students

Most or all of the student

1.2 Teachers approach/reasoning

8. Do the students understand what is going on in the classroom?
c
c
N/A
N/O
Students cannot articulate

what is expected of them.
Students can articulate

what is expected of them,
but is task oriented.
Students can tie goals

and experiences together
to explain teachers
approach and reasoning
in written or spoken
work. (i.e. understands
broader purpose)
Comments:
1.3 Mentor
9. Do the students view the teacher as a mentor?
c
c
N/A
N/O
Students avoid direct

interaction with teacher
Comments:
Students interact with

teacher when prompted
Student seeks out

interactions with the
teacher (as appropriate)
322
1.4 Students collaborate to solve problems

10. When called for, how well do the students collaborate to solve problems?
c
c
N/A
N/O
Students exercise no
initiative to create group
effort, solution or
consensus; one person
may solve the problem
Work toward group

solution or consensus
when prompted; utilize
resources of one or two
people in the group
Immediately move to
establish a group /
collaborative effort;
utilize resources of all
people in group
Comments:
2 Student engagement in course

2.1 Making sense of instruction/content
2.1.1 Student participates in or engages in discussion
2.1.1.1 Students challenge an assertion; elaborate/engage in an assertion
11. Do the students engage the teacher regarding an idea or to challenge an assertion?
c
c
N/A
N/O
Student is not engaged;

does not ask questions or
pay attention
Comments:
Student asks question but it

isnt completely relevant
or has already been
answered (e.g. not paying
attention)
Student challenges teacher

for clarity (e.g., raising
counterpoint), question
relevant or demonstrates
student is beginning to
understand, student takes
idea one step farther
2.1.2
Students asking and answering question
12. Do the students demonstrate initiative?

c
c
N/A
N/O
Students wait for prompting

before responding with needed
action or avoid action
Students responds with

but do not further
dialog/problem with
additional ideas
Students correctly
responds with needed
and extend
concepts/actions one step
farther
Comments:
13. Do the students ask and answer questions?

c
c
N/A
N/O
Students asks questions on

procedures and processes to
meet course requirements, or
fail to ask questions even
when not understanding
Comments:
Students ask questions for

clarification; focus on
current topic and
application
Students ask questions to

advance the idea to other
situations; apply
discussion to applications
outside of test items
324
2.1.3
Relates discussion to own experiences
14. How effectively do the students relate the discussion to their own experiences?
c
c
N/A
N/O
Students past experiences

hinder learning of new
concept (e.g., reject new
notion).
Students articulate new

concept but do not reevaluate past experiences
or new challenges in light
of new understanding
Students correctly apply

new knowledge to reevaluate a past experience.
Comments:
2.1.4
Peripheral conversations (side conversations)
15. Do the students engage in peripheral conversation?

c
c
N/A
N/O
Students participate in
unrelated peripheral
conversations
Comments:
peripheral conversations
related to topics covered in
the course
peripheral conversations
related to understanding
and solving problems
related to the course.
Dialog extends course
concepts to new situations.
16. Do the students effectively discuss their performance with each other?
c
c
N/A
N/O
Students do not talk about

their performance and/or their
ability to improve
Students share
performance results; only
focused on grade/meeting
standard.
Students discuss content,

in addition to
performance, and critique
each other, in a way that
focuses on application to
real world.
Comments:
17. How effectively do the students discuss how their actions can impact the rest of the group?
c
c
N/A
N/O
Students do not express any

impact their actions might
have on the group
Comments:
Students expresses the

impact of their actions on
the group
Students express how their

actions can impact the rest
of the group, and actively
adjust to maximize
positive effects.
326
2.2 Student is engaged in training/instruction (when not affected by third party or

circumstantial events)
2.2.1 Materials
18. Do the students have the necessary materials?
c
c
N/A
N/O
Students do not have the

necessary materials prepared
(e.g. did not do homework)
Students have the required

materials prepared.
Students have completed

required work and have
taken it one step further
(i.e. took ownership of
their learning)
Comments:
2.2.2
Body Language
19. Do the students exhibit body language that reflects engagement in the course?
c
c
Yes
No
c
c
N/A
N/O
If no, what negative behaviors were observed?

Poor posture (e.g. head down)
Lack of eye contact
Students are distracted (e.g., looking elsewhere, reading other material)
Other
Comments:
2.3 Third-party factors impacting engagement in training

20. Are there third party factors impacting student engagement in training?
c
c
Yes
No
c
c
N/A
N/O
If yes, what challenges were observed?

Time constraints
Schedule issues
Social distractions: cognitive (personal, family,
leave)
Time of day
Individual and class distractions
Computers (internet available)
Lighting levels
Location of projector screen
Orientation of classroom (seating, etc.)
Temperature
Comments:
2.3.1 Students generally maintain a high level of engagement or show a pattern for
increasing engagement
21. Do the students maintain a high level of engagement or increase engagement over time?
c
c
N/A
N/O
Students are not engaged; do

not take ownership of
individual learning; want to be
given what is necessary to
meet course requirements
Comments:
Students are somewhat

engaged; do not take
ownership of individual
learning complete only
required material.
Students are actively

engaged; take ownership
of their individual
learning; discussions and
assignments consistently
go one step beyond
requirements.
328
22. Does the teacher/student role blur as the course continues?

c
c
N/A
N/O
Strict hierarchical roles;

students never assume a peerlike relationship with the
teacher, and/or engage in
collaborative problem-solving
with teacher.
Students only take

ownership of learning tasks
when directed to by
teacher (i.e., told to interact
with group); Problems and
solutions mandated by
teacher.
Teacher takes on a peerlike relationship with

students; becomes a
facilitator of students
own learning; Students
actively help to determine
problems and solutions
with teacher support
Comments:
3 Students relationship with teacher

3.1 Students have an effect on direction of instruction
3.1.1 Students do not get overwhelmed or left behind
3.1.2 Students feel that they have the ability to influence the training
23. Do the students take initiative to catch up when feeling overwhelmed or left behind?
c
c
N/A
N/O
Students do not speak up and

ask for help; do not respond to
teacher queries
Comments:
Students do not speak up,

but respond to teacher
when queried
Students proactively speak

up and ask questions when
falling behind or confused
3.1.2.1 Students questions drive focus of discussion (within reason)

24. Do the students questions drive the focus of discussion?
c
c
N/A
N/O
Students do not ask questions

or questions have no effect on
focus of discussion
Student ask only

clarification questions;
discussion does not deviate
from current topic or
application
Student questions enrich

the discussion and drive it
to expand across other
related topics or
applications
Comments:
3.2 Students discover solutions to problems (process)

3.2.1 Students struggle to solve problem
3.2.1.1 Students continually get closer to a solution
25. Do the students struggle to discover a solution and solve novel problems?
c
c
N/A
N/O
Students execute standard

procedures even when not
appropriate for problem; fail
to solve problem
Comments:
Students make some

progress on novel problem,
but fail effectively solve
the problem
Students adapt training to

discover an effective
solution to a novel
problem
330
3.2.1.2 Student reacts to failure to solve a problem)

3.2.1.3 Students change path based on Teacher feedback
26. How do the students react to teacher feedback to an incorrect solution?
c
c
N/A
N/O
Students do not change

behavior until given the
solution

after being prompted by
teacher with significant
guidance (e.g., leading
questions)

after being prompted by
teacher with minor
guidance (e.g., reminder)
Comments:
27. Do the students demonstrate a sense of accountability for their actions?

c
c
N/A
N/O
Students do not articulate

what they did
Comments:
Students articulate what

they did
Students articulate what

they did and what were
the implications of their
actions
4 Student results
4.1 Students understand the why behind the instruction
4.1.1 Students discover solutions to problems (results)
4.1.2 Students show change in their past assumptions based on feedback
4.1.3 Paths to solution are not identical because of the environment
4.1.4 Students understand the principles
28. Do the students pursue alternate paths to discovering the solution?
c
c
N/A
N/O
Students cannot solve problem
Students can solve problem

under standard conditions
Students can solve

problem under varying
conditions; adapt approach
to novel context
Comments:
4.2 Performance on tests and quizzes

4.2.1 Factual (declarative) knowledge retention
29. Do the students retain and apply factual knowledge (demonstrate a technical and contextual
understanding)?
c
c
N/A
N/O
Students do not retain

knowledge (e.g., cannot
answer question)
Comments:
Students retain knowledge

(e.g., give textbook
answer)
Students retain and apply

knowledge to novel
problems (e.g., show
application to mission,
technical and contextual
understanding)
332

4.2.2 Student can execute the standard procedure correctly
4.2.2.1 When appropriate
4.2.2.2 Sometimes there is only one way
30. Can the students execute the correct standard procedure?
c Yes
c No
Comments:
c
c
N/A
N/O
4.3 Transfer of lessons learned (near and far transfer)

4.3.1 Apply lessons/principles to own behavior (near transfer)
31. Do the students apply lessons learned to improve performance?
c
c
N/A
N/O
Students do not apply lessons

learned to change behavior;
continue to make same errors
Comments:

in accordance with lessons
learned
Students integrate lessons

learned with existing
behavioral strengths to
yields optimized
individual approach
32. Do the students demonstrate the ability to perform under highly stressful situations?
c
c
N/A
N/O
Students fail to perform in

stressful or dynamic situations

marginal performance in
stressful or dynamic
situations (arrive at
solution, but in nonoptimal way, or very
slowly)

action in stressful or
dynamic situations to
arrive at solution
Comments:
33. Do the students integrate task and skills into future training (e.g., understands the context of
the skill, task)?
c
c
N/A
N/O
Students do not integrate

skills; skills stay isolated such
that more advanced tasks
cannot be solved.
Comments:
Students integrate some

skills into other activities
to arrive at solutions under
standard conditions
Students integrate skills

and lessons learned across
the training program to
arrive at solutions in
varying conditions and/or
novel contexts.
334
4.3.2 Feedback from students (far transfer; post training)

4.3.2.1 Does instruction achieve desired outcomes?
34. Were the course learning objectives met?
c
c
N/A
N/O
None of the learning

objectives were met
Comments:
Course-specific objectives
were met.
Course objectives were

met and outcomes
advanced in a positive
manner
Self Report Questions

1.1.4: Student demonstrates trust in the teacher
3. Do you have trust in your teacher to develop you individual skill set?
I dont think the teacher

cares if I do well or not

interested in my success in
this course

genuinely interested in
my success in my
mission
1.5 Students perceive course as valuable

1.5.1 Course content accurate
4. General: Do you feel the course content is reflective of your experiences?
Course content is/was not

reflective
Some course content was

reflective of my
experiences, but some was
not. Doctrinally correct,
but not useful in COE.
Course is reflective of
my experience and
relevant to me/COE;
5. ABIC: Is the course content consistent with what you expect to do as a teacher based
on your previous experience with instruction?
Course content is not

relevant; Not improving
my skill level

relevant, but some was not.
All the course relevant;

I am learning things I
did not previously
know
336
6. ABIC: Is the course content consistent with what you expect to do as a teacher based
on your previous experience as a Soldier?
Course content is not

relevant; Not improving
my skill level
All the course relevant;

I am learning things I
did not previously
know

relevant, but some was not.
1.5.2 Student perceives value in achieving the learning objectives

7. Did you perceive the course as valuable?
I am taking this course

because it is required for
certification.
I think this course will

benefit me as a teacher
I think this course will

benefit me as a teacher
and a Soldier.
8. How engaged were you throughout the course?
I was bored throughout the

course
I participated in order to
get a good grade or was
occasionally motivated by
the teacher
I participated because I
was fascinated and excited
by what I was learning
Results/ Individual Development

9. Do you feel that you are a better problem solver as a result of your last practice
exercise?
Not at all; no different than

before I participated in the
exercise.
Yes, but only within the

context of this course
I feel I am a better
problem solver overall as
an individual
10. Do you feel that you are a better able to create and/or identify a positive learning
environment as a result of the last practice exercise?
No, I dont know how to

recognize if students are
learning or are engaged
I can identify when

learning is occurring but
Im not sure how increase
student learning or
engagement
I can identify when

learning is occurring and
know how to increase
student learning and
engagement
11. Do you feel that you are more confident in yourself as a result your last practice
exercise?
I dont know what good

instruction is and I cannot
teach well.
I know what good

instruction is, but Im not
sure I can teach well most
of the time.
I know what good

instruction is and I can
teach well most of the
time.
338
12. Do you feel that you have a higher sense of personal accountability as a result of your
last practice exercise?
A teacher cannot make a

difference in student
development.
I understand that a teacher

can make a difference in
student development but
Im not sure I personally
can make a difference.
As a teacher, I can make a

difference in student
development.
13. Do you feel that you will take more initiative as a teacher in future situations as a
result of you last practice exercise?
No, I dont know what or how

to change in my teaching to
increase student development.
I understand how my
teaching can be changed to
increase student
development, but I cannot
change what or how I
teach.
I understand what could

be better and can change
what and how I teach to
foster student
development
14. Do you feel that you have more awareness regarding your personal capabilities as a
result of your last practice exercise?
I am uncertain of my
limitations and capabilities as
a teacher
I am aware of my
limitations and capabilities
as a teacher, but I am
uncertain about how best
leverage my skills
I am aware of and know

how to leverage my
limitations and capabilities
as a teacher
15. Overall, how much do you believe the students learned from your last exercise as a
teacher?
I dont think the students

learned anything or very little
I think the students learned

some
I think the students

learned a great deal
340
Appendix C: Command Climate

Appendix C: A Commanders View of Outcomes-Based Training and Education
COL Casey Haskins
United States Military Academy
Summary
Outcomes-based training and education (OBTE) is a different system for training and for training
management. By way of analogy, OBTE is to training what mission command is to operations:
trainers are given requirements but not directed how they must achieve them. They are then held
accountable for the results.
Whereas standard Army training methods seek to teach Soldiers and leaders how to apply
approved, doctrinal solutions to particular problems, OBTE seeks instead to teach them how to
frame problems and solve them, focusing on the results rather than the methods used to obtain
them. It is thus designed to create thinking, adaptive Soldiers and leaders who are capable of
applying what they know to solve problems they have previously not encountered.
Definition
OBTE is an approach to planning, managing, and delivering training and education. It results in
the attainment of a set of holistic, observable, and measurable skills and behavioral traits
(outcomes) in individuals and units. It does so by requiring a thorough understanding of the
underlying principles and increasing mastery of fundamentals, gained while progressing through
a series of increasingly challenging scenarios. These scenarios at all times require the trainer and
student to think and solve problems in context.
Description
OBTE relies heavily on the trainers growing expertise and ability and much less on scripts,
standard procedures, or external controls. It standardizes training by the results or outcomes. In
other words, success is judged when the student or unit demonstrates they can solve a new
problem to an acceptable level using their newly trained skills or knowledge.
It does not standardize training by the process (every student will first do A, then do B, then
advance to C, and finally qualify on D) or by inputs (each student will fire three rounds, spend 2.5
hours on this topic, watch these slides, do three iterations of this drill, etc.) It recognizes that
many skills and situations have more than one acceptable way of solving the problem and does
not mandate one particular solution. Nor does it specify one way to teach a particular skill or to
learn it.
OBTE explicitly focuses at every step on developing such intangible traits as critical thinking,
judgment, problem solving, initiative, and accountabilityin both the instructors and the
students.
Though there are similarities, Outcomes-Based Training and Education are not just new words for
old concepts. Outcomes are not merely terminal learning objectives. Measures of effectiveness
are not merely task standards. Many senior leaders, upon hearing of OBTE, claim theyve always
done it. This is a common misperception. The vast majority have not.
341
Haskins
Finally, OBTE requires no more resources than the standard Army training model. While it will
require a new method for allotting resources, since inputs (number of rounds, hours, miles, etc.)
no longer drive the training, a standard package of resources will still suffice, so long as the
trainers have more flexibility about how to use them.
Elements of OBTE.
Developing the Outcomes
This is the first step and arguably the most important. It requires clear thinking, and to succeed it
requires ownership by the leaders who will have to achieve them. The outcomes should be
simply stated in clear language. Ideally they are broad rather than detailed, and they are holistic.
In other words, they are not meant to stand alone, but together. That is important because it
prevents a leader from conducting training that might further one outcome while hindering or
even going backward in others. In the case of a unit, the outcomes are a statement of what the
unit should be able to do and they should describe its values and behavior. In the case of a
school, they specify what each graduate should be and be able to do.
Each outcome is then expanded one level into measures of effectiveness (MOE). These MOE
answer the question what does success in this outcome look like? Taken together, the MOE
define success. Furthermore, each MOE should be measurable or at least observable, so that they
form not just the basis for the training program, but also the basis for the assessment.
Developing the Training Plan
OBTE is most similar to standard training in this step. However, there are two main differences.
The training plan is not the point around which everything hinges; the outcomes are. So if
something isnt working, the plan changes. Traditionally, units have tended to do this well, but
schools were much more rigid. Using OBTE, the POI in a school will change from one cycle to
the next as leaders assess success or failure at achieving outcomes.
The other way in which OBTE differs from standard Army training plans is that trainers receive
more latitude. Less is directed by higher headquarters, but the plan correspondingly builds in
accountability. For instance, if land navigation is part of a school, the plan might build in several
days with little direction, available to junior leaders, and resourced with training areas and
transportation. There would be no expectation that each leader would use the time the same way.
A description of the types of navigation problems each student would be expected to solve on the
final day should provide sufficient guidance for trained leaders to teach their Soldiers; those
Soldiers performance will then show clearly how well each leader did and allow the chain of
command to hold the leaders accountable.
342
Conducting Training
This is the area in which OBTE most differs from standard Army training. The differences stem
mostly from an attempt to make training better fit human nature, rather than working against it as
Army training often has. It is also geared to fit better with the ambiguity and complexity of
todays fights.
Far more flexibility is granted to both teachers and students, with far fewer external
controls. This requires an investment in training leaders (both on the skills and on how to
teach) but an overwhelming majority of both leaders and Soldiers respond
enthusiastically and perform at a much higher level.
Training emphasizes principles rather than checklists, procedures, or standards. While

there are certainly minimum standards for what an acceptable performance is, they are
rarely discussed with the students, who therefore tend to strive to do their best.
Training emphasizes the why. Traditional Army training emphasizes the what and
how but too often neglects the why. For instance, every Soldier knows how to low
crawl, but surprisingly few can explain when it might make sense to do so. They
therefore avoid doing it at all or they tend to do it in inappropriate circumstances. OBTE
builds in the why from the beginning, and reinforces it at every step by requiring the
student to solve problems using principles and newly acquired knowledge and skills.
As students solve problems, and learn by doing, they are required to figure as much of it
out for themselves as they are capable of. This guided self-discovery, in which the
leader helps the student only through the bits of a problem that he cant quite figure out
for himself, helps the student to more thoroughly understand the concepts and to connect
them to their context. While it may take a bit longer to learn, the learning is deeper and
retained longer. Students in OBTE are conditioned from the beginning to think and to
solve problems. In much of traditional Army training, by contrast, students are told what
to do, whether or not they understand why. They are therefore being conditioned not to
think and not to solve problems, but rather to follow sequences and procedures. This
conditioning causes them difficulty when they are forced to adapt to new and unexpected
circumstances. Not only are they less experienced at solving problems, but they are
conditioned not to try.
Problems in OBTE are realistic and dont divorce the skill from its context. To take an
example, treating a casualty in combat is different from treating one in a classroom or on
a range. This situation changes the medical response. For instance, the need to move the
casualty to a less vulnerable location for further treatment not only changes the sequence
of treatment steps in controlling bleeding (a tourniquet is immediate rather than after
trying direct pressure and elevation) but also requires the Soldier to balance medical
treatment with security, control of the unit, and accounting for personnel. Together, this
results in a fundamentally different learning experience from traditional training that
isolates each task and trains them one at a time.
(Note: Don Vandergriffs Adaptive Learning Model is essentially a classroom variant of
this. Students are challenged with tactical decision exercises of increasing difficulty, so
that as they learn to solve complex tactical problems, they do so in realistically
challenging scenarios, requiring difficult decisions, tradeoffs, and follow-through.)
343
Haskins
Stress is reversed. Normally in an Army school, the student will experience a high level
of stress at the beginning. Whether induced intentionally or inadvertently through the
unfamiliarity of the situation, that stress usually interferes in mastering the fundamentals.
The student is more focused on avoiding trouble than in mastering the skills. OBTE, in
contrast, begins training with little or no stress. Treating the students as adults, trainers
explain skills thoroughly, breaking them down into their fundamentals, making sure
students understand principles and the why behind each skill, but without talking down
to them. On the other hand, as the student begins to master the basics, OBTE presents
him with increasingly difficult problems to solve with his new skills, even while he
struggles to master them. This results in increasing stress as the training progresses,
which helps the student to cement his skills and understanding, helps him to gain
confidence as he succeeds at handling more challenging problems, and helps him learn to
manage high levels of stress as he performsall with little or no yelling or other artificial
stressors. Students in traditional Army schools, by contrast, tend to experience decreased
stress as they adjust to the environment and get more comfortable. (Ranger students later
in the course require more yelling or bigger threats to induce a similar response to early
days.) This reduced stress results in incomplete learning, as the student often fails to
lock in the skills in a realistic combat context.
OBTE aims for a much higher level of mastery of individual and team skills. The
tyranny of throughput and external controls means that traditional Army training (task,
conditions, standards) results in everyone meeting minimum standards, but often
prevents Soldiers from achieving as much as they are capable of, and often penalizes
initiative and new approaches. This is certainly true in schools, but also often true in
units. The results are unfortunate but predictable. The level of individual expertise in
many units is too low to enable a unit to excel in combat situations. Worse, the
traditional approach can foster a climate where Soldiers and even leaders tend to wait to
be told what to do next, rather than exercising judgment and initiative.
Mistakes in OBTE are treated as opportunities for learning to occur. Students are
encouraged to try things for themselves and to learn from their mistakes. The leaders
role is to make sure that the students analyze why something went wrong and to draw
reasonable lessons from the experience. (Of course the leader is expected to exercise
judgment and not allow mistakes that will result in catastrophe.) Too often, Army
training treats mistakes as things to be avoided or nipped in the bud. Leaders are
rewarded for avoiding mistakes or correcting them immediately and are rarely
encouraged to allow mistakes to unfold so that their Soldiers can learn from them.
Accountability is built into OBTE. Leaders are given both responsibility and the
authority they need to conduct the training their own way. They can then be held
accountable for the results, which are usually clear. But if trainers are required to train a
task in a certain way, as is often the case in Army training, it is difficult to blame them if
their Soldiers dont learn. In OBTE, students too learn accountability. Required to solve
problems, and given the authority to try different solutions, they learn to accept
responsibility for their actions and decisions. Results matter. In almost every case,
students come to see accountability as an enabler, allowing them to solve problems their
own way.
344

How Training is Assessed
Soldiers are given problems to solve and then watched as they solve them. Those problems
should be unfamiliar to them, but which they could be reasonably expected to solve given their
level of skills and experience. If the problems are properly designed, the trainers who are
observing them will not only see clearly whether the students have mastered the skills, but will
see whether they truly understand why things are done, and will gain insight into such intangibles
as initiative, judgment, and accountability. An added benefit is that the assessment itself is good
training for the students being assessed.
Conclusion
In an Army of motivated volunteers, expected to succeed in difficult and rapidly evolving
battlefields, OBTE is a better way to prepare. It aligns more closely with the way people learn.
While results are preliminary, the evidence is clear that it results in superior mastery of skills,
better retention, higher levels of confidence, and improved judgment, initiative, and
accountability.
On the other hand, implementing it poses challenges. It relies much more on leaders abilities,
and so requires investment in better preparing leaders. They need to have the skills, they need to
be able to explain why things are done the way they are, and they need to be able to teach. It
requires the right command climate. Strict insistence on uniformity and standardization will stop
it dead, since OBTE requires both leaders and students to experiment with different solutions. It
requires commanders to create conditions that allow it, including neutralizing external agency
vetoes, and removing rules and processes that prevent initiative. Finally, it will require a different
method of allocating resources to training, and more flexibility in using them, since resources are
currently matched to tasks being trained rather than to skills attained. Nevertheless, it can be
done, and when done it works very well.
345
Riccio et al.
Appendix D: Warrior Ethos
Analysis of the Concept and Initial Development of Applications
From: Riccio, G., Sullivan, R., Klein, G., Salter, M., & Kinnison, H. (2004).
Warrior ethos: Analysis of the concept and initial development of applications.
ARI Research Report 1827. Arlington, VA: US Army Research Institute for the
Behavioral and Social Sciences. (pp. 1-12)
The U.S. Army has adopted a set of Army Values and as a part of basic training it inculcates
initial entry Soldiers with their importance. The values reflect societal beliefs to which most
American citizens would ascribe. The values are not Army or combat specific, yet they set a
foundation designed to develop right beliefs and responsible actions by the American Soldier.
While not unique to the profession of arms, the values are certainly important to the citizen turned
Soldier, whose actions within the institution of the Army must reflect the values of the Nation as
a whole. The Army Values of Leadership, Duty, Respect, Selfless Service, Honor, Integrity and
Personal Courage can stand alone.
In contrast to the Army Values, notions of a Warrior Ethos as an overriding credo for the
American Soldier provide a unique set of values to complement the seven Army Values. The
values of a Warrior Ethos would be peculiar to the needs of an Army which is required by the
Nation to fight, but at the same time is required to be in consonance with the character, sentiment
and beliefs commonly held by the American people. The purpose of a set of values used to
underpin or describe an Army level Warrior Ethos may be to ingrain the belief that failure by an
Army and its Soldiers is not acceptable while the means to fight exists.
This paper describes the first phase of a multi-phased approach to addressing the area of Warrior
Ethos. This work, done under the auspices of a Small Business Innovative Research (SBIR)
contract, centers on Warrior Ethos for initial entry Soldiers, but can easily be expanded to the
broader force to include officer training and reinforcement training for personnel already in units.
The first steps of the process focused on definition of Warrior Ethos, a dissection of the terms and
tenets of Warrior Ethos into their component parts, together with in depth definitions of each of
the tenets. Potential arenas for training-based Warrior Ethos interventions were identified, and
plans laid for their implementation.
Warrior Ethos is at the heart of the expectations of a warrior, a Soldier who performs required
duties in a harsh and unforgiving environment which directly involves killing and also provides
potential for being killed. Warrior Ethos is implicit in the Armys Code of Conduct; it is explicit
in the historical records of the Armys combat heroes, particularly those recognized by the
Congressional Medal of Honor, the Distinguished Service Cross and the Silver Star.
Current Understanding of Warrior Ethos
The Warrior Ethos Staff Primer (Training and Doctrine Command, 2003b), presented to Chief of
Staff of the Army (CSA) GEN Peter J. Schoomaker, is a seminal document from the Task Force
Soldier (Task Force Soldier, 2003) within the Training and Doctrine Command (TRADOC). The
basis of the work is the motivational principle that transforms American Soldiers into Warriors
the Warrior Ethos. Newly re-defined and adopted by the Army in November, 2003, the Warrior
Ethos is the heart and central focus of the new Soldiers Creed (Training and Doctrine Command,
2003a). The Creed is depicted in Figure 1, and the tenets of Warrior Ethos are completely
embedded therein.
346
Figure 1. The Soldiers Creed

The Warrior Ethos helps to ensure that all Soldiers, regardless of rank, branch or military
occupational specialty, are prepared to engage the enemy in close combat, while serving as a part
of a team of flexible, adaptable, well-trained and well-equipped Soldiers. Warrior Ethos is a part
of everything the Army does. It is the basis of leader decision-making, and has an impact on
Soldiers at every level from the most senior General Officer down to the most junior Soldier.
Warrior Ethos defines the American Soldier.
The concept of Warrior Ethos is not difficult to understand; far more difficult are decisions on
how to ensure the effective dissemination of the overall ethic of a Warrior. Initial Military
Training begins a process of immersive inculcation of Army Values to all new Soldiers. Trainees
learn about the Army and are exposed to attitudes and behavior associated with Warrior Ethos.
The challenge is to internalize Warrior Ethos to the greatest extent possible during the limited
timeframe and then to sustain it well beyond this initial training. A Soldiers lifestyle and daily
behavior must reflect Warrior Ethos. Warrior Ethos must continue with the Soldier to his or her
advanced individual training program location, then to the unit. The importance of sustaining
Warrior Ethos is indicated in vignettes describing actions that resulted in award of the
Congressional Medal of Honor (U.S. Army Center of Military History, n.d.).
Private First Class Melvin L. Brown, U.S. Army, Company D, 8th Engineer Combat Battalion
exhibited extraordinary heroism on 4 September 1950, and demonstrated the first three tenets of
Warrior Ethos.
While his platoon was securing Hill 755 (the Walled City), the enemy, using
heavy automatic weapons and small arms, counterattacked. Taking a position on
a 50-foot-high wall he delivered heavy rifle fire on the enemy. His ammunition
was soon expended and although wounded, he remained at his post and threw his
few grenades into the attackers causing many casualties. When his supply of
grenades was exhausted his comrades from nearby foxholes tossed others to him
and he left his position, braving a hail of fire, to retrieve and throw them at the
enemy. The attackers continued to assault his position and Pfc. Brown
347
Riccio et al.
weaponless, drew his entrenching tool from his pack and calmly waited until they
1 by 1 peered over the wall, delivering each a crushing blow upon the head.
Knocking 10 or 12 enemy from the wall, his daring action so inspired his platoon
that they repelled the attack and held their position. [Mission first, never accept
defeat, never quit]
A more recent and compelling example of the four tenets of Warrior Ethos is that describing the 3
October, 1993 actions and extraordinary heroism of Master Sergeant Gary I. Gordon and
Sergeant First Class Randall D. Shughart. The citations show that Master Sergeant Gordon and
Sergeant First Class Shughart distinguished themselves by actions above and beyond the call of
duty while serving as Sniper Team Leader, and Sniper Team Member, United States Army
Special Operations Command with Task Force Ranger in Mogadishu, Somalia.
Master Sergeant Gordon's sniper team provided precision fires from the lead
helicopter during an assault and at two helicopter crash sites, while subjected to
intense automatic weapons and rocket propelled grenade fires. When Master
Sergeant Gordon learned that ground forces were not immediately available to
secure the second crash site, he and Sergeant First Class Shughart unhesitatingly
volunteered to be inserted to protect the four critically wounded personnel,
despite being well aware of the growing number of enemy personnel closing in
on the site. After their third request to be inserted, they received permission to
perform this volunteer mission.
Equipped with only sniper rifles and pistols, Master Sergeant Gordon and
Sergeant First Class Shughart, while under intense small arms fire from the
enemy, fought their way through a dense maze of shanties and shacks to reach
the critically injured crewmembers. They pulled the pilot and the other
crewmembers from the aircraft, establishing a perimeter which placed them in
the most vulnerable position. They killed an undetermined number of attackers
while traveling the perimeter, protecting the downed crew. Their actions saved
the pilot's life. Sergeant First Class Shughart continued his protective fire until he
depleted his ammunition and was fatally wounded. After his own rifle
ammunition was exhausted, Master Sergeant Gordon returned to the wreckage,
gave a rifle with the last five rounds of ammunition to the dazed pilot with the
words, "good luck." Then, he radioed for help and armed only with his pistol,
Master Sergeant Gordon continued to fight until he was fatally wounded.
The preceding vignettes depict extreme examples of Warrior Ethos: I will always place the
mission first. I will never accept defeat. I will never quit. I will never leave a fallen comrade. It
is clear that Soldiers immediately recognize this when such historical deeds are described to
them. However, the average Soldier is not continually exposed to conditions within which
Warrior Ethos is clearly manifested and do not frequently experience the conditions that foster
Warrior Ethos. This is the case whether they are in garrison or in a combat situation. There is a
need and an opportunity to develop training curricula that foster the development and sustainment
of Warrior Ethos. There is a collateral need and opportunity to operationalize the definition of
Warrior Ethos so that progress toward achievement of such training objectives can be measured.
348
Purpose
The purpose of this report is to document, expand and clarify the concept of Warrior Ethos, to
present a foundation for the development of a training package to inculcate Warrior Ethos into
initial entry training, and to show how it can be done. A future report will document the trial
implementations and summarize the recommendations for further research that could, in
principle, be conducted.
Approach
The objectives of the project were to review, analyze and define Warrior Ethos and to determine
the most likely sites for training interventions. A literature review was conducted to provide
traceable conclusions about Warrior Ethos and to show why a Warrior Ethos training package
should be developed. A front-end analysis determined the components of the constructs
associated with Warrior Ethos, the fundamental attributes embodied therein, and environments
suitable to enhance these values. Training techniques were identified to determine if and how the
attributes of Warrior Ethos can be imparted to Soldiers. Discussion of the performance metrics
and venues was initiated. Warrior Ethos must include Combat Arms, Combat Support, and
Combat Service Support operations. The basic combat training (BCT) environment thus provides
a logical entry level training site, whether it is the gender-integrated BCT found at, for example,
Fort Jackson, SC, or the One Station Unit Training (OSUT) at Fort Benning, GA.
Expansion of the Definition of Warrior Ethos
The perspective of the CSA on Warrior Ethos (see, e.g., Training and Doctrine Command, n.d.)
and its centrality in the Soldiers Creed suggests that Warrior Ethos is a mindset of commitment.
This is taken as a first principle in the ongoing refinement of the understanding of Warrior Ethos.
In particular, it points to persistent causal factors in an individuals attitude or disposition, which
strongly influence the individuals perception of a situation and actions in a situation. Moreover,
it indicates a disposition of being intellectually or emotionally motivated to think, perceive, act,
or forbear with respect to something or someone to which the individual is bound beyond the task
at hand (see also Field Manual (FM) 7-1, Battle Focused Training, Department of the Army
(DA), 2003.). Treatment of Warrior Ethos, or its inculcation, must address the cognitive and
social implications of commitment.
In promulgating an appreciation of Warrior Ethos in the Warrior Ethos Staff Primer (Training
and Doctrine Command, 2003a), the CSA has endorsed the four specific principles or elements of
doctrine, which herein are referred to as tenets of Warrior Ethos. The tenets are:
place the mission first

never accept defeat
never quit
never leave a fallen comrade
These tenets reveal the nature of the Warrior Ethos commitment. There is an explicit commitment
to ones fellow Soldiers. The tenets also imply a commitment to an organization, to a group of
whatever size is necessary to execute a mission successfully, insofar as the mission is the raison
dtre for the organization or group. The groups to which a Soldier must be committed, and that
influence the Soldiers mindset, can be nested. Sometimes the commitment can be described as to
a single individual, another member of a team, or to the elements within a squad, to the platoon or
company within a larger operational unit, all the way up to the Army as a whole, and ultimately to
349
Riccio et al.
the Nation. The guidance of the CSA suggests that Warrior Ethos requires an understanding by all
Soldiers of the interrelationships of such nested groups, not only with respect to the objectives
that smaller groups derive from larger groups, but also because of the values that sustain larger
groups (e.g., an Army, a nation). Understanding the relevance of ones thoughts and actions to
this social context increases the likelihood that ones thoughts and actions will be motivated by
something larger than oneself.
The Warrior Ethos Staff Primer (Training and Doctrine Command, 2003) has implications for
each of the areas of Doctrine, Organization, Training, Materiel, Leader Development and
Education, Personnel, and Facilities (DOTMLPF) although the primary focus here is doctrine,
training, leadership and personnel. A full definition of Warrior Ethos requires identification and
assessment of these implications. Part of the mission of the recently concluded Task Force
Soldier was to review, analyze and make recommendations about training as a mechanism for
implementing Warrior Ethos or Warrior Spirit (Task Force Soldier, 2003). The Task Force
Soldier mission required coordination of such proposals with other Soldier initiatives and
addressing life long learning for Soldiers (see also FM 7-1, DA, 2003, Para 1-6). This mandate
implied a holistic approach to training in which different types or phases of training, as well as
education, are sequenced and integrated with respect to the objective of inculcating Warrior
Ethos. It also made possible this approach to solving the problem of sustaining the elements or
tenets of Warrior Ethos beyond initial training and on into the Soldiers career.
The Tenets of Warrior Ethos
Mission First. The concept of mission first is an expression of priority of tasks. The
prioritization of tasks occurs on the battlefield just as it does in training and allows Soldiers and
Leaders to rank order or prioritize what must be done. The foremost task is the mission of an
individual Soldier or that of the unit. The concept of mission first is based on the oath of office
or enlistment and the Uniform Code of Military Justice (UCMJ). All Soldiers begin their career
with the swearing in ceremony where they recite the oath of enlistment or commissioning. The
oaths are shown below.
The current oath of enlistment is: "I, _____, do solemnly swear (or affirm) that I will support and
defend the Constitution of the United States against all enemies, foreign and domestic; that I will
bear true faith and allegiance to the same; and that I will obey the orders of the President of the
United States and the orders of the officers appointed over me, according to regulations and the
Uniform Code of Military Justice. So help me God." (Title 10, US Code; Act of 5 May, 1960
replacing the wording first adopted in 1789, with amendment effective 5 October, 1962).
The current oath of office is: "I, _____ (SSAN), having been appointed an officer in the Army of
the United States, as indicated above in the grade of _____ do solemnly swear (or affirm) that I
will support and defend the Constitution of the United States against all enemies, foreign or
domestic, that I will bear true faith and allegiance to the same; that I take this obligation freely,
without any mental reservations or purpose of evasion; and that I will well and faithfully
discharge the duties of the office upon which I am about to enter; So help me God." (DA Form
71, 1 August, 1959, for officers.)
If an individual interprets the oath as a mission statement, it implies an obligation to follow
civilian and military leaders in the execution of duties. This oath creates a framework for putting
the mission first, swearing to support and defend the Constitution of the United States.
350
The leaders challenge is to accomplish the mission while at the same time looking out for the
welfare of the Soldiers. Soldiers must be available to assist the leader in accomplishing the
mission. Therefore, mission and Soldiers are both important. Placing the mission first requires
prioritization of competing tasks and requirements, including those not directly related to the
military mission. Soldiers must balance their learned needs (family, safety, comfort, etc.) and deconflict these needs with the requirements of the mission. This prioritization enables Soldiers to
accomplish multiple difficult and sometimes selfless tasks concurrently, with the end state result
of accomplishment of the primary mission as well as multiple sub-tasks. Delaying actions
because of conflicting priorities/requirements does not mean that the Soldier is not placing the
mission first; it indicates the Soldier is multi-functional and balanced and prioritizes. Soldiers
historically have sacrificed in order to accomplish their mission.
Never Accept Defeat. The concept of not accepting defeat is found in the Code of Conduct. The
Code requires every Soldier to fight and not to surrender of his or her own free will. All Soldiers
are expected to fight an enemy for as long as the Soldier and the unit have the means to resist.
The idea of not accepting defeat does not necessarily include censuring failure. The U.S. Army is
discrete in the use of the terms failure and defeat. In practice the Army Values Soldiers and
Leaders, who if temporarily defeated in battle, will seek ways and means to continue the fight and
achieve victory. Examples abound in events of U.S. Army history where Soldiers and units
seemingly defeated, continue to fight, or in some cases disengage, to live to fight another day.
This too is an exemplar of not accepting defeat.
Training methodologies must stress upon the Soldier that the individual is never removed from
the fight so long as the Soldier has a prudent means to resist and carry the fight to the enemy.
This does not mean that we train Soldiers to conduct suicide missions as currently practiced by
our enemies conducting the war on terror. We operate within the confines of the Rules of Land
Warfare, regardless of the actions of our enemy.
Current exercise rules of engagement (EXROE) used in most field training exercises dictate that
when a Soldier becomes a simulated casualty, the Soldier immediately removes him/herself from
the fight. These narrow training-based EXROE may impede the development of the Warrior
Ethos desired during real enemy contact where the Soldier continues to press the fight toward the
enemy until the Soldier is no longer physically capable of resistance and is forced to stop.
Training methodologies as applied in typical training environments must reward Soldiers for
creative thinking as they face imminent defeat. What cunning and adaptive thinking does the
Soldier demonstrate that enables the ability to overcome adversity?
Never Quit. When does a Soldier terminate the fight and surrender his force to a numerically
superior enemy? When is it prudent to save your Soldiers rather than to commit the needless
sacrifice of their lives? There are examples throughout history in which commanders surrendered
their force to a superior enemy force. Having surrendered, as demonstrated repeatedly by
Prisoners of War, it is possible to continue to resist ones captors for a great length of time. The
elements of physical toughness, mental acuity and mental toughness (not succumbing to mind
games), and spiritual fitness (faith in the unit, leaders, and a higher being) help inculcate the
never quit tenet. This toughness enables the Soldier to continue to pursue the mission against
seemingly insurmountable odds.
Never quitting implies not letting oneself down and not letting buddies, subordinates and
superiors down. This comes from the belief that others depend on you and that to quit will
endanger your friends. Wong, Kolditz, Millen, and Potter (2003) provide insights into this
phenomenon, particularly that the notion of quitting will endanger the members of the small unit.
351
Riccio et al.
They noted that social cohesion serves two purposes in combat motivation. First, because of the
close ties to other Soldiers, the unit cohesion places a burden of responsibility on each Soldier to
achieve group success and protect the unit from harm. The second role of cohesion is to provide
the Soldier the confidence and assurance that someone he/she could trust was, in effect, watching
out for them.
Never Leave a Fallen Comrade. Soldiers enter into otherwise perilous situations because they
have trust and confidence in their country and their leaders that they will not be abandoned. A
classic example of efforts made to recover fallen comrades is that of Joint Task Force Full
Accounting, established in 1992, and based on 1973 accounting efforts after Viet Nam. It may not
be possible to recover injured or killed comrades in the heat of battle, but we must return and
pursue their recovery at the earliest opportunity and continue this mission until closure is
obtained, until all have been retrieved. The Central Identification Laboratory in Hawaii continues
to recover remains of Soldiers lost in action during World War II and Korea. This relentless
pursuit to achieve closure for family members and fellow Soldiers provides the individual Soldier
the peace of mind required to maintain the never quit posture while being assured that if
wounded or killed, never leave a fallen comrade will prevail.
Faith in Themselves and Their Comrades. Wong, Kolditz, Millen, and Potters conversations
with Soldiers (2003) showed that once they are convinced that they are being looked after, and to
the maximum extent possible, their own personal safety will be assured by others, they feel
empowered to do their job. They can maintain the fight, knowing they are not alone. This
discovery is central to this tenet because it relates to protecting each other, and provides some
relief from stress. Soldiers want to know that if they are wounded in action their buddies and unit
will fight to prevent their capture. They expect to receive medical treatment in a timely manner,
and if needed, Soldiers and their families expect their remains to be repatriated at some point after
the battle or conflict. All of this provides a level of comfort and trust among Soldiers that is
essential to combat performance at the small unit level.
Clarifying the Definition of Warrior Ethos
Current thinking about Warrior Ethos is reminiscent of well-established beliefs about leadership.
The historical lessons learned about Army leadership provide a solid grounding for development
of a strategy to inculcate Warrior Ethos in all Soldiers. In addition, promulgation of certain
attributes of leadership to all Soldiers has implications for leader training as well as basic training
(see e.g., FM 7-1, DA, 2003, Para A-2, A-15). For these reasons, special consideration was given
to lessons learned as articulated in FM 22-100 Leadership (DA, 1999). Key considerations are
summarized below:
The will of Soldiers is three times more important than their weapon (Col. Dandridge M.
Malone, Small unit leadership: a commonsense approach, as quoted in FM 22-100, DA, 1999, p.
2-11). Will is especially important under adverse external conditions. In such situations, the
perseverance to complete the mission must come from an inner resolve that derives automatically
from a fundamental commitment to something larger than oneself.
The core of a Soldier is moral discipline. It is intertwined with the discipline of physical and
mental achievement. Total discipline overcomes adversity, and physical stamina draws on an
inner strength that says drive on (Former Sergeant Major of the Army William G. Bainbridge,
as quoted in FM 22-100, DA, 1999, p. 2-12). Self-discipline and will manifest in similar ways
and under similar conditions. Self-discipline refers more to the activities of thinking and acting,
the means to the end, while will refers more to the end objective. Self-discipline thus is linked
352
with mastery. Mastery requires hard training that exposes individuals to adverse external
conditions often enough to develop a habit of doing the right thing.
The leader must be an aggressive thinkeralways anticipating and analyzing. He must be able
to make good assessments and solid tactical judgments (BG John T. Nelson II, as quoted in FM
22-100, DA, 1999, p. 2-12.). Initiative is especially important in ambiguous situations. It requires
adaptability in the means to the end without undue risk to achieving the end objectives. Initiative
thus requires a keen understanding of a commanders higher-level objectives or intent and the
ability to differentiate this from the a priori plan to achieve the objectives. This capacity must be
developed through experience with situations in which one is allowed some license to innovate
and which foster an appreciation of means-end relationships among multiple actions and multiple
objectives.
I learned that good judgment comes from experience and that experience grows out of mistakes
(General of the Army Omar N. Bradley, as quoted in FM 22-100, DA, 1999, p. 2-13). Judgment
is closely related to initiative. Successful initiative depends on good judgment. Judgment refers to
the ability to deal with conflicting demands, conflicting information, and ambiguous situations. It
must be developed through experience, some trial and error, reflection on consequences of ones
actions for oneself and others, and an acceptance of the responsibility for ones actions.
There is an important distinction between self-confidence and false confidence. Self-confidence
comes from an understanding of ones capabilities and limitations. Development of such selfknowledge comes from experience with a variety of situations in which one is required to
demonstrate initiative and exercise judgment. The attendant ability to deal with the unknown
manifests itself in a demeanor that tends to control doubt and reduce anxiety in oneself and others
in adverse conditions (i.e., when it is most needed). The term self-knowledge is used to avoid
confusion with the broader self-awareness considered important for leadership and especially for
strategic decision-making (see, e.g., Army Training and Leadership Development Panel, n.d.).
Nevertheless, it will be useful to consider implications of work on self-awareness in leader
training for self-knowledge in basic training.
It is not genius which reveals to me suddenly and secretly what I should do in circumstances
unexpected by others; it is thought and meditation (Napoleon Bonaparte, as quoted in FM 22100, DA, 1999, p. 2-14). There are three important and related issues pertaining to intelligence.
First, the value of experience is increased immeasurably by reflecting on lessons learned,
conducting such analyses in the context of values and other military principles, and extrapolating
them to various what-if scenarios. This is the basis of the After Action Review (AAR).
Additionally, application of ones intelligence in this way allows any person to become smarter,
and this is more important than the fact that some individuals may be smarter than others. Finally,
there should be a commitment to continuous intellectual development in ones job or
responsibilities.
Individuals bring different talents to a situation. The special talents of individuals can be well
utilized if teams are assembled so that team members balance and supplement one another and if
there is an appreciation by each team member of the complementary skills of all team members.
This increases the effectiveness of teams in achieving common objectives. In addition, such
awareness increases the ability to understand the cultural differences of adversaries and noncombatants, which, in turn, make the Army a more effective force.
Again citing the Armys Leadership Manual, FM 22-100 (DA, 1999), cultural awareness is part
of a mutual respect that is at the heart of a team identity. Such respect fosters appreciation for
353
Riccio et al.
complementary talents and assignments within a team. The team identity goes beyond this
appreciation, though, to include trust in the selflessness of ones comrades in the most difficult of
times. This is important not only because it increases the adaptability and effectiveness of teams,
but also because it a key Army value that should be shared by all members of this culture.
Warrior Attributes Derived from the Tenets of Warrior Ethos
The review of current thinking about Warrior Ethos and leadership provides a foundation for an
analysis of the Warrior Ethos that helps build connections between the associated military
considerations and current understanding of related concepts in the scientific community. This
analysis, in turn, will provide for the development of a training curriculum to inculcate Warrior
Ethos. It will involve the development and integration of new training concepts to foster life
long training and education. This approach requires that identified needs and solutions be traced
to scientific theory. It also requires that solutions can be implemented and that hypotheses about
solutions (i.e., their relationship with needs) can be examined empirically with methods that can
be replicated by others. The intent was thus to identify a set of concepts relating to Warrior Ethos
that referred, as explicitly as possible, to individual dispositions (i.e., cognitive or socialpsychological attributes) which could be addressed, as directly as possible, with training
interventions.
Seven attributes of individual psychology and behavior have been identified as key cross-cutting
elements of the four tenets of Warrior Ethos:
Perseverance
Ability to Set Priorities
Ability to Make Tradeoffs
Ability to Adapt
Ability to Accept Responsibility for Others
Ability to Accept Dependence on Others
Motivated by a Higher Calling
These attributes of Warrior Ethos are further described in the following paragraphs along with
representative measurement concepts which could be developed into qualitative and quantitative
methods of validation and verification for the training package and, if operationally viable and
feasible (given time and resource constraints), would be developed in parallel with the associated
training concepts.
Perseverance. Warrior Ethos requires an ability to work through adversity, to persevere at all
times, and to embody each of the four tenets of Warrior Ethos (mission first - never quit do not
accept defeat leave no comrade behind). Soldiers will encounter friction, in the form of
resistance or impediments to performance, during the execution of planned tasks. Often, as a
result of such external obstacles, it will appear to be easier to quit than to complete the task at
hand. Individuals must readily and reflexively draw on sources of inner strength to endure
adverse conditions and persist, even when it is not immediately obvious that the objective can be
achieved. In principle, external conditions and accompanying stressors can be manipulated in a
manner that can be replicated in the training environment, with measurable behavioral effects
(e.g., choices).
Ability to Set Priorities. Warrior Ethos requires an ability to prioritize tasks for mission
accomplishment if mission first is to be its first tenet. Every Soldier will have multiple tasks to
perform on a daily, hourly and, sometimes, minute-by-minute basis. This necessitates the
prioritization of work based on an understanding of what is most important to least important in
354
the context that all specified and implied tasks must be performed. In principle, it ought to be
possible to shift or change the relative importance of specific training events or tasks to provide
measurable behavioral effects to show the ability to prioritize.
Ability to Make Tradeoffs. Warrior Ethos (mission first, never accept defeat) requires the ability
to make trade-offs in the application of tactics, techniques and procedures. Every battlefield
situation will present options with differing sets of opportunities and penalties if decisions are
poorly made. Frequently there are no right answers to any given situation, but recognition of
consequences of behavior must be gained through experience. Conflict and synergy among task
demands can be manipulated in a manner that can be replicated in a scripted training scenario,
with measurable behavioral effects and opinions.
Ability to Adapt. Warrior Ethos requires adaptability, by smooth reaction, to changes in mission
and unexpected, often unpleasant, surprise (never quit, never accept defeat). A change in mission
can be as disruptive to the expected flow of events as a surprise created by the enemy, weather or
unfamiliar terrain. Each will either further constrain or relax available options for mission
accomplishment. Soldiers must understand the significant constraints but continue to seek ways
to accomplish the mission with a minimum of friction or disruption. In principle, these external
constraints and friction can be manipulated and replicated (e.g., the conditions in which tasks are
executed) to provide behavioral effects and opinions that can be measured.
Ability to Accept Responsibility for Others. Warrior Ethos requires acknowledgement and
acceptance that each Soldier is counted upon by other Soldiers to perform his/her mission and
tasks (never quit, never leave a fallen comrade). Any failure to perform by one Soldier means that
another Soldier must assume that workload. This is not taken lightly if an individual is deemed
capable of performing but fails to do so. On the other hand, a Soldier who continues to perform
the mission despite being incapacitated by accident or enemy action is a team member who is
held in high esteem by others. In principle, opportunities or needs to demonstrate such reliability
can be manipulated in a manner that can be replicated (e.g., established at well-defined points in
the training event) and with consequent behavior and opinions that can be measured.
Ability to Accept Dependence on Others. Warrior Ethos connotes the capacity of each Soldier to
rely on fellow Soldiers to accomplish missions and tasks. A Soldier must recognize that he or she
depends on comrades for assistance and for personal security. Teamwork is important not just to
get the mission accomplished but to do so in such a way that provides some assurance that the
individuals of the team survive contact with the enemy (or severe weather or inhospitable terrain),
in order to fight again. In principle, opportunities or needs to demonstrate reliance on others can
be manipulated in a manner that can be replicated (e.g., established at well-defined points in the
training event) and with choices that can be measured. As an individual develops Warrior Ethos
and primary identity with a team, responsibility to others presumably becomes inseparable from
reliance (dependence) on others. At the outset of training, however, personal attributes may need
to be addressed separately to ensure full understanding.
Motivated By a Higher Calling. Warrior Ethos implies a primary motivation derived from the
values of the Army and belief in the cause for which the Army fights Duty, Honor, Country. It
is important for Soldiers to understand why they are fighting and for them to believe that it is
right. Higher calling cannot be tested experimentally in any obvious way but, in principle, it is
measurable in terms of opinions, attitudes and understanding about the meaning of or reason for
the individual Warrior skills and tasks as well as collective drills and objectives.
355
Riccio et al.
References
Department of the Army (1999). Army leadership (FM 22-100), Washington, DC: Author.
Department of the Army (2003). Battle focused training (FM 7-1). Washington, DC: Author.
Task Force Soldier (2003). Task Force Soldier CSA Directive. Retrieved August 14, 2004, from
U.S. Army Infantry Web site: http://www.infantry.army.mil/taskforcesoldier
Training and Doctrine Command (2003a). Soldiers Creed. Retrieved August 14, 2004, from
TRADOC Web site: http://wwwtradoc.army.mil/pao/Web_specials/WarriorEthos/SoldiersCreed.jpg
Training and Doctrine Command (2003b). Warrior Ethos staff primer. Fort Monroe, VA: Soldier
Task Force.
Training and Doctrine Command (n.d.). TRADOC begins new initiative: Warrior Ethos.
Retrieved August 14, 2004, from TRADOC Web site: http://wwwtradoc.army.mil/pao/Web_specials/WarriorEthos.
U.S. Army Center of Military History (n.d.). Medal of Honor citations. Retrieved August 14,
2004, from CMH Web site: http://www.army.mil/cmh-pg/ Moh1.htm.
Wong, L., Kolditz, T. A., Millen, R. A., & Potter, T. M. (2003, July). Why they fight: Combat
motivation in the Iraq War. Carlisle, PA: Strategic Studies Institute, U. S. Army War College.
Supplementary Work Product from Warrior Ethos Project
We found that the warrior attributes helped achieve a useful mapping among a variety of valuesbased concepts in current Army doctrine. While such relationships are not intended to suggest
one-to-one mappings or a full understanding of each concept, they do alleviate unnecessary
complexity in important sets of Army requirements and thus suggest ways in which the
requirements have actionable implications for Army training and education.
Appendix E: Indicators of Warrior Ethos

From: Bruny, T., Riccio, G., Sidman, J., Darowski, A., & Diedrich, F. (2006).
Enhancing warrior ethos in initial entry training. Proceedings of the 50th Annual
Meeting of the Human Factors and Ergonomics Society, San Francisco, CA.
The U.S. Army Infantry School Task Force Soldier, in 2003, adopted four Tenets of Warrior
Ethos: Mission First, Never Quit, Never Accept Defeat, Never Leave a Fallen Comrade. The
purpose of the research was to refine and operationalize the 2003 definition of Warrior Ethos and
to examine means for its inculcation into the Army. Warrior Ethos was translated into seven
values-based attributes exemplified by behavior that could be identified in combat and noncombat situations. The attributes thus make it easier to examine and develop the underpinnings of
Warrior Ethos early in a Soldiers training. Using a train-the-trainer procedure, trainers can be
provided with ways of looking at Soldier behavior within a framework for Warrior Ethos that can
be applied consistently throughout training.
Methods
The ultimate objective of the formalized observations and video analyses described below was to
identify trainee behaviors that exemplify each of the seven Warrior Attributes. Two visits were
made to Ft. Benning and Ft. Jackson in the summer of 2005, during which soldiers were
videotaped while executing the TDC stations. These tapes were then coded to identify behaviors
indicative of the seven Attributes, which will ultimately provide the foundation for a multimedia
Train the Trainer program. This program will help Drill Sergeants ground their AARs within
the framework of Warrior Ethos.
Participants
A total of 26 basic training soldiers participated. The Benning unit (n = 12) was all male, while
the Jackson unit (n = 14) was gender-integrated (10 males, 4 females).
Instruments and Facilities
In each data collection event, two video cameras and a parabolic microphone were used to record
trainee behaviors: one to gather global, and the other for relatively close-up, footage. In both
cases, substantial distances were maintained between the cameras and soldiers so as to minimize
interference. The TDC consists of 6 stations that resemble obstacle courses. In general, the goal
of each station is to maneuver the entire team, as well as some additional resources (e.g.,
ammunition) across a set of disconnected platforms and walls within a 20 minute time period
using limited resources and a strict rule set.
Procedure
Two individuals experienced with video analysis coded the video footage from both trials.
Initially, we used an iterative process to develop a common coding scheme between the coders,
including a complete list of attribute subcategories and their constituent behavior (Table 1). For
example, the Dependence attribute contained the following subcategories for which behavior
could be identified: full participation, idea solicitation, and using physical assistance.
Prioritization, on the other hand, contained the subcategories developing sound plans, considering
multiple courses of action, and managing resources.
Table 1. Warrior Attributes and associated teaching points

Observed Attitude or Behavior
Improve
Sustain
AAR
Dependence relate to never quit (tenet of Warrior Ethos) & Loyalty (Army value)
all individuals participated in task
solicited ideas or help from others
helped overcome individual limitations
Responsibility relate to never leave a fallen comrade (tenet) & Selfless service (value)
a team-oriented leader emerged
did not alienate teammates
understood consequences of ones own actions
Sense of Calling builds on dependence and responsibility; relate to Integrity (Army value)
took mission and tasks seriously
understood values in the details of task
did not cheat; allowed team to learn on its own
Prioritization relate to mission first (tenet of Warrior Ethos) & Duty (Army value)
developed sound plan
Selected best course of action
identified and managed resources
Trade-offs relate to never accept defeat (tenet of Warrior Ethos) & Honor (Army value)
considered multiple ideas without chaos
weighed recommendations from team members
team task-organized
Adaptability builds on prioritization and tradeoffs; relate to Respect (Army value)
rehearsed or assessed solutions
considered new alternatives from others
applied lessons learned from prior experience
Perseverance builds on everything; relate to Personal courage (Army value)
overcame frustration
identified ways to work smarter, not harder
sustained attitude
358
Appendix E: Indicators of Warrior Ethos
Once the coding scheme was established, we used a multi-pass method such that coding was
directed at one attribute independently per pass, allowing for a narrowed focus and nonorthogonal attributes. For each observed behavior the coders recorded the phase, time, attribute
and subcategory, and whether it was a positive or negative example (e.g., youre not allowed to
jump! and just jump! are positive and negative examples of Sense of Calling respectively,
within the rule set of the TDC that prohibit jumping).
Results
Qualitative Findings
Non-materiel gaps and solutions. The two most noteworthy observations at the TDC are (a)
Warrior Ethos can be identified with training behavior in ways that can be utilized by DS to
reinforce desirable behavior and attitudes, and (b) there is considerable variability among DS and
over time in the use of Warrior Ethos as a teaching construct. Thus, there is an opportunity to
provide DS with a framework for more consistent use. The clearest use of Warrior Ethos by DS
was in the AARs that were scheduled after each station in the TDC. DS could benefit from a
framework that would make it easier for them to make connections between Warrior Ethos and
observed behavior in TDC.
Materiel gaps and solutions. Spiral development of a training support package for the TDC over
more than a year, and with frequent feedback from DS, has led to a tool that should facilitate
teaching of Warrior Ethos at TDC. Checklists for the Attributes of Warrior Ethos also underwent
spiral development based on iterative use by investigators in the field and in post hoc analysis of
audio-video recordings.
Quantitative Findings
Reliability Assessment. To assess inter-rater reliability of coded video data, we conducted a
simple comparative behavioral frequency analysis by totaling each coders number of observed
behavior for each of the seven attributes and six TDC stations. A Pearsons r (r = .89, p < .05)
was calculated to assess correspondence between the two coders frequency counts.
Descriptive Analysis. A total of 467 behavioral events were identified and categorized.
Descriptive analysis was conducted in an effort to find patterns that indicate which behaviors
were observable, and on which stations within the TDC course. The analysis collapsed the data
across attribute subcategories from the Benning and Jackson trials, and then occurrence across
each of the six TDC stations (Table 2).
Table 2. Observed instances of behavior associated with the warrior attributes.
Interpretation of Behavioral Patterns. We identified several important findings in these data.

First, there were inconsistencies in the total frequency counts across attributes and stations. For
example, there were few instances of Adaptability relative to Responsibility, suggesting that
novice trainees may be less skilled in adapting to the station demands after establishing a course
of action. However, trainees also demonstrated a relatively high degree of Responsibility toward
both the tasks and fellow trainees. Second, we recognized several important behavioral patterns
by station. For example, the highest frequency of Dependence came from the Destroyed Bridge
station, which elicited many behaviors indicative of individuals using their teammates
unrequested physical assistance. This result is not surprising given the nature of this station,
which required the repetitive passing of boards while maintaining balance in close contact with
teammates. Further, the highest frequency of Prioritization came from the Quicksand station,
which elicited a large number of behaviors indicative of developing a sound plan and effective
resource management. This particular station required complex problem solving during mission
planning and managing a limited number of resources. These interpretations were made for each
station, allowing us to assess not only which attributes have the highest frequencies, but why this
was likely to occur, providing a solid foundation for training support package development.
Discussion
Having identified the relationships between attributes and stations, the videotapes were reviewed
to find salient behaviors exemplifying each attribute. Several of these video clips are being
included in a Train the Trainer program that is designed to assist DS in conducting AAR.
Within this program, DS will learn how to link behavioral examples to the higher level concepts
of Warrior Ethos.
Ideally, having familiarized themselves with how attributes manifest themselves as trainee
behaviors, the DS will be able to embellish typical questions and comments (e.g., What did we
do well?) with specific comments directly related to Warrior Ethos (e.g., You showed excellent
responsibility for others when...). Trainees are also likely to benefit from receiving feedback
that is grounded in specific examples. As such, they can begin to learn that Warrior Ethos is not
just a philosophical concept, but rather it is reflected in their own actions.

OBTE Monograph 01NOV2010

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

OBTE Monograph 01NOV2010

Uploaded by

Copyright:

Available Formats

UNCLASSIFIED

Implications for an Integrated

Copy edited by G. Riccio

U.S. Army Asymmetric Warfare Group

2282 Morrison St., Ste. 5355 Ft. Meade, MD 20755-5355

Asymmetric Warfare Group

Asymmetric Warfare Group

An Initiative in Outcomes-Based Training and Education

Outcomes Based Training & Education

Asymmetric Warfare Group

Riccio, Diedrich, & Cortes (Eds.)

Outcomes Based Training & Education

Asymmetric Warfare Group

Riccio, Diedrich, & Cortes (Eds.)

Outcomes Based Training & Education

Riccio, Diedrich, & Cortes (Eds.)

Asymmetric Warfare Group

Forward by AWG Commander

Asymmetric Warfare Group

Riccio, Diedrich, & Cortes (Eds.)

Asymmetric Warfare Group

Chapter 3. Principles and Practices of Outcomes Based Training & Education............................50

Riccio, Diedrich, & Cortes (Eds.)

Chapter 5. Passion and Reason in Values-Based Learning & Development ...............................118

Asymmetric Warfare Group

Chapter 8. Observations of Behavior and Communication in Rifle Marksmanship Training .....160

Asymmetric Warfare Group

Riccio, Diedrich, & Cortes (Eds.)

Section III. Further Development of OBTE as a Service System ..........................................206

Asymmetric Warfare Group

Asymmetric Warfare Group

Riccio, Diedrich, & Cortes (Eds.)

Appendix D: Warrior Ethos..........................................................................................................345

Asymmetric Warfare Group

Asymmetric Warfare Group

progenitors of OBTE directly influenced the development of measures utilized in the

Asymmetric Warfare Group

Figure 1. Evolution of the investigation as reflected in the chapters of this monograph.

Asymmetric Warfare Group

Section I. Development of Stakeholder Requirements for OBTE

Asymmetric Warfare Group

Preparation for Full Spectrum Operations

Asymmetric Warfare Group

Preparation for Full Spectrum Operations

Asymmetric Warfare Group

Preparation for Full Spectrum Operations

1.3 An Appraisal of Instruction with Respect to OBTE

Preparation for Full Spectrum Operations

Type of evidence and source

Was CATC perceived as valuable overall?

Self report data from Soldiers who participated

Was CATC perceived as valuable in

Source: Web-based survey conducted after

What suggestions did Soldiers have for

To what extent did DS exhibit the OBTE

Source: Observational rating form during

To what extent did Privates exhibit high

Source: Observational rating form during

Kirkpatrick Level 2 (Privates)

Were communication patterns consistent

Communication between DS and Privates

Kirkpatrick Level 2 (DS)

Source: Survey following return from

Asymmetric Warfare Group