COGNITIVE READINESS ASSESSMENT

AND REPORTING: AN OPEN SOURCE

MOBILE FRAMEWORK FOR OPERATIONAL
DECISION SUPPORT AND PERFORMANCE
IMPROVEMENT
Matthew Heric, PhD Jenn Carter

Cognitive readiness (CR) and performance for operational time-critical environments are
continuing points of focus for military and academic communities. In response to this need,
we designed an open source interactive CR assessment application as a highly adaptive and
efficient open source testing administration and analysis tool. It is capable of evaluating the CR
of individual personnel and combining these results into intuitive, visually based summaries of
readiness. The components, logic, and architecture are presented.

THE BROAD RESEARCH DOMAINS of cognitive
readiness (CR) and human performance for operational
time-critical environments such as emergency response
and military operations have long been critical focus
areas within military and academic research communi-
ties. While the research community has firmly estab-
lished assessment methods for measuring an individual’s
cognitive state and capability level, a current challenge
within ongoing dynamic field operations is having the
ability to employ these metrics proactively and obtain
accurate assessments of individual and group CR and
then identify supplemental training or preparation
needs. Given the advanced state of computer systems
today, an integrated mobile software application that
can employ subject matter expert (SME) measures to
evaluate CR effectively and accurately for mission readi-
ness is now feasible. The solution and methods must
offer rapid testing, together with reliable prediction and
evaluation measures, allowing supervisors to improve
their abilities to assess, monitor, and mitigate issues
leading to poor performance. Another primary intent is
to have this system fully functional for field deployment
environments, thus removing the dependency on class-
rooms and workstations and simultaneously taking
advantage of personnel periodic downtimes typically
associated with deployments.
Any CR software for field use must be easily customiz-
able, mobile-platform ready, and extensible. Similarly,
the reporting system must be able to aggregate indi-
vidual assessments to address the needs of small mission-
focused specialized subgroups and whole-group levels.
For this, we designed an open-source interactive cogni-
tive readiness assessment application (iCogRA2) adaptive
and efficient testing administration and analysis tool for
evaluating CR. For this, computer-based testing (CBT)/
computerized-adaptive testing (CAT), based predomi-
nantly on item response theory (IRT) methods, provide
highly informative platform structures minimizing work-
load.
iCogRA2 was functionally defined by technologists
in the fields of test design, measurement, and CR. The
system logic is purposefully linear and has an ability to
leverage inputs from willing contributors through an
open source design.

Performance Improvement, vol. 50, no. 7, August 2011

©2011 International Society for Performance Improvement
Published online in Wiley Online Library (wileyonlinelibrary.com) • DOI: 10.1002/pfi.20227 5

PFI20227.indd 5 8/17/11 11:05:55 AM

 BACKGROUND
The capacity to prepare personnel for uncertain and often
stressful environments is of great concern to emergency
response supervisors and military commanders charged
with maintaining group performance within physically
and mentally demanding environments (Salas & Cannon-
Bowers, 2000). Although there have been marked advance-
ments in technologies that aid personnel in completing
missions, these advancements also introduce more infor-
mation for them to process, learn, and implement (Salas,
Priest, Wilson, & Burke, 2006). With increasing opera-
tional requirements such as the recent need for soldiers to
understand and take into account sociocultural factors in
support of peace-keeping missions, the varied nature and
wide scope of knowledge and skill requirements can be
mentally taxing and require advanced tools to test for CR
in deployed personnel (Thompson & McCreary, 2006).
Human cognition is vital to overall mission effective-
ness in complex network-centric operations (Wesensten,
Belenky, & Balkin, 2005). Within these operational envi-
ronments, it is critical to survey the CR of both the team
and individuals and address performance measures as
they relate to achieving mission goals and maintaining
mental stamina (Morrison & Fletcher, 2002).
Cognitive Readiness and Assessment
for Time-Critical Environments
Organizations such as the U.S. military dedicate consid-
erable resources for personnel training and follow-on
performance assessment. The U.S. Army, for example, has
the highly regarded Army Readiness Assessment Program
for scenario-based safety training. Broadly, testing and
assessment are done with a focus on mission fulfillment,
yet demands on time-critical task performance continue
to strain the cognitive capacities of operational person-
nel to the point where their effectiveness is significantly
diminished and there are thresholds for performance
that must be closely monitored and understood to main-
tain a balance between operational effectiveness and
individual CR. For operations, however, the situation is
dynamic, and just as testing and evaluation are conducted
to ensure the readiness of new technologies, personnel
and leaders must be able to assess the readiness of the
units for which these technologies were created. In this
context, CR is the mental preparation—the skills, knowl-
edge, abilities, and motivations—an individual needs
to establish and sustain competent performance in the
complex and unpredictable environment of time-critical
operations (Morrison & Fletcher, 2002). Decreased CR
directly affects team cohesion and effectiveness and limits
human abilities to perform within diverse and complex
6 www.ispi.org • DOI: 10.1002/pfi • AUGUST 2011
PFI20227.indd 6
environments. Based on the needs of the military, public
safety, and other similar user groups, new developments
are needed to leverage laboratory CR to more operational
environments. In addition, extended approaches for col-
lecting and reporting CR are needed that can be applied
and customized across codependent operational groups
(Holmes-Rovner & Wills, 2002).
Although advancements in technology and systems are
important to improving and maintaining time-critical
capabilities, human performance remains the building
block for successful operations (Schoomaker, 2007).
Maintaining cognitive capabilities is a constant concern
of field officers and supervisors in charge of networked
forces; however, this performance factor is also difficult to
identify, measure, and address (Salomon & Perkins, 1989).
Historically, measurements of cognition, and specifically
metrics of CR, have been difficult to employ (Markham &
Medin, 2002). Evolving measurement techniques are con-
tinually being refined and improved, and a rather large
pool of literature and research deals with the metrics of
CR (Stevens’ Handbook of Experimental Psychology, 2004).
Leveraging these methods and metrics for development
of a mobile application for easy assessment, visualization,
and distributed learning is the foundation of iCogRA2.
Furthermore, for time-critical applications, CR com-
prises multiple inputs, including situational awareness,
transfer of training, decision making, and decision
support, among others (Fletcher, 2004). Processes for
measuring these are somewhat common, but efficiency
logistics in use are not in comparison. This is particularly
the case when collection and assessment must be done in
the field, away from classrooms and labs. Field collection
systems, for instance, need an ability to send data to a cen-
tral processor for review and decision support function-
ality, adequate feedback systems to track CR status over
time, and the ability to track both individual and group
CR dynamically. Such user networks would allow rapid
understanding of individuals’ statuses in relationship to
others in a subgroup or whole group (Beale, 2005). A dis-
tributed application would also need to update training
records, manage schedules, assess problem solving and
creativity, and determine mission assignments. Thus, as
systems’ functional needs broaden, the resulting capabil-
ity creep highlights the logistical complexities.
Measuring Cognitive Readiness
At its foundation, CR is a blend of efficiency and adapt-
ability that, through measurement and analysis, models
and predicts performance and effectiveness. CR mea-
sures combine competence for established responses with
adaptation to varying conditions and, in its most extreme,
unanticipated status. The combination is important
8/17/11 11:05:55 AM

CR is a blend of efficiency
and adaptability that,
through measurement
and analysis, models and
predicts performance and
effectiveness.
because no single attribute is sufficient in isolation. For
instance, a particularly noteworthy performer may be
inflexible and not have the abilities to manage in volatile
situations, while a compliant individual may acclimatize
quickly to operational variants but lack abilities to link
basic operating procedures to proper reactions. To pro-
vide an accurate measure of CR, the ability to perform
a task and to cope with changing environments must
be measured jointly. The relative importance between
competency and adaptability will be highly dependent on
the mission objective for the individual, the individual’s
position within the unit, and overall mission parameters.
A soldier tasked as a mechanic, for example, will place a
great deal of emphasis on competency, whereas a soldier
on a front line would likely value mental acuity predomi-
nantly (Fletcher, 2004).
For an initial iCogRA2 baseline, the need is to measure
psychological traits relative to established components
of CR. Using the content in Table 1, the initial iCogRA2
measures focus on aspects such as these:
• Memory tested through knowledge and skill retention
• Metacognition evaluated by having an individual pre-
dict his or her own performance
• Problem solving by matching a properly arranging set
of responses to maximize the chance of success

TABLE 1 THE 10 COMPONENTS OF COGNITIVE READINESS AS DEFINED IN FLETCHER (2004)

TRAIT DESCRIPTION

Situation awareness Ability to perceive and comprehend one’s place in the environment and how to react to
changing conditions

Memory The active, reconstructive ability to recall or recognize situation patterns

Transfer The ability to apply what is learned in one context to devise solutions in another performance context

Metacognition The executive functions of thought used to monitor, assess, and regulate one’s own cognitive processes

Automaticity The ability to regulate processes that require limited conscious attention

Problem solving The ability to analyze the current situation, devise strategies to understand the situation, and develop
resolutions through a series of executable steps

Decision making The ability to review various courses of action and then allocate sufficient resources to the problem

Mental flexibility and creativity The ability to generate, adapt, and modify courses of action in response to variable situations

Leadership A combination of technical, conceptual, ethical, and interpersonal competencies that encourage and
support others in carrying out a designated course of action

Emotion The ability to perform complex tasks under confusing, high-stress situations

Source. Fletcher, 2004.

Performance Improvement • Volume 50 • Number 7 • DOI: 10.1002/pfi 7

PFI20227.indd 7 8/17/11 11:05:55 AM

 Through an open source design, these measures could
easily expand to meet the requirements of the leader-
ship charged with mission success or specific person-
nel requirements at hand. Thus, the assessment of CR
becomes wholly requirements driven in a context of
operations and mission objectives.
Cognitive Readiness Assessment Techniques
and Shortcomings
While CR modeling has a rather established use in social
and behavioral research, limitations posed by assessment
availability and tandem reciprocal remediation can easily
limit the practical utility of computational CR models
(Logue, Sutton, Jarjoura, & Smucker, 2000). Recently
metrics have been offered that address the cognitive capa-
bilities underlying a soldier’s ability to understand his or
her situation within the battlefield and how best to act
and react to produce the best possible course of action
within network-centric environments (Wesensten et al.,
2005). These methods include cognitive performance
quantification (by direct or indirect assessment), which
then allows for predicting an individual’s operational
capacities. From our experience, limitations of CR meth-
ods include supervisor inaccessibility to the output and a
nagging lack of practical application for field uses (e.g.,
for deployed troops). In developing a mobile-based rapid
CR assessment application at the individual and group
levels, any new solution must address the shortcomings
of leadership accessibility.
SOLUTION DISCUSSION
The iCogRA2 solution is centered on handheld, touch-
screen devices such as smart phones and a centralized
FIGURE 1. ICOGRA 2 WORK FLOW
8 www.ispi.org • DOI: 10.1002/pfi • AUGUST 2011
PFI20227.indd 8
server. The combination is capable of measuring various
aspects of individual users’ CR, augmenting their CR
state, and collating groups of individuals’ CR states in a
high-utility report for relevant leadership.
The iCogRA2 foundation is defined in four segments:
Part 1: Design of Testing Questions and Methods
• CR assessment question-and-answer structures
• CAT definition
Part 2: Software Architecture and Mobile Platform
• Database and structure
• Test logic
• Application run-time environment and software
design
Part 3: Reporting and Analysis
• General subject-based assessment reports
• Single topic assessment
Part 4: Actionable Intelligence
• Results storage and exportation
• Training recommendations
This work flow supports the development, execu-
tion, and integration of the base system on which users
can augment the capabilities for additional capabilities
through the open source design. Again, the intent of
iCogRA2 is to leverage existing methods and push these to
practitioners. The resultant reciprocal feedback-reporting
work flow is captured in Figure 1.
A primary initial interest will be to evaluate iCogRA2
functionality in real-time, time-critical operational
8/17/11 11:05:55 AM
 environments and examine how these tools produce
accurate and timely results for leadership while not sig-
nificantly adding to a full schedule of daily tasks (Warm,
Matthews, & Finomore, 2008). With functional testing,
we expect to make the iCogRA2 application freely avail-
able for evaluation by Internet download (subject to any
international distribution restrictions as they may apply)
and provide an application programming interface (API)
for those interested in extending the code.
Part 1: Design of Testing Questions
and Methods
Using our collection of literature and research on CR, we
approached iCogRA2 from a viewpoint that SMEs have
already developed large amounts of data on how best
to measure and assess CR (Allen & Yen, 2001; Hancock
& Szalma, 2008; Potter, 1999; U.S. Army, 2007). We are
not proposing new measurement theories in this regard.
Instead, we leverage existing methods to assess dynami-
cally CR at the individual and group levels through the
use of rapid CAT rhetorical structures and IRT (Foster,
2009) accessed using mobile platforms. This approach
addresses both the CR of an individual at the time a test
is taken, as well as assessment of his or her learning or
understanding potential. The goal is to allow leadership
to determine state as well as what measures need to be
taken to maintain or improve CR state.
While the objective is usually to evaluate current per-
formance level, iCogRA2 CR use is for operations, and
measures need to assess ability rather than merely static
knowledge. The desired testing for time-critical scenarios
therefore must provide a dynamic assessment of cogni-
tion and potential: measuring perception, memory, and
problem solving. Using CAT (e.g., if a subject scores at a
consistently low level, the test will automatically begin to
assess his or her cognitive potential rather than current
state), the intent is to pinpoint differences between an
individual’s lack of knowledge (which may be a defi-
ciency in training procedures) versus lack of the ability to
learn. The dynamic assessment takes into account several
features that traditional static testing methods do not:
• Cognitive processes are adaptable, and assessment is
meant to ascertain the CR expansion instead of testing
merely for current functioning.
• Dynamic assessment showcases potential cognitive
capacity.
• The overall objective is aligned well with leadership
goals in that each assessment is intended to reveal
cognitive faculty, suggesting training intervention
aimed at enhancing and realizing the individual’s full
potential.
PFI20227.indd 9
CAT helps meet the need
for rapid and accurate
assessment for time-critical
domains, and it is a staple of
instructional assessment.
Again, these are rather well-understood metrics, but
iCogRA2 brings them to a new decision support domain.
Cognitive Readiness Assessment Question-and-Answer
Structures. Research has presented a variety of games and
examinations that address areas of CR not tied to domain
knowledge: adaptability, communication, decision mak-
ing, problem solving, metacognition, situation awareness,
teamwork, adaptive expertise, creativity, and critical think-
ing (Fletcher, 2004). Similarly, the U.S. military has created
examination materials for measuring personnel expertise
relative to optimal performance within battlefield opera-
tions (U.S. Army, 2007). It is critical that a field system have
a sufficiently broad logic format to allow dynamic storage
and presentation using an extensible tool kit with a suffi-
cient XML-based question schema. This will allow for the
inclusion of new metrics and theories as they arise.
CAT Definition. The use of CBT for field operations
offers several familiar advantages over traditional paper-
and-pencil tests: individualized and efficient administra-
tion, instantaneous scoring, and the ability for leadership
to easily visualize results and assign new training courses
as needed. iCogRA2 uses the specialized form of CBT
within the functional logic of CAT. CAT is unique in this
regard as it estimates a user’s knowledge or understand-
ing and then adjusts the questions being asked to glean
the maximum information about that user in the least
amount of time. The questions presented during a CAT
are based on the answer history and perceived under-
standing of each user; therefore, performance can be
quickly assessed and scored instantaneously, with reports
generated automatically for leadership.
CAT helps meet the need for rapid and accurate
assessment for time-critical domains, and it is a staple
of instructional assessment (e.g., the Armed Services
Vocational Aptitude Battery). When the question library
from which CBT pulls from is large, the CAT algorithm
randomly selects from a subset of equivalently rated ques-
tions, providing unique tests for each individual. By using
Performance Improvement • Volume 50 • Number 7 • DOI: 10.1002/pfi 9
8/17/11 11:05:55 AM
 a random sequence of questions rather than a stationary
test, it is possible to measure an attribute more accurately
with much shorter tests (Mills & Steffen, 2000). CAT is
generally recognized as one of the best ways to access lon-
gitudinal changes in knowledge and understanding and
detect misfits in testing (van Krimpen-Stoop & Meijer,
2000). Thus, employing the CAT framework will allow
leadership to:
• Evaluate how well assessments pinpoint CR factors
• Refine training and mitigation based on feedback
• Allow comparisons in personnel performance through
time
CAT heavily leverages rhetorical structures and IRT
in the design, analysis, and scoring of tests, and the tech-
niques are well documented (Embretson & Reise, 2000).
Although IRT has commonly been used to measure
knowledge or skills, it is certainly capable of measuring
psychometric qualities with properly designed questions.
IRT assumes that each question pertaining to a quality
has differing response probabilities based on the person’s
innate psychometric disposition—which is described by
the item response function. This method can also be used
to measure scaled responses.
A person who is above a certain CR level would find the
answer obvious and answer all questions correctly; those
with poor readiness traits may answer incorrectly most of
the time. By asking a series of questions, the test is able to
determine with high probability the CR of an examinee. The
next question selected will divide the existing belief region
into smaller ones. This process continues until an examin-
ee’s latent trait is believed to be in a sufficiently small region.
To summarize, the iCogRA2 CAT method is an iterative
algorithm with the following steps (also see Figure 2):
FIGURE 2. CAT STRUCTURE FOR CR ASSESSMENT
10 www.ispi.org • DOI: 10.1002/pfi • AUGUST 2011
PFI20227.indd 10
1. Assume the examinee has a uniformly distributed
belief function for a defined trait.
2. Select a question that divides the belief distribution
into discrete subsections.
3. Present a chosen item to the examinee, who then
answers it.
4. Update the psychometric distribution using Bayesian
mechanisms with all prior answers.
Steps 2 through 4 are repeated until the distribution is
sufficiently narrow; thus, the psychometric quality can be
measured accurately.
Part 2: Software Architecture and Mobile
Platform
We now turn to the iCogRA2 functional description: the
software performs as a tandem server-side/client-side
application. The server side handles data storage, test con-
tent generation, results analysis, and user validation and
assignment operations. Interactions with the data occur
from the client, implemented as a multiplatform hand-
held application, usable on current consumer devices
such as a modern multifunction phone. The overall soft-
ware architecture is presented in Figure 3.
Database and Structure. The server-side components
use standard databases attributes for storage. The struc-
ture is a cross-platform SQL database solution composed
of three matrices: a test library, a training library, and
a user library. The test library table stores all of the
information relevant to test administration and analysis.
Columns include test item identification numbers, test
item content, answer types (multiple choice, true or false,
or scaled), question rating, answer rating, and question
8/17/11 11:05:55 AM
 FIGURE 3. ICOGRA 2 ARCHITECTURE
categorization. This library is queried when generating
test content for users as well as when performing scoring
analysis.
To perform training recommendations based on user
test scores, the system needs information on available
training courses. To that end, the training library table
contains data organized into structure columns, includ-
ing training course name, course identification number,
identification numbers of prerequisite courses, and a list
of topics that the course in question addresses. These
categorizations serve as criteria for matching individual
questions (and thereby performance on tests including
those questions) to relevant training.
Users of the iCogRA2 software have various aspects
of their testing experience and training history tracked.
To this end, the system stores data relevant to indi-
vidual users, and the user library table sorts by indi-
vidual identification numbers (which need not be linked
to any confidential identification information within
the database) and contains the user’s testing history
(including individual test question responses), as well as
information to support supervisors’ testing assignments.
Individual identification numbers can then be assigned
(by an administrator) to a subsidiary role under a speci-
fied command user. These relationships are reflected in
stored data in the user library table. Test assignment may
be performed on the server side or from the handheld.
Administrators can simply select a test from a list of
available examinations and then select users to whom it
should be assigned.
Test Logic. The CAT-IRT logic is a server-side solu-
tion. The system generates tests for individual users
PFI20227.indd 11
appropriate to the subject of interest and their individ-
ual performance during the course of the examination.
Test questions are drawn from the iCogRA2 test library
database as needed, and scores update according to
each answer submitted by inputting the relevant answer
information to the user library table. Because IRT uses
a per item approach to performance analysis, the sys-
tem need not track users over their entire test session
(beyond ensuring that they answer enough questions
from the test library to fulfill the criteria of the exami-
nation) for purposes of content generation. Only their
response to the question at hand is necessary to deter-
mine which question should be selected and dispensed
to the user.
A key component of iCogRA2 is an SME input inter-
face for questions, answers, and precoded item response
function options defined by the data stored in the test
library. For example, as knowledge and research expand,
SMEs can establish the test question content and store
the input in defined XML format, describing the neces-
sary components of each question (e.g., question ID,
text, answer type, question rating). This information is
transformed at run time into a rich-text visualization
for the user. SMEs can also use multimedia content by
embedding links to images, video, and audio files in the
question and answer text using simple inline syntax and
HTML content tags.
Application Run-Time Environment and Software
Design. The iCogRA2 computations are performed
on the server, thus reducing the demands placed on
the handhelds. Because handheld software is somewhat
defined by the hardware, the systems essentially act
Performance Improvement • Volume 50 • Number 7 • DOI: 10.1002/pfi 11
8/17/11 11:05:55 AM
 as rich-text rendering clients for server-side applica-
tions that perform the analysis and data preparation.
Multiplatform compatibility is a rather logical matter.
Rich-text rendering can be performed using an agnostic
Web-media-focused interface, making porting between
multiple handheld platforms a simple matter of using
appropriate input APIs.
The handheld application is responsible for four tasks:
(1) allowing users to enter log-in information, (2) dis-
playing a history of that user’s tests (including tests they
have not yet completed but have been assigned), (3) ren-
dering the rich-text multimedia content of a question and
associative answer, and (4) transmitting the user’s answer
selection to the server. The testing interface is clear and
intuitive: (1) the user is presented with a prompt for log-
in; (2) a list of selectable tests is presented, color-coded
to indicate completion status; and (3) the multimedia
question content is offered with selector buttons for each
of the possible answers. By default, users who complete a
test can view their score results at the time of completion
and on the test history display at initial log-in. Figure 4
demonstrates the test user interface.
The tandem to the test interface is the administrator
interface. The test administration function first presents
a filterable list of tests (see Figure 5). Supervisors may
select tests individually or in a group and may filter tests
by content categories. The supervisor then selects users
from the list of presented IDs, and test assignments can be
designated to subordinates according to perceived testing
needs. Supervisors are given the option to hide user test
performance.
The server validates user access to tests, selects rel-
evant test questions based on the CAT algorithm, and
FIGURE 4. ICOGRA 2 USER INTERFACE
12 www.ispi.org • DOI: 10.1002/pfi • AUGUST 2011
PFI20227.indd 12
updates user records with scoring statistics. The server-
side software interfaces with standard desktops or secure
sockets layers (SSL) from handheld users. Because user
interactions are asynchronous, only one server process
is required, and the software need respond only when
an individual user sends or requests data. User log-in
validation involves standard secure connections with SSL
by using individual handheld device encryption keys.
Hashed encryption key values track devices across several
interactive sessions ensuring that users need not log in
more than once provided that their server interactions
are made within a reasonable window of time.
Once a remote test user establishes a connection to
the server, the application selects questions from the
test library data table using CAT-IRT methodologies.
Question data are transmitted over the SSL connection
to the client device, rendered by the local application. The
same is true of assignment or performance data being
viewed by an administrator. The server-side application
retrieves relevant information from the database, per-
forms relevant mathematical analysis, and transfers the
resultant data to the handheld device.
Part 3: Reporting and Analysis
Once users complete a test with their handheld, the results are
logged and made available in visual and numerical reports.
To this end, iCogRA2 makes easily understood administrator
summaries and breakdowns through a simplified interface,
and in our opinion, summary graphs provide tremendous
benefit for quickly understanding the results.
General Subject-Based Assessment Reports. Individual
test results can be combined to display results for the
8/17/11 11:05:56 AM
 FIGURE 5. ICOGRA 2 ADMINISTRATOR INTERFACE

entire group. For easy understanding, the scores are sum-
marized in interactive rose diagrams depicting relative
strengths of various metrics for the group or individu-
als. A rose diagram simultaneously presents continuous
or discrete values of multiple psychometric factors with
ease. The values for a group along a given axis are calcu-
lated by any function from the observed results of each
individual to an overall score that includes the mean,
median, or even multiple percentiles of the group. If
the user desires to view a higher-resolution account of a
given quality, he or she may click on the trait to expand
it into a rose chart detailing the subfactors that influence
the combined psychometric. For instance, a quality such
as knowledge could be broken down into a set of mea-
surements for each individual skill necessary for mission
success.
Figure 6, an example chart, graphs the average and indi-
vidual scores of several users side-by-side for comparison.

FIGURE 6. ROSE DIAGRAM VISUALIZATION OF INDIVIDUAL AND USER PERFORMANCE

Performance Improvement • Volume 50 • Number 7 • DOI: 10.1002/pfi 13

PFI20227.indd 13 8/17/11 11:05:56 AM


FIGURE 7. USER PERFORMANCE LEVELS
Each axis of the rose chart represents an individually
scaled score for one subject area with the radial line indi-
cating either the average of all users’ scores or a single
user’s score. The intersection point of the radial line with
the subject axis gives the performance of the group (or
that user) for that metric.
Single Topic Assessment. When considering a single
topic for assessment, the supervisor is presented with a
scatter plot (see Figure 7). The vertical axis represents score
and the horizontal axis individual users. In both the single-
subject and multisubject graphing interfaces, the overall
graph is color-coded for rapid analysis. According to preset
thresholds, a given score for an individual represents poor,
adequate, or good performance within that subject area.
Here, those categorizations are represented by scaled gray
shades (or by red, yellow, and green color coding on a
phone). Selection of an individual point on the scatter plot
or an individual line on the rose chart takes the adminis-
trator directly to test answers, training recommendations,
and test assignment options for the selected user.
Part 4: Actionable Intelligence
Results Storage and Exportation. iCogRA2 generates
time-stamped raw data for other programs to access
the storage tables in an internal relational database that
records individual soldier responses, date of response,
and time taken to respond. The database also stores
summary and analysis results of the raw data, including
unit averages and median and reports. Each individual’s
personal record is updated indicating the date, time,
and identification key for his or her results to support
harvesting by SQL queries. Mechanisms can be packaged
and exported from within iCogRA2 into .xls and .csv data
tables for easy access by standard graphical and analytical
14 www.ispi.org • DOI: 10.1002/pfi • AUGUST 2011
PFI20227.indd 14
software and for easy integration with standard C4ISR
(command, control, communications, computers, intel-
ligence, surveillance and reconnaissance) systems such as
Distributed Common Ground Systems, Command Post
of the Future, and Tactical Ground Reporting System.
Analysts may export the summarized or raw data for
an individual or entire unit with time trajectories of the
data (when assessments were conducted at different time
points). Since they are of fundamental importance for
the testing work flow, the generation of failure reports is
streamlined so proper corrective actions may be taken.
Again these reports detail failure at both the individual
and the unit levels.
Training Recommendations. The exported results can
be fed into a program that automatically recommends
and even schedules actions to remedy poor performance.
Each evaluation criterion is categorically labeled depend-
ing on the type and nature of the quality being measured.
These labels may be binary pass or fail, or they may have
greater granularity such as an A through F scale. The rec-
ommendation system requires a database consisting of
all of the courses and supplemental tutorials available to
augment categorical skills or knowledge.
Course recommendations are based on optimal
individual and group readiness criteria and can be
set accordingly, yet it remains the supervisor’s task to
determine if the result indicate adequate CR. Indeed,
logic similar to that of CAT could be used to assign
individuals to an appropriate task in accordance with
the recommendations, and this is essentially because
tasks intuitively have differing degrees of difficulty
(physical or intellectual). If an individual is found to
be unable to learn acceptable task details, the program
suggests reassignment to another task of lesser difficulty
where he or she may be better able to support opera-
tional objectives. Conversely, when group deficiency
for difficult tasks is observed, units with mastery of
marginally easier tasks may be selected to train in the
more complicated task. In each situation, shifting is
performed to maximize the individual’s value to the
group while simultaneously maximizing his or her CR
performance potential.
The final recommendation tool is based on temporally
tracked operational readiness metrics. Even if a group
shows competency on a given metric, there may be a
downward trend in the metric when viewed as a function
of time. Various modeling tools can be used to predict
the trajectories, and if the downward trend predicts that
a group will soon lose competency or readiness, the soft-
ware will indicate a preventive measure that is necessary
to maintain satisfactory levels.
8/17/11 11:05:57 AM
 CONCLUSION
iCogRA2 provides the foundation for an advanced CR
assessment testing and decision support solution. This
includes algorithmic approaches to CAT in the domain of
assessment, using IRT-based question selection and scor-
ing. This approach maximizes the accuracy of measure-
ment with a minimum number of required questions to
assess rapidly an individual or group and then automates
the reporting process for leaders. iCogRA2 also establishes
a baseline CR assessment question pool and an extensible
architecture to support future expansion of metrics by
SMEs, tailored to suit individual needs or new knowledge
or skill domains.
iCogRA2 is open source, and the core functionality is
envisioned to grow in subsequent development cycles by
expanding and adding services. For example, the software
could incorporate testing games that both test and develop
the psychological traits of CR in the guise of entertain-
ment. In addition, the recommendation program could
be expanded to perform many of the logistical functions
for automation and optimization of training schedules
to assist leaders in ensuring their group’s optimal perfor-
mance within the operational environment.
The use of computerized technology has played a
major role in training and evaluation, and the increasing
march of innovation has helped greatly in this regard.
Practitioners have much to gain by leveraging these
advancements through the use of applications such as
iCogRA2, and as more open source solutions emerge,
users can extend their training and performance assess-
ment and evaluation expertise into new domains, with
increased efficiency and at reduced expense. For those
interested in developing or extending their own solutions,
the information in this article provides a logical template
for such exploration.
References
Allen, M.J., & Yen, W.M. (2001). Introduction to measurement
theory. Longrove, IL: Waveland Press.
Beale, R. (2005). Supporting social interaction with smart
phones. IEEE Pervasive Computing, 4(2), 35–41.
Embretson, S.E., & Reise, S.P. (2000). Item response theory.
Florence, KY: Psychology Press.
Fletcher, J.D. (2004). Cognitive readiness: Preparing for the
unexpected. Alexandria, VA: Institute for Defense Analyses.
Retrieved from http://www.dtic.mil/cgi-bin/GetTRDoc?AD=
ADA458683&Location=U2&doc=GetTRDoc.pdf.
PFI20227.indd 15
Foster, M.R. (2009, September). Improving the output from
software that generates multiple choice question (MCQ) test
items automatically using controlled rhetorical structure the-
ory. In Proceedings of the 2009 International Conference RANLP
(pp. 29–34). Borovets, Bulgaria. Retrieved from http://www
.aclweb.org/anthology/R/R09/R09–2.pdf.
Hancock, P.A., & Szalma, J.L. (2008). Stress and performance.
In P. A. Hancock & J. L. Szalma (Eds.), Performance under
stress (pp. 1–18). Burlington, VT: Ashgate Publishing.
Holmes-Rovner, M., & Wills, C.E. (2002). Improving
informed consent. Medical Care, 4(9), V30-V38.
Logue, E., Sutton, K., Jarjoura, D., & Smucker, W. (2000).
Obesity management in primary care: Assessment of readiness
to change among 284 family practice patients. Journal of the
American Board Family Medicine, 13(3). Retrieved from http://
www.medscape.com/viewarticle/421558_1.
Markham, A.B., & Medin, D.L. (2002). Decision making. In
H. Pashler & D. Medin (Eds.), Steven’s handbook of experi-
mental psychology (2nd ed,. pp. 413–466). New York, NY: John
Wiley & Sons.
Mills, C.N., & Steffen, M. (2000). The GRE computer adap-
tive test: Operational issues. In W.J. van der Linden & C.A.
Glas (Eds.), Computerized adaptive testing: Theory and practice
(pp. 75–100). Norwell, MA: Kluwer.
Morrison, J.E., & Fletcher J.D. (2002). Cognitive readiness.
Alexandria, VA: Institute of Defense Analysis. Retrieved from
http://handle.dtic.mil/100.2/ADA417618.
Potter, R.W. (1999). The art of measurement: Theory and
practice. Upper Saddle River, NJ: Prentice Hall.
Salas, E., & Cannon-Bowers, J.A. (2000). The anatomy of
team training. In S. Tobias & J.D. Fletcher (Eds.), Training and
retraining: A handbook for business, industry, government, and
the military (pp. 312–335). New York, NY: Macmillan.
Salas E., Priest, H.A., Wilson, K.A., & Burke, C.S. (2006).
Scenario-based training: Improving military mission per-
formance and adaptability. In T. Britt, A. Adler, & C. Castro
(Eds.), Military life: The psychology of serving in peace and com-
bat (pp. 32–53). Westport, CT: Greenwood.
Salomon, G., & Perkins, D.N. (1989). Rocky roads to transfer:
Rethinking mechanisms of a neglected phenomenon.
Educational Psychologist, 24, 113–142.
Schoomaker, E.B. (2007). Military medical research on cogni-
tive performance: The warfighter’s competitive edge. Aviation,
Space, and Environmental Medicine, 78(5, Suppl.), B4–6.
Stevens’ handbook of experimental psychology. (2004).
Hoboken, NJ: Wiley.
Performance Improvement • Volume 50 • Number 7 • DOI: 10.1002/pfi 15
8/17/11 11:05:57 AM
 Thompson, M.M., & McCreary, D.A. (2006). Enhancing mental
readiness in military personnel. In T. Britt, A. Adler, & C.
Castro (Eds.), Military life: The psychology of serving in peace
and combat (pp. 54–80). Westport, CT: Greenwood.
U.S. Army. (2007). A guide to female soldier readiness. Aberdeen
Proving Ground, MD: U.S. Army Center for Health Promotion and
Preventive Medicine. Retrieved from http://www.hooah4health.
com/deployment/docs/Female_Soldier_Readiness_Guide.pdf.
van Krimpen-Stoop, E.M., & Meijer, R.R. (2000). Detecting
person misfit in adaptive testing using statistical process
control techniques. In W. J. van der Linden & C.A. Glas (Eds.),
Computerized adaptive testing: Theory and practice (pp. 201–
220). Norwell, MA: Kluwer.
Warm, J.S., Matthews, G., & Finomore, V. (2008). Vigilance,
workload, and stress. In P.A. Hancock & J.L. Szalma (Eds.),
Performance under stress (pp. 115–142). Burlington, VT:
Ashgate.
Wesensten, N.J., Belenky, G., & Balkin, T.J. (2005, Spring).
Cognitive readiness in network-centric operations. Parameters,
pp. 94–105.

MATTHEW HERIC, PhD, is CEO of IAVO Research and Scientific and e-Educational Systems and
Research and has been involved with military and educational research programs since the early
1980s. He has more than 25 years of experience as a researcher, consultant, and business owner. His
primary academic focus is the application of statistical models to evaluate learning and educational pro-
grams. He has led several instructional programs for the U.S. Air Force (U-2 program), NASA (Defense
Landsat Program Office), and the National Exploitation Lab (Washington, DC) and has worked as a
guest lecturer, invited speaker, and contract writer. He may be reached at mheric@iavo-rs.com.

JENN CARTER is the division director of the behavioral sciences group at IAVO Research and
Scientific and is leading three long-term Office of Naval Research projects in the field of human,
social, cultural, and behavioral research. She has expertise in rhetorical communications and has
worked with the North Carolina Population Center studying the human impact on land use and land
change in developing nations, automata model development, spatial analysis of human events, pre-
dictive model creation, and statistical applications. She may be reached at jcarter@iavo-rs.com.

16 www.ispi.org • DOI: 10.1002/pfi • AUGUST 2011

PFI20227.indd 16 8/17/11 11:05:57 AM

Copyright of Performance Improvement is the property of John Wiley & Sons, Inc. and its content may not be
copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written
permission. However, users may print, download, or email articles for individual use.