2012 IEEE International Conference on Systems, Man, and Cybernetics

October 14-17, 2012, COEX, Seoul, Korea

COSA2 - A Cognitive System Architecture with Centralized

Ontology and Specific Algorithms
Stefan Brggenwirth and Axel Schulte
Institute of Flight Systems
Bundeswehr University Munich
85577 Neubiberg, Germany
{stefan.brueggenwirth, axel.schulte}@unibw.de

Abstract—In this article, we present the architectural and
algorithmic details for a COgnitive System Architecture that
uses a Centralized Ontology with Specific Algorithms (COSA2 ).
COSA2 is a layered intelligent agent framework on the basis
of the modified Rasmussen model of human performance. It
encompasses integrated algorithmic support for goal-oriented
situation interpretation, dynamic planning and plan execution,
as well as provisions for reactive behavior. A unique feature is
the claim for an expressive, centralized knowledge representation,
used by all functions to ensure consistency. The framework is
being applied to different problems in the domain of uninhabited
aerial vehicles. This article focuses on high level, concept-based
behavior and illustrates the modeling and processing details by
a simplified UAV mission management example.
Keywords—Intelligent agents; Knowledge modeling; Plan exe-
cution, formation, and generation; Rule-based processing
I. I NTRODUCTION
Increasing demand for highly automated, uninhabited sys-
tems requires the development of generic, yet flexible software
architecture as the basis for the COSA2 framework. Then, we
describe each of its constituting subfunctions and the syntactic
and algorithmic details. Throughout the article we illustrate the
concepts by an example taken from UAV mission management.
The article concludes with open research questions.
II. T HE C OGNITIVE P ROCESS
The cognitive process describes a cyclic information pro-
cessing scheme that is used within the Cognitive System
Architecture. It is based on a model of human performance in
information processing published by J. Rasmussen [6] in 1983.
It furthermore serves as a guideline for knowledge acquisition
from domain experts and as an ontology guideline for domain
specific modeling.
,ƵŵĂŶWĞƌĨŽƌŵĂŶĐĞ
ƉƌŽĐĞĚƵƌĞͲďĂƐĞĚ ĐŽŶĐĞƉƚͲďĂƐĞĚ

K^ ŽŶĐĞƉƚƐ DŽƚŝǀĂƚŝŽŶĂůŽŶƚĞdžƚƐ dĂƐŬKƉƚŝŽŶƐ

ŵĂƚĐŚŝŶŐ ŐŽĂůƐΘ ƚĂƐŬ
ďĞŚĂǀŝŽƵƌ

frameworks to provide these systems with intelligent decision ŝĚĞŶƚŝĨŝĐĂƚŝŽŶͲ ĐŽŶĐĞƉƚƐ 'ŽĂů ĐŽŶƐƚƌĂŝŶƚƐ
WůĂŶŶŝŶŐ
ĂŐĞŶĚĂ
ƌĞůĞǀĂŶƚ /ĚĞŶƚŝĨŝĐĂƚŝŽŶ
making capabilities. The Institute of Flight Systems has suc- ĐƵĞƐ ĞƚĞƌŵŝŶĂƚŝŽŶ

cessfully applied its Cognitive System Architecture (COSA) dĂƐŬ^ŝƚƵĂƚŝŽŶƐ WƌŽĐĞĚƵƌĞƐ

ĐƵƌƌĞŶƚƚĂƐŬ ĂĐƚŝŽŶ
[1] in various domains, ranging from multiple uninhabited dĂƐŬ ;ŝŶƚĞŶƚͿ dĂƐŬ
ďĞŚĂǀŝŽƵƌ

ŝŶƐƚƌƵĐƚŝŽŶƐ
aerial vehicle (UAV) guidance [2], [3] to intelligent system ƚĂƐŬͲƌĞůĞǀĂŶƚ
ĞƚĞƌŵŝŶĂƚŝŽŶ džĞĐƵƚŝŽŶ
ĐƵĞƐ
configuration management [4]. Applications developed for
ƵĞDŽĚĞůƐ ^ĞŶƐŽƌŝͲDŽƚŽƌWĂƚƚĞƌŶƐ
COSA follow a conceptual processing scheme, called the cog-
ƐŬŝůůͲďĂƐĞĚ
ďĞŚĂǀŝŽƵƌ

&ĞĂƚƵƌĞ ĐƚŝŽŶ
nitive process, that resembles the human way of information &ŽƌŵĂƚŝŽŶ ĐŽŶƚƌŽůͲƌĞůĞǀĂŶƚĐƵĞƐ ŽŶƚƌŽů
processing. By using this analogy we belief that: (1) the
modeling and ontology creation by human domain experts, ƐĞŶƐĂƚŝŽŶƐ ƐĞŶƐĂƚŝŽŶƐ
ĞĨĨĞĐƚŽƌ
tŽƌŬ ĐŽŵŵĂŶĚƐ
as well as the interface to a human operator may benefit from ůŐŽƌŝƚŚŵŝĐ ĐůĂƐƐĞƐ ǁŝƚŚŝŶ K^͗ ŶǀŝƌŽŶͲ
the semantics used in the cognitive process and (2) intelligent /ŶĨĞƌĞŶĐĞͲĂƐĞĚ ^ĞĂƌĐŚͲĂƐĞĚ WĂƚƚĞƌŶͲĂƐĞĚ ŵĞŶƚ

agents using the COSA framework have the ability to act

in a flexible and comprehensible way. The first version of
the COSA framework used the cognitive processor Soar [5]
as a rule-based inference engine. Beside inference however,
it was found that the individual conceptual processing steps
specified in the cognitive process may benefit from further
algorithmic support provided by a generic cognitive system
architecture. In this article, we describe a novel COgnitive
System Architecture with a Centralized Ontology and Specific
Algorithms (COSA2 ) to facilitate each such processing step,
along with the embracing architecture to connect these sub-
processes into an intelligent processing framework.
This article is organized as follows: First, we describe
the cognitive process and the resulting overarching agent
Figure 1. The modified Rasmussen scheme [7] illustrating the processing
scheme of COSA2
A. The Modified Rasmussen-Scheme
The modified Rasmussen scheme [7] shown in Fig. 1,
laid the intellectual foundation in the attempt to establish
an analogous computation paradigm for intelligent software
agents. The modified Rasmussen scheme breaks down the
complex phenomena of human cognition into a number of
simplified, cognitive subfunctions, each responsible for certain
dedicated aspects of human cognition. These are shown as
boxes in the figure, with arrows indicating the primary flow
of information. During runtime, the cognitive process iterates

978-1-4673-1714-6/12/$31.00 ©2012 IEEE 307

through the different cognitive subfunctions. All cognitive sub-
functions are associated within one of three horizontal layers,
that describe increasing levels of abstraction and flexibility
in human behavior. The inner, light-blue box represents the
subset of human cognitive subfunctions that are algorithmically
supported by COSA2 .
Note that Rasmussen’s scheme is typically used, e.g. by
psychologists to describe the human cognition performance. In
its original interpretation, the different subfunctions, as well as
the three layers of behavior are essentially interwoven. Besides
the centralized knowledge representation, we have taken a
rather pragmatic, computer-science and application oriented
approach by separating the cognitive subfunctions. E.g. human
skill-based behavior (e.g. visual perception occurring as part
of the human Feature Formation process) is only partially
covered by the architecture.
B. Central Knowledge Representation
Even though each cognitive subfunction serves a specific
purpose in the cognitive process, we describe each subfunction
using a generic ordering scheme as shown in Fig. 1. The cog-
nitive process distinguishes static, a-priori knowledge specified
during design time (shown in blue, e.g. C ONCEPTS or TASK
O PTIONS) and situational knowledge, generated during run-
time (shown in red in Fig. 1, e.g. identification relevant cues or
matching concepts). As already indicated, gray boxes represent
the cognitive subfunctions that constitute the cognitive process.
Each cognitive subfunction performs a specific algorithm on
the basis of the indicated a-priori knowledge. Algorithms are
divided into the three classes inference-, search and pattern
matching-based, and explained further in section 3.
An important claim of the cognitive process is for a central
knowledge representation. This means, that both situational
and a-priori knowledge are specified exactly once, and accessi-
ble from all the different subfunctions. As a consequence, each
cognitive subfunction may generally read all situational knowl-
edge elements available in working memory. However, it writes
specifically only into the situational knowledge indicated by
the outgoing arrow. For example, the cognitive subfunction
Identification is implemented as an inference algorithm which
uses C ONCEPTS as a-priori knowledge. It may read from
anywhere in working memory (with its primary source be-
ing identification relevant cues) and exclusively modify the
situational knowledge on matching concepts.
C. Working Memory Representation
The situational knowledge forms the agent’s working mem-
ory. It is represented as a graph with weighted edges. Like
in traditional objected oriented programming languages, we
distinguish between static, prototypical class definitions and
concrete instantiations of these classes as unique objects.
During runtime, the agent may instantiate an arbitrary number
of concrete objects from these class templates as vertices. As
shown in Fig. 2, directed edges are used to store semantic
relations between the objects or attributes.
308
hs dĂƌŐĞƚ
ĚŝƐƚĂŶĐĞ
>Ăƚ сϰϴ͘ϬϮ ϴ͘ϱ >Ăƚ сϰϴ͘ϭϮ
>ŽŶсϭϭ͘ϰϵ >ŽŶсϭϭ͘ϳϮ
Figure 2. A graph with weighted edges is used to describe the situational
knowledge
Each edge in the WM graph represents a certain
fact in the agent’s world knowledge. An edge e1 =
(U AV, T arget, distance, 8.5) may represent the fact, that
UAV is 8.5 km away from Target. A condition is a set of
propositional expressions connected by logical connectives,
that allow the matching of complex patterns including AND,
OR, NOT relationships and path-expressions. The central
pattern-matching algorithm for all cognitive subfunctions GET-
M ATCHING I NSTANCES (condition, WM) searches the WM for
all sets of fact-tuples, that match a certain condition. For
example, the condition “distanceU AV,T arget < 10 returns the
fact e1 . Note, that multiple combinations of facts may match
the condition, and are hence found as individual instances.
D. Related Work
Compared to its predecessor COSA, COSA2 shifted the
focus from a cognitive architecture (developed to describe
human cognition processes, such as Soar or ACT-R [8]) to-
wards the behavior of rational agents investigated by computer
science or robotics. From this agent-theoretical point of view,
the approach resembles a hybrid agent architecture [9] with
horizontal layering, as found in Fergusons TouringMachines
[10] or other Three-Tier architectures [11]. The lowest layer
(skill-based) is reduced to converting subsymbolic input data
to an abstract symbolic representation (and vice versa). These
symbols are then fed in parallel into a reactive layer1 with
prestored procedures (procedure-based) and a deliberative,
goal-oriented layer (concept-based) comparable to a BDI-agent
design [12], augmented with dynamic planning capabilities
[13].
A major challenge for the centralized knowledge represen-
tation of COSA2 is a tight integration between action planning
and reasoning about the consequences within a single world
model. Among the first to address this issue was the possible
worlds approach by Ginsberg and Smith [14]. Classical plan-
ning languages like STRIPS [15] handle the framing problem
[16] well, but still suffer from the ramification problem. That
is, all consequences of executing an action, including indirectly
inferred effects, must explicitly be listed as postcondition.
Derived predicates, introduced later with the Planning Domain
Definition Language (PDDL) [17] help relief this pitfall, but
intentionally restrict the expressiveness of indirect effects. A
reactive planner, that can handle indirect effects was developed
by Williams [18] and applied in embedded intelligent systems.
The COSA2 framework offers the full expressiveness of its
inference engine to derive indirect effects.
1 Reactive behavior is realized only rudimentarily in our framework and
mentioned here for completeness
III. D ESCRIPTION OF THE C OGNITIVE P ROCESSING S TEPS cognitive subfunctions responsible within the cognitive process
As expressed by the Rasmussen scheme in Fig. 1, each to draw the situational picture. As shown in the figure, she
cognitive subfunction is responsible for certain aspects within distinguishes three levels of SA with increasing complexity.
the human cognition process. This analogy describes the func-
/ŶĨĞƌĞŶĐĞͲĂƐĞĚ ^ĞĂƌĐŚͲĂƐĞĚ WĂƚƚĞƌŶͲďĂƐĞĚ
tional requirements on each cognitive subfunction and serves
as a modeling guideline for the implementation of the cognitive WĞƌĐĞƉƚŝŽŶ ŽĨ
ĞůĞŵĞŶƚƐ ŝŶ
ŽŵƉƌĞŚĞŶƐŝŽŶ
ŽĨ ĐƵƌƌĞŶƚ
WƌŽũĞĐƚŝŽŶ ŽĨ
ĨƵƚƵƌĞ ƐƚĂƚƵƐ WůĂŶŶŝŶŐ
^ƚĂƚĞŽĨ ƚŚĞ WĞƌĨŽƌŵĂŶĐĞ
ƐŝƚƵĂƚŝŽŶ ĂŶĚ
process as an intelligent agent framework. ĞŶǀŝƌŽŶŵĞŶƚ
ĐƵƌƌĞŶƚ ƐŝƚƵĂƚŝŽŶ
ĞĐŝƐŝŽŶ
ŽĨ ĂĐƚŝŽŶƐ
>ĞǀĞůϭ
During runtime, COSA2 transitions through the individual
>ĞǀĞůϮ >ĞǀĞůϯ

states of the cognitive process as shown in Fig. 3. Besides the • 'ŽĂůƐ

execution state within the cognitive process, the architecture • KďũĞĐƚŝǀĞƐ
• džƉĞĐƚĂƚŝŽŶƐ

separately maintains the state of the current task agenda, as also

illustrated in the figure. The processing cycle can be divided Figure 4. Interpretation of situation awareness based on Endsley’s model
into three parts that use specific algorithms: (1) Environment
and goal-interpretation using inference algorithms (2) Action
planning and adjustment using search and constraint satisfac- Level 1 (L-1) simply represents the elementary building
tion algorithms (3) Plan execution using pattern matching. blocks of a perceived situation, comparable to Identification
Relevant Cues generated by Feature Formation.
WƌŽŐƌĞƐƐ
dŝŵĞůĂLJĞƌ
Level 2 (L-2) SA combines the different elements to derive
z

/ĚĞŶƚŝĨŝĐĂƚŝŽŶ
>ϮƌƵůĞ
ĨŝƌĞĚ
/ĚĞŶƚŝĨŝĐĂƚŝŽŶ
>ϯƌƵůĞ
ĨŝƌĞĚ
'ŽĂů hŶĞǀĂůƵĂƚĞĚ
ŐĞŶĚĂсс
WůĂŶŶĞĚ
a comprehensive representation of the situation, analogous to
Matching Concepts as deduced by Identification. In addition,
WůĂŶ^ŽůƵƚŝŽŶ WůĂŶͬ
>Ϯ >ϯ ĞƚĞƌŵŝŶĂƚŝŽŶ dŝŵĞůĂLJĞƌƐ͍
>ϮƌƵůĞƐ >ϯƌƵůĞƐ 'ƌƵůĞƐ ŶĂůLJƐŝƐ ZĞƉůĂŶ
ĐŚĞĐŬĞĚ ĐŚĞĐŬĞĚ ĐŚĞĐŬĞĚ ŐĞŶĚĂсс
>ϭƌƵůĞ &ĂŝůĞĚ
ĨŝƌĞĚ
ŐĞŶĚĂ^ƚĂƚĞ
E ŐĞŶĚĂ͗с
hŶĐŚĞĐŬĞĚ ŐĞŶĚĂсс
sĂůŝĚ
WůĂŶ
&Ăŝů
Endsley includes goals and objectives into L-2 SA, since they
hŶĐŚĞĐŬĞĚ
ZĞƐĞƚ
tD
ĚũƵƐƚ
ŐĞŶĚĂ ŐĞŶĚĂсс
critically influence the interpretation of a situation.
&Ăŝů
ĞƚĞƌŵŝŶĞ
WƌŽŐƌĞƐƐŝŽŶ;Ϳ ŐĞŶĚĂсс
Finally, level 3 (L-3) SA takes a projection of future
ĞƚĞƌŵŝŶĞ dŝŵĞĚ
WƌŽŐƌĞƐƐŝŽŶ;Ϳ dŝŵĞĚ
ŐĞŶĚĂсс
E
ĞƚĞƌŵŝŶĞ
WƌŽŐƌĞƐƐŝŽŶ
environment states into account. Based on this interpretation
sĂůŝĚ͍
ĚũƵƐƚ
ŐĞŶĚĂ;Ϳ
ĚũƵƐƚ
ŐĞŶĚĂ;Ϳ
z
of Endsley’s model, Feature Formation, Identification and
&ĂŝůĞĚ
^ŽůƵƚŝŽŶ
sĂůŝĚ Goal Determination create SA within the cognitive process.
dĂƐŬ dĂƐŬ

2) Feature Formation (L-1 Perception): The human Fea-

ŶĂůLJƐŝƐ;Ϳ
WůĂŶ;Ϳ
WůĂŶ;Ϳ
ŶĂůLJƐŝƐ;Ϳ
>ϭƌƵůĞƐ
ĐŚĞĐŬĞĚ
WůĂŶŶĞĚ
&ĞĂƚƵƌĞ
&ŽƌŵĂƚŝŽŶ;>ϭͿ
/ŶƉƵƚZĞĂĚ
ZĞĂĚ
/ŶƉƵƚ
^ƚĂƌƚ
Figure 3. State transition diagram of the cognitive processing cycle and the
task agenda
A. Situational understanding using an inference engine
Within the cognitive process, the three subfunctions Fea-
ture Formation, Identification and Goal Determination use
an inference mechanism to build up the situational understand-
ing of the agent, that sets the basis for subsequent computation
processes. The relevant a-priori knowledge associated with the
three subfunctions is given as production rules. A produc-
tion rule has the form [name] : (condition, effect, priority,
[duration]) and expresses a simple ’if condition =⇒ then
conclusion’ relationship: For each graph-pattern in the working
memory that matches the rule’s condition (see previous section
II-C), the rule is said to have fired, and a certain effect is
applied by adding or modifying edges in the working memory.
1) Endsley’s Model of Situation Awareness: To illustrate
the design choices made for the inference-based subfunctions,
it is worthwhile looking at a perspective on human cognition
based on Endsley’s model [19] (Fig. 4). The concept focuses on
the elements of situation awareness (SA) as observed by human
operators. It provides some analogies to the inference-based
ĞƚĞƌŵŝŶĂƚŝŽŶ džĞĐƵƚŝŽŶ
^ŽůƵƚŝŽŶ
ture Formation process encompasses elaborate perception
and signal processing tasks, in order to assemble semantic
information from an initially unfiltered stream of sensations.
tƌŝƚĞKƵƚƉƵƚ;Ϳ
Within COSA2 , it is assumed that appropriate preprocessing
steps take place outside the framework, so that assigning
meaningful symbols to incoming input-data becomes trivial.
Feature Formation is then reduced to being the first sub-
function, that starts a new processing cycle by sampling the
input interface of the agent. It uses a-priori knowledge on C UE
M ODELS to transform subsymbolic, numeric input data from
the work environment into a symbolic representation. These
symbols are then provided as identification relevant cues to the
concept-based layer or as task-relevant cues to the procedure-
based layer. A C UE M ODEL rule may for example express
a conclusion that each edge representing an input voltage
between 0-5V is to be interpreted as ’low’ voltage.
3) Identification (L-2 and L-3 Perception): The Identifica-
tion subfunction assembles a high-level representation of the
situation perceived by the agent. The associated knowledge
on C ONCEPTS comprises rules to reason about the immediate,
currently perceived situation (L-2) and about projections of
future states (L-3).
a) Reasoning about the current environment state using
L-2 rules: When reasoning about a situation, the Identification
subfunction typically matches upon identification relevant cues
created by the Feature Formation subfunction, and applies
its a-priori knowledge on C ONCEPTS to interrelate these and
expand a broader, comprehensive situational picture. As shown
in Fig. 6, the result is written into the matching concepts

309
 Ϭ &ƵůĨŝůůĞĚс&ΛϬ
/ŶƚĞƌƉƌĞƚDŝƐƐŝŽŶKƌĚĞƌΛϬ
>ϮZƵůĞ
WŚŽƚŽŐƌĂƉŚĞĚс&ΛϬ
dĂƌŐĞƚDŽǀŝŶŐΛϬͲϭϬ
>ϯZƵůĞ
Figure 5.
namespace in working memory. This cognitive subfunction
condenses various, unrelated symbols from the situational
feature space into meaningful, abstract concepts.
/ĚĞŶƚŝĨŝĐĂƚŝŽŶ
ŝĚĞŶƚŝĨŝĐĂƚŝŽŶͲ
ƌĞůĞǀĂŶƚĐƵĞƐ
ƐŝƚƵĂƚŝŽŶĂů
ĨĞĂƚƵƌĞƐƉĂĐĞ
Figure 6. Identification expands a comprehensive situational picture [7]
b) Reasoning about hypothetical environment develop-
ments using L-3 rules: Motivated by Endsley’s model of
situation awareness, external events or processes, occurring in
the environment with a temporal extent, are described as L-3
perceptions. L-1 and L-2 perception rules describe immediate
conclusions on facts that have already occurred and been
observed by the agent. An active L-3 rule, on the other hand,
describes an ongoing transition process in the environment,
that strives towards a steady state in the future, modeled by
the rule’s effects.
Each L-3 rule is therefore associated with a duration,
that determines the timelayer when the nominal, anticipated
effects of the transition process apply. Within this timelayer,
the anticipated situation is then interpreted by Identification
and Goal Determination rules. Effects of an L-3 rule are
considered hypothetical, until actually encountered in the envi-
ronment. Due to their hypothetical character, L-3 rules along
with its anticipated and derived effects are retracted, should
its precondition become invalidated. Usually, the precondition
of an L-3 rule describes a deviation criterion, measuring the
deviation between current situation and anticipated effects of
the L-3 rule. If the deviation between current and anticipated
future state, falls below a certain threshold, the transition
process described by the L-3 rule is no longer active and the
rule is retracted.
An L-3 rule might for example express that, if there is a
deviation between the commanded and the current position of
a UAV (precondition), the current position will strive towards
the commanded position (effect), and eventually reach it, at a
point in time given by the rule duration.
4) Goal Determination: The concept-based layer of
COSA2 is driven by goals. The Goal Determination step
ϭϬ
ĐŚŝĞǀĞDŝƐƐŝŽŶΛϬ
'ŽĂů;ZĞŵŽǀĞĚͲƚͲŶĚͿ
WŽƐϭсy͕zΛϭϬ
WŽƐϮсy͕zΛϭϬ ŝƐƚĂŶĐĞ сϭϬΛϭϬ
ŽŵƉƵƚĞŝƐƚĂŶĐĞΛϭϬ DĂŝŶƚĂŝŶŝƐƚĂŶĐĞΛϭϬ
>ϮZƵůĞ 'ŽĂů;DĂLJ EĞǀĞƌKĐĐƵƌͿ
Different perception levels during environment interpretation
checks the perceived, abstracted situation for a violation of
desirable states and constraints, modeled as M OTIVATIONAL
C ONTEXTS by the knowledge engineer. Like C ONCEPTS,
these are described as classes, with attributes to hold specific
ŽŶĐĞƉƚƐ
information on the goal and rules, that describe the condition
ϭ
under which the goal becomes instantiated. The semantic is
Ϯ
here, that an instance of a goal in the working memory is
͘͘
Ŷ
͘ created, when the agent is in an undesirable state. With respect
ŵĂƚĐŚŝŶŐĐŽŶĐĞƉƚсϭ
to the subsequent planning process, three different types of
goals are supported:
• RemovedAtEnd Hard goal, that needs to be satisfied,
after all planned tasks that have been executed. E.g. Goal
requiring the accomplishment of a certain mission order.
• MayNeverOccur Hard goal, that describes a critical state,
that may never be reached. E.g. Goal describing an out-
of-fuel condition.
• PenaltyOnOccurence Soft goal, describing a soft-
constraint, that increases a penalty counter when violated.
Among possible solutions that satisfy the hard goals,
COSA2 selects the one that minimizes the penalty counter.
E.g. Penalty counter is increased by 10 units when cross-
ing an unsafe area.
5) Summary and Example: Figure 5 summarizes the rule
inference of different cognitive subfunctions, used for en-
vironment interpretation of a simple UAV mission manage-
ment agent. The figure shows the higher-level, concept-based
subfunctions Identification and Goal Determination only.
Feature Formation is not covered here, as the associated L-1
rules are primarily used to process raw input-data.
The scenario contains the UAV and a moving target. The
ACU has a built-in goal MaintainDistance of type MayNever-
Occur, and another goal of type RemovedAtEnd to fulfill
the mission order, in this case to photograph the moving
object. The figure illustrates the currently perceived situation.
It contains all active rules (represented by colored boxes with
round corners), along with the edge modifications they created
(represented as labeled arrows). The @-symbol denotes the
timelayer, at which the rule fires and its edge modification
occur.
At T = 0, a regular L-2 rule (InterpretMissionOrder) fires,
that analyzes the mission order to determine, whether or not
the target has been photographed yet. As a result, the edge
Fulfilled is set to False (at T = 0), which in turn instantiates a
goal AchieveMission. The goal-type RemovedAtEnd indicates
to the subsequent planning process, that the goal must be
accomplished after execution of the plan. In addition, an L-
310
3 rule (TargetMoving starting at T = 0 with duration D = 10) facts to the working memory. All rules, that match on a
is triggered, that creates a second timelayer at T = 10. The modified fact in their condition, need to be checked again,
projected situation is then interpreted by the set of L-2 rules. and are therefore added to the RULE AGENDA based on their
In this example, the rule ComputeDistance would fire and priority. Once no more rules remain on the RULE AGENDA,
assess the new distance between the UAV and the target. the cycle has reached the quiescence state. In this state, the
This new distance might exceed the maximum admissible working memory content is considered updated and consistent
range as modeled by the MaintainDistance goal. The goal-type with the given knowledge in the rule base.
MayNeverOccur indicates to the subsequent planning process, The cognitive processing cycle starts by sampling the input
that the natural progression of the environment would lead interface and then calls I NFERENCE (C UE M ODELS ∪ C ON -
to a critical state. Hence, the agent must not stay passive CEPTS ∪ M OTIVATIONAL C ONTEXTS, WM) as described in
with respect to the moving object, either by preventing the Fig. 7. Even though all rules are concurrently relevant, the
anticipated effects of the L-3 rule or by performing other priority ranges are arranged as follows: C UE M ODELS (L-1)
actions to avoid the violation of the MaintainDistance-Goal. > C ONCEPTS (L-2) > C ONCEPTS (L-3) > M OTIVATIONAL
C ONTEXT rules. The rules are therefore generally checked
1: procedure I NFERENCE(knowledge, wm) in that order, e.g. the first M OTIVATIONAL C ONTEXT rule is
2: time ← 0 evaluated, after all active C ONCEPTS rules have fired.
3: loop A special convention holds for the management of L-3 rules
4: while ruleAgenda = empty do within the C ONCEPTS knowledge to ensure future time-layers
5: rule ← DEQUEUE F RONT(ruleAgenda) are evaluated in the right order. When an L-3 rule has fired, its
6: matches ← GET M ATCHING - effects are not applied right away, but added to an associative
I NSTANCES(rule, wm) array, that stores the occurrence time (current timelayer + rule
7: if rule ∈ L3 then  Postpone L3 Rule eval duration) for each instance of a matched L-3 rule. As shown in
8: L3Rules ← A DD(time + dur, matches) Fig. 3, the cognitive process progresses to the next timelayer,
9: continue by dequeuing the L-3 instances with the earliest occurrence
10: end if time and applying its effects. On the next timelayer, inference
11: for each instance in matches do of matching C ONCEPTS and M OTIVATIONAL C ONTEXT starts
12: modF acts ← FIRE RULE(instance) over until all timelayers are evaluated.
13: SUBMIT T OWM(modF acts, wm) B. Planning of own tasks as a search problem
14: touchedRules ← GET T OUCHED -
Planning describes the problem of creating a sequence of
C ONDITIONS(modF acts, knowledge)
tasks, to be executed by the agent with the aim of transitioning
15: ruleAgenda ← ADD(touchedRules)
from the current environmental state into some desired target
16: end for
state. The current environment state is primarily described by
17: end while
the matching concepts, whereas the target state results from the
18: if L3Rules = empty break
goals & constraints identified during the Goal Determination
19: SORT K EYS (L3Rules)  Next timelayer
phase. The set of possible actions of the agent is described as
20: time, matches ← DEQUEUE F RONT(L3Rules)
TASK O PTIONS, where a task is defined by a set of precon-
21: for each instance in matches do
ditions that must be satisfied to enable it and the expected
22: modF acts ← FIRE RULE(instance)
effects that result from its execution. If the planning process
23: SUBMIT T OWM(modF acts, wm)
was successful, the selected subset of tasks is stored as the
24: touchedRules ← GET T OUCHED -
task agenda for execution.
C ONDITIONS(modF acts, knowledge)
25: ruleAgenda ← ADD(touchedRules) >ϯͲŽŶĐĞƉƚ
26: end for
WƌŽũĞĐƚŝŽŶ
27: end loop
28: end procedure ŽŶĐĞƉƚƐ DŽƚŝǀĂƚŝŽŶĂůŽŶƚĞdžƚƐ dĂƐŬKƉƚŝŽŶƐ
ŵĂƚĐŚŝŶŐ ŐŽĂůƐΘ ƚĂƐŬ
ŝĚĞŶƚŝĨŝĐĂƚŝŽŶͲ
ĐŽŶĐĞƉƚƐ 'ŽĂů ĐŽŶƐƚƌĂŝŶƚƐ ĐĂŶĚŝĚĂƚĞ
Figure 7. Inference algorithm used in COSA2 ƌĞůĞǀĂŶƚ /ĚĞŶƚŝĨŝĐĂƚŝŽŶ
ĞƚĞƌŵŝŶĂƚŝŽŶ
WůĂŶŶŝŶŐ
ĐƵĞƐ
ƚĂƐŬ
ĂŐĞŶĚĂ

6) Algorithmic Details: COSA2 uses an graph-based infer-
ence engine, described in [20], [21]. Central to the algorithm
(Fig. 7) is a priority queue (RULE AGENDA), that stores an
ordered list of unchecked rules with modified conditions. The
interpreter removes the rule with the highest priority from the
RULE AGENDA and builds all active rule instances that match
the condition in the working memory. Each active instance is
then fired by applying it’s action and submitting the modified
Figure 8. The projection based planning approach reuses knowledge on
C ONCEPTS and M OTIVATIONAL C ONTEXTS
1) Projection based planning approach: The idea of central
knowledge representation suggests that a-priori knowledge
used elsewhere to describe portions of the ontology, is reused
whenever applicable. In traditional planning languages, the
new environment state is often modeled as a direct effect

311
of executing a task. In contrast, the approach taken in the
COSA2 framework is based on projection, as shown in Fig.
8: A new search state is expanded by applying the effects
of a feasible task candidate and subsequently interpreting the
resulting situation (using the inference-based subfunctions as
described in section III-A).
A physical effect to be applied by the agent is often split
into the active part of sending the respective command and a
passive part, caused by the anticipated environment reaction
to the command, modeled as an L-3 concept rule. If, e.g. the
mission requires the UAV to relocate, modeling of this process
is split into the activity of sending the command to the flight
management system (Task: FlyToX in Fig. 10), and the envi-
ronment reaction of transitioning the UAV to the new location
(L-3: EnrouteToX). The environment reaction is represented by
an L-3 concept rule, describing the anticipated achievement
of the new physical position after a certain duration. The
agnostic semantics of an L-3 rule as explained in section III-A3
are preserved: Assuming the command to program the new
waypoint (FlyToX) happens instantaneously and can never fail,
the time to reach the new position is naturally described by the
duration of the L-3 rule (EnrouteToX), which is automatically
monitored for failure by the architecture. This allows the
consistent integration with the perception process described
in the previous section.
2) Transitioning from perception to planning in the exam-
ple: Fig. 10 shows a valid agenda, that is a solution to the
mission management example given in section III-A5. During
the inference based subfunctions, the agent perceived a moving
target as indicated by the respective L-3 rule. As mentioned,
this describes the anticipated, nominal environment, should the
rule’s preconditions remain valid.
In the given scenario, the ACU may utilize the external
development to its own benefit, by scheduling an interception
course at the right time. This is indicated by the task FlyToX
(shown in red) and the L-3 rule EnrouteToX (shown in green) it
triggers. As before, the L-2 rule computeDistance determines
the new distance between the UAV and the position of the
target. A sufficiently low distance in turn establishes the
required precondition for two follow-up tasks. TakePhoto will
lead to the accomplishment of the mission order and the
retraction of the respective goal.
3) Planning architecture in COSA2 : In order to efficiently
tackle the computational complexity related to planning, three
different components are involved in the planning (and replan-
ning) process in COSA2 as shown in Fig. 9. The sequence,
in which the components are invoked is illustrated in Fig. 3.
The planning subfunction transitions through several substates,
depending on the state of the task agenda. Should the goal-
determination subfunction report unsatisfied goals or violated
constraints, the planning subfunction is initiated by setting the
state of the task agenda to UNCHECKED.
312
ĚũƵƐƚĞĚ^ŽůƵƚŝŽŶ;dĂƐŬŐĞŶĚĂͿ
>ŝŶĞĂƌ
tŽƌŬŝŶŐ
DĞŵŽƌLJ
ƵƌƌĞŶƚ^ŝƚƵĂƚŝŽŶ;hƉĚĂƚĞͿ KƉƚŝŵŝnjĂƚŝŽŶ WůĞdž
ĂŶĚ
^ĐŚĞĚƵůŝŶŐ
^ŽůƵƚŝŽŶ
ZƵůĞ
ŽŶƐƚƌĂŝŶƚƐ
ĂƐĞ WůĂŶ
K^Ϯ
ŶĂůLJƐŝƐ
ŽŵƉŝůĞƌ
ĂƐĞ
^ŽůƵƚŝŽŶ
W>
D> WůĂŶŶĞƌ W>
Figure 9. COSA2 planning architecture
4) PDDL planning: For initial planning or if replanning
is required, an external PDDL planner is used to generate a
baseline-solution, using nominal parameter values (shown in
orange). PDDL [17] is an open academic standard to describe
planning problems. It originated from the state-based STRIPS
[15] description of an action by a set of preconditions and
effects. A task may be selected for execution, if its precon-
ditions are fulfilled in the current state. Execution of a task
transitions to a follow-up state by applying the tasks effects.
During the compilation of the COSA2 knowledge model, each
task and rule is translated into an PDDL action. Weighted
edges of BOOLEAN type are represented as PDDL predicates,
INT or FLOAT types as PDDL fluents. Once a valid baseline
solution is found, the agenda state transitions to PLANNED.
The semantics of the inference-based projection phase are not
naturally available in PDDL and encoded as conditional effects
of actions.
5) Linear optimization and scheduling: Once a valid task
agenda is generated, the planning subfunction is reduced to
determining, whether the existent agenda is still valid and
progressing with its execution is still in line with the goals.
Comparable to the long-lasting intentions of a BDI-agent
design, we follow the convention to stick to the structure
of a generated agenda as long as possible. This avoids the
computationally expensive recomputation of a plan and gives
the framework a certain long-term momentum that prevents
numeric oscillations with minimal operational benefit. Besides
the structure of an agenda (that is, the order of selected tasks),
the numeric parameters of the agenda require constant adjust-
ment, when executing the plan in a real-world, continuous
environment. In particular floating point parameters, such as
scheduled execution times for tasks or threshold distances that
trigger new tasks, usually never precisely hit their nominal
values when executed in reality.
However, the essential information that is captured in the
structure of the agenda - rather than sticking literally to
nominal parameter values - is the set of logical constraints
among its elements, that must be fulfilled, in order to retain the
agenda’s causal structure (shown in the graph representation
of the agenda, e.g. Fig. 10).
 Ϭ ϭϬ
dĂƌŐĞƚDŽǀŝŶŐΛϬͲϭϬ
WŽƐϭсy͕zΛϭϬ
>ϯZƵůĞ
ŵĚWŽƐсdΛϬ
WŽƐϮсy͕zΛϭϬ
&ůLJdŽyΛϬ ŶƌŽƵƚĞdŽyΛϬͲϭϬ
dĂƐŬ >ϯZƵůĞ
Figure 10. Agenda graph during execution, representing a solution to the planning problem
a) Plan Analysis: Once the PDDL planner found a
solution, it is imported into COSA2 and its causal structure
is analyzed (shown in blue, Fig. 9). This happens by stepping
through each task and inference step, while recording the de-
pendencies between added effects and required preconditions
as shown in Fig. 10. A task, that adds a precondition that
is required later on by a rule, leads to a temporal ordering
constraint that enforces the task to be executed, before the
rule fires. In addition, numeric constraints on variables are
recorded, e.g. a maximum threshold for a distance that occurs
in a precondition of a rule contained in the structure of the
agenda. The state of the agenda is not modified by this.
b) Numeric Agenda Adjustment: Once the temporal and
causal constraints of the baseline-solution have been analyzed,
they are stored and fed into the constraint-satisfaction program
CPlex and solved for the decision variables on each subsequent
iteration of the cognitive process.
Decision variables are the start times of tasks and start / end
times of L-3 rules. Initially, the start time for task FlyToX is
a decision-variable, with an yet undetermined value. Timetags
are easily computed using the task-agenda graph (Fig. 10).
A time-tag for the firing of a rule equals the latest (highest)
timetag of all its precondition-edges. Outgoing edges of a rule
indicate a modification of the edge by the rule - the respective
timetag of the edge therefore equals the timetag of the rule.
While the start time for a task is not constraint any further, the
end time of an L-3 rule equals its start-time plus its duration.
Besides the temporal aspects, the architecture can keep track
of numerical parameters of a task, which are given a nominal
value for PDDL planning and an upper and lower bound for nu-
merical adjustment. COSA2 therefore traces the computations
and transformations performed on the parameter throughout
the agenda graph, together with the numeric preconditions re-
quired (e.g. distance computation within the ComputeDistance
rule). Together, temporal and numeric constraints encode the
causal structure of the task agenda as a constraint satisfaction
problem.
If the constraint satisfaction problem can be solved, the
numerical values of the task agenda are updated and the state
transitions to VALID. If no solution is found, execution of the
remainder of the agenda has become infeasible, the agenda
state transitions to FAIL and a complete replanning and plan
analysis cycle is initiated.
In contrast to the combinatorial planning problem required
to generate the structure of the agenda, which is known to
be PSPACE complete in general [22], incremental updates for
the plan adjustment can be solved using a linear optimization
algorithm, which is computed very efficiently. Adjustment of
ŝƐƚĂŶĐĞ сϬΛϭϬ WŚŽƚŽŐƌĂƉŚĞĚсdΛϭϬ &ƵůĨŝůůĞĚсdΛϭϬ
ŽŵƉƵƚĞŝƐƚĂŶĐĞΛϭϬ dĂŬĞWŚŽƚŽΛϭϬ /ŶƚĞƌƉƌĞƚDŝƐƐŝŽŶKƌĚĞƌΛϭϬ ĐŚŝĞǀĞDŝƐƐŝŽŶΛϭϬ
>ϮZƵůĞ dĂƐŬ >ϮZƵůĞ 'ŽĂů;ZĞƚƌĂĐƚĞĚͿ
the numeric parameters leads to greater flexibility in plan
execution.
c) Agenda progression monitoring: Once a valid agenda
has been found, a key function, located in-between envi-
ronment perception and planning, is to determine the new
progression stage within the agenda - that is, to determine
which events of the agenda have already occurred and what
is to be done next? As already indicated, dynamic execution
of the agenda cannot literally stick to the scheduled times or
numerical values, but must consider the causal relationship
within the agenda instead.
Again, L-3 rules play a critical role, as all numerical and
temporal offsets originate in deviations between expected,
nominal environment reaction and reality. Determining the
progression state therefore boils down to the question, which
environment reactions, represented as L-3 rules in the agenda,
have already occurred? This in turn, depends on the context of
the remaining agenda, as the observed environment reaction
needs to be “close enough” to allow the remainder of the
agenda to be executed. Note, that the deviation criterion used
for the perception of the L-3 rule, as explained in section III-A,
is not sufficient in general, as the progression stage transitively
depends on the remaining agenda.
For each currently perceived L-3 rule, the architecture
evaluates, whether the L-3 rule has already been achieved, by
inserting for each of its effects the current values, while using
the nominal values for the effects of all other L-3 rules. If the
constraint satisfaction problem can be solved, the respective
L-3 rule is marked as achieved. Also, all rules that transitively
depend on it are marked as achieved and their effects are fixed
to the solution values. Should an L-3 rule not be achieved after
its duration has elapsed, the architecture consideres it failed
and reports a execution monitoring violation.
Fig. 10 shows the agenda during execution with progression
stage indications. A yellow cap on the left of rule indicates
its preconditions are achieved, whereas a yellow cap on the
right indicates, that the rule’s effects are fixed. Note also, that
after the linear optimization phase, the decision variables have
been replaced by their solution values, e.g. the concrete timing
information is now attached. Also, the figure indicates two L-3
rules, occurring in the environment, with about a fifth of their
expected duration elapsed. The red line marks the time elapsed
since agenda was planned.
C. Task Execution and progression monitoring by reactive
pattern recognition
1) Subfunctions within the procedure-based layer: Accord-
ing to Rasmussen [6], human behavior is dominated by routine
procedures, optimized with respect to every-day situations.
313
This procedure-based behavior increases performance in en-
vironments with recurring situational patterns, that trigger a
predetermined action-sequence as reaction. In the modified
Rasmussen scheme [7], the Task Determination subfunction
is responsible for matching such patterns (TASK S ITUATIONS)
and associating with it a predefined, routine action-procedure
for fast, reactive behavior.
In the context of procedure-based behavior, this can be
rudimentarily emulated by directly triggering a task from a
Feature Formation rule. In the context of concept-based
behavior, as focused on in this article, the subfunction Task
Determination detects when to toggle the next task from the
agenda.
A task may itself be composed of several low-level actions
that are stored as automated procedures. Execution of the
procedure is controlled by the Task Execution subfunction,
while Action Control sends the low level actions to the output
interface.
2) Task Determination: Once a task agenda is generated,
the agent needs to successively execute the tasks in it, each at
the appropriate time. A task is executed, if (1) its preconditions
are achieved and (2) its scheduled start-time has elapsed.
Both conditions become trivial, since the reasoning on when
to initiate the next task is primarily done by the agenda
progression monitoring step of the concept-based planning
subfunction, as explained in the previous section.
IV. C ONCLUSIONS
In this article, we presented the architectural and algorith-
mic concepts for a new cognitive system architecture (COSA2 ).
The architecture is an attempt to build an intelligent agent
framework on the basis of the modified Rasmussen model of
human performance.
It encompasses unified functions for goal-driven situation
interpretation, planning and plan execution. A unique feature is
the claim for a centralized knowledge representation, enabling
the reuse of knowledge modeled once, whenever applicable to
ensure consistency. Three algorithmic classes (inference based,
search based and based on reactive pattern matching) have
been identified and prototypically implemented to close the
loop on deliberative behavior. Tighter coupling between the
subfunctions, including subsymbolic, skill-based behavior, as
originally described in Rasmussens model of human cognition,
is subject to current research. Along this effort, fast, reactive
behavior shall be integrated as part of the procedure based
layer.
A first successful flight experiment using the COSA2 frame-
work embedded on fixed-wing UAV was conducted last year
in collaboration with the Naval Postgraduate school, Monterey
[23]. Future areas of application for COSA2 will be in the
field of assistant systems and task-based UAV guidance. This
effort also encompasses the integration of the framework with
airborne hardware and sensors, to evaluate the performance of
the architecture in real-world scenarios.
314
R EFERENCES
[1] H. Putzer and R. Onken, “Cosa - a generic cognitive system architecture
based on a cognitive model of human behavior,” in Cognition, Technol-
ogy & Work, vol. 5, no. 2. Springer, 2003, pp. 140–151.
[2] A. Schulte, C. Meitinger, and R. Onken, “Human factors in the guidance
of uninhabited vehicles: Oxymoron or tautology? the potential of cogni-
tive and co-operative automation,” in International Journal on Cognition
Technology & Work. Heidelberg, Germany: Springer, 2008.
[3] J. Uhrmann and A. Schulte, “Task-based guidance of multiple uav using
cognitive automation,” in COGNITIVE 2011, The Third International
Conference on Advanced Cognitive Technologies and Applications, 2011,
pp. 47–52.
[4] W. Pecher, S. Bruggenwirth, and A. Schulte, “Using cognitive automa-
tion for aircraft general systems management,” in 5th International
Conference on System of Systems Engineering (SoSE). IEEE, 2010,
pp. 1–8.
[5] J. E. Laird, A. Newell, and P. S. Rosenbloom, “Soar : An architecture
for general intelligence,” in Artificial Intelligence, vol. 33. Heidelberg,
Germany: Springer, 1987, pp. 1–64.
[6] J. Rasmussen, “Skills, rules and knowledge, signals, signs and symbols,
and other distinctions in human performance models,” in IEEE Trans-
actions on Systems, Man, and Cybernetics, vol. SMC-13. Heidelberg,
Germany: Springer, 1983, pp. 257–266.
[7] R. Onken and A. Schulte, System-Ergonomic Design of Cognitive Au-
tomation: Dual-Mode Cognitive Design of Vehicle Guidance and Control
Work Systems. Springer, 2010.
[8] T. Johnson, “Control in act-r and soar,” in Proceedings of the Nineteenth
Annual Conference of the Cognitive Science Society, 1997, pp. 343–348.
[9] M. Wooldridge, An introduction to multiagent systems. Wiley, 2009.
[10] I. A. Ferguson, “Touringmachines: An architecture for dynamic, rational,
mobile agents,” Ph.D. dissertation, University of Cambridge, 1992.
[11] E. Gat et al., On three-layer architectures. Cambridge: AAAI Press,
1998.
[12] M. Bratman, Intention, plans, and practical reason. Harvard University
Press Cambridge, MA, 1987.
[13] F. Meneguzzi, A. Zorzo, and M. da Costa Mora, “Propositional planning
in bdi agents,” in Proceedings of the 2004 ACM symposium on Applied
computing, Nicosia, Cyprus, 2004, pp. 58–63.
[14] M. Ginsberg and D. Smith, “Reasoning about action i:: A possible worlds
approach,” in Artificial Intelligence, vol. 35, no. 2. Elsevier, 1988, pp.
165–195.
[15] R. Fikes and N. Nilsson, “STRIPS: A new approach to the application
of theorem proving to problem solving,” in Artificial intelligence, vol. 2,
no. 3-4. Elsevier, 1971, pp. 189–208.
[16] J. McCarthy and P. Hayes, Some philosophical problems from the
standpoint of artificial intelligence. Stanford University, 1968.
[17] M. Fox and D. Long, “Pddl2. 1: An extension to pddl for expressing tem-
poral planning domains,” in Journal of Artificial Intelligence Research,
vol. 20, no. 1, 2003, pp. 61–124.
[18] B. Williams and P. Pandurang Nayak, “A reactive planner for a model-
based executive,” in International Joint Conference on Artificial Intelli-
gence, vol. 15. Citeseer, 1997, pp. 1178–1185.
[19] M. R. Endsley and D. J. Garland, Situation Awareness Analysis and
Measurement. Mahwah, NJ: Lawrence Erlbaum Associates, 2000.
[20] S. Brueggenwirth, R. Strenzke, A. Matzner, and A. Schulte, “A generic
cognitive system architecture applied to the multi-uav flight guidance
domain,” in Proceedings of the International Conference on Artificial
Intelligence and Agents, Crawley, UK, 2010, pp. 292–298.
[21] A. Matzner, M. Minas, and A. Schulte, “Efficient graph matching
with application to cognitive automation,” in Applications of Graph
Transformations with Industrial Relevance. Berlin, Germany: Springer,
2008, pp. 297–312.
[22] T. Bylander, “The computational complexity of propositional STRIPS
planning,” in Artificial Intelligence, vol. 69, no. 1-2. Elsevier, 1994,
pp. 165–204.
[23] S. Clauss, S. Brueggenwirth, P. Aurich, A. Schulte, V. Dobrokhodov, and
I. Kaminer, “Design and Evaluation of a UAS combining Cognitive Au-
tomation and Optimal Control,” in Proceedings of Infotech@Aerospace.
The American Institute of Aeronautics and Astronautics, 2012.