2 views

Uploaded by Óscarillo Valdés

This article investigates causal relationships leading to emergence in an agent-based model of human behaviour. A new method based on nonlinear structural causality is formulated and practically demonstrated. The method is based on the concept of a causal partition of a model variable which quantifies the contribution of various factors to its numerical value. Causal partitions make it possible to judge the relative importance of contributing factors over crucial early periods in which the emergent behaviour of a system begins to form. They can also serve as the predictors of emergence. The time-evolution of their predictive power and its distribution among their components hint at the deeper causes of emergence and the possibilities to control it.

- Potochnik - 2014- Causal Patterns and Adequate Explanations
- 77
- Where Explanation Ends Understanding As
- AR_FINAL
- National Study on Neck Restraints in Policing - Technical Report - Canadian Police Research Centre
- Multiple Regression in Small-N Comparisons
- soc-sci-app (1)
- ProcessAnalysis
- Ibp Yp Key Concepts Zebra the Me
- Chap 7 Patterns
- A Study in Karma - A Besant
- caraka nyaya
- How to Read Critically
- Statistics Assignment
- Academic Pharase Bank Uni of Mancehster
- Bridges--Ecology of Reasons Talk for Berlin and Leipzig
- student work analysismg
- english language
- Summary 1 and 2 Approaches
- bbdlm3 5320

You are on page 1of 19

Complexity

Volume 2017, Article ID 8381954, 18 pages

https://doi.org/10.1155/2017/8381954

Research Article

Causal Analysis of an Agent-Based Model of Human Behaviour

1

Ustav Informatiky SAV, Dubravska Cesta 9, 84507 Bratislava, Slovakia

2

Airbus Defence and Space GmbH, Rechliner Strasse, 85077 Manching, Germany

Copyright 2017 Marcel Kvassay et al. This is an open access article distributed under the Creative Commons Attribution License,

which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

This article investigates causal relationships leading to emergence in an agent-based model of human behaviour. A new method

based on nonlinear structural causality is formulated and practically demonstrated. The method is based on the concept of a causal

partition of a model variable which quantifies the contribution of various factors to its numerical value. Causal partitions make it

possible to judge the relative importance of contributing factors over crucial early periods in which the emergent behaviour of a

system begins to form. They can also serve as the predictors of emergence. The time-evolution of their predictive power and its

distribution among their components hint at the deeper causes of emergence and the possibilities to control it.

indirectly serve as an auxiliary model validation tool, for

When we study a complex system whose model equations example, for operational validation in the sense of Sargent [1]

cannot be solved analytically, we typically turn to simulations. or Louie and Carley [2].

Social systems are investigated in this way too, especially their In general, emergent behaviour can be very difficult to

emergent behaviours. The preferred and natural approach explain, and we still lack powerful and general methods for

is then agent-based modelling. In agent-based models, even the purpose. The European Agent Technology Roadmap [3]

relatively simple rules governing individual agents tend to states that understanding the mechanisms that can be used

produce unexpected patterns of global behaviour. The phe- to model, assess and engineer self-organisation and emer-

nomenon of emergence poses two main challenges: gence in multi-agent systems is an issue of major interest.

(i) Validation challenge: how to ascertain whether the Similarly, the UK governmental report [4] claims that the

phenomenon observed in the simulation model can difficulty in forming rigorous causal characterisations of

also occur in the real system the aggregate behaviour of a complex system (rather than

the absence of regularity or predictability in this aggregate

(ii) Explanatory challenge: how to identify its causes and

behaviour) . . .is the more legitimate barrier to adopting

stages of manifestation

complex-systems approaches in an ICT engineering context.

In this article, we focus on the explanatory challenge. In this article, we do not attempt to tackle the problem

As an aside we may mention that the two challenges are in in its full generality; instead, we try to show how to build

fact interrelated since explanation is often a prerequisite to narrower methods tailored to the specifics of the system or

validation. Thus an explanation might reveal that a partic- model at hand on the basis of nonlinear structural theory of

ular phenomenon was caused by the simplifications of the causation formulated by Judea Pearl and other researchers [5

simulation model and should not occur in the real system 7]. In doing so, we extend and conclude our earlier attempts to

(rather the simulation model should be refined). In other apply structural causality to an agent-based model of human

cases, the explanation might suggest experimental conditions behaviour [8].

under which the emergent phenomenon could be observed in Broadly speaking, if we already have a theoretical descrip-

the real system too. For this reason, although our proposed tion of an agent-based model (ABM) and its executable

2 Complexity

(ABS), our approach consists of four steps: Looted

shop

(1) Model analysis: ABM equations are analysed from the

point of view of structural causality in order to define

additional causal analytical variables (CAVs) that

need to be calculated and logged during the simula-

tions. As we explain later on, each CAV quantifies the

effect of one causal factor on a given model variable

and represents one component of its causal partition Aggressive

individuals Looters

(2) Implementation and data provision: the algorithms for

computing and logging the CAVs are implemented in

the ABS, and appropriate simulation experiments are

Security patrol

then defined and executed

(3) Data analysis: logs of the simulation runs deemed

relevant for causal analysis are analysed by suitable

machine learning techniques (clustering, classifica-

tion, etc.) using CAVs as predictors in order to Figure 1: Initial setting of the simulation scenario.

generate and test hypotheses about the causes and

stages of the observed emergent behaviours

(4) Hypothesis validation by new simulation experiments:

It is surrounded by dots, each representing one agent. The

in this phase, we go beyond statistical hypothesis

dark ones are the looters; the white ones are the violence-

testing by taking advantage of the fact that simula-

prone individuals whose intention is to attack the soldiers.

tion models are fully observable and manipulable by

The soldiers are represented by the three medium gray dots

setting their initial conditions and parameters and,

in the bottom part of the figure.

if necessary, by modifying their software implemen-

Civilian agents are endowed with one default motive

tation. Thus we can ultimately prove or disprove

and a matching behaviour by which they try to satisfy it.

our causal hypotheses by directly manipulating the

For looters, this leads to looting and for the violence-prone

suspected causes and observing the effects of our

individuals to stone-pelting the soldiers. The agents also

interventions on the emergent behaviour of the sim-

monitor their surroundings. As the patrol approaches, this

ulated system.

may induce fear in some looters who then start leaving the

In the remainder of this article, we walk through this scene. The violence-prone individuals, however, do not get

process focusing mainly on steps (1), (3), and (4). We start by afraid but rather attack the patrol. The violence may impact

describing our agent-based model and scenario in Section 2. the remaining looters in two possible ways: they may either

In Section 3, we introduce structural causality and the con- get afraid and leave or get angry and join the attack. The ratio

cept of a causal partition. In Section 4, we incrementally of looters who get afraid to those who get angry depends on

develop the method of causal partitions in the context of our their motivational dynamics, which we explain next.

agent-based model. This section corresponds to the model A simplified diagram of the key factors affecting the

analysis step of the above process. In Section 5, we then behaviour of our civilian agents is shown in Figure 2. The

analyse the results of our simulation experiments, generate model draws primarily on Berkowitz [10], Prentice-Dunn and

hypotheses, and validate them by additional experiments. Rogers [11], Staub [12], and Canamero [13]. The main ideas

and processes underlying the model were developed by Air-

2. The Model and the Scenario bus Defence and Space (former EADS Deutschland GmbH)

(https://airbusdefenceandspace.com/) in collaboration with

Both the model and the scenario come from project EUSAS the Department of Social Psychology of the University of

(European Urban Simulation for Asymmetric Scenarios) Zurich and the chair for Operations Research at the Univer-

[9] financed by 20 nations under the Joint Investment Program sity of Passau on behalf of the German Bundeswehr.

Force Protection of the European Defence Agency (EDA). As depicted in Figure 2 (starting from the top left corner),

The project dealt with asymmetric security threats in which the number of people surrounding the agent, their actions,

security forces face rioting crowds, insurgents, or terror- and other events in the vicinity affect the agents emotional

ists rather than regular military forces. One such example, motives (fear, anger) and other internal variables (arousal,

which provided context for our simulation scenario, was the readiness for aggression). Besides events and actions, there is

peacekeeping ISAF mission in Afghanistan. In the scenario, also a direct social influence of other agents on the agents

a crowd of civilians is looting a shop and an approaching fear and anger. This was modelled according to Latanes

soldier patrol is supposed to stop the looting and disperse formula of strength, physical proximity, and the number of

the crowd. The scene is depicted in Figure 1. The black areas influencing agents [14].

represent buildings and barriers unreachable to agents. The Speaking qualitatively, the agents internal arousal

rectangle with gray interior near the top is the looted shop. depends on the number of people in the vicinity and their

Complexity 3

surrounding events &

people influence

actions

Arousal

Anger Fear

aggression

Other motives:

Feeling of Norms for looting,

Behaviour

group antiaggression will to attack, . . .

cohesion

Legend

External Internal

factors variables Motives Behaviour

violence: the higher the number and the more violent associated state transition functions conforming to general

they are, the sharper the increase of the agents arousal. systems theory.

Deindividuation means that the agent considers himself a The PECS also requires that some state variables produce

part of the crowd and no longer a separate individual: the or act as motives, that is, forces that drive agents to action. A

higher the agents arousal and the cohesion of his group, good example is the level of physical energy (a state variable

the higher the deindividuation. Readiness for aggression in the physical conditions slot) and hunger, a motive driving

(RFA) is jointly affected by the norms for antiaggression, us to search for food and replenish the energy. Our model

deindividuation, and external events as follows: (a) the includes four state variables that act directly as motives: fear,

higher the norms for antiaggression, the lower the RFA; (b) anger, looting motive (present only in looters), and will to

the higher the deindividuation, the higher the RFA; and attack (present only in violence-prone civilians). In the PECS

(c) the more violent actions are witnessed, the higher the model, the motives compete for control over the behaviour

RFA is. The form of the agents eventual aggression depends of the agent: the strongest wins and becomes action-guiding.

primarily on the RFA. In our scenario, the initial value of The motives thus need to be mutually comparable, which we

RFA was set to a level guaranteeing that civilians would have achieved by normalizing and restricting their values to

resort to stone-pelting the soldiers whenever they became the closed interval [0, 1]. Unlike fear and anger, which were

aggressive. endowed with complex dynamics described below, the looting

Regarding the agent-based architecture suitable for motive and will to attack were both set to constant values

implementation, project EUSAS opted for the PECS reference which fear and anger had to cross in order to affect the agents

model [15, 16]. The PECS acronym stands for Physical behaviour.

conditions, Emotional state, Cognitive capabilities, and Social Each motive, when it becomes action-guiding, pres-

status, the four kinds of internal factors that need to be elects a group of behaviours that can satisfy it. In our

modelled in order to achieve realistic agent behaviour. At model, for example, there are three fearful behaviours:

the same time, the PECS model only provides these four withdrawal (walking away), flight (running away), and panic

empty slots without specifying the factors to be modelled or flight (running away at extra speed with sensory perception

the level of modelling detail: these decisions are left to the blocked). Which of them is triggered when fear becomes

modeller as they depend on the task at hand. The PECS simply action-guiding depends on a secondary selection criterion,

requires that these factors be modelled as state variables with in this case the actual intensity of fear. This criterion can

4 Complexity

Table 1: Fear- and anger-related constants. Social influence of nearby agents on a motive variable

(where stands for either fear or anger ) of an

Constant Value Interpretation observing agent is defined by the following sum:

Sensitivity of the first derivative of fear

1 5

/. ( ) prestige sympathy

Maximum value of fear at which / = 4 [ ]. (3)

2 1.1 distance

becomes zero. =

[ ]

Inbuilt tendency of fear to increase, a

composite constant defined as the Here, the summation is over those agents (indexed by the

3 0.1 difference between the effect of the subscript ) who are not farther away from agent than a

expected negative consequences and the certain social influence radius (set to 100 metres for both fear

agents resilience to fear. and anger) and in whom the motive happens to be action-

Sensitivity to fear-related social influence guiding at the moment of evaluation. Each agent is assigned a

4 12.5 constant social rank prestige which modulates its influence

of other agents.

Sensitivity to fear-inducing events (see on others: group leaders enjoy higher prestige than ordinary

5 1 members and thus influence the others more. Analogously,

Table 2).

Sensitivity of the first derivative of anger there are constant sympathies assigned between groups:

1 2 sympathy captures the sympathy of agent s group toward

/.

Maximum value of anger at which agent s group and is interpreted here as the susceptibility of

2 1.1

/ becomes zero. the former to the influence of the latter. Finally, distance is

3 0.3 Resilience to anger. the physical distance between the two agents. In the original

Sensitivity to anger-related social model, variables and many constants were expressed in the

4 12.5 percentage scale [0, 100]. For the sake of simplicity, in this

influence of other agents.

Sensitivity to anger-inducing events (see paper, we have converted them into the ratio scale [0, 1] (see

5 1 Table 1).

Table 2).

During the simulation, differential equations (1) and (2)

for each agent are solved numerically by the Euler method

with a constant (but user-definable) time step . Resorting

to numerical method enabled us to ignore the specific form of

be arbitrary; for example, while anger preselects a group the equations right-hand side and consider the general case

of aggressive behaviours, the final choice of the form of

aggression depends on the readiness for aggression (RFA). As

= (, ) , (4a)

we have already mentioned, we set RFA so that aggressive

civilians would always resort to stone-pelting the soldiers.

In line with the PECS modelling methodology, agent = (, , ) , (4b)

behaviours are conceptualised as sequences of atomic, unin-

terruptible elementary actions, for example, one step in a where , are bounded (but not necessarily continuous)

certain direction or one stone-throw. When a new motive nonlinear functions. The Euler method approximates the new

becomes action-guiding, it only takes effect after the current values of fear ( + ) and anger ( + ) on the basis of the

elementary action is completed: the current behaviour pat- current ones (), ():

tern is then cancelled and a new one is activated.

As for the dynamics of the simulated emotions fear ( + ) () + ( () , ()) , (5a)

and anger, they comprise a continuous part and a discrete

part. The continuous dynamics of fear () is driven by the ( + ) () + ( () , () , ()) . (5b)

differential equation After calculating these new continuous values, discrete

dynamics come into play: the cumulative effects of the

= 1 (2 ) (3 + ) , (1) perceived external events on fear ( ) and anger () are

added in order to obtain the new total values:

where 1 , 2 , 3 are fear-related constants (see Table 1) and ( + ) = ( + ) + 5 , (6a)

is fear-related social influence of nearby agents. Analo-

gously, the continuous dynamics of anger () is driven by the ( + ) = ( + ) + 5 . (6b)

equation

In the next iteration, the new total values of fear

and anger will be used as initial conditions in the

= 1 (2 ) ( + 3 ) , (2) numerical solution of the differential equations representing

their continuous dynamics. This sequential coupling of the

where 1 , 2 , 3 are anger-related constants (see Table 1), continuous and the discrete dynamics qualifies our agent

is anger-related social influence of nearby agents, and is the models as sequential hybrid in the sense of Swinerd and

model variable arousal. McNaught [17].

Complexity 5

Table 2: Main event impacts on fear and anger. case of fear, for example, we want to know what portion of

its actual value at any time should be attributed to the social

Impact on fear Impact on anger influence of nearby agents (variable in (5a)) as opposed to

Event

Direct Indirect Direct Indirect the direct impact of external events (variable in (6a)). We

Effective shot 0.4 0.35 0.1 0.25 tackle this question in the sections that follow.

Warning shot 0.3 0.3 0.1 0.1 In contrast to our civilian agents, the soldier patrol

Stone thrown 0.002 0.002 0.18 0.15 characters were intended primarily as avatars to be controlled

by real people in a high-fidelity 3D cyber-environment of a

commercial battlefield simulator VBS2 (http://www.army-

technology.com/contractors/training/bohemia-interactive/).

The constants 5 , 5 (see Table 1) capture the agents In consequence, our soldier agents were not defined at the

individual sensitivity, while and are the sums of the same level of detail as the civilian ones. In our scenario, for

emotion-inducing impacts as per Table 2 for all the events example, they are just passing by and act in self-defence.

perceived by the agent during the time interval (, + ). Their rules of self-defence say that when a given civilian first

The direct values from Table 2 are used when the perceiving throws a stone at a particular soldier, that soldier responds by

agent is within 20% of the maximum perception distance. a warning shot in the air. If the same civilian throws a stone

When the agent is farther away than 40% of this maximum at the same soldier a second time, that soldier is permitted

distance, the indirect values are used. In the intermediate to use an effective shot aimed at the legs of the attacker

zone (from 20% to 40%), a weighted average is used, sliding in order to immobilize him. That is, of course, an extreme

down linearly from the direct value toward the indirect one. simplification, but it proved useful in the early phases of

Sensory perception of the agents is limited by a radius of 50 m project EUSAS for calibrating the civilian agents.

for events like throwing stones and 150 m for gun shots. While experimenting with this model and scenario, we

Besides the constants in Table 1 and the event impacts noticed that, for the parameter setting listed above, the emer-

in Table 2, emotional dynamics is greatly influenced by the gent collective behaviour of the civilians seemed to bifurcate

initial values of fear 0 and anger 0 . In our scenario, these along two different trajectories. In some cases, almost all

were set to 0 = 0.3 and 0 = 0.2. Additionally, our agents the looters got afraid and left the scene, while in others

underwent the moderating influence of emotions and fatigue: almost all got angry and joined the attack. Because our model

when the average of fear and anger (termed in our model incorporated an element of randomness, some variation in

the emotional moderator) crossed the level of 0.5, further its behaviour was expected, but the extreme variation we

sensory perception of external events was blocked. Moreover, witnessed was unusual and called for an explanation. This

as the physical energy of our agents decreased with expended spurred our search for analytical methods that could unravel

effort and sustained injuries, they slowed down and their causal chains and dependency in complex systems of this

actions took longer. kind. The method of causal partitions presented below is the

Like all models, this one too is no more than a useful result.

simplification of the enormously complex human psyche. Its

intended use was for virtual training of security personnel in 3. Structural Causality and Causal Partitioning

project EUSAS; therefore its equations were formulated in a

deterministic fashion so as to lead to reasonably predictable Structural causality starts from the concept of a structural

agent behaviour. Some measure of variation in its behaviour equation. In order to illustrate it, let us consider two simple

was subsequently achieved by randomizing agents initial electrical circuits shown in Figure 3. Both include a variable

positions and the duration of their elementary actions. resistor connected to an ideal source of (a) voltage or (b)

Practitioners of System Dynamics (SD) will have current.

undoubtedly noticed that we have modelled agent motives The relationship between the current passing through

by the same kind of equations as those used in SD. Here, the resistor and the voltage on its terminals depends on

of course, the context is different: our equations describe its resistance and conforms to Ohms law, which can be

an inside of a single agent, whereas in SD such equations expressed in many nearly equivalent ways:

would typically be used to model the whole agent community

without bothering to model its constituent individuals or = ;

their low-level interactions. Despite this difference in

modelling focus, the similarity of the equations used and = ;

the emphasis that SD puts on tracing the flow of causality

through the modelled system make it very likely that our (7)

= ;

method of causal partitions could be profitably employed

in SD context. In Conclusion, we therefore provide some

practical hints in this regard. = .

Numerical solution of this model by iterating through

(5a) and (5b) and (6a) and (6b) can give us a complete and From the algebraic point of view, all these formulations

detailed time-evolution of simulated fear and anger of civilian are permissible and any of them could be said to apply

agents. Our present goal, however, is more ambitious. In the to both circuits. Structural causal theory, by adding extra

6 Complexity

+

r1 r1

u0 i0

r2 r2

(a) (b)

rules governing the form of equations, manages to encode defend this model as realistic; it serves merely to illustrate the

in them additional information about the flow of causality. principles of structural causality and our method of causal

Essentially, it treats the equality sign as an assignment partitions.

operator in programming languages. Thus, on the left-hand The variables that occur on the left-hand side of structural

side of a structural equation, there can only be one variable. equations are called endogenous, that is, determined by the

Moreover, this variable has to be genuinely dependent on model. In our example, these are {, , }. The remaining

the right-hand side, which is meant to capture the causal variables {, } are exogenous: in their case, we are not inter-

mechanism determining (or assigning) its value in the ested in modelling the mechanisms that set their values and

system under investigation. Interpreted in this way, each of simply consider their values as given. The value assignment

the two circuits can only be represented by one form of Ohms to the exogenous variables, for example ( = 1 , = 1 ), is

law. In order to identify the correct structural forms, we called the background or context in which we try to solve the

need to contemplate the effect of an intervention by an model equations.

experimenter: what happens if he or she changes the value In general, according to Pearl [5], causal analysis can be

of the resistor from 1 to 2 ? We can immediately see that for applied to systems that are described by structural equations

(a) it is the current that changes, while for (b) it is voltage. of the form

Thus the right structural equations describing the circuits are

= ( , ) , = 1, . . . , , (9)

(a) = / and (b) = .

Let us now consider a slightly modified version of where stands for the set of parent variables of

Figure 3(b): instead of an ideal current source we might (i.e., endogenous variables directly determining the value

have a photovoltaic cell producing current dependent on the of through an autonomous causal mechanism captured

intensity of light (), and our resistor might be a thermistor by ) and represents the set of background variables.

whose resistance depends on temperature (). A simplified The autonomy of the mechanisms (and, consequently, of

structural model of such a circuit might look as follows: the equations) means that it should be possible to change

any of them by external intervention without affecting the

= () , (8a) remaining ones [18]. A set of such equations is called a

= () , (8b) structural model.

In structural theory, causation is interpreted as a relation

= , (8c) between events. A primitive event is defined as a model

variable assuming a value from its permitted range; for

where , are unspecified nonlinear functions. By including example, = 1 [6]. More complex events can be expressed

only these equations in the model we stipulate that there is as Boolean combinations of primitive events. We normally

no other significant dependency among the model variables, speak of an event when something happens as a consequence

at least in the space of parameter values under consideration. of model equations; for example, in our case, the event = 1

Thus, for example, we assume that the current produced by would imply that the temperature assumed a value 1 such

the photovoltaic cell does not depend on the resistance , a that (1 ) is evaluated to 1 . Causal thinking also requires

condition which probably holds just approximately and only a special kind of event called intervention (sometimes also

for a certain range of resistance values. We also assume that action): this is when we intervene from outside and impose

the temperature of our thermistor does not depend either the value assignment = 1 regardless of the temperature

on the intensity of light or on the current passing through or the causal mechanism (), for example, by replacing the

it; otherwise we should have represented this dependence thermistor with a normal resistor whose resistance equals

explicitly by a dedicated equation. It is not our intention to 1 . An intervention means that we wipe out the affected

Complexity 7

structural equation from the model and replace it with Thus, sustenance is (trivially) satisfied too. Last, = 1 =

another (typically a straightforward value substitution). The (1 ) is minimal since it involves only one model variable.

solution of this modified system of equations represents the Therefore, it qualifies as the actual cause. Moreover, we could

response of the model to the intervention. In our case, (8b) have equally easily demonstrated that = 1 = (1 ) also is

would be replaced with = 1 , giving the model response an actual cause of = (1 ) (1 ). This ease amounting

= (1 ) 1 . to triviality in fact signals a problem: applying structural

Structural approach to causality enables us to inquire causality in this form to models with real-valued variables is

whether one event ( = ) is a cause of another ( = ) not going to be very useful or instructive. Typically, all the

in a given context = . Various kinds of causes have been parent variables will be identified as actual causes of their

proposed in the literature, but we shall focus here on the dependent or child variable assuming its observed value: a

notion of actual cause as defined by Halpern and Pearl in result no doubt correct but not particularly enlightening. This

[6]. Informally, = is an actual cause of = if the stands in sharp contrast to highly nontrivial results obtained

following conditions hold (in general, , , can represent in models with binary or categorical variables, as documented

subsets of model variables. If = { | = 1, . . . , }, then by the many interesting examples in [6].

= denotes the logical conjunction of primitive events In our previous work [8], we suggested how to extend

=1 = ): the structural approach in order to cope with real-valued

(i) Actuality: both = and = are true (observed) variables. Instead of asking, What is the cause? we proposed

in the model in the context = to ask a modified question: In what proportion have all

the causes contributed to the effect? Continuing with our

(ii) Dependence: an intervention on changing its value example, we should try to determine the proportion in which

from to some other ( = ) must result in the change of from some previous level 0 to the present

a change of the value of from to some other

one 1 could be attributed to (or split among) its parent

( = ). This needs to be demonstrated for at least

variables and . Let us assume that = 0 was observed

one setting = of a suitable subset of the remain-

at time 0 together with = 0 , = 0 in the context

ing endogenous variables, which is to be imposed

= 0 = ( = 0 , = 0 ). For the sake of simplicity, let

through V

us further assume that our two observations were so close

(iii) Sustenance: the intervention = must insulate the in time that the state-space trajectory of the system between

event = from a restricted class of interventions them can be considered linear. In such a case, we can

on the value of . Specifically, if any subset of the approximate the change of by a scalar product of two

variables in is made to flip between the values vectors. The first is the gradient of along its parent variables

and ( being the original value of observed according to its structural equation (8c). The second is the

under the context = before the intervention = vector form of the change in these parent variables between

), this must not have any effect on the value of , so the two observations (contexts = 0 and = 1 ):

long as remains set to

(iv) Minimality: is minimal; no proper subset of grad () (, ) (10a)

satisfies the above conditions

This informal rendering of Halpern and Pearls definition ( , ) (, ) (10b)

is not entirely precise but suffices for our purposes (for

rigorous definition, please refer to [6] or [5]).

Armed with this definition, we can analyse our circuit + . (10c)

model. For example, we can inquire whether the event =

1 = (1 ) qualifies as an actual cause of event = (1 )

(1 ) observed at time 1 in the context = 1 = ( = Equation (10c) can be interpreted as a general recipe for

1 , = 1 ). The first condition, actuality, is satisfied, since quantifying the responsibility of the parent variables , for

both events hold. In other words, we obtain them as a solution the change of their dependent variable : the first summand

to the model equations ((8a), (8b), and (8c)) in this context. on the right-hand side ( /) captures the contribu-

Assuming that all our variables are real-valued and positive, tion of and the second one ( /) captures that

dependence is also easily demonstrated: it is trivial to show of .

that, for example, halving the value of would halve the Suppose that we trace further evolution of at times

value of . The fact that we could demonstrate this without 2 , . . . , as it assumes new values 2 , . . . , along some

intervening on the remaining endogenous variable makes trajectory representing our observational study or experi-

the sustenance condition also trivially satisfied: if we include ment. Expressing each of these changes of as per (10c)

in , then the value (corresponding to = 1 = (1 ) and summing up the contributions of separately from

observed in the context = 1 ) is the same as the value those of , we obtain a general formula quantifying the total

for which dependence was demonstrated. Flipping between contribution of each causal factor (parent variable) along

and then means that nothing actually changes, so the an arbitrary trajectory (implicitly approximated here by a

observed value of = (1 ) (1 ) cannot change either. piecewise linear curve):

8 Complexity

() ( () () ) (0 , ( () () ) ,

=1 =1 = () =1 = ()

=

=

(11) (14)

+ ( () () ) . ( () () )) .

= () =1 = ()

=1

=

=

Note that in this example the contribution of the initial

The above formula uses superscript indexing, where () value 0 remained constant throughout the trajectory; that

denotes the value of variable at time and () denotes its ()

is, 0 = 0 , = 1, . . . , . In general, however, this contri-

backward difference: () = () (1) . Partial derivatives bution can vary, as we shall see for simulated fear in our

()

are evaluated at midpoint ( , () ) of each linear segment of agent-based model of human behaviour.

()

the trajectory, so = 0.5(() +(1) ) and () = 0.5(() +

(1) ).

4. Causal Partitioning in Practice

In order to keep the total contribution of each causal The notion of a causal partition of a model variable intro-

factor separate, we proposed a new, vector-like representation duced in the previous section is the core of our new analytical

of model variables termed a causal partition. Let us consider method. Its development was in fact a long-term effort

variable driven by structural equation = ( , = 1, reported step by step in several publications, of which the

. . . , ), where { , = 1, . . . , } are its parent variables. most recent one is [8]. We present the derivation comprehen-

Let us assume that starts from = 0 and proceeds sively here along with some as yet unpublished improvements

through = , = 1, . . . , . Its causal partition vector and results.

at the end of this trajectory ( = ) is then expressed as

(0 , 1 , 2 , . . . , ), where each partition component 4.1. Basic Version of Causal Partitioning. Before applying

stands for the total contribution of the corresponding structural causality to our human behaviour model we need

parent variable along this trajectory and is calculated as to ascertain that it meets the criteria set for structural models.

In Appendix A, we demonstrate that when ordinary nonlin-

ear differential equations used in our model are converted

into difference equations (so as to solve them numerically

()

= ( () ) through simulation), they at the same time become struc-

=1 1 =1

tural.

..

. Let us now focus on the general form of differential

= ()

equation driving the continuous dynamics of fear of our

(12)

agents (4a). As explained with regard to (10a), (10b), and (10c)

in Section 3, function on the right-hand side of (4a) can be

() linearized on each discretized time interval through the scalar

= ( () ) ,

=1 1 =1

product of its gradient (/, / ) and the vector form of

..

.

the change in its parameters (, ):

()

=

+ , (15)

where ( () , . . . ,

() ) is the midpoint of the th linear

1

segment of the trajectory. where , , stand, respectively, for the difference in

the values of , , between the start and the end of the

The first partition component 0 has a special role: it

time interval under consideration. This equation enables us

denotes the contribution of the initial setting = 0 to

to determine the proportion in which the change in the value

the final value = . We introduced it in order to force

of can be attributed to the influence of the changes in

the partition components to sum up to the value of the repre-

its parameters , , which we interpret as their elementary

sented variable, which we found very helpful for interpreting

causal effect on . By summing up these elementary causal

causal partitions:

effects separately for each factor, we can partition the value

of at any moment into a sum of total contributions per

= 0 + ( ) . (13) factor:

=1

= + , (16)

This approach implies that, at the beginning of the where is the total contribution of fear to the value of

trajectory, is represented by the causal partition vector (summed since the start of the simulation) and is the total

(0 , 0, 0, . . . , 0). contribution of social influence . Substituting this partition

Going back to the example, variable at the end of into (5a), we get

our observational study would be represented by the causal

partition vector: ( + ) () + + . (17)

Complexity 9

If we further substitute (17) into (6a), we get 4.2.2. Nonzero Initial Conditions and Different Types of Events.

In all the versions of our method introduced so far we have

( + ) () + + + 5 (18) implicitly assumed zero initial value of fear (0 = (0) = 0).

In our simulation experiments, which we present in the next

which can be rewritten as section, this initial value was positive (0 > 0) and strongly

influenced simulation results. Its causal effects therefore need

( + ) () + + + . (19) to be taken into account. As explained in Section 3, this

can be achieved through a dedicated partition component,

This means the new total value of fear can be interpreted which we propose to denote by 0 for fear and 0 for

as the old one plus the contributions of its three causal its first derivative. The hybrid nature of our model (i.e., its

factors. By summing up these contributions separately since combination of the continuous and the discrete dynamics)

the start of the simulation, we can, analogously to (16), enables us to view the setting of the initial value of fear as

partition the value of at any moment into a sum of total a special kind of event which takes effect at time 0 = 0 just

contributions per factor: before the simulation starts. This amounts to conceptualising

the components 0 , 0 as being split off from the regular

= + + . (20) event components , .

As with the initialising event, we can also differenti-

The right-hand side of (20), written in a vector-like ate other kinds of events. For example, in our simulation

form ( , , ), represents the basic causal partition of fear experiments, we distinguish between the events (actions) of

as per the original version of our method introduced in [19]. civilians and the events (actions) of security forces. This is

We interpreted its first component as the extent to which easily possible since the cumulative event impact on fear

fear could be considered self-propelled. is defined as a straightforward sum of individual event

impacts which do not interact in any way:

4.2. Improved Versions of Causal Partitioning. Having worked

with the basic version of our method for some time, we = impact ( ) , (22)

realised that the concept of self-propelling behaviour was

problematic and complicated the interpretation of causal par-

titions. Eventually, we found a way to eliminate it. We present where impact is a function mapping each event perceived

the derivation of this enhanced version in Section B.1 of by the agent during a given simulation step to its numerical

Appendix B. In this version, the first derivative of fear is impact on fear . This function is defined in a tabular form

represented by the causal partition vector ( , ) and fear in Table 2. Events are results of agent actions, and for each

by the partition ( , ). Partition components indexed action we know whether its originator is a civilian or a

by represent the contribution of external events, while security staff, so we can sum the impact of civilian actions

those indexed by stand for the contribution of social () separately from that of security actions ( ). If

influence. we then allocate a dedicated partition component to each

type of event, we end up with fear being represented by a

4.2.1. Dependency among Causal Factors. So far we have causal partition (0 , , , ) and its first time derivative

treated fear and social influence as if they were indepen- by a causal partition (0 , , , ). The full derivation

dent. This assumption was in fact implied in the way we used of recurrence relations for each partition component is

(15) for separating the effect of fear on from that of . presented in Section B.3 of Appendix B.

In consequence, both the basic and the enhanced methods At this point it should be clear that the number of causal

silently stipulate that a change of fear has no effect on social partition components is not fixed in advance but rather

influence . But our model does not meet this requirement: depends on the problem at hand. As a matter of fact the above

by formula (3), is in fact a function of and of fear of all the structure turned out to be sufficient for our purposes, but

influencing agents (note that fear corresponds to variable this was by no means guaranteed. Rather, it was a matter of

in (3)). Because all the terms on the right-hand side of (3) are trial and error. If it were to be found insufficient, we would

nonnegative, increasing will simultaneously decrease . In have tried to get additional information by further splits.

order to account for this indirect effect of on through , For example, the civilians in our scenario were of several

the notion of a total derivative needs to be employed: types, so we might have tried to separate the event impact for

each civilian type. This would amount to splitting into

F 1 , 2 , . . . and into 1 , 2 , . . . as necessary. We

= + . (21) might have also likewise split social influence : individual

contributions in its formula (3) do not interact in any way

This total derivative represents the true (or total) sensi- (they are just summed up), so the splitting would again be

tivity of to changes in regardless of whether they are fairly straightforward.

direct (the first term on the right-hand side) or indirect (the This concludes our theoretical development of causal

second term). Based on this insight we can further improve partitioning in the context of the simulated emotion of fear

our method; the corresponding mathematical derivation is of our civilian agents. It enables us to express at any

presented in Section B.2 of Appendix B. moment as a vector-like structure (0 , , , ) whose

10 Complexity

Table 3: Predictive accuracy of partition components and MoE (in %) at the 90th second of simulated time.

Data set 0 0

100 ms 88.7 61.0 93.0 51.7 99.0 97.7 80.3 97.0 95.3 88.7

300 ms 80.0 67.7 48.7 60.3 97.0 93.0 91.0 98.0 93.0 92.0

components sum up to while at the same time isolating the was tracked by 0 , 0 . We also calculated five other relevant

portions of its value attributable to each causal factor. attributes:

Along similar lines we have developed and implemented (i) (number of effective shots)

a causal partitioning procedure for the simulated emotion

of anger of our agents. Their anger is driven by (2) (ii) (number of warning shots)

and the procedure partitions it into a vector-like structure (iii) (number of stones thrown)

( 0 , , , , ). Since the equation driving anger (iv) A-count (number of times that anger became action-

is similar to that driving fear, the only really new partition guiding in some civilian agent)

component is , which captures the effect of a new variable

(termed Arousal) in (2). At this point, then, we have at (v) F-count (number of times that fear became action-

our disposal two causal partitions through which we can guiding in some civilian agent)

analyse and interpret the emergent behaviour of our agent- , , are so-called measures of effectiveness (MoE)

based model. This analysis, however, requires real simulation that were used to evaluate scenarios in project EUSAS: for

experiments and data, which we present below. scenarios that turned aggressive we expected high MoE and

high A-count, while for the timid ones we expected low

5. Experimentation and Validation MoE and high F-count.

of Hypotheses We included MoE as a sort of competition to our

causal partitions. It was evident that MoE could classify the

This section describes the application of our method to the scenarios well, since aggressive developments implied high

data from simulations of our agent-based model and scenario. numbers of stones thrown as well as of gunshots. MoE,

We start by briefly summarizing our early experiments and however, lacked the explanatory power: they could not tell us

hypotheses in Section 5.1. These were already reported in anything about why a particular scenario turned aggressive

detail in [8]. Practical experience gained in this early phase or timid.

enabled us to improve our method, rerun the simulations, An initial clustering exercise in two-dimensional space

and ultimately identify the cause of the surprising emergent (A-count, F-count) revealed two distinct clusters (timid

behaviour of our model. We present these later activities in versus aggressive) as expected. The shape of the clusters did

Section 5.2. All data mining and machine learning experi- not appreciably depend on the length of the time step ,

ments were performed in Weka [20], version 3.7.9, which used which allowed us to rule out the discretization and rounding

EM (expectation maximization) algorithm. errors as a significant factor. The clustering supplemented our

data with a new attribute assigning each simulation into one

5.1. Early Experiments and Hypotheses. The set of early of the two clusters.

experiments consisted of 300 runs of our scenario with the We then trained several SVM classification models on

time step = 300 ms and 300 runs with the time step the scaled data from our data sets, choosing the cluster

= 100 ms. This enabled us to gauge the effect of the time assignment as the target class. When we examined the

step size (and of the resulting discretization and rounding predictive power of individual partition components, we were

errors) on the observed emergent behaviour. Since the time- surprised to find that 0 possessed the highest prediction

evolution of our scenario was rather fast, it was sufficient for accuracy (see Table 3). It was tempting to postulate the initial

each simulation to cover just 90 seconds of simulated time. value of fear 0 as the underlying cause, but we knew this

At the end of this period, the average values of fear and anger could not be, since we had kept it constant for all the 600

were recorded and their causal partition vectors were passed simulations. Its predictive power must have stemmed from

on to machine learning algorithms for further analysis. other factors, most likely the other two partition components

In this early phase, we have not yet distinguished between to which it was tied by the constraint = 0 + + .

the civilian and the security event impacts, so the final Judging by their accuracy, appeared to be more significant

average values of fear and anger were represented by the than .

partitions High predictive accuracy of these simple models proved

that causal partitions were relevant to the investigated emer-

= (0 , , ) , gent phenomenon. The question of their practical utility

(23)

= ( 0 , , , ) . remained unresolved, because by practical utility we meant

their ability to guide us toward that aspect of the model which,

Partition components , recorded the effect of all if modified, would suppress the bifurcation of simulation tra-

external events apart from the initialising one, whose effect jectories. We did not expect our method to directly compute

Complexity 11

the answer but rather assist us in the process of formulating element of randomness played no role (in fact it had to

and testing hypotheses. Toward this end, we first needed because without it all the simulations would have yielded an

to identify and interpret the key factors behind the good identical result) but that there were likely other deterministic

performance of our causal classifiers. The results presented factors amplifying and stabilizing the bifurcation.

above led us to conclude that the most important factors Our fourth hypothesis was that the external events and

appeared to be, first, external events acting through fear social influence acted together, perhaps as part of a two-

( ), followed by social influence acting through both anger stage or multistage process. However, in order to verify it, we

( ) and fear ( ). This was the kind of hint that machine needed to improve our method first. The improvements and

learning could extract from our causal partitions. In order to the new results are described in the next subsection.

proceed further, we needed to incorporate deeper technical

knowledge of our agent-based model into our hypotheses. 5.2. Method Improvements and New Results. The first

Our initial hypothesis was that early in the scenario, as a improvement aimed at the identification of the decisive

result of some unknown process or random fluctuation, there moment when the trajectory bifurcation began. We have

formed a nucleus of agents that were either angry or afraid solved this by logging causal partitions periodically every

(while the other agents were still driven by their standard two seconds. Later, off-line, we then identified the point

motives) and this nucleus then converted the rest of the when the partition components started exhibiting increased

agents by their social influence. If this was the case, we would predictive power.

expect the social influence components and to be The second improvement reflected our need for more

negatively correlated (i.e., working against each other) and detailed information. Instead of the combined effect of all

at the same time to be the best predictors for classification. external events lumped together, we recorded the effect of

Analysis of the simulation data, however, showed only a small civilian actions separately from that of security actions by

(albeit negative) correlation. Moreover, this pair was not the splitting each external event component into two: was

best predictor, since its combined accuracy was only about split into (the effect of civilian actions) and (the effect

95%. At this point our method was not yet so mature as to of security actions), and likewise into (civilian) and

enable us to reject this hypothesis outright, but its likelihood (security). The causal partitions of anger and fear thus

decreased. The main weakness of our approach was that we became

causally partitioned only the final values of fear and anger at

the end of the simulation, while the really decisive period = (0 , , , ) ,

(24)

seemed to be its early part. We felt the need to dynamically = ( 0 , , , , )

identify the moment in which the trajectory bifurcation

began and apply causal partitioning at that point. in accordance with the theoretical derivation of our method

Our second hypothesis dealt with and the early attack in Section 4.2.2.

by the violence-prone individuals. There was an element of Our new set of experiments started with 350 simulation

uncertainty as to how many stones they would be able to runs with the time step = 300 ms. Each run again covered

throw. As a rule they selected the closest soldier as their target, 90 seconds of simulated time and periodically logged the

and if they hit him twice, they were in turn immobilized by causal partitions as well as the other relevant attributes ,

an effective shot. Thus, in the worst case, they only threw two , , A-count, and F-count. After a preliminary review

stones, while in the best case, four (with three soldiers, the of simulation logs, we rejected four for showing signs of

fourth stone-throw always resulted in immobilization). Given numerical instability. Out of the 346 remaining ones, 196

that stone-throws incite anger and effective shots mainly fear, belonged to the aggressive cluster and 150 to the timid cluster.

the proportion of stone-throws to effective shots in the early Thus, even the trivial classifier assigning all the simulations

part of the scenario might be the tipping factor determining to the aggressive cluster could achieve approximately 56.7%

its subsequent aggressive or timid turn. If this hypothesis was accuracy (196/346). This means we should consider as infor-

true, then by adjusting the soldiers to use only warning shots mative only those causal partitions and classifiers that achieve

we should make all the scenarios turn aggressive. We tested higher accuracy than 56.7%.

this experimentally, but the bifurcation persisted. Thus the The shape of the two clusters, shown in Figure 4, is similar

second hypothesis had to be discarded as well. to that reported in [8]: timid scenarios congregate in the top

The above experiment also rendered unlikely our third left corner, while the aggressive ones appear to protrude from

hypothesis (that our agent-based system was simply dis- the bottom right corner toward the center of the figure.

playing chaotic behaviour). The first counterargument had The periodic logging of causal partitions allowed us to

already been furnished by the initial clustering exercise, in examine the time-evolution of their predictive power. It

which the system behaviour was shown to be robust, without is shown graphically in Figure 5 and tabulated in Table 4.

undue sensitivity to the time step size. We expected high The 10th second of simulated time turned out to be the

sensitivity if the observed bifurcation had been primarily earliest moment when the outcome at the 90th second could

due to random fluctuations. Furthermore, forcing our soldier be predicted with an increased accuracy, mainly thanks

agents to use only warning shots was a much more significant to (the effect of civilian actions on fear). In the 12th

change and yet the bifurcation persisted. We could therefore second, the prediction accuracy further increased, but here

safely conjecture that the bifurcation was caused by some the importance of faded, having been replaced by

stable and robust mechanism. This did not mean that the (the effect of security actions on fear). In the 14th and the 16th

12 Complexity

60

tabulated.

Time

Predictor

50 8s 10 s 12 s 14 s 16 s

ALL 54.6 72.0 87.6 98.3 99.1

MoE 56.6 70.5 70.5 61.8 56.6

40

56.6 70.2 50.6 85.5 86.4

56.6 56.6 87.3 85.5 79.2

F-count

20

Of course, the decrease was only temporary: our

earlier experiments mentioned in Section 5.1 had

10 shown MoE to regain nearly 100% accuracy toward

the end of the simulation.

0 (3) The MoE predictor was not the only one whose

0 10 20 30 40 50 60 predictive power did not monotonously increase with

A-count time; our causal predictors , also fluctuated.

Figure 4: Simulation clusters for the new set of experiments. Particularly in the case of we suspected that it was

polluted with irrelevant information after the 12th

second.

100.00

(4) The succession in which the individual partition

components took the lead in predictive accuracy

90.00

appeared to confirm our suspicion of a staged process.

Its first phase seemed to be related to civilian actions;

80.00

this would explain why took the lead at the 10th

second. Its second phase appeared to be linked with

(%)

70.00

security actions, conferring on them the predictive

lead at the 12th second. Finally, social influence

60.00

seemed to take over at the 14th second, conferring the

lead on .

50.00

8 10 12 14 16 It is safe to say that the time-evolution of the predictive

(s)

power of partition components and other relevant attributes,

whether displayed graphically as in Figure 5 or tabulated as in

ALL Fes Table 4, would be the principal tool of investigators using the

MoE Fs

method of causal partitions. The key to success is the ability

Fec

to map the peaks in predictive accuracy to their potential

Figure 5: Time-evolution of the accuracy of selected predictors. causes or at least to suspected phases or stages of the observed

emergence. In this case, our knowledge of this particular

model helped us map two of the three suspected phases

seconds, the prediction accuracy came close to 100%, but here mentioned above to what we knew about the behaviour of

the lead shifted to (the effect of social influence on fear). our agents:

Limitations of our graphical software forced us to label

as Fec, as Fes, and as Fs in the figure. The label (1) It was always the aggressive civilians that started

ALL represents the predictor built from all the nine causal the confrontation by stone-pelting the approaching

partition components, while MoE stands for the predictor soldier patrol, so this might be the first phase.

built from the variables , , . (2) The patrol members simply reacted to stone-pelting

Several interesting observations can be drawn from this by warning and effective shots, which might then be

figure and table: the second phase.

(1) The accuracy of the ALL causal model rose very This sequence was common to all our simulations. After-

quickly toward the theoretical maximum of 100%. We wards they bifurcated along two different routes. Regardless

considered this very significant and promising. of the route taken, the process seemed to be linked with social

(2) The competing MoE model could not keep the pace: influence, and this might then constitute a hypothetical third

its accuracy got stuck at 70% and later even decreased. phase.

Complexity 13

solution, yet we could not quite put our finger on it. We

were looking for the feature of our model that was causing 90

the bifurcation, so that by adjusting it we could force all

the simulations along one common route. Moreover, it had

80

to be something that differentiated the timid simulations

(%)

from the aggressive ones. We noticed the first signs of

differentiation in the third phase linked with social influence 70

and conjectured that the aggressive scenarios were fuelled

by social influence on anger while the timid ones by social 60

influence on fear. But even this appeared to be an ex post

phenomenon, a consequence of some invisible factor that just 50

kept eluding us. 1 1.5 2 2.5 3 3.5 4 4.5 5

As in our earlier experiments, we were ultimately forced (s)

to come up with hypotheses incorporating deeper technical

knowledge of our model. But this time we had one more Figure 6: Proportion of aggressive simulations as a function of

precious clue at our disposal: whatever the identity of the soldier reaction time.

elusive factor, it was clearly acting between the 8th second

and the 14th second of simulated time. We focused on this

early period and, using the graphs of selected model variables extreme is not so clear-cut because the soldiers cannot

provided by the MASON simulation framework, we managed start firing without a cause. The confrontation has to start

to identify a new prime suspect: by the 14th second, with a stone-throw, followed by a warning shot, followed

sensory perception of most of our agents was already blocked. by another stone-throw, followed by an effective shot. This

Although significant, it was again something common to event sequence too blocks the perception of the civilians and

all our simulations and, by itself, could not differentiate the incites in them the maximum achievable amount of fear.

timid from the aggressive ones. But in conjunction with Consequently, such scenarios should be timid.

other known facts it finally helped us to piece together a more If this hypothesis was correct, then we should be able to

adequate picture of the mechanism of emergence. Our new control the fate of our simulations by adjusting the reaction

hypothesis postulated that when the aggressive civilians start time of soldier agents. More specifically, by prolonging their

stone-pelting the soldiers, the fate of the scenario depends reaction time, we should exacerbate the civilian tendency

on how quickly and resolutely the soldiers react before the toward aggression. Soldier reaction time is a parameter which

sensory perception of the civilians gets blocked. Later actions is not obviously linked with either fear or anger and, had we

will not affect their emotions. There are two principal sources not investigated the process of emergence, we would have had

of variability in the scenario which impact the initial attack little reason to suspect that it played a significant role in it.

and the subsequent security response: In order to verify this hypothesis, we run several batches

of simulations, each consisting of 350 simulation runs and

(1) We slightly randomize the duration of the agents each with a different setting of soldier reaction time. The

actions so as to prevent massive synchronised replan- results, shown in Figure 6, confirmed our hypothesis. The

ning of their behaviour which we found to interfere standard value of soldier reaction time was one second, at

with smooth visual rendering and real-time operation which 56.7% of the simulation runs in the batch turned

of our system. aggressive. Increasing this time by just half a second increased

(2) At the beginning of each simulation, civilian agents the proportion of aggressive simulations to 78.2%. At two

are created in random positions near the shop. Before seconds, the proportion was already 88.2%, at three seconds

they can attack the soldiers, the aggressive civil- 98.3%, and at four seconds 99.7%. At five seconds, all the

ians have to find shelter behind the corners of the simulations turned aggressive.

buildings. Due to their random initial positions, the We would like to point out here that we were not surprised

exact timing and intensity of their attack are slightly simply by the fact that soldier reaction time affected the

different in each simulation run. subsequent development of simulation scenarios: that was

only natural and expected. What surprised us was that the

It might then be the variable sequence of events perceived reasons for the observed bifurcation fell into this category

by the civilians before their perception gets blocked that too. This at first appeared completely counterintuitive since

seals the fate of each simulation. In order to illustrate this in both timid and aggressive simulations the soldiers reacted

principle, let us consider a couple of event sequences and equally quickly when measured in absolute time; for example,

their emotional impact as per Table 2. At one extreme we in our baseline simulation batches (with soldier reaction time

have six successive stone-throws: these are enough to block set to one second), it was precisely one second after being

the perception of the civilians and incite in them a lot of hit by a stone that soldier agents emitted either a warning

anger but very little fear. Thus, whenever the aggressive or an effective shot, so how could this be at once too late

individuals manage to mount a quick and intensive assault, for simulations that turned aggressive and swift enough for

such scenarios will tend to turn aggressive. The opposite those that turned timid? Our surprising realisation lay in

14 Complexity

the fact that in this case of emergence the speed of response equations. This constrains the type of agent-based systems

was not to be judged by the time it actually took but rather and the types of emergence that can be investigated with it.

relatively to stones thrown, and the blocking of the sensory We do not claim that our approach can throw light on all

perception of civilian agents was the key accessory that kinds of emergence in agent-based systems. Specifically, we

helped shape the system response in this peculiar way. A at present view the emergent behaviour of simple rule-based

somewhat simplified expression of this dependence might be agents as outside its scope: in order to apply causal analysis to

the following: How many stones were thrown by the time the this case we would have to define, first, what we mean by cause

first warning shot occurred? If too many, then the simulation and effect in such a system and, second, how we quantify

would turn aggressive. causal effects.

Although we could in principle formulate further exper- Regarding future work, we see several directions. The

iments to elucidate finer aspects of the behaviour of our first is to investigate our agent-based model in greater depth

model, our main task is now essentially completed. Regarding and elucidate certain subsidiary aspects before we apply our

the bifurcation of simulation trajectories, we have uncovered method to other systems. The first such aspect is the signif-

its phases as well as its generating process and have even icance of social influence for the bifurcation of simulation

identified a nonobvious model feature through which we trajectories. At present, we tend to think that social influence

could suppress it. is not crucial; that is, the bifurcation would persist even if we

kept and in (1) and (2) at zero level, for example, by

6. Conclusion and Future Work setting the constant 4 in (3) to zero. Furthermore, we do not

expect that this would significantly affect the time-evolution

In this article, we introduced the concept of a causal partition of the predictive power of partition components shown in

of a model variable and developed the method of causal Figure 5. What we expect to change is the composition of

partitioning which can be used to investigate emergent the clusters; that is, some simulations might shift their cluster

phenomena in complex systems. affiliation, although we are at present unable to say which

We derived the method in the context of an agent-based cluster would grow and which would shrink. We also expect

model of human behaviour from project EUSAS, in which this effect to be relatively mild (if any). Another interesting

simulated emotions of fear and anger competed for the question is how many clusters are really there. In one of our

control of agent behaviour and were partly driven by ordinary early reports [21] we decided to work with two clusters, but

nonlinear differential equations. In one of our scenarios, this that was an arbitrary decision in order to keep the analysis

model exhibited a puzzling emergent behaviour: simulations simple. We were not at all sure that we would succeed and

with the same input parameter setting bifurcated along two were ready to consider more clusters if necessary. The fact that

different trajectories. The time-evolution of the predictive we could complete the analysis using only two clusters does

power of causal partitions with respect to this bifurcation not by itself settle the question; it might merely show that our

helped us identify its stages as well as its generating process. method tolerates some uncertainty in this respect.

This in turn helped us find a nonobvious model feature The second direction, and a natural next step, is to apply

through which we could suppress the bifurcation and force our method to other similar systems. By similarity we mean

all the simulations along one common route. that, besides being modelled through structural equations,

In general, the method of causal partitions comprises the variables and functions used in them should be numerical

four steps. The first, mathematically rigorous, includes causal (ideally, real-valued). The most straightforward application

analysis and the derivation of causal partitions for the would be to systems governed by ordinary differential equa-

system of interest. As a result, causal partitions produced by tions, but we believe an extension to systems driven by partial

simulations in the second step can be reliably interpreted differential equations should be possible as well. As already

as showing how much each causal factor contributed to the mentioned, agent-based models using structural equations

numerical value of the partitioned model variables. The third would make another legitimate and interesting analytical

and the fourth steps are more heuristic in character and target. We also see an exciting possibility to adapt our

combine elements of statistics, data mining, and machine method for use in artificial neural networks: these too are

learning. This phase starts with the investigation of the described by structural equations, because each neuron has

predictive accuracy of causal partitions with respect to the only one output (dependent variable) which is a function of

observed emergent phenomenon. It is important to keep one or more inputs. In this way we might be able to study,

in mind that high predictive accuracy of a given partition for example, the processes of learning in complex neural

component does not by itself guarantee that the causal factor architectures.

behind it is actually causing the phenomenon: it may be The discipline of System Dynamics merits special men-

merely associated with it. Additional validation activities are tion: it not only deals with dynamical systems but also

needed, separate experiments manipulating the suspected painstakingly maps the flows of causality through them.

cause and confirming or disconfirming its effect on the Therefore, in most cases, SD models equations should qualify

emergent phenomenon. In any case, the best early predictors as structural equations, which would make causal partition-

help us focus on the relevant aspects of the system in search ing applicable to them at least in principle. As an example,

for its real causes and generating processes. let us consider variable (modelled in SD as a stock) with

It will also be well to point out that our approach can inflows and outflows 1 , 2 , . . . , . For simplicity, let us

be applied only to systems modelled through structural assume that the dynamical process we are interested in starts

Complexity 15

at time = 0 with zero initial value of . Then its value at value of variable at time . Substituting from (A.2) into

an arbitrary subsequent point of time can be expressed as (A.1) leads to

the sum of all its inflows and outflows separately integrated

() (1) (1) (1)

(or summed) over the time interval [0, ]. Thus, if denotes

+ ( , ),

the integrated or summed flow over the concerned period, (A.3)

then () = 1 + + . We can then represent () by a = 1, . . . , , = 1, . . . , .

vector-like structure (1 , . . . , ) which closely corresponds

to what we have called a basic causal partition earlier in Now the relationship to (9) becomes clearer. The left-

the article. In this basic version of causal partitioning, hand side of (A.3) corresponds to variable in (9), which

we look on each flow as contributing to its own dedicated means that in this structural form the value of at time

partition component of . The purpose of partitioning is is considered a separate structural variable, distinct from

to gain additional information: we now know not only the the values of at other points in time. Most importantly,

()

resulting value of but also how much each flow contributed, in this form, the parent set of no longer contains this

which might help us understand some of the more puzzling variable but only the preceding values of in time. Thus,

behaviours of the modelled system, for example, through after the discretization, no structural variable depends on

exploring the time-evolution of the predictive power of itself. We can therefore conclude that the discretized version

partition components and their combinations with respect of our model does qualify as a structural model as defined by

to the observed behaviour. More sophisticated (enhanced) Pearl in [5].

forms of causal partitioning might try to redefine causal

partitions typically by splitting some components into two B. Improving and Fine-Tuning

or more in order to gain more information or by eliminating Causal Partitions

others in order to improve interpretability.

Finally, the last direction of future work comprises the Regarding mathematical notation used in this appendix, we

study of the methods mathematical properties, especially the view the process of simulation as effectively discretizing time

limits of its stability. Elsewhere in the article we mentioned into a sequence 0 , 1 , 2 , . . ., whose general member can be

that we had to reject some data because it exhibited signs calculated as = . The first simulation step then covers

of numerical instability. We could not go into details, but the period [0 , 1 ) or [0, ). In general, the th simulation

in [22] we mentioned how this could be handled, and we step spans the period [1 , ) which can be calculated as

intend to follow it up alongside our work on further practical [( 1) , ). With respect to model variables we

applications. employ superscript indexing where () stands for ( ).

Analogously, () denotes its backward difference at time

Appendix

: () = () (1) .

A. Applicability of Structural Causality to

B.1. Eliminating a Redundant Partition Component. We show

Ordinary Differential Equations

here how we managed to eliminate the component of

In this appendix we restrict our attention to (1)-(2) and their the basic causal partition of fear defined in Section 4.1. The

generalised form ((4a) and (4b)). Following the notation used elimination rests on a revised version of (15), in which the

in the prototypical structural equation (9), such equations term has been further analysed. Practically, represents

can be written as the change of over one simulation step which spans a period

of simulated time.

, ) , = 1, . . . , ,

= ( (A.1) If we apply (15) to backward differences at time 1 and

expand the term (1) , with a slight rearrangement, we get

where the function can be nonlinear. One potential

problem seems to be that instead of a model variable there is a

time derivative on the left-hand side; another problem is that (1) (2) + (1)

= (1)

(1)

derivative; that is, it appears to belong to its own parent set: (B.1)

. In order for such models to qualify as structural, (1)

+ (1) .

the differential equations need to be converted into difference = (1)

equations, for example, by approximating the derivatives by =

difference quotients:

() () (1)

Here, (1) is the value of at the beginning of the

current (th) simulation step, and (2) is its value at

= ,

1 (A.2) the beginning of the previous step. Partial derivatives are

evaluated at midpoint ( (1) , (1) ) of the previous step. Our

= 1, . . . , , = 1, . . . , ,

goal is to causally partition the total value of fear at the end

where indexes the original variables and indexes the ()

of the current simulation step . Equation (B.1) can be

()

discretized moments of time , so that stands for the interpreted as saying that the first derivative of fear at the

16 Complexity

beginning of the current step can be approximated on the In practice, we do not even need (B.6b) since we can

basis of the values from the previous step. Of course, we do handily calculate from the causal partition constraint: =

not need (B.1) in order to calculate (1) because, knowing .

(1) (1)

and , we can calculate it directly. The reason we Having partitioned the first derivative of fear , we

need (B.1) is to causally partition , which is a prerequisite can now partition fear itself. In order to partition its new

()

for causally partitioning fear itself. total value at the end of the current simulation step ,

(1)

At the same time, (1) is simply the sum of the discrete take its previous partition and add to it, component

and the continuous changes during the previous simulation by component, the continuous increment for the current

step: simulation step. This increment can be written in vector form

(1) (1)

(1) as ( , ). Then add the cumulative impact of

(1) = 5 + (2) . (B.2)

()

external events perceived during the current step 5

Substituting this into (B.1) eliminates fear as a cause of

directly to . Expressed in the form of recurrence relations

its first derivative :

it becomes

(1) (2) () (1) (1) ()

= + + 5 , (B.7a)

(1)

+ (5 + (2) ) (1) () (1) (1)

= (1) = + . (B.7b)

= (B.3)

Again, instead of calculating as per (B.7b), we can

(1) calculate it more handily from the causal partition constraint:

+ (1) .

= (1) = .

=

This can be rewritten as B.2. Handling Dependency among Causal Factors. On the

(1) (2) (1) (1) basis of (21), we can approximate the total effect of fear on

1 + 2 5 + 3 , (B.4)

the value of as the sum of its direct effect and its indirect

where 1 , 2 , 3 represent sensitivities or weighting factors effect:

based on the partial derivatives evaluated at the midpoint

(1) , (1) ) of the previous simulation step:

( = + .

(B.8)

direct

3 = (1) ,

indirect

= (1) (B.5a)

= We now need to restructure (15) in line with this more

general perspective, which can be done by simultaneously

adding and subtracting the indirect effect of fear on its right-

2 = (1) ,

= (1) (B.5b) hand side:

=

1 = 1 + 2 . (B.5c) [ + ]

These partial derivatives can be approximated numeri- total effect of fear

cally. (B.9)

Equation (B.4) can be directly translated into an improved

+[ ].

method of causal partitioning. In this version of the method,

the first derivative of fear is represented by the causal social influence residual

partition vector ( , ) and fear by the partition ( , ).

Partition components indexed by represent the contribu- We see that , which represents the total change of

tion of external events, while those indexed by stand for over one simulation step, can be approximated (and therefore

the contribution of social influence. interpreted) as the total effect of fear of our agent plus a

The improved method proceeds as follows: in order to term which we propose to call the social influence residual

causally partition the first derivative of fear at the begin- and denote by . We obtain this residual by adjusting the

gross effect of social influence for the indirect effect of fear

ning of the current simulation step (1) , take its previous

, so that it only includes the effect of fear of the remaining

partition (2) , multiply it by 1 , and add the weighted (influencing) agents. If we express the total effect of through

contributions of external events and social influence during the total derivative and compact the residual, we obtain

the previous simulation step to their respective partition

components and . This can be expressed through

recurrence relations, one for each partition component: + . (B.10)

(1) (2) (1)

= 1 + 2 5 , (B.6a) This equation holds for all simulation steps. As in

(1) (2) (1) the derivation of the enhanced version of the method in

= 1 + 3 . (B.6b) Section B.1, we apply it to the simulation step preceding the

Complexity 17

current one; that is, we use backward differences at time 1 . The above equation enables us to split the original causal

Expanding (1) and substituting for (1) from (B.2) partition component as defined in Section B.2 into three:

lead to a generalised form of (B.4) which does not require 0 , , . The idea of splitting means that the recurrence

and to be independent: relations for these new derived components will be analogous

to that for in (B.13a), except that each new component now

(1) only includes the appropriate kind of event impact:

(2) (1)

0

= 1 0

(2)

+ 2 5 0 ,

(1)

(B.16a)

= 1

(2)

+ 2 5 ,

(1)

+ (5 + (2) ) (1) (B.16b)

= (1)

= (1) (2) (1)

= 1 + 2 5 , (B.16c)

(1)

+ (1) (2) (1)

= 1 + . (B.16d)

(2) (1) (1)

1 + 2 5 + , (B.11b)

Likewise, will be split into 0 , , , with recur-

where

1

, 2

are sensitivities or weighting factors capturing rence relations analogous to that for in (B.7a):

the total sensitivity of to changes in (as opposed to the () (1) (1) ()

direct sensitivity captured by 1 , 2 in (B.5a), (B.5b), and 0 = 0 + 0 + 5 0 , (B.17a)

(B.5c)): () (1) (1) ()

= + + 5 , (B.17b)

2 = (1) ,

=

() (1) (1) ()

(B.12a)

(1)

= + + 5 , (B.17c)

=

() (1) (1)

= + . (B.17d)

1 = 1 + 2 . (B.12b)

This leads to recurrence relations for components of Finally, we also need to adjust the causal partition con-

resembling those of (B.6a) and (B.6b): straints:

() () () ()

(1)

= 1

(2)

+ 2 5

(1)

, (B.13a) () = 0 + + + , (B.18a)

() () () () ()

(1)

= 1

(2)

+ .

(1)

(B.13b) = 0 + + + . (B.18b)

As a result of working with total sensitivities, the compo- These more detailed causal partitions enable us to exam-

nent now represents just the social influence residual, that ine the effect of civilian actions on the emergent behaviour of

is, only the effect of fear of the influencing agents. the system separately from that of security countermeasures

Regarding fear , its recurrence relations remain the same or the setting of the initial value of fear.

as in (B.7a) and (B.7b), and the partition constraint = +

continues to hold as well. Competing Interests

B.3. Splitting Partition Components to Gain More Information. The authors declare that they have no competing interests.

In this section, we illustrate the process of splitting partition

components on the component which in the previous Acknowledgments

versions of causal partitioning held the total contribution of

all the perceived events regardless of their type. If we sum the This work was supported by EDA Project A-0938-RT-GC

effects of perceived events separately by type, we can express EUSAS, national project VEGA no. 2/0167/16, and the Slovak

their total effect as Research and Development Agency under the Contracts nos.

APVV-0233-10 and APVV-0809-11. The authors would like

= 0 + + , (B.14) to thank Dr. Ladislav Halada of the Institute of Informatics

of the Slovak Academy of Sciences for helpful suggestions

where 0 is the effect of the setting of the initial value of

regarding mathematical notation.

fear 0 , is the effect of civilian actions, and is that

of security countermeasures. Substituting this into (B.11b)

provides justification for further splits (or fine-tuning) of our References

causal partitions: [1] R. G. Sargent, Verification and validation of simulation mod-

els, Journal of Simulation, vol. 7, no. 1, pp. 1224, 2013.

(1) 1

(2) + 2

5

[2] M. A. Louie and K. M. Carley, Balancing the criticisms:

(B.15)

(1) (1) (1) (1) validating multi-agent models of social systems, Simulation

(0 + + ) + . Modelling Practice and Theory, vol. 16, no. 2, pp. 242256, 2008.

18 Complexity

[3] M. Luck, P. McBurney, O. Shehory, and S. Willmott, Agent [21] M. Kvassay, L. Hluchy, P. Krammer, and B. Schneider, Explor-

technology roadmap: a roadmap for agent based computing, ing human behaviour models through causal summaries and

Tech. Rep., University of Southampton on behalf of AgentLink machine learning, in Proceedings of the 17th IEEE International

III, 2005, http://eprints.soton.ac.uk/261788/1/al3roadmap.pdf. Conference on Intelligent Engineering Systems (INES 13), pp.

[4] S. Bullock and D. Cliff, Complexity and emergent behaviour 231236, San Jose, Calif, USA, June 2013.

in ICT systems, Tech. Rep., Hewlett-Packard Labs, Palo Alto, [22] M. Kvassay, L. Hluchy, and B. Schneider, Summarizing the

Calif, USA, 2004, http://eprints.soton.ac.uk/261478/1/HPL- behaviour of complex dynamic systems, in Proceedings of the

2004-187.pdf. IEEE 11th International Symposium on Applied Machine Intel-

[5] J. Pearl, Causality: Models, Reasoning, and Inference, Cambridge ligence and Informatics (SAMI 13), pp. 1520, IEEE, Herlany,

University Press, New York, NY, USA, 2000. Slovakia, February 2013.

[6] J. Y. Halpern and J. Pearl, Causes and explanations: a structural-

model approach. part I: causes, British Journal for the Philoso-

phy of Science, vol. 56, no. 4, pp. 843887, 2005.

[7] J. Y. Halpern and J. Pearl, Causes and explanations: a structural-

model approach. Part II: explanations, British Journal for the

Philosophy of Science, vol. 56, no. 4, pp. 889911, 2005.

[8] M. Kvassay, L. Hluchy, P. Krammer, and B. Schneider, Causal

analysis of the emergent behavior of a hybrid dynamical

system, Acta Polytechnica Hungarica, vol. 11, no. 4, pp. 2140,

2014.

[9] M. Kvassay, L. Hluchy, S. Dlugolinsky et al., A novel way

of using simulations to support urban security operations,

Computing and Informatics, vol. 34, no. 6, pp. 12011233, 2015.

[10] L. Berkowitz, Ed., Aggression: A Psychological Analysis,

McGrawHill, New York, NY, USA, 1962.

[11] S. Prentice-Dunn and R. W. Rogers, Deindividuation and the

self regulation of behaviour, in Psychology of Group Influence,

P. B. Paulus, Ed., pp. 89109, Lawrence Erlbaum Associates,

Hillsdale, Mich, USA, 1989.

[12] E. Staub, Predicting collective violence: the psychological and

cultural roots of turning against others, in Collective Violence,

C. Summers and E. Markusen, Eds., pp. 195209, Rowman &

Littlefield Publishers, Lanham, Md, USA, 1999.

[13] D. Canamero, Modeling motivations and emotions as a basis

for intelligent behaviour, in Proceedings of the 1st International

Conference on Autonomous Agents (AGENTS 97), pp. 148155,

ACM Press, Marina del Rey, Calif, USA, February 1997.

[14] B. Latane, Dynamic social impact, Journal of Communication,

vol. 46, no. 4, pp. 1325, 1996.

[15] C. Urban, PECS: a reference model for the simulation of

multiagent systems, in Tools and Techniques for Social Science

Simulation, R. Suleiman, K. G. Troitzsch, and N. Gilbert, Eds.,

pp. 83114, Physica Verlag, 2000.

[16] B. Schmidt, The modelling of human behaviour: the PECS

reference model, in Proceedings of the 14th European Simulation

Symposium and Exhibition (ESS 02), A. Verbraeck and W. Krug,

Eds., pp. 1318, SCS Europe BVBA, Dresden, Germany, October

2002.

[17] C. Swinerd and K. R. McNaught, Design classes for hybrid sim-

ulations involving agent-based and system dynamics models,

Simulation Modelling Practice and Theory, vol. 25, pp. 118133,

2012.

[18] J. Pearl, An introduction to causal inference, International

Journal of Biostatistics, vol. 6, no. 2, Art. 7, 61 pages, 2010.

[19] M. Kvassay, L. Hluchy, B. Schneider, and H. Bracker, Towards

causal analysis of data from human behaviour simulations, in

Proceedings of the IEEE International Symposium on Logistics

and Industrial Informatics (LINDI 12), Smolenice, Slovakia,

September 2012.

[20] Weka 3: Data mining software in Java, http://www.cs.waikato

.ac.nz/ml/weka.

Advances in Advances in Journal of Journal of

Operations Research

Hindawi Publishing Corporation

Decision Sciences

Hindawi Publishing Corporation

Applied Mathematics

Hindawi Publishing Corporation

Algebra

Hindawi Publishing Corporation

Probability and Statistics

Hindawi Publishing Corporation

http://www.hindawi.com Volume 2014 http://www.hindawi.com Volume 2014 http://www.hindawi.com Volume 2014 http://www.hindawi.com Volume 2014 http://www.hindawi.com Volume 2014

World Journal

Hindawi Publishing Corporation

Differential Equations

Hindawi Publishing Corporation

http://www.hindawi.com Volume 2014 http://www.hindawi.com Volume 2014

https://www.hindawi.com

Combinatorics

Hindawi Publishing Corporation

Mathematical Physics

Hindawi Publishing Corporation

http://www.hindawi.com Volume 2014 http://www.hindawi.com Volume 2014

Complex Analysis

Hindawi Publishing Corporation

Mathematics

Hindawi Publishing Corporation

in Engineering

Hindawi Publishing Corporation

Applied Analysis

Hindawi Publishing Corporation

Nature and Society

Hindawi Publishing Corporation

http://www.hindawi.com Volume 2014 http://www.hindawi.com Volume 2014 http://www.hindawi.com Volume 2014 http://www.hindawi.com Volume 2014 http://www.hindawi.com Volume 2014

International

Journal of Journal of

Mathematics and

Mathematical

Discrete Mathematics

Sciences

Function Spaces

Hindawi Publishing Corporation

Stochastic Analysis

Hindawi Publishing Corporation

Optimization

Hindawi Publishing Corporation

http://www.hindawi.com Volume 2014 http://www.hindawi.com http://www.hindawi.com Volume 2014 http://www.hindawi.com Volume 2014 http://www.hindawi.com Volume 2014

- Potochnik - 2014- Causal Patterns and Adequate ExplanationsUploaded byAnonymous DSATBj
- 77Uploaded byLip Stick
- Where Explanation Ends Understanding AsUploaded byhenriqueffc
- AR_FINALUploaded byAxle Vega
- National Study on Neck Restraints in Policing - Technical Report - Canadian Police Research CentreUploaded byEd Praetorian
- Multiple Regression in Small-N ComparisonsUploaded byDavi Barboza
- soc-sci-app (1)Uploaded byChris Valenzuela
- ProcessAnalysisUploaded byDr.Raj Mohan
- Ibp Yp Key Concepts Zebra the MeUploaded by52702378
- Chap 7 PatternsUploaded bynatsdorf
- A Study in Karma - A BesantUploaded byNicholas Jensen
- caraka nyayaUploaded byManoj Sankaranarayana
- How to Read CriticallyUploaded byShariBarry123
- Statistics AssignmentUploaded byVictor 李家铭 Lee
- Academic Pharase Bank Uni of MancehsterUploaded bySITI HAJAR ASWAD
- Bridges--Ecology of Reasons Talk for Berlin and LeipzigUploaded byNix Da
- student work analysismgUploaded byapi-284284138
- english languageUploaded byAyaz Saa'ir
- Summary 1 and 2 ApproachesUploaded byolaonly
- bbdlm3 5320Uploaded byapi-316237255
- 0 J2004 - Calling Krsna's bluff Non-attached action in the Bhagavadgītā.pdfUploaded bybudimah
- Secada - Descartes on Time and Causality - 1990Uploaded byFederico Orsini
- Effectuation Framework as an Alternative Paradigm to Opportunity Identification for Creation of New Venture EnterpriseUploaded byIOSRjournal
- Fales-Mystical-Experience-as-Evidence.pdfUploaded byDanielle Newman
- Cause-ConsequenceUploaded bycamille_faye
- Pornography Consumption and Satisfaction:A Meta-Analysis by Paul J.Wright et alUploaded byRoland Kristo
- The Science of Human Nature, By William Henry PyleUploaded byTheVoiceWithin
- Cause and Effect DiagramsUploaded byMohammad Jaid Alam
- Lincoln NaturalisticInquiryUploaded byGiusi Caforio
- Handbook of Political ExperimentUploaded byJairo Pimentel

- vialidad urbana sedesol.pdfUploaded byCesar Hernández
- Venas Abiertas de La Matemática FinancieraUploaded byMiguel Angel Ramirez Arellano
- 17014623 Precaris a La Deriva a La Deriva Por Los Circuitos de La Precariedad Femenina 2003(1)Uploaded byJorgelina Villarreal
- Computer Algebra Recipes - An Advanced Guide to Scientific Modeling Richard H Enns(Springer 2007 372S With Maple)Uploaded byÓscarillo Valdés
- Biological Fluid Dynamics, Non-Linear Partial Differential EquationsUploaded byÓscarillo Valdés
- Las Venas Abiertas de La Matemática FinancieraUploaded byÓscarillo Valdés
- Chaos Control and Synchronization via Switched OutputUploaded byÓscarillo Valdés
- Evolución reciente de la industria de cemento.pdfUploaded byÓscarillo Valdés
- Una Teoría de La Formación BiológicaUploaded byÓscarillo Valdés
- What is Tumblr a Statistical Overview and ComparisonUploaded byÓscarillo Valdés
- The Strategic Decision-Making as a Complex AdaptiveUploaded byÓscarillo Valdés
- The Strategic Decision-Making as a Complex AdaptiveUploaded byÓscarillo Valdés
- Chang 2016 Telematics and InformaticsUploaded byÓscarillo Valdés
- educación y valoresUploaded byÓscarillo Valdés
- On the Notion of CognitionUploaded byÓscarillo Valdés
- lacomunicacionPRLernomiacognitivafolletoUploaded byÓscarillo Valdés
- Equidad y Prevención de violencia AYUDA PARA EL MAESTRO BLOGUploaded byLop Wilbert
- EL TABERNACULOUploaded byÓscarillo Valdés
- EL TABERNACULOUploaded byÓscarillo Valdés
- EL TABERNACULOUploaded byÓscarillo Valdés
- Manual BullyingUploaded byzagr28
- softlibreUploaded byvertice
- Reforma Fiscal 2014 Paso Por PasoUploaded byÓscarillo Valdés
- Some Formal Problems With the Von NeumannUploaded byÓscarillo Valdés
- 22681711Uploaded byDoroteo Cortes Perez

- mp_sat_solution.pdfUploaded bybhaskar ray
- Battery Testing, Test Methods and ProceduresUploaded bya_mohid17
- FKA195: Lab Demo Thin Film Resistance MeasurementsUploaded bysaradayamini2005
- resistor smd 220 ohmsUploaded byXavier G Tesla
- Distribution System and Other Elements ModelingUploaded byfreddy rivera
- ElectricalUploaded byCesar Bl
- bca experimentsUploaded byNaveen Bhat
- fordUploaded bymiguel
- WorkShop Practice Lab MannualUploaded byAliHaider
- 03~model solutions 3 (2)Uploaded byChaitanya Bhargav Kotike
- Memristors DATAUploaded byGuruprakash Singh
- PS6000Uploaded byGermainJCR
- SDI_113Uploaded bylaurenjia
- Transformer Losses CalculationUploaded bynaveedfnd
- ELESUploaded byAnoop Sree Harshan
- AutomotiveUploaded byMos Craciun
- Incomplete List of Iec StandardsUploaded bySports Gallery
- GATE_S6_EE_2017Uploaded byPartha Sarathi Manna
- 57078197-Module-3Uploaded byJeremy Kumar
- SABH100 ManualUploaded byDee Formee
- SemiconductorUploaded byPaul Senen Didulo
- ThicknessUploaded byRajaEvans
- 2x1.20 Rm Flexible TTUploaded byjamil
- Guide to Interpreting I-V CurvesUploaded bysanjurk
- Diode (Lab Report)Uploaded byBenjamin Cooper
- Fire Sensor CircuitUploaded byNoel Saligue
- ESAT (Transmission Fundamentals)Uploaded byMalkiel Callisthenes Darja
- 7.Article Dynamic Analysis and Testing of OLTC by DRM ENU FinalUploaded byHuấn Phan
- Clas 10 Sa1 Physics WsUploaded byKamaluddin Azmi
- STEM_Physics 2 CG_with tagged sci equipment.pdfUploaded byOnyong Galarion