UNCLASSIFIED

U.S. ARMY COMBAT CAPABILITIES

DEVELOPMENT COMMAND –
ARMAMENTS CENTER
The Design of Swarm Experiments: A Systems Engineering Approach to
Swarm Research
2022 AI4SE & SE4AI Workshop, 21-22 September 2022
Distribution Statement A: Approved for Public
Melissa Jablonski, Jonathan Jablonski, Ross Arnold, Benjamin Abruzzo, Elizabeth Mezzacappa Release, Distribution is Unlimited.

US Army DEVCOM Armament Center, US Army Armament Graduate School

Presented by: Melissa Jablonski
21-22 Sept 2022 UNCLASSIFIED
 UNCLASSIFIED

OUTLINE

– Background
• Swarm Intelligence
• Computer Simulation Experimentation of Swarms
– The Design of Swarm Experiments Methodology
• Plan
• Design
• Execute
• Analyze
• Assimilate
– Conclusions

UNCLASSIFIED 2
 UNCLASSIFIED

BACKGROUND OF SWARM INTELLIGENCE

– Groups of entities of similar

characteristics that tend to move
together and work cooperatively
toward an overarching goal
– Swarm intelligence is based on the
concept of swarm theory: simple
entities behaving in a collaborative
manner can produce emergent effects
• Think beehives and ant swarms
– Application of general systems theory

UNCLASSIFIED 3
 UNCLASSIFIED

COMPUTER SIMULATION EXPERIMENTATION

– DroneLab is an example of a software

application designed to facilitate
simulation of large numbers of
unmanned aircraft systems (UAS)
operating cooperatively as a robot
swarm
– Here, DroneLab is used to simulate
swarms of UAVs after a natural disaster
– Certain parameters of the simulation
(e.g. input variables) are varied to
understand the effect each has on our
output of interest: Time to find survivors
UNCLASSIFIED 4
 UNCLASSIFIED

THE DESIGN OF SWARM EXPERIMENTS

METHODOLOGY
Plan
– Design of Experiments techniques Identify the objectives,
are used to systematically sample MoE/MoPs, & swarm
characteristics of
the design space interest

– Best practice method

Assimilate Design
• Generate valid, useful data Use the prediction Choose the specific
model to build an scenarios for the
• Support the development of an inductive framework DSE simulation or test
accurate model Methodology
• Describe and predict the behavior of
the system
Analyze Execute
– Supports continued research: plan, Conduct the
Fit a prediction model
design, execute, analyze, and to the data
simulations or test &
aggregate the data
assimilate new knowledge that leads
to planning the next research effort
UNCLASSIFIED 5
 UNCLASSIFIED

PLAN

– First and perhaps most important step of

Plan
the DSE process Identify the objectives,
• Upfront time and effort to properly MoE/MoPs, & swarm
characteristics of
define the problem generally leads to a interest
net reduction in overall project
resources
Assimilate Design
• Knowing the end goal allows for the Use the prediction Choose the specific
model to build an scenarios for the
planning of a path to get there inductive framework DSE simulation or test
– Three primary tasks Methodology
• Identification of the objective
• Identification of measures of
effectiveness/measures of Analyze Execute
Conduct the
performance Fit a prediction model
simulations or test &
to the data
• Identification of swarm/entity aggregate the data

characteristics

UNCLASSIFIED 6
 UNCLASSIFIED

PLAN
Identification of the objective

– Two primary questions to address

1. Why is your team conducting the research?
2. What does you team hope to accomplish as a result of the research?
– Consideration of Research and Engineering Objectives
• Research Objectives
– Broad qualitative goals emphasizing general aspects of swarm behavior
– Understanding the interactions between independent and dependent variables
– Building of a theoretical framework explaining the interactions
• Engineering Objectives
– Based on quantitative measures considered in the context of specific applications
– Often mathematically expressed as a constrained optimization process
– Potentially a binary comparison between a proposed design relative to a baseline design

UNCLASSIFIED 7
 UNCLASSIFIED

PLAN
Identification of measures of effectiveness & measures of performance

– These are the dependent or predicted variables,

the output of the simulation or result of the test
– The metrics that matter and can be measured
– Consider both the engineering and research
objectives, may be multiple ways of looking at the
same data
• Engineering MoP/MoE: Time to rescue 80% of survivors
• Research objective may have you delve further into the
performance over the entire mission time and differences
in curve shape

UNCLASSIFIED 8
 UNCLASSIFIED

PLAN
Identification of swarm/entity characteristics

– These are the independent predictor variables,

the inputs of the simulation or what is being
controlling in the test
– Type of variable
• Continuous, discrete, mixture, categorical
– Range or levels of the variable
• Minimum and maximum or discrete levels
– Constraints or combinations of variables that
can’t physically be tested or would not be useful
to test/simulate
• Example: higher flight speeds will result in lower battery
life

UNCLASSIFIED 9
 UNCLASSIFIED

DESIGN

– What combinations of inputs to Plan

simulate or test? Identify the objectives,
MoE/MoPs, & swarm
– Design of experiments (DOE) characteristics of
interest
approach
– Systematically vary the Assimilate Design
independent variables to identify Use the prediction Choose the specific
model to build an scenarios for the
the effect each has on the inductive framework DSE simulation or test

dependent variables Methodology

– 3 broad classes:
• Factorial/Optimal Designs Execute
Analyze Conduct the
• Space Filling Designs Fit a prediction model
simulations or test &
to the data
aggregate the data
• Hybrid Designs

UNCLASSIFIED 10
 UNCLASSIFIED

DESIGN
Recommended Designs for Drone Experiments
Factorial and Optimal Designs Space filling designs Hybrid designs
Place points in extremal corners Distribute points throughout the
to maximize leverage and design space, avoid repetition Combination of factorial/optimal
Description
achieve repetition in lower even in lower dimension and space filling designs
dimension projections projections
Low
Expected High
deterministic or low-noise
Experimental live testing, very noisy Low to Mid
simulations, high noise data that
Noise simulations
can be replicated
High
Available Low
typically at least enough runs for Low to Mid
Resources typically 10-50 unique runs
10x the number of factors
Expected Simple
Complex
Response can be described by a multiple In between
high order, non-linear behavior
Behavior linear regression model
Space filling augmented with I-
Full factorial designs, Fractional Latin Hypercube, Maximum
optimal points, Fractional
Examples factorial designs, D-optimal Projection, D-optimal Latin
factorial augmented with space
designs, I-optimal designs Hypercube
filling points

UNCLASSIFIED 11
 UNCLASSIFIED

EXECUTE

– Run the simulations or conduct Plan

the test Identify the objectives,
MoE/MoPs, & swarm
– Recommend putting some characteristics of
interest
upfront effort into automation
• Simulation setup, submission of Assimilate Design
runs, extraction of results, and data Use the prediction Choose the specific
aggregation model to build an
DSE scenarios for the
inductive framework simulation or test
– Track and report any errors and Methodology
outlier runs
• Analysis of this data can aid in
identifying implementation issues Analyze Execute
Conduct the
• Can lead to improved verification of Fit a prediction model
simulations or test &
to the data
the model/simulation aggregate the data

UNCLASSIFIED 12
 UNCLASSIFIED

EXECUTE
Sample Simulation Matrix

UNCLASSIFIED 13
 UNCLASSIFIED

ANALYZE

– The process of fitting a prediction model

Plan
to data is commonly called statistical Identify the objectives,
learning or machine learning MoE/MoPs, & swarm
characteristics of
– Fit a mathematical equation to predict interest
the output as a function of all the inputs
– Large number of model fitting methods Assimilate Design
exist; several are recommended for Use the prediction Choose the specific
model to build an scenarios for the
designed experiments and continuous inductive framework DSE simulation or test
response data Methodology
• Linear Regression
• Gaussian Process/Kriging
• Neural Networks Execute
Analyze Conduct the
Fit a prediction model
simulations or test &
to the data
aggregate the data

UNCLASSIFIED 14
 UNCLASSIFIED

ANALYZE
Recommended Model-Fitting Methods for Drone Experiments

Linear Regression Gaussian Process Neural Networks

A set of inter-connected
Linear approximation of Multidimensional
feedforward activation
response based on one interpolation using spatial
Description functions loosely inspired
or more predictor correlation to make
by the architecture of a
variables predictions
biological brain

Simple and highly

Excels at fitting highly
interpretable, Estimate of Excels at fitting highly
complex response
Advantages expected model error, complex response
surfaces, Estimate of
Estimate of expected surfaces
expected model error
error of predictions

Complex response
Simple response Complex response
Typical Uses behavior with large
behavior behavior
amounts of data

UNCLASSIFIED 15
 UNCLASSIFIED

ASSIMILATE

– What can be learned from the prediction

Plan
model? Identify the objectives,
– Model is a closed form equation that can MoE/MoPs, & swarm
characteristics of
near instantaneously emulate the interest
simulator or test
• Make predictions anywhere within the Assimilate Design
original variable bounds Use the prediction Choose the specific
model to build an scenarios for the
• Easily see relationships between inductive framework DSE simulation or test
inputs and outputs Methodology
• Numerically optimize for specific
scenarios or goals
• Rank the importance of independent Analyze Execute
Conduct the
variable on the outputs of interest Fit a prediction model
simulations or test &
to the data
aggregate the data

UNCLASSIFIED 16
 UNCLASSIFIED

ASSIMILATE
Research Goals

– Assessing old or formulating new

hypotheses
– Understanding general
interactions between independent
and dependent variables
– Identifying regions in the design
space which demarcate
fundamentally different regimes of
swarm behavior

UNCLASSIFIED 17
 UNCLASSIFIED

ASSIMILATE
Engineering Goals

– Optimizing a swarm design

relative to MoPs or MoEs
– Conducting trade studies in which
physical specifications or
behavioral roles are studied
– Assessing a swarm performance
in “edge-cases” or mission critical
scenarios

UNCLASSIFIED 18
 UNCLASSIFIED

CONCLUSIONS

– Design of swarm experiments is a novel, rigorous method to

systematically explore swarm and multi-agent systems in the face of
uncertainty
– Generally applicable across the research community
– Explore both deterministic and non-deterministic problems
– Applicable to live swarm robotic testing and simulation environments
– Results in the ability to thoroughly explore and understand the
behavior of multiple inputs and design parameters on the outputs of
interest

UNCLASSIFIED 19
 UNCLASSIFIED

QUESTIONS

?
Melissa Jablonski
Systems Analysis Division
DEVCOM Armaments Center
melissa.n.jablonski.civ@army.mil
973-724-9912

UNCLASSIFIED 20