Evaluating the Reproducibility of Multiagent
Systems
Alex Vitorino
Federal University of São João del-Rei
São João del-Rei, Brazil
alexvitorino@ufsj.edu.br
Erivelton Geraldo Nepomuceno
Federal University of São João del-Rei
São João del-Rei, Brazil
nepomuceno@ufsj.edu.br
Abstract—A multiagent system (MAS) is a computer based
model of multiple interacting intelligent agents within an envi-
ronment. One of the key aspects of such models is the ability
to reproduce the results in different software and hardware. In
general, the simulation of these models are carried out under
floating point standards. It is well known that by limitation
of computer arithmetic, the results may be different when the
combination of hardware and software is also different. In such
case, the evaluation of the reproducibility of MAS is harder than
deterministic models, as it is usually seen as stochastic process.
To overcome this problem, this paper presents a Kolmogorov-
Smirnov test to verify if two simulations of a MAS keep the same
statistical properties. Computational simulations illustrate the
efficience of the technique as a tool to verify the reproducibility
of a MAS.
I. I NTRODUCTION
Reproducibility is a key aspect to validate a computational
model. This technique consists in measuring the degree of
agreement or similarity between the results of the simulations
carried out under different computational platforms. In many
fields, modeling is a complex process that involves the ability
to deduce and inference. In engineering, one may have phys-
ical models and mathematical models. The first refers to an
arrangement of real parts and mechanisms, while the second
type involves the careful application of physical laws to obtain
a set of equations that adequately describe the behavior of the
system [1]. Individual-Based Model (IBM) has received great
attention over the past decades. This model considers each
individual to be treated as a single discrete entity with age and
at least one characteristic or property that changes throughout
the life cycle, such as social position. The relevance of these
type of model can be seen, for example, in Cho’s work [2], in
which the IBM has been used to evaluate the eradication of a
species exposed to natural and toxic survival rates.
Using the individual characteristics that are part of the IBM,
it is possible to provide more realistic simulations. On the
other hand, the IBM makes use of recursive functions which
can propagate errors in simulations. Recursive functions are
the basic mechanism for repetitions in the functional languages
that include errors propagated during the simulations [3].
978-1-5090-4886-1/17/$31.00
2017
Denise Fonseca Resende
Federal University of São João del-Rei
São João del-Rei, Brazil
denisesade@gmail.com
Márcio Júnior Lacerda
Federal University of São João del-Rei
São João del-Rei, Brazil
lacerda@ufsj.edu.br
Intended to solve this problem, in the 1970s and 1980s,
a group of scientists and engineers, led by the Institute for
Electrical Engineering and Electronics Engineers (IEEE), de-
veloped a standard for floating-point representation and arith-
metic [4] that can be applied to numerical computation [5], [6].
This guideline is a systematic approximation of real arithmetic,
so some properties of this arithmetic are not guaranteed for
the floating point [6]. Arithmetic computing is an area that
contemplates the standardization of the arithmetic system for
computers, as well as hardware issues and software implemen-
tations, testing and validations. It is an interdisciplinary area
that involves mathematics, computer science and electrical
engineering [7].
Although in many cases it portrays satisfactory and near-
expected results, numerical computation denotes still far from
being a tool that provides results compatible with reality. This
is because most of today’s computers make use of the IEEE
754-2008 standard for floating-point arithmetic, which is a
systematic approximation of the arithmetic of real numbers
and can be represented as a finite subset of the real numbers.
Consequently, some properties of the arithmetic of the real
numbers can not be assured for floating-point arithmetic [4],
[8], [9], [10].
Therefore, it is essential to understand that each number
stored by the computer actually represents a range of the set of
real numbers and it is fundamental to consider these ranges in
the operations performed by the hardware as a way of trying to
reduce or eliminate the problem of the error in the computers.
The remainder of the paper is organized as follows. The
preliminary concepts are introduced in Section II. Section
III presents the methodology. Section IV is dedicated to the
numerical experiments and Section V concludes the paper.
II. P RELIMINARY C ONCEPTS
It is known that the IEEE 754-2008 standard species accu-
racy only for storage and basic arithmetic operations [6], thus
it does not ensure absolute accuracy for solution of functions.
The numerical rounding process takes an infinitely precise
number, and if necessary modifies it to fit into the destination
format [6]. However, rounding processes can be implemented
and combined so that an infinitely precise number is adjusted
c IEEE
and aligned in order to guarantee that the inherent error of
this process is as small as possible and hence the stored value
closest to the infinitely precise result.
A. System Reproducibility
Reproducibility of systems is associated with the ability
to repeat a simulation or experiment by other researchers,
preferably exempt from the former, using different samples
and equipment. Verifying the validity of a model requires
reliable empirical results. The reliability of a result depends on
the possibility of it being reproduced under appropriate condi-
tions. In this sense, the question of reproducibility is a central
methodological factor of the contemporary sciences [11].
The lack of reproducibility proves to be a problem both in
experimentation and in system modeling. Results that can not
be reproduced do not add significant contributions to the sci-
entific researches. It is recurrent in the literature the existence
of works that have insufficient data or do not present detailed
information such as: the version of the simulation software,
the operating system, the architecture of the processor used,
among other relevant information needed to reproduce the
works [12].
B. Individual Based Model
According to Grimm [13], “each individual is treated as
a single, discrete entity having age and at least one other
property that changes throughout the life cycle, such as weight,
social position, between other”. Simulation models that de-
scribe individuals (agents) have been generally used in several
research fields [14]. The IBM is a computational tool that
allows simulation experiments, taking into account the individ-
ual characteristics and interactions between these individuals.
It considers heterogeneous populations, where characteristics
diversify between age, geographic location, social and cultural
factors and genetic characteristics for example. These factors
are ignored in other types of models. In this way, the IBM
can be expressed by a mathematical formulation, where an
individual is represented by
In,t = [Cn,1,t Cn,2,t . . . Cn,m,t ], (1)
where n is a sequential number that identifies an individual,
m is the number of features, t is the instant that the individual
presents a specific set of characteristics and Cn,m,t is a
characteristic of the individual. A population of individuals
is represented by
Pt = [I1,t I2,t I3,t . . . Im,t ], (2)
where Im,t is an individual at time t and P ∈ Rm×n is a
matrix. For proper administration of IBM, it is necessary to
adopt some premises listed below:
1) Constant population. The mortality rate is equal to the
birth rate, which is to say that d = µ.
2) Characteristics of the individual. An individual is char-
acterized by a set of n characteristics.
3) Category of the individual. There are three categories
for an individual: 0 (susceptible), 1 (infected) and 2
(recovered).
4) Statistical distribution. For mortality (and consequently
birth) the exponential distribution was adopted.
5) Infection process. It has been assumed that every contact
between a susceptible individual and an infected individ-
ual can provoke a new infected individual following a
uniform distribution.
6) Change of category.
a) 0, 1, 2 → 0. This means that the individual died
and another was born;
b) 0 → 1. A susceptible individual becomes in-
fected;
c) 1 → 2. An infected individual recovers.
This is a very general formulation that allows the elaboration
of a methodology to specify which mechanisms should be
included in the model.
C. Kolmogorov-Smirnov Test
Kolmogorov-Smirnov statistic provides a means of testing
whether a set of observations are from some completely
specified continuous distribution Fo (X) [15]. The test is
based on the maximum difference between an empirical and
a hypothetical cumulative distribution [16]. The use of the
Kolmogorov-Smirnov test for two independent samples aims
to validate if two samples come from populations X1 and X2
with the same distribution. These samples do not need to be
the same size [17].
In this article we used the kolmogorov smirnov test 2,
function of Octave Software (version 4.2.0), that performs
the Kolmogorov-Smirnov two-sample test to compare the
distributions of values in the two data vectors X1 and X2 .
The null hypothesis for this test demonstrates that X1 and X2
have the same continuous distribution. The contrary hypothesis
demonstrates that they have different continuous distributions.
III. M ETHODOLOGY
For the purposes of reproducibility of the IBM algorithm
proposed by [18], it was observed that the algorithm is
governed by stochastic relations and generates random values
for susceptible (S), infected (I) and recovered (R) individuals
of a given population. The IBM employed in the simulations
is presented in Algorithm 1. Figure 1 shows the infected
population signal graph obtained by simulating Algorithm 1
in Matlab software.
By using Matlab R2010b 64-bits software (Version
7.11.0.584) and Scilab 64-bits (Version 5.5.2) two hundred,
five hundred, one thousand and fifteen hundred sequential
simulations with three thousand iterations each were carried
out in order to obtain a satisfactory sampling for S, I and R.
The simulations were performed in two microcomputers with
the configurations presented in Table I.
It was observed that the use of 1500 simulations determine a
suitable region for the analysis of reproducibility of the model,
Algorithm 1 IBM Algorithm
1: N=1000; . Population size
2: dt=0.1; . Time interval
3: dd=1/3; . Infecting period
4: alpha=1/60; . Death rate (d=mi)
5: mu=1/60; . Life expectancy (mi)
6: ni=5*(1/alpha)/dt; . Number of iterations
7: infec=0.2; . Infection rate
8: S0=0.8*N; . Number of susceptible
9: I0=0.01*N; . Number of infected
10: R0=N-S0-I0; . Number of recovered Fig. 1: IBM (Infected Population) - Matlab Software
11: %Initial population
12: P=[zeros(S0,1); ones(I0,1); 2*ones(R0,1)];
13: P(:,2)=-(1/alpha)*dt*alpha*log(rand(N,1)); Computer Processor Clock Instruction Operational
Set System
14: P(:,3)=-(1/alpha)*log(rand(N,1)); 1 Intel Core I3- 2.1 GHz 64-bit Windows
15: indf=find(P(:,1)==1); 2310M (2 cores/4 7 Ultimate
16: nindf=length(indf); threads) X64
2 Intel Core I7- 2.8 GHz 64-bit Windows 10
17: P(indf,4)=(1/dd)*dt*dd*log(rand(nindf,1)); 4500U (2 cores/4 Home X64
18: P(indf,5)=-(1/dd)*log(rand(nindf,1)); threads)
19: %S,I,R - Initial
20: S(1,1) = sum(P(:,1)==0); TABLE I: Configuration of Microcomputers
21: I(1,1) = sum(P(:,1)==1);
22: R(1,1) = sum(P(:,1)==2);
since it is possible to obtain a mass of data with less influence
23: rest=0;
of the initial conditions and steady state. For the analysis, the
24: %Process until tf or number of iterations
interval between the time instants 200 and 1000 was used.
25: for k = 1:ni
Table II shows the different configurations considered in the
26: indd = f ind(P (:, 2) ≥ P (:, 3)); . Death step
tests:
27: P(indd,:)=[];
28: bth=dt*N(1)*mu*exp(-mu*rand(1))+rest;
29: bthf=floor(bth); Test Hardware Software Date (Dec/2016) Matrix
30: rest=bth-bthf; 1 Computer 1 Matlab/Scilab 19 1500x3001
31: if bthf > 0 . Birth step 2 Computer 1 Matlab/Scilab 21 1500x3001
3 Computer 1 Matlab 21 1500x3001
32: P(end+1:end+bthf,3)=-(1/alpha)*log(rand(bthf,1)); and 2
33: end 4 Computer 1 Scilab 21 1500x3001
34: N(k+1)=size(P,1); and 2
35: indr=find(P(:,4)>P(:,5)); . Recover step TABLE II: Tests (1500 simulations)
36: P(indr,1)=2;
37: P(indr,4:5)=0;
From the results obtained through the tests demonstrated in
38: for kk = 1:N(k+1) . Infection step
Table II, the transposition of the obtained matrices was carried
39: if P(kk,1)==0
out. The procedure was performed in the following way,
40: ind = ceil(rand(1)*N(k+1));
from the obtained matrix transposed (3001x1500), the time
41: if ind = kk %Except the same individual kk
instant 200 was selected with interval between simulations 200
42: if P(ind,1)==1 && rand(1) > 1 - infec;
to 1000, generating vectors of size 801.
43: P(kk,1)=1;
We used the tool from Octave to test if two samples
44: P(kk,4)=0;
belong to the same distribution. In the simulations of this
45: P(kk,5)=-(1/dd)*log(rand(1));
paper, the parameters X1 and X2 refer to distinct populations
46: end; end; end; end;
resulting from susceptible (S), infected (I) and recovered (R)
47: indf=find(P(:,1)==1);
individuals for each simulation of the Table II. The result H is
48: P(indf,4)=P(indf,4)+dt;
1 if we can reject the hypothesis that distributions are equal, or
49: P(:,2)=P(:,2) + dt;
0 if the hypothesis can not be rejected. We reject the hypothesis
50: %Number S, I, R
if the test is significant at the 5% level [17], [19].
51: S(1,k+1) = sum(P(:,1)==0);
52: I(1,k+1) = sum(P(:,1)==1); IV. N UMERICAL E XPERIMENTS
53: R(1,k+1) = sum(P(:,1)==2);
54: end; For the purpose of graphical analysis of the quantity or
absolute frequency of each value of S, I and R, histograms
have been generated from the data obtained in the sampling
of a thousand five hundred simulations. Figures 2-5 show
the results for the infected population (I). We chose to
demonstrate the results only of Matlab software because of its
popularity and relevance in research and engineering teaching.
Figure 2 shows the histogram obtained by means of the
vector extracted from the matrix of infected individuals (I),
instant of time 200, simulations of 200 to 1000 and β = 0.15.

Fig. 4: Histogram of Infected (I) values (Test 3 - Computer 1

/ Matlab - 21/12/2016)

Fig. 2: Histogram of Infected (I) values (Test 1 - Computer 1

/ Matlab - 19/12/2016)

Figure 3 shows the histogram obtained by means of the

vector extracted from the Matrix of infected individuals(I),
instant of time 200, simulations of 200 to 1000 and β = 0.15.

Fig. 5: Histogram of Infected (I) values (Test 3 - Computer 2

/ Matlab - 21/12/2016)

In order to verify if the samples belong to the same

continuous distribution, the function of Octave was applied,
which showed an H equal to 0 for all cases, demonstrating a
high reproducibility rate even with the variation of hardware,
software and data. It can also be verified, through the figures
presented, that the variation of the beta parameter causes
that the results no longer belong to the same continuous
distribution.
Table III shows a summary of the comparisons undertaken.
Fig. 3: Histogram of Infected (I) values (Test 2 - Computer 1
/ Matlab - 21/12/2016)
Test Hardware Software Date Result
Figure 4 depicts the histogram obtained by means of the (Dez/2016) KS Test
vector extracted from the Matrix of infected individuals (I), 1 Computer 1 Matlab/Scilab 19 0
instant of time 200, simulations of 200 to 1000 and β = 0.2. 2 Computer 1 Matlab 19 and 21 0
3 Computer 1 and 2 Matlab 21 0
Figure 5 presents the histogram obtained by means of the 4 Computer 1 and 2 Scilab 21 0
vector extracted from the Matrix of infected individuals (I),
instant of time 200, simulations of 200 to 1000 and β = 0.2. TABLE III: Results (1500 simulations)
 Table IV shows a comparison between the extended results [6] Institute of Electrical and Electronics Engineers (IEEE) (2008). IEEE 754
for different values of β: standard for floating-point arithmetic.
[7] Nannarelli, A., Seidel, P. M., and Tang, P. T. P. (2014). Guest Editors
Introduction: Special Sectionon Computer Arithmetic. IEEE Transactions
on Computers, 63(8), 1852-1853.
Test Hardware Software β Result KS Test [8] M. L. Overton. Numerical Computing with IEEE floating point arithmetic.
1 Computer 1 Matlab 0.2 / 0.15 1 Society for Industrial and Applied Mathematics, 2001.
2 Computer 1 Scilab 0.2 / 0.15 1 [9] D. Goldberg. What every computer scientist should know about floating-
3 Computer 1 and 2 Matlab 0.2 0 point arithmetic. Computing Surveys, 23(1):5.48, March 1991.
4 Computer 1 and 2 Scilab 0.15 0 [10] Moore, R. E., Kearfott, R. B., and Cloud, M. J. (2009). Introduction to
interval analysis. Society for Industrial and Applied Mathematics.
TABLE IV: Results for different values of β (1500 simula- [11] H. Atmanspacher R.G. Jahn (2003) ”Problems of Reproducibility Com-
tions) plex Mind-Matter Systems”, J. Scientific Exploration. vol. 17, pp. 243-
270.
[12] Robert McDougal, Anna Bulanova, and William Lytton. Reproducibility
in Computational Neuroscience Models and Simulations. IEEE Transac-
V. C ONCLUSION tions on Biomedical Engineering, 63(c):1-1, 2016.
We applied the Kolmogorov-Smirnov Test on two different [13] Grimm, V. (1999). Ten years of individual-based modelling in ecology:
What have we learned and what could we learn in the future?, Ecological
computers using Matlab and Scilab, well known and widely Modelling 115(2-3): 129-148.
used software. Through the analyzes performed in this article [14] Huston, M., Deangelis, D., , and Post, W. (1988). New computer-models
it can be observed that simulations of the same algorithm in unify ecological theory - computer-simulations show that many ecological
patterns can be explained by interactions among individual organisms.
Matlab and Scilab software present equivalent results. It also Bioscience, 38(10):682691.
can be seen that the studied IBM algorithm presents equivalent [15] Razali, N. M., and Wah, Y. B. (2011). Power comparisons of shapiro-
values for independent simulations of the hardware, that is, wilk, kolmogorov-smirnov, lilliefors and anderson-darling tests. Journal
of statistical modeling and analytics, 2(1), 21-33.
the simulation 1 performed in computer 1 is equivalent to the [16] Massey Jr, Frank J. “The Kolmogorov-Smirnov test for goodness of fit.”
simulation 1 performed in computer 2 and so on. Journal of the American statistical Association 46.253 (1951): 68-78.
This result shows the importance of testing the algorithm [17] Lilliefors, Hubert W. (1967). On the Kolmogorov-Smirnov Test for
Normality with Mean and Variance Unknown, Journal of the American
in different software and also the relevance of having access Statistical Association, Vol. 62, No. 318., pp. 399-402.
to the code. This is exactly what IEEE has just announced [18] Nepomuceno E. G., Takahashi R. H. C. and Aguirre L. A. (2016),
to provide in 2017, as it would be possible to view and run “Individual Based-Model (IBM): An Alternative framework for epidemio-
logical compartment models”, Revista Brasileira de Biometria. Vol. 34(1),
Algorithms in IEEE Xplore, in a new service called Code pp. 133-162.
Ocean [21]. [19] “Kstest2: Two-sample Kolmogorov-Smirnov test”, MathWorks, Dec.20,
Through the analysis of the histograms and the results 2016. [Online] Avaliable: https://www.mathworks.com/help/stats/
kstest2.html.
verified through KS Test it can be inferred that the method [20] Moore, R. E. (1979). Methods and applications of interval analysis.
used for verification of reproducibility was able to evaluate Society for Industrial and Applied Mathematics.
similar systems in the experiments with variation of hardware, [21] “Authors@IEEE Newsletter”, IEEE, Dec, 2016. [Online] Avaliable:
https://www.ieee.org/publications standards/publications/authors/
software and date and at the same time was able to identify authors newsletter archive.html.
Systems that do not have the same distribution for the case
where there was variation of the parameter of infection rate
(β).
The results presented also demonstrate the importance of
the role of the user in performing computational simulations.
As future work we intend to perform the simulation of other
models of complex systems, as well as perform the simulations
presented in this work considering other processors architec-
tures.
ACKNOWLEDGMENT
This work has been supported by the CNPq and CAPES. A
special thanks to Model and Control Group - GCOM.
R EFERENCES
[1] Felı́cio, L. C. (2007). Modelagem da dinâmica de sistemas e estudo da
resposta. RiMa.
[2] Cho, W. S., Nguyen, V. T., and Chon, T. S. (2012). Individual-Based Mod-
els Applied to Species Abundance Patterns in Benthic Macroinvertebrate
Communities in Streams in Response to Pollution.
[3] Nepomuceno, E. G. (2014). Convergence of recursive functions on
computers The Journal of Engineering, 1-3.
[4] Overton, M. L. (2001). Numerical Computing with IEEE floating point
arithmetic, Society for Industrial and Applied Mathematics.
[5] Goldberg, D. (1991). What every computer scientist should know about
floating-point arithmetic, Computing Surveys.