Experiments in computer science

2021/2022

Luís Paquete
University of Coimbra
Experiments in computer science

Contents

● Computer science as a discipline

● The scientific method and the engineering cycle
● Goals in computer science experiments
● Measurements, simulation and analytical modelling
● Properties of metrics
● Accuracy, precision and resolution
● Random and systematic errors
Experiments in computer science

What is computer science as a discipline?

"Computer science is in part a scientific discipline concerned with the empirical

study of a class of phenomena, in part a mathematical discipline concerned with the
formal properties of certain classes of abstract structures, and in part a technological
discipline concerned with the cost-effective design and construction of commercially
and socially valuable products."

P. Wegner, Research paradigms in computer science. In proc. of the 2nd International

Conference on Software Engineering, 322-330, 1976.
Experiments in computer science

Computer science as an empirical science:

● Data-collection phase: Collect large amount of data concerning phenomena

● Observed uniformities and differences between phenomena lead to classification
schemes and to (succint) models and theories on the observed data
● Phenomena in computer science: man-made entities and concepts developed in the
computer
Experiments in computer science

Graph structure in the Web

(Broder et al. 2000)

● Analysis on 200 millions pages and 1.5 billion links

● The number of links to individual pages is
distributed according to a power law
● The number of weakly connected components also
follows a power law: The Web does not depend on
the existence of nodes of high in-degree nodes
● If a path exists between two nodes, its average
length is close to 7
Experiments in computer science

The scientific method

● The scientific method is an empirical method for the construction of knowledge

Observation Hypothesis

Concrete
Prediction

Experimental
evaluation
Experiments in computer science

The scientific method

● The scientific method is an empirical method for the construction of knowledge

Falsifiability

Observation Hypothesis

Concrete

Reproducibility Prediction

Experimental
evaluation
Experiments in computer science

Computer science as mathematics:

● Development of abstractions from a class of phenomena

● Study of properties of the abstraction independently of the phenomena from which
they were derived
● Study of algorithms, computational complexity theory, formal methods, information
theory, criptography
Experiments in computer science

Quicksort
(Hoare 1959)
012345

● Quicksort is a divide-and-conquer algorithm to sort

0 12345
n numbers
● It takes O(n2) time in the worst case if the pivot is 1 2345
always the smallest element in the list
2 345

● It takes O(n log n) time on the average case 3 45

4 5
Experiments in computer science

Computer science as an engineering discipline:

● Efficient accomplishment of specific tasks and towards the development of tools that
will enable classes of tasks to be accomplished more efficiently
● Practicing engineer - takes a sequence of systematic selection or design decisions
which narrow down alternative options for accomplishing the task
● Research engineer – may use mathematics and physics in the development of tools
for the practicing engineer (but more practically-oriented than a empirical scientist or a
mathematician)
Experiments in computer science

Which sorting algorithm to use?

● A server needs to sort (in-place) 1GB of integers for each PageRank query
● The input data arrives almost sorted
● Server: 2 Intel Xeon Silver 4210R 2.4G, 10 Cores / 20 Threads, 9.6GT/s, 13.75M
Cache, 2 32GB RDIMM, 2933MT/s, 2 480 GB SSD SATA, OS Debian GNU/Linux 10
● Quicksort or merge sort?
● void qsort(void *base, size_t nitems, size_t size, int (*comp)
(const void *, const void*)) or std::sort() ?

● Java.utils.Arrays.sort() or Java.utils.Collection.sort()?
Experiments in computer science

A Keystroke-Level Model for user performance

(Card et al. 1980)

● A model that predicts the time taken by an expert

user to perform a given task on a computer
● It takes into account the time to keystroking (K),
pointing (P), homing (H), drawing (D), as well as the
user's mental state (M) and system response (R)

● The model gives a lower bound on the effectiveness

of new user interface proposals
Experiments in computer science

Engineering cycle

● Recall: Efficient accomplishment of specific tasks and towards the development of

tools that will enable classes of tasks to be accomplished more efficiently

System
Idea
design

Concrete
implementation

Experimental
evaluation
Experiments in computer science

(Research) Engineering cycle

● Recall: Efficient accomplishment of specific tasks and towards the development of

tools that will enable classes of tasks to be accomplished more efficiently
Falsifiability

System
Idea
design

Concrete

Reproducibility implementation

Experimental
evaluation
Experiments in computer science
Goals in computer science as an engineering discipline:

● Compare alternatives: provide quantitative information about which configurations

are best under specific conditions
● Determine the impact of a feature: determine the impact of adding or removing a
specific feature of the system
● System tuning: find the set of parameter values that produces the best overall
performance
● Performance debugging: apply tools and analysis techniques to determine why the
system is not meeting performance expectations
● Set expectations: set expectactions for what a system is actually capable of doing
A side note: Experiments with computers are usually very well controlled.
Experiments in computer science

Interaction with the system:

● Measurements: it provides the most reliable results since, given the necessary
measurement tools, no simplifying assumptions are required. However, direct
measurement may not be feasible for some systems.
● Simulation: a program that models important features of the system being analysed.
It can be easily modified to test different scenarios. It requires simplifying assumptions
since it may be impossible to model every detail of the system being studied, which
results in less accuracy.
● Analytical modeling: mathematical description of the system. It provides quick
insight into the overall behaviour of a system or its components. It may require even
more simplifying assumptions in order to be tractable.
Experiments in computer science

(Good) Properties of metrics:

● Linearity: the value of the metric should be proportional to the actual performance of
the system being measured.
● Reliability: a metric is reliable if system A always outperforms system B when the
corresponding values of the metric for both systems indicate that system A should
outperform system B.
● Repeatibility: a metric is repeatable if the same value of the metric is measured
each time the same experiment is performed.
● Easiness of measurement
● Consistency: a consistent metric is one which the units of the metric and its precise
definition are the same across different systems
● Independence: a metric should be independent of outside influences
Experiments in computer science

Nondeterministic measurements

● Measurements of the same system may vary significantly due to random events.
● Exmple: CPU-time of a program may be interfered by background operating systems
tasks, page and cache mappings, system load, etc. Therefore, program’s executation
time is nondeterministic.
● A way of dealing with nondeterministic metrics is to repeat the measurements
several times and report a value that summarizes the values collected (indice of
central tendency) and other that indicates how dispersed they are (variability).
Experiments in computer science

Characteristics of measurement tools

● Accuracy: absolute difference between a measured

value and the corresponding reference value
(usually, an agreed-upon standard)
● Precision: closeness of measurements to each
other.
● Resolution: smallest incremental change that can
be detected and displayed
Experiments in computer science

Errors in measurements

● Systematic errors are due to an experimental

"mistake” that introduces bias into the
measurements. They affect the accuracy of the
measurements.
● Random errors are unpredictable, nondeterministic
and need not be controllable. They are unbiased
since a random error has an equal probability of
either increasing or decreasing a measurement.
They afect the precision of the measurements.
Experiments in computer science

Observer effect

● Disturbance of an observed system by the act of

observation
● Measuring CPU-time implies increasing CPU-time
of the observed process.
● It may induce systematic error
Measurements

A gaussian model of errors

Assume that a single source of random error can change the value measured for
x by +E or –E with equal probability

Error Measured value Probability

–E x–E 1/2
+E x+E 1/2
Measurements

A gaussian model of errors

Assume that a two sources of random error can change the value measured for x
by +E or –E with equal probability

Error 1 Error 2 Measured value Probability

–E –E x – 2E 1/4
–E +E x 1/4
+E –E x 1/4
+E +E x + 2E 1/4
Measurements

A gaussian model of errors

● This produces a binomial distribution for the

possible measurements – for large n, this
distribution approximates a normal distribution.
Experiments in computer science

Recap:

● Computer science can be seen as a discipline that uses empirical, mathematical and
engineering paradigms
● From engineering perspective, the goals in computer science are: to compare
alternatives, to determine the impact of a feature, to perform system tuning, to identify
relative performance, to debug performance, to set expectations
● Techniques available: measurements, simulation or analytical modelling. Each one has
its own pros and cons.
● Metrics have several properties
● In case of nondeterministic measurements, take care of accuracy, precision and
resolution. Errors may affect accuracy and precision.
Experiments in computer science

Suggestions:

● Which paradigm are you following for your thesis project?

● Can you take direct measurements on your experiments or do you have to rely on
simulation and/or analytical modelling?
● Which are the properties of your metrics?
● Do you expect to have systematic errors on your experiments? How to handle them?
● Exercise 1 in UCstudent (identify independent and dependent variables)
Experiments in computer science

References:

● D.J.Lilja, Measuring computer performance, Cambridge University Press, 2002 (see

chapter 1)
● P. Wegner, Research paradigms in computer science. In proc. of the 2nd International
Conference on Software Engineering, 322-330, 1976
● A. Z. Broder, R. Kumar, F. Maghoul, P. Raghavan, S. Rajagopalan, R. Stata, A. Tomkins,
J.L. Wiener, Graph structure in the Web. Computer Networks, 33(1-6): 309-320, 2000
● S. K. Card, T. P. Moran; A. Newell, The keystroke-level model for user performance time
with interactive systems. Communications of the ACM, 23 (7): 396–410, 1980
● C. A. R. Hoare, Algorithm 64: Quicksort, Communications of the ACM, 4 (7): 321, 1961