A Network Monitoring Suite

Håkon Flatval, Summer Student at CERN 2019, supervised by Danilo Cicalese

Norwegian University of Science and Technology, Trondheim, Norway

Abstract
During my ten weeks at CERN, I have been working on two programs related
to network monitoring. The first program is called Apoll, which polls routers
for information periodically and puts it into a database. I have also created the
Network Topology Visualizer, a web application that visualizes traffic through
a network in near real-time. These programs work together with the external
projects InfluxDB [4], for database features, and Grafana [5], for data visual-
ization.

Keywords
Summer Student report; Network Monitoring; Grafana; InfluxDB.

1 Apoll
1.1 Introduction
Apoll has been developed in the past and was already working when I arrived. It periodically polls a user-
given list of routers for information about traffic and stores this information in the time-series database
InfluxDB. The repository is located at Gitlab [1].

1.2 Goals
My primary goal has been to cut down the time between polls from 30 seconds to 5 seconds. Previously,
we experienced holes in the output data for this polling frequency. In addition, the program was in need
of proper documentation, as there was little explanation as to how the program worked from before.

1.3 Work
We suspected that one of the reasons Apoll was not able to keep it up for small poll intervals, was its
interface to the database, which is visualized in figure 1.
As can be seen, Apoll output the data to files, which then is read by a Python script that runs
periodically, putting the data into the database. We wanted to let Apoll interact directly with the database.
There is no maintained InfluxDB client for the C language, so I created a small wrapper around libcurl
[7] to enabling posting through HTTP.
A sketch of the final version of the program can be found in figure 2.

Figure 1: The structure of Apoll at the beginning of my contract

 Figure 2: The structure of Apoll at the end of my contract

In addition to coupling Apoll directly to the database, I also wrote a Python script that fetches
mac-addresses from the same routers that Apoll is polling, and look up their names in CERN’s database
for network devices [6]. These names can be used by Apoll to annotate the data with device names
instead of just the name of the router and interface number, making the data easier to interpret. This
feature is optional.
In addition to this, I have fixed several bugs, stabilized the timing between polls, and made some
small refactorizations.

1.4 Result
Apoll can now be used for polling with 5 second intervals. The intervals may be larger in times of high
latency, but can be used with this frequency much more reliably. The data inserted into the database
is annotated with names from CERN. In addition, I have written several READMEs. They describe
primarily setup, configuration and use with Grafana.

2 Network Topology Visualzer

2.0.1 Introduction
The Network Topology Visualizer is a web application made to visualize the flow of traffic through a
network topology. It reads the data that Apoll inserts, from InfluxDB. This project uses NodeJS [10] and
ReactJS [9], and is made with visual components from ESNet’s react-network-diagrams project [8].
An open-source example of how ESNet uses their own components can be vied at https://my.es.
net/. This served as our inspiration.
The source code for the Visualizer can be found at Gitlab [2].

2.1 Goal
We wanted a web application that can give intuitive understanding of how the data flows through the
network, find anomalies and detect bottlenecks. In addition, we wanted it to be directly integratable into
Grafana.

2.1.1 Work
Developing this application has been a learning experience, as my previous web development skills
were limited. Although the components were already made, and our goals largely resembled the open-
source example from ESNet, a good deal of time went to changing and understanding the code so that it

2
 Figure 3: The topology visualizer

worked the way we intended. In addition, it had to be coupled with Influx, for which we used the client
node-influx [11].
An example of the visualizer in action can be found in figure 3.
I implemented a number of features in the visualizer:

– Each edge gets a color related to the current traffic flow through the edge
– Clicking on an edge brings up the numeric traffic flow between two nodes, and the link’s capacity
– Clicking on a node shows a list of all devices connected to that node (not only other nodes, but
devices Apoll identified to be connected to the router), and the traffic to and from them
– One can switch the coloring from being dependant of the absolute value of the traffic, to depend
on the traffic flow relative to the link capacity instead.

Another feature that was a goal for this project was to have it embedded in Grafana. Grafana
has support for embedding HTML pages, which makes this requirement easier. However, we wanted a
server-less solution, with everything needed to run the webpage present in a single file. To achieve this,
I have used different packages from NodeJS that embeds the scripts, topology and stylesheets directly
into the HTML page. The InfluxDB client also runs directly from the Javascript in the HTML page, and
not on a server. Additionally, this makes it easy to switch out the Visualizer when making new versions,
since all information is present in a single, long string.

2.2 Editor
In addition to creating the Visualizer, I have also created a topology editor, so that a user can create their
own topology to visualize with the Visualizer. The repository is also on Gitlab [3]. It is based on another
example that is included in ESNet’s react-network-diagrams [8].

2.3 Result
The Network Topology Visualizer works as intended. It visualizes data from Apoll in a human-intuitive
manner and allows for different topologies to be visualized.

3 Conclusion
Together, Apoll and the Network Topology Visualizer works well together in providing information on
the traffic flowing through the network. Turning down the time between router polls to five seconds
has made it possible to react more quickly to unexpected events, and to follow the development of the
situation more closely over time.

3
 The Network Topology Visualizer also allows for easier administration of the network, detection
of bottlenecks and locating anomalies physically.
Together with the code, I have provided documentation for setup and use of the programs described
above. This will without doubt be very useful for further work on these projects.

Bibliography
[1] The Apoll repository on Gitlab,
https://gitlab.cern.ch/network-monitoring/01_apoll
[2] The Network Topology Visualizer repository on Gitlab,
https://gitlab.cern.ch/network-monitoring/03_network_topology_visualizer
[3] The Network Topology Editor repository on Gitlab,
https://gitlab.cern.ch/network-monitoring/02_network_topology_editor
[4] InfluxDB, a time series database,
https://www.influxdata.com/
[5] Grafana, primarily a time series visualizer,
https://grafana.com/
[6] CERN’s device database,
https://network.cern.ch/sc/fcgi/sc.fcgi?Action=Main
[7] The CURL library,
https://curl.haxx.se/libcurl/
[8] ESNet’s react-network-components,
https://github.com/esnet/react-network-diagrams
[9] ReactJS, a Javascript frontend library, Facebook Inc.,
https://reactjs.org/
[10] NodeJS, Javascript application framework,
https://nodejs.org/en/
[11] node-influx, InfluxDB client for NodeJS,
https://github.com/node-influx/node-influx