ri

vHen k
pada
a
ls
Malk

JÄMFÖRELSE AV LATENCY MED OLIKA

TRANSPORT LAYER API:ER I UNITY

COMPARING LATENCY WITH DIFFERENT

TRANSPORT LAYER API:S IN UNITY

Examensarbete inom huvudområdet Informationsteknologi

Grundnivå 30 högskolepoäng
Vårtermin 2022

Arvid Karlsson
Tarik al Tayar

Handledare: András Marki

Examinator: Yacine Atif
 Abstract
This study aimed to examine transport layer API:s for the game development software Unity, and
conclude its effect on latency. This effect was examined by conducting a controlled experiment,
whereas three transport layer API:s, KCP, Telepathy, and Steamworks.NET were active on three
different open-source Unity online games. The underlying network infrastructure Mirror was
integrated to utilize each transport layer API, which also provided the components necessary to
capture latency at runtime. In a second unstable connection experiment, the packet loss rate of 30%
was configured to analyze the transport layer API:s performance during a poor connection. Although
previous work has analyzed Mirror, the corresponding transport layer API within Mirror, and its
effect on latency, have not been researched. The results suggest that Steamworks.NET achieves a
significantly increased latency compared to KCP and Telepathy, though Telepathy only significantly
increases from KCP under unstable network conditions.
Keywords: Latency, Networking, Unity, API
 Table of content
1 Introduction ..................................................................................................................................... 1
2 Background ..................................................................................................................................... 2
2.1 Latency ..................................................................................................................................... 2
2.2 Transport layers ......................................................................................................................... 3
2.3 Unity ........................................................................................................................................ 5
2.4 API: s ....................................................................................................................................... 5
2.5 Mirror ....................................................................................................................................... 5
2.6 Transport layer API:s ................................................................................................................. 7
2.7 Clumsy 0.2 ................................................................................................................................ 7
3 Related work ................................................................................................................................... 8
4 Problem ........................................................................................................................................... 9
4.1 Aim .......................................................................................................................................... 9
4.2 Methodology ........................................................................................................................... 10
4.3 Approach ................................................................................................................................ 11
5 Implementation ............................................................................................................................. 12
5.1 Setting up Unity....................................................................................................................... 12
5.2 Integrating Mirror .................................................................................................................... 12
5.3 Scenes .................................................................................................................................... 14
5.3.1 Pong ............................................................................................................................... 15
5.3.2 Tanks .............................................................................................................................. 16
5.3.3 Benchmark ...................................................................................................................... 16
5.4 API integration ........................................................................................................................ 17
5.4.1 Configure Custom NetworkManager ................................................................................. 17
5.5 Experiment setup ..................................................................................................................... 18
6 Result ............................................................................................................................................ 20
6.1 Raw data ................................................................................................................................. 20
6.2 Stable connection experiment analysis ....................................................................................... 21
6.3 Unstable connection experiment analysis ................................................................................... 24
6.3.1 Quality of sample............................................................................................................. 25
6.4 Summary of experiments .......................................................................................................... 25
7 Evaluation ..................................................................................................................................... 26
7.1 Conclusions............................................................................................................................. 26
7.2 Discussion ............................................................................................................................... 26
7.2.1 Societal benefit ................................................................................................................ 27
7.3 Future work ............................................................................................................................. 28
8 Ethics............................................................................................................................................. 30
8.1 Internal validity ....................................................................................................................... 30
8.2 External validity ...................................................................................................................... 31
9 References ..................................................................................................................................... 32
1 Introduction
Videogames that provide multiplayer services require a robust, scalable, and fast processing
architecture to ensure availability and reliability for the users. Good network infrastructure is
considered one of the most crucial points in any development of this theme, not only for players’
in-game experience but also for the company’s ability to stay in business (Sun & Leu, 2011).
One element that directly affects the playability of a multiplayer game and causes concern over the
design process is latency. Latency largely depends on the geographical distance between the player
and the server, and this delay between input and response is critical to be kept low to make the game
playable (Sun & Leu, 2011). Further on, a game also consists of visual and auditory content to make
the experience more enjoyable. For multiplayer games, these properties are also sent across the
network, adding overhead that must be considered when developing an online game (Apel, 2018).

The game engine Unity provides a powerful editor that allows anyone to develop games for free
(Menard & Wagstaff, 2014). A big part of Unity’s success is its ability to import and use third-party
assets to found complex services with little to no configuration needed from the developer. Such a
service is could be networking functionality to establish multiplayer (Menard & Wagstaff, 2014).
Although Unity already has built-in networking functionalities, many third-party assets expand these
to a wider scale of use (Stagner, 2013).

Mirror is one such infrastructure with over 100 000 downloads on Unity’s official community market
(Vis2k, 2021a). A few articles have measured the performance of Mirror and other networking
infrastructures (Polančec & Mekterović, 2017; Novotny et al., 2020; Sanchez, 2020). However, these
sources of related work did not consider which transport layer API that was configured within the
network infrastructure. The transport layer is an architectural layer of an online purpose system that
manages the real-time communication between the end-users (Taha Al-Dhief, et al., 2018). Its task is
to send and receive data and transfer this to and from the application layer.
In Mirror, the transport layer is an empty object that developers must configure themselves by
integrating a transport layer application programmable interface (API:s). There is a large supply of
transport layer API:s available for network infrastructures such as Mirror, but no research have
compared the different options in terms of what latency they achieve (Vis2k, 2021b).

This paper compared three Unity transport layer API:s which were applied to open-source games
using the third-party network infrastructure Mirror. Latency was measured by conducting controlled
experiments, to then calculate if there was a significant difference in latency between each transport
layer API. In addition, packet drop rate was manually increased using the tool Clumsy 0.2 to see if the
choice of transport layer API was relevant for the players with a poor connection as well.

The research provided in this article is beneficial for developers that aim to integrate a multiplayer
service into their Unity-based games. Using our research, a development team gets knowledge on
what latency each API achieves in multiple themes of games. This has the potential to act as a guide
during the design process which consequently saves the company time and money (Apel, 2018).
Similarly, the paper can be a guideline for further research regarding network infrastructures in Unity.

1
2 Background

2.1 Latency
In the book Distributed Systems. Concepts and Design (Coulouris, et al., 2012), latency is defined as
the time it takes in milliseconds for a transmitted network package to reach the destination node.
According to Kelly & Kumar (2021), a good rating of latency in multiplayer games is in the range of
10-20ms. At the same time, latency fluctuation of just a few milliseconds can go unnoticed by the
human brain. It is generally when latency is above 100ms that a human brain considers the game
unplayable (Kelly & Kumar, 2021).

Coulouris et al. (2012) mention multiple factors that affect latency. One factor is the underlying
network technology. These are specifications such as the download and upload speed of the network.
Also, if the connection is routed through the internet rather than a local area network, the latency can
be increased a great amount (Coulouris, et al., 2012). The same is said in the book Unity Networking
Fundamentals. Creating Multiplayer Games with Unity (Kelly & Kumar, 2021). When network
packets traverse the destination through the internet, they are managed by the recipient’s Internet
Service Provider (ISP). Some ISP:s consume more steps in order to move the packets to respective
router, resulting in an added latency for the ISP subscribers. This is known as the Last-Mile Latency
and it is not present if the client and server connect locally on the same router since no interference
with the ISP is made (Kelly & Kumar, 2021).

Coulouris et al. (2012) further elaborate on what fluctuates latency. For instance, the read and write
speed of the client’s and the server's hardware determines how fast a requested resource can be
transmitted. Also, short bursts of high latency can be experienced if transmitted packets are lost
during their travel to the destination (Coulouris, et al., 2012). If the underlying transport-layer is
reliable then the lost packet is retransmitted all over again, consuming more milliseconds to resend.
On the other hand, if the transport layer does not ensure reliability, then the lost packet will go
unnoticed and the receiving end experience a delay from listening for the next packet (Coulouris, et
al., 2012).

In the article Reducing Development Overheads with a Generic and Model-Centric Architecture for
Online Games (Apel, 2018), the multiplayer integration process of a Massive Online Multiplayer
Game (MMOG) is documented.
Apel (2018) writes that a considerable amount of effort for planning, creating, and operating the
network infrastructure is present when developing a game. Apel (2018) states that a working online
game is a symbiosis between the network infrastructure and the game-mechanic itself.
One aspect of the architecture and the network infrastructure that directly affects the latency of the
game is the overhead. Overhead is described as a part of the network infrastructure's effort to
communicate with the player, which creates overhead when not-so-relevant information is sent
between the nodes (Apel, 2018). This subsequently adds more information that needs to be processed
which results in an increased latency (Apel, 2018).
There are many features of a game that need to be transmitted multiple times every second to give the
player the experience of the game. For example, graphics and physics rendering, audio, game content
and authentication (Apel, 2018). The latency of the game is therefore dependent on how optimized the
game engine exchanges these features to the network infrastructure (Apel, 2018).

2
Similarly, Coulouris et al. (2012) write that overhead increases the risk of packet loss, a term for when
sent networked packets disappear somewhere on the route to the destination node.
Other causes for packet loss are also mentioned, such as if the message buffer at the receiving node is
temporarily full which means that a buffer overflow occurred (Coulouris, et al., 2012).

In the paper Deploying a massively multiplayer online game with a low-latency (Sun & Leu, 2011),
the geographical distance between the client and the server is claimed to have the most significant
effect on latency. To utilize this factor, it is common for gaming companies to develop their
multiplayer service on top of a geographically distributed architecture. This means that the serving
node consists of many servers distributed throughout the world so that the connecting client is
assigned to the geographically closest serving node (Sun & Leu, 2011).
According to the authors, this architecture is prone to enhance scalability and decrease latency.
However, dividing clients into different servers adds complexity to the infrastructure and increases the
risk of network failures such as packet loss (Sun & Leu, 2011).

2.2 Transport layers

The transport layer of a system is the architectural layer responsible for delivering and receiving
messages across the network (Coulouris, et al., 2012). More specifically, messages acquired from the
upper layers of the system are divided into networked packets. In the same way, the transport layer
listens (asynchronously or synchronously) for packets from nodes over a network and assembles them
back into messages which can be managed by the upper layers. There are multiple ways a transport
layer could proceed to do so, which is determined by its protocol properties.

As specified by Performance Comparison between TCP and UDP Protocols in Different Simulation
Scenarios (Taha Al-Dhief, et al., 2018), Transmission Control Protocol (TCP), and User Datagram
Protocol (UDP) are two of the core transport layer protocols used for communication between
application processes that are executing on different network hosts.
TCP is a connection-oriented protocol that communicates with clients using an end-to-end structure.
A property of TCP is that a dedicated communication line must be established between the server and
the client before packets can be sent (Taha Al-Dhief, et al., 2018). Establishing the connection
requires additional resources, which consequently slows down the speed of the transmission.
However, this feature ensures the reliability of the connection. In case of packet loss, TCP resends it
again, and the client-server connection remains established till the exchange is complete. Once a
connection is established using TCP, the corresponding operating systems will have to track ongoing
communication with more threads than that of a UDP connection. Therefore, a TCP connection
requires more resources from the processor as well (Taha Al-Dhief, et al., 2018). The overhead
elements of TCP are visualized in figure 1.

3
 Figure 1 TCP packet header

UDP is a connectionless protocol that sends datagrams through the network. Unlike TCP, datagrams
can be transmitted directly to a node socket without the need to establish a communication line first
(Taha Al-Dhief, et al., 2018). Performance-wise, this attribute makes UDP faster than TCP. In
contrast, UDP does not guarantee data delivery since lost packets are not automatically handled by the
protocol (Taha Al-Dhief, et al., 2018).

Figure 2 UDP Datagram header

Because UDP does not consider delivery, as well as ordering or duplication protection, it requires less
overhead (see figure 2). Therefore, a UDP datagram packet can consume fewer bytes compared to a
TCP packet, consequently making it faster to process which reduces latency for the client (Taha Al-
Dhief, et al., 2018).

In the article Analyzing the Effect of TCP and Server Population on Massively Multiplayer Games
(Suznjevic, et al., 2014), the TCP and UDP transport effects in multiplayer games are tested and
evaluated. Suznjevic, et al. (2014) state that it is a splinter group of games that use TCP as a transport
layer. It is not convenient to design multiplayer services that way, they elaborate (Suznjevic, et al.,
2014). The games that have succeeded with a TCP design all use a similar client-server architecture
whereas the client holds all application logic and 3D rendering, while the server only communicates
updates of specific entities for the clients. This way the number of transmitted packets, as well as the
overhead, can be minimized which is prone to decrease latency (Suznjevic, et al., 2014).
According to a more recent article, however, TCP have become a standard protocol within cloud
gaming services (Wu, et al., 2017). On the other hand, the authors describe barriers using TCP in
gaming. Latency is generally higher while packet loss also leads to latency spikes due to the
retransmission property of TCP (Wu, et al., 2017).

4
Another common transport protocol is a relay transport which is utilized on Peer-To-Peer (P2P)
networks (Coulouris, et al., 2012). In their book, Coulouris et al. (2012) writes that P2P systems aim
to share data on a large scale without having to manage each node separately. Instead, each peer (user)
contributes some resources to the P2P network. The process load is dynamically balanced among all
nodes in the systems. Meanwhile, all peers have the same functional capabilities and responsibilities
as everyone else, hence no managing server is needed, which is the case for UDP and TCP
connections.

2.3 Unity
As described in Game development with Unity (2nd. ed.) (Menard & Wagstraff, 2014), Unity
workspace provides an integrated game engine and a powerful editor that has been designed to
simplify development. For example, the developer can drag and drop components that contain
features or implement own features using C# scripts. Also, Unity offers services to developers like the
asset store, which is a web marketplace where the community can download and use premade assets.
These assets are free or paid resources that can be imported to the project. These resources could be
scripts, textures, components, or 3D models (Menard & Wagstaff, 2014).

2.4 API: s
Unity’s use of an Application Programming Interface (API) is described by Menard & Wagstraff
(2014) as an interface implemented under the hood of different systems or applications to allow
services to interact. For instance, an API could contain script libraries, functions, or data structures
that manages a specific service of the system. (Menard & Wagstaff, 2014). In the publication Unity
Multiplayer Games (Stagner, 2013), Unity’s networking API:s are addressed. Although Unity has a
built-in API for multiplayer development, there are many third-party API:s that expand these
functionalities to give the developer a greater area of use for multiplayer integrations (Stagner, 2013).

2.5 Mirror
Mirror is a free underlying network infrastructure for Unity. It is on top of the lower-level transport
real-time communication layer (Vis2k, 2021c). This API offers plenty of functionalities like the ones
provided by Unity’s built-in networking features. However, with Mirror, these have been extended
with more options to choose from (Vis2k, 2021c). In general, the provided functionality covers
message handling, serialization, and support for distributed systems. What makes Mirror unique is
that developers don’t have to worry about the ability to handle local clients and remote clients
differently. Unlike Unity’s built-in networking capabilities, Mirror automatically detects client nodes
and server nodes, thus no separate code needs to be written for each type of connection (Vis2k,
2021c). This is visualized as the “Connection management” layer of the Mirror infrastructure (see
figure 3). Fewer threads created for code assembly mean less overhead which is beneficial to
achieving low latency (Taha Al-Dhief, et al., 2018).

5
 Figure 3 Mirrors network infrastructure

In addition, Mirror serves as a server authoritative system (Vis2k, 2021c). Therefore, the developer
can decide what authority the client and the server share when it comes to synchronized events. For
instance, the developer can choose if connected clients should have authority to move their player
models on their local computer to then synchronize this position to everyone. Otherwise, each client
must tell the server to do the transition. In the latter case the server does all calculations which adds
security against cheating (Vis2k, 2021c). This flexibility of authority affects latency as well. Kelly &
Kumar (2021) concluded that a client without any authority to move and synchronize independently
introduces unacceptable miscalculations due to delay between the client and the server, at the same
time as it doubles the number of transmitted packages needed for events to be triggered.

Another benefit of Mirrors authoritative features is that it allows a player to act as a server and a client
at the same time, also known as a host (See figure 4) which decreases the necessary work to run the
game online (Vis2k, 2021c). The possibility still exists to implement a stationary server, but this
requires additional configuration in Unity.

Figure 4 Mirrors Authoritative system

6
2.6 Transport layer API:s
On the “Low-level API” layer of Mirrors infrastructure (see figure 3), a transport layer API must be
assigned. Mirror includes KCP and Telepathy which follows the UDP and TCP protocols, but other
transport layer API:s can be imported manually (Vis2k, 2021b).
KCP is the default transport-layer API included in Mirror which offers a UDP connection (Wei,
2022). The developer of the API, Wei (2022), claims that KCP reduces latency by 40% compared to a
TCP utilized protocol. On the other hand, manual configuration of IP address and port forwarding are
required to enable clients to connect through the internet (Novotny, et al., 2020).

Telepathy is also a transport-layer API included with Mirror. However, it is intended to handle
massively multiplayer online games with TCP protocol (Vis2k, 2021d). The ability to handle many
concurrent users is possible thanks to its reduced overhead architecture. Despite this, one
disadvantage is that it uses one processing thread for each node that the server, host, or client
communicates with (Vis2k, 2021d). As motivated by Taha Al-Dheif et al. (2018), this requires more
resources from the processor which can add latency between input and response. Like KCP, this
transport layer API also requires IP address and port forwarding to allow clients to connect (Novotny,
et al., 2020).

Steamworks.NET is yet another transport layer API for Unity that creates a Peer-To-Peer (P2P)
network for connecting with other users efficiently (Novotny, et al., 2020). Furthermore,
Steamworks.NET follows a relay transport layer protocol that utilizes the P2P network (Vis2k, 2021b).
The transport layer is a C++ Steamworks API converted to a C# library to provide integration with
Unity (Labrecque, 2022). Unlike KCP and Telepathy, Steamworks.NET provides multiplayer on a
global level without having to configure IP addressing and port forwarding. This is possible thanks to
Steamworks.NET’s asynchronous communication with geographically distributed servers that are
owned by Valve Incorporate (Labrecque, 2022).

Steamgames (2022), provides documentation for the use of their API. The documentation provides all
necessary as well as optional functions that are included (Steamgames, 2022). One such necessary
feature is the callback functionality. Callbacks are functions that retrieve data asynchronously from
the Valve Incorporates geographically distributed servers. Each connection must listen for changes
that could occur at any time in the online session, such as whenever a client joins or leaves a session
(Steamgames, 2022). Based on research made by Sun & Leu (2011), this introduces complexity to the
network infrastructure, which could increase latency.

2.7 Clumsy 0.2

The software Clumsy 0.2 utilizes as a filter to capture network packets and then manipulate them on
demand (Tao, 2022). There are several forms of packet manipulation, such as latency, packet loss, and
packet duplication. By doing so one can evaluate games and applications on poor connections (Tao,
2022). Clumsy 0.2 has previously been used in a related work where Kazakov (2021) proposed a
method to solve the problem of sending user input in wrong order in cloud gaming software. By
introducing packet loss functionality of 10%, 25% and 50% the problem could be simulated and
analyzed (Kazakov, 2021).

7
3 Related work
The official Unity page for networking offers all kinds of tutorials to help new Unity developers
understand multiplayer integration (Unity Technologies, 2021). Moreover, Polančec and Mekterović
(2017) discussed in their paper Developing of a prototype of a MOBA game using the Unity 5 game
engine how they integrated third party components like the Photon Unity Networking (PUN). Like
Mirror, PUN is an underlying network infrastructure for online game development in Unity that
synchronizes different features over multiple clients (Polančec & Mekterović, 2017).

In his master thesis, Wang (2017) argued Unity’s built-in networking framework is too low-level for
developing multiplayer online games. The author therefore proposed a demo project for multiplayer
integration in Unity using the underlying network infrastructure Ice RPC. However, the demo did not
consider latency nor the increase in user volume and consequently, it did not provide a good
availability (Wang, 2017).

Novotny et al. (2020) developed a framework in Unity for Multi-User VR Experiences. In addition,
the authors described useability of several networking libraries such as Mirror, Steamworks.NET, and
PUN. The framework was built with Mirror, and Steamworks.NET was used as transport layer API
for connecting to other players. Novotny et al.’s (2020) conclusions suggested that Mirror could be
used for multiplayer integration in Unity, while Steamworks.NET is a preferable transport layer API
to choose. However, the work did not compare any other transport layer API with Steamworks.NET
(Novotny, et al., 2020).

In his graduation report Realtime Networking Technologies for Unity, Sanchez (2020) researched
different networking topologies for Unity Realtime multiplayer games. The author provided an in-
depth comparison between the networking infrastructures Mirror and PUN. Sanchez (2020)
showcased the pros and cons of these, stating that Mirror was the best choice based on its great
usability (Sanchez, 2020). Sanchez (2020) did not conduct any experiment, such as measuring latency
for each infrastructure, but instead relied on documentation to determine their differences. In addition,
fundamental networking concepts that relates to multiplayer development were described as a part of
the report. To exemplify, Sanchez (2020) stated that relay transport layers (utilized on P2P network)
have an increased latency when many users are connected since messages need to pass all connected
clients in the P2P network until it reaches the destination node (Sanchez, 2020).

In the report GameOn: p2p Gaming On Public Transport (Zhang et al., 2015), latency was measured
while playing mobile online games using a P2P connection. The network infrastructure API GameOn
allowed multiple smartphones to join a P2P network through a Wi-Fi medium known as Wi-Fi direct.
Their study concluded that latency rapidly increases when the number of connected clients increased,
which also created gameplay issues such as packet loss to be present. One focus of their research was
environmental effects, which could have affected the latency since the P2P network connection was
wireless (Zhang, et al., 2015).

8
4 Problem
Developing multiplayer games suited to handle many users’ concurrency is no easy task. In their
publication, Apel (2018) states that a considerable effort for planning must be made before the
integration of multiplayer can begin. In addition, the developers must question how topics such as
graphics, audio, physics, and rendering shall be exchanged and visualized over the network (Apel,
2018). At the same time, players want a low latency, with a good range being 10-20ms (Kelly &
Kumar, 2021).
The game development software Unity provides different networking infrastructures that can integrate
multiplayer, such as Mirror or PUN (Stagner, 2013). These infrastructures require a transport layer
API to work. There are many transport layer API:s available for installation, whether it is UDP, TCP
or a Relay protocol (Vis2k, 2021b). Despite this, no research has compared them in terms of what
latency they achieve.

Novotny et al. (2020) developed a multiplayer framework in Unity. They implemented the underlying
network infrastructure Mirror with the transport layer API Steamworks.NET, and even compared
Mirror with other infrastructures such as PUN. These assets are well known in the Unity communities,
having an average of 100 000 installations each (Vis2k, 2021y; Hana Digital, 2021; Exit Games,
2022). However, Novotny et al. (2020) did not consider what transport layer API they had attached
within the network infrastructure. Had they used another transport layer API their framework might
have performed differently since different transport layer protocols have different properties (Taha Al-
Dhief, et al., 2018)

Stepping into the game-making territory, it is easy to notice the variation of valuable networking
transport layer API:s that can be used. In this study, a variety of different transport layer API:s are
benchmarked and analyzed regarding latency. Our conclusions can orientate developers who have not
yet decided which transport layer API to implement into their game. Also, the conclusions answers if
the choice of transport layer API is different for players with unstable network.

4.1 Aim
This study aimed to compare the Unity transport layer API:s KCP, Telepathy and Steamworks.NET.
The transport layer API:s were compared to determine if there was a significant difference in latency
from playing a set of Unity multiplayer games. This experiment was done two times. The first time
with a stable connection and the second time with a packet loss rate of 30%.

Research Questions
• RQ1: How do transport layer API:s compare to each other in terms of latency when
connection is stable?
• RQ2: How do transport layer API:s compare to each other in terms of latency when
connection is unstable?
• RQ3: Does the type of game influence latency regardless of transport layer API?

9
Objectives
The following objectives had to be fulfilled to satisfy the study and answer our research questions:
● Identify transport layer API:s that are compatible with Unity.
● Integrate multiplayer to a set of Unity games.
● Measure latency of every game using each transport layer API, with and without packet loss.
● Analyze the results.

Hypothesis
Transport layer protocols, whether it is UDP, TCP or Relay, have different approach to manage
communication with processes over the network (Taha Al-Dhief et al., 2018; Sanchez, 2020).
Therefore, testing Unity multiplayer games with the transport layer API:s KCP, Telepathy and
Steamworks.NET would result in different mean latencies. This motivates the first hypothesis H1:

• H1.0: There is no significant difference in latency when utilizing different transport layer
API:s in Unity online games.
• H1.1: There is a significant difference in latency when utilizing different transport layer API:s
in Unity online game.

Protocol differences are also relevant when encountering networking issues such as packet loss. For
instance, the UDP handles lost packets differently than TCP (Taha Al-Dhief, et al., 2018).
Furthermore, Relay transport layers have an increased complexity due to its architecture, which
increases risk of failure according to Sun & Leu (2011). The second hypothesis is therefore:

• H2.0: There is no significant difference in latency when packet loss is introduced.

• H2.1: There is a significant difference in latency when packet loss is introduced.

Lastly, Apel (2020) mentioned elements that affect latency, regardless of what transport layer API is
being used. If this is the case then the transport layer API:s should achieve different mean latencies
between games as well, which motivates our third hypothesis:

• H3.0: The type of game has no impact on latency.

• H3.1: The type of game has an impact on latency.

4.2 Methodology
A controlled experiment was conducted with stable network connection, where a locally connected
client monitored latency. Each selected Unity multiplayer game was run one at a time with each of the
integrated transport layer API:s, making a total of nine tests. This procedure was done a total of four
times to get reliable results.
For the experiment with unstable connection, the same controlled experiment was conducted, but with
a treatment of having 30% packet loss. For both experiments, the independent variable was the
transport layer API which purpose has been to send and receive networked packets, while the
dependent variable was the measured latency.

Motivation
In their book Experimentation In Software Engineering, Wohlin et al. (2012) listed some of the
possible methodologies and strategies to conduct research. Moreover, the authors showcased some
strategies and how they can be applied (Wohlin, et al., 2012).

10
Wohlin et al. (2012) showcased case studies which can be used to investigate an instance of a
phenomenon, individual or a group of people in the context of real life. In a case study, the research
variables are selected and not sampled.
If we were to use case studies, we could have let participants play a set of Unity online games, testing
each transport layer API, and rate the playability. For this study’s purpose however, this would have
been difficult to draw conclusions from since latency differences is a matter of milliseconds.

Surveys or interviews are two alternative methodologies that Wohlin et al. (2012) showcased. For
these methods, a sample of the population answers a list of questions to develop generalized
conclusions about a certain product or service (Wohlin, et al., 2012). Such methodologies would
reflect how the population, based on a sample, felt about latency in Unity multiplayer games. For
accurate measurements regarding latency, a controlled experiment was preferred. A controlled
experiment investigates the cause-effect relationship of variables (Wohlin, et al., 2012).
By conducting a controlled experiment, latency can be sampled which makes it easier to generalize
through statistical representations. Furthermore, this methodology suited the study because it provided
more freedom than the other strategies in terms of manipulating the independent variables to our
control. Therefore, the dependent variables were measured while the independent variable was
changed or applied with different treatments. This way, data could be compared statistically and
provide answers for the research questions.

4.3 Approach
Sanchez (2020) mentioned several networking infrastructures opted for Unity multiplayer. For this
study, we used Mirror as the underlying network infrastructure. The choice is motivated because of
the following reasons:
1. First and foremost, Mirror is free with no need for extra subscriptions or fees, unlike PUN
which cost 85€ on the Unity market (Exit Games, 2022).
2. Sanchez (2020) praised Mirror for being the best choice regarding the usability aspects for
new developers.
3. Novotny (2020) implemented and used Mirror as a network infrastructure, so there’s an
indication that the service should work.

Additionally, Novotny (2020), used Steamworks.NET as a transport layer API which motivated this
study to have Steamworks.NET as one of the independent variables. KCP was chosen as another
independent variables since it is the default transport layer API in Mirror following a different
transport layer protocol than Steamworks.NET (Vis2k, 2021b).
Suznjevic et al. (2014) stated that there is a splinter group of games that use TCP in multiplayer
games. A more recent study, however, argues that TCP have become a dominating protocol in cloud
gaming services (Wu, et al., 2017). Therefore, the TCP utilized transport layer API Telepathy was
included as the third independent variable.

To answer the second research question RQ2, the unstable connection experiment relied on the tool
Clumsy 0.2. Researcher Kazakov (2021) used Clumsy 0.2 in a similar study whereas the packet loss
rate was increased to 10%, 25% and 50% at separate occasions (Kazakov, 2021). This indicated that
the tool had quality. However, the number of treatments had to be reduced to be suitable for a study of
this theme. Therefore, Clumsy 0.2 only operated with 30% packet loss during the unstable connection
experiment, which is somewhat the average of Kazakov’s (2021) configuration.

11
5 Implementation
5.1 Setting up Unity
Unity version 2020.3.26f was downloaded. The choice of version would not matter in practice, as
long as the version is 2019.4.28 or a later release. In theory, however, every update to Unity will
amend some controlling which indirectly could affect the execution of each test.
Version 2020.3.26f was selected since it is a Long Time Support (LTS) version, meaning that it is a
merge of all improvements that have been done throughout the year. This further ensured that no
compilation defect was present during the experiment, which would be a threat to the internal validity
otherwise. To set up an experimental environment a new Unity 3D project was created with the
default settings.

5.2 Integrating Mirror

The latest supported version of Mirror, version 57.0.0, was downloaded to establish the baseline
network infrastructure. Once downloaded, Unity automatically imported the addon and assembled it
into the project. In the following section, all Mirror components that were necessary for the Unity
project are addressed.

NetworkManager
A NetworkManager component is a class provided by Mirror that controls all core aspects of
multiplayer sessions. The aspects include game states, spawning, and scene management.
Even though the NetworkManager component operates on the Game Control layer in Mirrors network
infrastructure (see figure 3), the manager needs a transport layer API reference to communicate to the
low-level API transport layer. Hence, there is a slot available for the developer to choose an
appropriate transport layer API (see figure 5).

Figure 5 NetworkManager component

12
After all transport layer API:s were integrated, the transport layer slot was switched to the
corresponding API we measured.
Tick-rate is the frequency of network updates per second. A higher value means that networked
information such as player positions gets more precise, while a low tick-rate can be incorrectly
interpreted as a high latency by the player (Coulouris, et al., 2012). As already mentioned, latency is
strictly the time it takes for packets to traverse the networks between the client and the server
(Coulouris et al., 2012). Tick-rate was therefore not of interest for this study and maintained the
default value of 30.
The Network Address field is a string variable for what the default IP address will be once a game is
launched. As suggested by Colouris et al. (2012), latency could be decreased if the client connect to a
host on a local area network. Moreover, Kelly & Kumar (2021) states that the internet service
provider (ISP) determines the Last-Mile Latency depending on how they route packets internally
before sending them to the internet. Therefore, every host and client connected locally for this study,
meaning they had the same ISP which excludes different Last-Mile Latencies from interfering with
the validity of measurements.
The default value of localhost could remain in the Network Address field, while only the client node
had to enter the host computers IP address.
Similarly, no authenticators had to be assigned. As stated on Mirrors official website, Mirror is an
authoritative system that allows a connected client to act as the server (Vis2k, 2021c). Because this
research only focused on doing experiments in a controlled environment, there was no need to add
extra security to who can access the server and how. This decision ceased the possibility of adding
latency, which is a problem described by Kelly & Kumar (2021).

Setting up CustomNetworkManager
Each game required a custom NetworkManager that handled all creations of sessions through buttons.
This was done by adding a new script to our Unity project. The script was attached under our
NetworkManager component to keep a reference to its current NetworkManager (see figure 6).

Figure 6 CustomNetworkManager component

13
The script inherited all API functions and variables were provided with Mirror to allow network
communication. To do this, using Mirror was added and Unity’s default MonoBehaviour was changed
to Mirrors NetworkBehaivour (see figure 7).
using UnityEngine;
using Mirror;

public class CustomNetworkManager : NetworkBehaviour {

// Network functionality
}

Figure 7 CustomNetworkManager script

Any implementations to the CustomNetworkManager script needed to be configured with account to

what transport layer API Mirror would use. We therefore returned to this script once all transport
layer API:s were integrated. Our goal was to let users interact with buttons to call functions in the
CustomNetworkManager script. The functions could then communicate with the NetworkManager
component, which had access to the low-level transport layer API through Mirrors
NetworkBehaviour.

NetworkPingDisplay
To measure latency, a component called NetworkPingDisplay was added next to our
NetworkManager component (see figure 8).

Figure 8 NetworkPingDisplay component

This component displayed the latency in the right corner of the screen at runtime, which was
measured by Mirrors class NetworkTime. NetworkTime follows the basic principle for measuring
latency, that is, start a timer as an empty packet is sent to the server and stop the timer when the server
responds to that packet. This time in milliseconds is then divided by two to find the estimated latency.
Refer to Appendix A for code documentation of the NetworkTime class.
The NetworkTime class was referenced in a custom C# script called CaptureComponent. Referencing
made it possible to access the temporary latency that is calculated at runtime. By doing so, latency
could be written and saved to a text file document using Unity’s Input and Output (I/O) extension
class. This benefitted the controlled experiment in such a way that all measured latencies were logged
in individual text files. See Appendix B for CaptureComponent implementation.

5.3 Scenes
This version of Mirror provided multiple example scenes to try different networking solutions on.
These have been programmed and optimized by the official Mirror developers, which therefore served
as the testing environments for each transport layer API:s (Vis2k, 2021e).
The scenes that were of interest to both experiments were Pong, Tanks, and Benchmark. The selection
is motivated as such that each game is different regarding graphical rendering, required network
transmissions, and the amount of code each game executes (see Appendix C). Moreover, latency
varied by running transport layer API:s on multiple games which increased accuracy regarding the
whole population for Unity online games.

14
5.3.1 Pong
Pong is a simple two-player 2D game where each player controls a left or a right bracket that can
travel up and down. Once two players are connected, a ball spawned with a vertical velocity in either
direction (see figure 9). The player moves their bracket to an appropriate position to make sure the
ball hits the bracket which caused it to change direction and head towards the other player.

Figure 9 Pong scene

The scene consists of the map containing white walls that surround the player. Meanwhile, the player
is a 2D cube with a movement script attached. Similarly, the ball which moves left and right has a
movement script that checks for collisions with players which would reverse force direction.
Pong is described as a simple game to introduce multiplayer development in Mirrors own
documentation (Vis2k, 2021e). Pong was therefore a low effort game to run because it only increased
or decreased the player's position values at runtime, while the ball changed direction upon a collision.
The position of the players and the ball were synchronized by Mirrors Network Transform component
(see figure 10), similar to Unity’s built-in networking serialize functionality described in section 2.3.

Figure 10 Network Transform component

15
5.3.2 Tanks
Tanks is a 3D game that lets up to four players join and play concurrently (see figure 11). The tanks
can move around and shoot missiles towards others. Each fired missile instantiates as a 3D object in
the scene. Meanwhile, the tanks listen for collisions from other missiles. This made Tanks represent a
physics-based platformer game that introduced both 3D rendering as well as a variety of physics and
mathematical computations that all needed to synchronize over the network.

Figure 11 Tanks scene

5.3.3 Benchmark
Benchmark is also a scene provided by Mirror that focuses on benchmarking Massive Multiplayer
Online Games (MMOGs). At runtime, a thousand entities are instantiated around the player which
simulated connected players (see figure 12). By using the same networking components as a
connected player would, the entities are updated and synchronized in the same way clients would be if
there were a thousand clients connected to the host node. Moreover, each entity moves through
remote procedure calls which adds traffic to the server. Benchmark was therefore a good example for
a MMOG whereas the client endured large traffic from a thousand surrounding nodes.

Figure 12 Benchmark scene

16
5.4 API integration
KCP and Telepathy were already included with Mirror, thus no installation needed. Steamworks.NET
however, had to be downloaded and imported manually from Unity’s asset store. For this project the
latest version 4.4.1 of Steamworks.NET was used.
For each scene the NetworkManager’s transport field had a reference to the transport layer API, one at
a time. All instances’ integrations generated nine different builds, each game with every transport
layer API assigned.

Figure 14 NetworkManager with Kcp Figure 13 Kcp Transport component

5.4.1 Configure Custom NetworkManager

For KCP and Telepathy, only two functions were needed to make the custom NetworkManager fulfill
its task. One for hosting a session and one for joining an existing session (see figure 15). The first
function HostGame checks whether the connection is in an offline state. If so, it proceeds to call
StartHost in the NetworkManager component. This function then creates a multiplayer session that
initializes the active transport layer API.
Like HostGame, the second function JoinGame checks the client's connection state. If the client is
considered offline, StartClient will be called from the NetworkManager component which establishes
a connection using the host address string and the active transport layer API port.

17
 using UnityEngine;
using Mirror;

public class CustomNetworkManager : NetworkBehaviour {

private NetworkManager networkManager;
private void Start(){ //Reference our NetworkManager component
networkManager = gameObject.GetComponent<NetworkManager>();
}

public void HostGame(){

if(!NetworkClient.active){ //If application is in offline state
networkManager.StartHost(); //Create session and join as client
}
}

public void JoinGame(){

if(!NetworkClient.active){ //If application is in offline state
networkManager.StartClient(); //Join session as client
}
}
}

Figure 15 CustomNetworkManager script

Steamworks.NET on the other hand is an API that asynchronously must communicate with Steam
servers which required more implementations to work. Most importantly, callbacks needed to be
made for every event that updated a session (Steamgames, 2022). For these games, there were
callbacks for session creations, join requests, and once a client connected. Refer to Appendix D to see
the full implementation of the CustomNetworkManager when using Steamworks.NET.

5.5 Experiment setup

The experiment was conducted locally on the same router to avoid any additional latency from being
added due to the Internet Service Provider (ISP) routing scheme, which is a network issue known as
the Last-Mile Latency according to Kelly & Kumar (2021).
The third-party asset Mirror provided many classes that help developers analyze their multiplayer
service in Unity. This experiment relied on Mirrors class NetworkTime to derive latency at runtime.
The custom C# script CaptureComponent, implemented in section 5.2, was responsible for storing this
information into a text file automatically.

Prior to each experiment, all irrelevant applications that were running in the background were stopped
to decrease the number of active processes on the computers. Note that we determined what
applications should and should not be active during the experiment. To assist our decision, the book
Windows 10, Inside Out (Bott & Stinson, 2019) was used. This book explained how background
processes could be terminated from Windows Task Manager component without interfering with the
operating systems functionality (Bott & Stinson, 2019). For these experiments, the processes that had
any activity with the CPU (visualized as a percentage) were terminated, except for the ones that are a
part of the operating systems functionality (such as Service Host executables).

Factors that affect latency were presented in section 2.1. Some of these factors depend on the
properties of the games’ integrated network infrastructure, such as what transport layer API the
infrastructure uses (Taha Al-Dhief, et al., 2018). On the other hand, there are factors that increases
latency regardless of the network infrastructures properties.

18
Colouris et al. (2012) stated that the hardware on the host and client fluctuates latency since reading
and writing network messages is a matter of delay due to processing power. Hardware specifications
that were used for our client and host have therefore been provided in figure 16 and 17.

Processor: Intel Core i7-6700k 4.0Ghz.

Graphics card: AMD Radeon R9 390 Nitro 8GB.
Memory card: WD Black SN850 M.2 NVMe SSD, 5300MB/s read and write throughput.
RAM: 16GB DDR4.
Operating-System: Windows 10 64-bit.

Figure 16 Client hardware

Processor: Intel Core i5-9400F 2.9Ghz.

Graphics card: NVIDEA GeForce GTX 1650 4GB.
Memory card: WD Black SN850 M.2 NVMe SSD, 5300MB/s read and write throughput.
RAM: 8GB DDR4.
Operating-System: Windows 10 64-bit.

Figure 17 Host hardware

19
6 Result
40 data points (latencies) were captured for each of the nine tests. This process was repeated four
times to generate randomness and increase accuracy regarding the true population. The same objective
was done for the unstable connection experiment, but with a manually configured packet loss rate of
30% using the tool Clumsy 0.2.

6.1 Raw data

Box plot diagrams have been produced to get a visual representation of the raw data. These can be
found in Appendix E. As for this section, outliers and other observations are addressed. The transport
layer API:s are color coded whereas blue is KCP, red is Telepathy and yellow is Steamworks.NET.

From the box plots in figure 18 there are two outliers for both KCP and Telepathy when playing the
game Benchmark. This means that a latency measurement was abnormally high compared to the rest.
Despite this, outliers did not appear at the same time for every test. This can be determined by simply
comparing each box plot diagram in Appendix E.
Observe that the mean latencies (marked as an X) changed between games, even when the transport
layer API was the same. To exemplify, for figure 18 specifically, KCP achieved a mean latency of
27,65ms when running on the game Benchmark. However, a decreased mean latency of 16,15ms was
measured when switching to the game Pong. The same goes with the other transport layer API:s as
well.

Figure 18 Raw data representation

20
6.2 Stable connection experiment analysis
For figure 19 and 20, the average mean latency and standard deviation (STD) have been calculated for
each transport layer API based on how they performed on all three games. An average of average
grouped by transport layer API. This presents how each transport layer API differs in terms of
latency. Note that the color scheme is the same as presented in section 6.1.

Figure 19 Mean and STD

Since Steamworks.NET achieved a high latency that cannot be visually represented in the same
diagram as with KCP and Telepathy it is instead represented it in a separate histogram (see figure 20).

Figure 20 Mean and STD - Steamworks.NET

Although Telepathy achieved a lower mean latency of 15,87ms compared to KCP’s 16,50ms, the
transport layer API KCP had a slightly more stable performance regarding fluctuation. More
specifically, KCP achieved a STD of ±4,13ms compared to Telepathy’s ±4,31ms.

21
Steamworks.NET on the other hand, experienced the highest mean latency as well as STD overall,
with 1391,36ms mean latency and a ±10,50ms STD.
Figure 21 & 22 shows the same mean latency but provides confidence interval (CI) of all transport
layer API:s, visualized as vertical black bars. If confidence intervals for two or more transport layer
API:s do not overlap it can be concluded that there is a significant difference between the pairs in
terms of latency. More specifically, it means that 95% of any similar test made will achieve a mean
latency that no other transport layer API can achieve. If they would overlap however, Anova is used
to determine a difference.

Figure 21 Mean and CI

Figure 22 Mean and CI - Steamworks.NET

22
Since KCP and Telepathy’s confidence interval overlap, an Anova test was conducted to determine if
there was a difference in means or not. The Anova test (presented in figure 23) concluded that there
was a difference in means regarding the transport layer API:s, when utilized with a stable connection.

Figure 23 Anova test

Further calculation was done using Tukey test. For this experiment, the null hypothesis was going to
be rejected if the generated p-value was less than 0.05, since that would render one or more transport
layer API:s as significantly different in terms of their mean latency.
The output of the Tukey test, provided in figure 24, stated that the null hypothesis for H1 was
rejected. Specifically, Steamworks.NET was significantly different from both KCP and Telepathy,
while KCP and Telepathy were not significantly different compared to each other.

Figure 24 Tukey test

23
6.3 Unstable connection experiment analysis
Figure 25 displays the mean latency of 56,87ms and a ±STD of 6,46ms for KCP. Steamworks.NET
achieved a mean latency of 1386,78ms and a STD of ±2,69ms. Telepathy experienced an increased
latency compared to the stable connection experiment, with a mean latency of 1055,56ms and a STD
of ±204,97ms.

Figure 25 Mean and STD with packet loss

From figure 26 visualizes confidence interval (CI) instead of STD. KCP calculated a CI of ±3,655ms,
Steamworks.NET ±1,522ms and lastly Telepathy ±115,971ms. The confidence intervals do not
overlap between any of the transport layer API:s which concluded that there was a significant
difference between all transport layer API:s when simulating poor network, thus rejecting null-
hypothesis for H2.

Figure 26 Mean and CI with packet loss

24
6.3.1 Quality of sample
The standard error (SE) of the mean is an estimate of how well the sample data (consisting of 40 data
points per test) represents the whole population. It tells us how much each sample mean will vary in
case the study is repeated using a different data sample from the main population. Low standard error
values indicates that the quality of the sample is good respective to the population.
Looking at the summary of the samples for both experiments (see Appendix F), it is prominent that all
SE are low relative to their mean. This meant that the quality of every sample was good, which also
was the motivation for using the samples for statistical analysis.

6.4 Summary of experiments

In this section the research questions and hypothesis are addressed and summarized.
From the stable connection experiment, we can conclude that the transport layer API:s KCP,
Telepathy and Steamworks.NET differentiate in mean latency when utilized on a set of Unity online
games. Furthermore, Steamworks.NET differentiates significantly from KCP and Telepathy, and
achieved the highest latency out of all transport layer API:s. This rejected our null hypothesis for H1.
The latency increased when simulating a poor connection. For the unstable connection experiment, all
three transport layer API:s were significantly different from each other, with Telepathy having the
highest mean latency. Like the stable connection experiment, this rejected our null hypothesis for H2.
To address the third hypothesis H3, the box plot diagrams showed that mean latency changed between
different games, even when the transport layer API remained the same (see Appendix E). This
observation rejects our null hypothesis for H3, stating that the type of game impacts latency.

To answer our first research question RQ1, Steamworks.NET achieves a higher latency to a 95%
certainty. Meanwhile, there is a negligible difference from using KCP or Telepathy.
If the connection is bad however, KCP will achieve the lowest mean latency while Telepathy and
Steamworks.NET renders the game unplayable, which also answers the second research question
RQ2. Regarding our third research question RQ3, the box plot diagram observed that the game has an
influence on latency since the mean latency changed between games, even if the transport layer API
remained the same.

25
7 Evaluation

7.1 Conclusions
This study aimed to measure latency using the Unity transport layer API:s Telepathy, KCP, and
Steamworks.NET and compare if there is a significant difference while playing a set of Unity
multiplayer games. In addition, the study also investigated if the choice of transport layer API was
relevant for players with poor network connection.

To approach this study’s aim, the Unity engine served as a controlled testing environment, with
Mirror being the underlying network infrastructure. On top of Mirror, the transport layer API:s KCP,
Telepathy, and Steamworks.NET were integrated. Also, Mirror provided a latency measuring class
modified to capture latency at runtime. Each of the transport layer API:s have been active on three
different Unity online games Pong, Tanks, and Benchmark. By generating sufficient data with our
experimental setup, we generalized our results on a wider population of Unity multiplayer games.

The stable connection experiment showed the contrast between the three different transport layer
API:s with regard to latency, while also answering if the game influences their performance.
Specifically, KCP and Telepathy had no difference in mean latency, while Steamworks.NET achieved
a significantly greater latency. The experiment also concluded that the type of game impacts latency,
regardless of what transport layer API is being used.
The unstable connection experiment was conducted with a treatment of 30% packet loss rate manually
configured using the tool Clumsy 0.2. The analysis concluded that a poor connection results in a
different outcome compared to the stable connection experiment. With a poor connection, there was a
significant difference in latency between all three transport layer API:s.

7.2 Discussion
For the transport layer API Steamworks.NET we have calculated a mean latency of 1391,36ms during
the stable connection experiment. However, the game Benchmark was the major factor for such a high
overall latency since Steamworks.NET experienced latencies over four seconds for that game, which
was not the case when running Pong or Tanks (Refer to Appendix F to see every sample).
This fact makes Benchmark using Steamworks.NET unplayable for humans, as Kelly & Kumar
(2021) stated that a latency above 100ms is enough to be troublesome for the human brain.
To address possible causes for this behavior, we relate to Sanchez (2020) research that was conducted
on the same underlying network infrastructure Mirror, with Steamworks.NET as accommodating
transport layer API. Sanchez (2020) stated that Relay Transport Layers such as Steamworks.NET
have an increased latency since messages need to pass all connected clients in the P2P network until it
reaches the destination node. This property of Relay Transport Layers could motivate why
Steamworks.NET captured such a high latency during the Benchmark experiment where a thousand
concurrent users were connected.
Zhang et al. (2015) measured latency for mobile games when utilized on a P2P network. Their
experiment encountered a similar behavior to what Steamworks.NET did when running the game
Benchmark. For their experiment, latency rapidly increased when more clients connected. This also
caused packet loss to occur (Zhang, et al., 2015). However, Zhang et al. (2015) did not use Mirror as
their network infrastructure and used a Wi-Fi medium rather than a transport layer for

26
communicating. Despite this, their study as well as ours share similar findings since the use of P2P
networks caused gameplay issues when number of concurrent users are high.

The average mean latency for the transport layer API Steamworks.NET could have been lower, had
Benchmark not been included as one of our three games since Steamworks.NET achieved a latency
below 30ms on the other games when operating under a stable connection. However, to make sure our
data was representative on a wider population of Unity multiplayer games, we decided to vary what
Unity games we tested on. In contrast, if we only had measured latency for one of the games then we
could not conclude if the type of game influence latency. In addition, we performed the same nine
tests four times, and for each run we calculated three average latencies from each transport layer API.
This resulted in 12 averages per transport layer API for the four runs which improved the accuracy of
our statistical calculations.

Telepathy managed to keep a low mean latency for the Benchmark test. The developer of the
transport, (Vis2k, 2021d), claims that Telepathy can handle many concurrent users thanks to its
reduced overhead architecture. From the Benchmark test, Telepathy’s results were intact with Vis2k’s
description which showed that Telepathy did not increase latency when the number of concurrent
users had increased.
On the other hand, Telepathy did not manage to keep this latency once packet loss was introduced.
It is documented that TCP uses one processing thread for every connection in a network (Vis2k,
2021d). This could motivate why the TCP utilized transport layer API Telepathy had issues handling
a session containing one thousand connections while simulating a poor connection. The increase of
packet loss could require additional resources from the processor, since TCP must resend every lost
packet (Taha Al-Dhief, et al., 2018). Another explanation is motivated by Wu et al. (2017), stating
that TCP in gaming introduces latency spikes when lost packets are retransmitted, which is one of the
properties for TCP.

Novotny et al. (2020) used Mirror as an underlying network infrastructure with Steamworks.NET as
the transport layer API. Their research demonstrated a framework to achieve multiplayer in Unity.
The experiments conducted in our report also included Steamworks.NET as one of the independent
variables. Novotny et al. (2020) claimed that Steamworks.NET was a preferable transport layer API to
use for multiplayer, but what latency their framework measured was never provided. In our study,
Steamworks.NET achieved the highest mean latency when utilized on stable network connection,
while KCP and Telepathy were better options.

Researchers Polančec and Mekterović (2017) used the network infrastructure PUN to build a massive
multiplayer online game (MMOG). The results by Polančec and Mekterović (2017) indicate that PUN
works as an alternative underlying network infrastructure to Mirror. Therefore, PUN would be a
comparable option to integrate multiplayer on from a developers standpoint. On the other hand, the
transport layer API:s available for PUN have not been addressed or measured.

7.2.1 Societal benefit

This research has been conducted to investigate the differences in three transport layer API:s, KCP,
Telepathy and Steamworks.NET in terms of their network latency performance. The result of the
research is primarily focused to be used in the game development field whether it is for companies or
developers that want to integrate a multiplayer service into a game that utilizes the Mirror networking
infrastructure. Secondly, the player wants to achieve as low latency as possible, with a good range

27
being 10-20ms (Kelly & Kumar, 2021). Some countries might not provide as good internet as we are
used to, so in these parts of the world the choice of transport layer API can be critical to determine if
the game is playable or not.

Furthermore, previous research has reviewed some of the most common network infrastructures,
Mirror being one of them (Novotny, et al., 2020). However, no reviewed article has evaluated the
different transport layer API:s that are available within the network infrastructure, which for example
could be KCP, Telepathy or Steamworks.NET. Since our study proved that the choice of transport
layer API is an important factor for what latency a client may achieve, previous research conducted on
third-party assets such as Mirror must be questioned.

Our report suggested that it is not ideal to integrate the Telepathy transport layer API if the developer
expects some clients to have a poor connection, since this would render the game unplayable.
Similarly, is not recommended to integrate Steamworks.NET for a massive multiplayer online game
(MMOG). KCP on the other hand, experienced no abnormal latencies during either of our
experiments, though Telepathy achieved no different latency than KCP when utilized in stable
network condition.

Mihaljević et al. (2021) have discussed deployment of augmented reality systems, in the paper Impact
of 5G Network Performance on Augmented. Society is striving towards becoming more digital, with
technologies such as augmented reality being used for various services nowadays. If such virtual
worlds should become our new reality, then cloud access between the client and the service must
happen quickly (Mihaljević, et al., 2021). Also, Mihaljević et al. (2021) state that these services are
likely to occur over fifth generation networks (5G) which demands reliability and better integrations.
The result of our report therefore suggests that the type of transport layer is chosen wisely for
augmented reality systems, since transport layer affects latency to significant differences.

7.3 Future work

The outliers, visualized in figure 18 and Appendix E, can be caused by three possible networking
issues as suggested by Coulouris et al. (2012).
1. Overhead (Coulouris et al., 2012)
2. Packet drops (Coulouris et al., 2012).
3. Buffer overflows within the transport layer itself (Coulouris et al., 2012).

All three factors could have been present during the experiment, but our scope has not been to capture
and analyze network issues regarding each transport layer API. However, we can speculate their
presence based on research provided in the background section. For instance, Steamworks.NET did
not capture as many outliers as KCP and Telepathy (this is observed in Appendix E).
According to Novotny et al. (2020), Steamworks.NET offers a P2P network to connect with users
efficiently (Novotny, et al., 2020). Moreover, Coulouris et al. (2012) stated that P2P has a reduced
overhead architecture since load of the application is balanced among all connected. Since
Steamworks.NET follows this principle then overhead could have been the cause of the latency spikes
plotted as outliers in Appendix E, thus motivating why Steamworks.NET did not capture as many
outliers as KCP and Telepathy. However, more research needs to be conducted on the transport layer
API:s regarding network issues.

28
As described by Apel (2018), different games require more or less processing power for graphical
rendering as well as events that need to synchronize over the network. More specifically, the
architecture of game together with the network infrastructure determines how much overhead the
game processes, which is why we wanted to answer our third research question RQ3.
To verify what Apel (2018) stated we can simply compare how a specific transport layer API
performed during different games in Appendix E. Although there is a clear distinction for mean
latency and STD when we switched games, future work needs to be researched to determine what
technical aspects of games that causes this change.

Telepathy significantly increased mean latency when executed with unstable connection. As
mentioned in section 7.2, hardware performance on the client could be the cause for such a behavior.
If that is the case, then Telepathy’s latency could have been significantly lower if we had used a better
CPU on the computer. Since this study did not observe CPU performance while playing the games, it
cannot be proven, however. Future work could therefore investigate CPU performance using different
transport layer API:s.

A user study could address playability aspects in a qualitative manner. For such a study, participants
could play a game using different transport layer API:s and rate each transport layer API with regard
to playability. The results would exclude minor latency differences from being as important since
humans are not sensitive to them, according to Kelly & Kumar (2018). In contrary, the user study
would be more informative regarding networking issues rather than average latencies since spikes of
100ms latencies could render a game unplayable (Kelly & Kumar, 2021).

Polančec and Mekterović (2017) built a similar network infrastructure using a different underlying
network infrastructure PUN. Their multiplayer service should therefore be compared with that of
Mirrors to determine where they differentiate in terms of latency. A controlled experiment could
determine if transport layer API:s affect latency to a significant differences, for that asset as well,
which is the case using Mirror according to this study.

29
8 Ethics
The authors of this report have been responsible for the interpretations of previous work, as well as
the interpretation and conclusion of this study’s results.
Information gathered from other sources has earned the credit they deserve, both In-Text as well as in
the reference section.
In this paper, all figures and images have been created by us and there were no external visual
resources used. However, figure 3 and 4 are interpretations of figures provided by Mirror's official
website (Vis2k, 2021c).
Terminologies that could indicate biases like best and worst have been avoided to keep a neutral
objective about the results.
More transport layer API:s could have been investigated from an ethical point of view. On the other
hand, the independent variables for the experiments are the ones that are most frequently occurring
according to Mirrors official documentation (Vis2k, 2021b).
The experiments were conducted on our own equipment, mentioned in section 5.5, and the generated
samples are given in Appendix F. The experiment was controlled by us and with no participants
involved, and all tests executed under the same circumstances. Also, no data was removed from the
samples, such as outliers, which have been motivated in section 7.3.
This study could include nonintentional errors for any section. On the other hand, we have handled
our interpretations, implementations, and experimentations with quality in mind and have corrected
any observation of error. For transparency, we have addressed all topics we had to do to perform this
experiment which have been critical in achieving our results. Likewise, all our Unity integration is
documented in section 5. This could come in handy if anyone wants to replicate our experiments.

8.1 Internal validity

As discussed in the experiment setup, section 5.5, we stopped all unnecessary running processes in the
background but in modern operating systems it is hard to determine whether some processes are
needed or not. Either way, background processes could influence the outcomes of the experiment.
Coulouris et al. (2012) claimed that the hardware components also affect latency. For instance, the
read and write speed of our computers add some processing delay (Coulouris, et al., 2012).

Telepathy is a TCP utilized transport layer API which according to Vis2k (2021z) could require more
CPU power than other transport layer API:s. Therefore, the hardware has affected the measured
latency which could motivate a future work focusing on CPU consumption when using different
transport layer API:s.

The complexity of ping functionality, that is the C# class we use to measure latency (see appendix A),
could be unoptimized from a programmer’s perspective. However, this is a subjectable topic, but it is
important to know that the tool could have a margin of error that we will not know. On the other hand,
we tested this tool a great amount before we conducted the actual experiment.

Unlike KCP and Telepathy, Steamworks.NET is dependent on geographically distributed Steam

servers outside our testing environment. With that said, the servers Steamworks.NET asynchronously
communicates to could variate in network performance which is beyond our control.

30
8.2 External validity
Are the generated data samples from the experiments a good representative of the real population?
To our defense, we integrated each transport layer API on different types of Unity online games, so
that our results targeted a bigger population of Unity multiplayer games. The games vary in the sense
that Pong is a 2D game, Tanks is 3D and Benchmark introduces large amount of network traffic. We
also ran the capturing process four times for both experiments to generate as general data as possible.
Lastly, we provide confidence interval that applies to every transport layer API tested in this
experiment which predict the results of any similar study.

Since we used more than one game to measure latency on, we had to calculate each transport layer
API:s average from three different games. These were then averaged again over four runs to derive a
mean latency representing that transport layer API from which we measure a significant difference.
This had to be done in order to represent the true population more accurately than just testing the
transport layer API:s on a single game. However, it led to more generalized data.

We have not found any peer-reviewed articles that document implementation on the underlying
network infrastructure Mirror, which we use as a framework. Therefore, we must rely on the
developers' documentation regarding implementation and use which we do not know the quality of.

Some of the tests captured outliers. In section 7.3 we determined that these are most likely caused by
our independent variables, and not from an external factor. With this assumption we had to keep the
outliers in our comparison. Otherwise, we would not represent their differences truly. However, we
cannot prove this assumption and it could be a motivation for a future study regarding networking
issues with transport layer API:s in Unity.

Some of our references like books and articles are older than 10 years. Since this research topic is
limited in terms of what related work exist, we have not had much flexibility in what sources we use.
However, they have been used in this study to gather some fundamental information about
terminologies or technologies that, from our expertise within Unity and networking, are still valid to
the day.

31
9 References
Apel, S., 2018. Reducing Development Overheads with a Generic. Jena, Germany, Department of
Computer Science, pp. 21-28.

Bott, E. & Stinson, C., 2019. Windows 10 Inside Out. 3rd ed. Unterschleissheim, Germany: Microsoft
Press.

Coulouris, G., Dollimore, J., Kindberg, T. & Blair, G., 2012. Distributed Systems: Concepts and
Design. 5th ed. Boston, USA: Pearson Education Limited.

Exit Games, 2022. Unity Asset Store: Photon PUN 2+. [Online]
Available at: https://assetstore.unity.com/packages/tools/network/photon-pun-2-120838
[Accessed 21 April 2022].

Hana Digital, 2021. Steamworks.NET release statistics. [Online]

Available at: https://hanadigital.github.io/grev/?user=rlabrecque&repo=Steamworks.NET
[Accessed 21 April 2022].

Kazakov, A., 2021. Cloud gaming controls. Journal of Physics, Volume 2134, p. 7.

Kelly, S. & Kumar, K., 2021. Unity Networking Fundamentals. Creating Multiplayer Games with
Unity. 1st ed ed. New York, USA: Apress.

Labrecque, R., 2022. GitHub: Steamworks.NET. [Online]

Available at: https://steamworks.github.io/
[Accessed 21 April 2022].

Menard , M. & Wagstaff, B., 2014. Game development with Unity. 2nd ed. Boston, USA: Cengage
Learning PTR.

Mihaljević, A., Kešelj, A. & Lipovac, A., 2021. Impact of 5G Network Performance on Augmented.
Dubrovnik, Croatia, SoftCOM.

Novotny, A., Gudmundsson, R. & Frederick, H., 2020. A Unity Framework for Multi-User VR
Experiences. Reno, USA, EasyChair.

Polančec, D. & Mekterović, I., 2017. Developing MOBA games using the Unity game engine.
MIPRO, 22-26 May, pp. 1510-1515.

Sanchez, R., 2020. Realtime Networking Technologies for Unity, Enschede, Netherlands: Saxion
university of applied science.

Stagner, A. R., 2013. Unity Multiplayer Games. Birmingham, England: Packt Publishing.

Steamgames, 2022. Steamworks API Overview. [Online]

Available at: https://partner.steamgames.com/doc/sdk/api
[Accessed 21 April 2022].

Sun, Y. & Leu, J. Y., 2011. Deploying a massively multiplayer online game with a low-latency. Inf
Technol Manag, 26 January, pp. 35-47.

Suznjevic, M. et al., 2014. Analyzing the Effect of TCP and Server Population on Massively
Multiplayer Games. International Journal of Computer Games Technology, Volume 2014, p. 17.

32
Taha Al-Dhief, F., Albadr, M. & Sabri, N., 2018. Performance Comparison between TCP and UDP
Protocols in Different Simulation Scenarios. International Journal of Engineering & Technology,
Issue 7, pp. 172-176.

Tao, C., 2022. Clumsy 0.2. [Online]

Available at: https://jagt.github.io/clumsy/
[Accessed 20 April 2022].

Unity Technologies, 2021. Unity Multiplayer Networking. [Online]

Available at: https://docs-multiplayer.unity3d.com/
[Accessed 19 February 2022].

Vis2k, 2021a. Unity Asset Store: Mirror. [Online]

Available at: https://assetstore.unity.com/packages/tools/network/mirror-129321
[Accessed 21 April 2022].

Vis2k, 2021b. Mirror Networking: Transports. [Online]

Available at: https://mirror-networking.gitbook.io/docs/transports
[Accessed 18 April 2022].

Vis2k, 2021c. Mirror Networking: General. [Online]

Available at: https://mirror-networking.gitbook.io/docs/general
[Accessed 20 February 2022].

Vis2k, 2021d. GitHub: Telepathy. [Online]

Available at: https://github.com/vis2k/Telepathy
[Accessed 21 April 2022].

Vis2k, 2021e. Mirror Networking: Examples. [Online]

Available at: https://mirror-networking.gitbook.io/docs/examples
[Accessed 21 April 2022].

Wang, Y., 2017. A framework for developing network based games using Unity and Ice, Madrid,
Spain: E.T.S. de Ingenieros Informáticos (UPM).

Wei, L., 2022. GitHub: KCP. [Online]

Available at: https://github.com/skywind3000/kcp/blob/master/README.en.md
[Accessed 21 April 2022].
Wohlin, C. et al., 2012. Experimentation in Software Engineering. 2012:th ed. New York, USA:
Springer.

Wu, J. et al., 2017. Streaming Mobile Cloud Gaming Video over TCP. IEEE Transactions on circuits
and systems for video technology, 27(1), pp. 32-48.

Zhang, N., Lee, Y., Radhakrishnan, M. & Balan, R. K., 2015. GameOn: p2p Gaming On Public
Transport. Portland, USA, MobiSys.

33
APPENDIX A:
NetworkTime.cs for Ping Display.

internal static void UpdateClient()

{
// localTime (double) instead of Time.time for accuracy over days
if (localTime - lastPingTime >= PingFrequency)
{
NetworkPingMessage pingMessage = new NetworkPingMessage(localTime);
NetworkClient.Send(pingMessage, Channels.Unreliable);
lastPingTime = localTime;
}
}
// executed at the server when we receive a ping message
// reply with a pong containing the time from the client
// and time from the server
internal static void OnServerPing(NetworkConnection conn, NetworkPingMessage message)
{
// Debug.Log($"OnPingServerMessage conn:{conn}");
NetworkPongMessage pongMessage = new NetworkPongMessage
{
clientTime = message.clientTime,
serverTime = localTime
};
conn.Send(pongMessage, Channels.Unreliable);
}
// Executed at the client when we receive a Pong message
// find out how long it took since we sent the Ping
// and update time offset
internal static void OnClientPong(NetworkPongMessage message)
{
double now = localTime;
// how long did this message take to come back
double newRtt = now - message.clientTime;
_rtt.Add(newRtt);
// the difference in time between the client and the server
// but subtract half of the rtt to compensate for latency
// half of rtt is the best approximation we have
double newOffset = now - newRtt * 0.5f - message.serverTime;
double newOffsetMin = now - newRtt - message.serverTime;
double newOffsetMax = now - message.serverTime;
offsetMin = Math.Max(offsetMin, newOffsetMin);
offsetMax = Math.Min(offsetMax, newOffsetMax);
if (_offset.Value < offsetMin || _offset.Value > offsetMax)
{
// the old offset was offrange, throw it away and use new one
_offset = new ExponentialMovingAverage(PingWindowSize);
_offset.Add(newOffset);
}
else if (newOffset >= offsetMin || newOffset <= offsetMax)
{
// new offset looks reasonable, add to the average
_offset.Add(newOffset);
}
}

Figure 27 NetworkTime.cs for NetworkPingDisplay

34
APPENDIX B:
C# code for CaptureComponent.

using UnityEngine;
using Mirror;
using System;
using System.IO;

public class CaptureComponent : MonoBehaviour

{
private string ms;
private string txtDocumentName;

void Start()
{
//Create folder
Directory.CreateDirectory(Application.streamingAssetsPath + "/Tests/");
txtDocumentName = Application.streamingAssetsPath + "/Tests/" + "TestFile" + ".txt";
InvokeRepeating("CaptureLatency", 10f, 1f); //Call capture function once a second
}

public void CaptureLatency()

{
ms = Math.Round(NetworkTime.rtt * 1000).ToString();
if (ms == "") //if offline
{
return;
}
File.AppendAllText(txtDocumentName, ms + "\n"); //Add latency on next row
}
}

Figure 28 CaptureComponent.cs

35
APPENDIX C:
C# code of scenes Pong, Tanks and Benchmark.

using UnityEngine;

namespace Mirror.Examples.Pong
{
public class Player : NetworkBehaviour
{
public float speed = 30;
public Rigidbody2D rigidbody2d;
// need to use FixedUpdate for physics
void FixedUpdate()
{
// only let the local player control the racket.
// don't control other player's rackets
if (isLocalPlayer)
rigidbody2d.velocity = new Vector2(0, Input.GetAxisRaw("Vertical")) * speed *
Time.fixedDeltaTime;
}
}
}

Figure 29 Player.cs for Pong

using UnityEngine;
namespace Mirror.Examples.Pong
{
public class Ball : NetworkBehaviour
{
public float speed = 30;
public Rigidbody2D rigidbody2d;
public override void OnStartServer()
{
base.OnStartServer();
rigidbody2d.simulated = true;
rigidbody2d.velocity = Vector2.right * speed;
}
float HitFactor(Vector2 ballPos, Vector2 racketPos, float racketHeight)
{
return (ballPos.y - racketPos.y) / racketHeight;
}
// only call this on server
[ServerCallback]
void OnCollisionEnter2D(Collision2D col)
{
// did we hit a racket? then we need to calculate the hit factor
if (col.transform.GetComponent<Player>())
{
float y = HitFactor(transform.position,
col.transform.position,
col.collider.bounds.size.y);
float x = col.relativeVelocity.x > 0 ? 1 : -1;
Vector2 dir = new Vector2(x, y).normalized;
rigidbody2d.velocity = dir * speed;
}
}
}
}

Figure 30 Ball.cs for Pong

36
using UnityEngine;
using UnityEngine.AI;

namespace Mirror.Examples.Tanks {
public class Tank : NetworkBehaviour {
public NavMeshAgent agent;
public Animator animator;
public TextMesh healthBar;
public Transform turret;
public float rotationSpeed = 100;
public KeyCode shootKey = KeyCode.Space;
public GameObject projectilePrefab;
public Transform projectileMount;
[SyncVar] public int health = 4;
void Update() {
healthBar.text = new string('-', health);
if (isLocalPlayer) {
float horizontal = Input.GetAxis("Horizontal");
transform.Rotate(0, horizontal * rotationSpeed * Time.deltaTime, 0);
float vertical = Input.GetAxis("Vertical");
Vector3 forward = transform.TransformDirection(Vector3.forward);
agent.velocity = forward * Mathf.Max(vertical, 0) * agent.speed;
animator.SetBool("Moving", agent.velocity != Vector3.zero);
if (Input.GetKeyDown(shootKey))
{
CmdFire();
}
RotateTurret();
}
}
[Command]
void CmdFire(){
GameObject projectile = Instantiate(projectilePrefab, projectileMount.position,
projectileMount.rotation);
NetworkServer.Spawn(projectile);
RpcOnFire();
}
[ClientRpc]
void RpcOnFire(){
animator.SetTrigger("Shoot");
}
[ServerCallback]
void OnTriggerEnter(Collider other){
if (other.GetComponent<Projectile>() != null){
--health;
if (health == 0)
NetworkServer.Destroy(gameObject);}
}
void RotateTurret(){
Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
RaycastHit hit;
if (Physics.Raycast(ray, out hit, 100)) {
Debug.DrawLine(ray.origin, hit.point);
Vector3 lookRotation = new Vector3(hit.point.x, turret.transform.position.y,
hit.point.z);
turret.transform.LookAt(lookRotation);
}
}
}
}

Figure 31 Tank.cs for Tanks

37
using UnityEngine;

namespace Mirror.Examples.Tanks {
public class Projectile : NetworkBehaviour
{
public float destroyAfter = 2;
public Rigidbody rigidBody;
public float force = 1000;
public override void OnStartServer()
{
Invoke(nameof(DestroySelf), destroyAfter);
}
// set velocity for server and client. this way we don't have to sync the
// position, because both the server and the client simulate it.
void Start()
{
rigidBody.AddForce(transform.forward * force);
}
// destroy for everyone on the server
[Server]
void DestroySelf()
{
NetworkServer.Destroy(gameObject);
}
// ServerCallback because we don't want a warning
// if OnTriggerEnter is called on the client
[ServerCallback]
void OnTriggerEnter(Collider co)
{
NetworkServer.Destroy(gameObject);
}
}
}

Figure 32 Projectile.cs for Tanks

using UnityEngine;

namespace Mirror.Examples.Benchmark
{
public class PlayerMovement : NetworkBehaviour
{
public float speed = 5;
void Update()
{
if (!isLocalPlayer) return;

float h = Input.GetAxis("Horizontal");
float v = Input.GetAxis("Vertical");

Vector3 dir = new Vector3(h, 0, v);

transform.position += dir.normalized * (Time.deltaTime * speed);
}
}
}

Figure 33 PlayerMovement.cs for Benchmark

38
using UnityEngine;

namespace Mirror.Examples.Benchmark
{
public class MonsterMovement : NetworkBehaviour
{
public float speed = 1;
public float movementProbability = 0.5f;
public float movementDistance = 20;
bool moving;
Vector3 start;
Vector3 destination;
public override void OnStartServer()
{
start = transform.position;
}
[ServerCallback]
void Update()
{
if (moving)
{
if (Vector3.Distance(transform.position, destination) <= 0.01f) {
moving = false;
}
else
{
transform.position = Vector3.MoveTowards(transform.position, destination, speed *
Time.deltaTime);
}
}
else {
float r = Random.value;
if (r < movementProbability * Time.deltaTime)
{
Vector2 circlePos = Random.insideUnitCircle;
Vector3 dir = new Vector3(circlePos.x, 0, circlePos.y);
Vector3 dest = transform.position + dir * movementDistance;
if (Vector3.Distance(start, dest) <= movementDistance)
{
destination = dest;
moving = true;
}
}
}
}
}
}

Figure 34 MonsterMovement.cs for Benchmark

39
 Appendix D:
C# code for CustomNetworkManager.cs when using Steamworks.NET transport layer API.
using UnityEngine;
using UnityEngine.UI;
using Mirror;
using Steamworks;
public class CustomNetworkManager : MonoBehaviour
{ //Callbacks
protected Callback<LobbyCreated_t> LobbyCreated;
protected Callback<GameLobbyJoinRequested_t> JoinRequest;
protected Callback<LobbyEnter_t> LobbyEntered;
public ulong CurrentLobbyID;
private const string HostAddressKey = "HostAddress";
private NetworkManager manager;
public GameObject HostBtn;
public Text LobbyNameText;

private void Start()

{
if (!SteamManager.Initialized) { return; }
manager = GetComponent<NetworkManager>();
LobbyCreated = Callback<LobbyCreated_t>.Create(OnLobbyCreated);
JoinRequest = Callback<GameLobbyJoinRequested_t>.Create(OnJoinRequest);
LobbyEntered = Callback<LobbyEnter_t>.Create(OnLobbyEntered);
}

public void HostLobby() {

SteamMatchmaking.CreateLobby(ELobbyType.k_ELobbyTypeFriendsOnly, manager.maxConnections);
}

private void OnLobbyCreated(LobbyCreated_t callback) {

if(callback.m_eResult != EResult.k_EResultOK) { return; }
Debug.Log("Lobby created succesfully");
manager.StartHost();
SteamMatchmaking.SetLobbyData(new CSteamID(callback.m_ulSteamIDLobby), HostAddressKey,
SteamUser.GetSteamID().ToString());
SteamMatchmaking.SetLobbyData(new CSteamID(callback.m_ulSteamIDLobby), "name",
SteamFriends.GetPersonaName().ToString() + "'s lobby");
}
private void OnJoinRequest(GameLobbyJoinRequested_t callback) {
Debug.Log("Request to join lobby");
SteamMatchmaking.JoinLobby(callback.m_steamIDLobby);
}

private void OnLobbyEntered(LobbyEnter_t callback) {

//Everyone
HostBtn.SetActive(false);
CurrentLobbyID = callback.m_ulSteamIDLobby;
LobbyNameText.gameObject.SetActive(true);
LobbyNameText.text = SteamMatchmaking.GetLobbyData(new CSteamID(callback.m_ulSteamIDLobby),
"name");
//Clients
if (NetworkServer.active) { return; }
manager.networkAddress = SteamMatchmaking.GetLobbyData(new CSteamID(callback.m_ulSteamIDLobby),
HostAddressKey);
manager.StartClient();
}
}

Figure 35 CustomNetworkManager.cs for Steamworks.NET

40
APPENDIX E:
The following figures show the raw data of the stable connection experiment, one run per diagram.
Each box shows mean (marked as X), median, quartile intervals and outliers (represented as dots).
Note that every test using transport layer API Steamworks.NET on the game Benchmark achieved a
high latency, and cannot be visualized in conjunction with the other boxplots. Hence, we have
included these box plots in its own diagram (see figure 40).

Figure 36 Box plots of first run

Figure 37 Box plots of second run

41
Figure 38 Box plots of third run

Figure 39 Box plots of fourth run

42
Figure 40 Box plots of Benchmark using Steamworks.NET

43
 Appendix F:
This is a summary of all samples we used. All nine tests were captured four times, hence there are
four tables for both experiments.
Terminologies:
➢ Mean: Mean latency
➢ SE: Standard Error of sample
➢ Median: Median latency from all 40 captures.
➢ STD: Standard Deviation around the mean.
➢ Min: Minimum captured latency.
➢ Max: Maximum captured latency.
➢ Count: Number of data points in sample.
➢ CI(95%): Confidence Interval, which we calculate using a 0.05 alpha.

Figure 41 Stable connection experiment summary

44
Figure 42 Unstable connection experiment summary

45