Network Performance Metrics

Network Performance Measurement
Ahmed Naail Abeer (aabeer)
Waris Sagheer (waris)
Disclaimer
Document Data Source: The document has compiled the data from multiple Internet sites including
Wikipedia, http://www.speedguide.net/ & others
1

Table of Contents
Introduction ................................................................................................................................................. 4
Bits, Bytes and Bandwidth Reference Guide ............................................................................................... 5
Bits vs. Bytes Confusion ........................................................................................................................... 5
Metric System ...................................................................................................................................... 5
Data Transfer Rate Representation ..................................................................................................... 5
Memory Size/Data Storage Representation ........................................................................................ 5
Bits/Bytes Reference Table ...................................................................................................................... 6
Bandwidth Reference Table ..................................................................................................................... 6
Layer 1, Layer 2, Layer 3 and Layer 4 Overhead .......................................................................................... 8
Layer 1 and Layer 2 Overhead ................................................................................................................. 8
Layer 3 and Layer 4 Overhead ............................................................................................................... 10
Method 1 ............................................................................................................................................... 13
Method 2 ............................................................................................................................................... 13
Frame/Packet Size Selection .................................................................................................................. 15
Frame Size 64 Bytes ........................................................................................................................... 15
Frame Size 1500 Bytes ....................................................................................................................... 16
Propagation, Serialization, and Queuing Delay ......................................................................................... 17
Serialization Delay .................................................................................................................................. 17
Packet Size vs. Latency ....................................................................................................................... 17
Queuing Delay ........................................................................................................................................ 17
Example: ............................................................................................................................................ 17
Propagation Delay .................................................................................................................................. 18
Example: ............................................................................................................................................ 19
Network Latency/ Round Trip Time (RTT) .................................................................................................. 20
Example: ................................................................................................................................................ 20
Bandwidth Delay Product (BDP) ................................................................................................................ 21
Window Size and Scaling ........................................................................................................................... 22
Optimizing the TCP Receive Window ..................................................................................................... 22
Optimal TCP Receive Window Calculation ............................................................................................. 22
Throughput ................................................................................................................................................ 23
Example ............................................................................................................................................. 23
2

Throughput vs. packet size .................................................................................................................... 23
Understanding Ethernet Clock PPM ...................................................................................................... 25
Final Thoughts ........................................................................................................................................ 30

3

Introduction
There is no single method to measure the network performance. Network engineers most often refer to
the performance of network devices by using the speed of the interfaces expressed in bits per second
(b/s). The term network performance is a very broad which is not only dependent on the rate but it also
includes other performance metrics like Latency, Maximum Transmission Unit (MTU), Throughput,
bandwidth and soon.
The purpose of this document is to give an overview of each parameter affecting network performance
and also allow reader to understand the commonly used network terms and their use in network
building calculation.
This document will include the calculation and theory of the following topics:
• Bit, Bytes Conversion and Bandwidth Reference Guide

• Layer 1, Layer 2, Layer 3 and Layer 4 Overhead
• Packet Rate Calculation
• Network Efficiency
• Maximum Transmit Unit (MTU) and Maximum Segment Size (MSS)
• Propagation, Serialization, and Queuing Delay
• Latency and Round Trip Time (RTT)
• Bandwidth Delay Product (BPD)
• Window Size and Scaling
• Throughput
• Understanding Ethernet ppm

4

Bits, Bytes and Bandwidth Reference Guide
This section discusses the basic conversion and provides the bandwidth reference guide. The basic
conversion is included in order to give an idea to that reader who does not feel comfortable in
mathematics. These conversions will then be used throughout the document for calculation and will be
quick reference guide for all other readers.
Bits vs. Bytes Confusion

There are different ways to use bits and bytes conversion. Bits and Bytes conversation are normally used
to represent rate and memory allocation.
Metric System
Let's start with some background information on prefixes, by mentioning the Metric system:

• Kilo (k)* = 10 ^ 3 = 1,000 thousand
• Mega (M) = 10 ^ 6 = 1,000,000 million
• Giga (G) = 10 ^ 9 = 1,000,000,000 billion
• Tera (T) = 10 ^ 12 = 1,000,000,000,000 trillion

Note: According to the Metric system, the lowercase "k" is used to describe decimal kilobits and capital
"K" is used for binary kilobytes.

Data Transfer Rate Representation
It is commonly used for measuring the amount of data that is transferred in a second between two end
points.

• 1 bit (b) = 0 or 1 = one binary digit
• 1 kilobit ( kb) = 10^3 bits = 1,000 bits
• 1 Megabit (Mb) = 10^6 bits = 1,000,000 bits
• 1 Gigabit (Gb) = 10^9 bits = 1,000,000,000 bits

Note: Although technically speaking, the term kilobit should have a lowercase initial letter; most
published reports capitalize it in abbreviation, resulting in "56 Kbps," or even the really confusing "56K
Memory Size/Data Storage Representation

Memory Size or Data Storage bits/bytes are generally calculated as multiple or exponent of 2.

• 1 byte (B) = 8 bits (b)
• 1 Kilobyte (K / KB) = 2^10 bytes = 1,024 bytes
• 1 Megabyte (M / MB) = 2^20 bytes = 1,048,576 bytes
• 1 Gigabyte (G / GB) = 2^30 bytes = 1,073,741,824 bytes
• 1 Terabyte (T / TB) = 2^40 bytes = 1,099,511,627,776 bytes

5

Although data storage capacity, such as on hard drives is generally expressed in binary Megabytes
(2^20), most Hard disk manufacturers, and some newer BIOSes use decimal megabytes (10^6), which is
slightly different and it gets confusing.

• 1 byte (B) = 8 bits (b)
• 1 Kilobyte (K / KB) = 10^3 bytes = 1,000 bytes
• 1 Megabyte (M / MB) = 10^6 bytes = 1,000,000 bytes
• 1 Gigabyte (G / GB) = 10^9 bytes = 1,000,000,000 bytes
• 1 Terabyte (T / TB) = 10^12 bytes = 1,000,000,000,000 bytes

Bits/Bytes Reference Table

Name Symbol Equivalent
bit b 0 or 1
byte B 8 bits
kilobit kb 1000 bits
kilobyte (binary) KB 1024 bytes
kilobyte (decimal) KB 1000 bytes
Megabit Mb 1000 kilobits
Megabyte (binary) MB 1024 Kilobytes
Megabyte (decimal) MB 1000 Kilobytes
Gigabit Gb 1000 Megabits
Gigabyte (binary) GB 1024 Megabytes
Gigabyte (decimal) GB 1000 Megabytes

Bandwidth Reference Table

DS0 (digital signal 0) is digital transmission rate of 64 Kbps, the bandwidth normally used for one
telephone voice channel. It is the base multiple for both T (North America) and E (Europe) DS (digital
signal) carriers.
The following table summarizes the set of signals and their relationship to the T-‐carrier and E-‐carrier
systems
DSx / Ex / Jx Data rate DS0 multiple Carrier

DS0(E0/J0) 64 kbps 1
128 kbps 2 ISDN
DS1 1.544 Mbps 23 (24) T1
E1 2.048 Mbps 30 (32) E1
DS1C / J1C 3.152 Mbps 46 (48) T1C / J1C
DS2 / J2 6.312 Mbps 92 (96) T2 / J2
E2 8.448 Mbps 120 (132) E2
J3 32.064 Mbps 480 J3
6

E3 34.368 Mbps 480 (537) E3
DS3 44.736 Mbps 644 (672) T3
STS1 51.84 Mbps 672 OC1
DS3C 89.472 Mbps 1288 (1344) T3C
J4 97.728 Mbps 1440 J3C
DS3X 134.208 Mbps 1932 (2016) T3X
E4 139.264 Mbps 1920 (2148) E4
STS-‐3 / SDH-‐1 155.52 Mbps 2016 OC3
DS4 274.176 Mbps 3864 (4032) T4
J5 400.352 Mbps 5760 J4
DS4E 411.264 Mbps 5796 (6048) T4E
STS-‐9 / SDH-‐3 466.56 Mbps 6048 OC9
DS4C 560.160 Mbps 7728 (8064) T4C
E5 565.148 Mbps 7680 (8592) E5 (4 E4 chnnels)
STS-‐12 / SDH-‐4 622.08 Mbps 8064 OC12
DS4X 822.528 Mbps 11592 (12096) T4X
STS-‐18 / SDH-‐6 933.12 Mbps 12096 OC18
DS5 1.120 Gbps 15456 (16128) T5
STS-‐24 / SDH-‐8 1.244 Gbps 16128 OC24
DS5X 1.400 Gbps 19320 (20160) T5X
DS5E 1.680 Gbps 23184 (24192) T5E
STS-‐36 / SDH-‐12 1.866 Gbps 24192 OC36
STS-‐48 / SDH-‐16 2.488 Gbps 32256 OC48
STS-‐96 / SDH-‐32 4.976 Gbps 64512 OC96
STS-‐192 / SDH-‐64 9.952 Gbps 129024 OC192
STS-‐256 13.271 Gbps 172032 OC256
STS-‐768 / SDH-‐256 39.813 Gbps 516096 OC768

Note: The following digital signal levels are not standardized and/or are not in common use:
DS3C, DS3X, DS4E, DS4C, DS4X, DS5, DS5X, DS5E, E5, J5, STS-‐9, STS-‐18, STS-‐24, STS-‐36, STS-‐256, STS-‐768.

7

Layer 1, Layer 2, Layer 3 and Layer 4 Overhead
The word overhead means extra load. Whenever we talk about Layers basically we are referring to
standard OSI model where Layer 1 is Physical Layer, Layer 2 is Data Link Layer, Layer 3 is IP Layer, and
Layer 4 is Transport Layer and so on. The OSI model is shown in Figure 1.
Figure 1: OSI Model 7 Layers
Each Layer has some overhead, which is called Header. The header is required in each level to
understand and process the packets properly. Usually header contains size, source and destination
addresses and CRC.
These Overheads play important role in order to measure the performance of the network. The
performance of network cannot be determined without knowing the overhead at each layer. Let’s
discuss the overhead at each Layer which later be used for calculations.
Layer 1 and Layer 2 Overhead

The typical Ethernet frame looks like the one shown in Figure 2.
8

Figure 2: Ethernet Frame Format
The typical layer 2 packets including following fields shown in Figure 2 and the distribution of bytes is
given below.
• Destination Address : 6 bytes

• Source Address : 6 bytes
• Ethernet Type : 2 bytes
• Data Field : Between 46 and 1500 bytes
• Frame Check Sequence : 4 bytes
• VLAN Tag : 4 Bytes

The size of each Ethernet frame excluding the Data Field equates to Ethernet overhead of 18 bytes ( 6
bytes source address + 6 bytes destination address + 2 bytes ether type + 4 bytes FCS or CRC = 18 bytes
overhead excluding VLAN Tag).

With VLAN Tag the Ethernet Frame will include another 4 bytes, which makes an Ethernet overhead of
22 bytes.

We can conclude the layer 2 overhead as:

• Layer 2 Overhead without VLAN TAG = 18 bytes
• Layer 2 Overhead with VLAN TAG = 22 bytes.

9

There is one more important factor which is missing here in Figure 2; for each frame that is transmitted
on the physical layer (Layer 1) or transmission medium (i.e. wire, cable or fiber) there is a 8 bytes
premble and a 12 bytes inter-‐frame gap (IPG) overhead added on the ethernet packet which is shown
below:
• Preamble : 8 bytes
• Destination Address : 6 bytes
• Source Address : 6 bytes
• Ethernet Type : 2 bytes
• Data Field : Between 46 and 1500 bytes
• Frame Check Sequence : 4 bytes
• Inter-‐Packet Gap (IPG) : 12 Bytes
• Min Frame Length : 64 bytes
• Max Frame Length : 1538 bytes

The preamble and IPG are mandated which occupied an additional 20 bytes (8 bytes Preamble + 12
bytes IPG) of overhead. This is important to include while determining Ethernet Frame efficiency on a
network. Why? Well it is built into the Ethernet specification; this is true for 10/100/1000/10000
Ethernet.
The overhead at Layer 1 and Layer 2 are:
• Layer 1 Overhead = 8 Bytes Preamble + 12 Bytes IPG = 20 Bytes

• Layer 2 Overhead = 6 Bytes Source Address + 6 Bytes Destination Address + 2 Bytes Ether Type
+ 4 Bytes FCS or CRC = 18 Bytes (without VLAN TAG)
• Total Layer 1 + Layer 2 Overhead = 18 Bytes + 20 Bytes = 38 Bytes
Layer 3 and Layer 4 Overhead

We discussed about the Layer 1 and Layer 2 overheads. Similarly, Layer 3 (IP Packets) and Layer 4 (TCP
Packets) also have header overheads. The IP and TCP Packet Format are shown below in Figure 3 and
Figure 4.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| IHL |Type of Service| Total Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Identification |Flags| Fragment Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
10

| Time to Live | Protocol | Header Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Destination Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Options | Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 3: Internet Datagram Header
• Version, IHL, TOS, total length is a total of 4 bytes

• ID, Flags, and Offset is a total of 4 bytes
• TTL, Protocol, an checksum is a total of 4 bytes
• Source address is a total of 4 bytes
• Destination address is a total of 4 bytes
Total Layer 3 IP Overhead = 20 Bytes
The Layer 4 TCP Packet Format is shown below in Figure 4.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Port | Destination Port |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Acknowledgment Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data | |U|A|P|R|S|F| |
| Offset| Reserved |R|C|S|S|Y|I| Window |
| | |G|K|H|T|N|N| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Checksum | Urgent Pointer |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Options | Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 4: TCP Header Format
• Source Port : 2 Bytes

• Destination Port : 2 Bytes
• Sequence Number : 4 Bytes
11

• Acknowledgment Number : 4 Bytes
• Data Offset, Reserved, Control Bits, Window : 4 Bytes
• Checksum, Urgent Pointer : 4 Bytes

Without getting into too much detail, just know that before the options in TCP, total bytes are 20 Bytes;
this is a layer 4 overhead. If you include the timestamp option, it is an additional 12 bytes; however, you
will also notice that options are a variable field that can be up to 44 bytes.

The total overhead at Layer 3 and Layer 4 is:
• Layer 3 IP Overhead = 20 Bytes
• Layer 4 TCP Overhead = 20 Bytes

12

Packet Rate Calculation
There are several ways to measure the packet transmit/receive rate. The two easiest methods to
calculate rate which is given below:

Method 1
Frame Size = 66 Bytes
Frame Rate = 10 Mbps

Frame Size is in Bytes, so we need to convert the rate to bytes first.

Frame Rate = (10 x 1,000,000)/8 = 1250000 bytes/second

Packet per Second = Network Bandwidth (Bytes) / Frame Size (including IFG and preamble)

Frame Size including IFG and Preamble = 66 bytes + 8 Bytes + 12 Bytes = 86 bytes

Transmit Rate in Packet per second (pps) = 1250000/86 = 14534.883 pps

Transmit Rate in Mbps = Packet per Second (PPS) * Frame Size (Bytes) * 8

Transmit Rate in Mbps = 14534.883 * 66 Bytes * 8 = 7.67441 Mbps
Method 2
Frame Size = 66 Bytes (including CRC)
Frame Rate = 10 Mbps
Each Frame Overhead = 20 Bytes (8 Preamble + 12 Inter-‐frame Gap)

Total Frame = Frame Size + Frame Overhead = 66 + 20 = 86 Bytes

Overhead Calculation in per% = [Overhead (Bytes) / Total Packet Size (Bytes)] * 100

Total Overhead = (20/86)*100 = 23.25 %

Transmit Rate = [100% -‐ Overhead (in per%)] * (Frame Rate/100)

Transmit Rate (TX) = 100% -‐23.25% = 76.75
= 10 Mbps * (76.75/100) = 7.675 Mbps
13

Network Efficiency Calculation
The efficiency of a network can be measured by calculating a total overhead of a network. Again there
are several ways to determine network efficiency. One straightforward method is given below to
measure the efficiency in percentage.

Let’s suppose the following parameters:
Frame Size = 66 Bytes
Frame Rate 10 Mbps

We have already calculated the packet rate (pps) in Packet Rate Calculation section for Frame size of 66
Bytes and Frame rate of 10 Mbps, which is 14534.883 pps.

Rate in Packet per second (pps) = 1250000/86 = 14534.883 pps

Layer 2 has overhead of 18 Bytes + Layer 1 20 Bytes Overhead

bps = PPS x Frame Size (Bytes) x 8
Overhead in per% = bps/bandwidth

• 23.2% of overhead is consumed by the preamble and IFG: 20 Bytes

14534.883*20*8 = 2325581.28 bps

Overhead per% = 2325581.28/10,000,000 = 23.25%

• 20.93% of overhead is consumed by the Ethernet Header including CRC: 18 Bytes (Assuming
Untagged Packet)

14534.883*18*8 = 2093023.152 bps

Overhead per% = 2093023.152/10,000,000 = 20.93%

Total Overhead = 23.25 + 20.93 = 42.18%

Network Efficiency in per% = 100% -‐ Total Overhead (in per%)

Network Efficiency% = 100 – 42.18
= 57.82%

The efficiency indicates that in each packet of 66 Bytes there will be 57.8% of IP Payload and rest will be
overhead.
14

Maximum Transmission Unit (MTU) and Maximum Segment Size (MSS)
Packet size, often referred to as MTU (Maximum Transmission Unit) is the greatest amount of data that
can be transferred in one physical frame on the network. Larger Frames in Ethernet Network are more
efficient than smaller frames. The larger MTU size is recommended for faster networks. Any frame size
greater 1500 bytes of payload are considered as a Jumbo Frame. Generally, jumbo frames can carry up
to 9000 bytes payload. The standard MTU size is 1500 Bytes.

Each transmission unit consists of header and actual data. The actual data is referred to
as MSS (Maximum Segment Size), which defines the largest segment of TCP data that can be
transmitted. Essentially,

MTU = MSS + (TCP and IP headers)
MSS = MTU – 40 (TCP and IP headers)
Frame/Packet Size Selection

There is no single answer for packet/frame size selection. MTU depends on the application. For example,
enabling MTU for jumbo frames has no real benefits from Internet Traffic point of view because
internet traffic is mostly based on HTTP whose average packet size vary from 64 Bytes to 512 Bytes but
generally Jumbo Frame are considered more efficient than Smaller Frame.
Let’s consider two frame sizes, one smaller packet size 64 bytes and other bigger packet size 1500 Bytes
on a 100 Mbps network in order to understand the statement “ bigger size frame are better than smaller
frame”.
Frame Size 64 Bytes

The 64 bytes frame has already included the Ethernet headers, so the actual IP payload is 46 bytes. The
extra 18 bytes are Layer 2 overhead as discussed above.

As discussed earlier, the packet per second is required to determine the efficiency.





Layer 2 has about 45 percent of overhead utilizing 64 byte frames

• 23.8 percent of overhead is consumed by the preamble and IFG: (20 bytes)

15

PPS x Frame Size x 8 = bps/bandwidth = percent
148809.5 * 20 * 8 = 23169520/100000000 = 23.8 %

• 21.4 percent of overhead is consumed by Ethernet header and trailer (18 bytes)

148809.5 * 18 * 8 = 21428568/100000000 = 21.4 %

Total IP Payload Utilization or Efficiency = 100 – 45.2 = 54.8

54% of IP payload is left which is not very efficient.
.
Frame Size 1500 Bytes

The 1500 bytes frame has already included the Ethernet headers, so the actual IP payload is 1482 bytes.
The extra 18 bytes are Layer 2 overhead as discussed above.

As discussed earlier, the packet per second is required to determine the efficiency.





Layer 2 has about 2.49 percent of overhead utilizing 64 byte frames

• 1.31 percent of overhead is consumed by the preamble and IFG: (20 bytes)

8223.684 * 20 * 8 = 23169520/100000000 = 1.31 %

• 21.4 percent of overhead is consumed by Ethernet header and trailer (18 bytes)

8223.684 * 18 * 8 = 21428568/100000000 = 1.18 %

Total IP Payload Utilization or Efficiency = 100 – 2.49 = 97.5%

97.5% of IP payload is left, which is efficient, compare to 64 Bytes Frame Efficiency.
16

Propagation, Serialization, and Queuing Delay
The combination of propagation, serialization and queuing delay often produces a complex and variable
network Latency. Serialization and queuing delay are very negligible on high speed networks, and at
most are not included in any real calculations.
Serialization Delay
Serialization delay is the amount of time it takes for a networking device (such as a router) to encode a
packet onto the wire for transmission.
Packet Size vs. Latency

(MSS + Header) * 8 bits/byte
-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ = delay (per hop)
1,544,000 bits/sec.

Let's examine a transfer of 1,500,000 bytes of data using different packet size over a T1 line
(T1=1,544,000 bits/sec) using the above formula.

Then, using different MTU values, we can calculate the relevance of packet size to latency.

If MTU = 1500, then: (1460+40) * 8 / 1,544,000 = 7.772 ms delay per hop
If MTU = 576, then: (536+40) * 8 / 1,544,000 = 2.924 ms delay

Assuming a transfer over 10 hops, the 1500 MTU would yield 77.72 ms delay, while a 576 MTU would
take 29.24 ms to transfer over a T1 line.

It is obvious that smaller packets will be transmitted faster, simply because of the throughput limitation
of the line.

So serialization is a factor for a T1 circuit; there is a fixed 7.8ms of latency (not very high, but high
compared to Fast Ethernet, and is why serialization is not a major factor for high speed networks).

Queuing Delay
Routers can only process certain number of packets at a time. If packets are arriving faster than the
router can process them, the router will place the packets into the queue which is called queuing delay.

When discussing the utilization of a circuit, it is an important to discuss using an average utilization over
some unit of time.
Example:
A 100 Mbps circuit that is 70% utilized is 30% empty; 70 Mbps utilized and 30 Mbps available. This does
not however, mean that the interface is 70% utilized, the interface is in use or it is not in use. When a
17

packet/frame arrives at a routers interface that needs to be routed out another interface, the router will
forward/route that packet, but at that moment in time, the interface is 100 % utilized. During the time
that a interface is in use (100% utilized) and another packet arrives at the same time that the interface is
in use, the packet must be queued by the router until the router has finished processing the first packet;
then it can process the packet that arrived after the first; Most Internet backbone routers will not have
any queuing delays unless the circuit is around 90 to 100% utilized (Considered congested).

Propagation Delay
The propagation speed of a medium refers to the speed that the data travels through that medium.
Propagation delays differ between mediums, which affect the maximum possible length of the Ethernet
topology running on that medium.

In the following table, ‘c ‘refers to the speed of light in a vacuum, or 300,000 kilometers per second.
Medium Propagation Speed
-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐
Thick Coax .77c (231,000 km/sec)
Thin Coax .65c (195,000 km/sec)
Twisted Pair .59c (177,000 km/sec)
Fiber .66c (198,000 km/sec)
AUI Cable .65c (195,000 km/sec)

From these values, the size of a bit on 10BaseT can be calculated. 10BaseT is twisted pair, which has a
propagation delay of 177,000 km/sec. 177,000 km/sec divided by 10 million bits per second is 17.7
meters, or the size of a single bit on a 10BaseT network.

The maximum propagation delay through the network can be calculated by dividing the maximum
length by the speed. For 10Base2 thin coax network, this is 185 meters divided by 195,000 km/sec, or
950 nanoseconds. If the actual propagation delay from one end of the network to the other is greater
than 950 nanoseconds, late collisions may occur.

Propagation delay depends on the speed of light; is speed of light constant? Well, you cannot make it go
faster (Front Velocity); however, you can slow it down. In reference of space (outer space) light is the
most effective, meaning that light travels the fastest in a vacuum, which is roughly 186,000 miles per
second. The index of refraction is a way of measuring how much the velocity of a wave is reduced inside
a medium (Vacuum, fiber, glass, etc.). The larger the index of refraction is, the slower the light travels in
that medium. Let’s take a known number: 186,000 miles per second, the index of refraction for light in a
space is 1. You can calculate index of refraction of another material by dividing the speed the speed of
light in a vacuum by the speed of light in another (n = c/v). The typical value for the core of a fiber is
1.48. Remind you, this latency is fixed based on physics. It is there and will always be there.

18

Example:
What kind of propagation delay should I expect from Denver to Seattle?
In this example, we will use Level3's Network Map: Level3 Map
Per Google maps, the distance to Denver to Seattle is 1327 Miles.
The distance from Denver to Seattle: 1327m
The speed of light in vacuum is 186,000 miles/Sec (m/s).
The speed of light in fiber is roughly 68% of the speed of light in vacuum. (n = c/v [1/1.48=.675]).
The speed of light in fiber is 186,000m/s * 0.68 = 126,480miles/s
The one-‐way delay to Seattle is 1327m / 126,480miles/s x 10^3 = 10.4ms.
The round-‐trip time to Seattle from Denver and back is 20.8ms via Level3.
Actual latency reported by ping is: 28ms (The difference of 20.8 and 28 is negligible)

There is some queuing and serialization delay, but this is very negligible.

If you are facing a very high latency where it is NOT expected; however, if you executed a “traceroute”
to a destination and only one hop had high latency and the very next hop had a much lower latency, this
does not necessarily indicate that there is an issue; the could be a result of how long it took the router
to generate the ICMP exceed TTL message (Router is just busy with other important traffic, its concern is
NOT ICMP, most cases ICMP is specifically rate-‐limited and de-‐prioritized). If there is an actual issue, the
latency will continue or increase for all future hops afterward.

In simple terms; roughly, every 125 miles, you will have 1ms of latency (assuming no queuing or
processing delays).

19

Network Latency/ Round Trip Time (RTT)
In simple terms, network latency is the delay (time consumed) sending a packet from one host to
another and receiving a response back. The time that it takes is also referred to as the round trip time
(RTT). This is very important to remember; why? For one TCP uses it and the TCP window size to
determine the sending rate. Just because your bandwidth is 10 Mbps does not mean you will be able to
achieve a maximum throughput of 9.4 Mbps (Remember the TCP and Ethernet overhead!). The concept
of "end to end" is used as a relative comparison with "hop-‐by-‐hop"; end to end delay is the sum of
delays experienced at each hop from the source to the destination. At each hop, there is a transmission
delay (Serialization Delay [Fixed]), Variable delay (Queuing and processing delay), and finally
propagation delay, which we have already covered above. These are important topics to understand,
because they contribute to the maximum throughput you can obtain.

Example:
Let's examine a client computer communicating with a server over a geosynchronous satellite link.
The client's request (every packet) has to travel 22,300 miles to the satellite, then 22,300 miles down
to the server.
Uplink Distance = 22,300 miles

Downlink Distance = 22,300 miles
Total Distance = 44600 miles
Then, when the server sends its response, it has to travel the same distance back to the client, adding
another 22,300 miles up + 22,300 miles down.
Thus, that simple packet of data traveled at least 89,200 miles (44600 x 2).
Speed of light = 3 x 10^8 m/s (186,000 miles per second),
Round-Trip delay (RTT) = Total Distance (miles) / Speed of light (miles)
Minimum Round-Trip delay (RTT) on a satellite = 89,200/186,000 = 0.479 ~ 0.5 seconds

20

Bandwidth Delay Product (BDP)
The Bandwidth*Delay product (BDP) for short, determines the amount of data that can be in transit in
the network (just as RWIN). It is the product of the available bandwidth and the latency (RTT). BDP is a
very important concept in a window-‐based protocol such as TCP, as throughput is bound by the BDP.

The BDP states that:
• BDP (bits) = bandwidth (bits/second) * latency (seconds), or
BDP (bytes) = bandwidth (KBytes/sec) * latency (ms), or
BDP (bytes) = bandwidth * 8 (Kbits/sec) * latency (ms)

What does it mean? The BDP and the TCP Receive Window limit our connection to the product
of the latency and the bandwidth. A transmission cannot exceed the RWIN / latency value.

In TCP connections, the large Bandwidth Delay Product (BDP) of high latency connections, combined
with relatively small TCP window sizes on many end systems, effectively causes the throughput of a high
latency connection to drop sharply with latency and packet loss. This can be rectified with various
techniques, such as increasing the TCP congestion window size; however, with many new operating
systems, CTCP, Window scaling, and Receive Window auto-‐tuning, there is no need to manually set your
RWIN settings. With auto tuning, the receiver buffer size (and TCP window size) is dynamically updated
(auto-‐tuned) for each connection. Older operating systems, some Linux 2.4 kernels, Windows 98, ME,
2000, and XP do not have RWIN Auto-‐tuning or window scaling implemented, so it was/still is necessary
to manually configure the RWIN values.

21

Window Size and Scaling
A number of extensions have been made to TCP over the years to increase its performance over fast
high RTT links ("long fat networks", or LFNs for short), this also includes Satellite links; these RF links
(Layer 1 media) have long delays. Why is the TCP Window size important? Well, it is the amount of
buffering allowed by the receiver before an ack is required. If you use a low RWIN value and latency
increases, your transfer rate will decrease, if you use a latency number that is too small, and your
latency increases, your RWIN will get filled and throttle your transfer rate. Most Server class OS's has a
value set from 32 Kbytes to 128 Kbytes. This allows a reception of 21 to 85 packets before an ack is
required assuming a MTU of 1500 (32000/1500 = 21 and 128000/1500 = 85).
Optimizing the TCP Receive Window

TCP receive window depends on guess what? Latency! This is why the low RTT from point A to B is
important! Also, what is important to understand that packet loss is interpreted by TCP as network
congestion, so TCP will reduce the size of the window scale Bandwidth delay product (BDP) is a term
primarily used in conjunction with the TCP to refer to the number of bytes (Amount of data) necessary
to fill a TCP path at any one time, in order to fully utilize the available capacity of that path. High
performance networks have very large BDPs.

To give a practical example

Bandwidth (In bytes) x latency = TCP Window size

Optimal TCP Receive Window Calculation

1. Determine maximum segment size (MSS for short, packet size minus TCP/IP headers. Displayed by
the SG TCP Analyzer), as well as the line's maximum anticipated latency and advertised
maximum bandwidth, (for example, 1460 MSS, 300ms max latency, 6Mbit/s max bandwidth).

2. Find the unscaled RWIN value (largest even multiple of MSS less than 65535):
65535 / 1460 (MSS) = 44.9
rounds down to even number = 44
44 x 1460 = 64240 (this is the optimal unscaled RWIN value for broadband connections with
1500 MTU/1460 MSS)

3. Multiply the unscaled RWIN by 2 until it is close to, or larger than BDP
Our BDP for 6Mbps @300 latency is:
6000 kbps x 300ms = 1800000 / 8 = 225000
64240 (unscaled RWIN) x 2 x 2 ... = 256960 (number larger than, or close to BDP, this is the
optimal RWIN)

22

Throughput
Throughput (or transfer rate) in data transmission is the amount of data moved successfully from one
place to another in a given time period.

For data networks, throughput is usually measured in number of bits per second (bps) that are
transmitted, also quoted as Kilobits per second (Kbps) or Megabits per second (Mbps).

Example
Consider 2 hosts on a switch, with the switchports configured for a bandwidth of 100 Mbps:

12500000 (100000000/8) X .001 = 12500 Byte RWIN

Now, consider a host; Host A configured for a bandwidth of 100 Mbps in a data center in Seattle and
Host B configured for a bandwidth of 100 Mbps in a data center in Denver and using the Internet to
transfer files and host A is the initiator. If the RTT between the hosts is 192 ms, then the RWIN of host B
should be 2.4 MB. 2.4 MB is needed to sustain theoretical throughput. Assume now that the 2 hosts are
Windows 2000 servers (In the original release versions of Microsoft Windows 2000 Service Pack 1 (SP1)
and Windows 2000 Service Pack 2 (SP2), the default size of the TCP receive window was 17,520 bytes.)
and the default is not changed, the maximum throughput you will be able to achieve is 730 Kbps when
there is 192 ms of latency and the bandwidth is 100 Mbps. Even in the absence of packet loss in a
network, windowing can limit throughput. This is due to the fact that the TCP transmits data up to the
window size before waiting for the packets, the full bandwidth of the network may not always get used.

The limitation caused by window size can be calculated as follows:

RWIN/RTT=Throughput
Throughput vs. packet size

Using the same formula from our previous example, let's assume we need to transfer 1 MByte file over
the same T1 line.

1 MByte = 1024 KB = 1,048,576 bytes.
If MTU = 1500, then: (1460+40) * 8 / 1,544,000 = 7.772 ms delay per hop
1 MByte / MSS = 1,048,576 bytes / 1460 = 718.2, or effectively 719 packets to transfer 1 MByte.
Then, to transfer 1Mbyte: 719 packets * 7.772 ms delay per hop = 5588.068 ms, or 5.588 seconds
per hop.
If we are transferring our 1 MByte file over 10 hops, it will ideally take us:
(1st packet * 10 hops * 7.772ms delay) + 718 * 7.772 = 5.658 seconds.

If MTU = 576, then: (536+40) * 8 / 1,544,000 = 2.924 ms delay per hop.
1 MByte / MSS = 1,048,576 bytes / 536 = 1956.3, or effectively 1957 packets to transfer 1 MByte.
23

Then, to transfer 1 MByte: 1957 packets * 2.924 ms delay per hop = 5722.268 ms, or 5.722 seconds
per hop.
If we are transferring our 1 MByte file over the same 10 hops:
(1st packet *10 hops * 2.924ms delay) + 1956 * 2.924 = 5.748 sec.

The difference comes from the fact that when using larger packets the overhead is smaller. To transfer 1
M Byte, if using MTU of 1500 there are 719 * 40 = 28,760 bytes of overhead, while if using MTU of 576
1957 * 40 = 78,280 bytes, additional 49,520 bytes of headers transferred each M Byte. For our 10-‐
hop transfer, the additional overhead accounts for just a fraction of a second, however it makes a
difference if you consider large transfers over long periods of time. This difference is an also higher in
practice, considering TCP options and the fact that modern TCP/IP implementations tend to use larger
headers (additional 12 bytes header space for Timestamps for example). Also, the above example uses
best case scenario, not considering higher latency and congestion.

24

Understanding Ethernet Clock PPM
This section explains a phenomenon in which packets drops or high latency are observed at line rate
(100%) traffic.
Problem Description
• When sending line rate (100%) traffic from test port #1 to test port #2
• You may observe packet drops
Reasons for Packet Drop
• Clock accuracy
• Transmit clock speed of two identical ports (for example 2 TenGig ports) are not exactly the
same.
What is ppm?
• Clock accuracy is defined in terms of ppm (parts per million)
• It is a convenient way of comparing accuracies of different crystal clock specifications
25

Ethernet clock specification
According to the IEEE 802.3 specification
• For 1GE, clause 35 (Table 35-‐8) states that TX clock frequency should be +/-‐ 100 ppm
• For XAUI, clause 47 (Table 47-‐1) states that the baud rate tolerance per lane should be +/-‐ 100
ppm
Clock Jitter/Wander
• Even the clock speed of a particular port may change over time due to clock jitter and/or
wander. This may be due to noise and/or ambient temperature
ppm v/s Packet Loss
• An error of 100 ppm means 100 clocks are lost every 1 million clocks
• 1 Byte is transferred every 125 MHz Clock
• 100 clocks lost translates to a loss of 1 packet (of size 80B)
• 64 Byte will drop more than 1 packet
• Since 1GE clock period is 8ns (125MHz), 1 second translates to 125 million clocks
• Hence, 125 packets can be lost per second (Larger packet sizes drop will be less and smaller
packet sizes drop will be more)
• For 10GE an error of -‐100 ppm results in 1250 packets lost per second
• 100 Bytes = 1 Million Clocks
• 1 second = 100 x 125 Bytes =>12500 Bytes
• For 64 Byte Packet:
o Number of packet loss 12500/84 = ~149 Packets
o (where 84 is 64 Byte packet + 20 Byte IPG and Preamble)
• For 1500 Byte Packet:
o Number of packet loss 12500/1520 = 8 Packets
o (where 1520 is 1500 Byte packet + 20 Byte IPG and Preamble)
Control packets
• When a switch forwards traffic, it also adds additional locally originated control packets to the
transmit side.
• The control packets may include:
o L2 – CDP, keep alive, L2 protocols, ARP, etc.
o L3 – OSPF, ISIS, etc.
26

• Normally, switch generates more control packets than they receive from test equipment.
o If a switch receives line rate traffic from test port and sends line rate traffic + some
control traffic, it causes oversubscription on the egress port
o The rate of latency increase or packet drops due to control plane traffic depends on the
rate of control plane traffic
Example
Test Result with 10Gig L3 Unicast Line rate (100%) Traffic
• When sending line rate (100%) traffic from test port 2/1 to test port 2/2
• Unicast L3 Traffic

• Traffic Profile: 10 Gig Input interface and 10 Gig output Interface
• Packet Size:64 bytes
• Traffic Rate: 10Gig line rate (100%)
• Packet Drops (crystal on Switch/Route is about 10 ppm slower than the IXIA)
Traffic Item Tx Frames Rx Frames Frames Delta Loss %

10G-10G 6544844466 6544789847 54619 0.000834535
Test Result with 10Gig L3 Unicast 99.90% of Line rate Traffic

• Traffic Profile: 10 Gig Input interface and 10 Gig output Interface
27

• Traffic Rate: 99.90% of 10Gig line rate
• No Drops

10G-10G 1931926417 1931926417 0 0
• When sending line rate (100%) traffic from test port 7/2 to test port 9/4

• Traffic Profile: 1Gig Input interface and 1Gig output Interface

1G-1G 654484438 654478972 5466 0.000835161
Test Result with 1Gig L3 Unicast Line rate (100%) Traffic with Switch Tweaked IPG

28

• IPG is tweaked to simulate faster crystal. No Drops

10G-10G 1931926417 1931926417 0 0
Test Result with 1Gig L3 Unicast 99.99% of Line rate Traffic

• Traffic Rate: 99.99% of line rate
• No Drops
1G-‐1G 193362200 193362200 0 0
29

Final Thoughts
Generally, it's logical to assume larger packets are better, because of all the following factors:
• network -‐ reduced number of headers, as illustrated above
• routers -‐ less routing decisions
• clients -‐ less protocol processing and device interrupts

However, if pure throughput is not the ultimate goal, smaller packets might be more "responsive" since
they take less time to travel throughout the network. That effect might be preferred in some
applications and online gaming, at the expense of throughput.
Ultimately, packet size should be decided based on the type of the desired result, considering the
underlying network as well, to avoid negative factors such as fragmentation of packets. Still one has to
realize the fact that larger packets will still transmit more useful data than smaller packets, and that
there is no single "best" solution for all applications.

30

Network Performance Metrics

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Network Performance Metrics

Uploaded by

Copyright:

Available Formats

Waris Sagheer (waris)

• Bit, Bytes Conversion and Bandwidth Reference Guide

Bits vs. Bytes Confusion

Memory Size/Data Storage Representation

Bits/Bytes Reference Table

Bandwidth Reference Table

DSx / Ex / Jx Data rate DS0 multiple Carrier

Figure 1: OSI Model 7 Layers

Layer 1 and Layer 2 Overhead

Figure 2: Ethernet Frame Format

• Destination Address : 6 bytes

The overhead at Layer 1 and Layer 2 are:

• Layer 1 Overhead = 8 Bytes Preamble + 12 Bytes IPG = 20 Bytes

Layer 3 and Layer 4 Overhead

Figure 3: Internet Datagram Header

• Version, IHL, TOS, total length is a total of 4 bytes

Total Layer 3 IP Overhead = 20 Bytes

The Layer 4 TCP Packet Format is shown below in Figure 4.

Figure 4: TCP Header Format

• Source Port : 2 Bytes

Frame/Packet Size Selection

Frame Size 64 Bytes

Frame Size 1500 Bytes

Packet Size vs. Latency

Uplink Distance = 22,300 miles

Speed of light = 3 x 10^8 m/s (186,000 miles per second),

Round-Trip delay (RTT) = Total Distance (miles) / Speed of light (miles)

Minimum Round-Trip delay (RTT) on a satellite = 89,200/186,000 = 0.479 ~ 0.5 seconds

Optimizing the TCP Receive Window

Optimal TCP Receive Window Calculation

Throughput vs. packet size

Reasons for Packet Drop

What is ppm?

According to the IEEE 802.3 specification

ppm v/s Packet Loss

• Unicast L3 Traffic

Traffic Item Tx Frames Rx Frames Frames Delta Loss %

• Unicast L3 Traffic

Traffic Item Tx Frames Rx Frames Frames Delta Loss %

• Unicast L3 Traffic

Traffic Item Tx Frames Rx Frames Frames Delta Loss %

• Unicast L3 Traffic

Traffic Item Tx Frames Rx Frames Frames Delta Loss %

• Unicast L3 Traffic

Traffic Item Tx Frames Rx Frames Frames Delta Loss %

1G-­‐1G 193362200 193362200 0 0

You might also like

1G-‐1G 193362200 193362200 0 0