744
4, APRIL 2020
A Dynamic Threshold Calculation for Congestion Notification
in IEEE 802.1Qbb
Waqar Khurshid , Imran A. Khan, Miss L. M. Kiah, Osman Khalid, and Sajjad A. Madani
Abstract— Ethernet has been a backbone of data centers since
inception of cloud computing. The reason is its simplicity, cost
effectiveness, wide existing base, and maturity. However, Ethernet
is unreliable when it comes to providing guaranteed transmission.
Consequently, IEEE 802.1Qbb standard has been introduced
to provide reliability based on congestion notification (CN)
mechanism of IEEE 802.3x. However, IEEE 802.1Qbb has some
issues and limitations, such as higher buffer space reservation,
lack of support for heterogeneous switching devices, and low
throughput and efficiency of the network. This letter proposes
a novel approach for modified CN mechanism that provides
44 percent less space reservation for CN mechanism, support
for heterogeneity, higher throughput, and efficiency.
Index Terms— Congestion notification, IEEE802.1Qbb, optimal
threshold selection, reliable layer-2 infrastructure.
I. I NTRODUCTION
I N THE past three decades, Ethernet has been the most
dominant communication technology worldwide. Ethernet
has also become increasingly ubiquitous as workplaces are
more automated with Internet connectivity. Ethernet is usually
considered unreliable for majority of applications due to the
fact that route between a source to a destination typically
consists of multiple intermediate nodes, and there is inade-
quate feedback about buffer occupancy between intermediate
hops [1], [2]. A reliable network has inherent mechanism
to provide reliability at infrastructure level. However, upper
layer protocols are responsible for reliability provision in
Ethernet which should not be the case, as the reliability
mechanism in Ethernet should already exist at layer-2 [2],
[3]. A network switch silently drops packets when it is faced
with congestion, and the upper layers infer the packet drop
incidence as network congestion. Unawareness of the sender
for such packet drops contributes towards the unreliability
of Ethernet. Novel Ethernet faces problems in flows where
reliability cannot be integrated in the upper layer protocols like
Fiber Channel over Ethernet (FCoE) or real-time applications
Manuscript received November 25, 2019; revised January 1, 2020; accepted
January 1, 2020. Date of publication January 13, 2020; date of current
version April 9, 2020. The associate editor coordinating the review of this
letter and approving it for publication was M. Khabbaz. (Corresponding
author: Waqar Khurshid.)
Waqar Khurshid is with the Department of Computer Science, COMSATS
University Islamabad–Abbottabad, Abbottabad 22060, Pakistan, and also
with HITEC University, Taxila 47080, Pakistan (e-mail: waqarkhurshid@
cuiatd.edu.pk).
Imran A. Khan and Osman Khalid are with the Department of Computer
Science, COMSATS University Islamabad–Abbottabad, Abbottabad 22060,
Pakistan (e-mail: imran@cuiatd.edu.pk; osman@cuiatd.edu.pk).
Miss L. M. Kiah is with the Faculty of Computer Science and Tech-
nology, University of Malaya, Kuala Lumpur 50603, Malaysia (e-mail:
misslaiha@um.edu.my).
Sajjad A. Madani is with the Department of Computer Science, COM-
SATS University Islamabad–Wah Cantt, Wah Cantt 47040, Pakistan (e-mail:
madani@comsats.edu.pk).
Digital Object Identifier 10.1109/LCOMM.2020.2966198
1558-2558 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Edith Cowan University. Downloaded on April 17,2020 at 11:47:56 UTC from IEEE Xplore. Restrictions apply.
IEEE COMMUNICATIONS LETTERS, VOL. 24, NO.
that use User Datagram Protocol [2]. Hence, a hop to hop
based congestion notification (CN) method is introduced in
IEEE 802.1Qbb to provide reliability to such applications
along with Priority-based Flow Control (PFC). This standard is
not just responsible for reliability provision for real-time flows
on a hop to hop basis, but it elevates reliability of infrastructure
as well, thus purges the need for upper layer protocols. In IEEE
802.1Qbb, layer-2 switches provide feedback regarding buffer
conditions to the uplink switches when the buffer occupancy
stretches to a certain threshold limit. However, this threshold
selection is based on worst-case delay scenarios that roots
to issues of higher buffer space reservation, limitation of
heterogeneity of switching devices, and less throughput and
efficiency due to increased pause instigation [2].
This letter proposes a novel approach to dynamic threshold
estimation mechanism for IEEE 802.1Qbb instead of worst-
case threshold implementation. The following are the major
contributions of the proposed work.
a) A dynamic buffer threshold calculation technique is
devised that shows 44 percent less buffer space reservation
for lossless operations
b) The proposed work provides support for heterogeneity of
switching devices
c) Compared to existing work, high throughput and greater
efficiency is experienced in queuing and storing network
packets
The efficiency of our approach in real-world congested
traffic scenario is extensively evaluated through simulations
carried in widely used Network Simulator 2 (NS2) [4].
II. IEEE 802.3X AND IEEE 802.1Qbb
CN mechanism can be either selective or non-selective.
Selective CN technique stops the selective flows that are
causing congestion, whereas permitting remaining flows to
continue normally. On the contrary, non-selective technique
pauses all the traffic on the link when a pause signal is
acknowledged by the sender [2].
In 1997, IEEE introduced a non-selective standard for CN to
fabricate a reliable layer-2 infrastructure called IEEE 802.3x,
where packet drops in the traditional IEEE 802.3 LANs could
be tackled at layer-2 [5]. In such implementation, the receiver
used a pause frame (64-byte MAC control frame) to convey
the buffer occupancy feedback of the receiver to the upstream
sender. This allows sender to stop sending packets to the
particular receiver [5]. The pause mechanism activates when
the receiver’s buffer occupancy reached a certain limit, called
threshold. The pause frame contained the pause duration
that was defined as quanta. One quantum was equal to the
time to transmit 512 bits at the current link speed onto the
network. When the threshold time expires, the sender resumes
KHURSHID et al.: DYNAMIC THRESHOLD CALCULATION FOR CN IN IEEE 802.1Qbb
Fig. 1. IEEE802.1Qbb pause frame format.
the transmission [1]. Moreover, the paused traffic can be
resumed by sending a pause frame with zero (0) quanta value.
The popularity of IEEE 802.3x was short lived due to its inef-
ficiency for real-time data and time-sensitive applications as
compared to traditional IEEE 802.3 Ethernet. In IEEE 802.3x,
once the receiver initiates pause, the sender is not allowed
to produce any traffic flow for that receiver; consequently,
stopping all the traffic that might entail different Quality of
Service (QoS) [1], [2].
In March 2008, IEEE authorized CN standard with selec-
tive CN mechanism along with priority realization. Conse-
quently, IEEE 802.1Qbb (PFC) was proposed in 2010 and
in June 2011 it became an active standard [2], [6]. IEEE
802.1Qbb was an improvement to the previous 802.3x pause
frame. IEEE 802.1Qbb PFC supported class of service (CoS)
for eight different traffic flows on a single link [6]. Moreover,
802.1Qbb PFC applied separate flow control for each CoS,
and each class was prioritized on the basis of flow repre-
sented by the specific class. Layer-2 switches maintained eight
different queues, one for each CoS. While a queue reaches
the threshold, a pause frame was instigated for only that
particular queue, and initiating CoS was temporarily paused
instead of all CoS. The threshold was selected on the basis of
worst case delay scenarios. The pause frame contained eight
different values in quanta that represented each CoS. Each
class had a quanta value of 2-byte, and the maximum value
was 65,535. Fig.1 shows the format of the pause frame for
IEEE 802.1Qbb PFC.
The concept of initial constant congestion window (ICCW)
was introduced in [7] based on injecting two different thresh-
olds to reduce congestion. However, the work in [7] deals
with Wireless Sensor Network (WSN) and based on end to
end congestion control.
The selection of an optimal threshold was forever been a
challenge for CN mechanisms [2], [8]. Whenever a pause
frame is sent by receiver it is most probable that the frame
will face some delays before it can be received by the sender.
During that delay, more traffic will be sent to downstream
receiver that needs to be accommodated in the receiver’s
buffer. If the threshold for pause is set to high, then receiver
will not be able to accommodate the incoming traffic. On the
contrary, if the threshold is set too low then there is a
Authorized licensed use limited to: Edith Cowan University. Downloaded on April 17,2020 at 11:47:56 UTC from IEEE Xplore. Restrictions apply.
745
possibility that some buffer space will not be utilized in
efficient manner [2]. To calculate the delay caused by pause
frame, following delays should be considered in order to find
out the optimal threshold [8].
a) Receiver’s maximum transmission unit delay
b) Receiver’s transceiver latency
c) Pause frame’s serialization delay
d) Pause frame’s propagation delay
e) Sender’s maximum transmission unit delay
f) Sendery’s transceiver latency
g) Sender’s response time
The aforementioned delays can occur from pause instigation
till pause actually initiated by the sender. During that time,
more data will be sent to the receiver to be accommodated
in the buffer. IEEE 802.1Qbb considers worst cases of these
delays. The delay value (DV) is as follows [2], [3].
DV = 2(Fmax ) + FP F C + 2(P ropc ) + T Xdr
+ RXdr + RXds + T Xds + HDs . (1)
Here, 2(Fmax ) is the MTU serialization delay of the sender
and the receiver for the ongoing packet, FP F C is the serializa-
tions delay of the pause frame, 2(P ropc ) is the propagation
delays of the cable for the sent pause frame and response by
receiver, T Xdr and RXdr are transceivers’ latencies of the
receiver, RXds and T xds are the transceivers’ latencies of the
sender, and the HDs is delay of higher layer for pause frame
creation.
In IEEE 802.1Qbb (PFC) there are 8 different CoS, and
DV is calculated for each CoS separately because each CoS
has its own buffer queue [3]. For any given device, the Inter-
face Delay comprises of both receive and transmit paths
(i.e., ID = RXd + T Xd) [3], resulting:
DV = 2(Fmax ) + FP F C + 2(P ropc ) + IDr + IDs + HDs .
(2)
As per IEEE 802.1Qbb, both the sender and the receiver are
considered to be communicating on the same technology [3].
Therefore, we get:
DV = 2(Fmax ) + FP F C + 2(P ropc ) + 2(ID) + HDs . (3)
The resulted DV in this scenario is calculated as:
DV = 2(16160) + (672) + 2(5556) + 2(12288)
+ (25504) = 145384 bit times. (4)
This delay value is calculated as a worst case scenario
roughly equivalent to 18KB for each CoS [3]. IEEE 802.1Qbb
assumed that sender and receiver are of same technology
thus limitation for heterogeneity is not addressed. Current
implementation of the standard in industry is still based on the
DV proposed based on worst case delays and mainly employed
by the CISCO nexus series switches for datacenters.
Based on the calculations above we are faced with the two
major concerns:
a) What will be the actual DV in real-world scenario?
b) How to employ heterogeneity in the real world scenario?
c) What are the effects of actual DV on the performance and
throughput of the network?
746
III. P ROPOSED T ECHNIQUE
To calculate the actual DV in real-world scenario and
employ heterogeneity, we utilize dynamic calculation of delay
values in the boot up process of the network as spanning tree
algorithm. To calculate the actual delay value we must first
analyze each delay individually in (1).
The parameter 2(Fmax ) in (1) is the serialization delay of
the sender and the receiver. If the sender or receiver started
transmitting a full size frame, then the pause instigation has to
wait for the serialization of the ongoing MTU that cannot be
predicted at the network boot up. FP F C is the time required
to create and serialize the pause frame by the receiver that
also cannot be predicted at the network boot up. However,
2(P ropc ) is the delay of the cable to send the pause frame
and receive the acknowledgement frame that is calculated
dynamically at network boot up. The parameters T Xdr and
RXdr , and RXds , and T Xds , and HDs are computed at run
time. The aforementioned delays are variable due to the length
and speed of the link, heterogeneity of network devices, and
the switching fabrics of different devices making such delays
unavoidable and impervious to be fixed. Therefore, to calculate
actual DV at network boot up, each switch creates and sends a
pause frame to its upstream switch and waits for the response.
Then, all the delays can be calculated by round trip time (RTT)
of the frame. So the RTT will be:
RT Tinitial f rame = 2(P ropc ) + T Xdr + RXdr
+ RXds + T Xds + HDs .
Now the delay value can be calculated as:
DV = 2(Fmax ) + FP F C + RT Tinitial f rame .
Moreover, we can calculate the DV as per IEEE 802.1Qbb
maximum MTU and pause frame size as:
DV = 2(16160) + (672) + (RT Tinitial f rame × 512). (7)
Maximum frame size in IEEE 802.1Qbb is 2000 octets +
preamble (16160 bits). PFC frame is 64 octets + preamble
(672 bits), and 512 bits per quanta value for the RTT frame [3].
In (7), RT Tinitialf rame is a variable that depends on the
interface delays of the switching devices, length of the wire,
and bandwidth of the network. The dynamic RT Tinitialf rame
calculation lift the restriction of same technology employed by
both the switches as each switch can contribute its own delay
values when the initial frame is processed in real-time. The
DV is the total number of bits required in the buffer before
initiating a pause frame. Therefore, the threshold for pause
instigation should be calculated as given in (8).
T hreshold = Buf f erSpace(P erLinkInterf ace) − DV. (8)
Pseudocode for calculating RT Tinitialf rame on each port
of the switch is expressed in Algorithm 1.
IV. E XPERIMENTAL R ESULTS
In this section, performance of the proposed technique
is analyzed. For simulation environment, NS2 was used to
investigate the effects of proposed technique on buffer space
Authorized licensed use limited to: Edith Cowan University. Downloaded on April 17,2020 at 11:47:56 UTC from IEEE Xplore. Restrictions apply.
IEEE COMMUNICATIONS LETTERS, VOL. 24, NO. 4, APRIL 2020
Algorithm 1 Compute RTT of the Initial Frame Output: RTT
1: initialize: frame = 64 byte initial frame (min MTU);
2: initialize: time = current system time;
3: initialize: t = 0;
4: initialize: q = time required to inject 512 bits on link;
5: send frame to upstream link;
6: while response frame not received do
7: wait q (one quanta);
8: end while
9: t = current system time - time;
10: RTT = t / q;
11: return RTT ;
reservation, end-to-end delay, throughput, and buffer utiliza-
tion. The results are also compared with the previous standard
that also employed the same technique using non-selective
CN technique. The simulated environment consists of 10Gbps
links and a primary link is created as a bottleneck to create
the back-pressure in order to trigger the congestion notification
mechanism. The network was purposefully overwhelmed with
traffic to depict the peak load scenario where this technique
comes into play in order to achieve lossless operation.
The proposed technique utilized 44 percent less buffer
space in order to achieve lossless operation as compared to
the technique proposed in IEEE 802.1Qbb. On a single port
our technique reserved 81KB for each CoS as compared to
(5) IEEE82.1Qbb that reserves 144KB for each CoS. This sums up
to <4MB space reservation on a 48 port switch in comparison
to IEEE 802.1Qbb that reserve approx. 7MB of buffer space as
(6) shown in Fig. 2(a). The results for IEEE 802.3x are not shown
because it does not reserve buffer space as it stops traffic for
all flows with a fixed buffer value that can be set as required.
Fig. 2(b) shows the end-to-end delay comparison of all three
techniques for 8 different CoS as IEEE 802.1Qbb supports
up to only 8 different CoS. It can be seen that proposed
technique has less end-to-end delay as compared to other.
Moreover, end-to-end delay for CoS 4 and above is much more
in proposed technique and IEEE 802.1Qbb because these flows
are low priority flows. In order to compensate the high priority
flows, the low priority flows suffer and result in higher end-
to-end delay. IEEE 802.3x does not support multiple flows;
therefore, all the flows have uniform end-to-end delay for all
flows including high priority flows. This results in 33 percent
higher delays as compared to proposed technique and IEEE
802.1Qbb.
Fig. 2(c) shows the average throughput of the bottleneck
link. The proposed technique and IEEE 802.1Qbb both out-
performed IEEE 802.3x because IEEE 802.3x stops all the
flows at the same time and when the flows are reinitiated,
it encounters delay that results in low utilization of the link
during delay. The proposed technique achieved 1 percent
higher throughput because it reserves less buffer space as
compared to IEEE 802.1Qbb resulting in more space available
at the receiver to accommodate more data instead of initiating
the pause thus results in better throughput.
The results in Fig. 2(d) show that the proposed technique
utilizes the buffer much efficiently than the others. This is
KHURSHID et al.: DYNAMIC THRESHOLD CALCULATION FOR CN IN IEEE 802.1Qbb
Fig. 2. Comparison of result parameters.
due to the fact that proposed technique does not reserve extra
space for implementation of pause operation. Here, extra space
refers to the space reserved by IEEE 802.1Qbb on the basis
of worst-case delays while the proposed technique reserve
space based on actual delay thus the difference between space
reserved by IEEE 802.1Qbb and the proposed technique is
referred as extra space. The proposed technique and 802.1Qbb
make use of buffer more proficiently than IEEE 802.3x
because buffer space is divided into eight queues for each flow.
Consequently, when the PFC mechanism is triggered, only
relevant data is paused, meanwhile the other flows transmit
normally.
V. C ONCLUSION
In this study, we proposed a novel technique to dynam-
ically calculate the threshold for congestion notification for
IEEE 802.1Qbb standard. Current implementation of IEEE
802.1Qbb employ the congestion notification mechanism
based on worst case delays. To resolve the issue, an algorithm
is devised to calculate the threshold dynamically in order to
avoid the worst-case based delay threshold. Also, the proposed
scheme was implemented by extensive simulations and was
evaluated for its performance.
ACKNOWLEDGMENT
The authors would like to thank Dr. C. Desanti for his con-
tribution towards better understanding of the IEEE802.1Qbb.
Authorized licensed use limited to: Edith Cowan University. Downloaded on April 17,2020 at 11:47:56 UTC from IEEE Xplore. Restrictions apply.
747
R EFERENCES
[1] Priority-Based Flow Control, IEEE Standard 802.1Qbb, 2011.
Accessed: May 12, 2019. [Online]. Available: http://www.ieee802.org/1/
pages/802.1bb.html
[2] W. Khurshid, M. L. M. Kiah, I. A. Khan, R. Salleh, A. T. Chronopoulos,
and S. A. Madani, “Comparative study of congestion notification tech-
niques for hop-by-hop-based flow control in data centre Ethernet,” IET
Netw., vol. 7, no. 4, pp. 248–257, Jul. 2018.
[3] IEEE Draft Standard for Local and Metropolitan Area Networks—
Virtual Bridged Local Area Networks—Amendment: Priority-Based
Flow Control, Standard IEEE P802.1Qbb/D2.3, (DRAFT Amendment
to IEEE Std 802.1Q -2005), Sep. 2010, pp. 1–40. [Online]. Available:
https://ieeexplore.ieee.org/document/5570062
[4] T. Issariyakul and E. Hossain, “Introduction to network simulator 2
(NS2),” in Introduction to Network Simulator NS2. Boston, MA, USA:
Springer, 2012, pp. 21–40.
[5] IEEE Standards for Local and Metropolitan Area Networks: Sup-
plements to Carrier Sense Multiple Access With Collision Detec-
tion (CSMA/CD) Access Method and Physical Layer Specifications—
Specification for 802.3 Full Duplex Operation and Physical Layer
Specification for 100 Mb/s Operation on Two Pairs of Category 3
Or Better Balanced Twisted Pair Cable (100BASE-T2), IEEE Standard
802.3-1997, 1997, pp. 1–324.
[6] IEEE Standard for Local and Metropolitan Area Networks–Media
Access Control (MAC) Bridges and Virtual Bridged Local Area
Networks–Amendment 17: Priority-based Flow Control, IEEE Stan-
dard 802.1Qbb-2011 (Amendment to IEEE Standard 802.1Q-2011
as Amended by IEEE Standard 802.1Qbe-2011 and IEEE Standard
802.1Qbc-2011, 2011, pp. 1–40.
[7] N. Aslam, K. Xia, A. Ali, and S. Ullah, “Adaptive TCP-ICCW conges-
tion control mechanism for QoS in renewable wireless sensor networks,”
IEEE Sensors Lett., vol. 1, no. 6, pp. 1–4, Dec. 2017.
[8] IEEE Standard for Local and Metropolitan Area Networks—Virtual
Bridged Local Area Networks Amendment 13: Congestion Notification,
IEEE Standard 802.1Qau-2010 (Amendment to IEEE Standard 802.1Q-
2005), 2010, pp. 1–135.