Introduction to TCP/IP over Ethernet
Sören Wellhöfer
soeren.wellhoefer@gmx.net
November 1, 2009
This document present a brief disquisition
about the fundamental concepts involved in
TCP/IP networking over Ethernet. The focus
is put on explaining broader concepts rather
than on technical details; it is attempted to
foster a wider general understanding of the
subject matter in those without prior knowl-
edge.
The text has a slight bias towards pinpoint-
ing bandwidth-weak spots in the area of em-
bedded computing with Linux.
1 Frames, Datagrams, and Packets
Most of the TCP/IP networking done today is
grounded in Ethernet. As a fundamental com-
puter networking technology it is classified to
belong to the lowest, the Physical Layer of the
OSI1 reference model.
Data is transmitted in big-endian byte or-
der2 in consecutive packets called frames; each
of them delimited by a distinctive bit pattern
called preamble.
An Ethernet frame provides a section of
space dedicated for a payload of arbitrary data;
it must be between 46 and 1500 bytes in size.
Usually this section is fully or partially used
up by IP datagrams. The Internet Protocol (IP)
sets a layer of abstraction on top of Ethernet
and defines an addressing scheme by which
it is possible for nodes on a network to com-
municate with each solely by knowing each
other’s IP address. With this IP implements the
Network Layer of the OSI reference model.
The Data Link Layer, which is an interme-
diate layer between the Physical and Network
one, is implemented by the Address Resolution
Protocol, or ARP. Through ARP it is possible
for computers on a network to find out each
1
other’s hardware or MAC (Media Access Con-
trol) address if only the higher-level IP ad-
dress is know. MAC addresses are unique to
each piece of hardware equipment; it must be
known because Ethernet frames are generally
sent to a specific MAC address.
An IP datagram, being encapsuled in an Eth-
ernet frame, in turn dedicates a specific section
of space for arbitrary data that can for instance
be used by more upper-layer protocols such
as TCP (Transmission Control Protocol) or UDP
(User Datagram Protocol).
Because Ethernet as a networking technol-
ogy and IP as a networking protocol are in-
herently unreliable, meaning packets sent are
never guaranteed to arrive, a more advanced
protocol becomes necessary to cater to the
needs of reliability.
TCP provides reliability by employing a
set of rather elaborate control and verification
mechanism, amongst those being the concept
of ports, checksums, and acknowledgements.
A simple one, for instance, is that TCP is a
stateful protocol whereas IP itself is stateless.
This simply means that TCP keeps track of
connection states between two hosts saying
whether they are connected, transmitting data
or are about to disconnect.
It must be pointed out that TCP’s responsi-
bilities only lie in point-to-point connections
between two computers; it is not at all con-
1. The OSI (Open System Interconnection) reference
model is a layered-abstraction model with which it is
attempted to characterize different computer networking
and communication technologies by associating them
with a particular layer. On the lowest layer, the Phys-
ical Layer, it categorizes close-to-hardware technologies
whereas on the highest layer it places those closer to the
end user.
2. Also sometimes seen referred to as network byte order.
cerned with addressing or packet delivery as
this is taken care of by IP and Ethernet, which
are underlying technologies.
Furthermore, TCP aims at creating the illu-
sion of having a continuous data stream be-
tween two applications on either side of the
network connection. This allows programs
running on two totally unrelated computer
systems to communicate with each other over
a channel that much resembles the interface
that pipes or Unix domain sockets provide for
applications running on the same host.
TCP implements the Transport Layer of the
OSI reference model that sits on top of those
previously mentioned and directly interacts
with the Session Layer, the first one that ex-
hibits indispensable significance for all pro-
grams settled on higher levels; those all rely
on the stream-based network connections pro-
vided by TCP.
Together IP and TCP are often said to form
the Internet Protocol Suite which is also some-
times plainly called TCP/IP; it is the founda-
tion of most of the Internet as we now know it
today.
The Ethernet payload value of 1500 bytes
mentioned earlier is often referred to as Max-
imum Transmission Unit, short MTU. As sug-
gested by the name itself, the MTU value rep-
resents the maximum amount of data that can
be transported over a network connection in
one go.
There exists another quantity named Maxi-
mum Segment Size, or MSS, which is the size of
the current MTU minus that which whatever is
occupied by the IP and TCP headers together;
this is usually 20 bytes for each, adding up to
a total of 40 bytes. So for a maximum MTU of
1500 bytes, the MSS would be 1460 bytes due
to the 40 bytes being used up already.
MSS as a notion expresses the amount of
data that is now truly available for any kind
of user data. The term “maximum” in its name
can be understood in the sense of that which is
truly useable as the payload of user data.
2
Oftentimes it is the case that for a given path
in a network along multiple hops, the MTU
is of a much smaller size than the maximum
allowable value of 1500. This is owed by the
fact that many routers, firewalls or switches
part of the path have to downsize this quan-
tity. Sometimes it is ineluctable for them to
do so because for them it represents a means
of controlling the traffic flowing through them,
lest their buffers get filled up too quickly and
it would become necessary to revert to more
drastic methods such as dropping whole pack-
ets altogether; understandably, this is even
worse.
If a packet of certain size is transmitted along
a network path and subsequently encounters
a station whose MTU is set to a smaller value
than the packet’s size, the router or firewall
will usually opt for fragmenting it up into
several pieces, that is into several IP packets.
This, as a rule, is deemed undesirable be-
cause it is quite inefficient - a whole Ether-
net frame including protocol headers and IP
datagram and that in most cases carries only
the left-overs byte chunks from a larger frag-
mented one is created as a result.
To avoid this inexpedient situation it is pos-
sible for a network controller to lower the Max-
imum Transmission Unit (MTU) of the packets
egressing its interface. Unfavourably, it can
only do so indirectly by altering the Maximum
Segment Size (MSS); the extra space taken away
by the TCP/IP headers from the datagram size
must remain unchanged.
Obviously, the efficiency in terms of useful
payload goes down when the MSS is lowered
further, as an increasingly larger percentage of
header information is used for a decreasingly
smaller percentage of actual data. This un-
doubtedly affects the overall bandwidth neg-
atively.
It is furthermore possible for computers at
the two endpoints of a path to dynamically
adapt their currently transmitted MTU de-
pending on the circumstances of the path itself;
this is called MTU path discovery and can help
to reduce fragmentation tremendously. More
on it is given in the next section.
2 TCP Flow and Congestion Control
As was said previously, the mechanisms em-
ployed by TCP are of a rather complex nature
and hard to grasp in one swift glance. Hence,
the effort is brought forth in this section to
briefly outline those portions of the TCP proto-
col which are crucial in gaining an understand-
ing of how flow control is achieved.
Flow control is a mechanism of administer-
ing the transmission rate between two con-
nected network nodes and is always done by
the receiving end of a connection; it reports
back to the sender and discloses to him how
much data it is currently able to handle. With
this information the sender is able to adjust
his output rate to such a degree that possible
sources of delay are obviated altogether, thus
smoothing the flow. Not accidentally, this
makes better use of the total bandwidth at
hand.
It was also mentioned earlier that TCP uses
many progressive algorithms for congestion
control, which disguises a related but yet fun-
damentally different concept in the discipline
of TCP networking. Congestion control is gen-
erally manged by the sender and is a way of
bypassing discommodious situations that arise
when a network is used by many demanding
hosts all at once, thus threatening to congest
the network in unfathomable ways. Once
congested, a network may have worsened re-
sponse times that can lead to many problems
such as heightened delay times with negative
effects on bandwidth as a whole.
The main emphasise in this document is,
however, placed on flow control because
in the area of embedded networking it is
rather unlikely to occur that a larger num-
ber of bandwidth-demanding embedded sys-
tems will be linked together in one all-
3
encompassing network that is endangered of
being flooded with data from many unlikely
sources.
Limiting factors that this paper is concerned
with the most are revolving around the issue
that the embedded systems might not be able
to handle extensive networking traffic when
under high loads because of the tasks running
on them besides the exclusive control of their
networking interfaces. Moreover, the user-
space applications run on embedded Linux
systems are supposed to do all kinds of actual
work with the data sent through the network,
thereby imposing additional work loads to be
coped with. The effects on responsiveness and
overall systems performance are things to be
monitored closely.
3 Acknowledgments, Transmission
Buffers, and the Sliding Window
Principle
Realiability in TCP mainly comes from the
fact that if a packet of certain size is sent and
subsequentially received by the receiving side,
the packet gets buffered first and an acknowl-
edgement packet is sent back to the sending
side, acknowledging this specific packet. Only
if the acknowledgement is received by the
sender will an acknowledgement be sent to
the sender. In this acknowledgement of the
acknowledgement, the sender acknowledges
that it has received the first acknowledgement
from the sender. Only now can the sender
move the data out from the receiving buffer
and free the space for other incoming packets.
This buffer is commonly called the receiv-
ing buffer; in Linux it has a size of about 65
kbytes. If it is full, the sender may not transmit
any data anymore because the receiver is not
capable of buffering it. Through this method,
the receiver does what is called flow control; it
tells the sender when it may send data so that
the receiver takes only this what it can take.
This is the basic idea behind the whole sub-
ject; however, there is still a little more com-
plexity behind all this which is expressed in
the notion of the receiver telling the sender
not only when to send, but also how much to
send. This concept is called the sliding window
principle.
What this basically means is that when the
the available size of the receive buffer shrinks,
the receiving side reports the amount it is now
capable of receiving to the sender by including
this number in the acknowledgment sent to the
sender for the previously received packet. This
size which the receiver reports (or advertises)
to the sender is called the advertised window
size. The sending side will never send more
data than is given as the advertised window
size. If the advertised window size is zero
the buffer is full and the scenario that was
described above occurs; no data will be sent
until the advertised window size is larger than
zero again.
The “sliding” part about the sliding window
principle is that the window of the amount of
data that the sender is willing to receive looks
as if it is sliding along. The front edge of this
window is represented by the amount of data
the that is already received an acknowledged;
the end is formed by the amount of data the
receiver is still willing to take.
All this makes the flow control more subtle
and fine-grained for the receiver to perform.
Recall that the whole idea behind all this is not
to overflow the receiving side and let it decide
how much it can take.
The large protocol overhead and the some-
times long lasting retransmission times render
TCP an ineffective protocol for uses in real-
time applications. However, for bulk and
streaming data traffic it offers advanced mech-
anisms of congestion and flow control while
guaranteeing the dilvery of segements of data
one hundredt percent; this might be efficacious
for the goal of optimizing total bandwidth use.
4 UDP
To throw another protocol into the discussion,
let us take a look at UDP (User Datagram Pro-
tocol), one of TCP’s close relatives. It was
designed to allow servers and other computers
on a network to send and receive messages
in the form of minimalistic datagrams. UDP,
unlike TCP, does not maintain an elaborate
header that would puff up the data frame size.
This would for once already eliminate the huge
data overhead created by TCP.
There is not communication prior to trans-
mitting data nor is there anything like the
ordering of packets that have arrived from a
particual host. This is why UDP is generally
said to be an unreliable protocol. By virtue of
this particular fact UDP is not suitable when it
is important to communicate information that
are to be received at the other end, not even for
smaller amounts of data.
Though not originally indented for, UDP
nowadays often finds an application in con-
texts where huge amounts of data must be
transferred and a certain loss rate is tolerable.
For instance for VoIP or video streaming a loss
of 40% total does still produce usable results.
Seen from the embedded side of computing,
UDP might be an option if there either aren’t
many sources of noise inside a network that
could defer the UDP datagram packet from
arriving, or if the application context closely
resembles that kind of bulk data transfer which
has been discribed in the previous paragraph.