Register online at www.chappellseminars.

com Trace File Analysis: TCP In-Depth

TCP Analysis In-Depth

www.chappellseminars.com

Presenter: Laura Chappell, Founder of Chappell University and Wireshark University

laura@chappellseminars.com
Follow me: www.twitter.com/LauraChappell
Read my blog linked at chappellseminars.com

You can set up a TCP/IP network – no sweat. Now things get a bit more complex as you
need to tap into the traffic and understand what is happening at the packet level.

This course focuses specifically on TCP (Transmission Control Protocol).

Copyright Chappell Seminars. All Rights Reserved. 1

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

These are the areas we will discuss in this seminar.

1. TCP functionality
2. Inside the TCP handshake
3. Key TCP options
4. Sequence/Acknowledgement Processes
5. Slow-Start Algorithm/Congestion Avoidance
6. Recovery from Packet Loss
7. The TCP Window Size
8. Graphing IO, Throughput, RTT and Time/Sequence Information

Start capturing your traffic to see how you make a TCP connection and how data is
transferred over TCP.

Copyright Chappell Seminars. All Rights Reserved. 2

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

TCP Overview
TCP offers a connection-oriented transmission that begins with a handshake between two
devices. Data is sequenced and acknowledged to ensure proper delivery. Where UDP may
be considered similar to the standard mail system, TCP would be considered similar to an
express carrier.

TCP supports windowing – the process of sending numerous data packets in sequence
without waiting for an intervening acknowledgment. The size of the window is based on
the amount of traffic the network can handle (the network congestion rate) and the
receiver’s available buffer space.

Copyright Chappell Seminars. All Rights Reserved. 3

Register at www.chappellseminars.com Trace File Analysis: TCP In-Depth

If all is going well, TCP data flows become more efficient over time. At the beginning of a
data transfer using TCP, the Slow Start Algorithm defines that for every packet
acknowledged, two more can be sent. This creates the exponential growth of the cwnd
value.

You won’t see the beauty of the slow start algorithm when transferring little files because
there won’t be enough data to allow the window to grow nicely. I use the Open Office
application as my sample “big fat file” for testing. It’s mirrored around the world so I can
see the different link latencies in my tests.

Copyright Chappell Seminars. All Rights Reserved. 4

Register at www.chappellseminars.com Trace File Analysis: TCP In-Depth

Ouch! This is where packet loss gets painful.

• 99.9% of the time, packet loss takes place at some internetworking device such as a
switch, router or firewall.

• If you see the original packet AND the retransmission, you are upstream from packet
loss – packet loss has not occurred yet.

• If you only see the retransmission, you are downstream from packet loss. You are on the
cruddy side of a packet-dropping device.

• Upstream is the direction of the sender of data. Downstream is the receiver direction.

• Filter on the TCP sequence number in the retransmission to determine if you saw the
original and the retransmission. To filter on this field, open the TCP details area, right
click on the TCP sequence number field of the retransmission packet and choose Apply
Filter > Selected.

Copyright Chappell Seminars. All Rights Reserved. 5

Register at www.chappellseminars.com Trace File Analysis: TCP In-Depth

We don’t go back into the Slow Start process after packet loss. We begin the Congestion
Avoidance process. It’s like knowing there is a ceiling above you and you’ll hit your head if
you stand up too quickly. We slowly, linearly increase the cwd knowing there must be a
point when we’ll lose packets again.

RFC 2581 indicates that:

“One formula commonly used to update cwnd during congestion avoidance is given in
equation: cwnd += SMSS*SMSS/cwnd” where SMSS is the sender maximum segment size
(commonly 1,460 bytes – you can see this value defined in the TCP options are of the first
two packets of the handshake).

Ok… so now we slowly begin to increase the cwnd… then BLAM! Packet loss again. We cut
the cwnd down to 50% of the current value and begin creeping back up again.

See how packet loss messes up a perfectly good file download?

Copyright Chappell Seminars. All Rights Reserved. 6

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

TCP Packet Structure

The TCP header is typically 20 bytes long, but supports an Options field that can extend the
header length. The Options field is seen during the handshake process.

Source Port Field

See UDP source port listing online at www.iana.org.

Destination Port Field

See UDP source port listing online at www.iana.org.

Sequence Number Field

This field contains a number that uniquely identifies the TCP segment (at the transport
layer, we refer to the data that is preceded by the TCP header as a ‘segment’). This
sequence number provides an identifier that enables TCP receivers to identify when parts
of a communication stream are missing. The sequence number increments by the number
of data bytes contained in the packet.

Each TCP device self-assigns its own sequence number. The process of incrementing this
sequence number is covered further in the “TCP Windowing” section.

Copyright Chappell Seminars. All Rights Reserved. 7

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

Acknowledgment Number Field

The Acknowledgment Number field indicates the next expected sequence number
from the other side of the communications.

Header Length/Data Offset Field

This defines the length of the TCP header. In actuality, it is defined in 4 byte
increments, so a value of 5 in this field indicates that the TCP header is 20 bytes
long. We need this field because the TCP header length can vary depending on the
TCP header options used. While the UDP option field is rarely used, the TCP option
field is almost always used during the TCP connection setup to establish the
maximum amount of data that can be placed after a TCP header.

Copyright Chappell Seminars. All Rights Reserved. 8

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

Flags Field
The following list describes the flags used in the TCP header:

CWR (Congestion Window Reduced): Set to ACK notification of congestion along a route
(optional)
ECN-Echo Explicit Congestion Notification : Set to indicate congestion along a route
(optional)

URG (Urgent): Indicates Urgent Pointer field should be examined.

ACK (Acknowledgment): Acknowledgment packet.
PSH (Push): Bypass buffering and pass data straight to upper layer.
RST (Reset): Close the connection.
SYN (Synchronize): Synchronize sequence numbers – handshake process.
FIN (Finish): Transaction finished, but don’t close connection.

These last 6 fields are important in understanding TCP communications. The following
slides give a brief interpretation of how you can use these field values in your analysis.

Copyright Chappell Seminars. All Rights Reserved. 9

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

URG: Rarely seen. This application wants you to read the data in this packet in a
special order.

ACK: Typical acknowledgment packet. If this is missing from the process, then the
data stream cannot continue to be sent. Back-to-back ACKs indicate a
packet is missing from a set (see RFC 2001).

PSH: There are two ‘TCP buffer’ areas. One TCP buffer gathers outgoing data so
the window is a decent size. The other is on the incoming side to receive
data and pass it up in an ordered fashion. The PSH flag indicates that this
TCP segment should not be held in the outgoing or incoming buffers. An
application that is very time and user driven (such as character-at-a-time
telnet) may set the PSH flag on every packet making TCP act in a ping-pong
(packet out, ACK in, packet out, ACK in, etc.) manner.

Copyright Chappell Seminars. All Rights Reserved. 10

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

RST: If an application does not send a RST when you shut it down, that means
that the application holds the connection open. The application may rely on
a TCP connection timeout to shut down the connection. If an application
encounters a fault, it may send a RST in the middle of the communication.

SYN: This host is attempting to establish a TCP connection by exchanging the

starting sequence number values. One form of a Denial of Service attack
sends back-to-back SYNs incrementing the sequence number in each
packet. Some firewalls can block SYN packets coming in from an untrusted
source to stop connections between the inside and outside world.

FIN: This indicates that a process has completed and the data stream has been
sent. The sender does not want to shut down the connection, however.
Many times the PSH flag will be set with the FIN flag.

Copyright Chappell Seminars. All Rights Reserved. 11

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

Window Field
This field indicates the size of the TCP receiver buffer in bytes. A window size of 0 (in a packet other
than a RST packet) indicates that a sender should stop transmitting – the receiver’s TCP buffer is
full.

Checksum Field
This TCP checksum is a bit strange, just like the UDP checksum. The checksum is performed on the
contents of the TCP header and data (not including data link padding) as well as a pseudo header
derived from the IP header. Refer to RFC 793 for more information.

Urgent Pointer Field (optional)

This field is only relevant if the URG pointer is set. If the URG pointer is set, the receiver must
examine this field to see where to look/read first in the packet. This is not a common function.

TCP Options Field(s) (optional)

One option you will see often is the Maximum Segment Size (MSS) option – it is used in the first two
packets of the three-way handshake process. The purpose of this option is to define what segment
size the hosts support. The hosts will use the lowest common denominator between the two MSS
values. Other TCP options can be found at www.isi.edu/in-notes/iana/assignments/tcpparameters.

Copyright Chappell Seminars. All Rights Reserved. 12

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

The TCP Startup/Handshake Process

TCP is much more interesting and complex than UDP communication. The primary
areas of interest when analyzing TCP communications are:

Startup sequence (handshake)

Sequence/acknowledgment
Window sizing

The SYN packets synchronize the sequence numbers to ensure both sides know
each other’s starting sequence numbers. This is how they will keep track of the
sequence of data exchanged between them.

Copyright Chappell Seminars. All Rights Reserved. 13

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

Filtering on TCP Traffic

The figure above shows the capture filter syntax and the Display Filter Reference for
TCP filtering.

In this course I provide examples of building and saving various TCP display filters.

Copyright Chappell Seminars. All Rights Reserved. 14

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

The TCP Sequencing/Acknowledgment Process

The sequencing/acknowledgment process guarantees that packets are ordered properly
and protects against missing segments. During the handshake process, each side of the
connection selects its own starting sequence number. They will increment this sequence
number by the amount of data included in each packet. When you are analyzing the
sequencing/acknowledgment process, keep in mind this simple equation:

Sequence Number In + Bytes of Data Received = Acknowledgment Number Out

This slide depicts how a sequenced communication may occur in simple terms/numbers
(remember, the acknowledgment number filed contains the value of the next sequence
number expected from the other side.

The Acknowledgment number field only increments when data is received. You should also
note that the example starts with Host 1 sending data and then reverses when Host 2 has
something to send. This is typical of two-way communications.

Note: During the TCP startup and teardown sequence there is a ‘phantom byte’ that causes
the sequence number and acknowledgment number fields to increment by 1 even though
no data is exchanged. Don’t let this confuse you when you are just learning the process.

Copyright Chappell Seminars. All Rights Reserved. 15

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

The TCP Windowing Process

TCP communications can send multiple packets in a row without requiring an intervening
Acknowledgment for each packet sent. This is called a ‘window.’ The size of the congestion
window (what the network supports as opposed to what the receiver supports) is
dependent upon these factors:

•The receiver’s TCP buffer space advertised

•The amount of traffic allowed on the network (network congestion/packet loss)

The window will always be the lower of the two values. Network congestion is defined as a
condition that causes packets be lost in transmission because the network itself cannot
support the data transfer rate. For example, on an Ethernet network suppose a receiver
advertises a window of 4096 bytes, but packets are lost in the transfer of multiple 1024
byte frames due to a high collision rate on a very busy network.

In this case, the actual window is not 4096 bytes –it will be about 50% of the last window
attempted when packets were lost. This process of altering the window in case of data loss
is eloquently defined in RFC 2001.

Copyright Chappell Seminars. All Rights Reserved. 16

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

The TCP Windowing Process (continued)

You may wonder why it’s called a ‘sliding’ window. If you look at the data that’s sent and
you move a window over it, the right side of the window is the data that’s been
acknowledged. The right side defines the boundary of data that can be sent based on the
receiver’s advertised window.

Data set A+B has already been sent and acknowledged. The current window has sent data
set C+D+E and the sender is waiting for an acknowledgment. The window will now move to
the right to send the next data set F+G+H. The window continues to slide to the left as
acknowledgments are received.

Copyright Chappell Seminars. All Rights Reserved. 17

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

Service refusals are never normal on a network.

The slide above shows a TCP scan underway. Notice the striping pattern.

If a host sends a TCP SYN and receives an ICMP response, that response is likely coming
from a firewall process on the host or along the path.

See the Hacked Hosts: Network Forensics course for more details on unusual TCP traffic
patterns.

Copyright Chappell Seminars. All Rights Reserved. 18

Register at www.chappellseminars.com Packet Loss and Latency: Network Hell

Here’s the fun part.

This trace shows lots of problems in it – high latency, packet loss, window zero condition… ugh.

Open this trace file and select View > Time Display Format > Seconds Since Previous Displayed
Packet. This is a great setting to use when troubleshooting slow networks. Look a the time column
values for packets 1 and 2 (the first two packets of the TCP handshake). Ugh… Over 167
milliseconds delay – pathetic. There’s a latency problem right there. It’s just a snapshot, but it’s
ugly.

Now select Analyze > Expert Info Composite. Click the Warnings tab.

Since I began my capture before the TCP connection was established, I should not see any ‘previous
segment lost’ notations unless we had actual packet loss. There are 100 of these stinkers.

Expand that notation and click on the first packet listed, packet 133. That packet will be selected in
the main Wireshark window’s summary pane.

Spend some time looking through this area of the trace – pay particular attention to the TCP SACK
options information.

Copyright Chappell Seminars. All Rights Reserved. 19

Register at www.chappellseminars.com Packet Loss and Latency: Network Hell

TCP Selective ACK

The client is downloading a file from the server. Each of the lettered squares represents one
TCP segment (which is typically 1,460 bytes). The client received segments A, C, D, E, and F.
Unfortunately, B is missing.

The client will send an acknowledgment (ACK) for A and, in that same packet, indicate that
it is expecting B next. It does this by placing the Sequence Number for B in it’s
Acknowledgment Number field.

Remember: The TCP Acknowledgment Number field is the next expected Sequence Number
from the other side of the conversation.

In that ACK, the client that uses SACK places a left-edge and right-edge value in the TCP
header options area. The left-edge would be the Sequence Number for C and the right-
edge would be the end of segment F (which is also the expected Sequence Number for
segment G, if there will be a segment G).

RFC 2018 defines a maximum of three left-edge/right-edge sets that can be defined in a
TCP options area.

Copyright Chappell Seminars. All Rights Reserved. 20

WSU03: Troubleshooting Network Performance Packet Loss and Latency: Network Hell

With Wireshark running at the client in this image, here is what the numbers refer to:

1. The time between the SYN and SYN/ACK gives us a nice snapshot of RTT and the wire
latency.
2. If there is a large gap between the SYN/ACK and the final ACK, then something is wrong
with the client’s TCP stack. Completion of the handshake should be almost immediate.
3. If the client waits before using that connection it just established, then I’d look at the
application that requires the connection.
4. If you make a request from the server and there is a delayed ACK, then we look at wire
latency issues.
5. If the server ACKs your request quickly, but then takes a while to send the requested
data, then we are looking at server processing latency.

Note: Watch out for Microsoft’s Delayed ACK usage. The server would wait up to 200 ms
before sending in an ACK. If another data packet arrived within that 200 ms time, the
server would ACK both received packets together.

Copyright Chappell Seminars. All Rights Reserved. Slide 21

WSU03: Troubleshooting Network Performance Packet Loss and Latency: Network Hell

Here’s an ugly RTT graph.

The vertical stripping may be due to packets being queued up and then released in sets.
Remember that stripes never look good on a network!!!

During the seminar I show the most common mistake made when creating an RTT graph.
This is a unidirectional graph – if you select a packet traveling in the direction of data flow
then you’ll be see a graph with plot points. If you select a packet going in the direction of
ACKs only, then your graph will be almost blank.

Notice that the Y axis is set by Wireshark based on the maximum RTT value seen in the
traffic.

Copyright Chappell Seminars. All Rights Reserved. Slide 22

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

The TCP Time-Sequence Graph depicts the individual TCP segments (“I” bars), the ACK
packets (shadowing the black I bars) and the receive window space (the hovering grey line
above the I bar).

When the receive window space and the I bars meet, the available window has dropped to
zero.

Copyright Chappell Seminars. All Rights Reserved. 23

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

Ok… Every client and server should support Window Scaling.

The “window” we are talking about here is the TCP receive window – the size of the bucket
that holds incoming data (and buffers outgoing data when needed). We NEVER want a
receiver to say “sorry… closed up shop – no receive buffer space – stop sending data.”

In this animated slide I showed you that as data arrived at the server (this is a file upload
process), the server changed the value in it’s ACK packets to reflect the shrinking receive
window buffer space. In the final ACK, the server sends a “window zero” packet which
causes the client to stop sending data. At the client, their application would simple seem to
run an interminably long time.

Copyright Chappell Seminars. All Rights Reserved. 24

Register online at www.chappellseminars.com Trace File Analysis: TCP In-Depth

Notes:

Well… thanks much for attending this course.

Please help us reach out to the IT community to let them know about these online
seminars. I will likely not be on the road much this year teaching in public so
chappellseminars.com and chappellU.com are my primary methods of training folks on
network troubleshooting, optimization and security. Spread the word.

Copyright Chappell Seminars. All Rights Reserved. 25