Latency Troubleshooting Tips

1. Using traceroute
2. Identifying where latency occurs
3. Causes of latency
4. Actions you can take to resolve latency issues

Please note: This is intended to be a brief introduction to troubleshooting latency issues

and is not intended to be a comprehensive guide.

Using traceroute

The first step in troubleshooting latency is to determine the path the traffic follows. The easiest
way to do this is to use traceroute, a tool available on most computers and routers. On UNIX
systems and Cisco routers the command is invoked by typing traceroute, while in Windows the
command is tracert (typed in a DOS window). Following the command, enter the IP address or
hostname of the destination you are trying to reach. The output of traceroute is a listing of each
router the traffic passes through on its way to the destination and the time it took to get to that
router from the originating system:

sl-gw8-joe#traceroute example-host.sprintcustomer.com
Translating "example-host.sprintcustomer.com"...domain server (10.10.10.2) [OK]

Type escape sequence to abort.

Tracing the route to example-host.sprintcustomer.com (10.151.27.3)

1 sl-bb22-joe.sprintlink.net (10.36.144.17) 1 msec 2 msec 1 msec

2 sl-bb25-bob.sprintlink.net (10.232.19.178) 16 msec 16 msec 12 msec
3 sl-bb21-sam.sprintlink.net (10.232.20.91) 54 msec 50 msec 50 msec
4 sl-bb20-sam.sprintlink.net (10.232.3.157) 50 msec 50 msec 54 msec
5 sl-bb22-ed.sprintlink.net (10.232.20.98) 52 msec 51 msec 55 msec
6 sl-gw14-ed.sprintlink.net (10.232.27.10) 51 msec 53 msec 52 msec
7 sl-customer-1.sprintlink.net (10.232.141.20) 63 msec 65 msec 61 msec
8 cust-router.sprintcustomer.com (10.151.27.10) 70 msec 68 msec 72 msec
9 example-host.sprintcustomer.com (10.151.27.3) 78 msec 75 msec 74 msec

Note: Each time listed is the total amount of time that one packet took to get from the originating
machine to that hop. It is not the amount of time from the previous hop to the following hop. The
output of different implementations of traceroute may look different than the example given
above (traceroute on a Cisco router), but they should all provide similar information.
Identifying where latency occurs

Once you know the path that your traffic follows, you can examine the traceroute output to
determine where the latency occurs. Look for a hop where the times shown increase dramatically
(usually by several hundred milliseconds). For example:

sl-gw8-joe#traceroute example-host.sprintcustomer.com
Translating "example-host.sprintcustomer.com"...domain server (10.10.10.2) [OK]

Type escape sequence to abort.

Tracing the route to example-host.sprintcustomer.com (10.151.27.3)

1 sl-bb22-joe.sprintlink.net (10.36.144.17) 1 msec 0 msec 2 msec

2 sl-bb25-bob.sprintlink.net (10.232.19.178) 15 msec 13 msec 12 msec
3 sl-bb21-sam.sprintlink.net (10.232.20.91) 53 msec 50 msec 52 msec
4 sl-bb20-sam.sprintlink.net (10.232.3.157) 55 msec 51 msec 54 msec
5 sl-bb22-ed.sprintlink.net (10.232.20.98) 54 msec 53 msec 51 msec
6 sl-gw14-ed.sprintlink.net (10.232.27.10) 52 msec 53 msec 57 msec
7 sl-customer-1.sprintlink.net (10.232.141.20) 763 msec 842 msec 801 msec
8 cust-router.sprintcustomer.com (10.151.27.10) 770 msec 867 msec 882 msec
9 example-host.sprintcustomer.com (10.151.27.3) 866 msec 901 msec 878 msec

In the example above, there is a large increase in the times listed at hop 7. Therefore, the latency
is likely occurring across the link between the routers at hop 6 and hop 7. You should also take
into account the distance between hops when determining if there is high latency in a path. For
example, the times increase somewhat between hops 2 and 3, but this may not be indicative of a
problem if the distance between the routers at those hops is hundreds or thousands of miles. You
may also see isolated high hop times, but these do not typically indicate a problem. These
isolated high hop times are due to the lower priority that routers place on ICMP packets (used by
routers to respond to traceroute packets), which occasionally causes the router's response to be
delayed. The occasional high response time in any traceroute or ping output is normal.

The latency problem may still reside elsewhere despite what the traceroute indicates. Traceroute
only gives you the forward path to the destination host, not the return path. The return path may
be completely different (asymmetric routing), and there may be latency on that path. Asymmetric
routing is not unusual on the Internet and is not in itself a problem. To see the return path, you
will need to obtain a traceroute performed on the destination host tracing the path back to you.

Note: Not every router that resolves to a hostname ending in "sprintlink.net" is owned or
maintained by Sprint. SprintLink customer routers also resolve to hostnames ending in
"sprintlink.net". These routers will typically have names of the form "sl-xxx-##.sprintlink.net"
where "xxx" is a Sprint internal code name for the customer, and "##" is some number. An
example can be seen in hop 7 above. Customer routers are not typically maintained by Sprint,
nor can Sprint upgrade any customer links or alter a customer's traffic flow or routing without an
order directly from the customer. Please see SprintLink Naming Convention for more details.
Causes of latency

Latency can be caused by several different problems, a few of which are listed here:

Link Overutilization: The most common cause of latency is overutilization of a link. When the
amount of traffic passing over an IP link increases to a high percentage of the total bandwidth of
that link, the latency across that link begins to increase. The closer the utilization gets to 100%,
the worse the latency becomes.
Router Overutilization: When routers become overutilized, (CPU utilization approaches
100%), their ability to route traffic quickly becomes impaired. This may be due to the router
attempting to handle more traffic than it was designed for, or a shortage of router resources such
as memory. Router overutilization may also be caused by malicious attacks on the router, such as
denial-of-service attacks.
Firewall Issues: Firewalls can be affected by many of the same problems as routers. These
include overutilization due to lack of resources and malicious attacks.

Actions you can take to resolve latency issues

Latency on SprintLink: If the latency appears to be on SprintLink, and you are a SprintLink
customer, please open a ticket and we will address the issue. See the SprintLink Contacts Page
for information on contacting our Service Management Center. If you are not a customer of
Sprint, but have identified the latency problem to be on the SprintLink network, please send an
email to noc@sprint.net, including all relevant information you have, and we will address the
issue. Sprint will not open trouble tickets for non-customers.

Please note that Sprint cannot affect the service or bandwidth of any Sprint customer without
explicit direction from that customer. We will not modify, alter, or affect in any way any
customer's routing or traffic flow at the request of a third party. Please see SprintLink Naming
Convention page for more details on how routers are named and identifying customer routers
versus routers on the SprintLink backbone.

Latency elsewhere: If the latency appears to be occurring on a network other than SprintLink,
including a SprintLink customer's network, we recommend you contact the administrators of the
destination site to pursue a resolution to the issue.

Sprint cannot access, modify, upgrade or otherwise affect in any way networks that are not
owned and maintained by Sprint. Sprint cannot divulge any customer information
including, but not limited to, bandwidth capacities, quantity of links or contact
information, to any outside party.