You are on page 1of 58

27/09/2011

Troubleshooting Methods

At the end of this lesson we will be able to


Given a scenario, implement the following
network troubleshooting methodology

Network+2009 Objective 4.6

27/09/2011

What we will cover


Troubleshooting Methodology
Quick Steps

Troubleshooting Methodology

27/09/2011

Troubleshooting Overview
Proceed logically and methodically
Follow recommended steps
Use experience when necessary
Logical approach avoids wasteful, timeconsuming efforts

Troubleshooting Methodology Steps


1.
2.
3.
4.
5.
6.
7.
8.
9.

Information gathering
Identify the affected areas of the network
Determine if anything has changed
Establish the most probable cause
Determine if escalation is necessary
Create an action plan
Implement and test the solution
Identify the results and effects of the solution
Document the solution and the entire process

27/09/2011

Information gatheringidentify
symptoms and problems

You are like a detective

Use proper
communications
technique

Ask questions
Answers help identify
symptoms

Information gatheringidentify
symptoms and problems
Dont jump to conclusions
about symptoms
Pay attention
Users, system and
network behaviors, and
error messages
Treat each symptom
uniquely

27/09/2011

For on the job and troubleshooting


success
Communications skills is as important as
technical knowledge

Troubleshooting Methodology Steps


1.
2.
3.
4.
5.
6.
7.
8.
9.

Information gathering
Identify the affected areas of the network
Determine if anything has changed
Establish the most probable cause
Determine if escalation is necessary
Create an action plan
Implement and test the solution
Identify the results and effects of the solution
Document the solution and the entire process

27/09/2011

Identify the affected areas of the


network

Troubleshooting Methodology Steps


1.
2.
3.
4.
5.
6.
7.
8.
9.

Information gathering
Identify the affected areas of the network
Determine if anything has changed
Establish the most probable cause
Determine if escalation is necessary
Create an action plan
Implement and test the solution
Identify the results and effects of the solution
Document the solution and the entire process

27/09/2011

Determine if anything has changed

Troubleshooting Methodology Steps


1.
2.
3.
4.
5.
6.
7.
8.
9.

Information gathering
Identify the affected areas of the network
Determine if anything has changed
Establish the most probable cause
Determine if escalation is necessary
Create an action plan
Implement and test the solution
Identify the results and effects of the solution
Document the solution and the entire process

27/09/2011

Establish the most probable cause


Verify user is performing network
tasks correctly
Try to re-create the problem
Ask user precisely what was done
before error
Be diplomatic, NEVER insult the user,
or accuse him/her of causing the
problem

Establish the Most Probable Cause


(contd.)
Verify Physical layer connectivity
Many network problems occur at Physical layer
Types of Physical layer problems
Segment, network lengths exceed standards
Noise
Improper terminations, faulty connectors, loose
connectors
Damaged cables
Faulty NICs

Software errors may look like physical connectivity


problem

27/09/2011

Establish the Most Probable Cause


(contd.)
Verify Physical layer connectivity (contd.)
Diagnosing Physical layer Problems
Ask questions
Verify connections between devices
Verify soundness of connection hardware

Swapping equipment: tests theories


Exchange suspect component for known-good one
Change patch cable

Establish the Most Probable Cause


(contd.)
Verifying physical
connectivity

27/09/2011

Establish the Most Probable Cause


(contd.)
Verify logical connectivity
Verify correct VLAN assignment
Verify correct network configuration
Software-based causes

Resource conflicts with NICs configuration


Improperly configured NIC
Improperly installed, configured client software
Improperly installed, configured network protocols,
services

Troubleshooting Methodology Steps


1.
2.
3.
4.
5.
6.
7.
8.
9.

Information gathering
Identify the affected areas of the network
Determine if anything has changed
Establish the most probable cause
Determine if escalation is necessary
Create an action plan
Implement and test the solution
Identify the results and effects of the solution
Document the solution and the entire process

10

27/09/2011

Determine if escalation is necessary


Know when and how to escalate
Follow given procedure

Help desk support hierarchy


First-level support
Help desk analysts
Proficient in basic workstation, network
troubleshooting

Second-level support
Network specialist

Third-level support personnel


Help desk coordinator, Vendor

Troubleshooting Methodology Steps


1.
2.
3.
4.
5.
6.
7.
8.
9.

Information gathering
Identify the affected areas of the network
Determine if anything has changed
Establish the most probable cause
Determine if escalation is necessary
Create an action plan
Implement and test the solution
Identify the results and effects of the solution
Document the solution and the entire process

11

27/09/2011

Create an action plan and solution


identifying potential effects
Be aware of security implications

Consider how solution affects users or network functionality

Scalability
Temporary fix organization will outgrow
Allow for future network additions, enhancements
Cost Weigh options carefully

Create an Action Plan and Solution


Including Potential Effects (contd.)
Use vendor information
Manufacturer documentation
Free online troubleshooting information
Searchable databases
Sophisticated web interfaces for troubleshooting their
equipment
Vendors technical phone support
Consult with others, within, outside your organization

12

27/09/2011

Troubleshooting Methodology Steps


1.
2.
3.
4.
5.
6.
7.
8.
9.

Information gathering
Identify the affected areas of the network
Determine if anything has changed
Establish the most probable cause
Determine if escalation is necessary
Create an action plan
Implement and test the solution
Identify the results and effects of the solution
Document the solution and the entire process

Implement and test the solution


Apply solution only after researching
itss effects

Use methodical and logical approach

Roll out solution in stages

Verify problem solved properly

13

27/09/2011

Troubleshooting Methodology Steps


1.
2.
3.
4.
5.
6.
7.
8.
9.

Information gathering
Identify the affected areas of the network
Determine if anything has changed
Establish the most probable cause
Determine if escalation is necessary
Create an action plan
Implement and test the solution
Identify the results and effects of the solution
Document the solution and the entire process

Identify the results and effects of the


solution
After testing solution implementation:Determine how and why solution was successful
Ensure solution created no unintended, negative
consequences

14

27/09/2011

Troubleshooting Methodology Steps


1.
2.
3.
4.
5.
6.
7.
8.
9.

Information gathering
Identify the affected areas of the network
Determine if anything has changed
Establish the most probable cause
Determine if escalation is necessary
Create an action plan
Implement and test the solution
Identify the results and effects of the solution
Document the solution and the entire process

Document the solution and the entire


process
Always record:
Problem symptoms and cause (or causes)
Solution

Use centrally located database


Accessible to all networking personnel
Alerts colleagues about problem, solution, and
any network changes made

Follow-up with user

15

27/09/2011

Help to Prevent Future Problems

QUICK STEPS

16

27/09/2011

Duh Stuff
Is the user login in properly
Do they have the correct permissions
Operator error - are they doing the tasks
correctly
Network cables plugged in, Double Duh
Receiving Power, Triple Duh

Is Hardware or Software Causing the


Problem?
Hardware

Software

Failure

Configuration setting

Device Driver

File corrupted

Hardware settings

Patch required
Malware

17

27/09/2011

Is It a Workstation or a Server
Problem?
A Server problem will often affect the clients
Make sure Server Services are working first

Determine which Segments of the


Network are affected
Cabling
Switches
Routers
Virtual LANs (VLANs)
Access Control List or Firewall Policies
Miss-configured IP Address, Subnet Mask, Default
Gateway

18

27/09/2011

Review
1.
2.
3.
4.
5.
6.
7.
8.
9.

Information gathering
Identify the affected areas of the network
Determine if anything has changed
Establish the most probable cause
Determine if escalation is necessary
Create an action plan
Implement and test the solution
Identify the results and effects of the solution
Document the solution and the entire process

Review
1.
2.
3.
4.
5.
6.
7.
8.
9.

Information gathering
Identify the affected areas of the network
Determine if anything has changed
Establish the most probable cause
Determine if escalation is necessary
Create an action plan
Implement and test the solution
Identify the results and effects of the solution
Document the solution and the entire process

19

27/09/2011

Troubleshooting Connectivity

At the end of this lesson we will be able to


Given a scenario, troubleshoot common
connectivity issues and select an appropriate
solution

Network+2009 Objective 4.7

20

27/09/2011

What we will cover


Physical issues
Logical issues
Issues that should be identified but escalated
Wireless Issues

PHYSICAL ISSUES:

21

27/09/2011

Cross Talk is signal leak from one


wire to another

Near End Cross Talk (NEXT) and PowerSum Near-End Cross Talk (PSNEXT)
Cable Certfier

22

27/09/2011

Attenuation
Signal

Signal
Strength

Maximum Segment Length

When too many devices are on shared


media, collisions becomes a problem

23

27/09/2011

Shorts occur when two conductors


touch

Signal

Wires

Open occur when one or both


conductors lose continuity
Break
A

24

27/09/2011

Impedance Mismatch (Echo)


Transmitted Signal
Resultant Signal

Reflected Signal

Interference

25

27/09/2011

LOGICAL ISSUES

Port Speed and Duplex Mismatch


Configuration NIC
(Speed/Duplex)

Configuration Switch
(Speed/Duplex)

Resulting NIC
Speed/Duplex

Resulting Catalyst
Speed/Duplex

AUTO

AUTO

1000 Mbps, Full-duplex

1000 Mbps, Full-duplex

Assuming maximum capability of Catalyst switch, and NIC


is 1000 Mbps, full-duplex.

1000 Mbps, Full-duplex

AUTO

1000 Mbps, Full-duplex

1000 Mbps, Full-duplex

Link is established, but the switch does not see any


autonegotiation information from NIC. Since Catalyst
switches support only full-duplex operation with 1000
Mbps, they default to full-duplex, and this happens only
when operating at 1000 Mbps.

Comments

AUTO

1000 Mbps, Full-duplex

1000 Mbps, Full-duplex

1000 Mbps, Full-duplex

Assuming maximum capability of NIC is 1000 Mbps, fullduplex.

1000 Mbps, Full-duplex

1000 Mbps, Full-duplex

1000 Mbps, Full-duplex

1000 Mbps, Full-duplex

Correct Manual Configuration

100 Mbps, Full-duplex

1000 Mbps, Full-duplex

No Link

No Link

100 Mbps, Full-duplex

AUTO

100 Mbps, Full-duplex

100 Mbps, Half-duplex

Duplex Mismatch 1

AUTO

100 Mbps, Full-duplex

100 Mbps, Half-duplex

100 Mbps, Full-duplex

Duplex Mismatch 1

100 Mbps, Full-duplex

100 Mbps, Full-duplex

100 Mbps, Full-duplex

100 Mbps, Full-duplex

Correct Manual Configuration2

100 Mbps, Half-duplex

AUTO

100 Mbps, Half-duplex

100 Mbps, Half-duplex

Link is established, but switch does not see any


autonegotiation information from NIC and defaults to halfduplex when operating at 10/100 Mbps.

10 Mbps, Half-duplex

AUTO

10 Mbps, Half-duplex

10 Mbps, Half-duplex

Link is established, but switch does not see Fast Link Pulse
(FLP) and defaults to 10 Mbps half-duplex.

10 Mbps, Half-duplex

100 Mbps, Half-duplex

No Link

No Link

AUTO

100 Mbps, Half-duplex

100 Mbps, Half-duplex

100 Mbps, Half-duplex

Link is established, but NIC does not see any


autonegotiation information and defaults to 100 Mbps,
half-duplex.

AUTO

10 Mbps, Half-duplex

10 Mbps, Half-duplex

10 Mbps, Half-duplex

Link is established, but NIC does not see FLP and defaults
to 10 Mbps, half-duplex.

Neither side establishes link, due to speed mismatch

Neither side establishes link, due to speed mismatch.

26

27/09/2011

Network Configuration Errors


Plugged into Incorrect VLAN port
Incorrect IP Address
Incorrect Subnet Mask
Wrong Default Gateway address
Wrong DNS server address

Issues that should be identified but


escalated:
Switching Loop
Broadcast Storms

Routing Loop

Route Problems
Proxy Arp

27

27/09/2011

Wireless Issues:

Interference (bleed, environmental factors)


Incorrect Encryption
Incorrect Channel
Incorrect Frequency
SSID/ESSID mismatch
Standard mismatch (802.11 a/b/g/n)
Distance
Bounce
Incorrect Antenna Placement

Review

Physical issues
Logical issues
Issues that should be identified but escalated
Wireless Issues

28

27/09/2011

Hardware Tools

At the end of this lesson we will be able to


Given a scenario, utilize the appropriate hardware
tools

Network+2009 Objective 5.3

29

27/09/2011

What we will cover


Cable testers
Protocol analyzer
Certifiers
TDR
OTDR
Multimeter
Toner probe
Butt set
Punch down tool
Cable stripper
Snips
Voltage event recorder
Temperature monitor

Cable testers

http://commons.wikimedia.org/wiki/File:Cable-tester-0a.jpg

30

27/09/2011

Protocol analyzer

Certification testersor Certifiers

31

27/09/2011

Time-domain Reflectometer (TDR)


Finds and describes
faults in cables
Works like Radar
Finds length of cable

Optical Time-domain Reflectometer


(OTDR)

TDR of Optical Fibre

32

27/09/2011

Multimeter Measures Voltage,


Resistance, Current

Toner probe

Toner Probe demo


http://www.youtube.com/watch?v=jP0AtN9hTP4

33

27/09/2011

Butt set

Punch down tool

34

27/09/2011

Cable Crimper

Cable Stripper

35

27/09/2011

Snips

Voltage event recorder monitor the


quality of power supplied
Monitor and Records
Sags
Spikes
Surges
or other power
variations

36

27/09/2011

Temperature and Humidity Monitor

Review
Cable testers
Protocol analyzer
Certifiers
TDR
OTDR
Multimeter
Toner probe
Butt set
Punch down tool

Cable stripper
Snips
Voltage event
recorder
Temperature
monitor

37

27/09/2011

Command Line Tools

At the end of this lesson we will be able to


Given a scenario, select the appropriate command
line interface tool and interpret the output to
verify functionality

Network+2009 Objective 5.1

38

27/09/2011

What we will cover

Traceroute
Ipconfig
Ifconfig
Ping
Arp ping
Arp
Nslookup
Hostname
Dig
Mtr
Route
Nbtstat
Netstat

Getting Help
On Windows:
Type the command followed by /? or a -?

On Unix or Linux
Type the command followed by --help
Type man followed by the command

39

27/09/2011

tracert (windows)

or traceroute (linux) print the


route packets take to network host

traceroute allspice.lcs.mit.edu.
traceroute to allspice.lcs.mit.edu (18.26.0.115), 30 hops max
1 helios.ee.lbl.gov (128.3.112.1) 0 ms 0 ms 0 ms
2 lilac-dmc.Berkeley.EDU (128.32.216.1) 19 ms 19 ms 19 ms
3 ccn-nerif22.Berkeley.EDU (128.32.168.22) 20 ms 39 ms 39 ms
4 128.32.197.4 (128.32.197.4) 59 ms 119 ms 39 ms
5 131.119.2.5 (131.119.2.5) 59 ms 59 ms 39 ms
6 129.140.70.13 (129.140.70.13) 80 ms 79 ms 99 ms
7 129.140.71.6 (129.140.71.6) 139 ms 139 ms 159 ms
8 129.140.81.7 (129.140.81.7) 199 ms 180 ms 300 ms
9 129.140.72.17 (129.140.72.17) 300 ms 239 ms 239 ms
10 * * *
11 128.121.54.72 (128.121.54.72) 259 ms 499 ms 279 ms
12 * * *
13 * * *
14 ALLSPICE.LCS.MIT.EDU (18.26.0.115) 339 ms 279 ms 279 ms

Ipconfig and ifconfig

Both provide the IP address and network


configuration information for the computer
you are using
ipconfig Windows
ifconfig Unix/Linux

40

27/09/2011

ipconfig

Ifconfig

41

27/09/2011

Ping Packet Internet Groper


Written by Mike Muuss in 1983
Ping is a little thousand-line hack that I wrote
in an evening
I named it after the sound that a sonar
makes
Use ICMP echo request and echo reply
One of the most useful troubleshooting tools

Successful ping

42

27/09/2011

Failed ping

Arp ping

arping utility on Unix/Linux/Mac OS X


Used to send an ARP request to a system on
the network.
IPv4 devices must respond to an ARP request,
unlike a normal ping
Not forwarded by routers

43

27/09/2011

arping command options


/sbin/arping
Usage: arping [-fqbDUAV] [-c count] [-w timeout] [-I device] [-s source] destination
-f : quit on first reply
-q : be quiet
-b : keep broadcasting, don't go unicast
-D : duplicate address detection mode
-U : Unsolicited ARP mode, update your neighbours
-A : ARP answer mode, update your neighbours
-V : print version and exit
-c count : how many packets to send
-w timeout : how long to wait for a reply
-I device : which ethernet device to use (eth0)
-s source : source ip address
destination : ask for what ip address

arping command output


/sbin/arping www.learnthat.com
ARPING 64.34.165.234 from 72.51.34.96 eth0
Unicast reply from 64.34.165.234 [00:00:0C:07:AC:CD] 1.406ms
Unicast reply from 64.34.165.234 [00:00:0C:07:AC:CD] 2.402ms
Unicast reply from 64.34.165.234 [00:00:0C:07:AC:CD] 1.887ms
Unicast reply from 64.34.165.234 [00:00:0C:07:AC:CD] 5.152ms
Unicast reply from 64.34.165.234 [00:00:0C:07:AC:CD] 76.378ms
Unicast reply from 64.34.165.234 [00:00:0C:07:AC:CD] 2.054ms
Unicast reply from 64.34.165.234 [00:00:0C:07:AC:CD] 2.322ms
Unicast reply from 64.34.165.234 [00:00:0C:07:AC:CD] 48.561ms
Unicast reply from 64.34.165.234 [00:00:0C:07:AC:CD] 109.788ms
Sent 16 probes (1 broadcast(s)) Received 9 response(s)

44

27/09/2011

arp utility
To view and modify a hosts ARP cache/table
ARP cache/table contains mappings between
IP and MAC Addresses

Nslookup Name

Server lookup

C:\>nslookup Default Server: ns1.example.net


Address: 218.10.244.45 > ? Commands: (identifiers are
shown in uppercase, [] means optional)
NAME
- print info about the host/domain
NAME using default server
NAME1 NAME2 - as above, but use NAME2 as server
help or ?
- print info on common commands set
OPTION - set an option all
- print options, current
server and host [no]debug
- print debugging
information [no]d2
- print exhaustive debugging
information
--- Output Cut ---

45

27/09/2011

Nslookup Name

Server lookup (cont.)

> learnthat.com Server: ns1.example.net


Address: 218.10.244.45
Non-authoritative answer:
Name: learnthat.com Address: 72.51.34.96

Hostname command

On Windows simply displays hostname


On Unix/Linux can display as well as set the
hostname

46

27/09/2011

dig provides detailed DNS information


provides technical DNS information f on Unixlike systems
Dig utility has very many options

dig command output


dig thatnetwork.com
;<<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 <<>> thatnetwork.com
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4448
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0,
ADDITIONAL: 0
;; QUESTION SECTION:
;thatnetwork.com.
IN
A
;; ANSWER SECTION:
thatnetwork.com.
3600
IN
A
64.34.165.234
;; Query time: 169 msec
;; SERVER: 216.187.125.131#53(216.187.125.131)
;; WHEN: Tue Nov 30 14:13:23 2010
;; MSG SIZE rcvd: 49

47

27/09/2011

my traceroute (mtr) command

Linux command
Combines ping and traceroute
Similar to windows pathping
provides details of the path between two
hosts (similar to the traceroute command)
Plus, additional statistics for each node in the
path based on samples taken over a time
period (similar to the ping command).

My traceroute (mtr) command output


My traceroute [v0.71]
example.lan Sun Mar 25 00:07:50 2007
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Hostname %Loss Rcv Snt Last Best Avg Worst
1. example.lan 0% 11 11 1 1 1 2
2. ae-31-51.ebr1.Chicago1.Level3.n 19% 9 11 3 1 7 14
3. ae-1.ebr2.Chicago1.Level3.net 0% 11 11 7 1 7 14
4. ae-2.ebr2.Washington1.Level3.ne 19% 9 11 19 18 23 31
5. ae-1.ebr1.Washington1.Level3.ne 28% 8 11 22 18 24 30
6. ge-3-0-0-53.gar1.Washington1.Le 0% 11 11 18 18 20 36
7. 63.210.29.230 0% 10 10 19 19 19 19
8. t-3-1.bas1.re2.yahoo.com 0% 10 10 19 18 32 106
9. p25.www.re2.yahoo.com 0% 10 10 19 18 19 19

48

27/09/2011

Route
Displays and Sets Routing information
Windows and Unix / Linux different syntax

Nbtstat utility show NetBIOS over TCP/IP


information

The nbtstat shows information for NetBIOS


over TCP/IP
NetBIOS over TCP/IP a Windows networking
protocol
H:\>nbtstat
Useful for seeing NetBIOS connections in a
Windows networking environment.

49

27/09/2011

netstat networking utility shows


network statistics
Very useful tool
Very many options
Available on both Unix and Windows different
options
H:\>netstat /?
Displays protocol statistics and current TCP/IP
network connections.
NETSTAT [-a] [-b] [-e] [-n] [-o] [-p proto] [-r] [-s] [-v]
[interval]

Summary

Traceroute
Ipconfig
Ifconfig
Ping
Arp ping
Arp
Nslookup

Hostname
Dig
Mtr
Route
Nbtstat
Netstat

50

27/09/2011

http://learnthat.com/2010/12/comptianetwork-2009-domain-5-network-tools/

Network Scanners

51

27/09/2011

Objectives
At the end of this lesson we will be able to
Explain the purpose of network scanners

Network+2009 Objective 5.2

What we will cover

Packet sniffers
Intrusion detection software
Intrusion prevention software
Port scanners

52

27/09/2011

Packet sniffers
Wireshark
Microsoft Network Monitor

Wireshark Screenshot

53

27/09/2011

Intrusion Detection Software (IDS)


Detect
network
attacks

Composed of:

Sensors

Console

Engine

Detect
security
events

Monitor and
control

Logs events
and generate
alerts

Some Free IDS


Snort
Untangle
Bro NIDS
Prelude Hybrid IDS
OSSEC HIDS
Flowmatrix NBAD

54

27/09/2011

Intrusion Prevention Software

Real-time
monitoring of
all network
traffic

Detects
malicious
code or
attacks

When attack
is detected
Drop the
offending
packets
Allows all other
traffic to pass

Free Intrusion Prevention Software


Snort
Untangle

55

27/09/2011

Untangle Screenshot

Port scanners
Scan open TCP or UDP ports on a target host
or network
Used for testing for network vulnerabilities
Freeware Port Scanners
Network Mapper (Nmap)
Angry IP Scanner

56

27/09/2011

Nmap 5 Screenshot

Angry IP Screenshot

57

27/09/2011

Review
Packet sniffers
Intrusion detection software
Intrusion prevention software
Port scanners

58