MITEL

Mitel Communications
Director for the 3300 ICP

Fault Finding & Diagnostics

Workshop
Product Release 5.0
Issue 5 – 25th July 2012
 NOTICE
The information contained in this document is believed to be accurate in all respects but is not
warranted by Mitel Corporation (MITEL). The information is subject to change without notice
and should not be construed in any way as a commitment by Mitel or any of its affiliates or
subsidiaries. Mitel and its affiliates and subsidiaries assume no responsibility for any errors or
omissions in this document. Revisions of this document or new editions of it may be issued to
incorporate such changes

Inter-Tel® is a registered trademark of Inter-Tel (Delaware), Incorporated.

Mitel® is a registered trademark of Mitel Networks Corporation.
All other trademarks mentioned in this document are the property of their respective owners,
including Mitel Networks Corporation and Inter-Tel (Delaware), Incorporated. All rights
reserved.

© 2012 Mitel Networks Corporation

Personal use of this material is permitted. However, permission to reprint/republish this material
for advertising or promotional purposes or for creating new collective works for resale or
redistribution to servers or lists, or to reuse any copy righted component of this work in other
works must be obtained from Mitel Networks Corporation.
 Fault Finding & Diagnostics Workshop

Table of Contents

Workshop Introduction

The Problem Solving Process

Logical process to follow when analysing and solving problems

Fault Finding Tools

Tools available, 3300 help, logs, maintenance commands, CCS Trace, Wireshark etc

Lab Exercises
Various labs to do using the tools to analyse situations

Submit Test Results

Email the results of your tests to the Instructor

1
Fault Finding & Diagnostics Workshop

Workshop Introduction

Introduction
This workshop is designed to show engineers how to use problem solving skills and diagnostic
tools to gather information from a system/network to analyse a fault and resolve issues on MCD
for 3300 ICP and LAN.

The key to successfully fixing problems is to have good and clear information about the
problem. This is done by asking the right questions of the right people; observation and
knowing how to use the diagnostic tools available. Using the information gathered you can then
work through a logical process to bring a fault to a successful conclusion with the minimum of
disruption to a customer and put in place solutions to prevent the problem happening again.

This workshop will give you practical fault examples to work through from the more basic wiring
faults to network faults and database/programming issues.

There are practical tests during the workshop which will require you to email test results back to
your instructor for marking.

Certification
This workshop is one of the optional modules in the Communications (MCD) specialism.
Engineers must attend a minimum of 2 of these optional modules as part of the MCP
Communications (MCD) accreditation.

You must successfully complete the practical tests to be certified. The pass mark is 80%.

2
 Fault Finding & Diagnostics Workshop

The Problem Solving Process

Many inexperienced engineers will ‘dive in’ to a fault looking at the obvious causes and trying
various methods to fix it based on their limited knowledge. This often leads to mistakes,
additional faults occurring and most likely the original fault not being fixed.
The more experienced engineer will adopt a logical process of analysing the problem, gathering
information and then using this knowledge and their experience to quickly start working towards
fixing the fault to minimise disruption to the customer.
The key steps to follow to analyse a problem or fault are as follows:
1. Define the problem
2. Describe the problem
3. Establish possible causes
4. Test the most probable cause
5. Verify the true cause

Step 1 involves asking questions and gathering information to get an accurate picture of what
the problem is and equally importantly, what it isn’t. This way you can eliminate unnecessary
checks and tests.
Step 2 requires you to apply the knowledge from step one to accurately describe what the fault
or problem is and what it can’t be. Again you use the process of elimination to narrow down the
area you need to concentrate on.
Step 3 gets you to work through probable causes of the problem. With the knowledge of where
the fault is you can now start working through the likely causes having already eliminated what
it isn’t.
Step 4 is where you apply experience and diagnostic tools to test what is the most likely cause
of the fault.
Step 5 is the final step where you verify the fault by testing and if possible reproducing it and
then provide a solution to prevent the fault occurring again. This is an important step. If the fault
was caused by some third party event or interference you need to be sure it cannot happen
another time

Example Scenarios
Interactive exercise to work through two scenarios

3
Fault Finding & Diagnostics Workshop

Fault Finding Tools

To successfully work through a fault you need to know what tools are available to you and how
to interpret what they tell you.
One of the most important is experience, the ability to intuitively use knowledge previously
learned to get the most out of the tools and to understand the information you have.

Mitel On Line
Tool What it provides Who uses/interprets
them
Knowledge Base Searchable database of documents, Technical Field engineer and above
Bulletins, Known Product Issues etc
Product Release Details of the software release, new features, Field engineer and above
Notes enhancements, bug fixes and software/hardware
versions
Software Download the latest version of the software Field engineer and above
Downloads
eDocs Technical Documentation for all products Field engineer and above

3300 ICP
Tool What it provides Who uses/interprets
them
Troubleshooting Download from eDocs. Comprehensive list of Field engineer and above
Guide troubleshooting steps, techniques and guides to
help resolve problems
Maintenance Interrogative commands to get information about Field engineer and above
Commands circuits, trunks and processes in the 3300 ICP
Maintenance logs Real time record of activity in the 3300 Field engineer and above
Audit log Real time record of who has logged in to the system Field engineer and above
and from where (IP address), and what they have
done (MCD 5.0)
Software logs Real time record of software event activity in the Third line and design
3300 only
Diagnostic logs Capture of diagnostic information including logs and Third line and design
ESM only
Voice Quality Quality of voice calls including packet loss, jitter, Field engineer and above
Statistics delay etc.
Bandwidth Call admission control statistics, calls Field engineer and above
Management allowed/stopped
statistics
CCS Trace Real time details of signalling on digital, IP and SIP Field engineer and above
trunks including numbers dialled, call states etc
EDT trunk Real time diagnostics of digital trunks Field engineer and above
commands
IP Phone Using the diagnostics menu on the IP phone Field engineer and above
diagnostics
SMDR logs SMDR logs show a lot of information about the Field engineer and above
digits dialled and call status on incoming and
outgoing calls.

4
 Fault Finding & Diagnostics Workshop

Network
Tool What it provides Who uses/interprets
them
IP Phone Real time activity and diagnostics of IP phones Field engineer and above
Analyser
Wireshark Comprehensive capture of real time network traffic Field engineer and above
letting you analyse voice and data on a local area
network

Troubleshooting Guide
There is a troubleshooting guide on Mitel On Line that takes you through different fault finding
scenarios including the following:

• Initial setup and installation

• Hardware
• Software
• System Features
• Trunks
• Tools and embedded applications
• Voice networking
• LAN
• Diagnosing problems
• Using Logs

You should use the guide to help you work through a sequence of tests to help find the
problem.

CCS Trace
This handout explains in detail how CCS trace works and how to interpret the messages that
appear in the trace.
It will be explained later in this course.

EDT Trace
This handout explains in detail how to decode the messages from the MCD EDT Trace
command.

Wireshark
This very comprehensive program allows you to monitor and gather information from the Local
Area Network.
It will be explained later in this course.

Capturing Traces & Logs

When using maintenance commands or reading logs you need to know how to capture the
information in case you need to email to technical support for later evaluation. With the 3300
maintenance command response screen (CCS Trace for example) simply highlight the text
(CTRL+A) copy and paste it into an email or Notepad. For 3300 maintenance logs use the print
or export functions.
Wireshark has a capture feature which will be discussed later in the course.

5
Fault Finding & Diagnostics Workshop

Lab Exercises 1

LOGS

Lab 1 – Audit Logs and Audit Trail

In this lab you will log into the 3300, create a new user account and then use the Audit logs to
see the activity of the ‘new’ user in the 3300.

Reference
3300 help files

Step Task Expected Result/Observation 

1 Login with the default ‘system’ and create a new
root level system administrator account with your
own name and login details
2 Logout and then login again with your new user
account
3 Create a new COS with a name
4 Open the audit trail to see the original login/logout
at ‘system’ and then your login and the change you
made.

6
 Fault Finding & Diagnostics Workshop

Lab Exercises 2

DIGITAL TRUNK FAULT FINDING

In these labs you’ll use various maintenance commands to check the status and programming
of a digital link and CCS trace to analyse calls to understand the information being presented to
you.

Lab 1 – Digital Trunks EDT commands

In this lab you’ll use the EDT commands get more information about digital trunks and link
status. Please note many of the EDT commands are for Mitel design to use and interpret.

Step Task Expected Result/Observation 

1 Use EDT SHOW LINK CONFIG PRI to give you
details about the link programming.
Response:
Protocol : PRI
Logical link number: 4
ESM link number: 10
Framing mode: E1
Protocol variant: EURO_ISDN
NetworkSide: TRUE
Termination mode nt/lt: LT
e1_CRC4: TRUE
Impedance: 120 ohms
t310Timer: 40
t304Timer: 30
Resiliency: Off
2 Use the EDT command to get information about
the PRI digital link status with it established and
also unplugged.
EDT SHOW LINK STATUS PRI
Response:
PRI Logical link number: 10
Layer 1 Status: RED
Layer 2 Status: DOWN
Interface Activated: TRUE
In this example the link layer 1 (physical
connection) is down and the alarm state is RED.

7
Fault Finding & Diagnostics Workshop

3 Use EDT SHOW FRAMER CONFIG PRI for more

information.
Response:
Logical link number: 4
Esm link number: 10
Protocol : PRI.
Framer Mode = E1
Pattern = CRC
Signalling Mode = CCS
NT/LT Mode = S/W controlled -> LT mode
Line Impedance = 120
Current Link Status = RED ALARM
D Channel Status = DOWN
Remote Loopback = DISABLED - RM
Counter (0 - 50) 0
Interrupts Connected = true and Enabled
LOSS Alarm Bit Mask = 0xa2
TDM Stream =1
Client Instance = 0x9818570 Callback =
0xa859a94 - Enabled
You can download WinSCP from
The EDT TRACE command lets you view the call
the Internet free if you want to use
information (similar to CCS Trace). You can view the Logtofile option and view the
the response in real time or write a log file. file in the /db directory.

Use the EDT Trace command to start an EDT

trace in real time.
Make a call into your system from the external
phone on the PRI.
Use the handout to interpret the call

Make an outgoing call from an extension to the

external phone and again, use the handout to
interpret the call.

8
 Fault Finding & Diagnostics Workshop

Lab Exercises 3

IP AND SIP TRUNK FAULT FINDING

Lab 1 – IP Trunks
In this lab you’ll use the following IP trunk commands to test the trunks.
STATE XNET ICP X – to verify the link is established or not
RMESSAGE – to verify the link to the other PBX is established.

Step Task Expected Result/Observation 

1 Use STATE XNET ICP X to verify the IP trunks are

established. If the trunks have not been used they
should be IDLE. If a call has been made they
should be ESTABLISHED
2 Use the RMESSAGE command to test the trunks
are established

Lab 2 – SIP Trunks

In this lab you’ll use the following SIP trunk commands to test the trunks.

SIP LINK STATE PEER < PEER_NAME> - Shows the status of the link for the
specified peer (in service or not).
SIP LINK STATE ALL – as above but for ALL peers programmed
SIP LINK STATUS – how many links are established or not
SIP SHOW ACTIVE PEER <PEER_NAME> - shows active calls on that peer
SIP SHOW ACTIVE ALL – shows all active calls on the SIP link
SIP BUSY PEER < PEER_NAME>
SIP BUSY FORCE PEER < PEER_NAME> - will busy out and RTS the peer.
SIP RTS PEER < PEER_NAME>

Step Task Expected Result/Observation 

1 Make a call into your 3300 from the external phone
over the SIP trunks. Use the SIP SHOW ACTIVE
PEER <PEER_NAME> to view the active call
Hang up the call when finished
2 Use SIP LINK STATE STATE PEER <PEER
NAME> to check the link status (established)
3 Use SIP REGISTRAR CONFIG to check the
configuration of the SIP

9
Fault Finding & Diagnostics Workshop

Lab Exercises 4

CCS TRACE AND SMDR

These labs explore the CCS trace commands and use both CCS and SMDR to check numbers
dialled.
Use the CCS Trace handout to view and capture the traces and then interpret the data.

There is a practical test with lab 3

Lab 1 – PRI Trunks

This lab explores the CCS trace messages on PRI calls.

Step Task Expected Result/Observation 

1 Enable CCS TRACE continuous. Best to use this
for these labs as the results appear in real-time.
2 Make an incoming and then outgoing call on the
PRI link (answer it in both cases and hang up).
Use the CCS Trace handout to evaluate the trace
to understand the sequence of messages.
Check the originating caller CLI, the number
dialled and call completion messages.
3 Make an incoming call on the PRI to an invalid
number. Again, check the CCS trace CCM/CRM
message for an invalid call.
4 Make a call between two IP phones so they are
busy. Then, call one of them over the PRI and get
busy tone. Again, check the CCS trace CCM/CRM
messages.
Note: you will need to set Camp on timer in the
COS to blank for this to work, otherwise the call
camps on.
5 Modify the trunk attribute for your PRI trunk so it
absorbs 0 digits. BT will still send you 6 digits.
6 Make a call into your system from the external
phone. The call will fail. Again check the trace to
see why and the call failure error code.
7 Configure ARS so that any 9118xxx number
ALWAYS dials out 9118747 (your preferred
number)
8 Make an outgoing call to 9118118 and use the
CCS trace to check the outgoing number is
correctly sent 9118747.
9 Repeat step 8 but this time look at SMDR to see
the same information.

10
 Fault Finding & Diagnostics Workshop

Lab 2 – SIP Trunks

This lab explores the CCS trace messages on SIP calls.

Step Task Expected Result/Observation 

1 Make an incoming call on the SIP trunk. Check the

CCS trace to understand the messages.
2 If you want to make an outgoing call on SIP first
disconnect the PRI link. ARS will use SIP as the
second choice route.

Lab 3 – IP Trunks
This lab explores the CCS trace messages on IP Trunk calls.
There is a practical test with this lab.
Keep the traces and answers and email them to the instructor
together with the results of the later tests.

Step Task Expected Result/Observation 

1 You will need to cooperate with your colleague on

the other system for this lab to avoid confusion on
CCS trace messages appearing.
1 Make a call in each direction between your two
systems. Check the CCS trace to evaluate the
messages. Look for the supplementary messages
for text display indicating a name.
2 Make a call between two IP phones so they are
busy. Then make a call to one of them from the
other system. Check the CCS trace message for
CRM/CIM.
3 Modify the trunk attribute for your IP trunk so it
absorbs 2 digits. This will produce the wrong
number when you receive a call. Make a call from
the other system to one of your extensions. Note
you see an ‘Invalid’ message on the calling phone.
Check the CCS trace CIM/CRM messages to see
which end the error came from.
Fix the trunk attribute.
4 Modify the ARS for the IP trunks so that 1 digit is
absorbed when dialling out. This will produce the
wrong number. Dial an extension on the other
system. Again you see an Invalid message. Check
the CCS trace CIM/CRM again for which end this
error has come from.
NOTE: With IP (and DPNSS) trunks you see an
‘Invalid’ message on the phone when something is
wrong. This could be caused by the far end not
recognising the digits (step 4 above) or by your
system. You should use CCS trace to determine
which end has sent the error message (CIM/CRM)
5 Fix the ARS and make sure your trunks are
working normally

11
Fault Finding & Diagnostics Workshop

Practical Test 1
Step Task Expected Result/Observation 

1 In this test you will capture CCS traces, interpret

them and email the traces and results to your
instructor.
2 Ensure CCS Trace is running

3 Ask you instructor to make a call on your system.

Check you have the CCS traces.
4 Examine the CCS trace and answer the following
questions. Annotate the CCS trace or copy the
relevant lines showing where the answer is found.
1. What was the number the call
ORIGINATED FROM?
2. What digits were received from the BT
exchange?
3. Was the call successful or did it fail?
4. What was the extension the caller dialled?
5. How do you know the call was successful
or failed (show the trace that details this)?
6. Which party hung up the call first?
7. Which PRI channel was used for the call?
8. What was the complete route of the call?

Keep the trace and answers ready to email later.

5 Make a second call yourself from the external

phone into an extension. Answer the call and leave
it up.
6 Use the RESOURCES command to display the
call process information.
7 Hang up the call.

8 Copy and save the CCS Trace and results of the

RESOURCES command.
9 Annotate the CCS trace and RESOURCES
response or copy the relevant lines showing where
the answer is found.
1. What was the terminating extension
number?
2. Can this number be determined from the
RESOURCES RESPONSE? If so, how?
3. What PRI trunk was used for this call?
4. Can this number be determined from the
RESOURCES RESPONSE? If so, how?

Keep the trace and answers ready to email later.

12
 Fault Finding & Diagnostics Workshop

Lab Exercises – 5

IP Phone diagnostics menu

This lab explores the use of the phone diagnostics menu.

Lab 1 – Accessing the IP Phone diagnostics menu

Step Task Expected Result/Observation 

1 On a working IP phone, press and hold down the

UP and DOWN volume key then release the UP
key (keep the DOWN one pressed) and dial 234
on the keypad (CFG – Config). Release the
volume keys. The phone enters the diagnostics
menu.
2 For more information about each menu, refer to
the Troubleshooting guide.

Lab 2 – Port Mirroring for Wireshark – IP Phone LAN port

Step Task Expected Result/Observation 

1 Connect the PC or your laptop to the phone layer 2

port. Using the phone ‘debug’ menu enable port
mirroring.
Note: You must have the PC/laptop plugged in
before enabling port mirroring otherwise it won’t
work.
2 This will be used later with Wireshark to monitor
the network.

13
Fault Finding & Diagnostics Workshop

Lab Exercises – 6

Local Area Network

These labs explore the tools to analyse and record network information as an aid to fault
finding.

There is a practical test with lab 3

Lab 1 – Configuring Wireshark to capture packets

In this lab you’ll configure Wireshark to capture packets on the network. If you are comfortable
using Wireshark then go straight to Lab 2.

Step Task Expected Result/Observation 

1 Download and install the latest version of http://www.wireshark.org/

Wireshark
2 Make sure your laptop/PC is connected to the
phone you enabled port mirroring on.
3 Run Wireshark and begin a new capture.

4 Make a call between two IP phones.

5 Stop the capture. Scroll through the trace to make

sure you’ve captured UDP packets from the
phones.
6 Apply a filter to view just packets from one phone
using the phone IP address:
ip.addr==xxx.xxx.xxx.xxx
7 Clear the filter and apply another filter for the
originating phone MAC address:
eth.addr==xx.xx.xx.xx.xx.xx.
8 It is important to check the capture to make sure
you have the data you expect for the device(s) you
are monitoring.
9 Stop the capture and save it as a file called
‘MCD_capture1’

Lab 2 – Capturing packets on an IP phone call and analysing them.

In this lab you’ll configure Wireshark to capture packets on an IP to IP phone call and analyse
them.
Note: due to the differences in the menus of newer versions of Wireshark the screen shots
below may not be exactly what you see on your PC/Laptop.

Step Task Expected Result/Observation 

1 Start a new trace, make a call from the IP phone

over the PRI to the external phone. Answer it and
hang up. Stop the trace.
2 Using the screen shots below you’ll open the RTP

14
 Fault Finding & Diagnostics Workshop

stream to view statistics about it and also attempt

to decode the audio stream.
3 Select a UDP line by clicking the mouse on it.

Click on Analyze on the menu bar, Decode as…

4 Select UDP both streams and then RTP. Click OK. This will show the RTP
packets.

15
Fault Finding & Diagnostics Workshop

5 Select an RTP line and highlight it. Then click Telephony and RTP – Show All
Streams

6 You’ll see the two streams. The IP addresses in this example (IP phone to PRI)
will be of the IP phone and the E2T card (or if no E2T the RTC card). Select the IP
phone and click Analyze

Note: In the screen shot above e ‘Payload’ shows the codec being used (G711 in
this case). If you make a call between two 5330/40/60 phones they will use the
new G722 code which the current version of Wireshark cannot evaluate so the
payload field will not show the codec in use.

16
 Fault Finding & Diagnostics Workshop

7 You can view the data as text or select the Graph to display the stats in graphical
format and Player to attempt to decode the audio

Change the X and Y axis scales to display the graph clearly. You’ll notice it is color
coded.

17
Fault Finding & Diagnostics Workshop

8 If you selected the option Player in step 7 you have to decode the stream first then
play it.

You cannot decode the audio between IP phones as it is both encrypted and uses
the MINET protocol.

Lab 3 – Capturing packets on a SIP trunk and analysing them.

There is a practical test with this lab.
You will need to save the Wireshark trace ready to email to your
instructor. If you wish to run the trace first for practice then please
do so. Repeat this lab from step 4 for the test.

Capturing the trace to get the SIP messages can also be done with a maintenance command in
MCD 5. This means you do not have to configure a port on the layer 2 switch. However, it is
very important you know how to do this anyway so both methods are shown below.

In this lab you’ll configure Wireshark to capture packets on a SIP trunk call and analyse them.
The practical test is to carry out the correct steps to capture the whole trace and email it back to
the instructor.

Step Task Expected Result/Observation 

1 Before you can capture SIP traffic you need to be

connected to the network ‘outside’ the 3300. In the
classroom this means configuring a port on the HP
L2 switch to monitor. You can also mirror the ports
on the 3300 itself - see step 3 below.
If you know how to do port mirroring on the HP
switch then proceed to monitor the port connecting
18
 Fault Finding & Diagnostics Workshop

your HP L2 switch to the Router/Network socket on

the workstation. The IP address of the HP layer 2
switch is 134.199.X.100.

If you are not sure how to do this, proceed to step

2
2 Login to the HP switch with your web browser and the IP address shown above.

Select Configuration – Monitor Port and in the Monitoring Port drop down select the
port your laptop/PC is connected to and from the list below scroll to and select the port
that is connected to the Router/Network socket. Apply the changes.
3 You do not need to do this but if required you can
enable Port Mirroring on a 3300 MXe or CXi
controller.
Connect to the 3300 either on the serial port or
telnet on port 2002. Enter the command:
mipsdb_debug 51, "destination_port source_port"

Example: mipsdb_debug 51, "3 1"

Port 1 is being mirrored and monitored on port 3.
Example: mipsdb_debug 51, "3 4 5 6 7"
Ports 4 to 7 are being mirrored and monitored on
port 3

The command to disable the mirroring is:

mipsdb_debug 50

Note:
These commands do not survive a reboot. If the
CXi reboots, you will have to reinitialize the
mirroring. Also, please note that when the
mirroring is enabled, the port that is doing the
mirroring cannot be used for any other purposes.
4 Start a new Wireshark capture.
This will be used in the practical test.
If you wish to run this lab first before doing the
trace for the test then please do so. Just make
sure you follow the instructions for the test trace.
5 To use SIP unplug the PRI link.
Make a call from an extension out over the SIP
trunk to the external phone. Please make sure you

19
Fault Finding & Diagnostics Workshop

say the following: “This is the practical test for

SIP”.
Hang up the call and stop the trace.
6 The trace shows the RTP packets.

7 You can view the call flow as the 3300 and SIP
peer setup the call and also decode and play the
audio stream
8 Select an RTP line click Telephony and VoIP Calls

9 The SIP call is shown. Click to highlight it and then click Flow

20
 Fault Finding & Diagnostics Workshop

10 The sequence of events during the call are now displayed with the SIP messages.

11 To play the audio stream select Player in step 8 above.

21
Fault Finding & Diagnostics Workshop

12 To analyse the data select the RTP line, click on Telephony and RTP - show all
streams.

13 Select the Forward or Reverse stream and click Analyze

You can view the data as text or select the Graph to display the stats in graphical
format and Player to attempt to decode the audio

22
 Fault Finding & Diagnostics Workshop

Change the X and Y axis scales to display the graph clearly. You’ll notice it is
colour coded.

Practical Test 2
Step Task Expected Result/Observation 

1 Save the trace in the previous lab on SIP as

“MCD_SIP”
2 1. From the packet analysis, the delta is the time
between packets captured. What is the highest
delta value and on which packet did it occur on
the Forward stream?
2. What is the IP address of the SIP peer?

23
Fault Finding & Diagnostics Workshop

Lab 4 – Capturing packets on a SIP trunk using the MCD

command.
The maintenance command results in a pcap file that you can download from
the MCD and open in Wireshark

The maintenance command to run the capture is:

SIP TCPDUMP ON
response
SIP TCPDUMP Started...

SIP TCPDUMP OFF

SIP TCPDUMP Stopped

The resulting pcap file is stored on the MCD hard drive in the /vmail folder as WS_xxx.pcap.
You will need an ftp program (such as WinSCP) to access the folder and retrieve the file which
will then open in Wireshark.

Step Task Expected Result/Observation 

1 Enter the command SIP TCPDUMP ON to start the

capture.

Make a call from the external phone to an

extension over SIP, answer it and hang up.

Enter the command SIP TCPDUMP OFF to stop

the capture.

Using WinSCP (or similar program) connect to the

MCD and navigate to the VMAIL directory and
copy the pcap file.

Open the PCAP file in Wireshark and using the

information in Lab 3 on page 18 examine the trace.

Lab 4 – Capturing packets from an IP phone during boot up.

In this lab you’ll configure Wireshark to capture packets on the network while an IP phone boots
up to see the sequence of events.

Step Task Expected Result/Observation 

1 Enable port mirroring on the layer 2 switch the

phone is plugged into. Run a new Wireshark trace
and power up the phone. In the subsequent trace
set the filter to ‘bootp’ to show the boot sequence
only. The example below is a single VLAN with the
3300 as the DHCP server.
2 Selecting the first line (frame 24 in the example) shows the initial DHCP discover
message from the IP phone

24
 Fault Finding & Diagnostics Workshop

3 Open ‘Ethernet II’ to show the MAC address of the IP phone which is sending out
a Broadcast message.
‘Open Bootstrap Protocol’ which shows the MAC address but no IP address
information at this stage.
4 Also in the ‘Bootstrap Protocol’ you’ll see an Option which identifies the ‘client’ as
an IP phone.

5 Now select the next frame, 25 (DHCP offer) and open the ‘Bootstrap Protocol’.
Under ‘Bootp flags’ you will see the IP address being offered, 134.199.12.15 and
the options DHCP is offering, Router = 134.199.12.254 (t=3), DNS =
10.129.21.100 (t=6) and Vendor Specific Information (t=125). If you select this last
one and look at the expanded view at the bottom of the screen you’ll see the
details of Option 125 being offered to the phone.

25
Fault Finding & Diagnostics Workshop

6 The next frame, 26 is the client (IP phone) accepting the offer and the final frame,
27 is the DHCP server acknowledging the phone is now using that IP address.
7 In this second example the network has a Windows server DHCP with option 43
programmed. This also sets the VLAN ID as 6 and L2 Priority as 6. The 3300 is
also running DHCP but is connected on the L2 switch on a port set to VLAN 6.
Therefore the phone when it boots up will by default send out untagged frames
which the L2 switch will put on VLAN1 so the phone cannot see the 3300 DHCP
server.

8 The phone sends out the initial DHCP discover on VLAN 1 and the Windows
server (134.199.12.200) responds (as shown above). The phone gets the VLAN
information and releases the IP addresses sending out a second DHCP Discover
message tagged VLAN 6 which the 3300 DHCP (134.199.12.2) responds to.
Please note: The VLAN tag information is not shown. This seems to be a quirk of
the HP L2 switch in use.

26
 Fault Finding & Diagnostics Workshop

Submit your test results

Email your instructor with the annotated CCS trace and Resource trace (or separate answers)
from Lab Exercises 4, and the SIP trace and answers from Lab Exercises 6.

Email subject: “MCD fault finding course test”

Include with your results the System name and software version.

27
Fault Finding & Diagnostics Workshop

Fault Finding Scenario – interactive exercise

Fault reported: “4 phones in our office aren’t working”.

Step 1 - Define the problem
There isn’t enough information to make a decision about the fault. Do you know this customer
and their location? If not you need to find out more. If you do then don’t make assumptions
based on previous knowledge. Make sure you have up to date information.
Questions:
1. What type of telephone system do they have? Is it a 3300 ICP or SX-2000? Which type
and what software version? Is it a single site or multi-sites? If multi, is it clustered, using
SDS? If the customer doesn’t know then you should get the information from the
maintenance reporting database for that customer.
2. What type of phones do they have? Analogue, IP, SIP or Digital (SX-2000). If they are
IP phones do the users have their PC’s/Laptops (or any other network devices) plugged
into the port on the back of them?
3. Verify how many phones are not working. Is it just the 4 in the office or are there others
in the same office, on the same floor or the building that are also reported as faulty. If
more are faulty than the original report indicate are they all the same type i.e. all IP or all
analogue? Verify how many phones ARE working. Is the fault localised to one office,
one floor or several floors.
4. Has this problem happened before? Is there a history? Was any work being carried out
in the building by maintenance people, for example cabling, electricity, air-conditioning?
Has the power been turned off in the building (or parts of the building) at all?
5. Talk to the switchboard operator (if they have one) as they are usually an amazing
source of information about the comings and goings in an office.

Step 2 - Describe the problem

From the above questions we now know this is a small business with 50 users and it is only the
the 4 IP phones in one office that are faulty. Other users elsewhere in the building seem to be
fine. It is a 3300 CXi controller. All 4 phones appear to be ‘dead’ with no power (no display).
This happened at 11am. Two of the users have their laptops plugged into the port on the
phones and they report no network connection. The other two users have desktop PC’s
plugged into sockets under their desks and they are working fine. The customer has no
knowledge of any work being done in their building.
The table below is useful to record the information about the fault to help you analyse it.
What – describes the problem
Where – the location
When – the timescales of when it happened
Extent – how widespread is the problem.
Is – lets you describes specific things the problem is.
Possible but is not – List the things it could possibly be but isn’t to eliminate them
Differences – this list things different between the IS and IS NOT columns
Changes – lists things that have changed that may account for the problem

28
 Fault Finding & Diagnostics Workshop

IS COULD BE BUT IS NOT Differences Changes

What What failed Similar systems/devices
that did not fail
Where Where did it fail Other locations that did
not fail
When When did it fail Other times when
systems/devices did not
fail
Extent What else is affected Other systems/devices
that did not fail

Running our scenario through this we get the following:

IS COULD BE BUT IS NOT Differences Changes

What Failure of 4 phones. Power failure to the Only phones Nothing has been
No power or office. affected not the changed in the office
connection. Network fault in the two PC’s plugged
office. into separate
Faulty phone. sockets.
PC plugged into phone
causing the phones to
fail.
Where Single location System wide network Only this office Nothing has been
failure or power failure changed in the office.
Not known if anything
elsewhere has been
changed.
When Suddenly during the Planned maintenance or None. Everything Not known
morning downtime on the network was working.
Extent Single office affected System wide 3300 or Only this office
network failure

Step 3 – Establish possible causes

This table lets you collate the information to test possible causes.
Potential root cause True if Probably root cause?
Faulty phones All 4 failed at the same time Unlikely
Cable fault to each phone All 4 cables failed at the same Unlikely
time

Fault on the PC’s plugged into the
back of the phones
Cable fault to the network sockets
in the office
Network fault in the office
Network fault just to the phones
All 4 phones had PC’s connected
but only two do.
PC’s and phones are connected
the same way
All network devices (all network
devices and phones) failed
The phones are cabled separately
to the PC’s
Unlikely
No, two PC’s separately
connected still work
No, two PC’s separately
connected still work
Yes if the phones and PC’s
sockets are separately cabled.

Step 4 - Test the most probable cause

From Step 3 the most likely cause of the problem is a network fault to the 4 phones which are
connected to sockets separate from the two PC’s in the office on their own sockets.

29
Fault Finding & Diagnostics Workshop

First test. Plug a PC into the phone socket and see if it has a network connection. The PC
doesn’t need power on the socket but you will see if the network is up. If it isn’t then check the
other 3 phone sockets.
Second test. Trace the building wiring back from the socket using the wiring/socket numbering
plan (if it exists). The phone sockets need to be connected to the 3300 CXi at some point and
there may be multiple patch leads, patch panels and layer 2 switches between them.
Is this example it is found that 4 patch leads have been removed from a patch panel by an IT
worker by accident. This was not planned. The person was doing some other patching and
simply made a mistake.

Step 5 - Verify the true cause

The fault is quite clear in this example. To ensure it doesn’t happen again the customer needs
to be aware how the IT staff work on the network. Are all patch cable and patch panels labelled
up correctly? Are there procedures in place for scheduled work and ad-hoc work like this?

30