 ISDN debugging and tracing

o Theoretical introduction
 BRI
 PRI
 ISDN Layers
 Layer 1
 Layer 2
 Layer 3
o CAPI based EICON/Dialogic interface cards
 Gathering traces with the dmonitor tool
 Gathering traces with the Diva Trace Assistant
 Take a trace as follows:
 How to convert the Trace to text format
 Gathering traces with mlog tool
 Analysing trace files
 Is the ISDN Network to blame?
 Is the Escaux UCS server to blame?
o Zaptel based Digium PRI interface cards
 Taking the right trace
 Add more traces in /etc/asterisk/logger.conf needed (ESCAUX or RESELLER)
 Restart asterisk (ESCAUX or RESELLER)
 Put trace on the PRI (ESCAUX or RESELLER)
 Wait an occurrence of the problem and analyze the log (ESCAUX)
 Check that Layer 1 (Physical layer) is up
 Check that Layer 2 (Link layer) is up
 Important parameters : CRC4 / Clocking
 Zaptel groups & channels
 Groups
 Channels
o Woomera based Sangoma interface cards

ISDN debugging and tracing

The information in this document is based on "EICON: Interpreting Diva Traces with DiTrace and XLOG". This
document not only contains information on how to analyse traces, but detailed information about ISDN.

Theoretical introduction
BRI
 Access to the network is called Basic Access (BA).
 It is provided through a Basic Rate Interface (BRI).
 This kind of interface is also called an S0 Interface.
 There are two channels that you can use.

The Basic Rate Interface (BRI) consists of two full-duplex 64 Kbps B channels and one full-duplex 16 Kbps D
channel. The BRI is intended to meet the needs of most individual users. The end user with a BRI connects to the
ISDN through an S0 interface.

PRI
 Access to the network is called Primary Rate Access (PRA).
 It is provided through a Primary Rate Interface (PRI).
  This kind of interface is also called an S2 Interface.
 There are either 30 channels (most of the world) or 23 channels (North America, Japan) that you can use.

The Primary Rate Interface (PRI) consists of 30 full-duplex B channels and one fullduplex 64 Kbps D channel. In
some countries a PRI only consists of 23 B channels. The PRI was designed to meet the needs of users with
greater requirements. The end-user with a PRI connects to the ISDN with an S2 interface.

ISDN Layers
The bulk of ISDN protocol specifications address user to network interface and signalling information carried over
the D channel. The signalling protocol Q.931, for instance, carries call messages only between the user and network
and has no user-to-user significance. The protocols used on the B channel perform the user-to-user transmission of
information. The three protocol layers for the D channel are shown below.

Layer 1

Describes the physical connection between the terminal equipment (TE) and the network termination (NT), including
connectors, line code scheme, framing and electrical characteristics. It is a serial, full-duplex connection that can be
point-to-point or multipoint. Time Division Multiplexing is used to carry data from the D and B channels over the
physical link.
Several Layer 1 states can occur, the two most important ones are:

 Down: this indicates that Layer 1 is deactivated, this means the Communicator is not establishing a Layer 1
connection to the telco ISDN switch
 Activated: this indicates that Layer 1 is up and that you have a connection to the telco

Layer 2

Uses LAP-B and LAP-D procedures (Q.921) to ensure error-free communications over the physical link and defines
the logical link between the user and the network. The protocol also provides the procedures for multiplexing
multiple TEs on a single physical channel at the S/T reference point.

Layer 3

Defines the Q.931 signalling messages used to request services from the network. You will notice that layer 3 of the
B channel is not defined. This is because user data may be transported in any protocol the user application defines.
Optionally, Layer 3 of the D channel will support X.25.
Detailed specifications of the Q.931 protocol can be found in the ITU document

CAPI based EICON/Dialogic interface cards

Gathering traces with the dmonitor tool

# The argument at the end indicates BRI number (=controller number)

/usr/lib/eicon/divas/divactrl dchannel -dmonitor -c 1

/usr/lib/eicon/divas/divactrl dchannel -dmonitor -c 2

/usr/lib/eicon/divas/divactrl dchannel -dmonitor -c 3

/usr/lib/eicon/divas/divactrl dchannel -dmonitor -c 4

Gathering traces with the Diva Trace Assistant
In certain cases, it may be useful to analyse a problem further by looking at a trace that was taken while the problem
was being reproduced. This can be done using the trace program that can be found in the /usr/lib/eicon/divas
directory.

Take a trace as follows:

1. To start a Trace, go to the /usr/lib/eicon/divas directory and type ./Trace and choose the "Standard trace level”
option. On the screen "activate selective tracing" select NO. Accept all other default values.
2. Reproduce the problem.
3· Stop the Trace by typing ./Trace and select the option 'Save Trace and compress file'.

How to convert the Trace to text format

Once the trace has been generated, a file called /var/log/ditrace.bin will be created. To convert the trace into a file
that can be read easily, type the following command in the /usr/lib/eicon/divas directory.
divactrl ditrace -i /var/log/ditrace.bin >ditrace.txt
This will create a file called ditrace.txt in the /usr/lib/eicon/divas directory. This file can now be sent to Customer
Support along with a complete problem description.

Gathering traces with mlog tool

/usr/lib/eicon/divas/divactrl mlog -o -b

Analysing trace files

See also http://www.dialogic.com/support/helpweb/divasvr/advanceddiag.htm

Is the ISDN Network to blame?

The Diva Server Diagnostics utility produces a lot of output, most of this information is only useful to a developer,
but some of the more useful types of message are listed below.
Particularly the 'Cause' information is of interest. Below is an example of what this would look like in a trace. The
contents of your message will almost certainly be different from this one, so just look for the word 'Cause' and take
note of the two groups of letters and numbers following 'Cause' - in this case '82' and 'd8'.

9:12:44.684 - EVENT: Call failed in State 'Outgoing call proceeding'

Q.931 CR8003 DISC
Cause 82 d8 'Incompatible destination'

The first byte '82' contains 'location' information - where the message comes from - while the second byte 'd8' is the
'diagnostic'.
Make a note of the two 'Cause' codes (82 d8 in the above example) and enter these codes into EICON's ISDN
Diagnostic Code Analyser. The code analyser will show you where in the network the problem was reported from,
describe the problem and suggest solutions.
You can often tell a lot about the mode of failure by only looking at the SIG-X and SIG-R messages. Look out for
these messages:

Some common types of bearer capabilities (present in the SIG-X SETUP message) are:
*Suggested Strategy for finding problems in a Diva Server Trace*
Layer1 & Layer 2 problems:
 It is normal to have this kind of traces on a multipoint line (ex: BA without DDI):

SIG-EVENT FFFF0A

SIG-EVENT FFFF00

SIG-EVENT FFFF0A

...

*STEP 1:* Use the Diva Server Trace to find the message !"LoadAdapter" in the Initialisation sequence. To capture
the initialisation sequence of the Diva Server card follow the instructions on the page How to take a Diva Server
trace
This LoadAdapter message reports that the Diva Server card is installed and loaded correctly.
Also look for the message L1_UP in the trace. If you do not see this, then the Diva Server card is not even talking to
the network at the physical layer. Check the cable and the switch type configuration.
_*STEP 2:* Look for SIG-EVENT FFFF messages._
This shows low-level problems with the card. Look to see if there are any D-X or D-R messages; if there are not,
then the DIVA is not communicating with the NT1 or the network.
*STEP 3.* Quickly scan through the trace and look for " EVENT ".
This may show a local or network failure of some kind.
Layer 3 Problems:
_*STEP 4.* Make sure there is a SIG-X containing " SETUP "_
This indicates that a call is being made by the Diva Server card. If there is no SETUP, perhaps the application never
tried to make a call, or perhaps there is a layer 2 problem (see above).
_*STEP 5.* Check if a SIG-R came in containing " CONN "._
If the SETUP receives a "CONN" (CONNECT) as a response, then a "B" channel was successfully connected. If the
SETUP gets "DISCON" (DISCONNECT) as a response, then there is a problem with the network, or with the
equipment at the far end. DISCON normally has Q.931 cause codes to tell you what the problem is.
_*STEP 6.* If the cause of a "DISCON" is not obvious, then check the SETUP message_
Make sure that the ISDN numbers are correct, and also that the Bearer Capabilities look right.
_*STEP 7.* If the " CONN " event is received then look for " B-X " and " B-R " messages._
If you can see one or more of these, then the ISDN network and connection are OK, and you need to go on to the
next stage.
Higher Level Problems:
1. If you see B-X messages, but no B-R messages, then this means that DIVA is sending data (e.g. PPP frames),
but the remote equipment is not responding. Perhaps the remote equipment is not using the same protocol on the
"B" channel?
2. If the B-X and B-R data starts with " *FF 03* " then this is PPP protocol, and the next logical step is to analyse the
PPP to look for negotiation problems.
For example, are both ends using the same protocol (IP, IPX, Netbeui etc), does the authentication (i.e. PAP,
CHAP) succeed?
3. All "B" channel protocols (X.75, X.25, V.120 etc) have their own activation sequence, so at this stage you need to
start analysing the B-channel data with protocol-specific tools. Needless to say, it is important to have some idea
what kind of protocols the equipment at the other end of the link supports.
You can sometimes guess what the protocol is from the first frames sent and received, for example:

 B-X or B-R Frame contains 01 3F or 03 3F (SABM frame)

This could be X.75 (LAPB) or it could be X.25 (which needs to activate LAPB first).
 B-X or B-R Frame contains 08 01 7F (SABME frame)
This could be V.120 connection. Note that sometimes other protocols are carried inside V.120 (PPP or
asynch PPP for example).
 B-X or B-R Frame contains FF 03
This is quite likely to be PPP.
 B-X or B-R Frame contains 7E FF 7D 23
This is quite likely to be Asynch PPP.

Example frames from traces:

[1] D Channel example.
11:18:07.942 - D-X(008) 00 81 08 0A 08 01 06 5A
This is a frame transmitted (X) on the "D" channel. The first four bytes are actually a LAPD header (this is a LAPD
"I" frame) and the next four are the Q.931 header (0x5A = RELEASE COMPLETE).
[2] D Channel example.
11:18:07.964 - D-R(003) 00 81 73
This frame is a received (R) LAPD frame (in this case a "UA" frame).
[3] B Channel example - X.75 data.
=11:18:07.892 - B1-R(068) 68 bytes
0x0000 03 20 01 00 47 0D 00 83 30 31 38 31 32 38 33 39 . .G...01812839
0x0010 39 39 39 00 00 00 00 00 00 00 00 00 00 00 00 00 999.............
0x0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x0040 00 00 00 00 ....=
This frame is a received on "B" channel 1.
[4] B Channel example - PPP data.
=16:12:17.275 - B1-X(014) 14 bytes
LCP : id 2, len 10 ==> Configure-Request
- Magic-Number 0x0007A77B
0x0000 FF 03 C0 21 01 02 00 0A 05 06 00 07 A7 7B ..@!........'{=
This is a transmitted frame on "B" channel 1. Outgoing frames are truncated, if they exceed 28 bytes. As you can
see, DITRACE interprets some of the data (in this case PPP), and puts a decoded version before the data buffer
itself.
[5] B Channel example - PPP data.
=16:12:17.261 - B1-R(044) 44 bytes
LCP : id 1, len 40 ==> Configure-Reject
- Async-CC-Map 0x000A0000
- Protocol-Field-Compression
- Addr-and-Control-Field-Compression
- Callback ==> Location is determined during CBCP negotiation
- Multilink-MRRU 1500
- Multilink-Endpoint-Discriminator
-- Locally Assigned Address
0x0000 FF 03 C0 21 04 01 00 28 02 06 00 0A 00 00 07 02 ..@!...(........
0x0010 08 02 0D 03 06 11 04 05 DC 13 13 01 98 01 00 00 ........\.......
0x0020 7B A7 07 00 4C E3 F0 C3 C3 22 00 00 {'..LcpCC"..=
Here the PPP LCP header has been decoded by DITRACE, and the LCP Options are all interpreted to save you
doing it by hand.
[6] SIG example - DISCONNECT
=11:18:07.908 - SIG-R(008) 08 01 06 45 08 02 80 90
Q.931 CR0006 DISC
Cause 80 90 'Normal call clearing'=
This is a common type of frame you will be looking for in the trace. It shows that a Q.931 DISCONNECT has been
received (i.e. a call has disconnected) and the Cause code is interpreted and shown in English.
[7] SIG example - SETUP
=11:18:06.753 - SIG-X(027) 08 01 06 05 A1 04 02 88 90 18 01 83 70 08 81 32 38 33 39 39 39 39
Q.931 CR0006 SETUP
MORE
Bearer Capability 88 90
Channel Id 83
Called Party Number 81 '2839999'=
This is a call SETUP frame, as decoded in a SIG-X element. This shows a call attempt being sent to the network by
the DIVA card. You can see that the number being called is "2839999". The Bearer Capability field shows the type
of call being selected.
[8] CAPI Example - CAPI_PUT_MESSAGE
=11:18:06.732 - CAPI20_PUT(026) APPL 0001 0000:00000001 LISTEN REQ
52 00 00 00 00 00 00 00 00 00 00 00 00 00=
CAPI_PUT_MESSAGE shows a message being sent by an application to the DIVA card's CAPI 2.0 interface. Then
numbers show the application id, the message number and the PLCI/NCCI of the message, then the last thing on
the line is the CAPI message type itself, in this case a LISTEN_REQ. Message-specific data (if any) is on the
following line, and you can interpret this with the help of the CAPI 2.0 specification.
[9] CAPI Example - CAPI_GET_MESSAGE
=11:18:07.657 - CAPI20_GET(015) APPL 0001 0024:00000201 INFO IND
07 80 00=
Here you can see an INFO_IND message being read by the CAPI application. The data on the next line, tells what
kind of information element was seen by the DIVA card (in this case 07 = Q.931 CONNECT).
[10] EVENT Example.
=16:51:07.913 - EVENT: Call failed in State 'Outgoing call proceeding'
Q.931 CR8003 DISC
Cause 81 d8 'Incompatible destination'=
It pays to quickly scan through the trace for EVENT messages, since this can quite often show you the fault. Here, a
DISCONNECT is being received in response to a SETUP, and the 'Incompatible destination' cause code shows that
the user has made a digital call to a non-digital destination (e.g. a phone).
[11] MDL-ERROR Example
0:0032:776 - D-R(004) 02 B7 01 03
0:0032:777 - D-X(004) 02 B7 01 03
0:0033:365 - D-R(003) 02 B7 7F
0:0033:365 - MDL-ERROR(F)
0:0033:366 - D-X(003) 02 B7 73
0:0035:492 - D-R(003) 42 2B 7F

In this example, the first two lines show normal D-channel idle activity in both directions, and then at line 3, the
network resets the LAPD level by sending a SAMBE. This unexpected reset is detected by the Diva Server card as
an MDL error (according to the definitions for ISDN, for example in the ETSI specification). The type of the error is
'F', which means 'received SABME, Peer initiated re-establish'. More information on MDL Errors.

Is the Escaux UCS server to blame?

 If calls are rejected and none of them show up in the asterisk console log and if you are sure that all external
number mappings are correct, the Diva Eicon CAPI stack might be to blame. Restart the Eicon card as
follows:
o Stop asterisk
o Stop ALL processes of tracing ISDN line
o Stop F-Secure anti-rootkit
o /usr/lib/eicon/divas/Stop to stop the card
o /usr/lib/eicon/divas/Start to start the card.

Zaptel based Digium PRI interface cards

Taking the right trace
In order to troubleshoot a problem, the minimum requirement is to activate the right trace:

Add more traces in /etc/asterisk/logger.conf needed (ESCAUX or RESELLER)

ESCAUX:

[logfiles]

messages => warning,verbose,debug,error

RESELLER:
Enable more trace in the asterisk module.

Restart asterisk (ESCAUX or RESELLER)

Put trace on the PRI (ESCAUX or RESELLER)

Asterisk Console

pri debug span x

Wait an occurrence of the problem and analyze the log (ESCAUX)

No HDLC errors should appear in /var/log/asterisk messages.

Examples:

Jul 23 01:09:28 DEBUG[1063] chan_zap.c: Got event HDLC Bad FCS (8) on D-channel for span 1

Jul 23 01:09:28 DEBUG[1063] chan_zap.c: Got event HDLC Bad FCS (8) on D-channel for span 1

Jul 23 01:09:29 DEBUG[1063] chan_zap.c: Got event HDLC Abort (6) on D-channel for span 1

Jul 23 01:09:29 DEBUG[1063] chan_zap.c: Got event HDLC Bad FCS (8) on D-channel for span 1

Jul 23 01:09:30 DEBUG[1063] chan_zap.c: Got event HDLC Abort (6) on D-channel for span 1

Jul 23 01:09:32 DEBUG[1063] chan_zap.c: Got event HDLC Bad FCS (8) on D-channel for span 1

Jul 23 01:09:32 DEBUG[1063] chan_zap.c: Got event HDLC Bad FCS (8) on D-channel for span 1

Jul 23 01:09:32 DEBUG[1063] chan_zap.c: Got event HDLC Bad FCS (8) on D-channel for span 1

If there are such messages, it is either the Escaux card or the line which had error. However, the communications
are not cut. If you can see yellow alarm or red alarms in /var/log/asterisk/messages then all the communication are
cut. If it is not case, look if a lot of DISCONNECT message comes in the same time from the provider. If it is the
case, ask why to the provider.
Check that Layer 1 (Physical layer) is up
 zttool command should show Digium TE110P T1/E1 or 405P Cards PRI green
 to be 100% sure AND the service is down, you can reconfigure all zaptel devices in order to refresh the
status of zttool (That cuts calls): ztcfg -vvvv
 The following messages can be found in the log files

o wcte1xxp: Setting yellow alarm

o wcte1xxp: Clearing yellow alarm

o Jul 20 18:27:50 WARNING[2652]: Detected alarm on channel 2: Red Alarm

o No D-channels available! Using Primary on channel anyway 24!

Check that Layer 2 (Link layer) is up

 pri show span 1 (or 2,3,4)
 NOT GOOD: Status = Down means we have no signaling

Primary D-channel: 16

Status: Provisioned, Down, Active

Switchtype: EuroISDN

Type: CPE

Window Length: 0/7

Sentrej: 0

SolicitFbit: 0

Retrans: 0

Busy: 0

Overlap Dial: 0

 GOOD: Status = Up means signaling

Primary D-channel: 47

Status: Provisioned, Up, Active

Switchtype: EuroISDN

Type: CPE

Window Length: 0/7

Sentrej: 0

SolicitFbit: 0

Retrans: 0

Busy: 0

Overlap Dial: 0

 If all is good at this moment issues a 'pri debug span x' where x is the PRI number and make an incoming
calls. You should see a SETUP Q931 message. GOOD:

< Protocol Discriminator: Q.931 (8) len=39

< Call Ref: len= 2 (reference 17407/0x43FF) (Originator)

< Message type: SETUP (5)

< [a1]

< Sending Complete (len= 1)

< [04 03 80 90 a3]

< Bearer Capability (len= 5) [ Ext: 1 Q.931 Std: 0 Info transfer capability: Speech (0)

< Ext: 1 Trans mode/rate: 64kbps, circuit-mode (16)

< Ext: 1 User information layer 1: A-Law (35)

< [18 03 a1 83 82]

< Channel ID (len= 5) [ Ext: 1 IntID: Implicit, PRI Spare: 0, Preferred Dchan: 0

< ChanSel: Reserved

< Ext: 1 Coding: 0 Number Specified Channel Type: 3

< Ext: 1 Channel: 2 ]

< [6c 0a 21 81 32 36 38 36 30 39 30 33]

< Calling Number (len=12) [ Ext: 0 TON: National Number (2) NPI: ISDN/Telephony Numbering Plan (E.164/E.163) (1)
< Presentation: Presentation permitted, user number passed network screening (1) '26860903' ]

< [70 09 a1 32 37 36 36 31 30 39 31]

< Called Number (len=11) [ Ext: 1 TON: National Number (2) NPI: ISDN/Telephony Numbering Plan (E.164/E.163) (1)
'27661091' ]

-- Making new call for cr 17407

-- Processing Q.931 Call Setup

-- Processing IE 161 (cs0, Sending Complete)

-- Processing IE 4 (cs0, Bearer Capability)

-- Processing IE 24 (cs0, Channel Identification)

-- Processing IE 108 (cs0, Calling Party Number)

-- Processing IE 112 (cs0, Called Party Number)

> Protocol Discriminator: Q.931 (8) len=10

> Call Ref: len= 2 (reference 50175/0xC3FF) (Terminator)

> Message type: CALL PROCEEDING (2)

> [18 03 a9 83 82]

> Channel ID (len= 5) [ Ext: 1 IntID: Implicit, PRI Spare: 0, Exclusive Dchan: 0

> ChanSel: Reserved

> Ext: 1 Coding: 0 Number Specified Channel Type: 3

> Ext: 1 Channel: 2 ]

Important parameters : CRC4 / Clocking

 Usually CRC4 is off. This is set in /etc/zaptel.conf. If you do not see crc4 in this file, then it means it is OFF.
 In the PRI interface on the SMP :
o "NET - Network Side PRI" means our PBX is the master for the clocking.
o "CPE - Customer Side PRI" means our PBX is slave for the clocking.
 If you need add/remove the crc4, you need to issue a ztcfg -vvvv (! conversations are cut) AND to restart
asterisk

Zaptel groups & channels

Groups

 If you see "Dial ZAP/g1" -> it means the call is going over the 1st PRI interface (i.e. the one that is defined
as group=1 in gen_zapata.conf)
 If you see "Dial ZAP/g2" -> it means the call is going over the 2nd PRI interface (i.e. the one that is defined
as group=2 in gen_zapata.conf)
Channels

 If you see "Zap/6-1 is ringing", it means zaptel sent the call to the Zaptel Channel nr 6.
 For more detail about zaptel channels : see zaptel.conf

Woomera based Sangoma interface cards