White Paper

Best Practices for

Monitoring Nortel
CS1000 VoIP Metrics
With eHealth for Voice
Dan Lewis
Voice Engineering
September 2005

Executive Summary
This document summarizes issues in maintaining quality
in Voice over IP network transmissions and discusses how
eHealth for Voice can be used to monitor these issues in
the Nortel Communication Server 1000 environment.
VoIP Call Quality and the Network
Environment
Need for Continuous Monitoring.
Voice over IP is intended to replace existing analog and
digital telephone applications in such a way as to be virtually
transparent to the end user. While the economic and strategic
benefits of VoIP are certainly notable, users of traditional
telephone technology, who have come to expect a high degree
of quality and reliability in voice transmissions, are not
likely to be willing to make significant sacrifices in those
areas to embrace the new technology.
Much of the economic benefit of deploying VoIP comes
from making use of existing network infrastructure as a
medium for voice communications. Yet existing network
structures have come into being to deliver data, primarily
in the form of discreet files, not in the continuous streams
which characterize voice and media transmission.
The vendors of today’s enterprise VoIP applications have had
remarkable success in developing and utilizing mechanisms to
overcome these limitations. Nortel CS 1000 servers, equipped
with voice gateway media cards, are capable of delivering
voice communication to IP telephone terminal equipment
with a quality and reliability that approaches that of traditional
telephony. But for that quality and reliability to be maintained
in an enterprise network, both the application and the network
must be monitored continuously so that potential issues may
be identified and corrected before they result in degradation
of service. In addition, it is important to manage and monitor
bandwidth utlization on the network, to insure that necessary
bandwidth can be provided for the continuous streams of
packets with voice transmissions require.
Quality of Service Mechanisms to Ensure
Bandwidth Availability
Bandwidth availability on shared networks tends to fluctuate
significantly since data transmission is bursty by nature.
Because VoIP packets require real-time transmission, it is
crucial that some means of ensuring bandwidth availability
for VoIP packets is instituted. This is generally accomplished
through QoS mechanisms which allow VoIP packets to be
given priority at routers, switches and endpoints. For effective
transmission of voice over IP, networks must be QoS enabled
and configurable to support packet priorities1.
2
Nortel CS 1000 systems enable QoS mechanisms by
classifying IP packets at point of origin:
• CS 1000 Signaling Server classifies packets as signaling
packets.
• Voice Gateway Media Cards classify packets as voice or
signaling packets.
• Internet Telephones classify their packets as voice or
signaling packets and mark them for priority.
For details on implementing QoS mechanisms on signaling
servers, voice gateway media cards, and internet telephones
at Layer 2 (802.1p) or Layer 3 (DiffServ), consult your
Nortel documentation.
Network Obstacles to Successful Voice Transfer
The Internet Protocol has become the de-facto standard
for data communication in both local area and wide area
networks. While it is remarkably successful in transmitting
quantities of data both large and small, it presents some
inherent difficulties to voice communications which must
travel point to point in a more-or-less continuous stream.
Latency, packet loss, and jitter (variable delay) are all
potential obstacles to successful voice transmission which
must be overcome. They are also network issues which
must be monitored.
Latency. Latency, or delay, is the time between the sending
of a packet and its receipt at a destination. Since no
communication is instantaneous, some latency is inherent
in all transmissions. For traditional data traffic delays on
the order of a few hundred milliseconds are not generally of
great importance. For voice communication, delays above
150 ms have been shown to be noticeable to users, and
delays above 250 ms are generally observed as
degradation in voice quality, especially with higher
compression codecs.
Packet Loss. The loss of individual IP packets during
transmission can be attributed to a number of causes.
Some packet loss is characteristic of all IP communication
and is compensated for in data applications by protocols
which allow for the retransmission of lost packets.
Because of the streaming nature of VoIP transmissions,
retransmission of packets is not generally feasible. Voice
applications are, however, somewhat more tolerant of
packet loss, since no one packet will contain more than a
fraction of a phoneme. In addition packet loss
concealment (PLC) is built into VoIP codecs to smooth out
the effects of packet loss by duplicating or synthesizing voice
from existing packets. PLC can typically repair a gap of 20 to
40 ms with no discernable effect on the vocalization. For
longer gaps (the loss of a number of successive packets)
the PLC algorithms are less effective and noticeable
clipping or choppy speech are the result.
Jitter. It is a characteristic of IP networks that packets do not
remain evenly spaced in time as they traverse the network.
Jitter, or variable delay, is the variation in latency between
adjacent packets in the voice stream. Voice applications
require that packets are fed to the decoder at a generally
constant rate; packets which do not arrive at the decoder
in time are considered lost. If jitter is very high, whole groups
of packets can be lost so that PLC algorithms cannot prevent
significant degradation of voice quality. Jitter generally
becomes more severe as available bandwidth becomes
compromised by network congestion, or as a result of
absent, or poorly configured QoS mechanisms.
Most VoIP applications make use of a configurable buffer
on the receiving end to smooth out the stream of packets
fed to the decoder. However, buffering adds to the overall
latency of the transmission end to end. Effective configuration
of the jitter buffer requires that it be just large enough to
smooth out the stream and minimize packet loss, without
being so large as to add significant delay.
Voice Quality Metrics
To a large degree, voice quality describes a subjective
experience. While the bottom-line difference between
understanding and not understanding spoken language
over a telephone connection is crucial and fundamental,
finer grain measurements of voice quality are not easily
quantifiable.
Mean Opinion Scores (MOS). The industry has for some
time used Mean Opinion Score (MOS) as a rough means
of quantifying voice quality at the destination end of a
communication link. Mean opinion scores, are based on
averaged ratings by a sample of listeners. Formally, MOS
is determined by asking a sample of listeners to evaluate
test sentences read over a communications link by male
and female readers. Listeners use a 1 to 5 scoring system:
1 (bad), 2 (poor) 3 (fair) 4 (good) 5(excellent). The
arithmetic mean of all scores is the mean opinion score for
the transmission. In practice, MOS serves as means of
expressing estimates of voice quality.
The E-Model and R Values. The E-Model Transmission
Planning Tool, developed by the ITU, is a complex
calculation which attempts to provide a quantifiable
estimate of overall voice transmission quality expressed as
a single value, R. Nortel uses a simplified E-Model
calculation which takes into account the characteristics of
Nortel's IP phones. The standard E-Model is described in
ITU-T standards E.107 and E.108. Nortel's R value calculation
is described in detail in Data Networking for Voice over IP
(Nortel document 553-3001-160).
3
The R value scale is from 0 to 100 and R values may be
approximately correlated to Mean Opinion Scores:
R MOS Opinion
90 4.5 Very satisfied
80 4.0 Satisfied
70 3.5 Some users dissatisfied
60 3.0 Many users dissatisfied
50 2.5 Nearly all users dissatisfied
Nortel recommends maintaining R values above 70.
Using eHealth for Voice to Manage
Nortel Communication Servers
Continuous monitoring of network performance is crucial
to maintaining voice quality in VoIP applications on shared
networks. Managers need to be able to monitor bandwidth
availability and VoIP packet latency, jitter and packet loss on
a continuous basis to identify problems before they seriously
impair voice quality. By collecting and organizing data from
the Nortel Server, eHealth for Voice is able to produce a
number of reports and data views which assist in monitoring
VoIP performance. In addition, the eHealth Policy Manager
can be configured to automatically alert operators and
managers when user defined thresholds are met.
Setting Baselines for Policy-Based Management
eHealth for Voice provides a number of reports to track
network issues and usage statistics. Running reports for
bandwidth utilization, call traffic statistics, QoS IP statistics, and
Operational Measurements on a regular basis can provide a
baseline picture of VoIP performance in the network environment.
Key reports should be set as scheduled reports to be
automatically delivered to appropriate personnel for daily
monitoring of conditions. Once the system has been monitored
in this way for several days, it should be possible to establish
reasonable thresholds for key values such as bandwidth
utilization, latency, jitter, packet loss, and overall voice
quality (R-value). These thresholds then can form the basis
of policies monitored via eHealth Policy Manager so that
operators may be alerted automatically (via page, console
alert, email, or third-party application) when thresholds
are exceeded.
As a continuing process reports should be analyzed and
thresholds adjusted as appropriate to meet changing
conditions. In addition to providing monitoring, the
collection and retention of data in the eHealth for Voice
database, allows analysis of issues and provides a basis
for further system and network management.
Direct Monitoring of Network Issues would like to know if peak bandwidth use increases going
A number of eHealth for Voice reports are designed to forward, so we consider a threshold of 30% and we set a
assist managers in monitoring network traffic conditions policy to alert if either interzone or intrazone peak bandwidth
affecting VoIP packets. In addition, data collected by exceeds 30 percent at any given point. An alert at 30%
eHealth for Voice from the Nortel Signaling Server, and will tell us that our bandwidth use for calls in a particular
organized into discrete tables within the eHealth for Voice zone is increasing, and it will warn us in time to investigate
database, may be viewed directly through the use of the the cause and to consider making changes to available
program’s query designer. Automatic alerting can be set bandwidth or the network topology if necessary.
up through the eHealth for Voice Policy Manager, so that
operators are alerted when threshold values are exceeded. We create a new node-based policy for the condition from
the eHealth for Voice console.
Bandwidth Utilization. eHealth for Voice stores bandwidth
utilization statistics in the Internet Telephone Zone Traffic
Data table. Data from this table can be viewed directly via
the Query Analyzer and filtered as appropriate. The Zone
Bandwidth Utilization report should be run on a regular
basis. Ideally it should be scheduled using the program’s
scheduled reporting feature, and delivered to appropriate
personnel for regular analysis. The report details interzone
and intrazone: total bandwidth, bandwidth usage and
bandwidth percentage used at peak. The statistics are
separately reported for each established bandwidth zone.

Once the report has been run for a period of time it should
be possible to set a reasonable baseline for peak performance
so that appropriate personnel are notified when the
threshold is exceeded.
Bandwidth Example. To illustrate the process of
establishing and monitoring thresholds, consider the
following bandwidth example:
Figire 2. Adding a New Policy
We create a condition consisting of two statements linked
by OR. The statements are selected via the interface and
reference the specific fields for interzone peak and intrazone
peak in the from the zone utilization table of the eHealth
for Voice database.

Figure 1. Selecting the Zone Bandwidth Report

Run the Zone Bandwidth Utilization report for several
days. Although here we are running the report manually
from the report selector in the eHealth for Voice console,
it may be preferable to schedule the report to run daily
and be distributed automatically to the voice network
manager (see your eHealth for Voice documentation).
Assume that the report indicates that peak bandwidth for
both inter and intra zone calls is generally in the 20% —
35% range. This appears to be a good level and it therefore
appears that there is plenty of bandwidth available. But we
Figure 3. Creating the Condition
Finally, on the conditions tab of the policy form we assign
operation interval (all hours, so that the policy is in effect
and bandwidth is monitored at all hours), we set the severity
level of the policy to “Warning” and assign an action plan.
Action plans specify what happens when the threshold
conditions are met. In this example we use an action plan
which will send an email message to an operator at any
hour, and an alert message to the operator’s console during
business hours when the console is monitored. eHealth for

4
Voice Policy Manager allows great flexibility in specification
of actions and action plans and can send messages to pagers
and to eHealth Live Health or other network monitoring
software as well as email and console alerts. For details on
setting up and maintaining eHealth for Voice policies via
Policy Manager see the eHealth for Voice documentation.
Once this policy is activated a notice will be sent anytime
peak bandwidth usage exceeds 30%. In this way eHealth
for Voice becomes an active watchdog monitoring the
bandwidth utilization of your Nortel VoIP traffic.
To assist the user in setting up policies for monitoring
VoIP performance, eHealth for Voice Policy Manager
includes a number of pre-configured policy templates which
may be modified as appropriate. There are templates for
bandwidth usage, inter, and intrazone, as well as templates
for monitoring latency, jitter, and voice quality. A detailed
list of policy templates is included in the eHealth for Voice
documentation.
Figure 4. The Completed Policy
Latency. Maximum and average latency statistics, reported
by internet telephone type, is collected by eHealth for Voice
and stored in the Operational Measurements table. The
data may be viewed directly via the eHealth for Voice
Query Designer, or reported in the Operational
Measurements report. In general, values above 150 ms are
seen as signs of deteriorating voice quality. In the same
manner described above, a baseline for average and
maximum latency should be established by consulting a
run of operational measurements reports. Once baseline
threshold values are established, a policy can be
established based on either average or maximum values
(or both), to notify an operator when latency values
exceed the established thresholds. This would provide a
timely warning that latency issues might be beginning to
impair voice quality. Solutions to latency issues might be
changing QoS configuration, reducing the size of the jitter
buffer, increasing link capacity, or changing network
topology to reduce the number of hops between nodes.
5
Jitter. Managing jitter in VoIP networks consists largely in
finding the optimum size for the jitter buffer. Nortel allows
jitter buffer size to be configured via the voice gateway
media card configuration as part of the DSP profile. A
buffer too small will result in discarded packets, while
increasing the jitter buffer increases overall latency. Nortel
generally recommends that internet telephone firmware be
configured with a jitter buffer of 3.
eHealth for Voice stores maximum and average values for
jitter broken down by telephone type. To monitor the amount
of jitter in your VoIP traffic, consult Operational Measurements
data, either through the eHealth for Voice query designer
or on the Operational Measurements report. Baseline and
alerting procedures would be similar to those outlined
above. In addition, you should run the Operational
Measurements report or consult the data any time the
jitter buffer is changed in the voice gateway media card
(VGMC) or telephone firmware.
Packet Loss. Packet Loss is perhaps the one area where
VoIP applications are somewhat more tolerant than most
data applications. Nortel’s packet loss concealment (PLC)
can typically handle losses of 40 — 60 ms without
noticeable loss of voice quality. Loss of a number of
consecutive packets, resulting in longer “holes” in the vocal
stream will, however become noticeable as clipping or
choppy speech.
Packet loss is also stored in the Operational Measurements
table and reported via the Operational Measurements report.
Packet loss is reported as a percentage of total at the gateway
and at the internet telephone. Monitor the Operational
Measurements report for percentage of packet loss. As with
other network issues, an eHealth for Voice policy may be
set up to monitor packet loss send notifications as appropriate.
Packet loss should be examined in conjunction with jitter to
determine if it is caused by excessive jitter or insufficient
jitter buffering. It should also be considered in conjunction
with bandwidth utilization. Examination of the Operational
Measurements report, where these items are reported
together is helpful in spotting trends and isolating issues.
Monitoring Voice Quality via R Value
The E-model attempts to quantify voice quality into a single
value, R, by analyzing a number of parameters that affect
voice quality (see discussion above). Another approach to
monitoring voice quality is to monitor the R value itself. In
most instances this should not replace monitoring latency,
jitter, and packet loss, but it does give another approach
which can be helpful in conjunction with monitoring the
“demons.” In fact, setting policies based on R-value may
prove to be the most direct way of developing alerts when
voice quality begins to be impaired. However, a reduced
R-value by itself tells little of its cause. For that the other
data is essential.
eHealth for Voice stores listening R-value data in the References
Operational Measurements table as Listening R-Factor. MCK Communications, Inc.
Nortel’s supplied R-value is used for Nortel IP telephones. Voice Over IP: a primer for telecom professionals.
For other hardware, eHealth for Voice calculates the R- MCK Communications, 2002
value using the standard E-model calculation.
Morrissey, Peter
By monitoring the R-value on the Operational Prepare Your Network for VoIP
Measurements Report it should be possible to establish a Network Computing, July 8, 2004 p.69-74.
reasonable baseline and threshold. Nortel recommends
that R-value be maintained above 70. Nortel Networks, Inc.
Data Networking for Voice over IP
Once thresholds have been determined, policies may be Document Number: 553-3001-160
created in the same way as with other elements to be Nortel Networks, Inc., October 2003
monitored. Policy Manager includes a number of pre-
defined policy templates for monitoring R-value and MOS Seligman, Dan
metrics. Introduction to IP Quality of Service Technology
(Concord Communications Technical White Paper)
Concord Communications, Inc., 2005
Succession QoS Thresholds
Through its QoS interface, Nortel Succession software
also allows storage thresholds for warning and
unacceptable levels of packet loss, jitter, latency, echo,
and R-value on the signaling server. eHealth for Voice
collects counts of threshold violations in the Internet
Telephone Zone Traffic table. If the QoS configuration is
set on the Succession, eHealth for Voice policies may be
set up based on number of violations.

Effective thresholds are arrived at by an ongoing process

of monitoring and refining. By storing data indefinitely and
making it available in trending reports, eHealth for Voice
can greatly assist in arriving at effective QoS thresholds.
Using eHealth for Voice Policy Manager in conjunction
with QoS output to alert appropriate personnel of
threshold violations, provides a 24/7 watchdog to insure
voice quality.

6
1
Quality of Service mechanisms can be monitored network-wide with other eHealth products, also available from CA. For a
discussion of the technology, see the white paper, “Introduction to IP Quality of Service Technology,” listed in the reference
section at the end of this paper.

Copyright © 2006 CA. All rights reserved. All trademarks, trade names, service marks and logos referenced herein belong to their respective companies. This document is for your informational
purposes only. To the extent permitted by applicable law, CA provides this document “AS IS” without warranty of any kind, including, without limitation, any implied warranties of merchantability,
fitness for a particular purpose, or non-infringement. In no event will CA be liable for any loss or damage, direct or indirect, from the use of this document, including, without limitation, lost profits,
business interruption, goodwill or lost data, even if CA is expressly advised of such damages. MP295620206