UMTS Performance Optimization and Acceptance Guidelines

UMTS Performance Optimization and Acceptance Guidelines
contents
1.
INTRODUCTION.............................................................................................
................ 6
1.1.
OBJECT..........................................................................................................
.............. 6
1.2. Scope of this
document.............................................................................................. 6
1.3. Key Performance Indicator values and
Definitions................................................ 7
1.4. Audience for this
document....................................................................................... 7
2. Related
documents................................................................................................ 7
2.1. Applicable
documents................................................................................................. 7
2.2. Reference
documents................................................................................................ 7
3. General
Methodology......................................................................................... 9
4.
Assumptions..................................................................................................
........... 10
4.1. General
Assumptions................................................................................................
10
4.2. Specific
Assumptions................................................................................................
11
4.2.1. Design
criteria.......................................................................................................
11
4.2.2. Performance
criteria............................................................................................. 11
4.3. Warnings on metric
definitions............................................................................... 11
4.4.
Notation........................................................................................................
.............. 11
5. Test
Areas.............................................................................................................
..... 12
5.1.
Routes...........................................................................................................
............. 12
5.2. In-building Test
Areas............................................................................................... 12
5.3. Special Test
Areas.................................................................................................... 12
5.4. Warranty
Zone..........................................................................................................
12
6. Simulated Traffic Load (if applicable)........................................................
13
6.1.
Uplink............................................................................................................
.............. 13
6.2.
Downlink.......................................................................................................
............. 14
7. Test
Setup.............................................................................................................
.... 14
8. Performance
Metrics......................................................................................... 14
8.1. Combined Metric: Successful Call
Rate................................................................ 15
8.2. Detailed
Metrics........................................................................................................
16
9. System Optimization
process.......................................................................... 16
9.1. Configuration
Audit.................................................................................................. 19
9.2. Cell
Shakedown....................................................................................................
.... 20
9.2.1.
Objective.......................................................................................................
........ 20
9.2.2. Entrance
Criteria................................................................................................... 20
9.2.3. RF BASIC
checking............................................................................................. 20
9.2.4. Drive
Routes.........................................................................................................
21
9.2.5. Test
Drives...........................................................................................................
21
9.2.6. Test Areas (indoor
Site)........................................................................................ 21
9.2.7. Exit
Criteria..........................................................................................................
. 21
9.2.8. Issue
List...............................................................................................................
21
9.3. Cluster
Optimization.................................................................................................
22
9.3.1.
Objective.......................................................................................................
........ 22
9.3.2. Entrance
Criteria................................................................................................... 22
9.3.3. Drive
Routes.........................................................................................................
23
9.3.4. Test
Drives...........................................................................................................
23
9.3.5. Data Analysis / Problem
Resolution..................................................................... 26
9.3.6. Exit
Criteria..........................................................................................................
. 26
9.3.7.
Reports..........................................................................................................
....... 27
9.3.8. Issue
List...............................................................................................................
27
9.4. System
Optimization..................................................................................................
. 27
9.4.1.
Objective.......................................................................................................
........ 27
9.4.2. Entrance
Criteria................................................................................................... 28
9.4.3. Drive
Routes.........................................................................................................
28
9.4.4. Drive
TESTs.........................................................................................................
28
9.4.5. Data Analysis / Problem
Resolution..................................................................... 28
9.4.6. Exit
Criteria..........................................................................................................
. 28
9.4.7.
Reports..........................................................................................................
....... 29
9.4.8. Issue
List...............................................................................................................
29
10.
Reports..........................................................................................................
............ 30
10.1. Performance Result
Report............................................................................... 30
10.1.1. Geodetic
Information......................................................................................... 30
10.1.2. Key Performance Indicators (KPIs)
Reports................................................... 30
10.2. Performance Measurement
Reports.................................................................. 31
10.3. Test Environment
Report..................................................................................... 31
10.4. Issue
List................................................................................................................
. 31
11. OCNS
unavailability...............................................................................................
32
12. Abbreviations and
definitions......................................................................... 32
12.1.
Abbreviations................................................................................................
.......... 32
12.2.
definitions.....................................................................................................
.......... 34
13. Appendix 1: Test Equipment
List...................................................................... 36
14. Appendix 2: Performance
Metrics.................................................................. 37
14.1. KPI Indicative
values............................................................................................... 37
14.2. Detailed metrics for acceptance and
optimization............................................. 38
14.2.1. Acceptance
Metrics.......................................................................................... 38
14.2.2. Optimization
Metrics......................................................................................... 44
14.3. Measurement
Uncertainty.................................................................................... 53
14.3.1. Tolerance Interval of One Data
Population....................................................... 53
14.3.2. Tolerance Interval of Several Data
Populations............................................... 54
14.3.3. Table z-values vs. confidence

levels................................................................ 56
15. Appendix 3: Overall planning for performance acceptance.......... 57
15.1. Optimization
resources:........................................................................................ 57
15.1.1. IS-95
Experience.............................................................................................. 57
15.1.2.
teams............................................................................................................
.... 57
15.1.3. Extrapolation to
UMTS...................................................................................... 58
1. INTRODUCTION
1.1. OBJECT
This document is intended to describe a generic procedure for performing
UMTS network performance optimization and acceptance tests as well as
the main performance metrics that require field assessment for the set of
services provided through this network.
Procedures, conditions and performance metrics for actual contracts may
have variations to those referenced, presented and described in this
document.
This document is a generic and internal document. It provides guidelines
and a base for writing the actual Network Performance Acceptance Test
Plan.
It must not be communicated directly to the customer.
1.2. Scope of this document
This document is intended to define two key processes before launching a
UMTS network: performance optimization and performance acceptance.
Performance Optimization
A detailed system optimization process is needed to ensure that each of
the System Performance Metrics can be achieved. As such, the objectives
of the performance optimization process are to:
uniform optimization procedure
uniform measurement standards (test set-up, route selection, test area
selection, etc...)
set up clear methods and way of measurements in agreement between
the customer and Nortel Networks.
set up a process of following up with issues and work yet to be

completed (issue lists)
tracking of performance improvement trends over time
Performance Acceptance
Performance Acceptance can be defined as the final performance test
before commercial launch of a network. The sometimes called in-service
performance acceptance (when the acceptance is actually done on a
commercial network) is outside the scope of this document.
The performance acceptance test validates the network design. It shows
the compliance with contractual commitments.
The criteria and process for performance acceptance must be clearly
defined in order to ensure consistency all over the system. The detailed
measurement processes for each metric will be defined including all
necessary sign off requirements.
Due to the unavailability of UMTS collection and analysis hardware and
software equipment at the moment, the presented methods will be
updated as the equipment becomes available.
1.3. Key Performance Indicator values and Definitions
The Product-KPI values and default definitions may be found in [R1].
However, the definitions have been recalled in this document since some
have to be slightly adapted to measurement methods and devices
(availability of points of measurements, post-processing ability...). The
Network-KPI values (including network engineering) must be evaluated on
a case by case basis to include contract specificities.
In any case, when writing the final and customized Acceptance Test Plan
for the customer, values and definitions must be adapted according to
customer requirements or network peculiarities.
1.4. Audience for this document
This document is internal and intended mainly for dressing generic
Performance Acceptance Guidelines that may be used to customize each
customers Acceptance Test Plan. The resulting Acceptance Test Plan is
required to be signed off by both the Nortel Networks and the customers
acceptance primes.
Part of the audience should be found in teams dealing with:
Network performance acceptance
RF engineering
Network monitoring (metric implementation)
Network performance contract negotiations
2. Related documents
2.1. Applicable documents

All customized UMTS network performance acceptance test plans.
2.2. Reference documents
[R1] UMT/SYS/DD/72 v01.02 UMTS Key Performance Indicators (KPI)
Design [Draft]
[R2] 3GPP TS 21.904 v3.3.0 UE Capability Requirements
[R3] 3GPP TS 22.105 v3.10.0 Service aspects; services and service
capabilities
[R4] 3GPP TS 23.003 v3.7.0 Numbering, addressing and identification
[R5] 3GPP TS 23.107 v3.5.0 QoS Concept and Architecture
[R6] 3GPP TS 24.008 v3.6.0 Mobile radio interface layer 3 Core Network
Protocols Stage 3
[R7] 3GPP TS 25.101 v3.6.0 UE Radio Transmission and Reception (FDD)
[R8] 3GPP TS 25.215 v3.4.0 Physical layer Measurements (FDD)
[R9] 3GPP TS 25.303 v3.7.0 Interlayer Procedures in Connected Mode
[R10] 3GPP TS 25.331 v3.6.0 RRC Protocol Specification (Release 1999)
[R11] 3GPP TS 25.413 v3.5.0 UTRAN Iu Interface RANAP Signalling
[R12] ITU-T I.356 Integrated Services Digital Network; Overall network
aspects and functions Performance objectives
[R13] ITU-T P.800 Methods for objective and sujective assessment of
quality
Objective quality measurement of telephone band (300-3400 Hz) speech
codecs
[R14] ITU-T P.861 Methods for objctive and subjective assessment of
quality
Methods for subjective determination of transmission quality
[R15] RFC-2330 Framework for IP Performance Metrics
[R16] RFC-2679 A One-way Delay Metric for IPPM
[R17] RFC-2680 A One-way Packet Loss Metric for IPPM
[R18] PE/IRC/APP/154 E2E Voice Quality User Manual
[R19] TBD 1xRTT RF Acceptance Guidelines
[-] TBD UMTS OCNS settings (not before first UMTS commercial release or
latter)
3. General Methodology
UMTS systems introduce a wide range of applications. Like roads, streets
or buildings are selected and chosen by both parties to be geographically
representative, a set of representative applications and metrics has to be
mutually agreed between the customer and Nortel Networks.
The suggested methodology tries to merge both efficiency and user
experience.
During the same tests, both CS and PS domains will be simultaneously

tested. On CS-domain, voice will be used as an application, when a
mixture of ping and file transfer by FTP will be used on PS-domain.
Only one combined metric (successful call rate) will characterized the
performance acceptance based on a pass-or-fail criterion.
If more in-depth tests are requested, a specific cluster is selected to test
thoroughly the targeted applications. Results are supposed then to be
extrapolated to the entire network, providing the same tuning are
duplicated.
Traffic (voice or data) should be generated by tools for repeatability and
controllability purpose.
All tests must be done in a controlled environment, that means all mobile
devices are strictly controlled by the acceptance team and only the test
equipment is running in the tested areas.
As defined in the standards [R4], the end-to-end QoS is understood as
including all parts of the network. Our goal, as a vendor, is to provide and
commit on performances on the network part we are providing to our
customer. Therefore, we will concentrate as much as we can on the UMTS
Bearer, excluding as long as possible, the external bearer and the TE/MT
local bearer.
As a reminder, the standards (please refer to [R3]) distinguish several
type of information transfer according to three main categories:
connection oriented vs. connectionless
Traffic type:
guaranteed bit rate vs. non guaranteed bit rate
constant bit rate vs. dynamically variable bit rate
real time vs. non real time
Traffic characteristics:
Point to point (uni-directional, bi-directional symmetric, bi-directional
asymmetric)
Point to multipoint (uni-directional multicast, uni-directional broadcast)
For optimization and acceptance purpose we will keep the focus on bidirectional point-to-point connection oriented services.
The same document [R3] defines the following attributes for information
quality:
Maximum delay transfer (end-to-end one-way delay)
Delay variation
Information loss (error ratio: bit error ratio, SDU error ratio, Frame error
ratio depending on which is the most appropriate)
Data Rate
Here are the UMTS Bearer Attributes defined in the standards [R4]:
Traffic Class
Maximum Bit Rate
Guaranteed Bit Rate

Delivery Order
Maximum SDU Size
SDU Format Information
SDU Error Ratio
Residual Bit Error Rate
Delivery of erroneous SDUs
Transfer Delay
Traffic Handling Priority
Allocation/Retention Priority
However, not all of these attributes apply to any traffic class. Here is the
table listing the applicability of the attributes according to the traffic class,
[R5] section 6.4.3.3 table 2 UMTS bearer attributes defined for each
bearer traffic class:
Traffic class
Conversatio
nal class
X
X
X
X
Maximum bit rate

Delivery order
Maximum SDU size
SDU format
information
SDU error ratio
X
Residual bit error ratio X
Delivery of erroneous X
SDUs
Transfer delay
X
Guaranteed bit rate
X
Traffic handling
priority
Allocation/Retention X
priority
Streaming
class
X
X
X
X
Interactive
class
X
X
X
Background
class
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
As far as optimization and acceptance metrics are concerned, we will

focus on information quality measurements.
4. Assumptions
4.1. General Assumptions
This document is written with the following assumptions:
Logging (call trace family) is an RNC feature available and fully
operational at optimization and acceptance time
Hardware equipment and corresponding software tools are available and
fully operational.
For loaded conditions (if applicable), OCNS is a feature available and

fully operational at optimization and acceptance time[1]
4.2. Specific Assumptions

4.2.1. Design criteria
The list of the design criteria must be provided in the acceptance
document. Some of the network performance metrics are influenced by
these criteria (e.g. dropped call rate, call setup access success, blocking
rate).
The list should include at least (but is not limited to):
cell edge reliability
coverage targets (per environment, per associated quality of service)
traffic assumptions (in kbps, per environment, uplink and downlink)
services to be offered (various transmission rates, per environment)
link budgets (including at the very least quality of service thresholds
such as Eb/N0 targets per service or BLER and FER targets; radio margin
assumptions; antenna assumptions)
4.2.2. Performance criteria
A reference should be provided to a table including all performance to be
reached. A threshold shall be provided for each of the metrics and is to be
met during the acceptance.
Values can be divided into two main groups: mainly-equipment-driven
values (such as call setup time, transfer delay) and mainly-design-driven
values (such as dropped call rate, call setup success rate). Please refer to
[R1] for the former and to the specific contract design assumptions for the
latter.
These values have to be mutually agreed by the customer and Nortel
Networks and must result from the design criteria and the network
element known or planned performance.
4.3. Warnings on metric definitions
The metric definitions in 14.2 (page 38) do not fully take into account the
measurement methodology.
Some of the messages may be only accessible through test mobiles or
logging features.
Counters may not be available or may not have a granularity accurate
enough to perform the desired calculations. In lots of cases, a context of
communication must be considered to link the different messages and
itemized the metrics according to causes that are notified in other
signaling messages during the call.

All metrics are intended to be calculated on a network-wide basis or at the
very least on a per-cluster basis.
4.4. Notation
In the following formulas, the messages are generally written in italic and
with the following notation:
ProtocolAbbreviation_MessageName(Cause)
The abbreviations are detailed in the Abbreviations and Definitions
chapter, page 32. The message name is the one used in the
corresponding standard. The cause is generally used to indicate a group of
causes that should be used to filter the related messages. The cause does
not correspond necessarily to a cause described in the standards
(explanations are given as far as possible to relate to the causes
described in the standards).
5. Test Areas
The coverage area for test consists of drive-test routes, in-building and
special test areas within the designed service area. The set of driven
roads, in-building measurement areas and special test areas shall be
mutually agreed between the customer and Nortel Networks.
5.1. Routes
The drive routes shall cover the areas of interest and include some
primary as well as secondary roads. In addition, routes shall be chosen to
ensure a sampling of the main encountered environment in the designed
service area (such as dense urban, urban, suburban and rural
morphologies).
During data collection, the test routes shall be driven at speeds as being
representative of normal subscriber behavior. To insure repetitive results,
speed shall be carefully checked all along the drive tests.
Should the acceptance be done with different performance thresholds
(such as high requirements in dense urban areas, medium requirements in
suburban areas and low requirements in rural areas), the routes will be
carefully designed to keep each drive-test in an area of one and only one
type of coverage requirements (e.g. proscribe a drive overlapping dense
urban and suburban in the previous example)
5.2. In-building Test Areas

As far as possible, in-building coverage from external site infrastructure
should not be tested.
However, building penetration margins specified in the design may be
verified at street level using attenuators at the mobile (only RSSI would be
checked since the Ec/N0 will stay unchanged due to the fact that it is a
ratio).
Specific buildings requiring indoor testing shall be clearly defined (building
location, service areas and test areas within the building, interaction with
external coverage). The customer shall provide access to Nortel Networks
to all defined test areas within these buildings.
5.3. Special Test Areas
Special test areas (such as trains, subways, water areas, etc.) may be
included in the performance acceptance. A specific process will be
designed for such areas in agreement between the customer and Nortel
Networks.
5.4. Warranty Zone
Maps defining accurately the warranty zone have to be provided and
appended to the Performance Acceptance Test Plan.
The tests will be confined in this zone and any measurement coming out
of this warranty zone shall be discarded.
The warranty zone shall be determined from predictions coming from
Nortel Networks prediction tool (iPlanner) and shall be based on real site
coverage (this means that all sites shall be CW-drive tested)[2]. Typically,
the warranty zone will be defined from the P-CPICH EC/I0 (downlink) and
the uplink budget.
As a definition, the warranty area is the design coverage area diminished
by deployment restrictions, exclusions and specific non-warranty areas
(e.g. tunnels, specific bridges...). In case some of the sites included in the
warranty area were not up and running at the beginning of the first drive
test, then the warranty zone would have to be reviewed to include only
up-and-running sites.
6. Simulated Traffic Load (if applicable)[3]

The RF performance of the UMTS network should be evaluated with mobile
units that are operating under nominal loading conditions. For
performance acceptance, tests may be conducted with simulated traffic
load in complement of the unloaded tests. This traffic load shall

correspond to the number of Erlangs and kbps for which the system has
been designed as defined in the Final Cell Plan. The general method of
loading shall be as follows.
6.1. Uplink
Uplink load simulation shall be conducted by using a fixed attenuator
(most likely 3 dB, but not to exceed 5.2 dB) inserted in the transmit path
of the mobile unit. The exact value of the attenuator shall be consistent
with the equivalent noise rise at the cell site receiver generated by the
traffic load (at the busy hour) used in the design assumptions. The
equivalent noise rise (interference margin) shall be determined by
expressing the number of channels n associated with the traffic load as a
fraction of the pole point P:
n = number of channels
P = sector pole point
r = n / P = fractional loading with respect to the pole point
The design value of r shall be 50 % by default (equivalent noise rise of 3
dB). It shall not exceed 70 % in any case (equivalent noise rise of 5.23
dB).
During load simulation testing, the noise floor rise should be measured to
verify the 3 dB theoretical value (at r = 50 % load).
It has to be noted that this setting is pessimistic. Actually, the probability
of several overlapping cells to be altogether at their maximum load is
close to zero in normal conditions (unsaturated network). Therefore, the
above settings should be corrected by a factor allowing taking into
account a given amount of load balancing between cells. However, the
amount of this factor is FFS.
6.2. Downlink
Downlink load simulation shall be conducted by using Orthogonal Channel
Noise Simulation (OCNS). This feature should be part of Nortel Networks
UMTS RNS[4] and will be used to broadcast downlink orthogonal
interference of appropriate level. This interference shall be consistent with
the traffic load (at the busy hour) defined in the Final Cell Plan. The
number, level and channel activity of interfering links shall be defined in a
mutual agreement between the customer and Nortel Networks.
OCNS settings will be detailed in a referenced document [FFS].
7. Test Setup
All tests shall be performed using characterized and calibrated test
mobiles that meet or exceed the minimum performance specifications as
specified in [R2] and [R6].
Test mobiles shall have fixed attenuators connected between the
transceiver and the antenna to compensate for penetration loss,
additional vehicle height, and performance of the test vehicle antenna as
appropriate (it assumes that duplexers are used to separate uplink
transmission and downlink reception). Additional attenuation may also be
added to simulate uplink system load (if applicable) and/or building
penetration loss (indoor coverage from outdoor sites).
This assumes to have adequate packaging: screened boxes providing
radio isolation at the attenuators need to be used with such a test set.
The required test equipment list is detailed in Appendix 1: Test Equipment
List, page 36.
8. Performance Metrics
This section describes the test procedures to be used to determine the
system performance metrics as defined between the customer and Nortel
Networks for the performance acceptance.
All tests should be completed in a totally controlled environment under
unloaded conditions.
Depending on the feature availability and the load and/or ramp-up load of
the network, tests under loaded conditions might be performed according
to the design-assumed load (by default 50% loading relatively to the pole
capacity).
All metrics will be measured along the drive test routes and in the specific
test areas.
The preferred way for simplifying acceptance tests is to focus on only one
combined metric linked to a simple pass-or-fail criterion.
8.1. Combined Metric: Successful Call Rate

Objective
The suggested method for acceptance is stating a pass-or-fail criterion
based on a unique successful call metric. The successful call metric
actually combined several of the typical metrics (see section 8.2 page 16).
Definition
For a call to be successful, it must correctly setup within a given time, hold
during the average holding time (AHT), if necessary successfully handover (within the 3G coverage), satisfy all defined quality criteria and
correctly release the resources.
The successful call rate is defined as the ratio between the number of
successful calls versus the total number of attempts.
As per this definition, the successful call rate metric combined the call
setup success rate, the call setup time, the dropped call rate, the quality
thresholds, the successful handover rate and the successful release rate.
It has to be emphasized that this metric should be applied to the 3G
coverage only and inter-RAT hard handover must not be part of this
metric.
Some examples for target values and quality thresholds for this metric can
be found in section 14.1, page 37.
Measurement Method
Calls (either voice calls or data calls) will be chained during the entire test.
To have enough samples, it is recommended to have more than one
mobile per tested application (e.g. two mobiles for voice calls, two mobiles
for data calls).
To get closer to a one-shot test, mobile-originated and mobileterminated calls could be mixed during the test (with a proportion to be
agreed between the customer and Nortel Networks, e.g. 50% mobileoriginated, 50% mobile terminated). In this case, the impact of paging
(paging success rate) must be considered in the target values (either a
longer setup time or a lower success rate).
Depending on the measurement tool capabilities, mobile-terminated and
mobile-originated calls could be physically passed on different physical
sets (e.g. one mobile for voice mobile-terminated calls, one mobile for
voice mobile-originated calls).
For each call, the post-processing tool should come with a binary result
pass or fail. A call passes if it is successful, as per the definition.
The final percentage of success call rate is then computed by dividing the
number of successful call by the total number of call attempts.
8.2. Detailed Metrics

On customers request, detailed metrics may be preferred by the
customer to the successful call rate.
In that case, the metrics that can be measured for acceptance are:
Dropped Call Rate
Call Setup Success Rate (originations)
Call Setup Time (originations and terminations)
Paging success rate (it might be combined with call setup success rates in
a call setup success rate (terminations) metric)
Bit Rate (minimum, average, maximum; both ways)
File Transfer Time (one-way, measured on both ways)
Voice Quality (typically MOS- or DMOS-like tests)
For tests under loaded conditions, the following metrics may be added to
the previous one:
Blocking (radio)
RAB Unavailability Rate
Please refer to section 14.2, page 38, for detailed descriptions of each
metric including metrics used for optimization but not for acceptance.
9. System Optimization process

The network optimization consists of four different stages, as described
hereafter.
Configuration Audit
Configuration audits are the preparatory work that has to be done before
the sites are on air. This includes system-wide activities and tests
performed on each base station (noise floor measurements, installation
inspections, etc).
Some or all of the work might have been already done during the
installation, commission and hardware acceptance.
Shakedown
Shakedown tests are done to verify the basic functionality of the RNS,
rather than performance. Transmit power, code allocation, RRM algorithm
parameters and basic call processing functions are tested on a per-sector
basis. Shakedown tests are completed when correct codes are set up,
power levels are measured, basic call setup and softer handoffs are
successfully executed during drive tests around the site.
Cluster Optimization
Cluster optimization provides the first pass of optimiza, tion and will adjust
settings for parameters, antenna tilts and antenna orientations where
required. It is estimated to take between 1 and 1.5 days per site for one
drive-test team[5] as a total across all rounds, depending on the number
of problems encountered (i.e. approximately 6 weeks per cluster of
approximately 30 cells with one drive-test team, all rounds included). For
more information about planning and resource estimation, please see
appendix 6, page 38.
System Optimization (Control Route Analysis)
System optimization along the control routes is intended to further tune
the system in preparation for commercial clients. In large markets, some
of the system optimization activities will be performed in parallel with the
optimization of remaining clusters. Tests will use routes over multiple
clusters.
Hereafter is a chart describing the process:
9.1. Configuration Audit

Each of the following criteria must be met prior to advancing to the next
phase of testing.
Core Network Elements and Access Network Elements
The NEs (Node-Bs, RNCs, WGs, MSCs, GGSNs and other core network
elements) should have passed the functional acceptance (all features
tested and accepted). Further testing might occur in the extent that they
are not interfering with performance optimization tests.
Additionally, network testing and inter-working must have progressed to
the point that basic call routing (voice and data) has been validated (i.e.
local calls can be completed) for each of the NEs.
Cell Site Information
The plan must identify the date that each cell site will be installed and
integrated. To the possible extent, the schedule for node-Bs installation
and integration should be based on the clusters defined in the RF plan.
Spectrum Clearance
The spectrum should be cleared on all the service area. Noise floor
measurements over the whole channel bandwidth for all the used carriers
should reach at least the minimum requirements (expected values for the
maximum noise power allowed in the channel bandwidth would be

typically around 105 dBm). A plot from at least 30% of all the sites in the
system (evenly geographically distributed), indicating the noise floor of
the system, should be provided.
This step is not correlated with Node-B installation. The measurement is
done with a spectrum analyzer (on uplink and downlink bands) and does
not need to have the Node-B equipment on site.
The noise floor can be also checked with a drive test receiver (such as the
one from Agilent referenced in 13, page 36)
RF Design Information
Details of the RF design must be provided including:
geographical maps from the RF design tool with site positions
predictive coverage plots (Ec/N0, downlink-uplink balance, uplink (mobile
transmitted power) and number of pilots)
predicted soft/softer handoff plots
CW drive test data
On-the-field RF data: as-built antenna heights (at the radiating center),
as-built antenna azimuths, as-built antenna tilts.
Antenna Sweep Tests
Antenna sweep data shall be provided for each cell.
Construction Issue List
An issue list should be provided for each site (issues such as bad antenna
orientation, bad tilts, cable problems, wrong connections...) as well as the
attached plan to solve them.
Directions to each site
Written directions to the location of each site in the system should be
provided (maps are welcomed to pinpoint the location).
Cell Database and Datafill
A complete information set (soft copy, DRF format) should be provided for
each cell:
database information with coordinates (longitude, latitude), number of
antennas (use of diversity or not), antenna type, antenna model, tilt,
construction drawing, cable lengths, equipment configuration; this
information should be itemized on a per sector basis when needed (such
as antenna height, azimuth, scrambling code, antenna type, model, tilts...)
datafill information: all parameter values from the in-service MIB.
All the appropriate information stated above is to be provided as a soft
copy prior to the Shakedown phase of the project. The information should
be gathered and placed in a binder for each site identified in the system.
For border cells (cells in more than one cluster) the information can be
photocopied and placed in appropriate binders.
9.2. Cell Shakedown

9.2.1. Objective
The objective of the cell shakedown is to ensure proper cell site operation
in the integrated system. Proper operation is verified through drive tests
around the site and evaluation of antenna orientation, cable
interconnecting and correct code assignments.
9.2.2. Entrance Criteria
Each of the following criteria must be met on each cell of the cluster prior
to starting cell verification tests.
1. Antenna sweep tests must be successfully completed. The associated
sweep test acceptance forms shall be supplied for the cell site. These
tests will be useful in verifying that the cells are properly cabled and are
ready to support RF.
2. Integration tests must be passed and accepted. Both customer and
Nortel Networks representatives must have accepted the hardware and
the connectivity. Integration tests are used to show construction
completion and basic network readiness.
3. RNS, NSS and WG datafill parameters must be verified with accuracy.
4. Cleared spectrum.
9.2.3. RF BASIC checking
Receiving Antenna Cabling
The transmission and especially reception cabling have to be checked. It
should have been done during I&C (Installation and Commissioning). If not
or if unsure, the cabling checking has to be part of these tests.
A given power (TBD) shall be transmitted in front of the receiving
antennas (main and diversity) and the received power shall be checked at
the Node-B, compared to the noise level recorded when the reception is
idle (no signal transmitted). This is assuming that the power coming from
each antenna can be read independently at the Node-B or at the OMC-B.
Effective Transmit Antenna Orientation

EC/N0 by code is analyzed to determine if all antennas are pointing in the
right directions. If an antenna is found to be incorrectly installed, then a
work order will have to be issued as soon as possible to fix the problem.
9.2.4. Drive Routes

The drive routes for shakedown are defined as circles around the cell at
approximately 30 percent of the expected cell coverage area.
9.2.5. Test Drives

These tests are performed by driving circular routes around the base
station.
Purpose: to test call setup in each cell, handoffs (softer) between cells
and to verify antenna orientation, Primary Pilot EC/N0 (PCPICH EC/N0),
scrambling code allocation for each sector according to the RF design and
neighbor list coherence with engineering requirements. UE transmitted
power will be analyzed to double-check possible reception cabling
problems.
Call type: voice call; file transfer (FTP).
Load: unloaded
Data collected: PCPICH EC/N0, UE transmitted power, DL UTRA carrier
RSSI, DL transmitted code power and UL RTWP.
9.2.6. Test Areas (indoor Site)
These tests are performed in selected indoor areas where coverage is
provided by a specific indoor site.
Purpose: to test indoor handoffs (softer, possibly soft and hard) between
cells, if applicable; to test handoffs (soft and possibly hard) between
indoor cells and outdoor cells, if applicable; verify antenna orientations,
PCPICH EC/N0, and code assignment according to the RF design.
Call type: voice call; FTP.
Load: unloaded
Data collected: PCPICH EC/N0, UE transmitted power, DL UTRA carrier
RSSI, DL transmitted code power and UL RTWP.
9.2.7. Exit Criteria
Cell testing is completed when correct code and reasonable power levels
are measured, and softer handoffs are successfully executed.
Calls on both domains (Circuit Switched and Packet Switched) must have
been completed successfully on the site.
9.2.8. Issue List

There should be no issue list after shakedown tests. All problems must
have been addressed and resolved for exiting this phase.
However if issues are remaining, the warranty area might be revised
before starting the next phase by excluding the areas affected by the
remaining issues. The purpose is to have a warranty area compliant with
the exit criteria, i.e. without remaining issues.
9.3. Cluster Optimization

9.3.1. Objective
The objective of cluster optimization is to perform RF design verification,
identify and categorize network or coverage problems, optimize for best
performances according to the acceptance criteria and serve as a first
pass of problem resolution.
Satisfactory system performance is quantified by the successful call rate
metric being no less than 50 % more failure than needed to reach the
agreed system-wide threshold.
Example: if the system-wide threshold is 95 % of successfull calls, each
cluster would be required to reach at least 92.5% successfull calls.
If detailed metric acceptance must be done, satisfactory system
performance is quantified by the agreed metric values within their target
thresholds. The system performance metrics need not to be met to exit
cluster optimization but should be close enough to the targets to ensure
acceptance during the system optimization stage. Areas not meeting
system performance must be included in the issue list.
Due to the high number of possible mix of applications, it is not possible
practically to test thoroughly all clusters with all applications.
Each of the following criteria must be met prior to starting optimization on
each cluster:
All cell sites in the cluster must be integrated and accepted.
If one or more sites are not ready at cluster optimization time, then the
cluster must be redefined in order to exclude these sites (see impact on
the warranty area too, page 12).
Shakedown tests must have been completed on all cell sites in the
cluster.
All cell sites must have cleared the shakedown test exit criteria.
The cluster must not have any UMTS mobiles operating other than those
from the optimization team (i.e. controlled environment).
System performance acceptance criteria are clearly defined and agreed
by both the customer and Nortel Networks.
System performance acceptance deliverables are defined and agreed by
both the customer and Nortel Networks.
Predicted coverage maps per cell are available.
Databases of system design and propagation tool reflect accurately the
system configuration.
The warranty area is clearly defined and agreed by both the customer
and Nortel Networks. Maps of the warranty area are issued.
If applicable, OCNS settings have been set up and agreed by both the
customer and Nortel Networks.
Maps of drive test routes and specific data collection spots (including
indoor sites) have been reviewed and mutually agreed by both the
customer and Nortel Networks.
The network is locked from any hardware or software changes, except for
maintenance purpose (e.g. hardware failure, critical software bug...).
All logistical hardware and software for performance acceptance
processing are on site and ready to work (e.g. calibration hardware and
software, coordination procedures...)
9.3.3. Drive Routes
Engineering judgment should be used to determine these routes.
Marketing concerns and design issues should also be considered when
defining the various drive test routes.
Drive test routes must also be designed according to the threshold
definitions: if several threshold levels have been defined, then each drive
test must be contained in areas where only one threshold set applies.
Example: Three different threshold levels are defined depending on
morphologies: one set for dense urban, one for suburban and one for
rural. Then, drive test routes shall be designed exclusively within dense
urban areas or within suburban areas or within rural areas.
On average, clusters should comprise around 30 cells. Drive routes for
each cluster will be mutually agreed by the customer and Nortel Networks
and will be chosen as follows:
Optimization routes
Optimization routes will be selected from primary (e.g. highways) and
secondary roads (e.g. city streets of more difficult morphology) within the
cluster. In some instance, the engineering team may determine that
additional roads should be included in the cluster to ensure adequate
performance of critical service areas that are not included in the primary
and secondary road classifications. These routes should encompass all

land classifications (morphologies) and areas of engineering concern and
should pass through all inward-facing sectors in the cluster. These drive
test routes must be mutually agreed to between the customer and Nortel
Networks. Drive test routes will not be defined through known bad
coverage areas in the test area (i.e. areas with missing cells or areas not
planned to be covered). It should take approximately five to ten hours to
drive all optimization routes within a typical cluster.
Control Routes
Control routes will be used for demonstrating system performance on an
individual cluster basis and on a system wide basis. These routes are used
primarily for performance progress monitoring and should reflect typical
network performance.
The control routes should be a subset of the optimization routes and
should require no more than 3 hours of driving for a 30-cell cluster. These
routes should also encompass all roads of primary concern to the
engineering staff and cover all land classifications defined within the
cluster. When possible, control routes should be chosen to be contiguous
with those from neighboring clusters. Control routes are extended as each
cluster is exited.
Note: time of drives may vary since the time is dependant on the size of
the cluster areas and may vary depending on how close the cells are
spaced apart.
9.3.4. Test Drives
To simplify the process and be closer to the user experience, only one type
of test setup will be done for all drive tests. However, iterations of these
drive tests have to be planned to improve the network; these iterations
are necessary and mainly due to the dynamic aspect of the CDMA
technology. Each iteration will have a different purpose.
Each drive tests (except the first one, pilot optimization drive) will get at
least four mobiles, two dedicated to voice calls, and two dedicated to data
calls.
The tests shall follow the following processes during each applicable drive:
MO
Setup the call
Hold the call during the holding time
[ping RTT, file transfer time]
Release the resources
[when holding time is off]
MO only
MO
Setup the call
Hold the call during the holding time
[MOS]
Release the resources
[when holding time is off]
MOMT
PS data calls
CS data calls
PS calls will follow a test pattern composed of initial ping, subsequent

pings, file transfer by FTP (either uplink and/or downlink), idle states (to
activate the downsize/upsize feature, also known as always-on).
Transferred files must be sized according to RAB rates, holding time and
test patterns.
The optimization engineers shall use their judgment in deciding whether
or not all test drive iterations are required and how often and in what
order each drive test needs to be performed.
In the same spirit, several hereafter proposed drive tests can be grouped
in the same run (e.g. pilot optimization and radio optimization, grouping
the scrambling code analyzer with test mobiles)
Note: RNC logging must be set up for all drives, except pilot optimization
drive.
It has to be emphasized that the following number of drive tests is higher
than it should be with a mature technology using as much as possible
remote optimization techniques and tools. This shall be updated and
integrated in the current process when these techniques and tools are
available.
Pilot Optimization Drive
Route: Optimization routes.
Load: unloaded
Purpose: to determine the actual pilot coverage for each cluster and
solve the main RF propagation issues (pilot shoot-up, pilot pollution,
scrambling code overlap...)
Equipment: Pilot scanner (e.g. Agilent E7476A).
Applications: none.
Data collected: Scrambling code analysis (Ec/N0, Delay Spread, Time),
top-N analysis with a window at 20.
Analyses: Per-PCPICH coverage; Best PCPICH coverage; 2nd, 3rd, 4th, 5th, 6th
PCPICH coverage; Number of PCPICH over a given Ec/N0 (typically 12dB in

IS-95 networks)[6].
Radio optimization DRIVE
Routes: Optimization routes.
Load: unloaded
Purpose: Mainly insure the RF coverage control and RF optimization
(antenna azimuths, tilts, power settings, neighbor list tuning)
Process: each call is setup, hold during the agreed holding time and
released; mobile originating and mobile terminating for voice; mobile
originating only for data.
Applications: voice, combined ping and FTP.
Data collected: Ec/N0, UE transmitted power, DL UTRA carrier RSSI, DL
transmitted code power, UL RTWP, Eb/N0, average number of radio links,
number of dropped calls, voice quality, file transfer time.
Metrics calculated: dropped calls, bit rates (FTP only), voice quality.
Loaded Uu Optimization Drive (If applicable)[7]
This test drive is performed on the optimization routes.
Load: simulated load as per page 13 (only if OCNS is available); unloaded
otherwise and only if substantial RF changes have been made.
Purpose: RF optimization under load
Applications: voice, combined ping and FTP.
number of dropped calls, voice quality, file transfer times, ping RTTs.
Metrics calculated: dropped callsbit rates (FTP only), transfer delay (FTP
only).
Optimization Check and Fine Tuning Drive
This test drive is performed on the optimization routes.
Load: unloaded; if the loaded UU optimization drive has been done, then
simulated load as per page 13.
Purpose: Check and fine-tune the optimization results.
Applications: voice (originated and terminated); combined ping and FTP.
number of dropped calls, voice quality, file transfer times, ping RTTs, call
setup success or failures, call setup times.
Metrics calculated: successful call rate (pass-or-fail), dropped call rate,
bit rates (FTP only), call setup success rate, paging success rate.
Acceptance Data collection and Final Cluster Check DRIVE

This test drive is performed on the control routes.
Load: unloaded; if drive #2 was done under loaded conditions, then
simulated load as per page 13.
Purpose: Check and collect acceptance measurement for this cluster.
Applications: voice (originated and terminated); combined ping and FTP.
number of dropped calls, voice quality, file transfer times, ping RTTs, call
setup success or failures, call setup times.
Metrics calculated: successful call rate (pass-or-fail), dropped call rate,
bit rates (FTP only), call setup success rate, paging success rate.
Supplementary drive tests might be needed if additional tests are required
(such as IMSI attach (resp. detach) success rate, IMSI attach (resp. detach)
time, location update / registration success rate, and so on)
9.3.5. Data Analysis / Problem Resolution
Following are the basic parameters adjusted during cluster optimization.
1. Neighbor list optimization (based on unloaded lists)
correct neighbor list entries (minimize neighbor list length)
ensure proper prioritization of entries
This can be done with a pilot scanner drive test system and the correct
processing tool (see page 36)
2. Coverage problems (can include holes, failures due to multiple or
rapidly rising pilots)
adjust antenna azimuths, heights and types
downtilt / uptilt antennas
decrease Node-B power to create dominant server and reduce pilot
pollution[8]
Antenna type and azimuth changes must be approved by local RF
engineer.
3. Network timer checking and tuning.
4. Handoff parameters checking and tuning

In order to exit each cluster, the successful call rate results[9] must be
close enough[10] to the agreed thresholds (see examples in section 14.1,
page 37) or the corresponding cluster must be put on the issue list.
Cluster optimization sign-off will be performed by the regional customer
manager or representative.
A sign-off form with the issue list must be created and agreed to with the
customer.
9.3.7. Reports
Following is a list of reports available to the customer. The Performance
Result Report and issue list will be provided for cluster exit on the control
routes of each cluster. The other data will be available for review during
optimization and provided to the customer upon network performance
acceptance.
1. Performance Result Report
Drive data information
Geographical information
Successful call rate results.
2. Performance Measurement Report (informative only)
Maps of recorded values
3. Test Environment Report
List of all cell parameters when exiting from a cluster
List of test equipment
Load simulation (OCNS) details (one time and if applicable)
4. Issue list
9.3.8. Issue List
The issue list shall contain at a minimum:
complete list of problems
opening date for each item
owners and dates of resolution
Depending on the project organization, one global issue list or one issue
list by regional entity might be created.
9.4. System Optimization

9.4.1. Objective
The objective of system optimization (or control route analysis) is to
optimize the sum of all clusters covering an entire region. A region can be
defined along administrative borders or as an economical entity but in any
case, the meaning of region has to be clearly defined between the
customer and Nortel Networks. The system performance optimization
starts with two optimized clusters and ends with all the clusters in the
region. The entire region is fine-tuned with remaining and new issues
resolved. In addition, action plans for future work are defined. The system
performance metrics are calculated for the entire region and serve as the
warranted values as per the agreement between the customer and Nortel
Networks.
All clusters to be optimized must have met the cluster exit criteria or
signed off by the regional RF manager or designate with mutually agreed
issue list.
9.4.3. Drive Routes
System optimization is typically performed on the control routes.
However, additional optimization drive test routes may be defined as
required for diagnostic testing of new problem areas that are identified as
a result of new clusters being switched on.
9.4.4. Drive TESTs
Each of the five drive tests used in Cluster Optimization (page 23) can be
used by the optimization engineers as required. RNC logging must be set
up for all drives, except the pilot optimization drive.
The cluster optimizations should have brought the system to a level where
most of the optimization problems are already solved.
Radio optimization checking
Hopefully, the first drive test (done with coupling test mobiles and pilot
scanner, equivalent to pilot optimization drive and radio optimization drive
merged) will show very few optimization settings to change.
Optional optimization drives

After possible changes in parameter values for optimization, another drive
might be necessary to capture the new tuning and check the resulting
improvement.
Acceptance data collection
The next drive shall then be used to collect acceptance data for checking
if the successful call rate metric is meeting the acceptance criteria.
9.4.5. Data Analysis / Problem Resolution
Since the System Optimization is based on a network composed of already
tuned clusters, most of the following parameters should have been
optimized:
total cell power and P-CPICH power (typically the ratio between both is
kept constant)
antenna azimuths, tilts, heights and types
Due to the cross-effects between clusters, handoff parameters, neighbor
list contents and neighbor prioritization might need changes at borders
between clusters. This is also a time that can be used to finish up the
tuning of difficult spots.
In order to exit each region, the system performance metrics must meet
the threshold requirements.
System Optimization sign-off will be performed by the regional RF
manager or representative. A sign-off form with the issue list must be
created and agreed to with both parties.
9.4.7. Reports
Interim reports
During the System Optimization phase, Nortel Networks will be preparing
Control Routes Testing reports as permitted by time. These should include:
Updated issue lists
Successful call rate results
Final antenna tilts and azimuths
Final deliverables
At the end of the System Optimization tests, Nortel Networks must submit
the following documentation:
1. Performance Result Report
Drive data information
Geodetic information
Successful call rate results
2. Performance Measurement Report
Maps of recorded values
3. Test Environment Report
List of all cell parameters as final network datafill.
Load simulation (OCNS) details (if applicable)
4. Issue list
9.4.8. Issue List
The issue list shall contain as a minimum:
complete list of problems
owners and dates of resolution for issues in the process of being solved
new or remaining untreated issues
Depending of the organization of the project, the issue list might be global
or there might be one issue list per market.
10. Reports
Following are the details of the various reports that are generated
throughout the optimization process.
10.1. Performance Result Report
The Performance Result Report will include measurements that can be
used to determine the overall quality of system performance and whether
the measured network meets, or is trending towards meeting the agreed
system performance metric values. The data collected for these reports
will be from control and/or optimization routes.
The first part of the Performance Result Report will provide the following
information for the latest set of data collected and reported:
Date of report and name of person who completed the report.
Number of cells included vs. the total for launch in the region. The
number of cells currently undergoing optimization should also be included,
as well as a list of these cells with their references.
Test setups[11] when collecting the data.
Version of the software loads and other equipment-related information.
Outstanding actions that will improve or affect overall performance of
the area with target dates.
10.1.1. Geodetic Information
The Performance Result Report should include geographical information in
the form of a map showing the control routes as well as areas of
acceptable or not acceptable quality in addition to areas not evaluated.
The objective of the geodetic information is to show the extent of the area
for which data has been collected and to ensure that the coverage is
maintained as the system is optimized. The following information needs to
be displayed on the map:
site locations
area tested, including drive test routes, where cluster optimization is
underway and where cluster optimization is complete
control routes
areas/places of coverage problems
any relevant information that helps interpreting the results
10.1.2. Key Performance Indicators (KPIs) Reports
The KPI graphs or maps will show both uplink and downlink results after
discarding the (1-r)% of the bins having the worst measurement results (r
% coverage reliability).
The report should also include information on the length and duration of
drive routes used to collect the latest displayed results and any pertinent
information on events.
The report should also include information on the number of calls of each
type made for the latest displayed results and any pertinent information
on events that have influenced the measurements.
Where failures are believed to be caused by issues other than RF, e.g.
handset problems or network failures, this information should be
indicated.
10.2. Performance Measurement Reports
Measurement scope is wider than the ones done for System Performance
Metric (successful call rate) calculations. Supplemental measurements are
used for optimization and especially for troubleshooting. They are mainly
collected in areas where system performance objectives are not met.
The performance measurement reports are gathering these metrics used
for optimization.
Drive test data may include the following measurements:
Dropped calls and unsuccessful call attempt plots, counts and
percentages
BLER plots
Ec/N0 plots
Eb/N0 plots
Mobile Transmit Power plots
Mobile Receive Power plots
Number of Radio Link plots (handoff state)
10.3. Test Environment Report

This report is gathering all information on the test environment:
List of test equipment, including test mobiles.
Equipment setting details.
List of all cell parameters with their final values.
Antenna tilts and azimuths
If applicable, load simulation details (including OCNS settings).
It may also include details on issues that affected the network during
performance acceptance measurements.
10.4. Issue List

One global Issue List will be kept by region. It will begin with the first
cluster and grow as new clusters are added. Some of the problems
identified on the issue list may not be regional but rather national
problems. National problems can be tracked down on a project basis.
Each
1=
2=
3=
issue will be assigned a priority:

currently impacting optimization work
must be resolved prior to commercial launch
other issues
11. OCNS unavailability
At the time this document is written, the UMTS OCNS feature is not
scheduled to be implemented before the second commercial UTMS
release, at the best.
Therefore, number of network performance acceptances will have to be
done without this feature. This means that there is no way to do the tests
under loaded conditions. In this case, all the tests will be kept under
unloaded conditions.
Assumption: the acceptance without OCNS will be done on early
networks (opening before the first General Availability version or soon
after). The traffic load of such networks when opening commercially will
be probably very light, due also to the tension on the mobile device
market. It is reasonable to do an acceptance under unloaded conditions
and a close follow-up of the launched network to optimize the parameters
as the traffic is ramping up.
Depending on the contract, optimization services have to be included or
proposed to the customer to assist its team in handling the increasing
load during the first stage of the network. According to the experience on
previous CDMA networks, it is critical to react quickly to adjust the right
parameters to avoid brutal service degradations.
12. Abbreviations and definitions
12.1. Abbreviations
ABR
AHT
BBLER
CCAIT
CBR
CC
CPICH
CRE
CRT
CS
CSSR
CST
CW
DCR
DL
DRF
Dst
FDD
FER
FFS
Available Bit Rate

Average Holding Time
Background (traffic class)
BLock Error Rate
Conversational Real Time (traffic class)
CDMA Air Interface Tester (Qualcomm)
Constant Bit Rate
Call Control (see [R7] for details)
Common PIlot CHannel
Cell Reference Event
Conversational / Real Time
Circuit Switched
Call Setup Success Rate
Call Setup Time
Continuous Wave
Dropped Call Rate
DownLink
Data Request Form
Destination (IP address of the destination source)
Frequency Division Duplex
Frame Erasure Rate
For Further Study
FQI
GGSN
GoS
HHO
IIE
IPPM
ISO
kbps
KPI
LA
mCR
MIB
MO
MP
MT
NAS
NB
NE
NSS
OCNS
OSI
OVSF
PCPICH
PCR
PDU
PS
PSR
QoS
RA
RAB
RANAP
RAT
RFC
RL
RNC
RNS
RR
RRC
RRM
RSSI
RTT
RTWP
SSAP
SCR
SDU
SDUER
SHO
Frame Quality Indicator

Gateway GPRS Support Node
Grade of Service
Hard HandOver
Interactive (traffic class)
Information Element
Internet Protocol Performance Metrics
International Standard Organization
kilobit per second (1000 bits per second)
Key Performance Indicator
Location Area
Minimum Cell Rate
Management Information Base
Mobile Originated
Measurement Point
Mobile Terminated
Non Access Stratum
Node-B
Network Element
Network Sub-System
Orthogonal Channel Noise Simulation
Open System Interconnect
Orthogonal Variable Spreading Factor
Primary Common PIlot CHannel
Peak Cell Rate
Protocol Data Unit
Packet Switched
Paging Success Rate
Quality of Service
Registration Area
Radio Access Bearer
Radio Access Network Application Part
Radio Access Technology
Request For Comments
Radio Link
Radio Network Controller
Radio Network System
Radio Resource Management
Radio Resource Control
Radio Resource Management
Received Signal Strength Indicator
Round Trip Time
Received Total Wideband Power
Streaming (traffic class)
Service Access Point
Sustainable Cell Rate
Service Data Unit
Service Data Unit Error Rate
Soft HandOff
SM
Src
SSHO
Str
TBC
TBD
TBR
TD
TDD
TE/MT
TrCH
Type-P
packet
UBR
UE
UL
URA
UTRAN
VBR
WG
Session Management (see [R3] for details)

Source (IP address of the source host)
Softer HandOff
Streaming (traffic class)
To Be Confirmed
To Be Defined
To Be Referenced
Transfer Delay
Time Division Duplex
Terminal Equipment / Mobile Terminal (parts of the UE)
Transport CHannel
Allows defining metrics and methodologies generically;
further on, type P might be explicitely defined (i.e., exactly
what type of packet is meant), partially defined (e.g., with a
payload of B octets), or left generic. For more details, please
look at RFC-2330 [R15]
Unspecified Bit Rate
User Equipment
UpLink
UTRAN Registration Area
Universal Terrestrial Radio Access Network
Variable Bit Rate
Wireless Gateway
12.2. definitions
Area availability
Average Holding
Time
B, KB, b, kb, Mb
Bin size
Call
area availability refers to the percentage of

geographical bins over the specified coverage area
that meets the system performance objective as
agreed while compensating for the appropriate
penetration margin. That is, if any geographical bin
within the specified coverage area fails to meet the KPI
objective while compensating for the appropriate
penetration margin, then a service problem exists.
Average time during which the call is sustained. This
duration is typically used to define the call duration for
the acceptance tests. Typical AHT is around 2 min.
B = Byte = 8 bits
KB = Kilobyte = 1,024 bytes = 8192 bits
b = bit
kb = kilobit = 1,000 bits
Mb = megabit = 1,000,000 bits
geographic bins are used to calculate area availability.
The bin size is 200m x 200m (or otherwise mutually
agreed to)
Successfully established instantiation of a service that
is provided through the network to a given user.
Cluster
Cluster exit data
Controlled
environment
Drive Route
Ec/N0
Final Cell Plan

Issue list
KPI
QoS
Reported set
Service Area
Test Area
Warranty Area
Typically around 30 sites that serve a contiguous

geographic area
The final set of data collected for a cluster that
demonstrates the overall performance of the cluster
along the specified drive routes
Area with clear spectrum (verifying the requirements
on noise floor level) and where all working mobiles are
test mobiles (no other mobile should be able to roam in
the area)
A segment of the major roads within a cluster that will
be targeted for optimization. During data collection,
the test routes shall be driven at speeds mutually
agreed to as being representative of normal subscriber
behavior. Drive routes will be classified as follows:
Optimization Routes: drive routes selected to
encompass all morphology classifications and areas of
engineering concern defined in a cluster.
Control Routes: a subset of the optimization routes
that include the roads of principle concern for the
regional engineering staff and cover most land
classifications within the cluster.
CPICH EC/N0 as defined in the standard [R8] section
5.1.6: the received energy per chip on the primary pilot
channel divided by the power density in the band
Final issue of the RF design specifying the design
coverage area
A list of system deficiencies (e.g. coverage, operation,
construction, etc.) noted during optimization. It will
include a description of the deficiency, a priority, a
responsible party, and the estimated resolution date.
Key Performance Indicator: considered as a metric that
is of particular importance for indicating the level of
quality of service
Quality of Service: referring to the UMTS QoS as
provided by the UMTS Bearer (see [R5])
Composed of the active set and the monitored set
(with a maximum of the six best received neighbors)
Area defined to be covered (may vary according to the
service)
This expression should not be mistaken with the
service area defined in the 3GPP TS, concerning only a
limited group of cells.
Area defined to be tested (consists of all drive test
roads, all specific in-building test areas, all special test
areas)
The design coverage area diminished by deployment
restrictions, exclusions and specific non-warranty areas

(e.g. tunnels, specific bridges...)
13. Appendix 1: Test Equipment List

Most of the tools and equipment are being studied and developed as of
this document is written.
Depending on the availability and the very first tests, this part should be
much more detailed in the future.
Agilent E7476A W-CDMW Drive Test System (currently available)
UMTS Diva-Agilent post-processing tool for SC scanner (currently
available)
Test-mobiles (3rd-party vendor, availability to be specified), possibly with
Qualcomm-CAIT or similar software (if available on WCDMA/UMTS)
Commercial mobiles (3rd-party vendor, availaibility to be specified)
Test isolation / insertion Loss Kit (within a suitcase): AITS (Air Interface
Test Suitcase)
Vehicle GPS receiver (Trimble, Garmin, Magellan, Lowrance...)
UMTS RFO-DIVA (test mobile post-processing software)
Logging post-processing software (also dependant on logging availability
in the equipment)
UMTS end-to-end tool (voice quality and data quality)
Radcom protocol analyzer with UTRAN-SPY (post-processing software)
Test UE
Circulator
Circulator
x dB
attenuator for simulated load
(if applicable)
y dB
attenuator for penetration losses
Antenna
Possible DL bypass
Sample test set up for Isolation and Losses

(x and y shall be defined according to contract values)
14. Appendix 2: Performance Metrics

14.1. KPI Indicative values
Most of the Network KPIs must be stated on a case by case basis to
include all network design and contract specificities.
Please refer to [R1] for example of Product-KPI values.
Hereafter is an example for successfull call rate.
Let us assume that three levels of coverage are designed, according to
the marketing targets:
1. Dense coverage: key marketing areas where coverage has been
designed to support all services at high quality levels
2. Medium coverage: areas where most ressource-consuming services
are expected to be less used allowing less density in the coverage
3. Light coverage: areas where basic services are provided with very
good quality but no high ressource-consuming services are expected.
The repartition for CS-voice call originations is supposed to be by default
50 % mobile-originated, 50 % mobile-terminated if no other repartition is
better suited to the clients needs.
The repartition for PS call is 100 % mobile-originated.
Successful call rate metric[12]
Per-cluster
Region threshold
Coverag
Service
threshol (no more
Call success conditions
e density
d
than 50 %
vs. Region)
CS-voice dense
95 %
92 %
over the call MOS 3.5
MO call setup delay < 4s
MT call setup delay < 5s
Holding time = 2 min (without call
medium
light
PS (I/B dense
64k-UL /
64k-DL)
[13]
medium
light
85 %
80 %
78 %
70 %
95 %
92 %
85 %
80 %
78 %
70 %
drop)
same as dense
same as dense
Initial 500 B (4 kb) ping RTT (call
setup) < 5.5 s
Continuous 500 B (4 kb) ping RTT
< 0.5 s
800 kb file transfer < 20 s
4 Mb file transfer < 85 s
Holding time = 2 min (without call
drop)
same as dense
same as dense
Due to tool capabilities, it is not recommended at this time to include CSdata tests in acceptance criteria.
14.2. Detailed metrics for acceptance and optimization

14.2.1. Acceptance Metrics
Warnings: for each metric, a measurement source is given as far as
possible. The measurement source, as well as the measurement
methodology, may vary depending on the tool or feature availability.
Moreover, when it is specified Iub or Iu (for spying with protocol analyzers),
it does not mean that both interface spying are equivalent but that one or
both at the same time may be required to get the enough information to
compute the metric.
Dropped call rate
Objective
Measure the reliability of a call within the network coverage (which is
strictly limited to the warranty zone). Please refer to the design
assumptions and [R1] for values to be met within the warranty area.
Definition[14]
The dropped call rate is the ratio of abnormally terminated calls vs. the
total number of established calls, which can be expressed by the following
formulas:
CS Domain:
where Abnormal_Release_Cause is any cause as defined in 3GPP TS

24.008 [R6], Cause IE, different from:
Cause #16: Normal Call Clearing
Cause #31: Normal, unspecified
PS Domain:
where Abnormal_Release_Cause refers to the following IE failure cause
values appearing in the preceeding RRC_Connection_Release as defined in
3GPP TS 25.331 [R10]: unspecified, pre-emptive release, congestion and
re-establishment reject.[15]
Measurement method
To verify the coverage continuity, long calls will be performed (call are
started and maintained as long as possible). The number of dropped calls,
the time and the geographical location when the failure occurs will be
recorded along each control route.
During the same drive, short calls will be also performed. The short call
duration will be the Average Holding Time (AHT) as agreed between both
parties (the customer and Nortel Networks).
The AHT may differ from one service to another (typically for speech
service, the AHT is between 90s and 120s).
Potential source of measurements:
1. Trace mobile with post-processing
2. Protocol analyzers on Iubs and post-processing tools
Call setup Success rate (Mobile Originated)

Objective
Measure the service availability when user wants to activate it. Please
refer to the design assumptions and [R1] for values to be met within the
warranty area.
Definition
The call setup success rate is the number of successfully established calls
vs. the total number of call setup attempts, which can be expressed by
the following formulas:
CS Domain:
* the Establishment Cause IE is defined in 3GPP TS 25.331 [R10].
Note 1: this metric could be itemized by traffic class (assuming that the
originating cause is correctly assigned when setting up the call)
Note 2: the domain (CS or PS) information appears in the
RRC_Initial_Direct_Transfer message following in the call setup message
flow (please refer to [R1] for more details).
PS Domain:
* the Establishment Cause IE is defined in 3GPP TS 25.331 [R10].
Note 1: this metric could be itemized by traffic class (assuming that the
originating cause is correctly assigned when setting up the call)
Note 2: the domain (CS or PS) information appears in the
RRC_Initial_Direct_Transfer message following in the call setup message
flow (please refer to [R1] for more details).
Measurement method
The test mobile tool will be configured to originate a sequence of calls to a
known non-busy number or IP address (depending on the application).
Each call will be terminated when established.
2. Protocol analyzers on Iubs and post-processing tools
Call Setup Time (Mobile Originated and Terminated)

Objective
Measure the responsiveness of the network when the user wants to
access it. It is the time it takes for a mobile originated (respectively
terminated) call to be set up across the network. Please refer to [R1] for
values to be met within the warranty area.
Definition
The call setup time is the time between the initial call establishment
request and the call establishment confirmation. This is therefore defined
by the following formulas:
CS Domain, mobile originated call:

[16], [17]
CS Domain, mobile terminated call:
[18]
PS Domain, mobile originated call:
[19], [20]
PS Domain, mobile terminated call:
Non applicable as of this document was written.
Measurement Method
This test is accomplished in conjunction with the call setup success rate
(for origination) and with the paging success rate (for termination).
For each call, the call setup messages will be time-stamped and recorded
and the time difference will be calculated as described in the definitions.
2. Protocol analyzers on Iu (mobile originated calls) and PSTN interface (CS
mobile terminated calls with post-processing software
Paging SUccess rate

Objective
Measure the network efficiency in finding the appropriate mobile. Please
refer to the design assumptions and [R1] for values to be met within the
warranty area.
Definition
The paging success rate is the ratio of number of paging responses vs. the
total number of paging attempts. This can be defined by the following
formula:
[21]
* the Establishment Cause IE is defined in 3GPP TS 25.331 [R10]; all
terminating causes must be grouped together.
the CN Domain Identity IE is defined in 3GPP TS 25.331 [R10]; only
pages from a given CN domain shall be considered for this metric
(meaning there is a Paging success rate for CS and a paging success rate
for PS)
Measurement method
The test mobile will be in idle mode. A sequence of incoming call will be
set up. Each call will be terminated when established.
Trace mobile with post-processing
Voice Quality, Mean Opinion Score (MOS)

Objective
The objective of this measurement is to provide a measurement of voice
quality.
Please refer to [R1] for values to be met.
Definition
The Mean Opinion Score is a subjective evaluation of voice quality [R14].
It is the result of an absolute rating (without reference sample) on a scale
going from 1 (bad) to 5 (excellent).
Measurement Method
Please refer to [R18].
The coverage reliability r must be considered in the measurements and
therefore the resulting metric must be computed on the r % of the best
samples only (the worst (1-r) % of the samples must be discarded).
An end to end tool shall be used to compare the received voice sample
with the original one.
Bit Rate (data only)

Objective
This metric measures the throughput in kbps delivered to the application
(usually at TCP level). The minimum, maximum and average shall be

measured to get an idea of the transmission regularity. Please refer to [R1]
for the values to be met.
Definition
The bit rate is the number of bit received at the application SAP (usually
brought down to the TCP level) over a given period of time (usually
measured in kbps).
Measurement Method
The measurement is taken from long calls. The best application for this
type of test looks like a file transfer through FTP. A big enough file will be
chosen in order to keep transferring during the entire drive test. The
requested data rate transfer will be chosen according to the design
assumptions in the driven areas.
Bit rates for downlink and bit rates for uplink will be processed separately.
If r is the coverage reliability, the worst (lowest bit rate) (1- r )% of the
bins on the downlink and the worst (1- r )% of the bins on the uplink shall
be excluded from analysis. For performance acceptance, the remaining r
% of the bins of each of the downlink and uplink shall satisfy the bit rate
requirements.
Minimum, maximum and average values will be calculated for each way.
If key or strategic marketing locations included in the planned designed
coverage area fall within the r % of bins that are discarded, these
locations shall be put on the issue list.
Trace mobile with post-processing
File Transfer Time (Data Only)

Objective
This metric will measure the transmission efficiency between the source
application SAP (usually at TCP level) and the destination application SAP
(usually at TCP level). Please refer to [R1] for thresholds to be met.
Definition
The easiest way of measuring the transfer delay is to measure a so-called
file transfer time defined as the time to receive without errors a file of a
given size (between the first bit of the first received packet and the last bit
of the last received packet completing the error-free transmitted file).
This definition has the great advantage not to require any synchronization
between both ends for the measurement tool.
According to 3GPP TS 23.107 [R4] the following definition applies for the
transfer delay:
Transfer delay (ms)
Definition: Indicates maximum delay for 95th percentile of the distribution
of delay for all delivered SDUs during the lifetime of a bearer service,
where delay for an SDU is defined as the time from a request to transfer
an SDU at one SAP to its delivery at the other SAP.
NOTE 3: Transfer delay of an arbitrary SDU is not meaningful for a bursty
source, since the last SDUs of a burst may have long delay due to
queuing, whereas the meaningful response delay perceived by the user is
the delay of the first SDU of the burst.
The file transfer time is then equivalent to adding all the SDU transfer
delays for completing the file at destination, except the first delay
between the time of the request and the transfer time of the first SDU.
However, we can consider this time as negligible compared to the total
file transfer time, assuming file sizes over 100 kb.
Measurement Method
The measurement can be done with the same method than the bit rate
measurement and should be done during the same tests.
The data processing should follow the same methodology than the one
used for bit rate processing.
Potential source for measurements:
An end-to-end tool must be used to monitor the test at both ends
(checking that the file was received without errors).
14.2.2. Optimization Metrics

In complement of the performance metrics that will be used (partly or
totally) for network performance acceptance, other metrics are measured
and computed for optimization purpose without being formally linked to
the acceptance process. This section is supposed to give a list and details
about these metrics. The list is probably non-exhaustive and will have to
be adjusted after the first UMTS network performance optimizations.
RF Performance
Primary CPICH Ec/N0

Objective
Troubleshoot for best server reception and interference reduction.
Definition
As reported by the UE in the Intra-frequency measurement quantity IE
(defined in TS 3GPP 25.331 [R10]).
Measurement Method
The measurements shall be taken during any drive tests (all opportunities
are welcome).
1. Scrambling-code scanner (such as the Agilent E7476A) with a postprocessing software
2. Trace mobile and post-processing software
3. RNC logging and post-processing software (measurement report
analyses)
4. Protocol Analyzers on Iub with post-processing software
UE Tx Power
Objective
Troubleshoot for best server reception and uplink interference reduction.
Definition
As reported by the User Equipment in the UE Internal measured results IE
(defined in TS 3GPP 25.331 [R10]).
Measurement Method
The measurements shall be taken during any drive tests (all opportunities
are welcome).

1. Trace mobile and post-processing software
Average Number of RLs in the active set

Objective
Neighbor list and SHO tuning.
Definition
within a given cell.
Measurement Method
The measurements shall be taken during continuous drive tests.
analyses)
3. Trace mobile
Average Number of PCPICHs in the Monitored set

Objective
Definition
Average calculated from the number of monitored cell set measured by
the mobile and transmitted to the UTRAN in the measurement report
messages or by UTRAN counters.
Measurement Method[22]

analyses)
3. Trace mobile
Average Number of RLs per call

Objective
Definition
within the test areas (either cluster or system)
Measurement Method
analyses)
BLER (UL and DL)

Objective
Check the radio link transmission quality.
Definition
The Block Error Rate (BLER) is the Transport Channel Block Error Rate as
measured by UTRAN in uplink (as specified in 3GPP TS 25.215 [R8]) and as
reported by the UE in downlink in the
RRC_Measurement_Report(quality_measured_results_list).
Measurement Method
The measurements shall be taken from all continuous drive tests.

Potential source of measurements (TBC):
1. Trace Mobile with post-processing software (DL-BLER only)
Radio Blocking Rate

Objective
Radio resource shortage.
Definition
[23]
Measurement Method
The measurements shall be taken from drive tests used to measure call
setup success rate.
1. RNC logging with post-processing software
2. Protocol Analyzer spying the Iub with post-processing tool
Network Access Performance

RAB Unavailability
Objective
RAB resource shortage. This metric is classified as a Tier-2 metric.
Definition
* where: RANAP_RAB_AssRsp is the RANAP RAB Assignment Response
message, RABs_failed_SetupOrModify is the RABs failed to setup or
modify part of the previous message, Congestion causes are the
following failure causes[24]:
Requested Traffic Class Non Available (#18),
Requested Maximum Bit Rate Non Available (#20),

Requested Guaranteed Bit Rate Non Available (#21),
Requested Traffic Delay Non Achievable (#22),
No Remaining RAB (#31),
Requested Maximum Bit Rate Downlink Non Available (#33),
Requested Maximum Bit Rate Uplink Non Available (#34),
Requested Guaranteed Bit Rate Downlink Non Available (#35),
Requested Guaranteed Bit Rate Uplink Non Available (#36),
No resource available (#114)
Details may be found in [R11]
Measurement Method
This metric applies only on loaded networks. Measurements shall be done
during the same drive tests as the ones used for call setup success rate
(short calls).
2. Iu spying with protocol analyzers and post-processing software
Mobility Performance
UTRAN 2G HHO Success Rate
Objective
Inter-RAT HHO reliability.
Definition
Limits: if the mobile cannot resume the connection to UTRAN (as specified
in 3GPP TS 25.331 [R10]), the failure will not be counted.
Measurement Method
Due to the cross-network specificity of this feature, it is highly
recommended to avoid including such a metric in the network
performance acceptance contracts.
It would be much better to isolate the test cases as a demonstration
purpose of HHO reliability and correct tuning.
Within this context, special drive-tests must be done in selected areas
handing calls down to 2G. Sufficient drives must be done to get enough
samples.
2. Iub spying with protocol analyzers and post-processing software
Average UTRAN 2G HHO time

Objective
Inter-RAT HHO tuning.
Definition
To be measured on the 2G (GSM) part as:
Measurement Method
Measurements should be made on previously selected areas of the
network as agreed between both parties (the customer and Nortel
Networks).
1. Dual-mode test mobile with post-processing (GSM messages)
2. GSM Call Trace?
ATM
ATM Cell Rate
Objective
Check the validity of the negotiated PCR for the ATM CoS CBR; the SCR for
VBR; the couple (MCR, PCR) for ABR.
Definition
The ATM Cell Rate is defined in this document as being the effective cell
rate in number of cells per second.
Measurement Method
Measurement should be made on all drives.
1. Passport counters.
2. Interface spying (either Iub or Iu) with protocol analyzers and postprocessing software
ATM Cell Delay Variation (CDV)

Objective
Check the ATM transmission constancy.
Definition
The CDV we will consider in the document is the one defined as 1-point
cell delay variation considering the Measurement Point (MP) at the end of
the communication link (either a UE or the fixed extremity of the call). As
described in ITU-T I.356 [R12]:
The 1-point CDV parameter describes variability in the pattern of cell
arrival (entry or exit) event at an MP with reference to the negotiated peak
cell rate 1/T (see Recommendation I.371); it includes variability introduced
(or removed) in all connection portions between the cell source and the
specified MP. It can be related to cell conformance at the MP, and to
network queues.
The definition of this metric in ITU-T I.356 [R12] is:
1-point cell delay variation is defined based on the observation of a
sequence of consecutive cell arrivals at a single MP.
Measurement Method
1. Passport counters ?
ATM Cell transfer Delay (CTD)

Objective
Measure the performance of the ATM transmission as far as the
transmission delay through the transmission network is concerned.
Definition
As defined in the ITU-T I.356 [R12]:
Cell Transfer Delay (CTD) is the time, t2 t1, between the occurrence of
two corresponding cell transfer events, CRE1 at time t1 and CRE2 at time t2,
where t2 > t1 and t2 t1 Tmax. The value of Tmax is for further study, but
should be larger than the largest practically conceivable cell transfer
delay.
Measurement Method
1. Passport counters.
IP
IP One-Way Packet Loss
Objective
Check the IP transmission reliability.
Definition
This metric can be defined as the ratio of lost packets vs. the total number
of transmitted packets which can be translated in the following formula:
* where the One_Way_Packet_Loss is defined in RFC-2680 [R17] as:
>>The *Type-P-One-way-Packet-Loss* from Src to Dst at T is 0<< means
that Src sent the first bit of a Type-P packet to Dst at wire-time* T and that
Dst received that packet.

>>The *Type-P-One-way-Packet-Loss* from Src to Dst at T is 1<< means
that Src sent the first bit of a type-P packet to Dst at wire-time T and that
Dst did not receive that packet.
Measurement Method
Measurement should be made on all data drives.
1. End-to-end data quality tool
IP Maximum One-Way Delay

Objective
Check the IP transmission efficiency.
Definition
The maximum value of the one-way delay metric for the 95th percentile of
the distribution of delay for all delivered packets during the lifetime of the
bearer service.
The one-way delay is defined in RFC-2679 [R16] as:
For a real number dT, >>the *Type-P-One-way-Delay* from Src to Dst at
T is dT<< means that Src sent the first bit of a Type-P packet to Dst at
wire-time* T and that Dst received the last bit of that packet at wire-time
T+dT.
>>The *Type-P-One-way-Delay* from Src to Dst at T is undefined
(informally, infinite)<< means that Src sent the first bit of a Type-P packet
to Dst at wire-time T and that Dst did not receive that packet.
Measurement Method
Measurements should be made during long data call drive tests.
IP Maximum Jitter
Objective
Check the IP transmission constancy.
Definition
The IP Maximum Jitter is defined as the maximum variation of the one-way
delay metric as defined here above.
Measurement Method
Measurement should be made on all data drives.
14.3. Measurement Uncertainty

This is mostly applicable when a set of different detailed metrics are
chosen for acceptance criteria.
(Please see [R19] for more details)
14.3.1. Tolerance Interval of One Data Population
All of the performance metrics that is used to assess the quality of a
network is estimated using a sample of events (calls, frames, etc.) Let p
be the metric to estimate from the data (for example p could be FER or
dropped call rate, etc.). Let n be the total number of events (frames, calls,
etc.) in the sample, and k is the number of successes (i.e., number of
events that meet a certain criteria such as frame erasures, dropped calls,
etc). Then p is estimated as the ratio:
It can be shown that the distribution of this error is approximately normal
(Gaussian) with zero mean, and a standard deviation given by:
The accuracy of the estimate is thus influenced by the sample size, with
more samples giving a more accurate estimate (smaller deviation of the
error). Therefore, this measurement error must be taken into account
when selecting pass/fail criteria for qualifying the network.
The pass criterion is selected such that the estimate is less than or equal
to the target value of the metric plus a Tolerance Interval. The Tolerance
Interval is the value expressed as a number of standard deviates that
achieve a given Tolerance Interval. That is,
where
Thus, the estimate should satisfy
where
p is the target value for the metric (between 0 and 1).
z is the number of standard deviates that corresponds to a given
Tolerance Interval.
n - sample size.
For acceptance test we will use a Tolerance Interval of 99%. In this case,
the number of standard deviates z = 2.33. Clearly, the estimate
converges to the target value as the sample size approaches infinity or
when the target value is close to 1 (this indicates that the success events
occur with high probability or certainty and the sample size doesnt need
to be large to estimate the metric).
Example:
Assume the metric under consideration is FER and the target/design value
is 2%. If the FER is estimated from 100 samples, then the tolerance
interval is given by . In this case, the estimate must satisfy On the other
hand, if the sample size is 1000, then and the estimate must satisfy
14.3.2. Tolerance Interval of Several Data Populations

In some cases, it may be desired to calculate the tolerance interval of
more than one data population with different sample sizes. For example, it
may be desired to calculate the tolerance interval of the system-wide
metric from the performance metrics of subsets of the systems (e.g.,
clusters). Or, it may be desired to calculate the tolerance interval
associated with the cluster, which is composed of several samples with
different sizes. This section provides a formula to calculate the effective
tolerance interval.
We assume that the measurements in all subsets (or data populations) are
statistically independent. Let the sample sizes of the data populations be
n1, n2, n3, nN, where N is number of data populations.
The standard deviation of data population i is given by:
Since the data populations are statistically independent, the standard
deviation of all data populations combined equals
The tolerance interval of the total population is given by (please see
section 14.3.1 for details)
where z = 2.33 for 99% confidence.
We note that if the sample size in all data populations is the same, then
(For equal sample size in all data populations)
Comparing this result to that of a single data population case, we note
that the tolerance interval decreases by the square root of number of

populations.
Example:
Assume the FER measurements are collected from 4 clusters. Let the
sample sizes of the clusters be 1000, 1000, 10000 and 10000. Let the
target/design criterion be p = 2%.
The tolerance intervals that are associated with the FER measurements in
the 4 clusters are given by 0.01, 0.01, 0.0033, and 0.0033.
The composite tolerance interval is given by
14.3.3. Table z-values vs. confidence levels
Confidence
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
z
0.524
0.553
0.583
0.613
0.643
0.674
0.706
0.739
0.772
0.806
0.842
0.878
0.915
0.954
0.994
1.036
1.080
1.126
1.175
1.227
1.282
1.341
1.405
1.476
1.555
1.645
1.751
1.881
2.054
2.326
For any other value, please refer to any table of the standard normal
distribution.
15. Appendix 3: Overall planning for performance acceptance

15.1. Optimization resources:
15.1.1. IS-95 Experience
The following figures are coming from an IS-95 network performance
acceptance project and are given for information only. This is considered
as one of the typical project but extrapolation to UMTS must be done
carefully. More accurate numbers will be provided with time after the first
UMTS network acceptance projects.
Cluster Size (in number of sites)
14-20
Sites (total number for the tested
225
network)
Analyst Engineers
5.5
Shakedown teams
2 (3 to 4 sites per team per day)
Drive test teams
4
Analysis Computers
5
Weeks
9
Shifts
2 (7/17 and 16/2)
(Example of CDMA-IS95 Telstra network performance acceptance,
Brisbane, Australia, 1999)
15.1.2. teams
Shakedown team
A shakedown team is composed of three persons:
1. a driver
2. a technician to collect data and check the RSSI on the received path
(main and diversity) at the Node B.
Drive Test Team
A drive test team is composed of one operator and one driver.
The operator runs data collection and controls the drive test equipment
for all calls (typically voice and data). In some cases, this might need an
extra operator to handle calls (e.g. specific data calls such as web
browsing).
The operator is also in charge of downloading and post-processing the
resulting data and managing the backups and the archives.

Analysis Team
The analyst team is composed of one experienced engineer. S/he analyses
the data from each drive tests and makes change recommendations.
The experienced engineer has also a coordination role: coordinate drive
test teams (briefing before the drive tests, debriefing after the drive tests,
advise to drive test teams when they encounter problems, possible realtime analyses with the on-site drive test team.
At-the-Switch (RNC) team
A logging operator has to be scheduled as well to start logging, stop
logging. This could be the on-duty switch (RNC) technician.
A technician has also to be present to do datafill changes, configuration
audit. He would be also in charge of downloading and post-processing the
logged data, as well as managing the backups and the archives.
Depending on the workload, this task may need a fully dedicated
technician.
15.1.3. Extrapolation to UMTS
Shakedown
1 h / site / team
Cluster
optimization
Data
collection
1.5 site / team /

shift[25]
Cluster
optimization
1 site / engineer /
shift
This does NOT include:

driving time to & from site
re-do if maintenance problems
(equipment failure, either site or
measurement equipment or network
problems)
This estimation includes:
pre-driving routes (feasibility, clear
description to insure all subsequent
drives will follow the same path)
time lost because of equipment
problems (either drive test equipment
failure or network equipment failure)
sanity check driving (check after
s/w upgrade without much analysis
needed)
driving to and from clusters
Beside analysis time, this includes:
generating change requests
Analyses
System
optimization
reviewing datafill dumps

advising drive test teams when they
call in with a problem
advising rigging crews
analysis abortion due to
miscellaneous reasons (h/w failure at
site, data collection problems such as
bad configuration, h/w failure, data
missing...)
2 weeks
estimation assuming a system of
(for one team, one around 10 to 20 clusters, each around
shift a day)
20 to 30 sites (200 to 600 sites)
These estimations are based according to the process described in the

document.
This would lead to around 20 days for a cluster of 20 sites and 30 days for
a cluster of 30 sites for one team (one shift a day).
However, if time or money supersedes network optimization (and
potentially network tuning accuracy), here is an order of importance of the
drive tests:
Drive test type (see
process)
Pilot Optimization
Cluster
optimization
mandatory
Radio Optimizaton
mandatory
Loaded Uu Optimization if OCNS available
Acceptance data
mandatory
collection
System optimization
recommended (merged with
next)[26]
mandatory
if OCNS available
mandatory
Taking the most risks on optimization and reducing at the minimum the
drive tests, this would lead to the following estimations:
Cluster
optimization
Data collection
Cluster
optimization
3 sites / team / Beside analysis time, this includes:

shift
generating change requests
missing...)
2 sites /
Beside analysis time, this includes:
engineer / shift generating change requests
Analyses
System
optimization
1 week

missing...)
estimation assuming a system of
around 10 to 20 clusters, each around
20 to 30 sites (200 to 600 sites)
This would lead to 10 to 15 days for clusters of 20 to 30 sites.

Warning:
The above figures are only estimations and shall be updated according to
the first on-field experiences.
An example of possible planning would be the following one:
Let us assume the following:
the network area (system) contains 300 sites
this area is divided in 10 clusters
4 clusters can be optimized simultaneously (meaning without impacting
each other)
4 analyst engineers and 4 drive test teams are on site and ready
the shakedowns have been performed on a site by site basis, following
the I&C and the network element acceptance steps
the cluster optimization can be started simultaneously with the stability
period
the ending of the cluster optimization (last few clusters) allow to do the
first drive test for system optimization.
With all these assumptions, we would have the following schedule:
With a 7-days-a-week schedule, this would reduce the overall time frame
to 3 months and with two shifts a day, we reach the minimum time of a
little bit less than two months.
End of DOCUMENT

UMTS Performance Optimization and Acceptance Guidelines

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

UMTS Performance Optimization and Acceptance Guidelines

Uploaded by

Copyright:

Available Formats

UMTS Performance Optimization and Acceptance Guidelines

14.3.3. Table z-values vs. confidence

set up a process of following up with issues and work yet to be

2.1. Applicable documents

During the same tests, both CS and PS domains will be simultaneously

Guaranteed Bit Rate

Maximum bit rate

As far as optimization and acceptance metrics are concerned, we will

For loaded conditions (if applicable), OCNS is a feature available and

4.2. Specific Assumptions

signaling messages during the call.

5.2. In-building Test Areas

6. Simulated Traffic Load (if applicable)[3]

load in complement of the unloaded tests. This traffic load shall

8.1. Combined Metric: Successful Call Rate

8.2. Detailed Metrics

9. System Optimization process

9.1. Configuration Audit

maximum noise power allowed in the channel bandwidth would be

9.2. Cell Shakedown

Effective Transmit Antenna Orientation

9.2.4. Drive Routes

9.2.5. Test Drives

9.2.8. Issue List

9.3. Cluster Optimization

and secondary road classifications. These routes should encompass all

PS calls will follow a test pattern composed of initial ping, subsequent

PCPICH coverage; Number of PCPICH over a given Ec/N0 (typically 12dB in

Acceptance Data collection and Final Cluster Check DRIVE

4. Handoff parameters checking and tuning

9.3.6. Exit Criteria

list by regional entity might be created.

9.4. System Optimization

Optional optimization drives

10.2. Performance Measurement Reports

10.3. Test Environment Report

10.4. Issue List

issue will be assigned a priority:

11. OCNS unavailability

Available Bit Rate

Frame Quality Indicator

Session Management (see [R3] for details)

area availability refers to the percentage of

Final Cell Plan

Typically around 30 sites that serve a contiguous

restrictions, exclusions and specific non-warranty areas

13. Appendix 1: Test Equipment List

Sample test set up for Isolation and Losses

14. Appendix 2: Performance Metrics

14.2. Detailed metrics for acceptance and optimization

where Abnormal_Release_Cause is any cause as defined in 3GPP TS

Call setup Success rate (Mobile Originated)

Call Setup Time (Mobile Originated and Terminated)

CS Domain, mobile originated call:

Paging SUccess rate

Voice Quality, Mean Opinion Score (MOS)

Bit Rate (data only)

(usually at TCP level). The minimum, maximum and average shall be

File Transfer Time (Data Only)

14.2.2. Optimization Metrics

Primary CPICH Ec/N0

Potential source of measurements:

Average Number of RLs in the active set