KPI Investigation Methodology Process UA7

KPI Investigation Methodology
Processes, methods and tools to troubleshoot
Agenda
1. Introduction
2. KPI Investigation Methodology Overview
3. First Level Investigation
4. Deep Investigation
Introduction
3 | UA7.1.3 KPI Investigation Methodology | Feb. 2011
Alcatel-Lucent Internal
Proprietary Use pursuant to Company instruction.
Introduction
KPI degradation after Swap/Software Upgrade / Hardware Replacement / Feature
Activation is a sensitive issue for customers.
KPI degradations need to be quickly detected through accurate monitoring.
Main objectives of this document:
Present the recommended Processes,and Tools to troubleshoot efficiently the
performance issues observed during Global Market.
In particular, clarify the roles of the on-site team and back-office teams during the
First Level Investigation and Deep Investigation steps of the investigation
4 | UA7KPI Investigation Methodology
KPI Investigation
Methodology Overview
KPI Investigation Methodology Overview

Investigation process (organizational view)
Monitor the network
(customer)
KPI degradation
Monitor the network

(ALU)
First Level Analysis
First Level
Investigation
Alarms,Configuration
Deep
Investigation
Open an AR
that should
include all
the outputs
from the
First Level
Analysis
Optimization/Suppo
rt/GPS
troubleshooting
experts
- Analyze the issue based on all

available data (including traces)
- Maintain a comprehensive
document (usually Powerpoint)
with investigation results and
current action plan
Deeply analyze parameter
settings and counters
subject matter
Alcatel-Lucent experts
Internal
(GPS group)
- Collect additional
traces (RNC and NodeB
internal traces,
Network Analyzer
traces, CTg, Call
Failure Trace, etc.)
- Change parameters
and perform the
related monitoring
- Perform drive tests
- Apply fixes and
assess performance
recovery

Investigation process (functional view)
Customer feedback
Check
parameter
changes
KPI degradation
Basic counter
analysis to identify
the TopN most
impacted NodeBs or
Cells
Daily monitoring
Check
alarms &
stability
Record predefined set of

Call Trace &
product traces
First Level
Investigation
- Logs are uploaded as recommended, then AR is opened

- Before considering going deeper (e.g. open CR, trigger more support), check the
Release Notes applicable to current SW load to verify if issue is a known issue
CS CDR
Troubleshooting
PS R99 CDR
Troubleshooting
HSxPA CDR
Troubleshooting
7 | UA7.1.3 KPI Investigation Methodology
| Feb. 2011
Throughput
Troubleshooting
Accessibility
Troubleshooting
Mobility
Troubleshooting
Deep Investigation
Optimization/Support/
GPS and
troubleshooting
experts

Weekly performance investigation meetings
For each identified performance issue , technical meetings ;these meetings are
usually chaired by a single matter expert or Optimization troubleshooting
expert should be scheduled on a weekly basis in order to:
Share the latest investigation results from each involved team
Synch-up on the recent key events and data logs
Agree on potential new axis of investigations
Determine a clear action plan (next steps) and assess its feasibility

Useful Links Internal tools (ALU-proprietary) [1/3]
WPS (Wireless Provisioning System):
https://wcdma-ll.app.alcatel-lucent.com/livelink/livelink.exe?func=ll&objId=63414868
1.
Launch WPS installation (file wips-Vxxx_windows.exe)
2.
When prompted during installation, load the relevant UMTS plugin

(file UMTS_Access-Vxxx.wipsar)
Call Failure Trace:
At above link can be found:
Presentations on CFT capabilities and CFT usage examples
CFT Reader tool
eDAT (Evolved Data Analysis Tool):
http://navigator.web.alcatel-lucent.com/index.htm
RNC Log Collection:
NodeB Log Collection:

Useful Links Internal tools (ALU-proprietary) [3/3]
RFO Product:
http://aww.srd.alcatel.com/icr/SDCT/sdct.php
The License must be requested through the SDCT (Software Delivery Center Tool) homepage
accessible at above link. Once under the SDCT homepage:
1.
Download the SDCT User Guide (Online documentation PowerPoint link at the top of the page)
and read it
2.
Download the W-CDMA RFO License Ordering presentation (W-CDMA_RFO_License link at the
bottom-right of the page) and read it
3.
Click on the Request link at the middle of the page to initiate the license request procedure.
- If you are not yet registered in the SDC database, you will first need to register
(Registering button)
- When prompted the template to select is RFO Product license

Useful Links External tools
Wireshark:
Wireshark: http://www.wireshark.org/download.html
WinPcap: http://www.winpcap.org/archive/4.1beta5_WinPcap.exe
- Packet capture and filtering engine of many open source and commercial network tools
- Used by Wireshark
Iperf:
https://wcdma-ll.app.alcatel-lucent.com/livelink/livelink.exe?func=ll&objid=57980928
First Level Investigation

Pre-Requisites: Collection of reference data before Upgrade [2/2]
The KPI Investigation Methodology is fully applicable only if the following reference data are
properly collected in case of Swap/the SW Upgrade / HW Replacement / Feature Activation
(continued):
Snapshot (.xml or .xcm file containing the parameter configuration data)
Daily network Snapshots should be collected during at least 1 week prior to the upgrade, to check
configuration (parameters) stability. The Snapshots should be stored in a LiveLink repository for easy
access by back-office teams.
Hardware Inventory
HW Inventory files should be collected.
Drive Test.
Call Trace.
Tool X: Internal tool

(ALU-proprietary)
Pre-requisites:
Tool Y: External tool
The KPI Investigation Methodology refers to many internal (ALU-proprietary) and external
troubleshooting tools
Tools needed for First Level

Investigation
Tools needed for Deep Investigation
WPS (Wireless Provisioning System)

NPO (Nw Performance Optimizer)
UTRAN Call Trace Wizard
HFB (Historical Fault Browser)
RNC Log Collection
NodeB Log Collection
BTS Set Traces
eDAT, to post-process UE traces and Call Traces

RFO Product, to post-process Call Traces
WQA, to post-process CTn and CFT traces
CFT Reader, to post-process CFT traces
Remote Connection to NodeB (through Telnet session
or by using TIL tool), to check the NodeB status and collect
(i.e. to collect additional logs and deeply analyze the issue)
specific NodeB internal traces (e.g. IMT logs)

Audit Tool, to produce the Audit and NodeB Profiling excel
files based on Snapshot and HW Inventory
Wireshark, Windump, Iperf, DU Meter to investigate E2E
Tput

Selection of TopN most impacted network elements
Customer feedback
Check
parameter
changes
KPI degradation
1/ Basic counter analysis to

identify the TopN most
impacted NodeBs or Cells
For this, use standard NPO Views

2/ Also try to roughly identify the
failure causes
Daily monitoring
Check alarms
& stability
- Focus on the
TopN NEs
determined
through
counter
analysis
- Use e.g. HFB
tool
Record pre-defined set of

Call Trace & product
traces
- Launch CTg+CFT and RNC
Log Collection on the TopN
NEs
- Launch NodeB Log Collection
and BTS Set Traces with
NodeB KPI AP template on
the Top3 NodeBs
- Logs are uploaded as recommended, then AR is opened

- Before considering going deeper, check the Release Notes
applicable to current SW load to verify if issue is a known issue
First Level
Investigation
Deep Investigation
Rule to select the TopN most impacted (or worst) network elements:
If degradation is observed at RNC level
Identify the TopN NodeBs
If degradation is observed at NodeB level Identify the TopN Cells

Parameter check [2/2]
Method to compute all the differences between 2 Snapshots with WPS tool:
Import in WPS the earliest available Snapshot after the observed KPI degradation.
Use Replace initial snapshot and discard existing workorders option.
Under the same WPS application, import on top of previously loaded Snapshot the latest
available Snapshot before the observed KPI degradation.
Use Resynchronize the planned configuration with new snapshot option.
Under the Workorders tab you will see the delta workorder to transition from old to
new Snapshot, showing all the differences.
You can analyze the computed differences here, or make a right-click on the table to
export them as an .html file and analyze them under Excel (recommended option).

Call
Trace [1/2]
CTB/CTg
Call Trace logs:
In order to collect the relevant data in CTg logs while capturing a sufficiently high number of
failures, the CTg session can be configured to capture only a few specific call types.
CFT Call Trace logs:

Since UA7.1 release, Call Failure Trace (CFT) logs have proven to be a powerful additional type of
Call Trace for KPI investigation. A CFT record is being captured each time an RRC Connection failure,
RAB Assignment failure or RAB Drop occurs.
Differently from CTg logs, CFT logs do not contain the whole call flow but are basically a detailed
snapshot of the call at failure time; CFT logs are typically smaller than CTg logs.
In typical live environment a CFT session will capture the great majority of call failures,
whereas a CTg session usually captures only a relatively small part of the total number calls.
A CFT session will attempt to capture all call failures, regardless to the call types specified in the
CTg Call Trace template CFT is particularly powerful for Multi-RAB types of KPI degradations
CFT logs should be collected each time a CTg session is launched. This can be
done by making sure to use a CTg Call Trace template that includes the record
called UeCallFailureFull.
CFT logs should always be made available together with the CTg logs under the
log storage location (e.g. in a folder named CallTrace).

WPS snapshot delta (e.g. .html format), 2 snapshots (latest snapshot before
the degradation, earliest snapshot after the degr.) in folder Snapshots
Output of RNC Log Collection in folder RNC logs
Output of NodeB Log Collection (at least 1 BTS) in folder NodeB logs
CTg traces in folder CallTrace
Call Failure Trace data in folder CallTrace
18 | UA7.1.3 KPI Investigation Methodology |
Deep Investigation
Deep Investigation
Overview
Customer feedback
KPI degradation
Daily monitoring
First Level Analysis
First Level
Investigation
- Before considering going deeper, check the Release Notes
applicable to current SW load to verify if issue is a known
issue.Alarms and Configuration.
Deep Investigation
CS CDR
Troubleshooting
PS R99 CDR
Troubleshooting
HSxPA CDR
Troubleshooting
Throughput
Troubleshooting
Accessibility
Troubleshooting
Mobility
Troubleshooting
Optimization/Support/
GPS troubleshooting
experts.
Deep Investigation
Counter-based in-depth analysis
Deeper counter-based analysis is needed in order to characterize the degradation. The

aim is:
To narrow down as much as possible where the degradation occurs (product, board,
procedure)
To identify a common profile of TopN most impacted cells (HW, configuration, traffic
profile, etc)
The analysis is done with:
Counter correlation => attempt to correlate the general counter showing the
degradation and a counter for a specific lower level procedure.
Comparison of distribution of failure types (e.g. which cause of drop has increased the
most)
Counters-based analysis is also used:
To quantify patterns identified by CTg analysis
To assess the KPI recovery after fix delivery
21 | UA7.1.3 KPI Investigation Methodology 1
Deep Investigation
CS CDR Troubleshooting cookbook
CS CDR troubleshooting
Analysis of relevant
counters for CS CDR
degradation
Analysis of
CFT data
Analysis of CTg
traces
Consolidate CTg + CFT + NPO analysis

(cause of drops, drop scenario, IMSI)
Consolidate global analysis

(using all tools outputs)
Launch CTg+CFT
session on TopN cells
Deep Investigation
PS R99 Troubleshooting cookbook
PS R99 CDR troubleshooting
counters for PS CDR
degradation
Analysis of
CFT data
Analysis of CTg
traces


23 | UA7.1.3 KPI Investigation Methodology |
Launch CTg+CFT
Deep Investigation
HSxPA CDR Troubleshooting cookbook
HSxPA CDR troubleshooting
counters for HSxPA
CDR degradation
Analysis of
CFT data
Analysis of CTg
traces

If needed, perform
Drive tests + UE traces
Launch CTg+CFT
Deep Investigation
Mobility troubleshooting cookbook
Mobility
troubleshooting
counters for Mobility
degradation
Analysis of CTg traces
Consolidate CTg + NPO analysis

Perform WQA CTn
analysis
If needed, perform
Drive tests + UE traces
Launch CTn
Deep Investigation
Accessibility troubleshooting cookbook
Accessibility
troubleshooting
counters for Accessibility
degradation
Analysis of
CFT data
Analysis of CTg
traces

In case of RRC degradation:
Perform drive-tests (UE traces)
Launch CTg+CFT
www.alcatel-lucent.com
www.alcatel-lucent.com

KPI Investigation Methodology Process UA7

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

KPI Investigation Methodology Process UA7

Uploaded by

Copyright:

Available Formats

KPI Investigation Methodology

Processes, methods and tools to troubleshoot

3 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

4 | UA7KPI Investigation Methodology

5 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

KPI Investigation Methodology Overview

Monitor the network

First Level Analysis

6 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

- Analyze the issue based on all

KPI Investigation Methodology Overview

Record predefined set of

- Logs are uploaded as recommended, then AR is opened

KPI Investigation Methodology Overview

8 | UA7.1.3 KPI Investigation Methodology

KPI Investigation Methodology Overview

WPS (Wireless Provisioning System):

Launch WPS installation (file wips-Vxxx_windows.exe)

When prompted during installation, load the relevant UMTS plugin

Call Failure Trace:

At above link can be found:

Presentations on CFT capabilities and CFT usage examples

CFT Reader tool

eDAT (Evolved Data Analysis Tool):

RNC Log Collection:

NodeB Log Collection:

9 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

KPI Investigation Methodology Overview

10 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

KPI Investigation Methodology Overview

11 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

First Level Investigation

12 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

First Level Investigation

13 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

First Level Investigation

Tool X: Internal tool

Tool Y: External tool

Tools needed for First Level

Tools needed for Deep Investigation

WPS (Wireless Provisioning System)

eDAT, to post-process UE traces and Call Traces

14 | UA7.1.3 KPI Investigation Methodology

(i.e. to collect additional logs and deeply analyze the issue)

specific NodeB internal traces (e.g. IMT logs)

First Level Investigation

1/ Basic counter analysis to

For this, use standard NPO Views

Record pre-defined set of

- Logs are uploaded as recommended, then AR is opened

15 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

First Level Investigation

First Level Investigation

CFT Call Trace logs:

17 | UA7.1.3 KPI Investigation Methodology

First Level Investigation

18 | UA7.1.3 KPI Investigation Methodology |

19 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

First Level Analysis

Deeper counter-based analysis is needed in order to characterize the degradation. The

21 | UA7.1.3 KPI Investigation Methodology 1

Consolidate CTg + CFT + NPO analysis

Consolidate global analysis

22 | UA7.1.3 KPI Investigation Methodology