You are on page 1of 38

Benchmarking Report

Template

Contents
Executive Summary
Summary of City
Benchmarking Details
Data Collection Project Summary
Drive Route
Test Overview
Voice Tests
Data
Strategic Locations
Voice & Data Quality Weightage
Performance Ranking

Executive Summary

<< narration of the Benchmarked networks


and the geographical area / city ,
demography etc. by the service delivery
organization >>

Data Collection project Summery

<< summary of the benchmarking data


collection project, drive routes, distance
covered etc. by the service delivery
organization >>

Drive Route

<< Drive route plot from PILOT Fleet >>


<< explanation of the data collection drive
routes, clusters etc. by the service delivery
organization >>

Test Overview-Voice
Continuous call for 3 minute duration with 10 seconds of
idle time. Speech samples (DL and UL) are sent and
received which provide the MOS score for voice quality.

Test Overview-Data
Number of Operators Tested
<< operator list >>
Data Test Alogorithm
Test Details
Specifications
Attach & PDP Activation
Before every Test
Ping
3 times - 32 bytes on www.google.co.in
FTP Download
10 Mb for EvDo and 15 Mb for UMTS from server
FTP Upload
1 Mb for EvDo and 1 Mb for UMTS to server
HTTP
10 different websites
Email send
100 Kb Email to SMTP server
Email Receive
100 Kb Email from POP3 server

No.
1
2
3
4
5
6
7
8
9
10

HTTP Wbsites
www.google.com
www.facebook.com
www.numbuzz.com
www.airtel.in
www.cricinfo.com
www.vuclip.com
www.youtube.com
www.admob.com
www.zedge.net
www.nokia.com

<< The above test cases are created in PILOT


Fleet to perform the data benchmarking tests >>

Test Overview-Strategic Locations


Number of Operators Tested
<< operator list >>
Test Details
WAP Test
FTP Download
Email Send
Email Receive
SMS
MMS

Data Test Alogorithm


Specifications
10 different WAP sites
10 Mb for EvDo and 15 Mb for UMTS from server
100 Kb Email to SMTP server
100 Kb Email from POP3 server
SMS send to same number (160 charectors)
MMS send to same number 15 Kb

No.
1
2
3
4
5
6
7
8
9
10

WAP Sites
wap.google.com
m.facebook.com
m.orkut.com
wapin.com
m.cricinfo.com
m.vuclip.com
m.youtube.com
m.admob.com
www.m.zedge.net
m.sonyericsson.com

<< The above test cases are created in PILOT


Fleet to perform the data benchmarking tests >>
<< explanation of the demography of the
strategic locations tested by the service
delivery organization >>
8

Voice & Data KPI Weightage

Coverage Calculation = (A%*10+ B%*8 + C%*6 +D%*1) / 10


Grade RSCP in dBm
A : 0 to -74
B : -74 to -82
C : -82 to -90
D : -90 to -98

Performance Ranking
Performance
Bad
Poor
Average
Fair
Good
Excellent

CST Rank Performance


15.0 0 Bad
12.0 2 Poor
10.0 4 Average
5.0 6 Fair
3.0 8 Good
2.0 10 Excellent

CSSR Rank Performance


90% 0 Bad
92% 2 Poor
94% 4 Average
96% 6 Fair
98% 8 Good
100% 10 Excellent

CCSR Rank Performance


90%
0 Bad
92%
2 Poor
94%
4 Average
96%
6 Fair
98%
8 Good
100% 10 Excellent

MOS Rank Performance Coverage Rank


1.0
0 Bad
90%
0
2.0
2 Poor
92%
2
2.8
4 Average
94%
4
3.5
6 Fair
96%
6
4.0
8 Good
98%
8
5.0 10 Excellent
100%
10

10

8
6
4
2

<< the network performance Ranks are to be


calculated using KPI weightage & above ranking
methods and presented by the service delivery
organization >>

10

Contents-Voice Benchmarking

Voice Quality Rank


Data Quality Rank
Voice Benchmarking
Call setup time.
Call Setup Success rate.
Call Completion rate.
Call Completion Success rate comparison with CCR.
Voice quality UL & DL MOS.
Coverage
Plot of UL MOS.
Plot of DL MOS.
Voice Codec Usage
Handover Performance
Dropped call analysis

Voice Quality Ranking

10

CST

10

CSSR

10

CCSR

10

MOS

10

COV

10

TOTAL

<< Voice quality ranks are calculated and


presented in graphical /table form by the service
delivery organization, along with review notes for
each KPI and networks tested >>
12

Data Quality Ranking

10

eMail

10

HTTP

10

SMS

10

MMS

<< Data quality ranks are calculated and


presented in graphical /table form by the service
delivery organization, along with review notes for
each KPI and networks tested >>
13

Data Quality Ranking

10

Attach

10

PDP
Context

10

FTP-DL

10

FTP-UL

10

Ping

10

TOTAL

<< Data quality ranks are calculated and


presented in graphical /table form by the service
delivery organization, along with review notes for
each KPI and networks tested >>
14

Call Setup Time (CST)

<< Call setup time (CST) minimum and average


graph and data table from PILOT Fleet

<< Result analysis and review notes by the


service delivery organization >>

15

Call Setup Success Rate (CSSR)

<< Call Setup Success Rate (CSSR) minimum and


average graph and data table from PILOT Fleet

<< Result analysis and review notes by the


service delivery organization >>

16

Call Setup Failure Plot

<< Call Setup Failure Plot from PILOT Fleet


<< Result analysis and review notes by the
service delivery organization >>
17

Call Completion Rate

<< Call Completion Rate graph from PILOT Fleet

<< Dropped call analysis and review notes by the


service delivery organization >>
18

Call Completion Success Rate


Vs. Call Completion Rate

<< Call Completion Success Rate comparison with


CCR graph from PILOT Fleet
<< Result analysis and review notes by the
service delivery organization >>
19

Dropped Call Plot

<< Dropped Call Plot from PILOT Fleet


<< Result analysis and review notes by the
service delivery organization >>

20

Voice Quality - MOS

<< MOS UL & DL scores graph from PILOT Fleet

<< Result analysis and review notes by the


service delivery organization >>
21

MOS Plot

<< MOS UL & DL plot from PILOT Fleet


<< Result analysis and review notes by the
service delivery organization >>
22

Receive Level

<< Receive Level plot from PILOT Fleet


<< Result analysis and review notes by the
service delivery organization >>
23

Coverage Plot

<< Coverage plot from PILOT Fleet


<< Result analysis and review notes by the
service delivery organization >>
24

Codec Distribution

<< Codec distribution graph from PILOT Fleet


<< Result analysis and review notes by the
service delivery organization >>

25

Handover Performance
Handover Details

Value

WCDMA soft handover(Num)


WCDMA soft handover failure(Num)
WCDMA soft handover success rate (%)
GSM handover(Num)
GSM handover failure(Num)
GSM soft handover success rate (%)
WCDMA -> GSM handover(Num)
WCDMA-> GSM handover failure (Num)
WCDMA -> GSM handover success rate (%)
WCDMA handover rate (%)
GSM handover rate (%)
WCDMA -> GSM handover rate (%)

3222
0
100%
203
1
99.51%
22
0
100%
93.50%
5.86%
0.64%

<< Handover performance data and plots from


PILOT Fleet
<< Result analysis and review notes by the
service delivery organization >>
26

Dropped Call Analysis

<< Dropped call analysis from PILOT Navigator

<< Result analysis and review notes by the


service delivery organization >>
27

Contents-Data benchmarking

FTP Performance

FTP DL & DL

HTTP Performance.
Email Performance
GPRS Attach & PDP Context
SMS
MMS
Technology distribution
Modulation distribution
RLC Throughput
CQI Distribution
MCS Distribution

FTP Performance

<< FTP performance graphs from PILOT Fleet


<< Result analysis and review notes by the
service delivery organization >>
29

HTTP Performance

<< HTTP performance


graphs from PILOT
Fleet >>

<< Result analysis and


review notes by the
service delivery
organization >>

30

Email Performance

<< Email performance graphs from PILOT Fleet


<< Result analysis and review notes by the
service delivery organization >>

31

Attach & PDP Context Performance

<< Attach & PDP Context performance graphs


from PILOT Fleet
<< Result analysis and review notes by the
service delivery organization >>

32

Technology Distribution

<< Technology distribution plot from PILOT Fleet


<< Result analysis and review notes by the
service delivery organization >>

33

Modulation Distribution

<< Modulation distribution charts from PILOT Fleet


<< Result analysis and review notes by the
service delivery organization >>

34

RLC DL Throughput

<< RLC throughput plots from PILOT Fleet


<< Result analysis and review notes by the
service delivery organization >>
35

CQI Distribution

<< CQI distribution plots & graph from PILOT Fleet


<< Result analysis and review notes by the
service delivery organization >>
36

MCS Distribution

<< MCS distribution chart from PILOT Fleet


<< Result analysis and review notes by the
service delivery organization >>
37

38