ITU Regional Standardization Forum for Africa

(Kampala, Uganda, 23-25 June 2014)

Monitoring QoS/QoE of mobile

networks-novel approach

Tahitii Obioha
Network Planning and Optimisation Engineer
Planet Network International, France.

Kampala, Uganda, 25 June 2014

 Why Monitor QoS/ QoE of mobile networks
Monitoring = observing, checking and keeping a continuous
record of the progress or quality of something.
It is the use of any available technical tool to assess permanently
or for a given period of time a particular QoS parameter, e.g. call
set up time. (ETSI EG 202 009-2)
Reasons
For end-users : Satisfaction, get the best value for their money
amidst others
For operators : end-user Satisfaction, Increase revenue,
achieve customer loyalty, maintain competitive
edge amidst others. Surveys have shown that
about 90% of customers will not complain before defecting.
For Regulators: protect consumers, ensure end-user satisfaction,
create a level playing field for operators to compete amidst others.

They all share a common Objective CUSTOMER SATISFACTION

Kampala, Uganda, 25 June 2014 2
 QoE Measurement

Quality of Experience(QoE) has become an important

concept since the growth of mobile services.
QoE is very SUBJECTIVE in nature as it involves human
dimensions.
Measuring and improving QoE though challenging,
should be undertaken in order to assess the most
accurate and complete vision of the value offered by
the provider to end-users.
If the QoE is high, the user is happy, satisfied and loyal
while poor QoE will result in dissatisfied customers.

Kampala, Uganda, 25 June 2014 3

 QoE Measurement
QoE KPIs
USER EXPECTATIONS CAN BE GROUPED UNDER TWO MAIN CATEGORIES

1. RELIABILITY (Service quality of Accessibility and Retainability KPIs)

2. COMFORT (Service quality of Integrity KPIs)

Reliability QoE KPIs

QoE KPI Most important measures
Service Local and global coverage included
availability(anywhere) How seamless it is for the user?
Service The success rate of user connections for any
accessibility(anytime) service

Service access time The delays in setting up any service connection

Continuity of service The retainability of the service connection and its
connection-interruption performance overtime
ratio
Kampala, Uganda, 25 June 2014 4
 QoE Measurement

Comfort QoE KPIs

QoE KPI Most important measures
Quality of Session Jitter (delay variation %)
Average throughput(kbits/s) towards mobile
Bearer stability
Application layer packet loss ratio
Average end-to-end delay (ms)

Ease of use How easy is it to use service offered by the network

Level of support How quick and easy is it to get customer support

Kampala, Uganda, 25 June 2014 5

 QoE Measurement
Two pratical approaches to measuring QoE
Service level approach using Statistical Samples
•Approach relies on a statistical sample of the overall network users
the QoE for all the users in the network. (Mean Opinion Score (MOS))

Network Management system approach using QoS parameters

•Approach relies on using NMS to collect QoS KPIs from network
and comparing them with predefined target

Knowing full well that the holy grail of subjective measurements is to predict it from
the objective measurements i.e. predict QoE from a given set of QoS parameters.
NMS approach provides high accuracy than Service level because of its deterministic
and objective nature.
Using both methods nevertheless provides a better and more accurate picture of the
users’ experience
Kampala, Uganda, 25 June 2014 6
 Relationship between QoS and QoE

QoE = f(AQoS, NQoS) + other subjective factors (expectations, mood, etc.)

Strong Correlation between QoS and QoE

Relationship Between QoE and QoS may be near-linear for a voice call
but non-linear for a video call

Kampala, Uganda, 25 June 2014 7

 Relationship between QoS and QoE

There is an exponential relationship between QoS and QoE

End-to-end QoS is an important enabler for QoE in otherwords

a better network QoS in many cases will result in better QoE

QoE can be predicted from a given set of QoS paramete

vice versa (ETSI TS 102 250-1)

QoE may be improved with the right QoS mechanisms.

Relationship Between QoE and QoS may be near-linear for a voice call
but non-linear for a video call
Kampala, Uganda, 25 June 2014 8
 QoS Measurement
Quality of Service(QoS) arrived with 2nd Generation Networks
Ability of a Network to provide a service with an assured service
level.
It is intrisincally a technical concept, measured and understood in
terms of networks and network elements.
An objective metric that refers to the ability of the network to
achieve a more deterministic behaviour.
Service Qualification is based on six 6 primary components as defined
in ITU-T Rec E-800:
Operability
Accessibilty 85% Key Performance
Performance Indicators(KPIs)
Retainability
Integrity
Security Key Quality (Assurance)
Assurance 15% Indicators(KQIs)
Support

QoS can be measured using KPIs and KQIs

Kampala, Uganda, 25 June 2014 9
 Relationship between NP and QoS

Strong Correlation between NP and QoS

In otherwords Poor Network Performance

ultimately results to Poor QoS

Network related QoS

QoS aspects of service usage(ETSI TS 102 250-1)

Kampala, Uganda, 25 June 2014 10

 Relationship between NP and QoS

Qos Criteria

Network related QoS criteria Non network related QoS

85% criteria 15%
Mapping

Network Performance
Indicators/Parameters Target - range or limit
CALL DROP RATE < 2%
CALL SETUP SUCCESS RATE < 90%
…… ……
PARAMETER N <= x%

Kampala, Uganda, 25 June 2014 11

 QoS Measurement
QoS
NP Non-NP

Objectiv
Subjective
e
Active Passive

Non- Surveys, churn-

Intrusive rate, etc…
Intrusive

Walk/Drive OMC-R Ex.NMS: RPM System, Net Act,

imanager M2000 …
Test Around Counters
test using NMS
Ex. DT Tools: Nemo, XCAL, TEMS …
Relationship between NP,QoS and QoE

NP QoS & QoS QoE

NP is a subset of QoS which in turn, is a subset of the overall QoE scope
NP parameters ultimately determine the QoS
QoE can be predicted from a given set of QoS paramet
vice versa (ETSI TS 102
Kampala, Uganda, 25 June 2014 13
 QoS/QoE Assesement

TWO DIFFERENT AND COMPLEMENTARY APPROACHES TO MOBILE

QoS (ETSI EG 202 057-3)

Walk/Drive- round tests

Measurements based on network element counters using NMS

QoS Assesment Target Best Suitable QoS
Approach(es)

One-time snap shot DT

Acceptance Procedure DT/ NMS
Continuous Monitoring NMS
Optimisation Cycle DT + NMS

Monitoring :use of any available technical tool to assess

permanently or for a given period of time a particular QoS
parameter. (ETSI EG 202 009-2)
Kampala, Uganda, 25 June 2014 14
 Difference Between NMS and DT
Network Management System
QoS experienced by users are taken into account as
the calls are made with the real terminals by the real
users.
Representativity is assured as real traffic provides
measurements from the entire network 24h/7
Track performance down to the cell level
Provides comparabilty of Congestion and network
failures
Follow up on Operators’ network expansion, type of
equipment deployed and mobile evolution
NMS reports are actionable as it tracks the availability
of network elements (Ex. BSC/RNC)
Indepedent of enviromental factors
Traffic carried over the network and associated grade
of service (GoS) are obtained using this approach
KPI Build flexibility from numerous counters of NEs
Relatively no OPEX involved
Passive and non- intrusive, best approach in
continious QoS monitoring.
Kampala, Uganda, 25 June 2014
Drive Test (DT)
The QoS experienced is not taken into account by
users depends to some extent on the design of their
terminals which may differ somewhat from those used
for formal tests.
Drive test does not provide a representative of the
performance of the whole network because of
variation in user density and the complexity of
providing service in different areas. (60% of calls
made in a network are made indoor).
Performance stops at Base station level
Misses out on planned Site outages and network
failures.
Impossible to track Network Expansion using DT
DT reports are simply informative
Depedent on environmental factors such as weather
conditions, seasons, location, timing etc
DT has no information on Traffic carried over a given
network.
Limited KPI observation as it measures the network
from an external point.
High OPEX as well as time and Labor intensive
Active and intrusive, best approach for QoS
(coverage) benchmarking purposes.
15
 Draw backs of DT illustrated

Kampala, Uganda, 25 June 2014 16

 Draw backs of DT illustrated

Kampala, Uganda, 25 June 2014 17

 Draw backs of DT illustrated

Kampala, Uganda, 25 June 2014 18

 Quest for a Monitoring Tool
Monitor simultaneously 24H/7 the state of all network elements/acess technologies
in an operator network passively without compromising their daily operations
Put a check on service degradation and sometimes outages to the benefit of the
government and the masses.
Quest for a system that can monitor the performance of mobile operators
independent of the vendor and technology 
Flexible Benchmark Audit; weekly, monthly or quarterly.
Track the evolution and expansion of the mobile operators network.
Reduce churn rate of end-users directly or indirectly
A robust  system that is easily upgraded to meet the rapid evolution of mobile
technology.
A platform that will facilitate  actionable decision making to the benefits of all
parties involved with little or no OPEX.
Create a level playing field for all operators and protect the interest of all
consumers not just those in the urban areas.

Kampala, Uganda, 25 June 2014 19

 Network Management Systems

EXAMPLES OF NMS solutions in the market today

For REGULATORS (Vendor-Independent)

Regulators (QoS)Performance Management (RPM)system- compatible

with all major network vendors (Huawei, ZTE, Ericcsson, Alcatel and NSN)

FOR OPERATORS (Vendor –dependent)

imanager M2000 for Huawei vendor

NetNumen U31 for ZTE vendor
NEtAct for Nokia Siemens Vendor

NetOp EMS for Ericsson and Omnivista 2500 for Alcatel vendor

Kampala, Uganda, 25 June 2014 20

 RPM SYSTEM OVERVIEW
Regulators’ (QoS) Performance Management System is the novel NMS solution that
handles the task of interfacing all the operators’/service providers’ network
monitoring systems, collect perfomance data records and create KPI reports that
renders a given network performance against published benchmarks.
Main Features
• QoS monitoring and reporting 24/7 all year round
• Unlimited number of Users/ Single License
• Multi vendor Multi access technology
• Automatic Reporting and 3D GIS
display(compatible with google earth
• Friendly and intuitive web user interface

• No running costs and upgrades easily as mobile

access technology evolves
• Built in compliance with ETSI TS 102 250-4&7/
ETSI EG 202 057-3 and ETSI EG 202 009-2
Kampala, Uganda, 25 June 2014 21
 Conclusion
The assessment of the QoS/QoE is expected to be evaluated in
checking criteria against reference values. These criteria are
measured either objectively via technical means or subjectively via
surveys amongst the users. Experts agree that a mix of
objective(intrusive and non-intrusive) and subjective measurements
remains the best means to get the whole QoS picture.

Today, four out of five Regulators only employ Drive Test(intrusive)

both for one-time snap shot, benchmarking and continuous QoS
monitoring(reactive sense). Given DTs drawbacks and inaccuracy
with respect to continuous QoS monitoring, its high-time Regulators,
gaining from operators’ perspective, adopted the novel, practical and
more efficient approach – the use of Network Management Systems
(NMS) for proactive continuous QoS monitoring.

Kampala, Uganda, 25 June 2014 22

 Recommendations
Regulators, as recommended in ETSI EG 202 009-1
clause 7, in the quest for effective continuous QoS
monitoring and assessment should perform the
measurements themselves and make the results
available publicly rather than entrust in mobile
operators to provide the QoS information

Thus Regulators should procure RPM system or any

other NMS that facilitates the complex task of
enforcing mobile operators to adhere to agreed
KPIs, put a check on service degradations /outright
outages and even penalities for repeated failure to
achieve minimum QoS/ QoE targets.

Kampala, Uganda, 25 June 2014 23

 THANK YOU FOR YOUR
ATTENTION
For more information on RPM
system or any other NMS
Feel free to contact me
or email
cblanchard@planetworkint.com

Kampala, Uganda, 25 June 2014 24