ITU Workshop on

“Workshop on Practical measurement of QoS/QoE

Parameters for Regulatory Compliance”
(Cotonou, Benin, 16-17 July 2012)

Equipment for Testing Mobile QoS

Joachim Pomy, SG 12 Rapporteur

Consultant, Opticom GmbH
Consultant@joachimpomy.de

Cotonou, Benin, 16-17 July 2012

 [Content]
ITRs: Setting the stage for
a connected world

Equipment for Testing Mobile QoS

Measurements
Test Scenarios
Trigger Points
Test Equipment
Testing for various Services
Practical Hints
POLQA - Voice Quality - Hot Topic
 Measurement of QoS in Mobile

Measured from the perspective of an

expert
Assessing the network to see if it delivers
the quality that the network planner
targeted (cf. Rec. ITU-T G.1000)
Tools and Methods for Operation and
Maintenance
Protocol analysers are connected to BTSs,
BSCs, and MSCs for a period of time to check
for problems in the cellular network
Drive test (also walk tests) are used for
monitoring operationally achieved QoS
Measurement of QoS in Mobile

Drive tests allow the mobile network to be

tested through the use of a team of people
who take the role of users
Rate the QoS of the network
Not applicable to the entire network
Always a statistical sample
In Operation and Maintenance Centres (OMCs),
counters are used for various events
Providing the network operator with vendor specific
information on the state and quality of the network
Also, customer complaints are a vital source of
feedback on the QoS, and must not be ignored
End-to-end QoS Testing in Mobile

Minimum requirement for QoS

measurement equipment to ensure that
for all QoS parameters
trigger points are well defined
values are determined following wqell defined
procedures
Test-equipment fulfilling the specified
minimum requirements
Will allow performing the proposed
measurements
In a reliable and reproducible way
 Parameters are based on Field
Measurements

Measurements from customers point of view

Full End-to-End perspective, taking into account needs of
testing
Assumed end customer can handle his equipment and
services to use
Operability is not evaluated at this time
For the purpose of measurement it is assumed that
service is not barred for any reason
routing is defined correctly without errors
target subscriber equipment is ready to process service request
speech and video quality values measured
Only to be applied to calls ended successfully
In order to receive proper statistical analysis
 Not yet any ITU Standards in Place

ETSI TS 102 250 part 4

Speech and multimedia Transmission
Quality (STQ); QoS aspects for popular
services in GSM and 3G networks; Part 4:
Requirements for Quality of Service
measurement equipment
globally recognized as "the requirement"
in the area
extremely compehensive
detailed instructions
 [Content]
ITRs: Setting the stage for
a connected world

Equipment for Testing Mobile QoS

Measurements
Test Scenarios
Trigger Points
Test Equipment
Testing for various Services
Practical Hints
POLQA - Voice Quality - Hot Topic
 Test Scenarios

Test scenarios need to distinguish the

following principal user cases
User-to-user services (typically
telephony)
Store-and-forward services (e.g. SMS)
Information services (e.g. accessing the
internet or FTP download)
Push services (sender initiated transfer)
 [Content]
ITRs: Setting the stage for
a connected world

Equipment for Testing Mobile QoS

Measurements
Test Scenarios
Trigger Points
Test Equipment
Testing for various Services
Practical Hints
POLQA - Voice Quality - Hot Topic
Considerations on trigger points
It can be assumed that
any feasible test equipment will contain some kind of
communication terminal (UE) which may be a special
type (e.g. a Trace Phone)
or a standard UE
Also, it can be assumed that each such device will
provide information from different communication
layers
from Application Layer (close to the user interface)
down to lower layers, e.g. operating-system events,
TCP/IP layer, or Layer 3 signalling information, which is
used as trigger points for QOS PARAMETERS processing
Typically AT commands can be used in majority of cases
Action is typically triggered
by some emulated user action
causing some action on the air interface
Considerations on trigger points
Process of event propagation is deterministic
allowing for kind of mapping between layers
associated with communication and processing delay in each stage.
Choice of the layer for trigger point information retrieval
determines the view expressed in a QOS PARAMETER
choosing lower-level events ( Layer 3) gives more network-centric
view
events on higher levels to produce views more user-related
for same QoS PARAMETER source layer for events used as trigger
points should be the same
In benchmarking, for all all networks under test
use same type of UE, and QOS PARAMETERS
use trigger points from the same layer.
Changing the source layer for a given trigger point
changes in QOS PARAMETERS should be expected
calibration measurements required
to assess influence on QOS PARAMETERS both quantitatively and
qualitatively.
 [Content]
ITRs: Setting the stage for
a connected world

Equipment for Testing Mobile QoS

Measurements
Test Scenarios
Trigger Points
Test Equipment
Testing for various Services
Practical Hints
POLQA - Voice Quality - Hot Topic
 Mobile QoS Test-equipment (MQT)

Geographical data collection during a

mobile measurement
Position, speed and heading
Taken from GPS whenever possible.
If no GPS available use other tools
e.g. navigation on a geo-referenced bitmap
 [Content]
ITRs: Setting the stage for
a connected world

Equipment for Testing Mobile QoS

Measurements
Test Scenarios
Trigger Points
Test Equipment
Testing for various Services
Practical Hints
POLQA - Voice Quality - Hot Topic
 Telephony Voice

Means to generate necessary calls or to

answer automatically incoming calls
Received speech sample is measured
via the analogue output of the User Equipment
hardware setup to include necessary electrical
adaptation
Hardware setup of such a system is
essential for the correctness of the
measured QoS Parameters
Includes a UE specific part for the electrical
adaptation
audio output of handset to input of the soundcard
 Video Telephony

Means to generate necessary calls or to

answer automatically incoming calls
Capability to provide the following
measurement methods.
Video Telephony using the User Equipments
application; or
Video Telephony using a Video Telephony
Stack on the PC.
 MQT for FTP

FTP client to support the following

functions:
Active / Passive Mode
Common Firewall support
Downloading from subdirectories.
Uploading to subdirectories
Mobile based measurement equipment

Functionalities of the MQT can also

be realised on a single mobile phone
Different types of QoS tests can run
Test results can be logged for post
processing
Mobile based measurement equipment
may be controlled
by an application, remotely or
may be operated by a human
 [Content]
ITRs: Setting the stage for
a connected world

Equipment for Testing Mobile QoS

Measurements
Test Scenarios
Trigger Points
Test Equipment
Testing for various Services
Practical Hints
POLQA - Voice Quality - Hot Topic
 Practical Hints

Which Mobile Phone to use ?

should be one offered in the shop ?
should be one used by new customers ?
should be one hipe to young people ?
should be one with good QoS ?
BUT
it MUST be a so-called Trace Mobile
ASK YOUR PHONE PROVIDER !!!
they are RARE !!!!
 [Content]
ITRs: Setting the stage for
a connected world

Equipment for Testing Mobile QoS

Measurements
Test Scenarios
Trigger Points
Test Equipment
Testing for various Services
Practical Hints
POLQA - Voice Quality - Hot Topic
 Hot Topic:
POLQA™ - Rec. P.863
 PESQ POLQA
Acoustic measurements  Not easy 
Correct scoring with high background
noise  
AMR vs EVRC codec comparison
 
Representative scoring of reference
signals  
Effects of speech level in samples
 
Narrowband (300Hz -3400Hz)
 
Wideband (100Hz-7000Hz)
 Use SWB
Superwideband, SWB
(50Hz – 14000Hz)  
Linear Frequency distortion sensitivity
 

POLQA Introduction - (c) OPTICOM GmbH 2010 30

 What we Perceive …
In a subjective ACR experiment POLQA, PESQ and
human beings perceive the following distortions:

Factor Human POLQA PESQ

Level too high or too low X X no

Strong linear filtering X X no

Noise in the reference signal X X no

High timbre in reference X X no

Level variation X X poor

SWB noise on NB/WB signal X X no

POLQA Introduction - (c) OPTICOM GmbH 2010 31

Performance Validation
• The ITU has validated POLQA on:

• 47000 file pairs across

• 64 subjective experiments
• Languages included in the POLQA validation:

• American English and • German

British English • Swiss German
• Chinese (Mandarin), • Italian,
• Czech, • Japanese,
• Dutch, • Swedish
• French,

POLQA Introduction - (c) OPTICOM GmbH 2010 32

Any questions ?

Contact:
Consultant@joachimpomy.de