Voice Quality Optimisation

Velizy, FR
15.03.2018

Salvatore Piazzolla
MNT, Switzerland
Confidentiality

Terms of Release of Non-Binding Confidential Information

This document contains confidential, forward-looking, non-binding product statements, concepts and
information for discussion only. These concepts are to be regarded as work-in-progress, and nothing
contained in this document is intended to constitute a formal or informal commercial or technical offer.
No warranty regarding commercial availability, specification or other aspects of functionality is expressed
or implied by this document. Rohde & Schwarz reserves the right to amend any and all details relating to
these concepts at any time without notice or recourse. Any products that might eventually be developed
from these concepts may vary substantially and materially from those expected, anticipated, intended,
planned, believed, sought, estimated or projected herein. Rohde & Schwarz does not intend or assume
any obligation to update or revise these forward-looking statements, concepts or information in light of
developments which differ from those currently anticipated. All information in this document is regarded
as commercial valuable, protected and privileged intellectual property, and is provided under the terms
of existing Non-Disclosure Agreements.
Our areas of technology expertise
Engineering & Design
ı Decades of experience in designing and engineering
products for Mobile Network Testing

Testing true QoS & QoE

ı End-user device-based testing concepts
ı Stable test environments for fully reproducible, validated
and accurate results
ı The most complete range of test scenarios and KPIs

Internal applied research

ı Defining new standards for ITU-T, ETSI and VQEG

3
Innovation from RF to QoE testing
ı The first scanner measuring eMBMS, MIMO
4x4 and 5G

ı Our POLQA algorithm selected in 2010 as the

new world standard for perceptual voice
ı Unique audio delay measurements for VoLTE
testing
ı The new world standard ITU (J343.1) for
video MOS

ı The only solution measuring true RF

performance

4
VoLTE-VoWiFi-WhatsApp-etc.
How are we with the old, good voice?
How about VoLTE
News and introduction

ı VoLTE testing in Germany (08-10

2016)
 Germany, August 2016, Sony Z5, Drive 1
 66% VoLTE-VoLTE (out of 7‘500 calls)

 Germany, October 2016, Sony Z5, Drive 2

 67% VoLTE-VoLTE (out of 8‘000 calls)

 Both sides are mobile and moving

PA * PB = PVoLTE (~80% VoLTE at one side)

6
 Voice over LTE – Large scale real field results
Call statistics

ı VoLTE has a significant

lower Call Setup Time
ı CS has lower voice
quality (at this time)

VoLTE
Call CSFB /
-
status CS
VoLTE

Avg.
Setup 6.03 s 10.78 s
Time
Avg.
3.85 3.35
MOS

7
Voice over LTE – Large scale real field results Max MOS
AMR 23.85
Speech Quality - Discussion
40000

Normal situation: 35000

ı CS and VoLTE-to-CS apply AMR-WB 12.65 kbit/s 30000

Max MOS
ı VoLTE-to-VoLTE applies AMR-WB 23.85 kbit/s AMR 12.65
25000

Number of values
ı Under perfect network conditions 20000
AMR NB
ı AMR-WB 23.85 (avg. 3.85) best MOS: 4.15…4.20
ı AMR-WB 12.65
15000
(avg. 3.35) best MOS: 3.95…4.00
10000
Today’s evolvement:
ı CS-to-CS (3G) apply AMR-WB 23.85 kbit/s too 5000

0
1.0 ‐ 1.1 1.5 ‐ 1.6 2.0 ‐ 2.1 2.5 ‐ 2.6 3.0 ‐ 3.1 3.5 ‐ 3.6 4.0 ‐ 4.1
P.863 score

Voice-wise there is almost no difference

between CS and VoLTE anymore
VoLTE/VoLTE CS/CS

8
How about ‘WhatsApp Call’
Is it a serious competition to VoLTE?

ı WhatsApp Voice Call available since begin 2016

ı It is a free OTT VoIP service

ı WhatsApp: Drive test in Rome area

 October 2016, Samsung S6
 ~250 VoLTE / ~250 WhatsApp Calls
 Good network conditions!

9
Real field results Rome Oct. 2016
WhatsApp vs. VoLTE – Speech Coding and Quality
VoLTE
~4.20

ı VoLTE-to-VoLTE AMR-WB 23.85 kbit/s 70%

ı WhatsApp Opus 20 kbit/s? …wideband 8kHz VoLTE Rome

60%
WhatsApp Rome

50% WhatsApp
~4.35
ı Speech Quality Average (good network conditions)
40%
ı VoLTE avg. MOS: 4.03
ı WhatsApp avg. MOS: 4.02 30%
WhatsApp
Very bad calls
20%

Is that the whole story? 10%

0%

1.0
1.2
1.4
1.6
1.8
2.0
2.2
2.4
2.6
2.8
3.0
3.2
3.4
3.6
3.8
4.0
4.2
4.4
4.6
4.8
5.0
MOS (P.863 superwideband)

10
Real field results Rome Oct. 2016
WhatsApp vs. VoLTE – CST and Reliability
VoLTE
~4.20

ı WOW! 70%
 Slightly higher speech quality under good conditions VoLTE Rome
60%
 CallSetupTime: 5.4s (VoLTE Rome: 7.3s) WhatsApp Rome

50% WhatsApp
ı But ~4.35

 ~3% Failed / Dropped (VoLTE < 1%) 40%

 ~3% very bad calls (VoLTE 0.0%)

30%
WhatsApp
 ~3% missed voice (VoLTE <0.2%) Very bad calls
20%

10%

WhatsApp is still a bit less reliable but 0%

very close!

1.0
1.2
1.4
1.6
1.8
2.0
2.2
2.4
2.6
2.8
3.0
3.2
3.4
3.6
3.8
4.0
4.2
4.4
4.6
4.8
5.0
MOS (P.853 superwideband)

11
Real field results Rome Oct. 2016
WhatsApp vs. VoLTE – Uncompensated Frame Jitter

ı WhatsApp uses a default bearer and ‘common’ IP

ı Leads to higher frame jitter
70%

VoLTE Rome
ı Some jitter cannot be compensated by the jitter buffer 60%

 Audio signal gets warped 50% WhatsApp Rome

 Measurable as ‘variable audio delay’
40%

30%

20%

10%

0%
0 - 20 20 - 40 40 - 60 60 - 80 80 - 100 - >120
100 120
Delay variation / ms

12
How is the relation to ‘VoWiFi’
Is it different to VoLTE or to WhatsApp?

ı Applies and uses the VoLTE Call Flow

ı Uses a common IP-connection,
prioritization depends on host network

ı Static tests in German WiFi hotspot

 November 2016, Samsung S6
 ~50 VoWiFi Calls
 Good network conditions!

13
Real field results 2016
VoWiFi vs. VoLTE vs. WhatsApp VoWiFi
~4.10
VoLTE
~4.20

ı VoLTE-to-VoLTE AMR-WB 23.85 kbit/s

ı WhatsApp Opus 20 kbit/s? …wideband 8kHz
ı VoWiFi-to-VoWiFi AMR-WB 12.65 kbit/s (??)
WhatsApp
~4.35
ı Speech Quality Average (good network conditions)
ı VoLTE avg. MOS: 4.03
ı WhatsApp avg. MOS: 4.02 VoWiFi
ı VoWiFi avg. MOS: 3.95 Medium range calls

Is that again the whole story?

14
Real field results 2016
VoWiFi vs. VoLTE vs. WhatsApp

ı VoWiFi (the same as WhatsApp) uses

a default bearer or ‘common’ IP
ı Leads to higher frame jitter

15
Real field results 2016
VoWiFi vs. VoLTE vs. WhatsApp VoWiFi
Summary ~4.10
VoLTE
~4.20

ı VoLTE has very good ‘base quality’ and almost no

degrading factors
ı VoWiFi is almost as good as VoLTE
ı WhatsApp is superior in ‘base quality’ but may have very WhatsApp
bad samples from time to time ~4.35

ı WhatsApp has no managed SVRCC but stays connected

in case of 4G -> 3G handover
ı WhatsApp is easy to use and has a large user base

WhatsApp is very close to VoLTE and a very

serious competition to legacy telephony,
much more than Skype ever was!

16
Addition: Real field results Netherlands Oct. 2017
WhatsApp vs. VoLTE – Quality, CST and Reliability

ı WOW!
 Speech Quality Average (good network conditions)
ı VoLTE avg. MOS: 3.65
ı WhatsApp avg. MOS: 4.20
ı Although WhatsApp shows reliability issues, it is very close
to VoLTE

ı The lack of EVS definitely puts VoLTE quality behind most

advanced OTT
Addition: Real field results Netherlands Oct. 2017
WhatsApp vs. VoLTE – Quality, CST and Reliability
Telephony over IP – What’s next?

ı VoLTE (VoWiFi, CS too) will introduce EVS coding ı VoLTE is still more reliable
 Audio bandwidth default 14…16kHz
 Quality is superior to AMR-WB and Opus (MOS 4.5…4.6) ı VoLTE is easy to use
 WhatsApp may improve too in the future (?) (others will get close)

ı Many, many other free voice clients ı VoLTE has more subscribers
 Traditional SIP call providers (interconnection with CallApps)
 ‘Closed’ Apps as WhatsApp, Skype, Viber, Line,…

ı Differentiation by pricing?
ı VoLTE will extend to video telephony (ViLTE) …or by privacy?

 Skype, WhatsApp, Viber, Line have it already

19
 And now EVS codecs
ı Support of the new full-HD
voice (EVS) codecs recently
introduce
ı Benchmark of EVS vs. legacy
and WB-AMR codes
ı Advanced voice capturing
needed on test device to
cope with EVS
ı POLQA (ITU-T P.862) voice
MOS assessment
 Average MOS >4.6!
More info at: http://www.mobile-network-testing.com/en/detail/rs-mobile-
network-testing-and-net-check-are-first-to-test-new-evs-codec-in-live-network-/

20
How can we improve the voice quality?

…May 2017
Voice over LTE – Codecs and bitrates
Speech Quality – Introduction by example data

 CS (2G / 3G) AMR-WB 12.65

kb/s
 Some vendors: AMR-WB 23.85 kb/s
 Fallback AMR 12.2 (or lower)
Max MOS
 VoLTE AMR-WB 23.85 Max MOS EVS 24.4
AMR 23.85
kb/s
 Some MNOs apply AMR-WB
15.85, 12.65 Max MOS
AMR 12.65
 Some MNOs apply EVS 24.4
AMR NB
kb/s

 Under perfect network conditions

 AMR 12.2 best MOS: 3.20…3.30
 AMR-WB 12.65 best MOS: 3.95…4.00
 AMR-WB 23.85 best MOS: 4.15…4.20
 EVS 24.4 best MOS: 4.55…4.65

22
Voice over LTE – Codecs and bitrates
Speech Quality – Introduction by example data

Under perfect network conditions:

 AMR 12.2: best MOS: 3.20…3.30 Same data rate, but
 AMR-WB 12.65: best MOS: 3.95…4.00 different MOS, different QoE

 AMR-WB 23.85: best MOS: 4.15…4.20 Same data rate, but

 EVS 24.4: best MOS: 4.55…4.65 different MOS, different QoE

Bitrate ≠ Voice quality, but codec type is decisive

23
Voice Quality Analysis
Vodafone (avg. MOS: 4.10)

EVS

AMR-WB

AMR-NB
 Real field data Germany
 Mobile – to – Mobile
 ~50% LTE, ~45% 3G, ~5% 2G

 Under perfect network conditions

AMR-WB AMR-WB
 AMR 12.2 best MOS: 3.20…3.30
T-Mobile (3.93) Telefonica (3.82)
 AMR-WB 12.65 best MOS: 3.95…4.05
AMR-NB
 AMR-WB 23.85 best MOS: 4.15…4.25
 EVS 24.4 best MOS: 4.55…4.65

24
Voice Quality Analysis
VFD avg 12.65 3.92 3G (+ 2G)
VFD avg EVS 24.4 4.28 LTE

TMO avg 12.65 3.82 3G (+2G)

TMO avg 23.85 4.07 LTE

TEF avg 12.65 3.86 3G (+ 2G)

Limitation by (codec)
TEF avg 23.85 3.93 LTE technology

 Today VFD has the advantage of EVS

(technology profit)
 TEF has still considerable AMR-NB
fallbacks
 In general networks are good
 Technology dominates
Network issues

25
Voice Quality Analysis
How to improve?

Technology
 Severe quality issues
 Inter-RAT and Inter-FQ handovers
 RTP Loss / Delay in backbone
 Bad RF condition (of course)

Non-optimal
 Non-optimal transmission Severe Quality Issues transmission

 Uncompensated RTP Jitter (audio

distortions)

Technology improvement
Codec / Bitrate
Transcoding free operation
Use of EVS IO mode

26
Voice Quality Analysis
How to improve? Severe quality issues

 Check individual cases of inter-RAT calls

- Improvement of SVRCC interruption duration
- Avoid transcoding
- Use of EVS IO mode

- Examples of EVS 24.4 -> AMR 12.2 (in 3G)!

- Examples of EVS 24.4 -> AMR-WB 12.65

- On LTE side EVS stays in 24.4
(and network applies transcoding)
 Use of EVS IO mode if possible

27
Voice Quality Analysis
How to improve? Severe quality issues

 Check individual cases of inter-FRQ HO

- Individual cases with >500ms interruption

 Distortion in radio transmission

LTE Band 3 -> Band 20

MOS = 1.65

Sample direction B->A

 RTP loss visible at A-side!

TransportBlocks requested
for re-transmission ≥3 times

28
Voice Quality Analysis
How to improve? Severe quality issues

 RTP Jitter and Loss

- Heavy RTP Loss
 No TransportBlocks lost in Downlink
 BUT: Heavy loss of TransportBlock in Uplink
at sending side

MOS = 1.15

Downlink
Near-End Uplink
Sample direction A->B Far-End
 RTP loss visible at B-side!

29
Packet-switched Voice Transmission
How to deal with packet jitter?

o Circuit Switched: Fix-scheduled speech frames

o Packet Switched: Packet / frame Jitter

o Jitter Buffer for compensation

30
Packet-switched Voice Transmission
How to deal with packet jitter?

o In case the buffer runs empty, the first missing packets are ‘artificially constructed’, then audio pauses (perceptible)
o Whenever possible, natural pauses are used for warping

o Audio gets ‘re-warped’ usually be shortening natural pauses later on

31
Packet-switched Voice Transmission
How to recognize packet jitter / loss on lower layers?

RTP Loss / Jitter RTP Loss / Jitter refers to

refers to peer-to-peer peer-to-peer
 can be the transcoder

TrFO  TransCoding 

Core Network Core Network

UE TransportBlock Statistics only UE UE UE

refer to actual airlink

32
Voice Quality Analysis
How to improve? Non-optimal

 To measure at audio-side  Audio signal

warping (delay spread)
 This is ‘uncompensated RTP jitter’

 EVS compensates less RTP Jitter than

AMR-WB

 uncompensated audio delay

spread in LTE
(statistically the same RF / network)
AMR-WB (LTE): ~30ms
EVS (LTE): ~50ms
Minimizing RTP jitter (very generic task)
Changing EVS implementation
(increasing jitter buffer by one or
two frames)

33
Voice Quality Analysis
How to improve? Non-optimal

 Small cause, big issue

 1 transport-block is coming too late
(≥3 retransmissions)
 Packet (Frame) gets not lost finally
(RTP Loss: 0%) but is too late

MOS = 1.56

34
Voice Quality Analysis
How to improve? Non-optimal

 Small cause again

 1 DL frame is coming too late
(≥3 retransmissions)
 Frame gets not lost finally
(RTP Loss: 0%), but is too late

MOS = 2.71

35
Voice Quality Analysis
How to improve? Non-optimal

 1 RTP-packet (frame) gets lost

 Radio in UL and DL are ok

MOS = 3.75 (EVS 24.4!)

36
Voice Quality Analysis
How to improve? Technology

+ 0.12 for 3G
 Biggest improvement: AMR-WB 23.85 in 3G
- Requires Huawei infrastructure
- Competitors may do this

 Would improve 3G and 3G – 4G calls

37
Voice Quality Analysis
How to improve? Technology

AMR-WB 12.65 AMR-WB 23.85

 AMR-WB Details (normed @ y-axis)

Good position at 12.65

VFD avg 12.65 3.92 3G (+ 2G)

VFD avg EVS 24.4 4.28 LTE

TMO avg 12.65 3.82 3G (+2G)

TMO avg 23.85 4.07 LTE

TEF avg 12.65 3.86 3G (+ 2G)

TEF avg 23.85 3.93 LTE

Would improve to
~4.3 if EVS is used

38
Voice Quality Analysis
How to improve? Technology

VoLTE at A-side
 Minimize 4G – 3/2G calls
- AMR-WB (today 12.65) determines the MOS
- Use of EVS IO mode

- Dependency on campaign setup!

 geographically separated A and B side
will increase ratio of Inter-RAT calls
AMR-WB

39
Voice Quality Analysis
What to improve: SUMMARY

Average MOS Bad Sample Rate GoodOrBetter

MOS < 2.0 MOS > 3.6

 Worst cases, very low MOS scores

 Looking impressive! 4.10 -> 4.12
~1% -> 0%
~87%
 Solving all issues of MOS <2.0 But ~6% of calls are No considerable
covering at least one change
bad sample (-> 0%)

 Improving non-optimal transmission

4.10 -> 4.22 No change ~87% -> ~92%
 Solving 50% of issues
(Max MOS - 2.0)

 Technology Improvement 4.10 -> 4.17 No change ~87%

No change
 AMR-WB 23.85 in 3G

40
Voice Quality and Optimizing Networks
Conclusions

ı Maximum Voice Quality is determined by the used technology

ı Each MOS score considerably lower than this technology threshold must have a reason!

ı Voice communication is live and real-time, a network problem will cause a Voice Quality
drop within milliseconds

 Pre-requisites for voice quality optimization

41
Q&A
Thanks for your attention