Speech Quality Measurement

In TEMS Investigation

Speech quality measurement

AQM overview How AQM works in TI Accuracy of AQM in TI Typical AQM values in different environments What if AQM worse than expected

AQM overview
Estimate the user experienced quality. ITU recommendation P.862.1 Compares a distorted speech recording to the original.

Reference speech (original) Convert into psychoacoustic domain

Cognitive

PESQ value

Transmitted speech (distorted) Convert into psychoacoustic domain

modelling

AQM results: MOS-LQO

MOS 5 Meaning Excellent Good Fair Poor Bad

MOS Scale results from subjective tests (listening tests). MOS-LQO = Mean Opinion Score Listening Quality Objective result from measurement tools using AQM 862.1

4 3 2 1

VoCoder

Reference 0 1 2 3 0 1 2 3

Scale is: 4.5 (best) to 1.0 (worst). Best possible score in a live network depends on the speech coder used: - Best EFR: 4.2

EFR EFR EFR EFR EVRC EVRC EVRC EVRC

PESQ [MOS-LQO] 4.22 4.16 4.15 4.13 3.96 3.63 3.82 3.57

AQM in TEMS
AQM works by sending speech references and comparing the recording to the same known reference. TI uses four speech references ,two female and two male speakers. Each reference contains of two sentences and each sentence is 5.5 sec long. The references are in accordance with the ITU standard P. 830, which stipulates how subjective MOS tests should be performed.. One AQM score is given for each reference.
Reference 0

Sentence 0

Sentence 1

Test units
The AQM algorithm is implemented in the Call Generator and in the AQM Module of the TEMS Investigaton. Call Generators can be connected to analogue lines or to ISDN. Call Generators and AQM module store the recorded audio for later analysis. Results from UL and DL are joined into the database so you can see them together in a route or make statistics on both.

DL audio measurement
Digital Cellular Network

UL audio measurement Call Generators

Public Switched Telephone Network (PSTN)

Measurement procedure
DTMF syncs the call Half duplex

CG

Start when last digit received

Play 5.5 sec speech

Record 5.5 sec

FreqAQM FreqAQM

Play 5.5 sec speech

AQM DL
FreqAQM FreqAQM

AQM UL
FreqAQM

AQM DL
FreqAQM

TI

Send DTMF to order call. Start after last digit.

Record 5.5 sec and synch to the CG

Play 5.5 sec speech

Record 5.5 sec and synch to the CG

Accuracy of AQM in TI
Three steps of verification: 1. ITU standardization
Chose the algorithm which compares best to listening tests.

3) Live tests prove PESQ in TA system

2.

Conformance tests
When a measurment tool implements the AQM algorithm it must pass the conformance test. The test consists of 1800 speech sentences and only three scores may differ more than 0.05 MOS from the ITU version.
2) Conformance tests prove the PESQ algorithms in TA

3.

Live tests
Both the audio collection parts of a measurement tool and the handset can degrade the recorded audio. To verify that the degradations are insignificant we compare live tests to simulations and listening tests.
1) ITU has held a competition were PESQ was proven to be the best speech quality algorithm

Live tests
TEMS has a lab network with base station and BSC, which makes it possible to test with controlled radio environment. Results are then compared to simulations.
AMR Codec rate PESQ measured on live network on DL 4.11 3.81 3.50 3.21 PESQ measured on live network on UL 4.08 3.84 3.50 3.18 PESQ measured on simulation

12.2 7.95 5.90 4.75

4.17 3.84 3.50 3.26

Typical AQM values

Environment Best EFR / AMR FR (12.2) Best AMR HR (7.95) Best CDMA EVRC Best 3G (AMR 12.2) BLER = 1% DL Best 3G (AMR 12.2) BLER = 1% UL Typical 3G (AMR 12.2) BLER = 1% Good 3G/2G Acceptable AQM 4.15 3.85 3.8 4.15 3.85 3.8 >= 3.5 >=3.0

AQM measurements
PESQ algorithm (ITU Recommendation P.862)
Estimates the listening speech quality Speech-quality score based on the comparison between the original speech and the speech as degraded by the network Option in TEMS Investigation 8.0 MO PESQ voice calls (command sequence) UL and DL MOS score Frequent AQM Presentation of AQM scores in
TEMS Investigation/Data Collection TEMS Investigation/TEMS Discovery

AQM in TEMS Investigation

Supported terminals
Sony Ericsson K790i, K790a Sony Ericsson K800i LG CU-500

AQM measurements
MOBILE-TO-FIXED AQM/PESQ MEASUREMENTS
TEMS Investigation Command sequence
Reference audio

AQM Module

Test Terminal

Transmitted audio

PESQ DL Score

Received degraded audio (from Call Generator)

Log file containing PESQ DL scores Reference audio

Voice call

PESQ UL Score

PESQ UL scores in XML files (copy/download) Merged log file containing both PESQ DL and UL scores

TEMS Investigation Route Analysis (Converter)

Call Generator (with PESQ module)

AQM/PESQ MEASUREMENTS
[ SERVICE QUALITY TESTING ]

TEMS Investigation supports AQM/PESQ measurements - Mobile-to-mobile and mobile-to-fixed AQM is more than PESQ TEMS Investigations AQM/PESQ solution - TEMS Investigation with AQM license option - UE that supports AQM measurements
- Sony Ericsson W995, W995a, and W995 EDGE - Kyocera KX5 - Samsung SPH-M320 and M330

AQM module including cable kit Call Generator and/or Mobile Receiving Unit (MRU) Stand-alone or mounted in backpack or equipment case

Full reference speech quality measurements

[ SERVICE QUALITY TESTING ]

MOBILE-TO-MOBILE AQM/PESQ MEASUREMENTS

TEMS Investigation Command sequence
Initiate AQM call

AQM Module

Test terminal

PESQ DL Score

Received degraded audio (from Mobile Receiver Unit)

Logfile containing PESQ DL scores

Voice call Reference audio

Answer call

Logfile containing PESQ DL and positioned DL Frequent AQM Scores

TEMS Investigation Export functionality

TEMS Investigation

Mobile Receiver Unit (MRU)

AUDIO QUALITY MEASUREMENTS

Use TEMS Investigation to monitor end-to-end speech quality Perform AQM/PESQ measurements with a Call Generator, mobile receiving units, or a mix of both Mobile-to-fixed and mobile-to-mobile Note the difference between the two solutions, in areas like measurements, dimensioning, troubleshooting, and workflow PESQ UL and DL vs. PESQ DL One radio link affecting each PESQ score vs. two radio links affecting each PESQ score Impact of PSTN vs. impact of MRU location Capacity of Call Generator vs. number of MRUs Workflow and control Use SQI measurements as a complement, when troubleshooting

Create your own solution

AQM is worse is it bad radio on DL?

1. Check the radio for the DL:
GSM: SQI low (< 30)? Analyze further: Check no of HO;s. Each HO gives an avg drop of ~0.8 MOS. RxQual (5-7) and FER (>0) Is it AMR HR, old FR or old HR? 3G: SQI low? Analyze further: Avg BLER < BLER target. Check BLER target. A BLER target of 1% is default and gives AQM =~3.8 MOS. Is it AMR HR? Check no of Inter RAT HO.

AQM is worse is it bad radio on UL?

2. Check the radio for the UL:
1. All: Use UE traffic recording. 2. All: Look at Frequent AQM does it ever reach maximum? Check codec used: Is it AMR HR, FR or HR? 3. GSM: Look at Frequent AQM does it vary a lot? Analyze further: Check DL radio. Bad DL is clue to bad UL. HO directly affect UL. 4. 3G: Remember for 3G there is always BLER on the UL! Will give varying AQM and Frequent AQM scores.

AQM is worse not radio

No or too few AQM scores. Reason No synch.:
Voice message. Analyze by setting CG to record no sync . Round trip delay too long. CG trace parameter Start of Recording > 2600 samples?

Bad AQM due to low or high volume:

Check DL volume, should be > -15dB and < 2dB. Typical 6dB. Check UL volume, should be > -15dB and < 2dB. Typical 2dB.

Bad AQM due to background noise:

Record DL or UL audio. Look and listen at audio in silent parts in the beginning and end.

Some questions from customers

How do you know you show the correct AQM scores?
See Accuracy of AQM in TA

Agilent / QVoice shows lower AQM scores than TEMS. WHY?

The audio recording equipment is not the same. Our equipment has low background noise, very little filtering, correct volume settings from factory. The mobiles are not the same. Audio references are different. Remember we have verified our scores to simulations and in a controlled test environment (Techlab).