KPI Definitions in TEMS Products

NT13-25359, ver 2.0, 9/25/2013

 Ascom. All rights reserved.
TEMS is a trademark of Ascom. All other trademarks are the property of their respective holders.

NT13-25359, ver 2.0, 9/25/2013

 Contents

1 Introduction 1

2 General Aspects of KPI Data Collection and Computation 1

2.1 KPI Reporting.............................................................................................................. 1
2.2 ETSI Compliance ........................................................................................................ 1
2.3 Complete vs. Timed Measurements ........................................................................... 1
2.4 IP Capture Options ..................................................................................................... 2

3 KPIs by Service 2
3.1 KPIs for FTP Download .............................................................................................. 2
3.1.1 Timed Measurements .............................................................................................. 3
3.1.2 SFTP (Secure Shell FTP) ........................................................................................ 3
3.2 KPIs for FTP Upload ................................................................................................... 3
3.2.1 Timed Measurements .............................................................................................. 3
3.2.2 SFTP (Secure Shell FTP) ........................................................................................ 4
3.3 KPIs for HTTP Get/Download ..................................................................................... 4
3.3.1 Timed Measurements .............................................................................................. 4
3.4 KPIs for HTTP Post/Upload ........................................................................................ 4
3.4.1 Timed Measurements .............................................................................................. 5
3.5 KPIs for Ping ............................................................................................................... 5
3.5.1 Timed Measurements .............................................................................................. 5
3.6 KPIs for CS Voice ....................................................................................................... 5

4 Notes on Other Services: Email, Iperf, Streaming, CS Fallback 6

4.1 Email ........................................................................................................................... 6
4.2 Iperf............................................................................................................................. 7
4.2.1 TCP ......................................................................................................................... 7
4.2.2 UDP ........................................................................................................................ 7
4.3 Streaming over HTTP ................................................................................................. 8
4.3.1 Streaming Player Download .................................................................................... 8
4.3.2 Start of Streaming Video Replay .............................................................................. 9
4.3.3 Streaming Video Session ........................................................................................ 9
4.4 CS Fallback............................................................................................................... 13

NT13-25359, ver 2.0, 9/25/2013

 1 Introduction
This document describes the implementation of ETSI circuit-switched and packet-
switched service KPIs, as well as some closely related non-ETSI KPIs, in TEMS
products.
The document is valid for the following TEMS product versions:
TEMS Automatic 10.1 and later
TEMS Investigation 16.0 and later
TEMS Symphony 7.4 and later

2 General Aspects of KPI Data Collection and

Computation
2.1 KPI Reporting
For PS data services, current TEMS products report KPI events, each of which
contains either
an instance of the KPI itself (such as a session duration or throughput figure), or
an instance of an occurrence whose probability is measured by the KPI (such as a
session failure).
These in turn underlie the aggregation of KPI statistics in TEMS Discovery or some
other post-processing tool. The probability KPIs obviously need to be based on a fair
number of sessions for a good estimate of the probability in question to be obtained.
The data-collecting TEMS products do not in themselves output values of these latter
KPIs; rather, they need to be calculated during post-processing.
TEMS Symphony users should note that earlier versions of TEMS Symphony (version
6.x and older) did not report data in this fashion, but only trigger points on which to
base the KPI computation: for example, session start and end times.
Regarding CS voice, see section 3.6.

2.2 ETSI Compliance

In TEMS products, timestamps for KPI computation are captured as close to the source
as possible: that is, from the packet capture driver within the operating system kernel.
Calculations adhere to the specification ETSI TS 102 250-2 V2.2.1 (2011-04) unless
otherwise stated. This specification is hereafter referred to as ETSI 102 250-2.

2.3 Complete vs. Timed Measurements

The standardized KPIs have been defined with complete transactions in mind.
Transactions that are aborted after a fixed duration specified by the TEMS product user
(timed measurements) have not been considered. In case of timed measurements, a
best-effort solution is provided. Where triggers do not fully match ETSI 102 250-2, they
have been aligned as closely as possible to their ETSI counterparts in a manner

NT13-25359, ver 2.0, 9/25/2013 1

 compliant with ETSI TS 102 678 V1.2.1, except that the following item stated in
section 4.1 of that specification has not been implemented:

The connection should be checked to be still alive at the end of the transfer period td. The
reception of any data packet sent by the server on the data connection after the end of the
transfer period is a valid indicator that the data connection is still alive. An appropriate
timeout of e.g. 3 to 5 multiples of typical RTT in the measured network might be used while
waiting for the desired packet.

Session Time KPIs, of course, make sense only for sessions that have been allowed
to run to completion.
For details, see the various subsections of chapter 3.

2.4 IP Capture Options

The default IP data capture that is done by TEMS products during PS data service
testing is designed to obtain sufficient input for KPIs. No special activity needs to be
used in Service Control scripts for this purpose.
It is however also possible to conduct a more thoroughgoing IP sniffing by means of
the Start IP Sniffing script activity. This activity has a Filter parameter whose
possible settings are given here for convenience.

Filter: Type of filtering to apply to IP packets.

o Optimized Performance: Capture of IP packets is reduced to the minimum needed to
compute KPIs. (Packets are filtered with respect to IP address, protocol [TCP/UDP], and
port.) Not supported for on-device measurement.
o Headers Only: The first 54 bytes of every IP packet are captured.
o None: No filtering of IP packets. Note: This means that all packets belonging to any
service used by the PC will be captured; packets are filtered with respect to IP address
only. Be aware that full packet capturing can cause crashes on high speed networks
where sustained high throughput is achieved.

3 KPIs by Service
This chapter deals with TEMS product KPIs for each service.

3.1 KPIs for FTP Download

ETSI-to-TEMS Mapping Table
ETSI KPI FTP
Sect. TEMS KPI Event
{Download|Upload}
Service Non-Accessibility 6.1.1 FTP Download Service Not Accessible
Setup Time 6.1.2 FTP Download Setup Time
IP-Service Access Failure Ratio 6.1.3 FTP Download IP Service Access Failure
IP-Service Setup Time 6.1.4 FTP Download IP Service Setup Time
Session Time 6.1.6 FTP Download Data Transfer Time

NT13-25359, ver 2.0, 9/25/2013 2

 ETSI-to-TEMS Mapping Table
ETSI KPI FTP
Sect. TEMS KPI Event
{Download|Upload}
Mean Data Rate 6.1.7 FTP Download Mean Data Rate
Data Transfer Cut-off Ratio 6.1.8 FTP Download Data Transfer Cutoff

Note: Method A and Method B values are differentiated wherever applicable.

See ETSI 102 250-2 section 4.2.

3.1.1 Timed Measurements

For timed measurements, the end trigger is defined as Last incoming packet
containing content. This trigger is valid for both completed and aborted transfers.
Since this definition conforms to the ETSI standard, timed FTP downloads will have
ETSI compliant KPIs.

3.1.2 SFTP (Secure Shell FTP)

If SFTP is used for the file transfer rather than FTP proper, no KPIs are obtained. ETSI
102 250-2 does not define any KPIs for SFTP, and some metrics would not be possible
to obtain owing to the SSH encryption.

3.2 KPIs for FTP Upload

ETSI-to-TEMS Mapping Table
ETSI KPI FTP
Sect. TEMS KPI Event
{Download|Upload}
Service Non-Accessibility 6.1.1 FTP Upload Service Not Accessible
Setup Time 6.1.2 FTP Upload Setup Time
IP-Service Access Failure Ratio 6.1.3 FTP Upload IP Service Access Failure
IP-Service Setup Time 6.1.4 FTP Upload IP Service Setup Time
Session Time 6.1.6 FTP Upload Data Transfer Time
Mean Data Rate 6.1.7 FTP Upload Mean Data Rate
Data Transfer Cut-off Ratio 6.1.8 FTP Upload Data Transfer Cutoff

Note: Method A and Method B values are differentiated wherever applicable.

3.2.1 Timed Measurements

For timed FTP uploads, the end trigger is defined as Reception of ACK for last packet
containing content. This trigger is not ETSI compliant: ETSI 102 250-2
(section 6.1.6.3) defines the trigger as Stop: Reception of the [FIN, ACK] for the last
data packet containing content. That is, the FIN indicator has been removed from the

NT13-25359, ver 2.0, 9/25/2013 3

 condition. This is because the FIN indicator is sent during connection teardown, a
procedure which does not necessarily take place during a timed measurement.

3.2.2 SFTP (Secure Shell FTP)

If SFTP is used, no KPIs are obtained; compare section 3.1.2.

3.3 KPIs for HTTP Get/Download

This section deals primarily with HTTP downloads conducted with the TEMS browser.
The same KPI values apply also for Internet Explorer browser usage.

ETSI-to-TEMS Mapping Table

ETSI KPI HTTP Sect. TEMS KPI Event
Service Non-Accessibility 6.8.1 HTTP Service Not Accessible
Setup Time 6.8.2 HTTP Setup Time
IP-Service Access Failure Ratio 6.8.3 HTTP IP Service Access Failure
IP-Service Setup Time 6.8.4 HTTP IP Service Setup Time
Session Time 6.8.6 HTTP Data Transfer Time
Mean Data Rate 6.8.7 HTTP Mean Data Rate
Data Transfer Cut-off Ratio 6.8.8 HTTP Data Transfer Cutoff

Note: Method A and Method B values are differentiated wherever applicable.

3.3.1 Timed Measurements

There is no difference between timed and complete measurements; the same triggers
can be used in both scenarios.

3.4 KPIs for HTTP Post/Upload

ETSI does not define KPIs for HTTP Post/Upload. The TEMS product KPIs have been
designed to closely resemble the ETSI KPIs for other services, such as HTTP Get and
FTP (compare sections 3.13.3).

ETSI-to-TEMS Mapping Table

ETSI KPI HTTP (Get) Sect. TEMS KPI Event
Service Non-Accessibility 6.8.1 HTTP Service Not Accessible
Setup Time 6.8.2 HTTP Setup Time
IP-Service Access Failure Ratio 6.8.3 HTTP Post IP Service Access Failure
IP-Service Setup Time 6.8.4 HTTP Post IP Service Setup Time
Session Time 6.8.6 HTTP Post Data Transfer Time
Mean Data Rate 6.8.7 HTTP Post Mean Data Rate

NT13-25359, ver 2.0, 9/25/2013 4

 ETSI-to-TEMS Mapping Table
ETSI KPI HTTP (Get) Sect. TEMS KPI Event
Data Transfer Cut-off Ratio 6.8.8 HTTP Post Data Transfer Cutoff

Trigger points for the HTTP Post session are as follows:

Start: First packet containing content.
End: Last incoming ACK packet for content.
No Method A vs. Method B distinction exists for this service.

3.4.1 Timed Measurements

Since ETSI does not define any KPIs for HTTP Post, the session end trigger was
defined in such a way as to work for both timed and complete measurements.

3.5 KPIs for Ping

ETSI-to-TEMS Mapping Table
ETSI KPI Ping Sect. TEMS KPI Event
Round trip time 6.3.1 Ping Roundtrip Time

Trigger points for Ping round trip:

Start: ICMP Echo Request sent.
End: ICMP Echo Reply received by the sender.
The round-trip time is that calculated by the Windows API function IcmpSendEcho2Ex.
This is the lowest-latency way available in Windows to send Ping requests and capture
replies. Reference:
msdn.microsoft.com/en-us/library/windows/desktop/aa366050(v=vs.85).aspx
The difference between the round-trip time measured by Windows and that derived
from the network trace is in the range 0.5 ms. Since the round-trip time is reported in
milliseconds in the KPI, the difference after rounding is not noticeable.
TEMS products also define a timeout error message Ping Timeout signifying a failed
Ping request.

3.5.1 Timed Measurements

Not applicable.

3.6 KPIs for CS Voice

ETSI-to-TEMS Mapping Table
ETSI KPI: Telephony Sect. TEMS Data (see also below)
Service Non-Accessibility 6.6.1 Blocked Call event

NT13-25359, ver 2.0, 9/25/2013 5

 ETSI-to-TEMS Mapping Table
ETSI KPI: Telephony Sect. TEMS Data (see also below)
Setup Time 6.6.2 Call Setup event with call setup time
Speech Quality on Sample Basis 6.6.4 PESQ/POLQA information elements
Cut-off Call Ratio 6.6.5 Dropped Call event

For CS voice, no special KPI events are generated. However, call setup time is carried
as extra information by the Call Setup event, and speech quality scores are reported in
information elements belonging to the Media Quality category. Supported speech
quality measures are PESQ, POLQA NB and POLQA SWB.
TEMS products also have an additional, non-ETSI speech quality measure called
Speech Quality Index (SQI).
CS voice KPIs are computed for mobile-originated (MO) as well as mobile-terminated
(MT) calls.

4 Notes on Other Services: Email, Iperf,

Streaming, CS Fallback
For the following services, TEMS products do not have any KPIs defined. However,
they do exhibit certain information elements and/or events with similar content.
Average throughputs and all percentages apply to the period following the latest
network connect (triggering of Network Connect event) and are reset at network
disconnect (when the Network Disconnect activity has completed).

4.1 Email

IE Name Range/Unit Description

Email Receive 0 ... 350000 Average throughput for receiving of email.

Average kbit/s
Throughput

Email Receive 0 ... 172800 Elapsed time for current email receive
Transfer Time s session. (Not an average.)
(= 48 h)

Email Send 0 ... 350000 Average throughput for sending of email.

Average kbit/s
Throughput

Email Send 0 ... 172800 Elapsed time for current email send
Transfer Time s session. (Not an average.)

NT13-25359, ver 2.0, 9/25/2013 6

 4.2 Iperf
Iperf testing is conducted with the Network Bandwidth activity in Service Control
scripts. The testing can be done over either TCP or UDP.

4.2.1 TCP

IE Name Range/Unit Description

TCP Download 0 ... 350000 Average throughput for TCP download.

Average kbit/s
Throughput
(kbit/s)

TCP Download 0 ... 172800 Elapsed time for current TCP download
Transfer Time s session. (Not an average.)

TCP Packet Loss 0 ... 100 Retransmissions (in percent) on the

% downlink over the TCP protocol during the
last second. (Not an average.)
Note: For this percentage to be correct, IP
sniffing must not be set to Optimized
Performance. See section 2.4.

TCP Upload 0 ... 350000 Average throughput for TCP upload.

Average kbit/s
Throughput
(kbit/s)

TCP Upload 0 ... 172800 Elapsed time for current TCP upload
Transfer Time s session. (Not an average.)

4.2.2 UDP

IE Name Range/Unit Description

UDP Download 0 ... 350000 Average throughput for UDP download.

Average kbit/s
Throughput
(kbit/s)

UDP Download Text UDP jitter: the mean deviation (in ms) of
Jitter the difference in packet spacing at the
receiver compared to the sender, for a
pair of packets.

UDP Download 0 ... 100 Retransmissions (in percent) on the

Packet Loss % downlink over the UDP protocol. This
measurement is obtained once every
second. (Not an average.)

NT13-25359, ver 2.0, 9/25/2013 7

 IE Name Range/Unit Description

UDP Download 0 ... 172800 Elapsed time for current UDP download
Transfer Time s session. (Not an average.)

UDP Upload 0 ... 350000 Average throughput for UDP upload.

Average kbit/s
Throughput
(kbit/s)

UDP Upload 0 ... 172800 Elapsed time for current UDP upload
Transfer Time s session. (Not an average.)

4.3 Streaming over HTTP

4.3.1 Streaming Player Download

These information elements relate to the task of downloading the streaming player.
Please note that not all of the Streaming Player elements are obtained if the
streaming server is an HTTPS server, since the relevant packets are then encrypted.

IE Name Range/Unit Description

Streaming Player 0 ... 100 Percentage of attempts to download the

Service IP Access % streaming player that failed before the first
Failure Ratio packet of the download was received (IP
access failure).

Streaming Player 0 ... 172800 Time taken for IP access when

Service IP Access s downloading the streaming player (from
Time user request to receipt of first packet).

Streaming Player 0 ... 100 Percentage of attempts to download the

Download Data % streaming player that failed at the data
Transfer Failure transfer stage (after receipt of first packet
Ratio and before receipt of last packet).

Streaming Player 0 ... 172800 Time taken to download the streaming

Download Data s player (from receipt of first packet to
Transfer Time receipt of last packet).

Streaming Player 0 ... 100 Percentage of attempts to download the

Session Failure % streaming player that failed at some stage
Ratio (IP access or data transfer).

Streaming Player 0 ... 172800 Total session time for streaming player
Session Time s download (from user request to receipt of
last packet).

NT13-25359, ver 2.0, 9/25/2013 8

 4.3.2 Start of Streaming Video Replay

IE Name Range/Unit Description

Streaming Service 0 ... 172800

The duration of a service access from
Access Time s
requesting the stream at the portal until
the reception of the first stream data
packet.

Streaming Service 0 ... 100 Percentage of attempts to request

Non Accessibility % streaming that failed (never receive
response from streaming host).

Streaming 0 ... 100 The duration of the delay after streaming

Reproduction % request was responded and before the
Start Delay video replay was started.

Streaming Text Indicate that there is error occurs after

Reproduction streaming request was responded and
Start Failure before the video replay was started.

Streaming Video 0 ... 100 Percentage of attempts to start streaming

Play Start Failure % video replay that failed (playing never
Ratio started).

Streaming Video 0 ... 172800 Time from user video replay request until
Play Start Time s playing started.

Streaming 0 ... 100 Percentage of failures throughout the

Reproduction Cut % video playing session.
off Ratio

4.3.3 Streaming Video Session

Overall

IE Name Range/Unit Description

Streaming Video 0 ... 100 Percentage of streaming video replay

Session Failure % session that failed at some point before
Ratio the last video packet was received.

Streaming Video 0 ... 172800 Total time of streaming video replay

Session Time s session from user request to receipt of last
video packet.

VQmon Metrics: Video and Audio Perceptual Quality

NT13-25359, ver 2.0, 9/25/2013 9

 IE Name Range/Unit Description

Streaming 0 ... 5 Average absolute video MOS for the

Absolute MOS-V MOS stream.

Streaming 0 ... 5 Average audio MOS for the stream.

MOS-A MOS

Streaming 0 ... 5 Average audiovideo MOS for the stream.

MOS-AV MOS

Streaming 0 ... 5 Average relative video MOS for the

Relative MOS-V MOS stream. The difference between relative
and absolute video MOS is that the
relative metric does not consider frame
resolution, thus constituting a score which
is relative to the ideal for the current video
format.

These average scores are also reported in a Streaming Quality MOS event at the end
of a streaming session.

Bandwidth

IE Name Range/Unit Description

Streaming 0 ... 350000 Average audio bandwidth excluding

Average Audio kbit/s transport packet overhead, FEC, and
Received retransmissions.
Bandwidth

Streaming 0 ... 350000 Average bandwidth for video transport

Average Video kbit/s packets, excluding IP overhead, FEC, and
Received retransmissions.
Bandwidth

Packet Loss, Packet Errors

IE Name Range/Unit Description

Streaming 0 ... 100 Percentage of audio packets lost or

Average Audio % discarded (minus those corrected).
Effective Packet
Loss Rate

Streaming 0 ... 100 Percentage of video packets lost or

Average Video % discarded (minus those corrected).
Effective Packet
Loss Rate

NT13-25359, ver 2.0, 9/25/2013 10

 IE Name Range/Unit Description

Streaming Audio 0 ... 100 Percentage of audio packets received with

Packet Corrected % errors, but corrected using error correction
Rate algorithms.

Streaming Audio 0 ... 100 Percentage of audio packets discarded by

Packet Discarded % the receiving jitter buffer.
Rate

Streaming Audio 0 ... 100 Percentage of audio packets identified as

Packet Out Of % out-of-sequence, possibly due to high jitter
Sequence Rate levels or the use of load-sharing devices.

Streaming Video 0 ... 100 Percentage of video packets received with

Packet Corrected % errors, but corrected using error correction
Rate algorithms.

Streaming Video 0 ... 100 Percentage of video packets discarded by

Packet Discarded % the receiving jitter buffer.
Rate

Streaming Video 0 ... 100 Percentage of video packets identified as

Packet Out Of % out-of-sequence, possibly due to high jitter
Sequence Rate levels or the use of load-sharing devices.

Buffering and Rebuffering

IE Name Range/Unit Description

Streaming Video Text Number of times the video stream replay

Interruption Count was interrupted for rebuffering.

Streaming Video 0 ... 172800 Total duration of video stream replay

Interruption s interruptions for reasons of rebuffering.
Duration

Streaming Playout 0 ... 350000 Equal to Streaming Playout Buffer Empty

Buffer Delta Rate kbit/s Rate minus Streaming Playout Buffer Fill
Rate.

Streaming Playout 0 ... 100 Percentage of the total session time

Buffer Empty % during which the playout buffer was
Proportion empty.

Streaming Playout 0 ... 350000 Rate at which data is emptied from the
Buffer Empty Rate kbit/s streaming players playout buffer.

Streaming Playout 0 ... 350000 Rate at which the streaming players

Buffer Fill Rate kbit/s playout buffer is filled with new data.

NT13-25359, ver 2.0, 9/25/2013 11

 IE Name Range/Unit Description

Streaming Playout 0 ... 100 Percentage of the total session time that
Buffer Rebuffering % was spent rebuffering.
Proportion

NT13-25359, ver 2.0, 9/25/2013 12

 4.4 CS Fallback
The following events report on various aspects on CS fallback procedures:

Event Name Description

CSFB Blocked Call A CS fallback call was blocked. This can happen in
several ways:
The Extended Service Request timed out without
any response from network, or the network
responded with Service Reject. In this case the CS
fallback procedure never reaches the RAT change
stage.
RAT change to UTRAN/GERAN failed, as indicated
by the event EUTRAN RRC Connection Release
Redirected Failure.
After successful RAT change to UTRAN/GERAN,
the CS call setup failed (CS Blocked Call event
generated).
Extra information:
Call direction (MO/MT)
Block type

CSFB Call Setup A CS fallback call was set up. Triggered by the Layer 3
message Alerting.
Extra information:
Call direction (MO/MT)
Target technology (WCDMA/GSM)
Call setup time (measured from CSFB Call
Attempt, i.e. the first call attempt)
User setup time (measured from CSFB Call
Initiation, thus more accurately reflecting the user-
perceived setup time. Obtained only for MO calls in
scripts. If no CSFB Call Initiation event was
generated, the user setup time cannot be
computed.)

EUTRAN Reselection This event carries performance information on the

Time After CSFB Call switch back to LTE after hangup of a CS fallback call.
Extra information:
Idle to LTE time: Time in seconds from entering
idle mode (in UMTS) to reception of System
Information Block on LTE.
SIB 19 to LTE time: Time in seconds from
reception of System Information Block Type 19
(UMTS) to reception of System Information Block
on LTE.

NT13-25359, ver 2.0, 9/25/2013 13

 Event Name Description

PS Data Interruption This event reports the IP interruption time during RAT
Time Due To CSFB change to UTRAN/GERAN due to initiation of a CS
fallback call.
Extra information: Interruption time in ms. Measured
from last received IP packet in EUTRAN to first
received IP packet in UTRAN/GERAN.

NT13-25359, ver 2.0, 9/25/2013 14