You are on page 1of 51

Workshop on “Monitoring Quality of

Service and Quality of Experience of Multimedia
Services in Broadband/Internet Networks”
(Maputo, Mozambique, 14-16 April 2014)

Session 11: LTE QoS, QoE and performance
Seppo Lohikko
Principal Consultant, Omnitele Ltd
seppo.lohikko@omnitele.com
Maputo, Mozambique, 14-16 April 2014

topics discussed





Introduction to LTE
LTE and mobile services
LTE and WWW browsing QoE
Considerations on LTE QoE
Use cases

Maputo, Mozambique, 14-16 April 2014

2

LTE introduction |

why LTE?

Key drivers of LTE: Capacity, QoS, Cost, Competition from other
techs (e.g. WiMAX). 3GPP work started 2004 with target definitions.

Wireline
evolution drives
data rates

MBB success
drives
capacity

Business case
drives costefficiency

LTE
Targets

LTE Targets [1]:



2-4 fold spectral efficiency compared to
R6 HSPA (the “14.4 Mbit/s”)
Peak rate 100Mbit/s DL and 50Mbit/s UL
RTT < 10ms possible
Optimised for PS transmission

Maputo, Mozambique, 14-16 April 2014

3




High level of mobility and security
Optimised terminal power efficiency
Frequency flexibility with allocations
below 1.5MHz up to 20MHz
Lower CapEx & OpEx

LTE introduction |

Selected solutions to meet the targets

ARCHITECTURE
 Flat architecture: No RNC-element
 Optimised for PS, no CS domain in place

Spectral efficiency &
latency gain

RADIO
 DL Radio: OFDMA, 64QAM and 2x2 MIMO
 UL Radio: SC-FDMA, 16QAM modulation

Spectral efficiency
and peak rate gain

DEPLOYMENT FLEXIBILITY:
 BW options: 1.4; 3; 5; 10; 15; 20 MHz
 ~45 FDD and TDD frequency bands
Maputo, Mozambique, 14-16 April 2014

4

Easy refarming, wide
adoption expected.

279 live LTE networks in 101 countries*

*)

Maputo, Mozambique, 14-16 April 2014

5

14-16 April 2014 6 .topics discussed      Introduction to LTE LTE and mobile services LTE and WWW browsing QoE Considerations on LTE QoE Use cases Maputo. Mozambique.

Mozambique. 14-16 April 2014 7 .LTE AND QoS: End of tech talk…let’s explore what LTE really means for subscribers! Maputo.

news. NOT NW PERFORMANCE   They consume VoIP.Application Performance is the Make It or Break It END-USERS MIND SERVICE QUALITY. etc They don’t consume average throughput  focus on the services – not the networks Maputo. Mozambique. 14-16 April 2014 8 . video. social networking.

Mozambique.3 % 5% 1.0 % 20% 7% 10% 3.1 % 0% General Browsing Maputo.WWW & Video dominate usage MOBILE DATA USE CASE POPULARITY AND DATA VOLUME 80% 72% 70.3 % VoIP Source: Allot MobileTrends Report Q2/2013.5 % 70% Subscribers using (%) 60% Share of data volume (%) 50% 40% 30% 20% 20.7 % 0. 14-16 April 2014 Video Streaming 9 P2P Gaming 5% 4. CISCO VNI 2012 M2M .

voice is still a challenge  Higher throughput – some impact on service experience  Latency gains – More robust VoIP  Mobile HD video conferencing would require LTE…Demand? Maputo. Mozambique. simple data services UMTS: CS64 Video Telephony – a true killer-app? HSDPA: Real access to mobile data and rich web content WHAT LTE BRINGS?  No new services – Even worse. 14-16 April 2014 10 Re-inventing the wheel? .No revolution expected from LTE… SHORT RECAP OF MOBILE SERVICES HISTORY    GSM: reliable digital voice & SMS.

14-16 April 2014 11 . Mozambique.topics discussed      Introduction to LTE LTE and mobile services LTE and QoE Considerations on LTE QoE Use cases Maputo.

is there at least a QoE improvement? Maputo.LTE AND WWW BROWSING QoE If LTE doesn’t come with new services. 14-16 April 2014 12 . Mozambique.

14-16 April 2014 13 . end-user QoS and QoE … service specific QoE end-user QoS NW performance KPIs …to see LTE impact on QoE. Mozambique.recap | QoE – QoS – NW performance In Session 10 we linked together NW performance. we just need to explore the same path in opposite direction! Maputo.

4 25. Mozambique.com/2013/omnitele-benchmarks-mobile-service-quality-in-sweden/ . 14-16 April 2014 15.1.5 Huawei E3276 http://www.2 Samsung Galaxy Note2 LTE 14 32.omnitele.4 8.2 Apple iPhone5 Maputo. NW Performance | LTE real life bitrates Q1/2013 Omnitele Benchmark in Sweden: LTE outperforms DC-HSPA with factor of ~3 QoS? Mbit/s AVERAGE BITRATE 40 35 LTE DC-HSDPA LTE LTE: ~30M 30 25 Operator A 20 Operator B 15 HSPA: ~10M 10 5 0 10.6 28.

14-16 April 2014 15 25 30 35 40 . End-user QoS | WWW page waiting time WWW Page Waiting Time [s] 1000KB WWW PAGE WAITING TIME VS. BITRATE 4 QoE? HSPA LTE 3 2 HSPA: ~2s 1 LTE: ~1s HSPA: ~10M LTE: ~30M 0 - 5 10 15 20 Bitrate [Mbit/s] Maputo. Mozambique.2.

09 Average(s) 5 In our example group. 14-16 April 2014 16 14 16 18 20 .97 HSPA: ~2s 14.83 QoE [MOS] 4 5.3. QoE | LTE: Virtually zero gain in WWW QoE Web Browsing QoS (download time) and QoE (MOS) MOS 2.78 1 0 2 4 6 8 10 12 WWW Page Download Time [s] Maputo. Mozambique. both give perfect QoE (MOS=5) 3.99 3 LTE: ~1s 2 9.

14-16 April 2014 17 CTO TARGET 90% of subs get >3. CTO targets l find required NW performance QoS: WWW page waiting time [s] 1000KB WWW Page Waiting Time vs.5 Mbit/s in 4G >4. Mozambique.5 Mbit/s in 3G . Bitrate HSPA 10 LTE unsatisfactory region (www download time > 3s) 8 6 accepted region 4 (www download time < 3s) 2 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 NW Performance: Bitrate [Mbit/s] Maputo.4.

14-16 April 2014 18 . HSPA summary NW performance end-user QoS QoE +200% +100% +0% with LTE with LTE with LTE LTE does not revolutionise WWW browsing experience Maputo.WWW Browsing QoE: LTE vs. Mozambique.

14-16 April 2014 19 .topics discussed      Introduction to LTE LTE and mobile services LTE and QoE Considerations on LTE QoE Use cases Maputo. Mozambique.

Mozambique. 14-16 April 2014 20 .LTE QoE CONSIDERATIONS Maybe the example of WWW QoE with LTE didn’t still tell the whole truth… Maputo.

Mozambique.Legacy 3G and user expectations In our study. LTE gave 1s gain over HSPA in www page waiting time…but the measured HSPA network was in extremely good shape Q Maputo. 14-16 April 2014 21 Can we actually generalise LTE’s QoE improvement at all? .

30 Mbit/s Median = 1.0 1.0 3..0 HSPA+ R7: 3.12 Mbit/s Median = 3..9s DC-HSDPA: 6.3s Though LTE has no impact on the content quality. Mozambique.0 7.0 6.1s LTE: 15.LTE QoE improvement with YouTube? YouTube 720p Initial Buffering Time Signalling Buffering Time Practical Minimum Buffering Time Practical Range [s] 0.0 5.0 4..0 9. the buffering time is notably lower Maputo. 14-16 April 2014 22 8..6 Mbit/s Median = 5.0 ...0 2.

but helps a lot…at least in short term  All missteps taken with 3G are easy to reproduce with LTE… Maputo.key messages | LTE is not a game changer  LTE won’t redefine mobile broadband experience like HSDPA did on its time  LTE is not likely to solve all QoE problems. Mozambique. 14-16 April 2014 23 .

topics discussed      Introduction to LTE LTE and mobile services LTE and WWW browsing QoE Considerations on LTE QoE Use cases Maputo. 14-16 April 2014 24 . Mozambique.

OPERATOR CASE: CUSTOMER EXPERIENCE BENCHMARK competitive positioning of YouTube experience Maputo. Mozambique. 14-16 April 2014 25 .

5 0 0 T-Mobile Vodafone KPN T-Mobile Vodafone higher bitrate. 14-16 April 2014 26 Vodafone KPN .5 20 1 10 0.4G smartphone benchmark in Netherlands big difference in NW performance… CEM sol ut i on …thin margins in customer experience bitrate [Mbit/s] buffering time [s] 40 2 30 1. but KPN faster YouTube Maputo. Mozambique.

STRAIGHTFORWARD | TRUSTED | INTELLIGENT CUSTOMER EXPERIENCE BENCHMARK IN SWEDEN Country-wide comparison of subscriber perceived data service quality in two mobile networks Omnitele Report | 5 March 2014 STRAIGHTFORWARD | TRUSTED | INTELLIGENT .

executive summary introduction benchmark results methodology Maputo. 3. Mozambique. 4. 14-16 April 2014 28 . 2.TABLE OF CONTENTS 1.

introduction | project objectives Operator A assigned Omnitele to conduct an independent customer experience benchmark of mobile services in Sweden 1. Mozambique. Measure mobile data service quality with iPhone 5s smartphones in A and B mobile networks across Sweden Operator A 2. 14-16 April 2014 29 . Report and publish the survey results for the general public in clear and understandable fashion Operator B Maputo. Analyse and compare the customer experience of WWW browsing and YouTube video streaming services for both operators VS 3.

iPhone 5s WWW browsing TESTED WEB PAGES: Aftonbladet Google search Wikipedia http://aftonbladet.introduction | test cases Three test cases were measured for both WWW browsing and YouTube video streaming. Test sources were selected by Omnitele to represent typical use cases of Swedish mobile subscribers. 14-16 April 2014 30 Volvo Trucks – The Epic Split feat. Mozambique.org iPhone 5s YouTube video streaming TESTED YOUTUBE VIDEOS: Test video 1 (1:16) Test video 2 (0:33) Test video 3 (0:51) Maputo.se https://www. Van Damme Harlem Shake (original army edition) What Your Body Does in 30 Seconds .google.wikipedia.se/search?q=zlatan https://sv.

Mozambique. 100 rural.introduction | measurement campaign TEST CAMPAIGN     tests conducted Jan 20 – Feb 14. 14-16 April 2014 31 . 2014 total 101 test days by 5 measurement teams 680 test locations across Sweden (550 city. 30 holiday) 8 160 individual mobile data use case tests Test Execution Benchmarked Operators Number of cities Number of test locations Test days Totals 2 79 680 101 Sample Counts / operator WWW page download attempts YouTube video stream attempts Totals 2040 2040 Maputo.

14-16 April 2014 32 2. test case success rate Probability that the user can successfully initiate and complete the use case Maputo. the winning operator is defined by test case usability (2) 1.introduction | test parameters and quality positioning End-user centric analysis methodology and competitive quality positioning    For each of the 2 test cases 3 individual measurement samples are collected in every test location For each test case. the better operator in a given location is defined primarily based on success rate (1) If both operators have equal success rate in a specific test location. Mozambique. test case usability WWW: web page waiting time [s] YouTube: buffering time [s] .

14-16 April 2014 33 . 2. 4. executive summary introduction benchmark results methodology Maputo. Mozambique.TABLE OF CONTENTS 1. 3.

DASHBOARDS AND OVERVIEW     customer experience summary WWW browsing geographical benchmark YouTube streaming geographical benchmark 4G network availability 2. WWW BROWSING DETAILS     WWW WWW WWW WWW browsing: browsing: browsing: browsing: whole country urban areas rural areas holiday locations 3.benchmark results | table of contents 1. Mozambique. 14-16 April 2014 34 whole country urban areas rural areas holiday locations . YOUTUBE STREAMING DETAILS     YouTube YouTube YouTube YouTube streaming: streaming: streaming: streaming: Maputo.

14-16 April 2014 35 78% . Mozambique. Operator A scores higher than Operator B in 73% of the test locations. Numeric results however show that from typical mobile subscriber perspective the differences are rather marginal. 27% Operator B Telia Telenor Operator A 73% WWW Browsing YouTube 22% 32% 68% Maputo. Omnitele concludes that both Operators provide outstanding customer experience compared to any international references and industry standards.dashboards and overview | customer experience summary CUSTOMER EXPERIENCE Considering all performed tests.

Mozambique. 14-16 April 2014 Operator B 36 19% 31% 3.0 0. The absolute difference in WWW page waiting time is marginal.5 1.5 3.dashboards and overview | WWW browsing geographical benchmark Northern In WWW browsing test Operator A scores slightly better in most parts of Sweden. In Western Sweden Operator B results are better.0 Stockholm 76% . WWW page waiting time (s) WHOLE COUNTRY (680) 59% Western 47% 53% 41% Central-Eastern 69% 81% Southern 2.5 2.5 0.0 Operator B Maputo.0 24% 1.

21% 79% Western 24% YouTube buffering time (s) WHOLE COUNTRY (680) 1. Mozambique.dashboards and overview | YouTube streaming geographical benchmark Northern In YouTube video streaming test Operator A scores slightly better than Operator B consistently across Sweden.4 Southern 1.6 76% Central-Eastern 12% 88% 1.6 0.2 1.2 0.0 Operator B Maputo.8 18% 0.0 0.4 0. 14-16 April 2014 Operator A 37 Stockholm 82% 19% 31% 69% 81% . The difference in video buffering time is however hardly noticeable.

4G availability with iPhone 5s terminal is slightly wider for Operator A.dashboards and overview | 4G network availability 4G is no longer a rarity in Sweden but instead widely available across the country with both operators.9 % Maputo. Considering the tested locations. Mozambique. 14-16 April 2014 38 95. OPERATOR B WHOLE COUNTRY 4G OPERATOR A WHOLE COUNTRY 4G 4G 3G 3G 92.5 % .

4. 2. Mozambique. 3. 14-16 April 2014 39 .TABLE OF CONTENTS 1. executive summary introduction benchmark results methodology Maputo.

Mozambique.  Commercial state of the art smartphones used for capturing best available end-user quality  Devices sourced from retail stores and SIM cards from operator stores  Test locations chosen independently by Omnitele in blind test fashion  Frequency band and technology (2G/3G/4G) selection as per network parameterisation  All tests conducted in stationary state inside car Maputo.  Sundays: No testing.00 – 00:00.methodology | user-centric test methodology Measurement methodology designed to capture true end-user experience TESTING TIMES METHODOLOGY  Test days: Monday . focus on morning and night busy hours  Saturdays: Measurements between 10:00 – 00:00.Saturday  Test hours: 06. No measurements in business or university areas. 14-16 April 2014 40 .

Mozambique. Maputo. More information: http://www. Test cases included WWW browsing and YouTube video streaming.com/businesse s/network-testing/products NEMO CEM Mobile data testing was conducted with commercial iPhone 5s terminals and Nemo CEM measurement system. 14-16 April 2014 41 .anite.methodology | measurement equipment MEASUREMENT EQUIPMENT MOBILE NETWORKS Operator B Operator A INTERNET Nemo CEM is a flexible and scalable set of tools for monitoring smartphone data services from the end user perspective.

Mozambique.#setup failures . success rate includes service accessibility and retainability.#connection drops) #attempts YouTube video streaming Maputo. 14-16 April 2014 T2: time WWW page content downloaded [s] T1: time user request WWW page [s] YouTube Buffering time [s] = T3 + T4 T3: initial buffering time [s] T4: total rebuffering time [s] 42 . TC1 WWW Browsing TC2 YouTube video streaming Sequence 3 x WWW page download Sequence 3 x YouTube video stream (60s each) Interval 10s interval between WWW requests Interval 10s interval between WWW requests Time-out 30s time-out limit for WWW download Time-out 30s time-out for setup and re-buffering Reference point Public internet Reference point Public YouTube Methodology Stationary test in a car Methodology Stationary test in a car Test parameters are defined based on ETSI standardisation (TS 102 250-2). test case success rate usability WWW page waiting time [s] = T2 – T1 WWW browsing (#attempts .methodology | test cases and test parameters Three test cases were measured for both WWW browsing and YouTube video streaming. Test sources were selected by Omnitele to represent typical use cases of Swedish mobile subscribers.

2. result tables measurement location details statistical significance analysis test devices and SIM cards Maputo. 14-16 April 2014 43 . Mozambique.ANNEXES 1. 3. 4.

018856 0. urban areas WWW page waiting time (s) Number of locations Standard deviation (s) Confidence interval (s) Statistically significant difference? Telia B Operator Telenor Operator A 3.09395854 1. holiday locations WWW page waiting time (s) Number of locations Standard deviation (s) Confidence interval (s) Statistically significant difference? 44 Operator Telia B Operator A Telenor 2.4 2 2 0.19 NO Locations with failures Locations without failures WWW page success rate Confidence interval WWW browsing (Mbit/s).0 2.03 NO 0 147 98.0% 0. rural areas Number of locations Telenor Operator A 149 Statistically significant difference? WWW success. rural areas WWW page waiting time (s) Number of locations Standard deviation (s) Confidence interval (s) Statistically significant difference? Telia B Operator Telenor Operator A 2.0% 2 100.2 2.0% NO TeliaB Operator Telenor Operator A 24 24 0 0 24 100.92426963 1.26 0. urban areas Number of locations URBAN (550) Maputo.7 123 123 2.0% 24 100.0% NO . rural areas Number of locations HOLIDAY (30) 149 2 Statistically significant difference? WWW success.8 2.4 24 24 0.24772308 0. 14-16 April 2014 Telia B Operator TeliaB Operator NO Operator TeliaB Operator A Telenor 2 2 0 0 2 100.1 2.0% 0.0% Statistically significant difference? WWW success.0% 1. whole country Number of locations WWW browsing.28 NO Locations with failures Locations without failures WWW page success rate WWW browsing (Mbit/s).0% 0.ANNEX 1 | www browsing.705966 0.31 0. whole country WWW page waiting time (s) Number of locations Standard deviation (s) Confidence interval (s) Statistically significant difference? WHOLE COUNTRY (680) Telenor Operator A 3.249532 0.22 NO Locations with failures Locations without failures WWW page success rate Confidence interval RURAL (100) WWW browsing (Mbit/s). Mozambique.7% 149 100. details WWW success.0% 2.2% 0.4% 123 100.37 0.34 0.8% Confidence interval Locations with failures Locations without failures WWW page success rate Confidence interval Statistically significant difference? 0.7 149 149 1.0% 0.174307 0.0% NO TeliaB Operator Telenor Operator A 123 123 2 0 121 98.65776225 0.

69905192 0. Mozambique.6% 146 98. holiday YouTube buffering time (s) Number of locations Standard deviation (s) Confidence interval (s) Statistically significant difference? 45 Telia B Operator Telenor Operator A 1.8% 0. rural areas Number of locations YouTube video streaming (Mbit/s).302874 0.26581636 0.10 0.2 1.1 24 24 0.64409219 0.05 YES YouTube video streaming (Mbit/s).4% Locations without failures YouTube success rate Confidence interval 3.05 YES 149 Locations with access failures 3 1 Locations with connection drops 2 2 144 96.09 YES YouTube video streaming (Mbit/s).15 YES 2. whole country Number of locations YouTube video streaming.11 0.4 1.07 0. details YouTube success.1 123 123 0. rural areas Number of locations HOLIDAY (30) 2.105359 0.3 1. whole country YouTube buffering time (s) Number of locations Standard deviation (s) Confidence interval (s) Statistically significant difference? WHOLE COUNTRY (680) Operator A Telenor 1.3 1.04831896 0. 14-16 April 2014 Operator Telia B TeliaB Operator NO Telia B Operator 2 TelenorA Operator 2 Locations with access failures 0 Locations with connection drops 0 0 2 100.1 149 149 0.0% 23 95.9% Statistically significant difference? YouTube success. rural YouTube buffering time (s) Number of locations Standard deviation (s) Confidence interval (s) Statistically significant difference? Telenor Operator A 149 YouTube success.12 0.0% Locations without failures YouTube success rate Confidence interval Statistically significant difference? 0 NO .212602 0.3% NO TeliaB Operator Telenor Operator A 123 123 Locations with access failures 3 0 Locations with connection drops 2 2 118 95.2% NO Telia B Operator Telenor A Operator 24 24 Locations with access failures 0 1 Locations with connection drops 0 0 24 100.0% Locations without failures YouTube success rate Confidence interval Statistically significant difference? YouTube success.0% 0.9% 121 98.0 2 2 0. urban YouTube buffering time (s) Number of locations Standard deviation (s) Confidence interval (s) Statistically significant difference? Operator Telia B Operator A Telenor 1.287732 0.0% 2 100.0% Locations without failures YouTube success rate Confidence interval RURAL (100) Telia B Operator Telenor Operator A 1.0% 8. urban areas Number of locations URBAN (550) Maputo.ANNEX 1 | YouTube video streaming.5% Statistically significant difference? 2.0% 0.

2. Mozambique. 3. result tables measurement location details statistical significance analysis test devices and SIM cards Maputo.ANNEXES 1. 14-16 April 2014 46 . 4.

Mozambique. Norm. Maputo.96) Standard Error SE of the sample mean. is defined as s / 𝒏.975.α/2 = 0.Inv[0. In case two operators have differing mean values. the sample-based estimate of the standard deviation of the population).ANNEX 3 | statistical significance analysis In order to analyse the statistical reliability of the results.96 x SE.CI. with 95% probability the true population mean is within the sample mean +/. where  SE is equal to the standard error for the sample mean. That is. the difference is statistically significant.. 100% Statistically significant difference 99% 98% 97% 96% Operator 1 Operator 2 Statistically insignificant difference 95% KPI1 KPI2 CALCULATION The calculated confidence interval CI is based on (two-tailed) confidence level of 95%. where  s is the sample standard deviation (i. the observed difference is not statistically significant. Confidence Interval can be considered as the error margin of the reported results. see example figure below.e. 14-16 April 2014 47 . This is achieved by dimensioning the projects with sufficient test sample counts.975] = 1. That is. The CI is calculated as 1. The error margins are visible in the error bars of report graphs.05  1. and  1.96 is the . Omnitele applies confidence interval analysis on the calculated KPIs.95  α = 0. Omnitele targets to conduct the measurement campaigns so that the confidence intervals allow sufficient accuracy for the conclusions. If the error bars don’t overlap. the error margins are smaller than truly significant QoS differences from end-user point of view.975 quantile of the normal distribution (CL = 0. but overlapping error bars. and  n is the size (number of observations) of the sample.

2. result tables measurement location details statistical significance analysis test devices and SIM cards Maputo.ANNEXES 1. 3. Mozambique. 14-16 April 2014 48 . 4.

0. Mozambique.4 (11B554a) Apple Store Online Operator A SIM profile Xxxxxxxxx 16GB Operator B SIM profile Yyyyyyyyy 10GB . 14-16 April 2014 49 Terminal Model Modem Firmware OS Version Sourced from iPhone 5s 32GB Silver/Space Grey ME435KS/A ME436KS/A 1.03.ANNEX 4 | test devices and SIM cards Maputo.01 7.

14-16 April 2014 50 . Mozambique.End of Session 11… Questions? Maputo.

14-16 April 2014 51 .com www.omnitele.com Follow us at www. Seppo Lohikko Principal Consultant Omnitele Ltd.com/channel/UCyMeM3j2L9MQoB0qeDCev3g Maputo.linkedin.Any further enquiries: Mr.youtube.lohikko@omnitele.com/company/omnitele http://www. Mobile: +358 44 2793811 seppo. Mozambique.