Network Field Tests

WALK & DRIVE TEST GUIDE

21st August, 2019

C2 – Safaricom Internal C1 - Public
 Drive & Walk Test | Guide Summary
Airtel
Identification.
• Carry introduction letter, have company ID
• Be courteous

Test cases are standardized for uniformity.

• Indoor & Outdoor -Voice must be a mobile to mobile alternate calling LTE preferred &
Safaricom GSM
• Data test for indoor – HTTP DL/UL and Ping only – LTE Preferred all operators &
Safaricom UMTS ( Safaricom WiFi in some cases)
• Data test for outdoor benchmarks include the following at a minimum HTTP DL/UL,
Video, Ping & Web browsing. only – LTE Preferred all operators & Safaricom UMTS

Report structure is standard Telkom

• Log file storage is neatly labelled in folders to specify date, floor & building or location
(see slide 4)
• Summary table to indicate voice and KPI see next slide (slide 3)

2 C1 - Public
 Summary Drive Test Benchmark Score Card

Voice Services (indoor / outdoor)

Call setup Time (sec) Speech Quality
Call Setup Call Drop CSFB
Mean Max
NETWORK Success Ratio Ratio Average 90th Percentile CSFB Ratio Call Attempts Setup Fail Call Drops Attempts
Operator A 4G Pref 98.98% 0.40% 3.54 9.89 3.26 4.46 87.43% 1766 18 7 1544
Operator S 4G Pref 99.78% 0.16% 3.16 5.41 3.89 4.46 95.17% 1841 4 3 1752
Operator T 4G Pref 99.38% 0.10% 3.78 6.51 3.74 4.48 93.02% 2091 13 2 1945
Operator S 3G Pref 99.68% 0.05% 4.63 7.39 2.83 3.98 _ 1894 6 1 _

Data Services (outdoor)

DL Samples GT DL Samples GT UL Throughput UL Samples GT
NETWORK DL Throughput Kbps Median Video Reproduction Video Play Start
10Mbps (%) 3Mbps (%) Kbps 3Mbps (%) Latency(ms) Without Interruption (%) Time sec
Airtel Data 4G 6,938 17.53 86.69 6,104 71.83 37 99.03 2.58
Faiba Data 4G 6,762 26.26 61.71 3,710 42.13 38 98.34 3.18
Safaricom Data 4G 14,044 64.51 87.39 12,743 65.58 34 98.73 3.91
Safaricom Data 3G 6,795 23.46 70.86 1,732 15.31 49 97.13 5.32
Telkom Data 4G 8,290 31.31 82.42 3,664 36.25 33 99.18 3.00

Data Services (indoor)

DL Samples GT DL Samples GT UL Throughput UL Samples GT
NETWORK DL Throughput Kbps Median
10Mbps (%) 3Mbps (%) Kbps 3Mbps (%) Latency(ms)
Airtel Data 4G 6,938 17.53 86.69 6,104 71.83 37
Faiba Data 4G 6,762 26.26 61.71 3,710 42.13 38
Safaricom Data 4G 14,044 64.51 87.39 12,743 65.58 34
Safaricom Data 3G 6,795 23.46 70.86 1,732 15.31 49
Telkom Data 4G 8,290 31.31 82.42 3,664 36.25 33
3 C2 - Safaricom Internal
 LOG Files| Organization
Airtel

Telkom

For Outdoor tests, arrange logs according to date & location

For indoor walk test, arrange logs according to building &

floor

4 C2 - Safaricom Internal
 Airtel

WALK TEST REPORT

TEMPLATE
Telkom

5 C2 - Safaricom Internal
 LOCATION AND IMAGE
Airtel

Telkom

6 C2 - Safaricom Internal
 METHODOLOGY Airtel

Voice Service
• 70 seconds call with a pause duration of 15 seconds ( Applies for an MS on Automatic mode) to
allow for reselection.
• Mobile to Mobile alternate calls

Data Service:
• HTTP Download and Upload for all operators on LTE
 DL 5MB file Telkom
 UL 2MB file
• Separate Data Test done for Safaricom’s UMTS

Objective
• To Compare indoor Voice and Data performance for Safaricom and Competition.
• Propose remedial measures to improve Safaricom services

7 C2 - Safaricom Internal
 OVERALL SERVICE SUMMARY
Airtel

Call setup Time (sec) Speech Quality

Call Setup Call Drop CSFB
Mean Max
NETWORK Success Ratio Ratio Average 90th Percentile CSFB Ratio Call Attempts Setup Fail Call Drops Attempts
Airtel 4G Pref 98.98% 0.40% 3.54 9.89 3.26 4.46 87.43% 1766 18 7 1544
Safaricom 4G Pref 99.78% 0.16% 3.16 5.41 3.89 4.46 95.17% 1841 4 3 1752
Telkom 4G Pref 99.38% 0.10% 3.78 6.51 3.74 4.48 93.02% 2091 13 2 1945
Safaricom 2G Pref 99.68% 0.05% 4.63 7.39 2.83 3.98 _ 1894 6 1 _

DL Samples GT DL Samples GT UL Throughput UL Samples GT Median

NETWORK DL Throughput Kbps
10Mbps (%) 3Mbps (%) Kbps 3Mbps (%) Latency(ms)
Airtel Data 4G 6,938 Telkom
17.53 86.69 6,104 71.83 37
Faiba Data 4G 6,762 26.26 61.71 3,710 42.13 38
Safaricom Data 4G 14,044 64.51 87.39 12,743 65.58 34
Safaricom Data 3G 6,795 23.46 70.86 1,732 15.31 49
Telkom Data 4G 8,290 31.31 82.42 3,664 36.25 33

Voice Service
• GSM coverage was poor in upper basement hence poor CSSR. We recommend extra indoor
coverage antenna…

Data Service:
• Safaricom LTE coverage is sufficient in entire building. UMTS suffers from interference on 2nd Floor

8 C2 - Safaricom Internal
 RF SUMMARY Airtel

Telkom

9 C2 - Safaricom Internal
DATA SERVICE SUMMARY BY FLOOR
Airtel

Telkom

10 C2 - Safaricom Internal
 COVERAGE BY FLOOR Airtel

1. Have subsequent slides that indicate coverage as;

• RSRP,RSCP, RxLev
• RSRQ,EcNo, RxQual
• PCI,PSC,BCCH, RFCN

2. Floor summary comments Telkom

11 C2 - Safaricom Internal
 Airtel

OUTDOOR DRIVE TEST REPORT

TEMPLATE

Telkom

12 C2 - Safaricom Internal
 TEST ROUTE
Airtel

Telkom

13 C2 - Safaricom Internal
 METHODOLOGY Airtel

Voice Service
• 70 seconds call with a pause duration of 15 seconds ( Applies for an MS on Automatic mode) to
allow for reselection.
• Mobile to Mobile alternate calls

Data Service:
• HTTP Download and Upload for all operators on LTE & Safaricom’s UMTS
Telkom
 DL 5MB file
 UL 2MB file
 DL 500MB file
 UL 200MB file
 Ping test
 You tube test

Objective
• Propose remedial measures to improve Safaricom services in addition to benchmarking operators

14 C2 - Safaricom Internal
 BENCHMARK HIGHLIGHTS

 Speech quality improvement seen in most towns

 Safaricom is on the lead for voice performance along AAA-yyy & ccc-xxx highway
 A number of call drops in xxx , bbb, vvv, & yyy needs to be addressed
VOICE
 Safaricom lags in zzz & mmm.
 Safaricom leads in voice metrics on the highways
 Safaricom leads on data performance in Towns, roads and highways.
 Low throughputs in Xyz town.
DATA
THROUGHPUT

 Video performance is stable on most roads

VIDEO

15 C2 - Safaricom Internal
 OVERALL SERVICE SCORE
Airtel

Call setup Time (sec) Speech Quality

Call Setup Call Drop CSFB
Mean Max
NETWORK Success Ratio Ratio Average 90th Percentile CSFB Ratio Call Attempts Setup Fail Call Drops Attempts
Airtel 4G Pref 98.98% 0.40% 3.54 9.89 3.26 4.46 87.43% 1766 18 7 1544
Safaricom 4G Pref 99.78% 0.16% 3.16 5.41 3.89 4.46 95.17% 1841 4 3 1752
Telkom 4G Pref 99.38% 0.10% 3.78 6.51 3.74 4.48 93.02% 2091 13 2 1945
Safaricom 2G Pref 99.68% 0.05% 4.63 7.39 2.83 3.98 _ 1894 6 1 _

DL Samples GT DL Samples GT UL Throughput UL Samples GT

NETWORK DL Throughput Kbps Median Video Reproduction Video Play Start
10Mbps (%) 3Mbps (%) Kbps 3Mbps (%) Latency(ms) Without Interruption (%) Time sec
Airtel Data 4G 6,938 17.53 86.69 6,104 71.83 37 99.03 2.58
Telkom
Faiba Data 4G 6,762 26.26 61.71 3,710 42.13 38 98.34 3.18
Safaricom Data 4G 14,044 64.51 87.39 12,743 65.58 34 98.73 3.91
Safaricom Data 3G 6,795 23.46 70.86 1,732 15.31 49 97.13 5.32
Telkom Data 4G 8,290 31.31 82.42 3,664 36.25 33 99.18 3.00

16 C2 - Safaricom Internal
 VOICE MOBILITY SUMMARY
Airtel

WCDMA
GSM GSM/WCDMA WCDMA Inter WCDMA/GSM
HANDOVERS GSM Intra Cell ActiveSet
Handover Handover Frequency Handover
Update
Safaricom 4G Pref 0.00 _ _ 0.00 3.77 3.03
AIRTEL 4G Pref 1.62 0.00 26.36 0.00 4.60 1.94
Failed[%]
Telkom 4G Pref 0.55 0.00 _ 0.00 1.56 1.65
Safaricom 2G Pref 0.47 0.00 _ _ _ _

Safaricom 4G Pref 0 _ _ 0 6 2
AIRTEL 4G Pref 4 0 29 0 63 9
Failed #
Telkom 4G Pref 1 0 _ 0 1 2
Safaricom 2G Pref 16 0 _ _ _ _

Safaricom 4G Pref Telkom108 _ _ 25213 153 64

AIRTEL 4G Pref 243 136 81 12684 1306 456
Success # Telkom 4G Pref 182 2 _ 23587 63 119
Safaricom 2G Pref 3360 49 _ _ _ _

17 C2 - Safaricom Internal
 Data Performance Score Cards | By Town

Video Proportion
DL Throughput DL session UL Throughput UL Session Median Without Video Video Play UMTS_RAT GSM_RAT_Proportio
NETWOK TOWN (Kbps) SR (Kbps) SR Latency (ms) interruptions Resolution Start Time (sec) LTE_RAT_Proportion _Proportion n
Safaricom_Data_3G HOMABAY 4,020 99.9 2,029 100.0 62 91.0 378 4.6 0.0 100.0 0.0
Safaricom_Data_4G HOMABAY 10,431 100.0 8,076 99.3 49 95.3 1,007 2.6 98.0 2.0 0.0
Telkom_Data_4G HOMABAY 1,850 100.0 1,465 100.0 36 98.8 616 3.1 82.1 17.9 0.0
Airtel_Data_4G HOMABAY 1,013 98.6 712 98.8 87 98.4 366 3.6 0.0 68.7 31.3

Video Proportion
DL Throughput DL session UL Throughput UL Session Median Without Video Video Play UMTS_RAT GSM_RAT_Proportio
NETWOK TOWN (Kbps) SR (Kbps) SR Latency (ms) interruptions Resolution Start Time (sec) LTE_RAT_Proportion _Proportion n
Airtel_Data_4G RONGO 966 100.0 790 100.0 59 100.0 1,080 3.3 0.0 0.0 0.0
Safaricom_Data_4G RONGO 11,918 100.0 12,478 100.0 48 100.0 1,080 2.4 100.0 0.0 0.0
Safaricom_Data_3G RONGO 4,481 100.0 2,056 100.0 42 100.0 240 3.6 0.0 100.0 0.0
Telkom_Data_4G RONGO 2,122 100.0 1,980 100.0 34 100.0 1,080 2.4 0.0 0.0 0.0

18 C2 - Safaricom Internal
 Voice Score Card | By Town
Call Setup Call Drop Call setup Speech CSFB
NETWORK TOWN Call Att Setup Fail Drop CSFB Att Success Ratio Ratio Time (sec) Quality Ratio
SAFARICOM 2G Pref Migori 130 7 0 _ 94.6% 0.0% 5.4 2.8 _
AIRTEL 4G Pref Migori 110 10 1 106 90.9% 1.0% 3.5 2.6 96.4%
TELKOM 4G Pref Migori 141 17 1 87 87.9% 0.8% 6.3 3.0 61.7%
SAFARICOM 4G Pref Migori 118 1 1 114 99.2% 0.9% 4.8 3.3 96.6%

Call Setup
Setup Success Call Drop Call setup Speech CSFB
NETWORK TOWN Call Att Fail Drop CSFB Att Ratio Ratio Time (sec) Quality Ratio
SAFARICOM 2G Pref Homabay 147 3 0 _ 98.0% 0.0% 5.2 2.9 _
AIRTEL 4G Pref Homabay 140 4 0 136 97.1% 0.0% 3.5 3.5 97.1%
TELKOM 4G Pref Homabay 141 1 0 74 99.3% 0.0% 5.7 3.6 52.5%
SAFARICOM 4G Pref Homabay 142 2 0 133 98.6% 0.0% 4.5 3.5 93.7%

Call Setup
Setup Success Call Drop Call setup Speech CSFB
NETWORK TOWN Call Att Fail Drop CSFB Att Ratio Ratio Time (sec) Quality Ratio
SAFARICOM 2G Pref Rongo 63 1 0 _ 98.4% 0.0% 5.1 2.9 _
AIRTEL 4G Pref Rongo 63 0 0 0 100.0% 0.0% 3.6 3.6 0.0% A number of Call drops in Kisii town
TELKOM 4G Pref Rongo 69 4 0 68 94.2% 0.0% 4.9 3.5 98.6%
SAFARICOM 4G Pref Rongo 65 1 0 63 98.5% 0.0% 4.2 3.6 96.9%

19 C2 - Safaricom Internal
 Highway Coverage| Nairobi - Mombasa
Signal Quality AIRTEL Frequency band Coverage
SAFARICOM NBI -MSA TELKOM
used

a) Kyumvi – UMTS poor coverage [1 km] –

b) Konza Junction –GSM/UMTS/LTE poor coverage [1 km] –
c) Emali – 4km before Emali– UMTS poor coverage [2 km]
d) Emali- Simba stretch
e) 11km past Mtito Andei–UMTS poor coverage [2.3km]
f) 4km past Voi – intermittent GSM/UMTS/LTE coverage [5km]
g) 16 Km before mariakani GSM/UMTS/LTE coverage[2.25km]

20 C2 - Safaricom Internal
 Airtel

TEST EQUIPMENT
CONFIG

Telkom

21 C2 - Safaricom Internal
 Airtel
Voice Mobile to Mobile Calls

• 2 cars calling each other and driving in the same city/territory/highway (but different location/cells) at the same time
• Both handsets on the same location/car allowed only in very special cases (for example building walk test)
• A mobile to mobile call should be considered as a single call and not two legs
• Perform All 3 Operators4G <-> 4G calls in CSFB and Safaricom2G <-> Safaricom2G (no mixed scenarios required)
• DIAL message in originating side is used as NEW START trigger point ,while CONNECT message in originating side is
used AS STOP trigger point for analysis and official reporting. These trigger points have to be considered for call setup
success rate and call setup time KPIs.
• POLQA algorithm for speech quality evaluation.
• Official Call Setup time timeout for KPI reporting: 15 s,
• Call duration: 70 s Telkom
• Pause between calls: 30 s( Outdoors) 10s(Indoors)

22 C2 - Safaricom Internal
 Airtel
Voice Mobile to Mobile Calls

• HTTP DL FDTT Multisocket (each file 500 Mbytes) - Minimum Throughput in order to consider a session as
successful: 500 Kbps for downlink; Data transfer Timeout: 7 sec (ie download for fixed time of 7 sec
• HTTP UL FDTT Multisocket (each file 500 Mbytes) - Minimum Throughput to consider a session as successful: 500
Kbps for uplink; Data transfer Timeout: 7 sec
• Latency Ping 32 bytes

• For outdoor, 3 YouTube videos played in sequence with 10 pings in between;

Video ID Description URL Duration [s]
1 Audi R8 https://www.YouTube.com/watch?v=uyPPAZUq66I 47
2 Nestle’ Telkom
https://www.youtube.com/watch?v=QoHDdunfcQ8 45
3 Al Jazeera English Live https://www.youtube.com/watch?v=jL8uDJJBjMA 45

23 C2 - Safaricom Internal