Professional Documents
Culture Documents
4G KPI and Trouble Shooting PDF
4G KPI and Trouble Shooting PDF
Analysis of Tracing
Case Study
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 2
Poor Accessibility Reasons
Common causes
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 3
Poor Accessibility Reasons
Parameter, RF channel, Interference and Transmission
Improper parameter setting
Use of non-optimal parameters
Improper license sharing policy
RF channel fault
GPS clock lost and ineffective subframe offset configuration(TDD)
Intermodulation interference (FDD)
Incorrect transmission IP address
SCTP link intermittence
Packet loss on or long delay of a transmission link
Devices
UE’s are faulty
Bad access performance of the UE
ENodeB fault(Fault alarms, VSWR alarm, board alarm)
Sleeping cell
EPC fault(abnormal operations, message response timeout, NAS procedure exception)
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 4
Content
Analysis of Tracing
Case Study
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 5
Checklist Of Required Actions For Access Problems
Action Prerequisite Purpose
1.Determine whether the problem is a top cell problem or
network-wide problem.
Action 1: performing
scope identification, KPI None 2.Analyze the major causes of access failures and come up with
trend analysis, and cause priority actions.
resolution
3.Submit a sleeping cell problem to R&D for analysis.
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 6
Checklist Of Required Actions For Access Problems
Action Prerequisite Purpose
Perform proper actions based on site conditions:
Action 4: checking version None 1.Check for version difference on the top sites.
differences and known issues
2.Check for known issues.
Action 5: checking network Perform this action based on the 2.Check whether an access problem is caused directly by resource
planning and optimization analysis results provided by action congestions or indirectly by improper parameter configurations. For details
1. how to analyze a congestion problem caused by the system CAPS over-
capacity, see LTE performance-Basic Network Planning and Optimization.
3.Check whether an access failure is caused by weak coverage.
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 7
Checklist Of Required Actions For Access Problems
Action Prerequisite Purpose
For top-cell problems, check top users. 1.Check whether access-related KPI deterioration happens on individual user(s).
Action 7: checking top UE types and
top users For network-wide problems, check top 2.Check whether the access problem is caused by a certain type of UEs.
UE types.
Perform this action for analyzing a 1.Provide the general principles for analyzing a complained problem.
Action 10: repeating a complained complained problem.
problem For other problems, perform this action
when problems needs to be repeated. 2.Repeat the problems and collect the related logs.
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 8
Content
Analysis of Tracing
Case Study
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 9
Access Performance Evaluation
› Random access and NAS failures are not considered in eNodeB performance management,
but these factors also impact user experience
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 10
Accessibility- RRC Setup Success Rate
Formula L.RRC.ConnReq.Succ/L.RRC.ConnReq.Att*100%
Associated A: L.RRC.ConnReq.Att
Counters C: L.RRC.ConnReq.Succ
Target 99%
KPI 2.03
Remark
RRC Overall
RRC Setup
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 11
Accessibility- RRC Setup Success Rate (Service)
KPI Name E UTRAN RRC Setup Success Ratio
(Service)
Measurement Scope Network Level
Target Monitoring
KPI 2.031
Remark RRC Setup
RRC Service
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 12
Accessibility- RRC Setup Success Rate (Signaling)
Formula L.RRC.ConnReq.Succ.MoSig/L.RRC.ConnReq.Att.M
oSig*100%
Associated A: L.RRC.ConnReq.Att
Counters C: L.RRC.ConnReq.Succ Mosig
Target Monitoring
KPI 2.032
Remark
RRC Signaling
RRC Setup
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 13
Accessibility- S1 SIG Setup Success Rate
Associated A: L.S1Sig.ConnEst.Att
Counters B: L.S1Sig.ConnEst.Succ
Target 98.5%
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 14
Accessibility- eRAB Setup Success Rate
Associated A: L.E-RAB.AttEst
Counters B: L.E-RAB.SuccEst
Target 99%
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 15
Accessibility- eRAB Setup Success Rate (QCI=i)
Formula L.E-RAB.SuccEst.QCI.i/L.E-
RAB.AttEst.QCI.i*100%
A: L.E-RAB.AttEst.QCI.i
Associated
B: L.E-RAB.SuccEst.QCI.i
Counters
(i=1,2,3,…,9)
Unit Percentage (%)
Target Monitoring
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 16
General Process
• Determine the scale of problem
› From the performance statistic, we can determine the scale of the problem, is it global
problem, or cell level problem or just some individual UE problem. Then we can select
bottom N cell as optimization target
› Customer complain is an effective way to locate the individual UE problem
• For large scale access failure
› Check eNodeB alarm and basic configuration
› Check RF channel problem
» Once RF channel is abnormal, such as high VSWR, low RSSI Too Low or RSSI is
unbalanced, then the access performance should be significantly affected.
• For individual UE problem, signaling analysis can be used to locate the detailed cause
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 17
Handle Enodeb Generic Alarm
• Tools: M2000 or WebLMT
• Procedure
› Check the generic hardware fault alarm, such as hardware fault, board unusable , or any
other alarms which are related with hardware failure or link failure.
» Solution: Replace the faulty component.
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 18
Check DL Power Configuration
» PA = -10lg(Pb+1)
• Related command:
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 19
Check CRPI Rate
• If the bandwidth is 10MHz or above, the CPRI rate should be equal or more than 2.5Gbps, otherwise,
the cell can’t be setup
• If the rate is not correct, change the CPRI module
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 20
Check S1 Interface Status
• If SCTP link is faulty, check the SCTP
configuration, such as peer IP address, SCTP
port number
• If S1intrface state is faulty, check the PLMN
and operator ID.
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 21
Check Cell Barred Status
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 22
Check RF Channel Problem
• Due to the bad feeder quality, incorrect feeder connection or strong interference, it
might affect RF channel performance which has impact on UE access.
• Relevant problem:
› High VSWR: ALM-26529 RF Unit VSWR Threshold Crossed
› Low RSSI: ALM-26521 RF Unit RX Channel RSSI Too Low
› RSSI unbalance: ALM-26522 RF Unit RX Channel RSSI Unbalanced
» RSSI of the main RX channel and the RSSI of the diversity RX channel exceeds 10 dB.
› High RSSI: Caused by interference
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 23
Cause Of High VSWR
• VSWR: Voltage Standard Wave Ratio, indicate the if feeder impedance is match or
not. The normal VSWR range is 1~1.5. If current VSWR is more than a specified
threshold, then eNodeB will generate relevant alarm.
• Alarm generation:
› Cell is activated
› RF power is more than 34dBm/channel
› VSWR is more than a specified threshold
• Possible cause:
› Incorrect VSWR alarm threshold
› RF unit hardware fault
› RRU/RFU wrong connection
› Bad feeder quality or nonstandard feeder installation
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 24
Solution For High VSWR
• Step 1: Check if the VSWR alarm threshold is correct (default is 2dB).
› Note: RRU TX channel will automatic closed if VSWR is extremely high (more than post- processing threshold)
• Step 2: If alarm threshold is correct, then check if relevant feeder installation and RRU connections are meet the
standard.
• Step3: After the feeder tuning, if TX channel is closed, then activate TX channel again
› MOD TXBRANCH
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 25
Cause Of Low RSSI
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 26
Solutions For Low RSSI
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 27
Cause Of Imbalance Of RSSI
Alarm generation: RSSI of the main RX channel and the RSSI of the diversity RX
channel exceeds 10 dB.
• Possible causes
› High interference
› RRU cross connections
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 28
Solution of Imbalance of RSSI
• UL interference check
› From web-LMT: Perform spectrum detection to evaluate UL interference
› From M2000 client: Perform interference detect monitoring
› Find out interference source
• Check RRU connections, avoid cross connections, show as below
A A
N N
T T
1 2
RRU1 RRU2
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 29
Classification Of Interference
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 30
How To Located Inter-modulated Interference
• This kind of interference increases with eNodeB transmit power, so we can simulate the DL load,
and check RSSI, if it obviously increases with the DL load incensement, then we can determine the
interference
• Load simulation:
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 31
How To Located External Interference
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 32
Analysis Of Radio Network Fault
• Possible cause
› UU message process failure due to UE capability (UE doesn’t support relevant service)
› eNodeB failure during ERAB setup
• Solutions:
› Check UE capability
› Check eNodeB status
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 33
Analysis Of No Radio Resource
• Possible cause
› Radio admission failure for ERAB
• Solution:
› Check if admission parameters is correct
› Consider to enable load balance algorithm
› Consider capacity extension
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 34
Analysis Of Security Mode Failure
• Possible cause:
› UE security capability is not identical with eNodeB security capability
• Solution:
› Modify the eNB security algorithm to match it.
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 35
Analysis Of EPC Fault
• Cause:
› S1 message validity check failure, including
» Error exist in S1 message
» Duplicated ERAB ID
» Conflict with other procedures
• Solution:
› Contact EPC engineer to locate the problem
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 36
Analysis Of Transport Fault
• Possible causes:
› Transport admission failure
› S1 link failure during S1 message exchange
› Incorrect IP path configuration
› Wrong QoS configuration in EPC
• Solutions:
› Check transport admission algorithm, consider extend S1 bandwidth
› Check S1 interface
› Check IP path configuration
› Check relevant QoS configuration, AMBR should be more than 0, and for initial ERAB setup,
the QCI must be 6~9
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 37
Content
Analysis of Tracing
Case Study
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 38
Analysis Of Tracing
• Driver test tracing analysis
› We can get all UU interface messages, internal UE message as well as RSRP value
• eNdoeB tracing analysis
› We can get UU interface message and S1 interface message
• Form the above tracing analysis, we can locate the problem with the following
causes:
› Random access failure
› RRC setup failure
› NAS procedure failure
› ERAB setup failure
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 39
Random Access Failure
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 40
RRC Connection Failure
UE eNodeB
RRC_CONN_REQ
RRC_CONN_REJECT
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 41
RRC Connection Failure (Cont.)
UE eNodeB
RRC_CONN_REQ
RRC_CONN_SETUP
RRC_CONN_REQ
• Scenario 2: UE doesn’t receive “RRC connection setup” message, and repeat sending
request message until timer is out
• Possible cause:
› Poor DL coverage, check RSRP and RSRQ
• Solution:
› Perform RF tuning
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 42
RRC Connection Failure (Cont.)
UE eNodeB
RRC_CONN_REQ
RRC_CONN_SETUP
Waiting “RRC
Compelet”time out
• Scenario 3: eNodeB send RRC setup message, but doesn’t get complete message
• Possible cause
› If UE doesn’t reply RRC complete, it should be UE problem
› If UE already send RRC complete, then it should be poor UL coverage
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 43
NAS Procedure Failure MME
eNodeB
UE
S1AP_INITIAL_UE_MSG
S1AP_DL_INFO_TRANSF
RRC_DL_INFO_TRANSF
RRC_UL_INFO_TRANS
S1AP_UL_INFO_TRANSF
S1AP_UE_CONTEXT_REL_CMD
S1AP_UE_CONTEXT_REL_CMP
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 44
eRAB Setup Failure
eNodeB MME
UE
Uu:RRC/NAS S1-MME:S1AP
S1AP_INITIAL_CONTEXT_SETUP_REQ
RRC_SECUR_MODE_CMD
RRC_SECUR_MODE_CMP
RRC_UE_CAP_ENQUIRY
RRC_UE_CAP_INFO
RRC_UE_CAP_INDICATION
RRC_CONN_RECFG
RRC_CONN_RECFG_CMP
S1AP_INITIAL_CONTEXT_SETUP_RSP
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 45
eRAB Setup Failure Cause
• If any failure occurs during ERAB setup, we can directly get the failure cause form
performance statistic, the relevant counters are:
› L.E-RAB.FailEst.NoReply (No response from UE)
› L.E-RAB.FailEst.MME (EPC fault)
› L.E-RAB.FailEst.TNL (Transport fault)
› L.E-RAB.FailEst.RNL (Radio network fault)
› L.E-RAB.FailEst.NoRadioRes (No radio resource)
› L.E-RAB.FailEst.SecurModeFail (Security mode setup fault)
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 46
Faulty Signaling of No UE Response
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 47
Fault Signaling of Other Cause
• Scenario: eNodeB receives “initial context setup request” or “E-RAB setup request”
message, after that, if failures happen, then eNB replies response message with
failure cause
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 48
Content
Case Study
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 49
Case 1 – IP Path Configuration Leads To Low Access
Ratio
• Description: In one live network, ERAB setup ratio is very low, but it’s not zero.
• Analysis:
› Check the performance statistic of ERAB release, we found that most of failures are
caused by transport resource not available
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 50
Case 1 –IP Path Configuration Leads To Low Access Ratio
(Cont.)
• Now we can locate that the problem is due to IP path configuration. We check eNodeB
configuration and find a valid IP path, the connection is valid. And it is also weird that this
problem occurs with a probability
• Then we analyze the message again, and find that in the “UE context setup request”
message delivers variable GTPU address, and these address is not identical with eNodeB
configuration.
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 51
Case 1 –IP Path Configuration Leads To Low Access Ratio
(Cont.)
• Conclusion :As the SGW deliveries multiple GTUP address to eNodeB, but we only configure one
IP path to SGW, so it causes the partly ERAB setup failure
• Solutions: We confirm this with EPC engineer, SGW pool function is used, so there are multiple IP
addresses for SGW. Then we make up the IP path for all address, the problem is cleared.
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 52
Case 2 – Multi-mode UE Attach Failure
S1 trace
UU trace
• Description: During the commissioning of one live network, we found that UE attach fails.
• Alarm info: None
• Message Tracing: From the message tracing, we can see that most of access procedures are
normal, after ERAB setup, MME release the connection with the cause “normal release”
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 53
Case 2 – Multi-mode UE Attach Failure (Cont.)
• Analysis
• Since the most of procedures are normal, and eNodeB consider that it’s a normal release, so
we locate that the problem is due to NAS failure.
• So we continue to analyze the previous NAS message, and in the DL NAS message, we can
see that MME response the attach with the cause “MSC-temporatily-not-reachable”, as show
above. now we can confirm that this is root cause of attach failure
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 54
Case 2 – Multi-mode UE Attach Failure (Cont.)
• Analysis (Cont.)
› Why should UE need attach to MSC ? We check the UE model, it ‘s Huawei E398, with
multi-modes(GSM/UMTS/LTE). So we assume that this UE performs combined attach.
From the attach request message we verify our assumption.
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 55
Case 2 – Multi-mode UE Attach Failure (Cont.)
• Conclusion:
› We check that there is no CS domain configuration in current EPS network, so MME only replies PS attach
accept, and also inform that MSC can’t reachable
• Temporary solution:
› Change UE attach mode with PS only
› Add CS domain configuration in EPC
• Final solution:
› Update MME to compatible with combined attach even there is no CS domain
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 56
Case 3: eRAB Setup Low due to RRC License Limited
• In Thailand True LTE Network, several Clusters cannot pass KPI E-RAB Setup Success Rate;
• Checked cell level KPI and found the failures, several sites have much high number of eRAB
Setup failures, which cause the KPI not pass;
• The major faiure is due to “No Resource”;
• Checked the number of RRC user, which reached the licensed number;
• After expanded license, the KPI became normal.
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 57
Page 57
Thank You
HISILICONTECHNOLOGIES
HUAWEI SEMICONDUCTOR
CO., LTD. Page 58