Analysis Report on the TN12OBU101 Board Failing to Go

Online
RSL-MHL & RSL-SSR OLTs are down
Severity: Critical

Issue 01
Date 21st May 2019

HUAWEI TECH INVESTMENT CO., LTD.

 About This Document

Huawei Tech Investment Co., Ltd.

Address: Dalal Complex, 10th Floor,

Plot 1 A/2, Salem Al Mubarak St.
Salmiya Block 4, Kuwait
Website: http://www.huawei.com
Email: support@huawei.com
About This Document

About This Document

Prepared by FO/FM
Approved by Hamada Mohammed Mattar
Approval Date 2019-05-21

Huawei Proprietary and Confidential

Copyright © Huawei Technologies Co.,
Ltd
 Contents

Contents

1 Problem Description.............................................................................................1-1
2 Problem Analysis / Root Cause............................................................................2-1
2.1 Incident & Resolution classification…………………………
2.2 Root cause …………………………………………………….
3 Corrective Actions and Resolution......................................................................3-1
3.1 Corrective Actions…………………………………………….
3.2 Preventive Resolution…………………………………………
 Problem DescriptionProblem Description

1 Problem Description

On Tuesday May 21st 2019 at 12:10PM RSL-MHL & RSL-SSR OLTs went down,
“Communication with the device failed” alarm observed in the U2000. The same was
informed to Mr.Elwan from FM* (Field maintenance) team and to Technical Director
Mr.Wang. Viva team doing Preventive maintenance work caused outage by switching
off rectifier and removing battery load at the same time. Hence due to no electrical
power, OLTs went down.

Summary
Issue: RSL-MHL & RSL-SSR OLTs are down.
Severity: Critical
Incident time: 12:11:22 21st May 2019
Resolution time: 12:15:50 21st May 2019
Remarks:
a) Viva Preventive maintenance team doing PM at RSL CO.
b) Viva team performing alarm generation and creation for EMS.
c) In order to generate Load breaker alarm team removed the load breaker
while they switched off the rectifier at the same time.
d) Causing RSL-MHL, RSL-SSR OLTs down and all the connected MDU and
ONT are down.
e) Informed to switch on the rectifier immediately and then OLTs came up.
f) Outage occurred for 4minutes
 Problem Analysis / Root causeProblem Analysis / Root
cause

2 Problem Analysis / Root cause

2.1 Incident & Resolution classification:

Time Remark
5/21/2018 12:11 Incident time: Observed RSL-MHL & RSL-SSR OLTs and 18 MDUs went down.
5/21/2018 12:11 Observed "Communication with the device failed alarm" in U2000.
5/21/2018 12:12 Created a TT - INC20190521-00000001 and assigned to FM(Mr.Sherif)
5/21/2018 12:12 Informed to TD and Management in parallel.
Viva team already working for preventive maintenance work. So we contacted
5/21/2018 12:13
them to know the status of OLT at CO.
Investigated with Viva team and found that while testing the rectifier they
switched off the rectifier and at the same time they removed the load breaker, so
5/21/2018 12:13
the power got disconnected to the OLTs and went down. And all the MDUs went
down consecutively.
Immediately informed to Switch on the Rectifier, PM team switched on the
5/21/2018 12:14
Rectifier
5/21/2018 12:15 We found the OLTs and MDUs came up.
5/21/2018 12:15 Recovery time: Both OLTs are online and MDUs are online and working fine.
5/21/2018 12:45 Resolution time: Observed for 30Minutes and found OLTs and MDUs are stable
 Problem Analysis / Root cause

Please find the alarm status in the U2000 & neteco system

U2000

Neteco

2.2 Root Cause:

1- Switching off rectifier and removing battery load at the same time: While
generating the load breaker alarm, Viva Team removed the load breaker at the
same time switched off the rectifier, so the power got disconnected to the RSL-MHL
& RSL-SSR OLTs and went down.

Huawei Proprietary and Confidential 2

Copyright © Huawei Technologies Co.,
Ltd
 Corrective Actions and ResolutionCorrective Actions
and Resolution

3 Corrective Actions and Resolution

3.1 Corrective actions

1. Team should ensure the battery backup status before generating Rectifier
alarms.
2. Team should follow the Check list Step by step, Should not jump in between.
Should finish a test and then another test –one by one.
3. Viva team will be penalized for this mistake

3.2 Preventive Resolutions

1. Update Check list and make sure it is followed by Viva team during doing PM.
2. Training case has to be prepared and to be shared to all the PM team members.
3. Share this case study to all the PM teams to learn more from this mistake.