9067387 Blade no longer ready, I/O error, Link monitor die

Detail Information Chronology 9067387 Blade no longer ready, I/O error, Link monitor die
Config synced and traffic
STATUS Prepared spare module and NTT & TSEL continue Successfully replace the
meanwhile NTT team is on the way Fortinet release the analyze result FIM through to FW-GI-TBS1 and
Ticket Number: 9067387 NTT Open Ticket to troubleshoot at TBS
Around 18.30 NTT all back to normal
Open Date: 23 Dec 2023 at 7:44 PM Fortinet (9067387) to TTC TBS
Team found massive
Continue troubleshooting with TAC: Config FIM2 can’t be
Resolution: 27 Dec 2023 around 10:00 AM email alert All FW TBS Remote session with RMA Approved by
a. Shut and no shut LAG interface synced and escalate to
Ticket Priority: P2 status collected TAC Engineer Fortinet Team
Fortinet
b. Reseat FIM-7920-E on FW-GI-TBS1
Device: FW-GI-TBS1 c. Gathered logs to analyzed by Fortinet Ensure Service still
TAC Maintain
PROBLEM DESCRIPTION
LAG (aggregate Link) to Gi-Switch was down on FW-GI- 23/12 23/12 23/12 23/12 23/12 24/12 00:30 - 24/12 24/12 Christmas 26/12 26/12 26/12 26/12 27/12
TBS1 due to boot devices degrade indicated by error 18:30 19:06 19:44 19:53 23:50 04:00 15:03 Day 09:00 10:30 17:01 19:00 10:00
message "EXT2-fs (sda3): previous I/O error to
superblock detected” Day-1 Day-2 Day-3 Day-4 Day-5
SYMTOMPS
Captured logs on FW-GI-TBS1 when the issue appeared Interface status FW-GI-TBS1 during issue Session FW-GI-TBS during issue
1. Message "EXT2-fs (sda3): previous I/O error to
superblock detected” appeared
2. Message “blade in slot 2 of chassis 1 is no longer
ready to process traffic” appeared
3. LAG Interface Down
FW-GI-TBS1
IMPACTED SERVICES
DNS, HTTP & SMTP but those services re-routed
automatically to FW-GI-TBS2 & 3 (Services Maintain)
Interface status FW-GI-TBS1 after replace FIM-7920E
ACTION FROM FW POV
4. Open ticket to Fortinet
5. Coordinate with surrounding team Session FW-GI-TBS1 after replace FIM-7920E
6. Isolate/shutdown interface from peer device (GI
Switch 3&4)
7. Prepare spare and Replace FIM-7920E
FW-GI-TBS2

ROOT CAUSE Diagnose Status of

Boot Device in FIM-7920E degraded shown by this Modules FW-GI-
message: "EXT2-fs (sda3): previous I/O error to TBS1
superblock detected” FW-GI-TBS1

NEXT PLAN
Periodically to monitor the health check of FW-GI-TBS

STATUS
FW-GI-TBS3
NTT & TSEL agreed to keep this ticket open for next 2
days and monitor
Sensitivity Label: General