2019-BSG-IRT-03

INCIDENT REPORT
ISM
27721
INC#:

INCIDENT INFORMATION
Client Name: Sunlife
Site Address: 5th Ave, Taguig, 1634 Metro Manila
Contact Person: Rhea Benigno
Contact Number:
Email Address: Rhea.Benigno@sunlife.com
Date/Time Incident Reported: June 17, 2021 / 2220H
Restoration Date/Time: June 29, 2021 / 2120H

ENGINEER INFORMATION
Name: Rose Ann Alvarez

Division/Department: CXBSG Alcatel Collab TASK 122883

ID#:

EQUIPMENT DETAILS
Part / Dongle / System ID Equipment Description Serial Number Remarks

CASE STATUS
(Kindly provide EXPLANATION on the Case Status Selected in the Remarks Field below. Such explanation should answer WHY or HOW such case status was
selected)
Case Code Code Description Remarks

CLO Closed Client confirmed that RM are now accessible.

RES Resolved

WIP Work In Progress

PWC Pending with Client

PAR Pending With Parts

PRI Pending With Principal

PWR Pending With Requester

Copyright 2019, TRENDS

Form Title Version Author Released by Released

November 2019
TRENDS Incident Report 3.0 BSG Customer Services Group
Page 1 of 5
 2019-BSG-IRT-03

INCIDENT OVERVIEW
(Replace this info with a brief Executive Summary of the problem, including who was affected.)
Request assistance to check Verint as encountered issue on server's recorder manager

INCIDENT DETAILS
(Chronological sequence of events. Be as in depth as possible.)
Date Time Activity
June 17, 10:35 Issue reported
2021 PM
10:40 Case Assigned
PM
10:45 Joined in the ongoing remote session with Sir Bok, Sir Jonathan and Sir Jhenix.
PM
 Checked the services if running, verified running.
 Checked the error encounter when accessing RM, verified that I’m not familiar on the
error encountered. Below image was prompt for both Recorder server when accessing
RM on WFO portal and when accessing directly on Recorder server.

11:04 Request assistance to Sir Dhex to check the said error.

Copyright 2019, TRENDS

Form Title Version Author Released by Released

November 2019
TRENDS Incident Report 3.0 BSG Customer Services Group
Page 2 of 5
 2019-BSG-IRT-03

PM
11:06 Sir Dhex joined the remote session for checking.
PM
Following items was checked by Sir Dhex:
 Checked the services, verified running.
 Checked the tomcat, verified running.
 Tried if tomcat was accessible, verified accessible.
 Tried to rebuild the RM folder to check if corrupted then checked the RM if will be able
to access, verified that issue was still encountered after rebuilding.
 Tried to reboot the server, verified that issue was still encountered.
 Checked the tomcat logs.
 Tried to Save config and then refresh cache, issue was still encountered.
June 18, 12:00 Raised the said issue on Verint for further investigating.
2021 AM
Sir Charles was assigned on the raised case on the said issue. Contacted Sir Charles to request
12:10
help in checking the said issue. As discussed with Sir Charles he is on leave that day and advised
AM
us on the needed checking, verified that all needed checking was already checked by Sir Dhex.
06:38 Request Remote assistance to check the said issue via email.
AM
10:30 Verint Support CT, Akbar was joined on the remote session for checking.
AM
Following items was checked by Akbar:
 Checked the services.
 Checked the tomcat services.
 Checked the task manager.
 Checked the logs.
12:55 As advised by Verint Support he will be checked this internally and raised the said issue on T2
PM support. He added that he will give us an update within an hour.
01:53 Request an update on Verint support for the updates.
PM
June 22, 03:40 Verint support request remote session to check the said issue.
2021 PM
Following action has been done by Verint support.
 Try to refresh the Tomcat\Work\Catalina\localhost
 Try to refresh the RM folder, the same issue encountered
 Initiate manual Distribution for the Recorder, still, the process failed
 Check the EMA for Application and Recorder, failed communication encountered.
 Check the server.xml and user.xml for possible corruption
 Check the Framework-Management user account and re-input credentials but no luck.
As advised by Verint support via email, this issue requires to raise on their T2 or higher tier due to
abnormality encountered to all servers and not only on both Recorder portals. The EMA
09:25
functionality was also affected for the Application Server. The Distribution was also failed even in
PM
manual mode. Any additional setting will not take effect on the entire EM and RM. Verint support
also added that they will give us an update coming from T2 support once available.
June 23, 08:00 Verint support requested remote session for checking and gathering of additional logs needed by
2021 PM Verint T2 support.
10:00 Session was ended and already sent the logs via Aspera.
PM

Copyright 2019, TRENDS

Form Title Version Author Released by Released

November 2019
TRENDS Incident Report 3.0 BSG Customer Services Group
Page 3 of 5
 2019-BSG-IRT-03

Verint support was requesting to have a remote session tomorrow around 1:30PM for the checking
June 24, 09:26
of the said issue and as discussed with Ma’am Rhea, the requested remote session tomorrow at
2021 PM
1:30 PM was approved.
June 25, 01:30 Remote session with Verint support.
2021 PM
The T2 support investigates the issue and checks the Central DB and Application server first
02:30 before the RM. They found the below error messages and to eliminate the alarms, they updated
PM the WLSAdmin credentials. The change required a reboot and the session resume after the
production.
06:00 Continuation of checking and for the reboot to take effect the config made by Verint support.
PM
08:00 Session ended and advised us to continue troubleshooting on Monday at 7:00PM.
PM
June 28, 04:55 As advised by Verint support, this day’s session was cancelled and will re-schedule tomorrow at
2021 PM 7:00PM.
June 29, 07:00 Remote session started.
2021 PM
As per checked by Verint support, the port 8080 should be used by TOMCAT but in this case, it
has been used by some other application. As confirmed by the customer, they have installed
JAVA on the Verint servers, and we could see that port 8080 is used by JAVA which is causing
the error.

Resolution steps done by Verint Support:

 We confirmed that port 8080 is used by JAVA.
 By selecting the PID, we killed the JAVA process from Task Manager.
 In the mentioned path, we renamed the JAVA file.

When Verint support changed the name of JAVA on the application server and the two-recorder
servers, Killed the java task via task manager, restarted the recorder tomcat service that is to allow
the tomcat to use port 8080.
The customer confirmed that RM are now accessible.

09:20
PM

Copyright 2019, TRENDS

Form Title Version Author Released by Released

November 2019
TRENDS Incident Report 3.0 BSG Customer Services Group
Page 4 of 5
 2019-BSG-IRT-03

ROOT CAUSE ANALYSIS

(Replace this info with the detailed information concerning how the incident was remedied or alleviated.)

 Port 8080 should be used by TOMCAT but in this case, it has been used by some other application. And verified
that JAVA application was using the Port 8080.

RECOMMENDATION(S)
(Replace this info with the measures that can and should be taken to prevent this issue from recurring, including specific service level monitoring
recommendations – for short and long term.)

 Recommended by Verint support to patch on the latest WFO Package and Security Kit 14 to eliminate the JAVA
that has been update.

Copyright 2019, TRENDS

Form Title Version Author Released by Released

November 2019
TRENDS Incident Report 3.0 BSG Customer Services Group
Page 5 of 5