PIR – Services in stopped state after maintenance

Case Number Dev/Cloud Reference 24447 / RCA 24500

Case Start Date/Time 25-6-2023 06:00 Case End Date/Time 26-6-2023 11:30
Incident Summary
Incident description:

Customer UCCs were unavailable due to the services being in a ‘stopped’ state.

Summarized observations of the incident, impact and investigative actions taken:

After the Application Server maintenance/patching, all UCC services were started at the same time after the server
startup. This caused a license file to be in use and as a result, not all UCC services where able to start properly.
Normally, this is prevented due to the use of the automation that will stagger the startup of the UCCs to prevent any
conflicts.

Fix Provided During Restart the affected UCC Services

Incident Triage:
Final Root Cause

What was the cause of the incident and why did it happen:
Maintenance Patching of the Application Servers required restart of the Servers. Normally, these restarts are
automatically carried out in a stagged manner in order to ensure that resources and capacities are well distributed
and to prevent conflicts and locked files. This maintenance window, the UCC services were all started at the same
time due to an error on the Anywhere365 side.

When did the incident occur:

A few incidents were reported during late afternoon on Sunday 25th of June 2023 that seemed isolated and
quickly mitigated.
The majority of the issues were reported on the morning of June 26th, 2023 when a major incident was identified.

How the root cause was found, which observations and investigation was done to identify the incident:
The root cause was identified when reviewing the issue, investigating log files and review steps taken during
maintenance.

What action(s) was done to resolve the incident:

Restart the affected UCC Services.

Post Incident Action Items

Actions taken/required/lessons learned from Anywhere365:

Anywhere365 has reinforced the procedure to be used for maintenance and restarts. Focussing on the automation
to ensure the startup of UCC services in a controlled and staggered manner to prevent conflicts and locked license
files.

Additionally, Anywhere365 is reviewing current monitoring capabilities to identify events like these in a timelier
manner.

Actions taken/required from customer/partner:

No actions required from Customer/Partner side.