Ant

SERVICE NOW- HOW TO UPDATE EXECUTIVE SUMMARY
=============================================
Issue Description : KIN Server Down

Observed Business Impact: RMU's, Punchtools, Layway are down
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: Bypassed UPS / Powered on KIN via BX / Restored RAID /
Booted KIN with F6 to accept array rearrangement / Rebooted KIN server / Restore
files, directories from Tape / Complete Restoration done from tape / Hotbox
replaced
Alerting:K_Server was notified by an automated monitoring alert / User Raised
Ticket
Issue Description : KIN DB Down

Observed Business Impact:
Restoration Details: Rebooted KIN server / Replaced Processor card
Alerting:User Raised Ticket
Issue Description : Virtual KIN Server Down

Restoration Details: Smith, Terence fixed the issue (started the KIN server in
VMware)
Alerting:K_Server was notified by an automated monitoring alert
Issue Description : UPS Down

Observed Business Impact: No battery backup for V-KIN Server (RMU's, Punchtools,
Layway will down)
Restoration Details: New UPS installed for V_KIN
Alerting:K_Server was notified from V-KIN Server down alert
********************************************************************************
Issue Description : RX server Down

Observed Business Impact: Pharmacist not be able fill prescription
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Bypassed UPS and powered on RX server.
Alerting: K_Server was notified by an automated monitoring alert / User Raised
Ticket
Issue Description : RX System is thrashing -Pageln 13840:00 PageOut: 2739.00

Observed Business Impact: May be degrade the performance
Restoration Details: PageIN & PageOUT ratio came down normal
Alerting: K_Server was notified by an automated monitoring alert
Issue Description : PDX Down due to upgrade Failure

Restoration Details: Restored PDX from tape backup.
Alerting: User raised a ticket.
Issue Description : CPU Workload High. CPU Aggregate -1 is 99.99% busy in RX / IVR
Restoration Details: CPU utilization came down automatically and its normal now /
Rebooted RX
Alerting: Netcool Alert
********************************************************************************
Issue Description : IVR Server Down

Observed Business Impact: IVR is not available, No Automated Refill available.
Observed Member Impact: None
Restoration Details: Bypassed UPS / Powered on via ILO / Replaced systemboard /
Replace Dialogic Card
Alerting: K_Server was notified by an automated monitoring alert. / User Raised
Ticket
Issue Description : IVR System -Too many defunct/zombie processes

Restoration Details: Killed Zombie / Defunct processes & Restarted the SEA agent
Alerting: Proactive alert
===================================================================================
================================
Issue Description : Daily backup not performed on Evault.
Observed Business Impact: None
Restoration Details: L3 has Reinstalled the agent and Re-registered, sync'd and
updated job.
Alerting: Evault Portal alert
Issue Description : Hard drive failure in KIN / RX / IVR

Observed Member Impact: NoneSERVICE NOW- HOW TO UPDATE EXECUTIVE SUMMARY
=============================================

replaced
Ticket
SERVICE NOW- HOW TO UPDATE EXECUTIVE SUMMARY
=============================================
replaced
Ticket


VMware)

Layway will down)
********************************************************************************

Ticket


Rebooted RX
********************************************************************************

Ticket

===================================================================================
================================
updated job.

Affected Business Unit(s): Pharmacy / Store
Restoration Details: Replaced defective hard drive
Ticket
Issue Description : Tape backup issue in RX server

Restoration Details: Replaced Tape drive / Cleaned Tape drive / Replaced
systemboard, scsi cable, cage etc
Alerting:K_Server was notified by an automated monitoring alert. / User Raised
Ticket
Issue Description : RX / IVR server not reporting in SRM

Observed Business Impact: Server not reporting in SRM
Restoration Details: Restarted the SRM agents
Alerting: K_Server was notified by an Monitoring team
Issue Description : ITM Monitoring Offline Agent Restart - RX / IVR server

Observed Business Impact: Unable to monitoring the server
Restoration Details: Restarted the ITM agents
===================================================================================
======
Hi Terry,
The store having new KIN server (Dell - 430)

Spoke with Office Associate - Pamela Shaffer,
She said this morning VKIN was hung, hence they rebooted...
After reboot VKIN session not invoking
Please do the needful.
INC0363236
Tivoli team
Eduardo Manenti De Araujo XXX
Fabio Antonio Baldim XXX
Dipsmita Das2/India/IBM
Isaque Benatti Souza
=====================
Knowledge article KB0022536:
Symptoms
a. Procedure to install the new V-KIN server.
b. Procedure for V-KIN conversion.
c. Procedure to return the old KIN, Digi, monitor and tapes
Information:
1. The KIN conversion is to be performed two business days after the new V-KIN
server was installed.Hardware Deployment will call store the day before conversion
to confirm the conversion will take place the next morning.
2. Click here for the roll out schedule
3. The servers are configured at SHI, this includes imaging and installing a USB
flash drive for both the server and NAS.
4. The existing KIN server is on IP .202, after the conversion the new VKIN server
will be on .202
Cause
New V-KIN Server is replacing the IBM Netfinity 5600 Server
Resolution
Follow these steps to fix the issue
***IMPORTANT***: If you have any issues during the V-KIN Server

installation/conversion, call Hardware Deployment – Do NOT call the Help Desk.
1. Click here for the procedure to install the new V-KIN server
2. Click here for V-KIN conversion procedure
3. Click here for the Old KIN Removal procedure
========================

Observed Business Impact: when kin is down, Store cant end cash, End day, Can't do
online orders, RMUs offline, they can't punch in or out.
Observed Member Impact:: Associates can't perform any kin related activities
Affected Business Unit(s): Kmart
Restoration Details: Bypassed UPS / L3 Powered on kin VM / Rebooted KIN server /
Restore files, directories from Tape / Complete Restoration done from tape /
Powered on the UPS.
Alerting: Netcool Alert / User Raised Ticket / None

Observed Business Impact: Pharmacy can't able to work on pdx application.
Observed Member Impact: Pharmacists not able to fill prescriptions
Affected Business Unit(s): Kmart Pharmacy
Restoration Details: Bypassed UPS and powered on RX server / Powered on the UPS /
Complete Restoration done using Lifeboat CD with 10 Oct Tape / Rebooted RX server
due to hung state.


Restoration Details: Replaced Tape drive / Cleaned Tape drive / Reseated Tape drive
/ Replaced systemboard, SCSI cable, cage etc
Alerting: User Raised Ticket / None
Issue Description : zombie / Defunct Process in IVR / RX server

Restoration Details: Killed Defunct processes and restarted the SEA agents.
Issue Description : High CPU workload RX server
Observed Business Impact: No impact to business as it happen during non business
hours.
Observed Member Impact: No impact to business as it happen during non business
hours.
Restoration Details: Restarted lcfd agents, now CPU workload is normal
Alerting:Netcool Alert
Issue Description : New UPS to be installed.

Restoration Details: With help of store person, connected the v-kin and nas to the
new UPS.
Alerting: Proactive ticket
Issue Description : Need to check the UPS status for V-KIN

Restoration Details: store person confirmed that V-KIN power cords are connected
to UPS.

Issue Description : Hard drive failure in V-Host (V-KIN)

Alerting: vCenter Alert
Issue Description : Server not reporting in SRM

Affected Business Unit(s): None
Restoration Details: Restarted SEA Client & SRM Agent
Alerting: Proactive auto alert by BEM

Restoration Details: issue resolved automatically.
Proactive auto alert by BEM
Issue Description : ITM6 Monitoring Agents Offline

Restoration Details: Restarted ITM Agent
Alerting:K_Server was notified by Tivoli team.


Issue Description : Unable to Archive backup.

Restoration Details: Backup Archived to IVR
Alerting:Netcool Alert / User Raised Ticket / None

Restoration Details: Resolved automatically and CPU workload is normal
Issue Description : System is Thrashing

Restoration Details: Rebooted RX server / Resolved automatically
Alerting:K_Server was notified by an automated monitoring alert.
Issue Description : NAS not pinging.

Restoration Details: L3 worked with store and fix the power source issue.
not required. end user would not know if the device were functioning properly or
not.
Issue Description : iDrac not pinging.

Restoration Details: Reset the iDrac setting as per article #KB0031179
Issue Description : HOST & iDrac not pinging.
Restoration Details: Bounced the n/w Port by Network Team and Reset the iDrac
setting
Alerting: L3 Alert
Issue Description : KIN Power Supply is not Redundant

Observed Member Impact:: None
Restoration Details: Reseated power cables.
Alerting: Vcenter Alert
Issue Description : Tomcat appears to be stuck

Observed Business Impact: They can't punch in or out.
Observed Member Impact:: Associates can't perform punch in / out activities
Restoration Details: Issue resolved automatically by itself / Restarted Tomcat and
its running fine.
Alerting: BEM Alert
\Use Generic CI if we dont find the CI in ServiceNow.
PowerEdge T430 Server-GENERICASSET01178

HP ProLiant ML350 G3 Server - GENERICASSET01147
HP ProLiant ML310 G5 IVR Server - GENERICASSET00858
T750 IVR UPS-GENERICASSET01169
EATON 1500 UPS - GENERICASSET01158
EATON Powerware 5125 UPS - GENERICASSET00058
Information:
You can use the System Management Homepage to check the status of the hard drives
(and other hardware within the server).
Open Internet Explorer
In the address field, key: https://rx.NNNN.st.kmart.com:2381 Where NNNN is the
store number (kmart is the pass)
Click Smart Array 6balkuma1@in.ibm.com41 Controller in Slot N (2 for G3, 3 for G4),
under Storage, for hard drive info.
2381 Where NNNN is the store number (kmart is the pass)

Click Smart Array 641 Controller in Slot N (2 for G3, 3 for G4), under Storage, for
hard drive info.
Issue Description : NAS backup not happening

Observed Business Impact: There is no impact to the business.
Observed Member Impact:: There is no impact to the Members
Restoration Details: Replaced defective hdd in NAS device and configured nas.
Alerting: proactive ticket.
NCC configured fes2u1 port 16 to vlan9. pxe booted into redhat install.
425118 HP ProLiant ML310 G5 IVR Server - GENERICASSET00858

425170 HP ProLiant ML350 G3 Server - GENERICASSET01147
511265 PowerEdge T430 Server-GENERICASSET01178
141566 DS715 Disk Station NAS server-1540MUN539805
CI #435465 - NAS STORAGE
Initiate the Manual Backup on Carbonite Server Backup portal
rx7413 -Backup Failed -Carbonite Server Backup
Issue Description : Backup issue in RX server

Observed Business Impact : There is no impact.
Observed Member Impact : There is no impact.
Restoration Details:Ran manual backup
Alerting: eMail Alert


VMware)

Layway will down)
********************************************************************************

Ticket

Rebooted RX
********************************************************************************

Ticket

===================================================================================
================================
updated job.

Ticket
Ticket


===================================================================================
======
Hi Terry,

INC0363236
Tivoli team
=====================
Symptoms
Information:
will be on .202
Cause
Resolution

========================

Powered on the UPS.

due to hung state.




hours.
hours.

new UPS.

to UPS.











not.


setting
Alerting: L3 Alert


its running fine.
Alerting: BEM Alert

Information:

hard drive info.




Ticket

Ticket

===================================================================================
======
Hi Terry,

INC0363236
Tivoli team
=====================
Symptoms
Information:
will be on .202
Cause
Resolution

========================

Powered on the UPS.

due to hung state.




hours.
hours.

new UPS.

to UPS.











not.


setting
Alerting: L3 Alert


its running fine.
Alerting: BEM Alert

Information:

hard drive info.




Ant

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ant

Uploaded by

Copyright:

Available Formats

SERVICE NOW- HOW TO UPDATE EXECUTIVE SUMMARY

Issue Description : KIN Server Down

Issue Description : KIN DB Down

Issue Description : Virtual KIN Server Down

Issue Description : UPS Down

Issue Description : RX server Down

Issue Description : RX System is thrashing -Pageln 13840:00 PageOut: 2739.00

Issue Description : PDX Down due to upgrade Failure

Issue Description : IVR Server Down

Issue Description : IVR System -Too many defunct/zombie processes

Issue Description : Hard drive failure in KIN / RX / IVR

Issue Description : KIN Server Down

Issue Description : KIN DB Down

Issue Description : Virtual KIN Server Down

Issue Description : UPS Down

Issue Description : RX server Down

Issue Description : RX System is thrashing -Pageln 13840:00 PageOut: 2739.00

Issue Description : PDX Down due to upgrade Failure

Issue Description : IVR Server Down

Issue Description : IVR System -Too many defunct/zombie processes

Issue Description : Hard drive failure in KIN / RX / IVR

Issue Description : Tape backup issue in RX server

Issue Description : RX / IVR server not reporting in SRM

Issue Description : ITM Monitoring Offline Agent Restart - RX / IVR server

The store having new KIN server (Dell - 430)

Please do the needful.

Knowledge article KB0022536:

a. Procedure to install the new V-KIN server.

b. Procedure for V-KIN conversion.

c. Procedure to return the old KIN, Digi, monitor and tapes

2. Click here for the roll out schedule

Follow these steps to fix the issue

***IMPORTANT***: If you have any issues during the V-KIN Server

2. Click here for V-KIN conversion procedure

3. Click here for the Old KIN Removal procedure

Issue Description : KIN Server Down

Issue Description : RX server Down

Issue Description : IVR Server Down

Issue Description : Tape backup issue in RX server

Issue Description : zombie / Defunct Process in IVR / RX server

Issue Description : New UPS to be installed.

Issue Description : Need to check the UPS status for V-KIN

Issue Description : Hard drive failure in KIN / RX / IVR

Issue Description : Hard drive failure in V-Host (V-KIN)

Issue Description : Server not reporting in SRM

Issue Description : Server not reporting in SRM

Proactive auto alert by BEM

Issue Description : ITM6 Monitoring Agents Offline

Issue Description : KIN DB Down

Issue Description : PDX Down due to upgrade Failure

Issue Description : Unable to Archive backup.

Issue Description : High CPU workload RX server

Issue Description : System is Thrashing

Issue Description : NAS not pinging.

Issue Description : iDrac not pinging.

Issue Description : KIN Power Supply is not Redundant

Issue Description : Tomcat appears to be stuck

\Use Generic CI if we dont find the CI in ServiceNow.

PowerEdge T430 Server-GENERICASSET01178

2381 Where NNNN is the store number (kmart is the pass)

Issue Description : NAS backup not happening

425118 HP ProLiant ML310 G5 IVR Server - GENERICASSET00858

Initiate the Manual Backup on Carbonite Server Backup portal

rx7413 -Backup Failed -Carbonite Server Backup

IMPORTANT: If you have any issues during the V-KIN Server

IMPORTANT: If you have any issues during the V-KIN Server

IMPORTANT: If you have any issues during the V-KIN Server