SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

SBI – DR DRILL DOCUMENT

Prepared by Tushar Gaikwad Page 1

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

CONTENTS
(Press ctrl + link to directly go the required page)

Contents

1. Prerequisite Checks for DR Drill......................................................................................................................3

2. Steps of DR Switch...........................................................................................................................................4
3. Application shutdown at DC............................................................................................................................5
4. Database Switch...........................................................................................................................................11
5. Application Start at DR..................................................................................................................................18
6. DNS changes...................................................................................................................................................20
7. Host Entry changes........................................................................................................................................20
8. Configuration changes...................................................................................................................................21
9. Sync AL11 directories in RP1 and transport filesystem................................................................................23
10. STMS configuration in FGP........................................................................................................................24
11. Sanity Test of Cloud Connector.................................................................................................................28
12. Mail connection in RP1 system..................................................................................................................29
13. License installation.....................................................................................................................................30

Prepared by Tushar Gaikwad Page 2

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

1. Prerequisite Checks for DR Drill

 
1. The Oracle& Kernel Patch Levels should be same of Primary & Standby Databases.

2. ORACLE_HOME directory should have sufficient space for switch over activities.

3. The Primary & Standby Database should be in Sync.

4. Server time should be sync before starting the DB at DR side.

5. Check with customer and make sure no critical and long running jobs scheduled during the DR drill.

6. Take a server reboot of all DR servers and make sure the servers are working fine.

7. Filesystem /oracle/ZPSG/ZPSG_EN/ should be empty at DR site or same as DC site. It should not

contain salary files older than 3 months.

Prepared by Tushar Gaikwad Page 3

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

2. Steps of DR Switch

1. Remove the application servers from load balancer

2. Stop the applications and CI
3. Make sure Primary & DR DB are in Sync
4. Switch Primary DB as STANDBY system
5. Switch New DR DB as PRIMARY system
6. Start CI & App Servers
7. Check connectivity in SM51, SM66, SICK
8. Change RFC FGPCLNT500 in RP1 system
9. Change the DNS of entries of RP1, FGP, EP1 and SMP systems (HRMS IT team)
10. Add the application servers in Load balancer
11. Sync all AL11 directories of RP1 system and transport directories
12. Check STMS in FGP, change RFC of TMSADM only for FGP connection
13. Check CHARM
14. Check SAP Cloud connector
15. Install SAP license (If required)
16. Check Mail connection
17. Host entry file in RP1 for Festival advance and Aadhar service

Prepared by Tushar Gaikwad Page 4

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

3. Application shutdown at DC
1. Login to load balancer and disable the application servers of RP1 and FGP systems

https://10.189.6.71:8888

Got to -> SLB->Real Services -> Search for the application server and click on Actions button and
select disable selected real services

Prepared by Tushar Gaikwad Page 5

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

Prepared by Tushar Gaikwad Page 6

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

2. Log on to the application servers of systems as per IP address sheet and stop the sap applications
and after that stop CI.

Login to PIMS using credentials : https://pims.onlinesbi.com/frmLoginACMO.aspx

Prepared by Tushar Gaikwad Page 7

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

3. Login to the server and run the below command to stop application server

Command : stopsap r3

After stopping the application check the instance number and run the command

Please refer to below excel sheet for the commands for all Prod systems

SAP stop
command.xlsx

Command: sapcontrol -nr 08 -function StopService

Prepared by Tushar Gaikwad Page 8

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

After that run command cleanipc 08 remove twice till the output shows the number of IPC-objects 0

Command : cleanipc 08 remove

Prepared by Tushar Gaikwad Page 9

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

Run the command ipcs | grep <sid>adm to check if any shared memory objects are left or not.
IF output shows no result it means everything is OK.

e.g. For RP1 system, use command : ipcs | grep rp1adm

For FGP system, use command : ipcs | grep fgpadm

After stopping the all the application servers, stop the CI in the same way after that start with DB
switch activity

Prepared by Tushar Gaikwad Page 10

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

4. Database Switch

 
I)  PRI & STDBY: SQL> select name, open_mode from V$Database;
 
            The open_mode 'READ WRITE' describes the open database and 'MOUNTED' explains the standby
database.
 
II) PRI: SQL> select max(sequence#) from v$archived_log;
    PRI: SQL> Archive log list;

 
            Both commands above give the current sequence of the archived logs.
 
    STDBY: SQL> select max(sequence#) from v$archived_log where applied='YES';
 

Prepared by Tushar Gaikwad Page 11

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

III)After ensuring that PRI & STDBY are in sync issue the commands below.
 
    PRI: SQL> alter system switch logfile;
       

        This command creates archive log files.

        Issue this commands for 4-5 times which creates archive log files & check the PR-DR Sync.
 

Prepared by Tushar Gaikwad Page 12

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

IV) Issue the below command to display the switch over status of Primary & Standby Databases.
 
    PRI & STDBY
         SQL> select name,switchover_status from v$database;

 
    The result would be 'To Standby' or 'session active' on PRI and on STDBY it would be 'session active'.

Prepared by Tushar Gaikwad Page 13

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

V)  Issue the command below on Primary to switch the Database role from Primary to standby.
 
    PRI: SQL> alter database commit to switchover to physical standby with session shutdown;

VI) Issue the command below on Standby Database to commit the role to Primary.
 
    STDBY: SQL> alter database commit to switchover to primary;

Prepared by Tushar Gaikwad Page 14

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

VII) Issue the commands below on PRI to change the Archive Destination.

  PRI: SQL> startup nomount;

    PRI: SQL>alter system set log_archive_dest_state_2='defer';
    PRI: SQL>shutdown immediate;
 

    Now the PRI System is the DR System.

    Startup the database as described below.
   

Prepared by Tushar Gaikwad Page 15

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

    PRI (Current Standby Database)

 
    PRI: SQL> startup nomount;
    PRI: SQL> alter database mount standby database;
    PRI: SQL> alter database recover managed standby database disconnect from session;
 

VIII) The Actual DR System is the PR System now. Shut it down and start the database in normal mode as
described below.
 
    STDBY (Current Primary Database)
 

Prepared by Tushar Gaikwad Page 16

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

    STDBY: SQL>alter system set log_archive_dest_state_2='enable';

    STDBY: SQL>shut immediate;
    STDBY: SQL> startup;

5. Application Start at DR
Prepared by Tushar Gaikwad Page 17
 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

1. After DB switch start the CI and after that application servers using below command

Command: startsap r3

2. After starting all the applications check the T-codes SM51 for all applications servers

Prepared by Tushar Gaikwad Page 18

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

3. Check T-code sick

4. Login to the load balancer of DR site and enable the application servers of RP1 and FGP

https://10.176.189.155:8888

6. DNS changes
Prepared by Tushar Gaikwad Page 19
 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

Contact HRMS IT team for DNS changes for RP1, FGP, EP1 and SMP systems

7. Host Entry changes

Festival advance and Aadhar update services communicates with different SBI departments so need to
do changes the host entry file in RP1 system. (Note: Do the changes only if SBI HRMS department is
performing DR drill)

#DR-10.176.1.90 eispri.sbi.co.in EISPRI.sbi.co.in

DC-10.191.171.90 eispri.sbi.co.in EISPRI.sbi.co.in (enable this entry when only SBI HRMS department is
performing DR drill)

Service port numbers for telnet check:

tcp/7805, tcp/7806, tcp/7807, tcp/7815, tcp/7816, tcp/7817,tcp/7819

8. Configuration changes

Prepared by Tushar Gaikwad Page 20

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

1. Login to RP1 system and go to T-code SM59 and change the RFC FGPCLNT500

Change the Target Host from fgpprdci to fgpdrcci and set the password for user RFC_ADM

Peform connection test and authorization test, it should be OK

Prepared by Tushar Gaikwad Page 21

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

2. Check the Portal

https://hrms.onlinesbi.com/sap/bc/ui5_ui5/ui2/ushell/shells/abap/Fiorilaunchpad.html

9. Sync AL11 directories in RP1 and transport filesystem

Prepared by Tushar Gaikwad Page 22

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

1. Login to RP1 system and go to T-code AL11 and take list of filesystem and sync it (if any new filesystem
is created)

Important Note: Make sure that while performing DR Drill the filesystem /oracle/ZPSG/ZPSG_EN
should be empty at DR site or it should be same as DC site. Because DR drill happens in every 6
months so if old salary files are present then remove them.

Command to sync: rsync -avuzx /oracle/ZPSG/ rp1adm@10.189.6.197:/oracle/ZPSG/

Below is the list of filesystem of RP1

Filesystem
/oracle/CDSbackup/
/oracle/cds
/oracle/CDSOLDDATA
/oracle/FORM16A
/oracle/Doc_Info14
/oracle/PSG_SBI_DATA
/oracle/zfestival
/ZGRNOMCAO/
/oracle/ZPSG
/oracle/ZREPORT
/oracle/ZSALREG
/oracle/ZSALREG_Temp
/oracle/ZSALREPORT
/oracle/ZTDS
/oracle/ZTRICKLE
/oracle/ZVACATION
/oracle/bank_transfer
/oracle/bank_transfer_OLD

2. Transport directories needs to be sync for all Prod and Non-prod systems

Filesystems
/usr/sap/trans/data
/usr/sap/trans/cofiles
/usr/sap/trans/buffer
/usr/sap/trans/log

10. STMS configuration in FGP

Prepared by Tushar Gaikwad Page 23

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

1. Login to FGD, FGQ and FGP systems and go to T-code SM59 and change the RFC
TMSADM@FGP.DOMAIN_FGD

Change the hostname from fgpprdci to fgpdrcci and change the password of TMSADM

Prepared by Tushar Gaikwad Page 24

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

Prepared by Tushar Gaikwad Page 25

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

Check the connection test and authorization test it should be OK

Go to T-code STMS-> Overview-Systems and select all the systems and Go to Menu-> SAP systems-
>Check-> Transport Tool

Prepared by Tushar Gaikwad Page 26

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

Click on Yes button

The output should be All Green

Prepared by Tushar Gaikwad Page 27

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

11. Sanity Test of Cloud Connector

Login to SAP cloud connector using https://10.1.147.172:8443/ link and check if the connections are active
or not.

If the connections are in red color then click on connect button from Action column.

Prepared by Tushar Gaikwad Page 28

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

12. Mail connection in RP1 system

Change the IP address in T-code SCOT if mails are not getting delivered

PR:  10.189.213.228 Port 587

DR:  10.176.200.27 Port 587 (use this only if mail are not getting delivered)

Prepared by Tushar Gaikwad Page 29

 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2

13. License installation

Download SAP license from SAP Market place for All DR systems and install it. By default temporary
license get installed which is valid for 1 month but if we are going to stay at DR site for more than 1
month then install the license or after 20 days you will get message while login for license expire.

Go to T-code SLICENSE-> Menu Edit-> Install license and select the license file.

Prepared by Tushar Gaikwad Page 30