2017-590 HowTo Setup DR ctabase (ANY PLATFORM) using Oracettg Data Guard (Physical Standby) - Managed Hosting ternal - Confluence Prot Pages /... | OLD_DB HowTo HowTo - Setup DR database (ANY PLATFORM) using Oracle11g Data Guard (Physical Standby) Created by Vertta Moody, last modified by Sujatha Golla on Aug 28, 2015 This document provides step by step instructions for setting up DR and are Platform Independent including AP & FG7mode ‘or cross platform, If you are searching for documentation for BBCS steps please click here HowTo - Build DR database using Oracle11g Data Guard (Physical Standby) Prerequisites: Setup Standby Configuration Verification of Setup Configure archive log cleanup on DR server via oracle cron Configure monitoring Latency between production and DR standby database - Gap Sequence on DR server via oracle cron: = Troubleshooting Prerequisites: Hardware and Software Requirement: + Primary and Standby database must have enough memory for production SGA and other memory parameters to fit. RAM size does not have to be identical but every effort should be made to match, + Operating system running on the primary and standby systems must be Redhat (some system will have a lower version of Redhat) + QSM of DBO1 or DBOO1 should be establish and broken, this will provide all of the Oracle data that is needed + Primary database must be in archive log mode, if not contact requestor to arrange a downtime for production + Enable Force logging on primary database to protect against unlogged direct writes in the primary database that cannot be propagated to the standby database. + Ensure DBA team has CR for creating the Oraclettg data guard database. Also make sure, CR is submitted for storage team to mirror the DB volume and restore the snapshot of database taken on production server to DR server which will be used to configure standby database. + Archive log destination should be created by storage team NOTE: If this setup is AP or a VSM cannot be completed, a QSM can be taken and Oracle does not have to be installed. All oracle software will be a part of the QSM. Once QSM is broken the location should be mounted as Jhomeloracte (AP) , home/oracie/db01 or db001 FG7. If this is FG7, the /etc/fstab should be updated by ITG Platform OPS Example: 10.2.173.2:/vollfapedr_100261_s{089718/db01 /homelorace nfs ‘actimeo=0,nw,bg,hard,nointr suid, tep,noatime, timeo=600,rsize=32768,wsize=32768,vers=3 0 0 10.2.173.2:/vollfapa_dbarch/apedr_100261_sf089718 Jhomeloracle/arch nfs actimeo=0,rw,bg,hard,nointr,suid,tep,noatime,timeo=600,size=32768, wsize=32768, vers=3 0 0 + Make sure there is a CR created for DR configuration for a database server + Gather information about the production database server and DR server for which you are setting up standby database Information Example Client Name University of Newcastle Is the client aRAC system? NO Production DB server hostname —_apcprd_100261_sf0451 26_db01 ‘apedr_100261_sf08971 8 dbo DR DB server Hostname hip ieouence hippo 8443fpaqesviewpage action?pageld=21263196 12017-5:30 How/To- Setup DR database (ANY PLATFORM) using Oracletig Data Guard (Physical Standby) - Managed Hosting Internal ~ Confluence Proc. Information Example Production DB server Location SYD1 DR DB server location sYD2 Make sure DR DB personas are similar to production DB personas in resources such as CPU and physical memory. If resources on DR DB nodes are much less compared to production DB personas, bring it to SDM's attention to make sure itis fixed. If the SGA can at least fit into the memory and CPU are less, is fine. Also, make sure the DR server is physically located in different data center than that of production server, On Primary Production Database Do these steps before contacting the storage team to start the mirror 4. Make sure production database is in archive log mode. If not, coordinate required downtime to put the database in archive log mode. a) Check if the database is in archive log mode, please enable archive log by coordinating downtime. SQL > select log_mode from v$database; If database is in no archivelog mode, notify requester to establish a downtim SQL> shutdown immediate SQL> startup mount SQL> alter system set log _archive_dest_1="locati 2. Make sure to enable force logging SQL> alter database force logging; 3. Make sure archive_lag_target is set to 3600 seconds(1 hour) to generate an archive log at least once every hour. SQL> alter system set archive_lag target=3600 scope-both sid='* 4, Create a standby control file and Pfile on primary database which will be used to configure standby database on DR server. Create Directory "DR" for use below. SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS — ' /home/oracle/DR/standby_controlfile.ct, SQL > CREATE PFILE="/home/oracle/DR/initdr_stby.ora’ FROM SPFILE; hip ileontuence hpdio 8444paqesvlewpage.action?pageld=213263195 282017-5:30 How/To- SetupDR database (ANY PLATFORM) using Oracletlg Data Guar (Physical Standby) - Managed Hosting Internal ~ Confluence Product. 5. Contact storage team to setup DR DB volume on DR server, Storage team is responsible for creating a snapshot of production DB volumes and restoring to DR server under Flexgen BBCS. ‘SPECIAL NOTES ‘Submit CR assigned to Storage OPS with priority marked as high. Please send email to Service Desk and Storage OPS with CR details. Wiki Link with storage setup details for BBCS ONLY: Disaster Recovery - FlexGen BBCS - SetupiFlexGenBBCS-Setup-StorageSetup APIFG7 mode, QSM can be established to bring over db01/db001 database. Once the mirror is broken, exports for the volume should be setup so that the DR server can mount location. ‘AP mount location as Jhome/oracle FG7 mount location as /home/oracle Proceed with next steps. Once storage confirms DR DB volumes are setup, Verify and proceed with standby database configuration. Setup Standby Configuration 1. Start Listener on Standby Server. -bash-4.1$ Isnectl start -bash-4.1$ Isnrct] status 2. Enable the following parameters on the primary database for the dataguard configurat Implement and Vrify all parameters are enabled as above before proceeding further. SQL> alter system set log archive_des| scope=both; system altered. SQL> alter system set log_archive_dest_1='location=/home/oracle/arch/ORACLE VALID_FOR=(ALL_ system altered. SQL> alter system set log_archive_dest_2="SERVICE-ORACLE_S VALID_FOR=(ONLINE_LOGFILES, PRIM systen altered. SQL> alter system set FAL_SERVER-ORACLE_S scope-both; system altered. hip ileontuence hpdo 8444pagesvlewpage.action?pageld=213263195 382017-5:30 How/To- Setup DR database (ANY PLATFORM) using Oracletig Data Guard (Physical Standby) - Managed Hosting Internal ~ Confluence Product. SQL> alter system set FAL_CLIENT=ORACLE_P scope=both; systen altered. SQL> alter system set standby _file_management-auto scope=both; system altered. SQL> alter system set log archive DEST_state_2="ENABLE’ scope=both; 3 Modify the initdr_stby.ora file to update all the necessary parameters for dataguard configuration on Standby server. Verify all Mount points/ directory structure exists on DR server. Location: /homeforacle/DRYinitdr_stby.ora ‘Add the following parameters in standby init.ora file via vi editor which was originally created as copy of production init.ora file in step5 which is located under /homeforacie/DR. *Jog_archive_dest='thome/oracte/archORACLE/! *.FAL_SERVER=ORACLE_P *FAL_CLIENT=ORACLE_S * standby_file_management=auto 4, Copy the standby_controlfile.ct! in /homeloracle/DR to all respective locations listed in initdr_stby.ora Example cp standby controlfile.ctl /home/oracle/dbf/ORACLE/control@1..ct1 cp standby_controlfile.ct] /home/oracle/redo/rdo1 /ORACLE/controle2.ct1 cp standby controlfile.ctl /home/oracle/redo/rdo2/ORACLE/control@3.ct1. 5, Update Production and DR and production server tnsnames.ora file with the entries needed for Data guard configuration as below sample. Add this to both servers where ORACLE_P is the primary production server ORACLE_S is the standby server cat /hone/oracle/product/11.2.0/network/admin/tnsnames.ora ORACLE_P = (DESCRIPTION = (ADDRESS = hip ileontuence hpdo 8444pagesvlewpage.action?pageld=213263195 49,2017-5:30 How/To- Setup DR database (ANY PLATFORM) using Oracletig Data Guard (Physical Standby) - Managed Hosting Internal ~ Confluence Product. (PROTOCOL = TCP) (Host = apeprd-115298-44130-db@1) (Port = 1521) (CONNECT_DATA = (SID = ORACLE) ORACLE_S = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (Host = fgdr-115298-44130-dbee1 ) (Port = 1521) (CONNECT_DATA = (SID = ORACLE) 6. Create spfile from initdr_stby.ora SQL > Create spfile from pfile='/home/oracle/DR/initdr_stby.ora'; SQL > STARTUP NOMOUNT; SQL > ALTER DATABASE MOUNT STANDBY DATABASE; 7. Configure standby redo logs on Standby Database with same size as current production redo logs SQL > select * from v$log; SQL> alter database add standby logfile group 4 ‘/home/oracle/redo/rdo1/ORACLE/redo_g4_fil hip ileontuence hpdo 8444pagesvlewpage.action?pageld=213263195 582017-5-30 Hon To- Setup DR database (ANY PLATFORM) using Oracletig Data Guard (Physical Standby)- Managed Hosting ternal - Contuence Product. SQL> alter database add standby logfile group 5 ‘/home/oracle/redo/rdo1/ORACLE/redo_g5_fil SQL> alter database add standby logfile group 6 '/home/oracle/redo/rdo1/ORACLE/redo_g6_fil < » 8. Enable automatic managed recovery so that standby syncs with primary database automatically SQL > ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION; 9. Check Alert log on both Primary database and Standby database to make sure there are no errors. ‘Alett log Location: /home/oracieladmin/ORACLE/bdumplalert_ORACLE.log Verification of Setup Dataguard Verification and log Transfer 1. Switch couple of log files on primary database and verify if its shipped and applied on Standby database on DR server SQL > SELECT sequences, first_time, next_time FROM v$archived_log ORDER BY sequence#; SQL> ALTER SYSTEM SWITCH LOGFILE; Check the protection mode by running the below query SQL> SELECT protection_mode FROM v$database; SQL> Archive Log list; 2. Vel ationilatency check on Standby Database: SQL> Archive Log lists SQL> select applied, notappld, notappld-applied delta from (select max(sequence#) applied from v$archived_log where applied="YES') A, (select max(sequence#) notappld from v$archived_log) 8; 3. Check alert log on primary ‘There should be no errors on this server Alert log on primary should look normal with no errors hips cortuence bps io: S443pagesivewpage.action?pageld=213263196 682017-5:30 How/To- Setup DR database (ANY PLATFORM) using Oracletig Data Guard (Physical Standby) - Managed Hosting Internal ~ Confluence Product. Current log# 2 seq# 2613 mem# @: /home/oracle/redo/rdo1/ORACLE/redo_g?_file1.log Current logt 2 seqt 2613 memt 1: /home/oracle/redo/rdo2/ORACLE/redo_g2_file2.1og Thread 1 advanced to log sequence 2614 (LGWR switch) Current log# 3 seqit 2614 mem! @: /home/oracle/redo/rdo1/ORACLE/redo_g3_file1.1log Current log 3 seq# 2614 mem# 1: /home/oracle/redo/rdo2/ORACLE/redo_g3_file2.1og Fri Jan @9 10:23:25 2015 Archived Log entry 898 added for thread 1 sequence 2613 ID @x675beS1e dest 1: -bash-3.2$ 4. Check Standy Server alert log Qutput should look similar to this Thu Jan 08 18:27:50 2015 Primary database is in MAXIMUM PERFORMANCE mode RFS[4]: Assigned to RFS process 6208 RFS[4]: Selected log 9 for thread 1 sequence 2616 dbid 1734033694 branch 860956254 Thu Jan @8 18:27:5@ 2015 RFS[S]: Assigned to RFS process 6210 RFS[5]: Selected log 19 for thread 1 sequence 2615 dbid 1734033694 branch 860956254 Thu Jan @8 18:27:51 2015 Archived Log entry 10 added for thread 1 sequence 2615 1D @x67SbeSie dest 1 Thu Jan @8 18:27:51 2015 Media Recovery Log /home/oracle/arch/ORACLE/arch_1_2612_860956254. arc Media Recovery Log /home/oracle/arch/ORACLE/arch_1_2613_860956254. arc Media Recovery Log /home/oracle/arch/ORACLE/arch_1_2614 860956254. arc Nedia Recovery Log /home/oracle/arch/ORACLE/arch_1_2615_860956254. arc Media Recovery Waiting for thread 1 sequence 2616 (in transit) hip ieouence ppd jo 8443}paqesviewpage.acion?pageld=213263196 792017-5:30 How/To- Setup DR database (ANY PLATFORM) using Oracletig Data Guard (Physical Standby) - Managed Hosting Internal ~ Confluence Product. Con ure archive log cleanup on DR server via oracle cron Configure below cron job to run every 30 minutes to remove already applied archive logs on standby database server keeping only last 10. -bash4.1$ crontab 4 "130 ** ** Imntiasplutils/dbidribinideleteStbyArchLogs.bash 10 > /Imp/deleteStbyArchLogs.cron.Jog 2>81 Script usage:-bash-4.1$ /mnl/asp/ults/dbidr/bin/deleteStbyArchLogs.bash USAGE: /mnt/asplutils/db/dribin/deleteStbyArchLogs. bash This would keep last 10 applied archive logs and remove all others Configure monitoring Latency between production and DR standby database - Gap Sequence on DR server via oracle cro Configure below cronjob to run every 15 minutes to check the status of archive logs and if there is any latency between production and Standby database on DR server -bash-4.1$ crontab 4 */15** ** Imntiasp/utils/db/dr/bin/dr_monitor_gap.bash 10.5.12.77 ORACLE. blackboard.com 30 > Itmp/monitor_gap.cron.log 2>81 Usage: mntlasp/utils/db/bin/dr_monitor_gap.bash 10.3.3.60( Prod host name) ORACLE (DB Service name) 30 ( Threshold) IMP NOTE: By default the script take DB Service name as "ORACLE. blackboard.com’. If you have the DB Service name different than ORACLE, then please specity in cron job accordingly as below. ‘Syntax to check DB SERVICE NAME: -bash-4.1§ Isnretl services LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 28-AUG-2015 10:26:53 Copyright (c) 1991, 2011, Oracle. All rights reserved. Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521)) Services Summary... Service "FCPS6.blackboard.com” has 1 instance(s). Instance "FCPS6", status READY, has 1 handler(s) for this service... Handler(s): "DEDICATED" established: 788 refused:0 state:ready LOCAL SERVER ‘The command completed successfully In this Example, FCPS6.blackboard.com is the DB service name. Hence, setup the cronjob as below: Eg:7/15 * ** * mnt/asp/utils/db/dr/bin/dr_monitor_gap-bash 10.5.12.77 FCPS6.blackboard.com 30 > ‘Itmpimonitor_gap.cron-log 2>&4 Latency difference by default is "20" and can be modified by setting in cron as above which is set to "30" archive logs. Troubleshooting If you receive this error FAL[server, ARCO]: Error 12541 creating remote archivelog file ‘ORACLE_S' , this means the archivelogs cannot be transferred to standby database. SOLUTION: Check to make sure that the database and listener are up; start both database and listener for standby database. Then switch log files are few time on primary production database. Check alert logs for errors hip ieouence hippo 8443fpaqesviewpage action?pageld=21263196 892017-5:30 How/To- SetupDR database (ANY PLATFORM) using Oracletlg Data Guar (Physical Standby) - Managed Hosting Internal ~ Confluence Product. If the standby server is mistakenly shutdown. Restart the server in mount and restart applying logs: Startup nomount alter database mount standby databas: dr hip ileontuence hpdio 8444paqesvlewpage.action?pageld=213263195 2