Disaster Recovery System

A knowledge transfer session
December 2008 (Updated March 2009)

Nikos Plevris Principal Service Delivery Manager

Agenda
‡ ‡ ‡ ‡ ‡ ‡ Purpose & Scope Standby Database Environment Compatibility and Operational Requirements Standby modes Setting up the standby database for managed recovery Troubleshooting
‡ Standby Db Configuration ‡ Gap Sequence before initiating Managed Recovery
<Insert Picture Here>

‡ ‡ ‡ ‡ ‡

Standby Db Administration & Maintenance The SDR Activation/Operation procedure Graceful Switchover and Switchback Summary Q&A

Purpose & Scope
‡ Disaster protection for Information System
‡ Ensure Business Continuity ‡ Faster recovery and lower MTTR ‡ Potentially no data loss (if online redo logs are applied)

‡ Protection against data corruption ‡ Supplemental reporting ‡ Based on:
‡ Standby database operation ‡ Failover and Switchback functions

Standby Database Environment
‡ Managed standby environment
‡ Automatic shipment and application of archived logs from primary

‡ Procedural graceful switchover and switchback functions in effect ‡ Considerations on Administration Costs :
‡ Network connection maintenance ‡ Additional system resources and storage space ‡ Ensure mirroring of structural operations performed on primary ‡ Extra loss of time due to graceful procedure

Standby Database Environment ‡ Managed standby environment ‡ Automatic shipment and application of archived logs from primary ‡ Procedural graceful switchover and switchback functions in effect ‡ Considerations on Administration Costs : ‡ Network connection maintenance ‡ Additional system resources and storage space ‡ Ensure mirroring of structural operations performed on primary ‡ Extra loss of time due to graceful procedure .

Compatibility and Operational Requirements ‡ The primary db must run in ARCHIVELOG mode ‡ The redo logs that you apply to the standby db must be either archived or noncurrent online redo logs ‡ Same version and release of the operating system and database on the primary and standby hosts ‡ The standby host can use a different directory structure ‡ Primary and Standby db cannot share the same control file ‡ An activated standby db becomes a normal primary database .

Standby modes Managed Recovery mode Read-only mode Manual Recovery mode .

.f¶.Setting up the Standby Database ± Managed Recovery1 ‡ Backup primary database ‡ Connect to the primary database and create the standby control file SQL>ALTER DATABASE CREATE STANDBY CONTROLFILE AS ¶/oracle/dbs/stbycf.

Setting up the Standby Database ± Managed Recovery2 ‡ Copy the backup datafiles and control file from the primary site to the standby site ‡ On the primary site Create a service name for the standby database ‡ tnsnames.ora standby_service_name = (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp) (PORT=port_number) (HOST=host_name)) (CONNECT_DATA=(SID=standby_sid))) .

ora STANDBY_LISTENER = (ADDRESS_LIST= (ADDRESS=(PROTOCOL=tcp) (PORT=port_number) (HOST=host_name)) ) SID_LIST_STANDBY_LISTENER = (SID_LIST= (SID_DESC= (SID_NAME=standby_sid_name) (ORACLE_HOME=/oracle_home)) ) .Setting up the Standby Database ± Managed Recovery3 ‡ Configure the listener on the standby site so that it can receive the archived redo logs from the primary site ‡ $TNS_ADMIN/network/admin/listener.

add new archiving destinations ‡ LOG_ARCHIVE_DEST_n ‡ OPTIONAL or MANDATORY ‡ LOG_ARCHIVE_MIN_SUCCEED_DEST=m # first local archiving destination # LOG_ARCHIVE_DEST_1 = LOCATION=/oracle/arc/ ‡ LOG_ARCHIVE_DEST_STATE_n MANDATORY¶ # # second local archiving destination ‡ REOPEN=s LOG_ARCHIVE_DEST_STATE_2 = ENABLE LOG_ARCHIVE_DEST_2 = ¶LOCATION=/oracle/arc2/¶ # # third standby archiving destination LOG_ARCHIVE_DEST_3 = ¶SERVICE= standby_service_name OPTIONAL REPOEN=2¶ .Setting up the Standby Database ± Managed Recovery4 ‡ Set the initialization parameters for the primary database .

Setting up the Standby Database ± Managed Recovery5 ‡ Set the initialization parameters for the standby database ‡ most initialization parameters at the primary and standby databases should be identical ‡ Parameters in standby init.ora ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ = Primary db¶s DB_NAME DB_NAME CONTROL_FILES <> Primary db¶s DB_NAME Distinguishes standby¶s datafiles DB_FILE_NAME_CONVERT When the standby and primary databases share a host LOCK_NAME_SPACE Location of archives during manual recovery LOG_ARCHIVE_DEST Audit trail of archived log progression LOG_ARCHIVE_TRACE LOG_FILE_NAME_CONVERT Distinguishes standby¶s logfiles Location of archives during managed recovery STANDBY_ARCHIVE_DEST .

Managed Recovery . RECOVER MANAGED STANDBY DATABASE. ALTER DATABASE MOUNT STANDBY DATABASE.Managed Recovery Mode Primary Db Online redo Archived redo Transactions Remote archiving Via Oracle Net PRIMARY_HOST SDR Standby Db Archived redo STARTUP NOMOUNT.

ALTER DATABASE MOUNT STANDBY DATABASE.Read-Only mode Primary Db Online redo Archived redo Transactions Remote archiving Via Oracle Net PRIMARY_HOST SDR Standby Db Archived redo STARTUP NOMOUNT. Managed Recovery Queries . ALTER DATABASE OPEN READ ONLY.

becomes primary database .Failover to standby database Primary Db ALTER DATABASE ACTIVATE STANDBY DATABASE. PRIMARY_HOST Charybdis Transactions Online redo Archived redo Standby database activated.

LOG_ARCHIVE_DEST_STATE_n ‡At the standby site Check: listener. STANDBY_ARCHIVE_DEST SQL> SELECT dest_id "ID". 3> destination "Archive_dest". 4> error "Error" 5> FROM v$archive_dest. 2> status "DB_status".Troubleshooting the Standby Database Configuration1 ‡ The Standby Site Does Not Receive Logs Archived by the Primary Database ‡ Immediate Action: Query V$ARCHIVE_DEST ‡At the primary site Check: tnsnames. LOG_ARCHIVE_DEST_n. .ora.ora.

..Troubleshooting the Standby Database Configuration2 ‡ Cannot Mount the Standby Database ‡ The standby instance is not started in NOMOUNT mode ‡ The standby control file was not created with the ALTER DATABASE CREATE STANDBY CONTROLFILE . Statement ‡ Unacceptable control file backups: ‡ RMAN-created backup ‡ An operating system-created backup ‡ A backup created using an ALTER DATABASE statement without the STANDBY option .

ie MAXDATAFILES ‡ ALTER DATABASE CLEAR UNARCHIVED LOGFILES ‡ a RESETLOGS operation on the primary database invalidates the standby database .Resolving a Gap Sequence Before Initiating Managed Recovery1 ‡ A gap sequence is a range of archived redo logs needed by the standby database ‡ When standby db is created from an old backup ‡ Standby db is generated from a hot backup ‡ Shutdown of the Standby Database When the Primary Database Is Open ‡ Primary db¶s Controlfile change.

Resolving a Gap Sequence Before Initiating Managed Recovery2 ‡ To avoid creating gap sequences: ‡ Start the standby databases and listeners before starting the primary database ‡ Shut down the primary database before shutting down the standby database ‡ dynamically change MANDATORY to OPTIONAL before shutting down the standby database ‡ Network Failure Preventing the Transfer of Logs to the Standby Site ‡ To prevent set LOG_ARCHIVE_DEST_2 = ¶SERVICE=standby1 MANDATORY¶ ‡ WARNING: Primary might stall as it cannot switch into an unarchived online redo log .

( SELECT thread#.thread# = high. v$datafile WHERE checkpoint_change# <= next_change# AND checkpoint_change# >= first_change# ) GROUP BY thread# ) low WHERE low.thread#.next_change# > gap1 ) GROUP BY thread# ) high.thread#. MIN(sequence#) "LowGap#" FROM ( SELECT thread#. ( SELECT thread#.thread# AND a. "LowGap#".thread# = b. sequence# FROM v$log_history. a.sequence# FROM ( SELECT * FROM v$archived_log ) a.thread#. MIN(sequence#)-1 "HighGap#" FROM ( SELECT a. "HighGap#" FROM ( SELECT thread#. MAX(next_change#)gap1 FROM v$log_history GROUP BY thread# ) b WHERE a. .Resolving a Gap Sequence Before Initiating Managed Recovery3 ‡ To determine if Gap Sequence Exists (Standby site) SELECT high.

Resolving a Gap Sequence Before Initiating Managed Recovery4 ‡ Transmit the Logs in the Gap Sequence to the Standby Site ‡At the primary site determine filenames ‡At the standby site review the settings for STANDBY_ARCHIVE_DEST and LOG_ARCHIVE_FORMAT to determine archived log filenames ‡Copy archives to standby site SQL> CONNECT sys/sys_pwd@primary SQL> SELECT name 2> FROM v$archived_log 3> WHERE sequence# in (460. 202). .

ora SQL> ALTER DATABASE MOUNT STANDBY DATABASE. SQL> RECOVER AUTOMATIC STANDBY DATABASE. .Resolving a Gap Sequence Before Initiating Managed Recovery5 ‡ Apply the Logs in the Gap Sequence to the Standby Database SQL> CONNECT sys/sys_pwd@standby1 AS SYSDBA SQL> STARTUP NOMOUNT pfile=/oracle/admin/pfile/initSTBY.

first_change# AS "FIRSTSCN". SQL>SELECT thread#.Resolving a Gap Sequence Before Initiating Managed Recovery6 ‡ Determine which archived logs have been received by the Standby Site ‡ Tracing ALTER SYSTEM SET LOG_ARCHIVE_TRACE = 32 SQL> SELECT thread#.completion_time AS "TIME" FROM v$archived_log.archived. max(sequence#) AS "LAST_APPLIED_LOG" 2> FROM v$log_history 3> GROUP BY thread#. sequence# AS "SEQ#". next_change# AS "NEXTSCN". . deleted. name.

Administration & Maintenance1 .

.f¶ AS ¶/private1/stby/t_db2.Adding Tablespaces or Datafiles to the Primary Database ‡ CREATE TABLESPACE redo adds the new filename to the standby control file ‡ Manually create datafile to standby SQL> RECOVER MANAGED STANDBY DATABASE CANCEL. SQL> RECOVER MANAGED STANDBY DATABASE.f¶. SQL>ALTER DATABASE CREATE DATAFILE ¶/private1/stby/t_db2.

SQL> ALTER DATABASE MOUNT STANDBY DATABASE. Transfer the standby control file and archived log files SQL> RECOVER MANAGED STANDBY DATABASE CANCEL.Refreshing the Standby Database Control File SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS ¶filename¶. SQL> RECOVER MANAGED STANDBY DATABASE. SQL> STARTUP NOMOUNT. SQL> SHUTDOWN IMMEDIATE. Primary db Standby db .

PRIMARY_HOST SDR Secondary Disaster Recovery system (SDR) .

The SDR Activation and Operation internal procedure ‡ Graceful shutdown of the production information system ‡ Change of the configuration files on user site ‡ Interfaces to external systems transition ‡ Rolling forward to SDR system ‡ Switch back to production system (PRIMARY_HOST) .

Graceful Switchover and Switchback ‡ Avoid opening the targeted db with resetlogs ‡ Switchover ‡ From Primary (Source) -> Standby db (Target) ‡ Switchback ‡ From Standby (Source) -> Primary db (Target) PRIMARY_HOST Primary db Standby db SDR .

Graceful Switchover and Switchback Prerequisites ‡ ‡ ‡ ‡ ‡ ‡ PRIMARY_HOST SDR Primary db is shutdown normal/immediate No loss of any unapplied archived logs All archives are applied to standby db Standby db is shutdown normal/immediate Source db¶s online redo logs are available Target db¶s files. controlfile intact ± Same resetlogs version as Source db ‡ Loss of unrecoverable transactions (direct loads) is acceptable .

PRIMARY_HOST SDR ‡ Database undergoes a resetlogs version change: ‡ Above commands are required after: ‡ RECOVER DATABASE UNTIL [Cancel/Time/Change]. . ‡ RECOVER MANAGED STANDBY DATABASE. ‡ RECOVER STANDBY DATABASE. ‡ ALTER DATABASE ACTIVATE STANDBY DATABASE.What prevents a Graceful Switchover and Switchback ‡ ALTER DATABASE OPEN RESETLOGS.

Production system Interfaces to external systems transition External system ABS AIANET ATC PALLAS BHS COSMOTE CUTE DOAS DYSI FB/LB FIDS / Conrac HTML Information Desk Staff Pages IDS LCD boards/Matrix MAIN MCL MIS NMS RMS SAFCO file interface OIC file interface VB Pages applications OTENET file interface SITA PUBLIC S.A file interface Collaborative Decision Making (CDM) interface Aeronautical Billing system AIA Intranet Pallas System Baggage Handling System COSMOTE SITA CUTE system Differential Optical Absorption Spectroscopy DYSI System FB/LB Displays Flight Information Displays HTML / information desk ATC Tower IDS System LCD Boards/Matrix LCD UFIS AS GmbH Help Desk Germany Master Clock Management Information System Noise Monitoring System Resource Management System SAFCO Fueling Co OIC Fueling Co OTENET internet site SITA Messaging System Public S.A ftp server Ourania (AIA server)     Interface Disaster recovery site failover type Incoming Auto TNS Manual Outgoing Both Outgoing Outgoing Incoming Outgoing Outgoing Incoming Outgoing Outgoing Outgoing Outgoing Both Auto Change of the /etc/hosts file Auto Auto Change of the /etc/hosts file Auto Auto Change of the /etc/hosts file Auto Auto Manual Manual Outgoing Outgoing Outgoing Outgoing Outgoing Outgoing Incoming Outgoing Outgoing Outgoing Auto Auto Auto Auto Auto (config file should be changed) (config file should be changed) Manual Activate (tick) new service Auto Auto Information Desk Pages file interface   Auto Auto Changes to VPN UFIS AS) Outgoing Auto Incoming Auto (e mail to be send to TNS .

The Switchover1 1 shutdown 4 Online redo Logs shipped To Standby 7 Mount NEW STANDBY PRIMARY_HOST Charybdis 2 Standby apply archives 3 Standby shutdown 5 ‡Create controlfile ‡Startup NEW Primary 6 ‡Create standby controlfile ‡Ship to NEW STANDBY Timeline .

SQL>SHUTDOWN IMMEDIATE. ‡Online redo logs are shipped to Standby site ‡CREATE CONTROLFILE script sent to Standby site . SQL>ALTER DATABASE BACKUP CONTROLFILE TO TRACE NORESETLOGS.Graceful Switchover2 steps Primary site SQL>ALTER SYSTEM ARCHIVELOG CURRENT. SQL> SELECT sequence# from v$loghist.

SQL>ALTER DATABASE CREATE STANDBY CONTROLFILE AS µstandby. SQL>ALTER DATABASE OPEN. SQL>ALTER SYSTEM ARCHIVE LOG ALL.ora Run CREATE CONTROLFILE script from primary SQL>RECOVER DATABASE. SQL>STARTUP NOMOUNT PFILE=init.` SQL>RECOVER MANAGED STANDBY CANCEL.Graceful Switchover3 steps Standby site SQL>SELECT MAX(fhrba_seq) from x$kcvfh. CREATE CONTROLFILE script is sent to Standby site . SQL>SHUTDOWN IMMEDIATE.ctl¶.

Date 06/02/2008 19:36:28 DB Name "LNX817" Database flags = 0x00005447 Controlfile Creation Timestamp 06/02/2008 19:36:29 Incmplt recovery scn: 0x0000. File size=556=0x22c File Number=0. Max data members = 1 Arch list: Head=4. Max data members = 1 Arch list: Head=1. #Online files = 10 Database checkpoint: Thread=1 scn: 0x0000. File size=556=0x22c File Number=0. Head=1. File Type=4 BACKUP CONTROL ************************************************************************ DATABASE ENTRY ************************************************************************ (blkno = 0x1.nymfes.005cefe8 Resetlogs Timestamp 06/02/2008 19:38:50 Prior resetlogs scn: 0x0000. image: oracle@icho.1) 2008-12-16 10:38:22.483 DUMP OF CONTROL FILES. File Type=1 CONTROL ******************************************************************** DATABASE ENTRY ******************************************************************** (blkno = 0x1.717 SQL>ALTER SESSION SET EVENTS DUMP OF CONTROL FILES.0071e3b5 Threads: #Enabled=1. Tail=1. last-recid= 0) DF Version: creation=0x8100000 compatible=0x8000000.0) enabled threads: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 *** 2008-12-16 10:38:22.gr (TNS V1-V3) *** 2008-12-16 10:46:02. Db Name='LNX817' Control Seq=588=0x24c. size = 192.005cefe8 Resetlogs Timestamp 06/02/2008 19:38:50 Prior resetlogs scn: 0x0000.0071e3bd Controlfile Checkpointed at scn: 0x0000.0071e3f2 12/15/2008 17:07:36 thread:1 rba:(0x3b. Seq # 588 = 0x24c FILE HEADER: µimmediate trace name controlf Software vsn=135266304=0x8100000.0071e3f0 Controlfile Checkpointed at scn: 0x0000. #Open=1. last-recid= 0) DF Version: creation=0x8100000 compatible=0x8000000. Blksiz=8192. Vsn=134217728=0x8000000 Db Id=51525935=0x312392f. #Online files = 10 Database checkpoint: Thread=1 scn: 0x0000. Seq # 608 = 0x260 FILE HEADER: Software vsn=135266304=0x8100000.484 *** SESSION ID:(8.nymfes. in-use = 1.1) 2008-12-16 10:46:02. Head=1.0071e40d 12/16/2008 10:45:41 thread:0 rba:(0x0. Tail=4.Graceful Switchover4 Technical aspects Machine: i686 Instance name: STDBY Redo thread mounted by this instance: 1 Oracle process number: 9 Unix process pid: 1500. Date 06/02/2008 19:36:28 DB Name "LNX817" Database flags = 0x00004001 Controlfile Creation Timestamp 06/02/2008 19:36:29 Incmplt recovery scn: 0x0000.00000001 Prior resetlogs Timestamp 07/19/2005 18:18:23 Redo Version: creation=0x8100000 compatable=0x8100000 #Data files = 10. Force scn: 0x0000.0071e40d Threads: #Enabled=1.gr (TNS V1-V3) Standby Machine: i686 Primary Instance name: LNX817 Redo thread mounted by this instance: 1 Oracle process number: 9 Unix process pid: 1960.00000000 Resetlogs scn: 0x0000. Blksiz=8192. Compatibility Vsn=134217728=0x8000000 Db Id=51525935=0x312392f.0. #Open=1. image: oracle@icho. in-use = 1. Tail=1 enabled threads: 01000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 Max log members = 2.717 *** SESSION ID:(8. Force scn: 0x0000.0071e3f2 Resetlogs scn: 0x0000. max = 1. Db Name='LNX817' Control Seq=608=0x260. Compatibility level 10¶.00000001 Prior resetlogs Timestamp 07/19/2005 18:18:23 Redo Version: creation=0x8100000 compatable=0x8100000 #Data files = 10.0) enabled threads: 01000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 . size = 192.2. max = 1. Tail=1 enabled threads: 01000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 Max log members = 2.

ora parameters have been set .ora files exist in both sites ‡ All expected init.Scenario: Standby db has been activated ‡ ALTER DATABASE ACTIVATE STANDBY DATABASE. ‡ SHUTDOWN IMMEDIATE ‡ STARTUP ‡ Prerequisites: ‡ $ORACLE_HOME/dbs/Init.

‡ . ‡ ALTER TABLESPACE <> END BACKUP. . ‡ ALTER SYSTEM ARCHIVE CURRENT LOG.. ‡ host cp <source_datafile> <dest_datafile> ‡ . ‡ Backup current redo: ‡ ALTER SYSTEM ARCHIVE LOG ALL.Scenario: Fallback to Primary db ± Steps1 ‡ Hot backup of current primary db (ex-standby db): ‡ ALTER TABLESPACE <> BEGIN BACKUP....

‡ STARTUP NOMOUNT ‡ CREATE CONTROLFILE REUSE DATABASE "<>" RESETLOGS ARCHIVELOG ‡« ‡« .Scenario: Fallback to Primary db ± Steps2 ‡ Copy archive logs to primary db ‡ Create controlfile creation script: ‡ ALTER DATABASE BACKUP CONTROLFILE TO TRACE... ‡ Shutdown immediate current primary db (ex-standby db) ‡ Startup primary db and recreate controlfile.

Scenario: Fallback to Primary db ± Steps3 ‡ Apply the archivelogs and CANCEL upon the nonexistent archivelog: ‡ RECOVER DATABASE USING BACKUP CONTROLFILE UNTIL CANCEL ‡ ALTER DATABASE OPEN RESETLOGS .

‡ Move standby controlfile to standby site (where init.ora control_files expects) ..Scenario: Recreate Standby db ± Steps1 ‡ Hot backup of primary db: ‡ ALTER TABLESPACE <> BEGIN BACKUP. ‡ ALTER TABLESPACE <> END BACKUP. ‡ host cp <source_datafile> <dest_datafile> ‡ . ‡ ... ‡ ALTER DATABASE CREATE STANDBY CONTROLFILE AS '<>'..

. ‡ Set standby db to managed mode: ‡ RECOVER MANAGED STANDBY DATABASE. ‡ ALTER SYSTEM ARCHIVE CURRENT LOG. ‡ Copy archive logs to standby db ‡ Check if any sequence gap exists ‡ Apply the archivelogs and CANCEL upon the nonexistent archivelog: ‡ RECOVER STANDBY DATABASE.Scenario: Recreate Standby db ± Steps2 ‡ ALTER SYSTEM ARCHIVE LOG ALL.

ora Control_file Control_file .ora Init.Primary Db StandBy Db Init.

someone else with less ability might have your job. James A.Summary Be thankful for problems. If they were less difficult. . Lovell.

Sign up to vote on this title
UsefulNot useful

Master Your Semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master Your Semester with a Special Offer from Scribd & The New York Times

Cancel anytime.