“This presentation is for informational purposes only and may not be incorporated into a contract or agreement.

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decision. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Logical Standby Unleashed
Session 938

Introduction
• Who’s Who? • Requirements of a Typical SQL Apply User • Thomson Legal & Regulatory and SQL Apply
• Who are they? • Database Technology Challenges • Using SQL Apply

• How Data Guard met Thomson’s Challenges • Looking forward – Now and in the Future

Who’s Who?
• Larry M. Carpenter
• Oracle • Principal Product Manager

• Dan Dressel
• Thomson Legal & Regulatory • Database Architect

• Joydip Kundu
• Oracle • Senior Manager, Software Development
“This presentation is for informational purposes only and may not be incorporated into a contract or agreement.”

Requirements of a Typical SQL Apply User
Larry M. Carpenter

Requirements of a Typical SQL Apply Customer
• Recovery Point Objective (RPO) is Zero
• No Data Loss

• Recovery Time Objective (RTO) is Zero
• Failure of Server, Disk, Instance or Site

• High Availability Objective (HAO)
• Up-to-date standby database accessible for reads at all times • Zero Downtime Standby creation • Minimally affected by Oracle Upgrades

Thomson Legal & Regulatory (TLR)
Dan Dressel

Who are We?
• Customers
• More than one million customers worldwide
• All of the top 100 global law firms • All Big 4 accounting firms

• More than 120,000 daily online users • More than 53 million web site page views per month

Our Markets
• Primary Markets Served
• • • • • • • Legal Tax and Accounting Corporate Consulting Specialized libraries Government Law Schools

Our Brands

TLR is about Information Solutions
• Headquartered in Eagan, MN.
• Two world class data centers
• 200,000 square feet. • More than 700 TB of enterprise class disk. • Eight diesel generators with 16 MW of power backup.

• Technology is critical to our business.
• We make heavy use of relational database technology. • We’ve had them all – Oracle, DB2, SQL Server, Redbrick, Informix, MySQL, Sybase, ADABAS.

Database Technology Challenges
• Drive the level of availability up • Make solutions incrementally scalable • Ensure scaling strategy is transparent to applications and developers • Don’t sacrifice performance • Improve workload management capabilities • Make the environment easy to setup and administer • Make it cheaper

Our Solution
Online Users

Site A
Primary Location

Site B
Standby Location

LGWR SYNC Primary Database

Data Guard

Standby Database

Read Write

Read

System & Network Configuration
• 4-node RAC Primary
• IBM E325 Servers
• Each with 2 CPUs and 12GB RAM

• 4-node RAC Standby
• IBM E325 Servers
• Each with 2 CPUs and 12GB RAM

• SUSE9 64bit (United Linux 1.0) • NetApp Storage • Gigabit network

Data Guard Configuration
• Oracle Database 10.1.0.3 SQL Apply • Maximum Availability
• (LGWR SYNC AFFIRM)

• • • •

Real Time Apply 500GB database growing to 3TB 2.8MB/sec peak redo generation LOB intensive workload.
• Average LOB size 4-8k, but 5% of workload is comprised of Large Objects greater than 1MB

Lessons Learned
• Make sure software/hardware choices are all certified to work with one-another.
• Get vendor contacts for break/fix early.

• Leverage clusters and Data Guard to their fullest.
• Don’t build new configurations for each application.
• Establish simple, cookie-cutter models. • Make reusability a priority. Stay away from one-offs. • Compensate for performance issues with additional hardware rather than complex, highly-specialized tuning when possible to preserve standard models.

In Conclusion
• Hire good Network Engineers. • Hire good DBAs. • Hire good System Administrators.

Meeting Thomson’s Challenges with Data Guard
Joydip Kundu

Thomson’s Requirements
• Recovery Point Objective (RPO)
• No Data Loss

• Recovery Time Objective (RTO)
• Failure of Server, Disk, Instance or Site

• High Availability Objective (HAO)
• Up-to-date standby database accessible for reads at all times • Zero Downtime Standby creation • Minimally affected by Oracle Upgrades

Meeting the RPO
Transactions LGWR LNS
LGWR SYNC

RFS

Primary Database

Online Redo Logs ARCH

Standby Redo Logs

ARCH Archived Redo Logs

Archived Redo Logs

Configuring for the RPO
• Use Log Writer Synchronous Redo Transport
• LOG_ARCHIVE_DEST_2=‘SERVICE=wlp01a LGWR SYNC NET_TIMEOUT=30 REOPEN=15 VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=wlp01a’

• Add Standby Redo Logs • Set the Protection mode
• SQL> ALTER SYSTEM SET STANDBY 2 TO MAXIMIZE AVAILABILITY;

Meeting the Server Level RTO
• RAC
Site A
Primary Location

Primary Database

Read Write

Meeting the Site RTO
LGWR SYNC

RFS

An up-to-date Logical Standby Database

Standby Redo Logs

LSP

ARCH

Real Time Apply

Site B
Standby Location

Archived Redo Logs

Configuring for the RTO
• Turn on Real time apply
• SQL> ALTER DATABASE START LOGICAL 2 STANDBY APPLY IMMEDIATE;

• That’s it!

Behind the Scenes
T1

Logical Change Records not grouped into transactions Redo Records Reader
Standby Redo Log Threads

T2

T3

Preparers

LCR LCR :
Shared Pool

Builder

T4

Log Mining Apply Processing
Transaction groups

Applier Transactions to be applied

Coordinator Transactions sorted in dependency order

Analyzer

Standby Database

LOB Performance Increase
• Large Object Datatypes (LOB) supported since first release of SQL Apply in Oracle9i Release 2. • Ordering of LOB information in the Redo stream changed for better Performance in Oracle Database 10g Release 1. • Requires compatibility 10.1.0.0.0

LOB Handling Oracle9i
• LOB Blocks appear in Redo before necessary information to identify the table where they belong. • All LOB modifications must be staged in memory until necessary information identifying the base table row being modified arrives in the Redo. • With larger out-of-line LOB data the amount of memory in the LCR can be high.

LOB’s in Oracle Database10g
• Table identifying information now logged into the redo stream before the LOB blocks. • Redo Mining can now send identifying information to the Apply components earlier. • Further LOB blocks for the same transaction are then passed directly to the Apply processes without staging in the LCR cache.

More Performance Enhancements
• SQL Apply in Oracle Database10g Release 1 has been enhanced to provide the following performance improvements
• Redo records associated with tables that are not maintained by SQL Apply are filtered out early, and thus no longer occupy space in LCR cache. • The internal cache where tables maintained by SQL Apply has been improved to occupy less space. • SQL Apply will batch inserts together whenever possible to improve performance

Meeting the RTO Oracle Database 10g Release 2
• New Metrics to Monitor your RPO and RTO
• V$DATAGUARD_STATS

• Estimated Failover Time (seconds) • Apply Lag (seconds) • Transport Lag (seconds)
• V$STANDBY_APPLY_SNAPSHOT

• Redo Apply Rate (KB/second)

• Additional metric in Grid Control
• Redo Generation Rate (KB/second)

Meeting the RTO Oracle Database 10g Release 2
• Fast-Start Failover
• Data Guard automatically fails over to the target standby without requiring any manual steps. • Requires
• A Broker configuration with a new Broker capability • The Observer, which monitors the environment and triggers a failover if necessary • Maximum Availability protection mode • Flashback Database

• Broker automatically reinstates the old primary database as a new standby database. • Specialized events generated to handle Client Failover.

Fast-Start Failover
Primary Site
Observer

Standby Site

1. 2.

Data Guard in steady state – transmitting redo Observer monitoring state of the configuration

Fast-Start Failover
Primary Site
Observer

Standby Site

3.

Disaster strikes the primary – connections lost

Fast-Start Failover
Primary Site
Observer

Standby Site

4. 5. 6.

Observer times out Observer validates connection with target standby Observer begins Fast-Start Failover

Fast-Start Failover
Primary Site
Observer

7.

Target standby automatically becomes new primary

Fast-Start Failover
Standby Site
Observer

Primary Site

8. After old primary is repaired, Observer re-establishes connection 9. Observer automatically reinstates old primary to be a new standby 10. Redo transmission starts from new primary to new standby

Thomson – Test Results for Fast Start Failover
• Data Guard 10g Release 2 test results:
• Fast-start failover can execute in under 10 seconds • Reinstatement of old primary in under 5 minutes
Failover Characteristics
Automatic delay set on failover (user configurable) Time for database failover Time for DBA to respond Total elapsed time to failover Before Fast-Start Failover Not possible 10 minutes Up to 30 minutes Up to 40 minutes After Fast-Start Failover 30 seconds Less than 10 seconds Zero Less than 40 seconds

Thomson – User Experiences with Fast Start Failover
“Fast-start failover testing has shown great potential. Our testing has shown failover to a standby database completes automatically in less than 40 seconds of elapsed time for most failures. The original primary database can be reinstated as a new standby in less than 5 minutes once the initial failure has been corrected. Fast-start failover is changing the way we think about database high availability.” Thomson Legal & Regulatory

Meeting the HAO Up-to-date Readable Standby
• Logical Standby Databases are always open.
• Read access to Production system data
• No change of database role required • Real-Time Apply keeps the data current

• Read Write Access to additional data
• User added tables • Global Temporary Tables

Meeting the HAO Standby Creation
• Improving Logical Standby Database Creation
• Oracle9i
• Cold backup (downtime) or • Quiesce users (R/W downtime)

• Oracle Database 10g Release 1
• Zero Down Time Creation! • Meeting the Production HA Requirement

• Oracle Database 10g Release 2
• Still Zero Down Time Creation but even easier!

Zero Down Time Instantiation Oracle Database 10g Release 1
1
On-Line Backup

3

5
Restore

Recovery

2
Primary Database Create and Copy Logical Standby Control File

Physical Standby Database

4

Transport Service

Zero Down Time Instantiation Oracle Database 10g Release 1
6
Physical Standby Database Activation

7
Change DBNAME and DBID Logical Standby Database!

8
Start SQL Apply Services

Simple Instantiation Oracle Database 10g Release 2
1
Create a Physical Standby

2

3
Physical Standby Database!

Primary Database

Perform Dictionary Build on Primary database

Recover to Logical Standby

Logical Standby Database! Start SQL Apply Services

4

Meeting the HAO Upgrading Oracle
• Starting with Oracle Database 10g Release 1 SQL Apply databases can be used to perform Rolling Upgrades to Oracle patch sets or major releases. • Read access during the process is not hindered at all. • Read Write access suffers only minutes of interruption.

SQL Apply – Rolling Upgrades
Upgrade Redo
A B

Clients

Logs Queue

A

B

Patch Set Upgrades Major Release Upgrades

Version X 1

Version X 2

X

X+1

Initial SQL Apply Config

Upgrade node B to X+1

Redo Upgrade
A B A

Redo
B

Cluster Software & Hardware Upgrades

X+1

X+1 3

X

X+1

4 Switchover to B, upgrade A

Run in mixed mode to test

Airbus – User Experiences with Rolling Database Upgrades
“Data Guard's rolling upgrade capability is amazing. Airbus tested upgrading from Oracle Database 10g Release 1 to Oracle Database 10g Release 2. The elapsed time for the database upgrade was about 3 hours. But applications were available for all but less than 2 minutes while a Data Guard switchover was executed, quickly transitioning production from Release 1 over to Release 2. Oracle Data Guard has made great strides in helping us achieve our 100% availability goal.” Werner Kawollek Application Management Operations Airbus Deutschland GmbH

In Conclusion
Larry M. Carpenter

Requirements Met!
• Recovery Point Objective (RPO) is Zero
• No Data Loss occurs at Failover

• Recovery Time Objective (RTO) is Zero
• Handles Failure of Server, Disk, Instance or Site

• High Availability Objective (HAO)
• Standby database is accessible for reads at all times and is up-to-date. • Standby creation requires no downtime • Oracle Upgrades have minimal effect

MAA Best Practices Home Page
http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm • HA Best Practices for Oracle Database
• • • • • • • • • • Oracle Database High Availability Overview 10g Release 2 - Documentation Oracle Database High Availability Architecture and Best Practices 10g Release 1 Oracle Database 10g Best Practices: Data Guard Redo Apply and Media Recovery Oracle Database 10g Best Practices: Data Guard SQL Apply Oracle Database 10g Best Practices: Data Guard Role Transitions and Streams Using Recovery Manager with Oracle Data Guard in Oracle Database 10g Oracle Database 10g Best Practices: Migration to Automatic Storage Management Best Practices for Creating a Low-Cost Storage Grid for Oracle Databases Oracle9i Data Guard: Primary Site and Network Configuration Best Practices Oracle9i Fast-Start Checkpointing Best Practices

High Availability Demos/Sessions From Oracle Development
Demogrounds - Monday, Sep 19 – Thursday, Sep 22
Oracle Data Guard ILM and Storage Oracle Secure Backup RMAN, Flashback, and Online Redefinition

Sessions - Monday, Sep 19
1:30-2:30 pm, Room 303 - Optimizing Linux I/O 3:00-4:00 pm, Room 104 - The Future of Database Information Technology 4:30-5:30 pm, Room 103 - What They Didn't Print in the DOC - HA Best Practices by Gurus from Oracle's Maximum Availability Architecture Team

Sessions - Tuesday, Sep 20
3:00-4:00 pm, Room 104 - Logical Standby Unleashed 4:30-5:30 pm, Room 104 - Best Practices for Oracle Database 10g Backup and Recovery

High Availability Sessions From Oracle Development
Sessions - Wednesday, Sep 21
11:00 am-12:00 pm, Room 104 - Improve Your Tape Backup Results with Oracle Secure Backup 3:00-4:00 pm, Room 304 - Implementing Information Lifecycle Management (ILM) using the Oracle Database

Sessions - Thursday, Sep 22
1:00-2:00 pm, Room 104 - Minimizing Application Development Time Using Flashback: A Customer Case Study 2:30-3:30 pm, Room 104 - Best Practices To Achieve Business Continuity Using Oracle Applications and Oracle Database Technology 4:00-5:00 pm, Room 104 - Best Practices for Automatic Failover Using Oracle Data Guard 10g Release 2

QUESTIONS ANSWERS