Maximum Availability Architecture: Oracle Best Practices For High Availability

Deploying Oracle Maximum Availability Architecture with
Exadata Database Machine

Oracle Maximum Availability Architecture White Paper
August 2013
Maximum
Availability
Architecture
Oracle Best Practices For High Availability
Oracle White PaperDeploying Oracle Maximum Availability Architecture with Sun Oracle Database Machine and Oracle Exadata
Executive Overview ........................................................................... 2

Exadata MAA Architecture................................................................. 3
Initial Deployment .............................................................................. 5
Hardware Components .................................................................. 8
Software Components: .................................................................. 9
High Performance........................................................................ 10
Post Deployment Exadata MAA Configuration.............................. 10
Database Archivelog Mode and Enable Database Force Logging 11
Fast Recovery Area ..................................................................... 11
Oracle Flashback Technologies ................................................... 11
Backup, Restore, and Recovery .................................................. 13
Oracle Data Guard and Oracle Active Data Guard ...................... 13
Comprehensive Data Corruption Protection................................. 14
Application Clients - Failover Best Practices ................................ 16
Database Consolidation Best Practices ........................................... 16
Operational Best Practices for Exadata MAA................................... 17
Importance of a Test Environment ............................................... 19
Conclusion ...................................................................................... 21
Appendix 1: Exadata MAA Outage and Solution Matrix ................... 22
Unplanned Outages..................................................................... 22
Planned Maintenance .................................................................. 25
Appendix 2: Exadata MAA Performance.......................................... 27
References ...................................................................................... 29
Executive Overview
The integration of Oracle Maximum Availability Architecture (MAA) operational and

configuration best practices with Oracle Exadata Database Machine (Exadata MAA)
provides the most comprehensive high availability solution available for the Oracle
Database.
Exadata MAA is a very mature and is pre-optimized, pre-configured, integrated system of
software, servers, storage and MAA configuration best practices that comes ready-built to
implement the highest database and application availability and performance. Many
extremely mission critical applications in various industries including our top financial
companies, telecoms and web customers run and rely on Exadata MAA today. The full
Exadata system from database to disks has gone through many very thorough availability
evaluations both internally and with our top mission critical customers. Every critical
issue found in these evaluations have been corrected and integrated back into the
engineered system. So with Exadata, every customer benefits not just from Oracle's
efforts, but also the efforts of the most demanding customers in the world.
This paper provides Exadata MAA insight for rapid deployment and efficient operation of
Exadata Database Machine. It is divided into four main areas:
1. Exadata MAA Architecture
2. Initial Deployment
3. Post Deployment: Exadata MAA Configuration
4. Operational Best Practices for Exadata MAA
Exadata MAA Architecture

Exadata MAA architecture shown in Figure 1 is designed to tolerate unplanned outages of all
types, providing both high availability (HA) and data protection. Exadata MAA also minimizes
planned downtime by performing maintenance in a rolling fashion. The range of potential
outages and planned maintenance addressed by Exadata MAA are described more fully in
Appendix 1 of this paper: Exadata MAA Outage and Solution Matrix. For real world examples
of how Exadata achieves end-to-end application availability and near zero brownout for various
hardware and software outages, refer to this Exadata MAA video
(http://vimeo.com/esgmedia/exadata-maa-tests).
Figure 1. Basic Oracle Exadata Database Machine Configuration
The Exadata MAA architecture consists of the following major building blocks:
1.
A production Exadata system (primary). The production system may consist of one or more
interconnected Exadata Database Machines as needed to address performance and scale-out
requirements for data warehouse, OLTP, or consolidated application environments.
2.
A standby Exadata system that is a replica of the primary. Oracle Data Guard is used to
maintain synchronized standby databases that are exact, physical replicas of production
databases hosted on the primary system. This provides optimal data protection and high
availability if an unplanned outage makes the primary system unavailable. A standby Exadata
system is most often located in a different data center or geography to provide disaster
recovery (DR) by isolating the standby from primary site failures. Configuring the standby
system with identical capacity as the primary also guarantees that performance service-level
agreements can be met after a switchover or failover operation.
Note that Data Guard is able to support up to 30 standby databases in a single
configuration. An increasing number of customers use this flexibility to deploy both a local
Data Guard standby for HA and a remote Data Guard standby for DR. A local Data Guard
standby database complements the internal HA features of Exadata Database Machine by

providing an additional layer of HA should unexpected events or human error make the
production database unavailable even though the primary site is still operational. Low
network latency enables synchronous redo transport to a local standby resulting in zero data
loss if a failover is required. A local standby database is also useful for offloading backups
from the primary database, for use as a test system, or for implementing planned
maintenance in rolling fashion (e.g. database rolling upgrades). The close proximity of the
local standby to the application tier also enables fast redirection of application clients to the
new primary database at failover time. Following a failover or switchover to a local standby
database, the remote standby database in such a configuration will recognize that a role
transition has occurred and automatically begin receiving redo from the new primary
database - maintaining disaster protection at all times.
While the term standby is used to describe a database where Data Guard maintains
synchronization with a primary database, standby databases are not idle while they are in
standby role. High return on investment is achieved by utilizing the standby database for
purposes in addition to high availability, data protection, and disaster recovery. These
include:
3.
Oracle Active Data Guard enables users to move read-only queries, reporting, and
fast incremental backups from the primary database, and run them on a physical
standby database instead. This improves performance for all workloads by bringing
the standby online as a production system in its own right. Oracle Active Data
Guard also improves availability by performing automatic repair should a corrupt
data block be detected at either the primary or standby database, transparent to the
user.
Data Guard Snapshot Standby enables standby databases on the secondary system
to be used for final pre-production testing while they also provide disaster
protection. Oracle Real Application Testing can be used in conjunction with
Snapshot Standby to capture actual production workload on the primary and replay
on the standby database. This creates the ideal test scenario, a replica of the
production system that uses real production workload enabling thorough testing
at production scale.
Data Guard Standby-First patching (My Oracle Support Note 1265700.1) or Data
Guard Database Rolling Upgrades are two methods of reducing downtime and risk
during periods of planned maintenance. This is a key element of Exadata MAA
Operational Best Practices discussed later in this paper.
A development/test Exadata system that is independent of the primary and standby Exadata
systems. This system will host a number of development/test databases used to support
production applications. The test system may even have its own standby system to create a
test configuration that is a complete mirror of production. Ideally the test system is
configured similar to the production system to enable:
o
Use of a workload framework (e.g. Real Application Testing) that can mimic the
production workload.
Validation of changes in the test environment, including evaluating the impact of

the change and the fallback procedure, before introducing any change to the
production environment.
Validation of operational and recovery best practices.
Some users will try to reduce cost by consolidating these activities on their standby Exadata
Database Machine. This is a business decision that represents a trade-off between cost and
operational simplicity and flexibility. In the case where the standby Exadata Database
Machine is also used to host other development and test databases, additional measures may
be required at failover time to conserve system resources for production needs. For example,
non-critical test and development activities may have to be deferred until failed system is
repaired and back in production.
The Exadata MAA architecture provides the foundation needed to achieve high availability.
Equally important are configuration and operational practices described in the sections that
follow.
Initial Deployment
Oracle Exadata Database Machine is a pre-optimized, pre-configured, integrated system of
software, servers, and storage that comes ready-built to implement Exadata MAA. This section
focuses on what is pre-configured and the HA best practices that apply to initial deployment of
the Exadata Database Machine.
Oracle recommends that the following configurable deployment options be specified when
completing the Exadata Database Machine Configurator spreadsheet prior to delivery of the
system:
Choose HA bonded networks for client access.
Choose Oracle Automatic Storage Management (ASM) high redundancy for DATA and
RECO disk groups for best protection from the following storage outage scenarios:
a.
Double partner disk failure 1: Protection against loss of the cluster and Oracle
ASM disk group due to a disk failure followed by a second disk failure of a
partner disk.
b. Disk failure when Oracle Exadata Storage Server is offline: Protection against
loss of the cluster and Oracle ASM disk group when a storage server is offline
and one of the storage server's partner disks fails. The storage server may be
offline because of planned maintenance, such as online storage server patching,
or because of a storage server failure.
c. Disk failure followed by disk sector corruption: Protection against data loss
when latent disk sector corruptions exist and a partner storage disk is
unavailable either due to planned maintenance or disk failure.
MAA best practices allow the use of Oracle ASM Normal Redundancy for all disk
groups when the primary database is protected by a Data Guard standby database
deployed on a separate Exadata Database Machine and the operational practices are in
place to fail over in case of any of the above storage failures. Data Guard provides realtime data protection and supports availability objectives by enabling a failover to the
standby database should a double failure impact the primary database.
Refer to the Exadata Storage Server Software Users Guide section About Oracle ASM for
Maximum Availability and note that putting Oracle clusterware voting files in high
redundancy disk groups require at least 5 Exadata cells (failure groups) in the disk group.
Also in the case of disk failure or cell storage failure, Oracle ASM will automatically
rebalance to restore redundancy; however, you will need enough disk group free space
to ensure this is achievable. An Exadata health check with exachk (described later) has a
configuration check for minimum Oracle ASM free space.
Depending on the Oracle ASM redundancy level, each disk has a partner disk found in a separate
Exadata storage cell. If a write or read errors occurs on one disk, then Oracle ASM can consult the
Partner Status Table (PST) to see whether any of the disks partners are online. If too many partners
are offline, Oracle ASM forces the dismounting of the disk group. Refer to Oracle Automatic Storage
Management Administrator's Guide for more information about disk partnering.
Select the check box to generate the Oracle Enterprise Manager Grid Control
configuration files and help deploy Oracle Enterprise Manager Grid Control.
Fill in the entries to configure and enable Auto Service Request (ASR) so service
requests are automatically opened for specific hardware faults.
Select the option to enable Oracle Configuration Manager (OCM) to collect

configuration information and upload it to the Oracle repository. When the
configuration information is uploaded daily, Oracle Support Services can analyze the
data and provide better service.
Oracle Exadata Deployment Assistant (OEDA) creates the configuration file used with the
Oracle OneCommand utility. Before using OEDA, work with your network and database
administrators to evaluate the current network settings, such as current IP address use and
network configuration. Next, define the settings for Oracle Exadata Database Machine, such as
network configuration and backup method.
It is important to run OEDA and generate the configuration file prior to delivery of Oracle
Exadata Database Machine. After the Exadata Database Machine is delivered, an Oracle systems
engineer will validate the hardware and run a final check for any components that may have been
damaged while in-transit. Next, Oracle Advanced Support Services or Consulting Services will
execute the Oracle OneCommand utility using the supplied configuration files.
Within a few days after hardware arrival the system and database will be fully functional with:
Validated hardware, InfiniBand network, and Exadata storage cells that are available and
performing according to specifications.
Recommended operating system, firmware, and Oracle software and patches as

described in My Oracle Support Note 888828.1. Oracle Advanced Support Services or
Consulting Services may apply additional fixes as described in My Oracle Support Note
1270094.1 or there may be a slight deviation if the MOS note has just recently changed.
An Exadata Storage Grid consisting of DATA, RECO, and DBFS ASM disk groups.
DATA and RECO disk groups are configured using the Oracle ASM redundancy
option(s) previously selected through OEDA.
Network infrastructure that includes InfiniBand networks for all communication

between database and Exadata storage servers. The network is configured for both
client and management access.
Oracle Real Application Clusters (Oracle RAC) database and Grid Infrastructure
preconfigured using Exadata MAA configuration settings described in My Oracle Support
Note 757552.1.
Email alerts for cell alerts, Oracle Enterprise Manager Grid Control (initial configuration
files), Automatic Service Request and Oracle Configuration Manager base setup will be
automatically configured on the Exadata Database Machine if options checked as part
of the OEDA wizard.
Oracle Exadata Database Machine is engineered and preconfigured to achieve end-to-end

application availability with every hardware fault such FANs, PDUs, batteries, switch, disk,
database server, motherboards, and DIMMs for example. We cannot over-emphasize the amount
of the engineering and integrated testing involved in evaluating and configuring the "full stack
solution" which includes hundreds of integrated HA tests performed on a daily basis.
The HA characteristics inherent in a preconfigured Exadata Database Machine are described in
the following sections.
Hardware Components
The following hardware and component redundancy is common to all models of Exadata
Database Machine X3-2, X3-8, X2-2 and X2-8:
Redundant database servers
Each Exadata Database Machine comes with multiple preconfigured industry-standard Oracle
Database servers running Oracle Database 11g Release 2 (11.2) and Oracle Real Application
Clusters. Oracles engineering and testing teams ensure the firmware, software, and hardware
configuration is tuned and pre-configured to provide high availability and scalability. The
database servers are clustered, and they communicate with each server using the high bandwidth,
low latency InfiniBand network. With this configuration, applications can tolerate a database
server or Oracle RAC instance failure with minimal impact.
Redundant storage
Exadata Storage components database nodes local drives, Exadata disks, and Oracle Exadata
Storage Server (Exadata cell) are all redundant. The storage grid is preconfigured with Oracle
Automatic Storage Management (Oracle ASM) and Exadata so that your application can tolerate
hard disk, flash disk, flash card, and other Exadata storage cell failures. Exadata cells are
network-accessible storage devices with Oracle Exadata Storage Server Software pre-installed.
Database data blocks and metadata are mirrored across cells to ensure that the failure of an
Exadata disk or Exadata cell does not result in loss of data or availability. Disk drives are hot
pluggable.
Redundant connectivity
Redundant InfiniBand network connectivity using dual-ported Quad Data Rate (QDR) Host
Channel Adapters (HCAs) and redundant switches are pre-configured. Configuring the same
redundancy for client access is recommended and can be done at deployment time.
Redundant power supply
Each Exadata Database Machine has redundant power distribution units (PDUs) for high
availability. The PDUs accept separate power sources and provide a redundant power supply to:
Oracle Database nodes
Exadata Storage Cells
InfiniBand switches
Cisco network switch
Power supplies for Oracle Database nodes, Exadata Storage Cells, InfiniBand and Cisco switches
are all hot swappable.
Software Components:
The following are standard Oracle software components explicitly optimized and validated for
Exadata Database Machine.
Firmware and Operating System
All database and Exadata storage servers are packaged with validated firmware and operating
system software preinstalled.
Database Server Tier
Grid Infrastructure (Oracle Clusterware and Oracle ASM) and Oracle RAC software is installed
and patched to recommended software version at deployment, enabling applications to tolerate
and react to instance and node failures automatically with zero to near-zero application brown
out. As described in Appendix 1, all Grid Infrastructure patches and most database patches can
be applied in rolling fashion.
Storage Tier
The storage tier with ASM and Exadata software supports high availability by:
Tolerating hard disk, flash disk, flash card and Exadata cell failures
Applying many software changes in a rolling manner
Exadata storage cells include Oracle Hardware Assisted Resilient Data (HARD) to provide a
unique level of validation for Oracle block data structures such as data block address, checksum
and magic numbers prior to allowing a write to physical disks. HARD validation with Exadata is
automatic (setting DB_BLOCK_CHECKSUM is required to enable checksum validation). The
HARD checks transparently handle all cases including Oracle ASM disk rebalance operations and
disk failures.
High Performance
Oracle Development teams who focus on high performance for OLTP and Data Warehouse
applications have optimized the configuration defaults that are pre-set on Exadata Database
Machine. In some cases, there will be different default settings for Exadata Database Machine
X3-2, X3-8, X2-2 and X2-8. These settings are the result of extensive performance testing with
various workloads, both in Oracle labs and in real-world Exadata Database Machine
deployments.
Validated configuration defaults have also undergone fault injection testing in a full MAA
environment that includes Oracle RAC, Oracle ASM, RMAN, Flashback, and Data Guard.
Post Deployment Exadata MAA Configuration

Oracle strongly recommends that you DO NOT use initialization settings from previous
Oracle configurations. Begin with the default configuration delivered with Oracle Exadata
Database Machine and then tune if required. Most customers will find minimal need for tuning
database parameters in the post deployment phase. If more databases are added for
consolidation, system resources may have to be adjusted as described in the MAA paper: Best
Practices For Database Consolidation On Oracle Exadata Database Machine.
Additional configuration using HA best practices in each of the following areas is required for
complete implementation of Exadata MAA:
Database Archivelog Mode and Enable Force Logging
Fast Recovery Area
Oracle Flashback Technologies
Backup, Restore, and Recovery
Data Guard and Active Data Guard
Comprehensive Data Corruption Protection
Automatic Failover of Application Clients
Additional MAA configuration changes
10
Database Archivelog Mode and Enable Database Force Logging

Archivelog Mode and Database Force Logging are prerequisites to insure all changes are
captured in the redo and applied during database recovery operation. Enable database archive
logging and database force logging to prevent any nologging operations.
SQL> ALTER DATABASE ARCHIVELOG;
SQL> ALTER DATABASE FORCE LOGGING;
Optionally you may set logging attributes at the tablespace level if it is possible to isolate data that
does not require recovery into a separate tablespace. For more information refer to Reduce
Overhead and Redo Volume During ETL Operations in the technical white paper, Oracle Data
Guard: Disaster Recovery for Oracle Exadata Database Machine [7].
Fast Recovery Area

The Fast Recovery Area (FRA) is Oracle-managed disk space that provides a centralized disk
location for backup and recovery files.
The FRA is defined by setting two database initialization parameters.
The DB_RECOVERY_FILE_DEST parameter specifies the location for the Fast Recovery Area.
Use the RECO disk group, and MAA recommends leveraging Exadata storage grid instead of
external storage for the best performance.
The DB_RECOVERY_FILE_DEST_SIZE parameter specifies (in bytes) a hard limit on the total
space to be used by database recovery files created in the recovery area location.
Oracle Flashback Technologies

Oracle Flashback technologies provide fast point-in-time recovery to repair logical database
corruptions, most often caused by human error. There is a full suite of Oracle Flashback
technologies that enable repair at an optimal level of granularity to achieve the fastest possible
repair with the minimum amount of data loss. These technologies include:
Flashback Database
Flashback Database uses flashback logs to rewind an entire Oracle Database to a previous point
in time. Flashback Database is used for fast point-in-time recovery from logical corruptions,
most often caused by human error, that cause widespread damage to a production database. It is
also used to quickly reinstate a failed primary database as a new standby database after a Data
11
Guard failover, avoiding the costly and time-consuming effort of restoring from a backup. See
section 13.2 of Data Guard Concepts and Administration for more information on fast reinstatement
using Flashback Database [8]. Customers are also creating Oracle Flashback Database
Guaranteed Restore Points prior to major changes including software changes for a quick
fallback.
Issue the following command on both primary and standby databases to enable Flashback
Database.
ALTER DATABASE FLASHBACK ON;
Oracle Database 11.2.0.2 and 11.2.0.3 releases includes substantial performance enhancements
that reduce the impact of flashback logging on the primary database, and in particular its impact
on load operations and initial flashback allocations. This enables Flashback Database to be used
with OLTP and data warehouse applications with minimal performance impact. Oracle MAA
tests have achieved a load rate of 3 TB/hour using Oracle Database Release 11.2.0.2 with
Flashback Database enabled. This rate is possible due to the large I/O bandwidth of Exadata
Database Machine, 11.2.0.2 optimizations and MAA configuration best practices:
Flashback
Use
Database best practices documented in My Oracle Support Note 565535.1.
of local extent-managed tablespaces.
Re-creation
of objects instead of truncating tables prior to direct load.
Additional Oracle Flashback Technologies for More Granular Repair
More granular levels of repair are enabled by other Oracle Flashback technologies that include:
Flashback Query, Flashback Version Query, Flashback Transaction, Flashback Transaction
Query, Flashback Table, and Flashback Drop.
Flashback Drop requires configuration of a Recycle Bin. All of the other features use automatic
undo management and require that you allocate sufficient disk space to achieve your desired
undo retention guarantee or how far back in the past you want to be able to recover.
Configure undo retention guarantee
Undo retention period (in seconds) is specified by setting the UNDO_RETENTION
initialization parameter to a value that is at least double (2x) the desired detection period to
ensure Oracle Flashback operations can operate. Note that if you are using Oracle Active Data
Guard to offload read-only workload to a standby database, please see additional guidance on
setting undo retention period provided in Oracle Active Data Guard Best Practices [13].
See Section 13.2.7 of Oracle Database High Availability Best Practices [3] for detailed best practices on
the full complement of Oracle Flashback Technologies.
12
Backup, Restore, and Recovery

Oracle Database backup technologies, Oracle Recovery Manager, and Oracle Secure Backup
perform especially well on Exadata Database Machine with its high bandwidth InfiniBand
network and Exadata Storage Server Grid. Examples of performance in Oracle Database release
11.2.0.2 or higher include:
Disk-based backups taking full-image copies can be done at 20-25 TB/hour.
Effective backup rates of up to 50 TB/hour using incremental backups.
Database restore rates of over 20-25 TB/hour into normal redundancy disk group and 14
TB/hour into a high redundancy disk group.
Database redo apply rate of up to 2.8 TB/hour (840 MB/sec), or 1.9 TB/hour (568MB/sec)
with Flashback Database enabled (the recommended best practice) and Exadatas write back
flash cache for load operations.
Tape-based backups of up to 7 TB/hour for a full backup, and an effective backup rate of up to
50 TB/hour using incremental backup. Higher full backup rates are possible by configuring
additional media servers and tape drives.
These above rates are achieved with a database CPU utilization of less than 5% on the targeted
database servers, leaving plenty of CPU bandwidth for concurrent user workloads.
Backup and restore rates with Oracle Exadata Storage Expansion rack is higher. Refer to Oracle
Exadata Storage Expansion Rack Data Sheet.
For more information, see the MAA technical whitepaper Backup and Recovery Best Practices and
Performance for Exadata Database Machine appropriate for your release, either Oracle Database
11.2.0.2 and 11.2.0.3 [4] or Oracle Database 11.2.0.1 and prior releases [5].
Oracle Data Guard and Oracle Active Data Guard

Oracle Data Guard is the disaster recovery solution prescribed by the Maximum Availability
Architecture (MAA) to protect mission-critical databases residing on Oracle Exadata Database
Machine. Data Guard is also used to maintain availability should any outage unexpectedly impact
the production database, and to minimize downtime during planned maintenance. Data Guard is
included in the Enterprise Edition license for Oracle Database.
Oracle Active Data Guard provides the following extensions to basic Data Guard functionality
(Oracle Active Data Guard requires an additional license to access its advanced features).
A physical standby database may be open read-only while it applies updates received from the
primary. This enables offload of read-only workload to a synchronized standby database,
increasing capacity and improving the performance of the primary database. It is significant to
13
note that read-only queries on an Oracle Active Data Guard standby database have the same
guarantee of read-consistency as queries executing on the primary database no physical
replication solution supplied by any other DBMS vendor can provide this level of readconsistency.
RMAN block-change tracking can be enabled on the physical standby database, enabling fast
incremental backups to be offloaded from the primary database.
Automatic block repair is enabled for the Oracle Active Data Guard configuration. Should
Oracle detect a physical corrupted block on either the primary or standby, it will automatically
repair the physical block corruption using the good copy from the other database, transparent to
the application and user.
Data Guard is the disaster recovery solution used by Exadata MAA because:
It provides data protection superior to storage remote-mirroring [11].
It is a highly efficient physical replication process that supports all Oracle data types and Oracle
Database features, providing a level of simplicity and performance that is superior to logical
replication.
It also addresses planned maintenance, including support for database rolling upgrades when
upgrading to new Oracle Patch-Sets (e.g. 11.1.0.7 to 11.2.0.3) or future Oracle Database Releases
using Transient Logical Standby [6,7].
It provides a zero-risk, simple method for installing Oracle patches, including Oracle Exadata
Database Machine bundled patches, Oracle Exadata Storage Server Software patches, Patch Set
Updates (PSU), Critical Patch Updates (CPU), or Patch Set Exceptions (PSE) using StandbyFirst Patch Apply. Standby-First Patch Apply is supported for certified software patches for
Oracle Database Enterprise Edition Release 2 (11.2) release 11.2.0.1 and later. Refer to My Oracle
Support Note 1265700.1 for more information and the README for each patch to determine if a
target patch is certified as being a Standby-First Patch.
Oracle Active Data Guard provides high return on investment on standby systems while they are
in standby role by enabling read-only workload to be offloaded from primary systems.
For the complete set of best practices for optimal data protection and availability when
configuring Data Guard, see the technical white paper, Oracle Data Guard: Disaster Recovery for
Oracle Exadata Database Machine [6] and the documentation for Configuring Oracle Data Guard
in Oracle Database High Availability Best Practices [3].
Comprehensive Data Corruption Protection

Oracle Database includes comprehensive database-aware capabilities to either prevent or
automatically detect and repair data corruption that could otherwise lead to substantial downtime
and data loss. Data corruption occurs when a data block is not in a valid Oracle Database format,
14
or whose contents are not internally consistent. Corruption can result from either hardware or
software defects.
Listed below are the Exadata MAA recommended initialization settings for optimal data
corruption protection. The recommended settings place a greater emphasis on protection
compared to the default settings that favor minimizing performance impact. Where different
from the default value, the recommended initialization settings will need to be configured
manually. Note that a Data Guard standby database is necessary for complete protection.
Oracle strongly recommends that you read My Oracle Support Note 1302539.1 for additional
background and for important insight into the potential performance tradeoff. This tradeoff
requires that you conduct performance testing for your application workload prior to production
go-live.
On the Primary Database
DB_BLOCK_CHECKSUM=FULL (default TYPICAL on Exadata)

DB_BLOCK_CHECKING=FULL (default OFF on Exadata)
Some applications are not able to set DB_BLOCK_CHECKING=MEDIUM or FULL on the
primary due to performance impact. MAA recommends that you set
DB_BLOCK_CHECKING=MEDIUM or FULL on the physical standby as a minimum
practice to prevent the standby from various logical block corruptions.
DB_LOST_WRITE_PROTECT=TYPICAL (default TYPICAL on Exadata)
Enable Flashback Technologies for fast point-in-time recovery from logical corruptions that are
most often caused by human error and for fast reinstatement of a primary database following
failover.
On the Data Guard Physical Standby Database
DB_BLOCK_CHECKSUM=FULL
DB_BLOCK_CHECKING=OFF
DB_LOST_WRITE_PROTECT=TYPICAL
Setting DB_BLOCK_CHECKING=FULL provides maximum data corruption detection and
prevention on the standby; however redo apply performance can be reduced significantly.
Because redo apply performance is so fast, especially with Exadatas write back flash cache, most
customers can set DB_BLOCK_CHECKING=FULL or MEDIUM on the standby and still
maintain the required redo apply rates and availability service levels.
Enable Flashback Technologies for fast point-in-time recovery from logical corruptions that are
most often caused by human error and for fast reinstatement of a primary database following
failover.
15
Use Oracle Active Data Guard to enable Automatic Block Repair (Data Guard 11.2 onward).
Application Clients - Failover Best Practices

High application availability is only achieved if an application reacts and fails over transparently
when an unplanned outage or planned maintenance activity impacts the availability of the
production database. Automate the detection and failover process by following the best practices
described in the technical white paper, Client Failover Best Practices for Data Guard 11g Release 2 [9].
In addition, there are Exadata-specific best practices for Oracle packaged applications. Please
refer to the Oracle Technology Network for MAA Best Practices for Exadata Database Machine [10]
for additional recommendations for Oracle e-Business Suite, Siebel, PeopleSoft, and other
packaged applications.
Additional MAA Configuration Changes

The Exadata MAA development team working with Oracle support, Oracle test teams, mission
critical Exadata customers, and other development organizations may find new critical
configuration practices over time. These checks and recommendations are added to the Exadata
MAA health check (exachk) as well as optimizing the Exadata deployment configuration defaults
so all customers can take advantage of consolidated knowledge. It is recommended that you
download the latest exachk from My Oracle Support Note 1070954.1 and follow any additional
configuration, software, or hardware recommendations from the exachk report.
Database Consolidation Best Practices

Exadata is optimized for Oracle Data Warehouse and OLTP database workloads, and its
balanced database server and storage GRID infrastructure make it an ideal platform for database
consolidation. Oracle database, storage, and network GRID architectures combined with Oracle
resource management provide a simpler and more flexible approach to database consolidation
than other virtualization strategies (e.g. hardware or O.S. virtualization). Exadata does not
support virtual machines or Solaris Containers/Zones, but relies on the simplicity of Oracle
resource management for efficient database consolidation.
Refer to the following MAA white papers for database consolidation best practices on Oracle
Exadata Database Machine: Best Practices for Database Consolidation on Oracle Exadata Database
Machine [14], Oracle Exadata Database Machine Consolidation: Segregating Databases and Roles [15].
16
Operational Best Practices for Exadata MAA

The following operational best practices are required for a successful Oracle Exadata Database
Machine implementation:
1.
Document your high availability and performance service-level agreements (SLAs) and
create an outage/solution matrix that maps to your SLAs.
Understanding the impact to the business and the resulting cost of downtime and data loss is
fundamental to establishing Recovery Time Objectives (RTO) and Recovery Point
Objectives (RPO). RTO measures your tolerance for downtime while RPO measures your
tolerance for data loss. It is also likely that RTO and RPO will be different for different
classes of outages. For example, server and disk failures usually have RTO/RPO of zero. A
complete site failure may have larger RTO/RPO as well as less stringent performance SLAs.
This is due to managing the trade-off between the potential frequency of an outage
occurring and the cost or complexity of implementing HA/DR.
The range of outages that must be planned for in an Exadata MAA environment are
described in Appendix 1, Exadata MAA Outage and Solution Matrix. It is important to
document your procedures for detection and repair for each type of outage. In some cases
you will take advantage of Oracle software infrastructure to configure an automatic
response. In other cases, Oracle software infrastructure will provide automatic notification
but resolution will occur via human intervention. In all cases is it required to validate that
each detection and repair procedure is able to meet SLAs. A similar process is followed to
implement performance monitoring and rapid response to ensure that performance SLAs,
such as throughput and response time requirements, are also achieved.
2.
Validate HA and Performance SLAs. Perform fault-injection testing to validate the expected
HA response and all automatic, automated, or manual repair solutions. Ensure that the
application RTO and RPO requirements are met. Ensure that application performance is
acceptable under different scenarios of component failures. For example, does the
application continue to meet performance SLAs after node failure, Exadata Storage cell
failure, and Data Guard role transition?
3.
Test and upgrade to software recommended in My Oracle Support Note 888828.1.

Exadata Database Machine will be delivered and deployed with the then current
recommended HA software and system components. Once deployed it is necessary to
periodically check My Oracle Support Note 888828.1 for new software updates and
recommendations. It is a best practice to apply the recommended patches and upgrades
quarterly or bi-annually. Validate any change on test systems prior to applying them to a
production system. The standby database can also be used for final pre-production testing,
and to implement the change in a rolling manner.
17
4.
Check for any critical software issues in My Oracle Support Note 1270094.1 for any issues
applicable to your environment.
The information in this note describes any critical issue for Exadata Storage Server 11.2 and
Oracle Database 11g Release 2 using Exadata storage. If you are running an affected release
and the problem statement is relevant to your specific application, it is highly recommended
that you either employ the recommended workaround, or install the recommended patch.
Refer to this note every month or run the latest exachk to get the latest critical issues checks
and refer to the exachk report.
5.
Follow the Exadata testing and patching practices described in My Oracle Support Note
1262380.1.
Pre-production validation and testing of software patches is one of the most important ways
to maintain stability. The high-level steps are:
a.
Review the patch and upgrade documentation.
b. Evaluate any rolling upgrade opportunities in order to minimize or eliminate planned

downtime.
c.
Evaluate whether the patch qualifies for Standby-First Patching, described in My Oracle
Support Note 1265700.1.
d. For all database software changes, use Out-of-Place Software installation and patching
for easier fallback, described in section 14.2.4.3 Out-of-place Software and Installation
Patching in Oracle Database High Availability Best Practices.
6.
e.
Validate the application in a test environment and ensure the change meets or exceeds
your functionality, performance, and availability requirements. Automate the procedure
and be sure to also document and test a fallback procedure.
f.
If applicable, perform final pre-production validation of all changes on a Data Guard

standby database before applying them to a production system.
g.
Apply the change in your production environment.
Execute the Exadata MAA health check (exachk), as described in My Oracle Support Note
1070954.1.
Before and after each software patch, before and after any database upgrade, or minimally
every month, download the latest release of exachk and run it in your test and production
environments to detect any environment and configuration issues. Checks include verifying
the software and hardware and warning if any existing or new MAA, Oracle RAC, or
Exadata hardware and software configuration best practices need to be implemented. An
MAA score card has been added with MAA configuration checks and best practices. An
upgrade module has been added to proactively detect any configuration issues pre and post
database upgrade.
18
7.
Execute Data Guard role transitions.

Periodically execute Application and Data Guard switchovers to fully validate all role
transition procedures. We recommend conducting role transition testing a minimum of once
per quarter. In addition to Data Guard documentation, refer to My Oracle Support Notes
for switchover best practices for Data Guard Physical Standby (11.2.0.2); see either Note
1304939.1 if using SQL*Plus, or Note 1305019.1 if using the Data Guard Broker/Enterprise
Manager.
8.
Configure Exadata Database Machine monitoring and Automatic Service Request.

Incorporate monitoring best practices for the most comprehensive monitoring of the
Exadata Database Machine as described in the Oracle Enterprise Manager 12c: Oracle
Exadata Discovery Cookbook and My Oracle Support Note 1110675.1. See
http://www.oracle.com/asr and refer to the Oracle Exadata Database Machine Owners
Guide (delivered with Exadata Database Machine) for additional information about
Automatic Service Request.
9.
Check the Exadata and MAA repositories for the latest Exadata best practices.
See the following repositories of information:
Exadata best practices at My Oracle Support Note 757552.1, but all the configuration
checks are encapsulated into Exadata health check (exachk).
MAA OTN Portal [1]
Oracle 11g Release 2 Oracle Database High Availability Best Practices [3] documentation
for general MAA configuration and operational practices, including security best
practices, change control procedures and test plans.
Importance of a Test Environment

Investment in sufficient test system infrastructure is essential to Exadata MAA. Unfortunately,
test systems are frequently given lowest priority, with scarce capital dollars being reserved for
production environments. The table below describes the benefits and trade-offs of various
strategies for deploying test systems.
TABLE 1. TRADEOFFS FOR DIFFERENT TEST AND QA ENVIRONMENTS
TEST ENVIRONMENT
BENEFITS AND TRADEOFFS
Full Replica of Production Database
Validate all patches and software changes. Validate all functional tests.
Machine
Full performance validation at production scale
19
TABLE 1. TRADEOFFS FOR DIFFERENT TEST AND QA ENVIRONMENTS
TEST ENVIRONMENT
BENEFITS AND TRADEOFFS
Full HA validation especially if the replica includes the standby system.

Standby Database Machine
Validate most patches and software changes. Validate all functional tests.
Full performance validation if using Data Guard Snapshot Standby but this can
extend recovery time if a failover is required.
Role transition validation.
Resource management and scheduling is required.
Shared Exadata Database Machine
Validate most patches and software changes. Validate all functional tests.
This environment may be suitable for performance testing if enough system
resources can be allocated to mimic production.
Typically, however, a subset of production system resources, compromising
performance testing/validation.
Resource scheduling is required.
Smaller Exadata Database Machine
Validate all patches and software changes. Validate all functional tests.
No performance testing at production scale.
Common example is production using
Limited full-scale high availability evaluations.
High Performance (HP) disks and standby

using High Capacity (HC) disks or standby
being a smaller rack Exadata Database
Machine.
Older Exadata Database Machine
Validate most patches and software changes. Limited firmware patching test.
Validate all functional tests unless limited by some new hardware feature
Limited production scale performance tests.
Limited full-scale high availability evaluations.
Non-Exadata Database Machine
Validate database and grid infrastructure software and patches only.

Validate database generic functional tests.
Limited testing of Exadata specific software features (e.g., EHCC, IORM,
Storage Index, etc.)
Very limited production scale performance tests
Limited high availability evaluations.
20
Conclusion
Exadata MAA provides a full stack, end-to-end availability solution and provides the highest
performance and most available platform for running Oracle databases. The integration of MAA
architecture and MAA operational and configuration best practices are essential to maximize
application availability. This technical whitepaper has highlighted the HA capabilities that are
delivered pre-configured, and post-delivery configuration and operational best practices used to
fully deploy Exadata MAA [11].
21
Appendix 1: Exadata MAA Outage and Solution Matrix

Unplanned Outages
Use the following outage and solution matrix in Table 2 as a baseline when performing high
availability testing. The MAA recommended solution is provided for each type of outage along
with the expected application recovery time (RTO). Oracle recommends that you perform tests
by injecting these faults while running a real-world workload on an Exadata MAA test system.
Most outages should incur zero database downtime and a minimal application brownout for any
affected connections. For real world examples of how Exadata achieves end-to-end application
availability and near zero brownout for various hardware and software outages, refer to this
Exadata MAA video (http://vimeo.com/esgmedia/exadata-maa-tests).
Whether you are deploying manual or automatic failover, evaluate end-to-end application failover
time or brownout in addition to understanding the impact that individual components have on
database availability. The table includes links to detailed descriptions in Chapter 13, "Recovering
from Unscheduled Outages" in Oracle Database High Availability Best Practices [3].
TABLE 2 OUTAGE/SOLUTION MATRIX
OUTAGE SCOPE
FAULT INJECTION PROCESS
EXADATA MAA
Seconds to 5 minutesFootref 1
site failure
Database Failover with a Standby Database

Complete Site Failover
Application Failover
clusterwide failure
Seconds to 5 minutes
or production
Database Failover with a Standby Database
Exadata Database
Machine failure
computer failure
(node) or RAC
database node
failure (simulating
1. Unplug or power off or reboot a database

node
2. Wait 30 seconds or more
3. Restore power and power up database
node, if needed
4. Wait for database node to be fully up
the impact of
Small application downtime for cluster

detection, cluster reconfiguration and instance
recovery.
Managed automatically by Oracle RAC
Recovery for Unscheduled Outages
hardware failure,
reboots or
motherboard
failure)
Database Instance
failure
or RAC database
Kill -11 PMON background process or

shutdown abort the target instance
Small application downtime for affected

connections with sub-second application
failover is possible. For the affected
22
OUTAGE SCOPE
instance failure
EXADATA MAA
connections of the failed instance, brownout

will consist of cluster reconfiguration (1sec)
and instance recovery which is significantly
faster on Exadata with Exadata write back
flash cache. No database downtimeFootref 3
Managed automatically by Oracle RAC
Recovery for Unscheduled Outages
Exadata Storage
Server failure
1. Unplug or power off or reboot cell

2. Wait longer than ASM disk repair timer
Small application brownout with sub-second

repair using our InfiniBand fabric fast
detection mechanism.
(simulating a
storage head
failure)
Exadata Storage
Follow Exadata Owners guide procedure to
Server reboot
gracefully shutdown Exadata Storage
No application brownout.
server
Exadata cellsrv
process dies
1. Kill -11 cellsrv process on cell (simulate

SIGSEGV)
Small application brownout of maximum

couple seconds, no database downtime.
(simulating
Cellsrv quickly restarts automatically by cells
Exadata cell
resource manager background process. No
software failure)
loss of redundancy.
Exadata disk
failure or disk pull
1. Pull out spinning disk drive

3. Plug the same disk drive back in the
same slot
Zero application brownout especially with

Exadata write-back flash cache. Exadata and
Oracle ASM tolerate storage failures and
quickly redirect I/O to mirror(s) with minimum
service level impact.
Note: Disk pull: Oracle knows the difference
between a user pulling a good disk and a true
failed disk. For a disk pull and plug, a simple
ASM resynchronization is done of the delta
changes. A true disk failure on the other
hand will result in an immediate drop of the
failed disk and subsequent ASM rebalance
with no service level impact. Starting in
Exadata cell 11.2.3.2.0 or higher, a blue LED
light will indicate the failed disk.
Exadata flash disk

or flash DOM
failure
InfiniBand spine
No customer visible simulation.

Internal fault injection occurs. Oracle will
provide a software fault injection for
customers to inject this fault in the future
Application brownout of maximum one second
1. Pull IB cable on leaf switch connecting to

spine switch
No downtime
with write back flash cache and fast repair of

stale data.
23
OUTAGE SCOPE
switch failure or
EXADATA MAA
2. Plug IB cable back into leaf switch
port failure
InfiniBand leaf
switch failure or
port failure
1. Pull IB cable on leaf switch connecting to

cell or database node
3. Plug IB cable back into leaf switch
Small application brownout of couple seconds

due to redundant InfiniBand switches.
Bonded failover with minimal service level
impact pre-configured.
No application downtime
failure
1. Remove access to Cisco switch port

3. Restore access to Cisco switch port
Power failure or
1. Pull power to one of the PDU
No application brownout due to redundant
Cisco switch
PDU failure or loss
power failure.
of power source or
supply to any
computer or
Exadata cell
storage server
Client access
network failure
with HA bonded
network
human error
2. Pull Ethernet cable from eth1 interface

on database node
4. Plug Ethernet cable back into eth1
interface on database node
Small application brownout
< 30 minutesFootref 5
Recovering from Human Error
hangs or slow
See Oracle Database High Availability
down
Overview documentation for solutions for

unplanned downtime
Footnote 1 Recovery time indicated applies to database and existing connection failover. Network connection changes and
other site-specific failover activities may lengthen overall recovery time.
Footnote 2 Recovery time consists largely of the time it takes to restart the failed system.
Footnote 3 Database is still available, but portion of application connected to failed system is temporarily affected.
Footnote 4 Storage failures are prevented by using ASM with mirroring and its automatic rebalance capability.
Footnote 5 Recovery times from human errors depend primarily on detection time. If it takes seconds to detect a malicious
DML or DLL transaction, then it typically only requires seconds to flash back the appropriate transactions, if properly rehearsed.
Referential or integrity constraints must be considered.
24
Planned Maintenance
Table 3 shows the preferred approaches for performing scheduled maintenance on the Exadata
Database Machine after testing and patching best practices have been followed. The table
includes links to detailed descriptions in Chapter 14, "Reducing Downtime for Planned
Maintenance" in Oracle Database High Availability Best Practices [3]. Also see My Oracle Support Note
888828.1 for additional information.
TABLE 3. SOLUTIONS FOR SCHEDULED OUTAGES ON THE PRIMARY SITE
PLANNED MAINTENANCE
PREFERRED ORACLE SOLUTION
ESTIMATED
DOWNTIME
Site maintenance
Site, Hardware, and Software Maintenance Using
< 5 minutes
Database Switchover
Exadata Database Server
Apply at the standby and evaluate.
Firmware, Operating System
Oracle RAC rolling upgrade and service relocation (see
No downtime
Automatic Workload Management for System

Maintenance)
Hardware maintenance or system software
maintenance (node impact)
POracle RAC rolling upgrade and service relocation
No downtime
(see Automatic Workload Management for System

Maintenance)
Oracle Grid Infrastructure (GI) Upgrades
No downtime
Oracle Grid Infrastructure rolling patch upgrade (see

your platform-specific Oracle Clusterware Installation
Guide for complete details)
Oracle patch set or software upgrade for the
Oracle Database rolling upgrade with Data Guard
database
(transient logical standby)
< 5 minutes
If Data Guard is not applicable or if less downtime is

required by using active-active replication, consider
Oracle GoldenGate.
Oracle patch upgrade for the database,
Apply using Standby-First patching and evaluate.
Exadata Bundle Patch which includes Critical
Oracle RAC rolling patch upgrade using opatch (see
Patch Updates (CPUs), Patch Set Updates
Oracle RAC Patches)
No downtime
(PSEs) and database fixes

Oracle diagnostic "one-off" patches
Apply Standby First and evaluate.
No downtime
Online Patching
Exadata storage upgrade or patch
No downtime
25
TABLE 3. SOLUTIONS FOR SCHEDULED OUTAGES ON THE PRIMARY SITE
PLANNED MAINTENANCE
PREFERRED ORACLE SOLUTION
ESTIMATED
DOWNTIME
(firmware, operating system, Exadata
Exadata rolling upgrade
software)
or Switchover
< 5 minutes
Switches
Small
(InfiniBand, Ethernet)
Apply and restart switch
application
brownout
Power distribution unit (PDU)
Keyboard, video, mouse (KVM)
Apply on primary
Database object reorganization or redefinition
Online object reorganization with
No downtime
No downtime
DBMS_REDEFINITION (see Data

Reorganization and Redefinition)
Database platform or location maintenance
Database Platform or Location Migration
< 5 minutes
Note: Refer to the patch README to determine if a patch is Oracle RAC Rolling Upgradeable
or eligible for Standby-First patching. If not, then use either Data Guard database rolling
upgrades using the transient logical standby process [7], or Oracle Goldengate.
26
Appendix 2: Exadata MAA Performance

Exadata MAA is designed for OLTP and Data Warehouse applications, and for database
consolidation. Exadata Database Machine has been engineered for very high performance and is
able to support the most demanding applications where traditional architectures typically fall
short.
Exadata MAA performance tests have demonstrated substantial performance improvements
over non-Exadata configurations. Test results for archiving, standby redo apply, and disk
backup/restores are provided in Table 4. The Exadata configuration included: Oracle RAC
(w/forced logging and archiving), Oracle ASM, RMAN, Flashback, and Data Guard (ASYNC
redo transport).
.
TABLE 4 KEY PERFORMANCE IMPROVEMENTS FOR EXADATA DATABASE MACHINE
PERFORMNCE
IMPROVEMENT OVER
TEST
NON EXADATA
NON EXADATA
EXADATA DATABASE MACHINE X2-2

FULL RACK (RELEASE 11.2.0.2 OR 11.2.0.3)
CONFIGURATIONS
Archiving
5 X Faster
40 MB/sec per instance
214 MB/sec per instance

5.8 TB/hour for full rack
Instance
Up to 15 X for OLTP
recovery
Redo Apply
251 seconds for IOPS/OLTP
15 seconds
intensive workload
Up to 4 X for
237 seconds for Sqlload/OLTP
Loads/OLTP
intensive workload
60 seconds
6 to 12 X Faster
25 MB/sec OLTP
Up to 300 MB/sec OLTP
100 MB/sec Direct Logged
800+ MB/sec Direct Logged Loads
Loads
Local Disk
5 to 6 X Faster
Less than 5 TB/hour
Backups and
depending on the network
Restores
bandwidth, disk, controller, and

ability to scale out
Up to 24 TB/hour for full disk backups

Up to 50 TB/hour for incremental backup
14 TB/hour for full disk restores to high
redundancy disk group
Additional MAA tests evaluated the ability of an Exadata MAA configuration to sustain high
load rates. These tests are especially applicable for Data Warehouse applications that require
batch cycles to complete within a limited window of time. Experience has shown that users of
non-Exadata systems will disable important high availability features such as logging and
27
archiving, or database flashback logging due to concern for their impact on performance. The
high performance of the Exadata Database Machine, however, enables users to run with an
Exadata MAA configuration and still meet their performance objectives.
The following load rates were achieved with a fully compliant MAA configuration (archivelog
mode, database forced logging, Flashback Database, all best practices for database corruption
settings, Oracle RAC, Oracle ASM redundancy, and Data Guard with asynchronous redo
transport):
3 TB/hour load rates with Oracle Database 11g Release 2 (11.2.0.2)
1.5 TB/hour load rates with Oracle Database 11g Release 1 (11.2.0.1)
Note that other internal Oracle load tests on Oracle Database 11g Release 2 utilizing Oracle
compression technologies achieved up to 5 TB/hour load rates. The Exadata MAA load test
reported above did not utilize compression, nor was it designed to be a comprehensive
performance benchmark its purpose is to demonstrate the performance potential of Exadata
Database Machine using a simple parallel direct loader test with all HA best practice
recommendations implemented.
28
References
1.
Oracle Maximum Availability Architecture

http://www.otn.oracle.com/goto/maa
2.
Oracle Database High Availability Overview

http://www.oracle.com/pls/topic/lookup?ctx=db112&id=HAOVW
3.
Oracle Database High Availability Best Practices

http://www.oracle.com/pls/topic/lookup?ctx=db112&id=HABPT
4.
Backup and Recovery Performance and Best Practices for Exadata Database Machine,
Oracle Database 11.2.0.2 and 11.2.0.3
http://www.oracle.com/technetwork/database/features/availability/maa-tech-wp-sundbm-backup11202-183503.pdf
5.
Backup and Recovery Performance and Best Practices for Exadata Database Machine,
Oracle Database 11.2.0.1 and prior releases
http://www.oracle.com/technetwork/database/features/availability/maa-tech-wp-sundbm-backupfinal-129256.pdf
6.
Oracle MAA white paper: Oracle Data Guard: Disaster Recovery for Exadata
Database Machine and Exadata Cell
http://www.oracle.com/technetwork/database/features/availability/maa-wp-dr-dbm-130065.pdf
7.
Database Rolling Upgrades Made Easy

http://www.oracle.com/technetwork/database/features/availability/maa-wp-11g-upgrades-madeeasy-131972.pdf
8.
Data Guard Concepts and Administration

http://www.oracle.com/pls/topic/lookup?ctx=db112&id=SBYDB
9.
Client Failover Best Practices for Data Guard 11g Release 2

http://www.oracle.com/technetwork/database/features/availability/maa-wp-11gr2-client-failover173305.pdf
10. Oracle Technology Network: MAA Best Practices for Exadata Database Machine
http://www.oracle.com/technetwork/database/features/availability/exadata-maa-best-practices155385.html
11. Data Guard and Storage Remote Mirroring

http://www.oracle.com/technetwork/database/features/availability/dataguardremotemirroring086151.html
12. Exadata Database Machine Best Practices OTN portal

http://www.oracle.com/technetwork/database/exadata/index-089737.html
29
13. Oracle Active Data Guard Best Practices

http://www.oracle.com/technetwork/database/features/availability/maa-wp-11gr1-activedataguard-1128199.pdf
14. Best Practices For Database Consolidation On Oracle Exadata Database Machine
http://www.oracle.com/technetwork/database/features/availability/exadata-consolidation522500.pdf
15. Oracle Exadata Database Machine Consolidation: Segregating Databases and Roles
http://www.oracle.com/technetwork/database/focus-areas/availability/maa-exadata-consolidatedroles-459605.pdf
30
Deploying Oracle Maximum Availability

Architecture with Exadata Database Machine
and Oracle Exadata
August 2013
Author: Lawrence To
Contributors: Mike Nowak, Miike Smith, Rene
Kundersma, Doug Utzig, Joe Meeks, Dan
Copyright 2013, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and
Norris, Ron Weiss, Viv Schupmann, Virginia
the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other
Beecher
warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or
fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are
Oracle Corporation
formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any
World Headquarters
means, electronic or mechanical, for any purpose, without our prior written permission.
500 Oracle Parkway

Redwood Shores, CA 94065
Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective
U.S.A.
owners.
Worldwide Inquiries:
0109

Maximum Availability Architecture: Oracle Best Practices For High Availability

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Maximum Availability Architecture: Oracle Best Practices For High Availability

Uploaded by

Copyright:

Available Formats

Deploying Oracle Maximum Availability Architecture with

Exadata Database Machine

Executive Overview ........................................................................... 2

The integration of Oracle Maximum Availability Architecture (MAA) operational and

Exadata MAA Architecture

Figure 1. Basic Oracle Exadata Database Machine Configuration

standby database complements the internal HA features of Exadata Database Machine by

Validation of changes in the test environment, including evaluating the impact of

Validation of operational and recovery best practices.

Choose HA bonded networks for client access.

Select the option to enable Oracle Configuration Manager (OCM) to collect

Recommended operating system, firmware, and Oracle software and patches as

Network infrastructure that includes InfiniBand networks for all communication

Oracle Exadata Database Machine is engineered and preconfigured to achieve end-to-end

Post Deployment Exadata MAA Configuration

Database Archivelog Mode and Enable Database Force Logging

Fast Recovery Area

Oracle Flashback Technologies

Database best practices documented in My Oracle Support Note 565535.1.

of local extent-managed tablespaces.

of objects instead of truncating tables prior to direct load.

Additional Oracle Flashback Technologies for More Granular Repair

Backup, Restore, and Recovery

Oracle Data Guard and Oracle Active Data Guard

Comprehensive Data Corruption Protection

DB_BLOCK_CHECKSUM=FULL (default TYPICAL on Exadata)

Application Clients - Failover Best Practices

Additional MAA Configuration Changes

Database Consolidation Best Practices

Operational Best Practices for Exadata MAA

Test and upgrade to software recommended in My Oracle Support Note 888828.1.

Review the patch and upgrade documentation.

b. Evaluate any rolling upgrade opportunities in order to minimize or eliminate planned

If applicable, perform final pre-production validation of all changes on a Data Guard

Apply the change in your production environment.

Execute Data Guard role transitions.

Configure Exadata Database Machine monitoring and Automatic Service Request.

MAA OTN Portal [1]

Importance of a Test Environment

TABLE 1. TRADEOFFS FOR DIFFERENT TEST AND QA ENVIRONMENTS

BENEFITS AND TRADEOFFS

Full Replica of Production Database

Full performance validation at production scale

TABLE 1. TRADEOFFS FOR DIFFERENT TEST AND QA ENVIRONMENTS

BENEFITS AND TRADEOFFS

Full HA validation especially if the replica includes the standby system.

Shared Exadata Database Machine

Smaller Exadata Database Machine

Common example is production using

Limited full-scale high availability evaluations.

High Performance (HP) disks and standby

Older Exadata Database Machine

Non-Exadata Database Machine

Validate database and grid infrastructure software and patches only.

Appendix 1: Exadata MAA Outage and Solution Matrix

TABLE 2 OUTAGE/SOLUTION MATRIX

FAULT INJECTION PROCESS

Database Failover with a Standby Database

Database Failover with a Standby Database

Complete Site Failover

1. Unplug or power off or reboot a database

Small application downtime for cluster

Kill -11 PMON background process or