Database Administration

DATABASE HIGH AVAILABILITY BEST PRACTICES

James Viscusi, Oracle Corporation Pradeep Bhanot, Oracle Corporation

EXECUTIVE OVERVIEW
A much-discussed aspect of The internet changing everything has been the increasing focus on availability in open systems applications. Seemingly overnight, application developers and support staffs were faced with the latest buzzword 24x7 and were given sometimes ill-defined requirements to have systems and applications available at all hours. IT management, developers, support staff, DBAs, system and network administrators were then faced with the problem of translating those requirements into an achievable reality, integrating solutions from multiple hardware and software vendors. Oracles philosophy with our latest generation of products is to provide a set of components that will present to the users of a system a seamless picture of application availability, even though any one component may be experiencing a failure. Through out the remainder of this paper, we will review the justifications for developing a highly available solution, justifying the expense and trouble, explore some of the key components available in the Oracle product set and illustrate their use with a sample system architecture Oracle has been serious about providing high application availability to customers since it first introduced support for a clustered database in Oracle version 6.2 on VMS in 1990. Even back then Oracle consulting would write scripts to provide a simple standby database running recovery at a backup site. Today Oracle is setting the pace in open systems High Availability with its new tagline: Unbreakable.

UPTIME WHAT DOES AN APPLICATION REALLY NEED?

While Oracle continues to make the implementation of highly available systems easier, there are still associated architecture, planning, development and maintenance costs. The first step in the development of any system deemed to be highly available is to determine the true requirements. Many developers say they have a 24x7 database without a clear understanding of what that means to their application and a clear understanding of the actual business processing requirements. When assessing your application to determine if it requires the precautions necessary to make it highly available consider questions like, Is there a short window of time at any point (daily, weekly, monthly) when the database can be unavailable? If so, it may be possible to use some of the less expensive/more time-consuming alternatives for availability.

Paper 541

Database Administration

MYTHS OF UPTIME NUMBERS

The following table provides a guideline when defining high availability requirements. Downtime for hardware and software upgrades also counts when considering availability requirements. Implementing and maintaining a highly available system can require a large investment in hardware, software and a further investment of staff time to construct the recovery procedures, test the solution and monitor the system to reduce the overall recovery time. Business users, management and IT staff should all understand at design time the investment required in setting up and maintaining a 24x7 system.

Availability Needed 99.00% 99.25% 99.50% 99.75% 99.90% 99.99% 99.999%

Downtime (min/year) 5,256 3,942 2,628 1,314 526 53 5

Downtime (hour/year) 87.6 65.7 43.8 21.9 8.76 0.88 0.08

System designers often discuss availability in terms of 100% uptime. Our goal is to make the users perception that the system is available all the time or 24x7 even though an individual component may have failed. We want to do this while keeping the cost of implementing such a system affordable and the management of the overall solution straight forward and easy. A business needs to balance the cost of providing a given level of availability vs. cost of downtime. According to a survey by Contingency Research Planning, Livingston, N.J., the leading causes of computer downtime for more than 12 hours were 31% Power Related (Surge etc.), 20% for storm Damage, 16% for Burst Pipes, 9% for Fire and Bombing, 7% Earthquakes and 4% attributed to Other causes. A study conducted by the University of Texas uncovered that all companies who suffer from major data loss and extended downtime: 6% survive, 43% never reopen and 51% close within 2 years (source CIO Magazine April 1998). It is interesting to note that, although 98 percent of CIOs polled believe it is important to have a disaster recovery plan, 25 percent do not have one in place. This is according to a poll conducted by RHI Consulting and published at (http://www.cio.com/archive/040198_disaster.html).

WHAT COULD HAPPEN, WHAT COULD HAPPEN TO YOU?

The next part of understanding the issues required to design a highly available application is to understand the potential problems and pitfalls. One part of this can be understood by understanding common causes of downtime in the industry. As a sample, the results of a recent

Paper 541

Database Administration

survey on reasons for data loss as published in the Disaster Recovery Journal are summarized in the chart below.

2001 Survey of 184 sites

From this data, we can identify the two most likely causes of an outage are Hardware /Software errors and human errors. When discussing equipment errors, this data allows us to understand and justify the cost of appropriate, fault tolerant hardware and software solutions. Its interesting to not that the second largest cause of outage and data loss is human error. This only reinforces the point that appropriate and detailed procedures must be in place and review (and practiced) regularly to insure the highest levels of uptime. In addition to the categories of errors described here, it would be prudent to review the particular application environment for additional potential sources of outage and develop appropriate plans.

TEST IT!
While discussing human error, it is worthwhile to mention an often over looked best practice for a true HA environment. This is the creation and maintenance of an adequate test environment that is capable of producing load representative of the production environment. As was discussed earlier, human error was the second largest cause of data loss in a survey of companies that experienced an outage. Along with automation, scripting and clear procedures, testing is key to reducing the chance that human failure will cause or extend an outage. Oracle considers maintaining an appropriate test environment a key best practice to assure continued High availability for any system. An example of a site where testing really paid off is Merrill Lynch. On September 11th, Merrill Lynch one of the world's leading financial management and advisory companies, who test critical applications for disaster readiness quarterly, where challenged with an previously unthinkable disaster, when terrorists struck the Twin Towers in New York. A core data processing center within two blocks of ground zero made a decision to redirect operations to a backup data center in the City of London, England. Within 11 minutes of making the decision, London was able to take over operations by activating their Oracle standby databases with no loss of data.

Paper 541

Database Administration

CAPTURING THE DEFINITIONS THE SERVICE LEVEL AGREEMENT (SLA)

Once the systems requirements are defined, the need to be documented and maintained in a fashion that is understood by all parties. An industry standard practice is to create a Service Level Agreement (SLA) that would clearly define the periods of uptime, allowed downtime for maintenance windows, and recovery time for each type of anticipated outage. In addition to defining the uptime requirements, the procedure for recovering based on the outage and command structure should also be defined. Issues such as who should be notified, and how they should be contacted should be clearly spelled out. Also, the issue of who should declare an outage and start the failover process should be defined. For a system with very high availability requirements, much of the outage window could be lost if there is not a clear decision to start the failover process or if that process is not completely automated. Also these procedures should define the exact steps for activating the failover plan. As many of those steps as possible should be scripted and tested in advance. These scripts and plans should be tested in advance of their activation in a real failure and should for the basis of an ongoing test methodology.

THE GOALS OF THE ORACLE SOLUTION

As a part of our continuing improvements to the Oracle product set in the areas of Manageability and availability Oracle has made improvements to our products with the following goals in mind: Prevent failures before they happen through tested and certified configurations Detect obvious and subtle failures quickly with fault detection agents Capture diagnostic information Resume services to end users quickly, and with minimal disruption Analyze failures offline to prevent re-occurrence

Once you have a common definition of uptime, and a common understanding of what any given systems uptime should be, plus an understanding of the errors that you may encounter, choose from the array of solutions that Oracle provides. The next section of this document will review some of the new or key High Availability features with these features in mind.

BASE FEATURES
The Oracle9i release of Data Guard has built upon some features in the core database server product that contribute directly to the potential availability of any system.

FEATURES THAT IMPROVE UPTIME

Fast Restart of the server using FAST_START_MTTR_TARGET. This parameter when set determines how long it would take an instance to recover in the event of a failure. The value is expressed in seconds. This parameter directly supports the SLA concept discussed earlier as it gives the DBA the ability to bound and plan for a consistent recovery time.

Paper 541

Database Administration

Dynamic Reconfiguration of initialization parameters using alter system command. These changes can be made persistent or limited to the life of a particular instance Online Redefinition This new package allows the DBA to modify the storage parameters for any given object, move table a table from one tablespace to another, partition an existing table recreate or re-organize a table or change structure the structure of an existing table by adding or dropping or redefining a column.

FEATURES THAT ADDRESS HUMAN ERRORS

Given Human Error is so commonly involved in data loss, Oracle9i has many features to address this exposure to a customers data. Data Recovery Performing a data recovery following an accidental update or deletion is often used to restore data to a previous state. However this can be time consuming when compared to other options mentioned here. Flashback Query End users can be empowered to recover their own data by using the ability of an Oracle database to store transaction undo information for many days which can be used to view a database a past point in time that the user chooses. LogMiner - LogMiner scans the log/logs it is interested in, and generates, using the dictionary file meta-data, a set of SQL statements which would have the same effect on the database as applying the corresponding redo record. It is important to note that LogMiner will not show the individual SQL statements in a log, but rather the end result statement that would need to be applied to bring the database to the final state for the range of records reviewed. A DBA can use this tool to undo accidental changes to a database. Oracle Data Guard A standby database maintained by Oracle Data Guard can have changes immediately shipped from a Production system, the application of changes can be delayed. This delay provides a window for an administrator to be aerated, and act upon to human errors before they are applied to the Standby database image.

RECOVERY MANAGER (RMAN)

Oracles Recovery Manager is Oracles strategic tool to manage backup and recovery operations. Recovery Manager provides a tightly integrated method for creating and managing backups, restoring and recovering the Oracle database. There are a number of significant benefits to using Recovery Manager. These are possible because the Oracle database server is managing the backup and restore operations. Automated database backup and recovery Allows backing up of the entire database or any logical unit such as the controlfile, datafile, tablespace or archive log files Incremental backups Backups can be performed while the database is open or closed Two types of backup: image copies or backup sets Intelligent archive log management for both backup and recovery

Paper 541

Database Administration

Tablespace Point-In-Time Recovery support Integration with the Oracle Enterprise Manager Backup Manager GUI Omission of empty blocks during backup for optimization No extra redo is generated during online backups reducing any potential for a performance penalty

RMAN greatly simplifies recovery, by automatically identifying the appropriate backups and archive logs that are required to recover the database.

BLOCK MEDIA RECOVERY (BMR)

The block media recovery feature is a technique for restoring and recovering an individual corrupted data block or set of corrupted data blocks within a datafile. When a small subset of blocks in the database require media recovery, it is more efficient to selectively restore and recover just those blocks. Only blocks being recovered need to be unavailable, allowing continuous availability of the rest of the database during recovery. Block media recovery provides two main benefits over filelevel recovery: lowering the Mean Time To Recover (MTTR) and allowing increased availability of data during media recovery. BMR will reduce the MTTR in situations where only a small subset of blocks in a database file in need of media recovery. Without block-level recovery, if even a single block is corrupt the administrator must restore a backup of the entire file and apply all redo changes generated for that file since that backup.

HARDWARE ASSISTED RESILIENT DATA (HARD) INITIATIVE

Oracle's Hardware Assisted Resilient Data Initiative is a comprehensive program designed to prevent data corruptions before they happen. Data corruption, while rare, can have a catastrophic effect on a database, and therefore a business. By implementing Oracle's data validation algorithms inside storage devices, Oracle will prevent corrupted data from being written to permanent storage. This type of end-to-end, high level software to low level hardware validation has never been implemented before. HARD will eliminate a large class of failures that the database industry has so far been powerless to prevent. RAID (Redundant Array of Independent Disks) has gained a wide following in the storage industry by ensuring the physical protection of data, HARD takes data protection to the next level by going beyond protecting physical bits to protecting business data. The classes of data corruption that Oracle plans to address with HARD include: Writes of physically and logically corrupt blocks Writes of blocks to incorrect locations Erroneous writes by non-Oracle programs to Oracle data Partially written blocks Writes from cluster nodes that are considered dead Lost Writes Corrupted third party backups

Paper 541

Database Administration

The HARD initiative includes several technologies that can be embedded in storage devices to prevent all these classes of corruption. Oracles storage partners will roll out these technologies over time.

STORAGE CONFIGURATION
Configuring the storage subsystem optimally for the database is a very important task for most system and database administrators. A poorly configured storage subsystem can result in I/O bottlenecks and reduced protection from device failures. The SAME configuration model offers a a scheme that addresses the availability and performance challenges administrators face. SAME, an acronym for Stripe and Mirror Everything has been developed by Oracle experts who have done significant work in researching optimal storage configuration for Oracle database systems. This model is based on four simple proposals (i) stripe all files across all disks using a 1 megabyte stripe width, (ii) mirror data for high availability, (iii) place frequently accessed data on the outside half of the disk drives, (iv) subset data by partition and not by disk.

STRIPE ALL FILES ACROSS ALL DISKS USING A 1 MEGABYTE STRIPE WIDTH
Striping all files across all disks ensures that full bandwidth of all the disks is available for any operation. This equalizes load across disk drives and eliminates hot-spots. Parallel execution and other I/O intensive operations do not get unnecessarily bottlenecked because of disk configuration. Since storage cannot be reconfigured without a great deal of effort, striping across all disks is the safest, most optimal option. Another benefit of striping across all disks is that it is very easy to manage. Administrators no longer need to move files around to reduce long disk queues, which frequently has been a non-trivial drain on administrative resources. The easiest way to do such striping is to use volume level striping using volume managers. Volume managers can stripe across hundreds of disks with negligible I/O overhead and are, therefore, the best available option at the present time. The recommendation of using a stripe size of one megabyte is based on transfer rates and throughputs of modern disks. If the stripe size is very small, the disk head has to move a lot more to access data. This means that more time is spent positioning the disk head on the data than in the actual transfer of data. It has been observed that a stripe size of one megabyte achieves reasonably good throughput and that larger stripe sizes produce only modest improvements. However, given the current trend in advances in disk technology the stripe size will have to be gradually increased.

MIRROR DATA FOR HIGH AVAILABILITY

Mirroring data at the storage subsystem level is the best way to avoid data loss. The only way to lose data in a mirrored environment is to lose multiple disks simultaneously. Given that current disk drives are highly reliable, simultaneous multiple disk failure is a very low probability event. In addition to disk mirroring, Oracle offers its own internal mirroring also. However, Oracle mirroring is limited to redo logs and control files only. It is a good practice to use Oracle mirroring on top of disk mirroring. This is because Oracle mirroring is more immune to logical data corruption. In the case of disk mirroring, logical data corruption in a redo log will be reflected on both the mirrors, while in the case of Oracle mirroring, data corruption in one redo log will likely not be duplicated on the mirror of that log. Therefore, both redo logs and control files should also be mirrored at the database level for high availability purposes.

Paper 541

Database Administration

PLACE FREQUENTLY ACCESSED DATA ON THE OUTSIDE HALF OF THE DISK DRIVES
The transfer rate of a disk drive varies for different portions of the disk. Outer sectors have a higher transfer rate than inner sectors. Also outer portions of a disk drive can store more data. For this reason, datafiles that are accessed more frequently should be placed on the outside half of the disk. Redo logs and archive logs can undergo significant I/O activity during updates and hence should also be placed on the outside portion of the disks. Since this can lead to an administrative overhead on the part of the database administrator, a simple solution is to leave the inside half of a disk drive empty. The is not as wasteful an option as might appear because due to the circular shape of the disks, typically more than 60% of capacity is on the outside half of the disk. Also the current trends of increasing disk drive capacities make it an even more viable proposition.

SUBSET DATA BY PARTITION AND NOT BY DISK

Sometimes it is necessary to separate or partition the overall data set. For instance, a database administrator may want to separate read only data from updated data on different volumes. In such cases, the partition should be created across all disks, and then concatenated and striped to form a separate volume. This separates the data files logically without compromising any of the advantages of the SAME methodology. The primary benefit of the SAME model is that it makes storage configuration very easy and suitable for all workloads. It works equally well for OLTP, data warehouse, and batch workloads. It eliminates I/O hot spots and maximizes bandwidth. Finally, the SAME model is not tied to any storage management solution in the market today and can be implemented with the technology available today. For more details on SAME methodology, please refer to Oracle technical paper Optimal Storage Configuration Made Easy at http://otn.oracle.com/deploy/availability.

ORACLE9I DATA GUARD

The goal of Data Guard is to maintain a real-time copy of a Production database to protect against corruptions, human errors, and disasters. In the event of a failure of the Production (or Primary) database, a Standby database can be activated to act as the new Primary database. Oracle9i Data Guard is the management, monitoring and automation software layer running above proven foundations of the Oracle9i Recovery and Standby database infrastructure components. Orace9i Data Guard - Standby database was first delivered as part of the initial release of the Oracle9i Database in the summer of 2001. A Standby database maintains one or more, bit-for-bit replicas of the Primary database it protects. A SQL maintained Standby database is an integral part of a future release of Oracle9i, and is used to maintain a logical replica of a Primary database. A Data Guard configuration is comprised of a collection of loosely connected systems, consisting of a single Primary database and a number of Standby databases, which can be a mix of both traditional Standbys and SQL Miantained Standby databases. The databases in a Data Guard configuration can be in the same data center (LAN attached) or geographically dispersed (over a WAN) and connected by Oracle Network Services. When OLTP users add data or change information stored within an Oracle database, the database buffers changes in memory pending a request to make those changes permanent (when an application or interactive user transaction issues a COMMIT statement). Before the COMMIT operation is acknowledged by the database, that record is written to database redo log files in the

Paper 541

Database Administration

form of a Redo Log record, which contains just enough information to redo the transaction in case of a database having to restart after system crash or during data recovery following data loss. When using a standby database, as transactions make changes to the Primary database, the Standby database is sent redo log data generated by changes, in addition to being logged locally. These changes are applied to the standby databases, which runs in managed recovery mode in the case of a traditional standby database or applied using SQL regenerated from archived log files. Whilst the Primary database is open and active, a traditional Standby database is either performing recovery (by applying logs) or open for reporting access. In the case of an SQL maintained type of Standby database, changes from the Primary database can be applied concurrently with end user access. The tables being maintained by SQL generated from a Primary database log will, of course be read only to users of the SQL Maintained Standby. These tables can have different indexes, and physical characteristics from their Primary database peers, but have to maintain logical consistency from an application access perspective, in order to fulfill their role as a Standby data source.

BENEFITS PROVIDED BY ORACLE9I DATA GUARD

A standby database configuration can be deployed for any database. This is possible because, its use is transpartent to applications, as no application code changes are required to accommodate a standby. Customers can tune the configuration to balance data protection levels, application performance impact, network transport method and cost variables. A standby database can be used to reduce downtime for planned outages such as operating system or hardware upgrades by performing a graceful role reversal or switchover operation between the primary and one of its standby databases. Some of the key benefits of Data Guard in this architecture are: Ease of Use Data Guard provides all the interfaces and tools required to create, manage and monitor a Standby database Automated Failover Data Guard can improve overall system availability by automating the failover process. Protection from human error Data Guard can delay the application of logs to a Standby database for a user configurable time period. This allows the DBA some breathing room in the event of a accident in the production database. Using the ability to stagger the application of changes to a standby database enables the protection of production data from data center disasters, user errors or corruptions, file system failures, or overwritten disk volumes. . If, for example, a drop table is issued against a production database the DBA can halt the application of logs against an object in the standby, and copy the deleted object back in to the production database by a variety of means.

Offline Reporting Capability A Standby database can be opened in a read only mode, allowing reporting work to be moved from the production objects to the standby instance. An SQL maintained Standby database allows changes from a Primary database to be applied at the same time as users are running queries against the Standby database. These capabilities make a standby database a popular high availability solution.

Paper 541

Database Administration

REAL APPLICATION CLUSTERS

Real Application Clusters harnesses the processing power of multiple, interconnected computers. Real Application Clusters software and a collection of hardware, known as a cluster, unite the processing power of each component to become a robust computing environment. In Real Application Clusters environments, all active instances can concurrently execute transactions against a shared database. Real Application Clusters coordinates each instances access to the shared data to provide consistency and integrity. Oracle Real Applications Clusters supports additional Oracle features that enhance recovery time and minimize the disruption of service to end-users. These features provide fast and bounded recovery, enforce primary node /secondary node access if desired, automatically reconnect failed sessions, and capture diagnostics data after a failure. In addition, Oracle Real Applications Clusters can be integrated with the cluster framework of the platform to provide enhanced monitoring for a wide range of failures, including those external to the database. Oracle Real Applications Clusters can recover more quickly from failures than traditional cold failover solutions. Because both instances are started an the database is concurrently mounted an opened by both instances, there is no need to fail over volume groups and file systems, because they are already available to all nodes as a requirement of Real Applications Clusters. There is also no need to start the Oracle instance, mount the database, and open all the data files. If there are many connections that must be reestablished after failover, this task can be time consuming. Oracle Real Applications Clusters supports pre-connections to the secondary instance. In this case, the memory structures needed to support a connection are established in advance, speeding reconnections after a failure. Lastly, since both instances are running, the cache can be warm in both instances. In an active/active configuration, the cache on both nodes is warmed automatically. Real Application Clusters Guard is built upon Real Applications Clusters running in a primary node/secondary node configuration. Much of the complexity in installation, configuration and management of the various components of traditional high availability solutions is avoided with the use of Real Application Clusters Guard. In this configuration, all connections to the database are through the primary node. The secondary node serves as a backup, ready to provide services should an outage at the primary occur. Also, Real Application Clusters Guard is tightly integrated with the cluster framework provided by the system's vendor to offering enhanced monitoring and failover for all types of outages. The result of this integration of the cluster framework with Real Applications Clusters is the look and feel of traditional cluster failover. This is important as it means the solution works with all types of third-party applications that are designed to work with a single Oracle instance. However, the Real Applications Cluster Guard solution provides much faster detection and bounded failover in the event of a failure. It also provides improved performance after failover thanks to pre-connected secondary connections provided by Transparent Application Failover (TAF) and a pre-warmed cache on the secondary node.

Paper 541

Database Administration

TRANSPARENT APPLICATION FAILOVER

Transparent Application Failover automatically restores some or all of the following elements associated with active database connections. It is this feature that truly allows an Oracle based system to present a user with the highest perceived level of availability. In any evaluation of availability, it is user perception that must be considered a key component. A user, presented with a clear message indicating a slight delay in processing, does not even need to know that a failover or recovery operation is occurring in the background. The follow features of TAF provide the DBA and developer with some tools for masking a failover from the user Basic TAF On a failure, a connection is started to a predetermined instance. Connection is automatic, however there is a time when transactions can not be processed by the client TAF Pre-connect On a failure, a connection to a predetermined instance is already opened (such as another node of a cluster or a standby database). The time that transactions cannot be processed is minimized and the cost of maintaining the connection and the available failover resource. OCI Failover Callback - On a failure, a client application call-out can be provided that will be executed after instance failure. This callout can be used at the application level to check for the presence of a failure and execute some failover logic (retry transaction, present a message to the user, etc.)

Paper 541

Database Administration

ORACLE9I EXAMPLE ARCHITECTURE

There is no single high availability and data protection architecture that fits every application deployment. What follows is an example architecture we are using to illustrate the benefits of implementing a comprehensive Oracle based high availability and data protection scheme. This example can be augmented using third party RAID and media management technology.

Example architecture providing system and data protection

While all these components add capabilities to the DBA and developers toolkit for building highly available solution, the beauty of these components if how they can be brought together to develop a highly availability architecture. Before tying these features together it is important to note a last few general concepts: Hardware any Oracle system is only as available as the underlying hardware components. While it is beyond the scope here to review the many solutions available for different hardware vendors, redundant disk and clustered systems are key to any HA solution Manageability - keep the DBAs free to focus on adding high-level business value through automation of mundane tasks such as shuffling disks for performance, which can be eliminated using SAME. Administrators should be using modern tools such as Enterprise Manager to ease or eliminate many tasks by using alerts, advisories, capacity planning and monitoring tools. Administrators should exploit manageability features such as automatic undo management and automatic space management within database files.
Paper 541

Database Administration

Automation/Testing/Training - There is no substitute for adequate preparation. Period. The best HA solution in the world will not be successful unless all the players know and understand their roles in the advent of a failure. Understand who will declare and outage, under what circumstance and the role of each member of the team is vital toward meeting any high service level.

BENEFITS OF EACH COMPONENT:

RAC - This is the keystone of our availability solution. Real Application Clusters provides the mechanism for multiple instances to be accessing the same physical database. Depending on our availability goals we have a few equally valid options. If we configure for active/active RAC, users could be accessing from all nodes equally. In the event of a failure in this case, affected users could reconnect to the surviving instance and continue processing without affecting users on other nodes. If we configure an active/passive scheme using Real Application Clusters Guard, we can still take advantage of all the benefits of multiple instances accessing the same database with additional monitoring capabilities to detect and manage the failover process Standby (local and remote) The standby database functionality provides multiple functions in this architecture. Running locally, it provides a robust backup solution, taking the place of the traditional hot backup. By applying the logs from the production database with a relatively short delay, a complete copy of the production database can be maintained locally, allowing for object level recovery as well as file-based recovery. Using a remote standby provides protection from a critical failure of production hardware. In addition, running standby databases allows us some flexibility in applying the logs, providing protection from human error as described earlier and providing a copy of the database for read only processing away from the production instance. RMAN In this architecture, RMAN provides us an Oracle aware method for completing our architecture with an offline copy of the database. In addition to automating this process, RMAN check the integrity of the backup as it is being taken, catching any corruptions at the time of the backup instead of the restore. This is crucial, because if this is the last line of defense for the database and the backup is corrupted, the database many not be recoverable. TAF Finally, using the facilities available with Transparent Application Failover, we can speed the application recovery time by defining in advance the resources available to an application in the advent of an outage. With the failover callback we can also abstract many potential failures from the user. HARD and SAME Using these features described above make a configuration more robust, ease administration while making optimal use of available disk storage.

SUMMARY
Oracle9i provides a broad array of features easing the challenges of maintaining high availability for any application. While have provided sample architecture and described its benefits, each application deployment will have its own unique HA requirements, constraints and infrastructure. We hope this discussion better arms the reader to more fully exploit the capabilities Oracle provides to achieve high application availability. More information on Oracle high availability projects can be found at the URL, http://otn.oracle.com/deploy/availability.

Paper 541