Introduction to the choice of implementing Oracle RAC Oracle’s Real Application Clusters (RAC) product is a great temptation for

DBAs and businesses alike. What could be better than 24x7 availability, true scalability, rock-bottom hardware prices due to commodity servers, high performance, and maximum user concurrency? It all sounds like a miracle piece of software, well worth the extra cost and implementation. However, there are things you need to know about RAC before you commit your business to using it. We will divide these concerns into three main sections: • • • 100% Uptime Performance Scalability

For each of these sections, there are definite pros and cons. This presentation aims at providing a non-biased view based on years of experience with Oracle RAC to help you choose the best path for your business.

We need RAC to stay competitive! Find out what RAC is and report back immediately. What is Oracle RAC? Perhaps you heard about RAC from an Oracle sales sheet and were captivated by the wide range of benefits covered. Or maybe you heard about it from a colleague or an article in InfoWorld, where you saw another business implemented it and cut their downtime right to the 0.00001% mark. Wherever you hear about RAC, you usually only receive a single viewpoint. Some people really love it because of the benefits they’ve gotten, though at a cost, where others absolutely hate it because of the trouble it has caused them. I have worked with companies that have felt both ways. Before we actually delve into all these pros and cons to gain a whole view of a RAC implementation, let’s take a look at what RAC is from a high-level perspective. The Oracle RAC System Though we all refer to our implementations of Oracle as “the database,” a complete Oracle system is actually formed of two parts: the database and the instance. Component 1: The Database The database is simply our files on disk. An Oracle database consists of three specific required file types. • • Datafiles Control Files

The reason we call them groups is that there can be mirrored copies of the online redo log files in each group. Many times. It keeps track of the current state of the datafiles and redo logs. Oracle will write redo to the first log. Control Files in RAC The Control File is the record keeper of the Oracle Database. All data that is inserted. but they are all copies of each other. this means that Oracle can recover from a crash without the DBA having to do anything other than just telling the database to startup. . they are regularly recorded in the online redo logs. A tablespace is a logical disk area that Oracle objects such as tables and indexes can be stored in. Objects are further broken down into extents and blocks. but this is beyond the scope of this explanation. Each of these individual online redo logs is known as an online redo log group. If you lose a control file. the object is placed logically in the tablespace and physically in a data file. Each redo log group can have one or more members. Oracle will switch to the second log and write the same redo. one instance) you may have multiple control files. and re-apply lost transactions back into the database. just like you might record a movie on your VCR. and when the first log is full. These files are physically stored on disk resources. In a standard Oracle system (one database. and the database itself. there are two tablespaces that are absolutely required: SYSTEM and SYSAUX. In Oracle 10g and beyond. As changes occur. Oracle can “replay” the saved transactions in the redo logs. archive logs. Redo Logs in RAC clusters Think of redo logs as a tape recorder that records every change in the Oracle database. or deleted will make its way to the datafiles (eventually) once they are committed. At a minimum. Component 2: The Instance The Oracle Instance is the actual runtime aspect of Oracle. When a DBA or developer creates an object. the instance will crash until a recovery of some sort is performed. updated. These files make up what are known as tablespaces.• Redo Logs Datafiles in RAC Oracle Datafiles are the final storage location of our data. Like control files. The instance is made up of: • • Binary Processes RAM Memory Binary Processes in RAC Oracle actually runs as five critical and required binary processes that are activated when you start your instance. Oracle requires that you have two redo logs assigned to the database. This is known as multiplexing. it’s a good idea to have multiplexed copies of the redo logs. Each copy of a redo log file within a log group is called a “redo log member”. Also like VCRs. Oracle’s Control File is a required file.

though these must be viewable by all nodes. • DBWR – Database Writer. single database system. Data is flushed from this pool via DBWR to the datafiles. or systems. In a single instance environment. when the buffer is 1/3rd full. this results in downtime. DBWR is used to write blocks to datafiles (transition from instance to database) • • LGWR – Log Writer. Each instance in the “cluster” will have its own SGA (RAM areas) and binary processes. Also stores data written with inserts. • Shared Pool – Caches the means by which SQL can be executed. When SQL is run. called an execution plan. If any of these processes fail. SMON is primarily used to recover a crashed instance. • Log Buffer – Stores change data to be flushed to the current redo log file.EXE. LGWR writes redo information to the redo log files. This cluster interconnect allows each node of the RAC cluster to share cached data located in the buffer cache with any other node on the cluster. A RAC environment uses something called Cache Fusion to bring all the instances in the cluster together. In a RAC environment. however. in the cluster. Each instance has its own Buffer Cache. CKPT – Checkpoint. Oracle fuses these caches together into a single global buffer cache. as we saw in the previous section. The SGA is broken down into pools where data can be temporarily stored before being discarded. or DML). if the execution plan is cached in the shared pool. They will also have their own control files and redo log files.The Process Monitor. it must be parsed. . Cache Fusion for RAC RAC provides us a multiple instance. and deletes (called Data Manipulation Language. RAM Memory and RAC Oracle stores data in RAM in an area called the System Global Area (SGA). the parse phase is sped up considerably. are: • Buffer Cache – Stores cached blocks of data from Oracle datafiles when queried. Please note that on Windows.• SMON – The System Monitor. I will explain this in a moment. updates. or flushed to disk. These pools. Note that the Buffer Cache is very important for RAC. there is one shared set of datafiles. This occurs over a private network called a private or cluster interconnect. or memory areas. when it reaches 1MB. • PMON . on checkpoint. overwritten. CKPT assists in keeping all files in sync. the entire instance of Oracle crashes. or when required by DBWR. PMON cleans up dead processes and registers network services for the instance. these five separate processes are threaded under a single process called ORACLE. every three seconds. Flushing occurs every commit.

they can build up. acres and acres of spare room. think Oklahoma. love. All connections that would have pointed to Instance 2 (and in some cases connections that were already pointing at instance 2) will fail over Scalability and RAC There are two ways to scale your hardware: horizontally and vertically.Figure 1. but it definitely beats going to disk for it! High Availability and RAC RAC also gives us the benefit of High Availability. RAM is much faster than disk. We add CPUs. they do not need to build taller buildings. There is a lot of land available. or a fatal error of some sort on the system. they cannot build new buildings. In this case. If instance 2 above crashes. and therefore have the skyscrapers we all know. taller. the data will be cached in Instance 1’s Buffer Cache. RAM. For example. Now notice that Instance 2 runs a query that wants a row that Instance 1 already has cached. due to a power plug being kicked loose. When new developments are needed. However. Once this query has been executed and fetched. think of Manhattan. We all know about vertical scaling. If Instance 1 were to require any of this data again. etc until the system we are on is full. A simple view of Cache Fusion at Work Notice in the image above that Instance 1 (server 1) queries the centralized storage to find all employees between 1 and 10. as the case may be). Instance 2 would receive the data over the high speed network interconnect using Cache Fusion. they build out. and so the query would return much quicker. Is That It? . and sometimes avoid! Scaling horizontally is the practice of adding new systems to the cluster. This RAM to RAM transfer over the network isn’t as fast as local RAM. scaling upon the horizontal plane (or plains. There is no more room on the horizontal plane. we build up. Instead. To visualize scaling vertically. for instance. Instance 1 will take over the user load. it would have to look no further than local RAM.

Unplanned Downtime and RAC databases Unplanned downtime was mentioned above. Networks must be set up just so to allow data to transfer freely from node to node. and millions of dollars lost. Applications.No. Complex locking mechanisms must be in place to make sure data is reliable and secure. a systems administrator kills a required process such as SMON) • • • Oracle Internal errors Hackers Localized disasters (coffee spill on the new Sun server) Planned Downtime and RAC Planned downtime is more graceful than unplanned of course. expert systems. they all require a steady stream of data to keep them alive. and can happen because of some of the most simple or unexpected issues. of course RAC does much more internally. Not all downtime is “bad. Some examples of events causing unplanned downtime: • • • Power failure Overheating server room Kernel panic • Fat fingered mistake (for instance. DSS. It makes downtime more bearable by providing a multiple nodes to connect to. and allow your business to continue. but a crashed database could end lives. There is software called Clusterware that must make the bridge between the nodes. without a single second of downtime. it can and will cause massive amounts of error. possible data corruption. but in some ways can be worse than .” Downtime comes in two categories: planned and unplanned. or servers. Disks must be set up properly in order to allow this shared storage. reporting. of the cluster. your business requires data to stay above ground. Oracle RAC is a High Availability (HA) system. three nodes will take over immediately. analytics. with the hundreds of planes in the air at all times? Or if the database powering a just in time provider of organs for transplant were to suddenly crash because the janitor pulled the plug? It sounds incredibly dramatic. and thus. Imagine if the data powering the FAA’s air traffic control systems was suddenly lost. others can be more horrible still. And though this seems like a horrible loss. However. What Does RAC Do For My Business? The primary goal of RAC can be summed up in a single word: Uptime. It can last from seconds to hours in extreme situations. Uptime and Oracle RAC Data drives business. If you have a four node RAC cluster and a single node crashes. There is much more than this diagram to a fully functional RAC system. and is generally regarded as the worst type. it provides us enough meat to start talking about the pros and cons of using Oracle 10g RAC. If a bank loses its core transaction database for even a single hour.

Work can progress in a “rolling” fashion. it can actually decrease costs by decreasing hardware requirements. multiple power sources. we can use 4 16 CPU systems instead of a single 64 CPU server. In a RAC environment. What Could Possibly Go Wrong? After the previous section. RAC allows us to connect multiple low cost machines together in order to provide the same capability of a single large system. Whereas a single instance database can use Direct Attached Storage (DAS). multiple network cards. but this usually requires expensive software to manage. They usually end up costing hundreds of thousands of dollars. Some developers and administrators want daily maintenance periods. you must use some sort of shared storage device. other vendors. ranging from Host Bus Adapters (HBA) to the SAN itself. RAC has its drawbacks as well. Implementation of Oracle RAC RAC is a complex system to implement. For each server. RAC alleviates these issues by allowing you to have a single server down at a time. capable of connecting to many servers. The concurrent processes will be balanced across all the nodes of your cluster. We’ve all seen spreadsheets for new project implementations where we list all the new hardware we will need to purchase. you will want multiple Host Bus Adapters. Yet another cost is the network connection. but the CPUs are at only 50% capacity. etc. The multiple HBA cards are used in case a single one fails. usually through fibre-channel connections. thereby allowing your operation to remain online. it could require frequent restarts in order to keep things updated. We have all seen the requests for huge multi-processor systems that are upgradeable to somewhere around 128 gigabytes of RAM and over 90 CPUs. and even run into the range of millions. most administrators require redundancy within each server as well.unplanned downtime. which can cause planned downtime to be the bulk of your total downtime. Earlier we learned that the RAC system requires a . from networking to disk drives to Clusterware to Oracle itself. you are wasting half your CPUs. which is an array of inexpensive disks connected to a single server. and consultants may mention that RAC is good because of the price. with the added benefit of high availability. We will probably save money using the lower-cost hardware. For instance. a single system may have underutilized resources. from Implementation up to Usage. Most companies I have worked with require a consultant to come in to help plan their move to RAC and for the actual installation itself. There are many different pieces to the RAC environment. Depending on the software on the server. Scalability and RAC Oracle. there are some costly disk requirements. In addition. In order to implement a RAC system. and double the hardware equals double the cost. an added cost per CPU on top of what you are already paying for Oracle. and it can get very costly. you must now use what is known as a Storage Area Network (SAN). Redundancy can also be costly. you may be thinking “Where do I sign up?” It’s not that simple. If the system is waiting on a RAM resource. On top of that. Even though you have multiple servers to fail over to. whereas our 64 CPU system may be maxed out. This requires a unique set of hardware. and will therefore have a better chance to use otherwise unclaimed resources. and now we can add new servers if we run out of capacity. where one server at a time comes down. This means doubling up on hardware. we can utilize every server to the max. Though at first glance it seems expensive. A SAN is much more expensive.

high bandwidth with low latency. interconnect latency. Setting up and administering the hardware mentioned in the previous section on Implementation is no small task! Network Administrators will have to learn how to work with the new interconnect. Remember also that humans are fast becoming the most expensive part of the IT environment. listeners. and troubleshooting for clusters. Heck. However.cluster interconnect in order to accommodate RAM-to-RAM transfers of data blocks. consulting. In addition. Another note on usage comes from the architecture of RAC as a whole. and possible even training and consulting. and dealing with employees with some great new marks on their resumes! Usage for RAC Thankfully. multiple levels of training may be required. or to purchase other means of disk storage such as Solid State Disk. your volume manager or filesystem of choice. by the time you’re done. or a 3rd Party CFS). and more in-depth metrics such as cache coherency. In addition. they’re still not as fast as a local RAM read. Though RAC does provide horizontal scalability. if your cluster interconnect cannot handle the traffic. DBAs that are RAC proficient are usually better paid. While this does not sound like much. If you use a specialized interconnect such as Infiniband. The only way around this issue is to change your entire application to accommodate RAC. requires more implementation. The DBA must keep everything in the RAC environment monitored. and running perfectly. disk times from multiple systems. Of all the staff. They will have to understand how to set up and administer Clusterware. RAC learning for DBAs & System Administrators There is a definite learning curve when it comes to RAC. System Administrators will have to learn how to work with the disk resources. up to date. it is possible for more things to go wrong. With so many components. and the administrator will have to assist in setup. the RAC specific features of Oracle. The DBA must monitor the cluster. it’s nice. lots of trial and error. but it’s not always a surefire winner. and even a little bit of “miracle work” at times. With hardware costs falling on a daily basis while manpower costs remain the same. OCFS. it makes up many days of training. training and consulting may be required. it costs more money. Remember the Cache Fusion component we learned about in the last section? Well. you may pay a hefty fee for the administration of this complex environment. Oracle’s goal is to provide transparency for all users. Complex SAN environments such as EMC and NetApp can require training of their own. While RAM-to-RAM transfers over the network are indeed faster than reading from disk. you the manager may require some training in dealing with setting up training sessions. Because of all the different components that make up a RAC environment. you may need more DBAs than you previously did to keep everything in top notch shape. the database. You may notice key queries slowing down where they . While tools such as Grid Control help perform this monitoring. extra servers will actually degrade your performance instead of helping it. all instances. This interconnect must be very fast. DBAs will have the greatest learning curve. but are very expensive. it behaves much like a normal database. Oracle RAC will only function when using specific disk setups (ASM. the shared disk setup. RAW. and many other things. once a RAC system has been implemented. so no one ever knows they’re even touching a complex RAC environment. Interconnects such as Infiniband and Myrinet can accommodate this. this does not apply to the DBA. ASM or OCFS if they’re in use.

. zero data loss is allowed. costs can range from dollars per minute to tens-of-thousands of dollars lost for every minute the database is unavailable. In fact. it must protect us from varying levels of unplanned downtime ranging from single server outages (which RAC covers) to entire data center loss (which RAC does not cover). it does not account for all possible problems. it could be that your performance hits rock bottom. Some businesses choose not to follow all the guidelines for maximum availability. and it performs well. and can only be protected against with extensive (and usually expensive) disaster recovery methods. however. We have learned about instance failure. In addition. these tools must protect us from planned and unplanned downtime. Yes. Downtime can be expensive. resulting in the loss or corruption of data. usually as the result of some sort of natural disaster. most database professionals find it easier to tune a single instance system than a RAC environment. Data failure is the worst of the three we have seen thus far (instance and system failure). due to the lower level of complexity and resources required for management. However. No. RAC only makes up one piece of the MAA. Lastly. A hurricane. for a system that allows direct access by the end user. This is by far the worst unplanneddowntime scenario. An advertising company may allow hours of downtime. corruption can occur if hardware or software bugs result in inappropriate data being written to the datafiles. which is roughly the same as server failure. it may be that hours or even days of data could be reloaded. if excessive disks are lost it is possible that production data could be lost as well. High Availability. flood. all of which make up the Maximum Availability Architecture (MAA). requiring some form of recovery. but it’s not the end all be all of performance. In this case. If you bog down the interconnect with too many nodes. this time may come sooner than you think. Data Center Loss occurs when a system is completely lost. remember that all data will be in centralized storage. RAC is scalable. for instance. There is still a possibility of data failure or data center loss. Disaster Recovery. RAC protects us against this issue by providing multiple servers to which we may connect. or tornado may destroy or seriously disable an entire data center resulting in a combined loss of servers and disk. As we have seen in the previous section. However. In addition. Even then. When considering a high availability strategy. we learned in the last section on Implementation that the interconnect MUST be very fast with low latency in order to sustain your RAC cluster. RPO defines the allowable data loss if a failure occurs. and the disk mirror will provide no protection.used to be lightning fast due to the application pointing at varying nodes of the cluster. if a disk is mirrored with hardware or software RAID. a bank will usually allow no downtime whatsoever. If batch processes load our data. Some disk failures are non-disastrous. User error can also cause data loss if an operating system user removes database files with a command such as rm. The MAA provides us with redundancy on all components and employs different Oracle tools. the file will be removed. such as an online store or ATM machine. Oracle provides many options for preventing downtime and data loss. the DBA must consider: • • • • Recovery Time Objective (RTO) Recovery Point Objective (RPO) Downtime Cost-per-Minute Available Resources The RTO defines the allowable downtime for the database. Depending on the system.

each using parallel query.However. RAC only provides support for part of the availability spectrum. In addition. Using nologging with create index can make index rebuilding up to 30% faster. you can submit many index rebuild jobs simultaneously. it will be in the form of employees. 4. Parallel index rebuild jobs: Also. . The only danger with using nologging is that you must re-run the create index syntax if you perform a roll-forward database recovery. and other little mentioned components of a RAC system. consultants. SSD: Some shops will use temporary solid-state disk to speed-up the initial index writes and move the index to platter disk storage at a later time. sort of a parallel parallelism for fast index rebuilds.scan. On a large server you can simultaneously rebuild dozens of indexes. as we have seen here. 2. uptime is expensive as well. but you can also enjoy scalability with lower priced hardware. it is your credibility if you jump into a project without having a full view of its possible repercussions. Question: I have to rebuild several very large indexes and I need to know the fastest way to rebuild the index. In the previous sections we’ve talked about how costly RAC can be for your business. Partition the index: If you have purchased the partitioning option. you can rebuild a local partitioned index faster than a single large index. The cost will not only be in licensing. But do not forget that these things come with a cost. property of Oracle Corporation) Conclusion RAC provides businesses with some outstanding benefits. NOLOGGING: You can also use the NOLOGGING option for super-fast index rebuilding. and this full-scan can be parallelized according to your cpu_count. hardware. remember that while your employees are hopefully top notch and know what they are It is important for managers to understand these concepts before embarking on the RAC quest. you have several tuning options to make it faster: 1. 3. PARALLEL Scan: Start an index rebuild with a full. remember that if you have the spare CPU and the indexes are on different disks. Not only can you be much closer to 100% uptime. now we see that even more may be required for a fully bulletproof system. Figure 2: Example of an HA Configuration using MAA Best Practices (Oracle. What are the performance options for a fast index rebuild? Answer: When Oracle rebuilds an index. training. software. 5. Other costs will have to be endured to become fully bulletproof. and possibly even a higher user load.

Sign up to vote on this title
UsefulNot useful