This action might not be possible to undo. Are you sure you want to continue?
ORACLE - From Dream To Production
BGOUG – Gabrovo 22.04.2005
,, Let someone k n o w ”
Describe the major steps in RAC implementation Show and explain the main problems and obstacles Show and explain solution and workarounds Give some practical ideas and variations Try to look at the problems from different angles Non-Goals Explain RAC concepts and fundamentals Show ALL aspects of RAC
Introduction RAC Installation High Availability Configuring Testing and Tuning RAC Implementing RAC in production
RAC Installation Problems Solutions .
High Availability.Problems Need to evaluate and purchase cluster hardware Need of knowledgeable OS. Network and Storage professionals All this takes time .
These configurations are not certified or supported by Oracle Support Services .Solutions RAC on Single Node RAC on Single VMware Node RAC on Multiple VMware Nodes RAC and Network Block Device Other Solutions All of the solutions presented here are for testing purposes ONLY.
0 with OCFS) Key Points No need of “fencing” configuration No need of using clustered file system or raw devices No need of multiple oracle homes (ORACLE_HOME) .Single Node (Oracle9i 9.2.1 .RAC on Single Node Why not? Metalink Note:241114.Step-By-Step Installation of RAC on Linux .
rac2. …) zyumix:/# su .oracle oracle@zyumix:~$ .RAC on Single Node (cont’d) Key Points (cont’d) One oracle user with 2 or more profiles one for every instance. rac2 oracle@zyumix:~$ echo $ORACLE_SID rac2 . (e.rac1.g. . rac1 oracle@zyumix:~$ echo $ORACLE_SID rac1 zyumix:/# su .oracle oracle@zyumix:~$ . .
Disadvantages .RAC on Single Node (cont’d) Key Points (cont’d) Oracle Universal Installer needs Clusterware software in order to install RAC option.
RAC on Single Node (cont’d) Client side load balancing Listener ‘A’ Server side load balancing Listener ‘B’ Server A Instance ‘A’ Server B Instance ‘B’ Database .
Linux (RHEL/SLES) and Oracle RAC (again on single node) VMware Workstation (90-day free eval. registration required) RHEL3 (not free) .html An easy. hands-on. step-by-step guide describing how-to install VMware.oracle.com/technology/tech/linux/vmware/cookbook/index.RAC on Single VMware Node Even easier !!! The Oracle-on-Linux VMware Cookbook - http://www. SLES8 (not free) Disadvantages .
You can create very complex environments – multiple NICs. disks etc. provided the disk in question is a SCSI disk.RAC on Multiple VMware Nodes VMware GSX/ESX Server permits the sharing of plain disks with multiple virtual machines running on the same host. This approach is very powerful but complex. Now there are a number of nodes although virtual Disadvantages . switches.
.Network Block Device (NBD) that maps remote files to local block devices (e. /dev/nb0) via TCP/IP network.g. One computer (not necessarily Linux machine) serves as data storage for all cluster nodes (Linux machines) instead of expensive disk array. In this solution.RAC and Network Block Device This solution allows you to build scalable and high available database system only with common Intel PCs connected into Ethernet network. a standard shared disk subsystem is replaced by a native Linux technology .
. which will reply with the data requested. The remote resource doesn't need to be a whole disk or even a partition. it will send a request to the server via TCP.RAC and Network Block Device (cont’d) With this thing compiled into the kernel. Every time the client computer wants to read /dev/nd0. It can be a file. Linux can use a remote server as one of its block devices.
RAC and Network Block Device (cont’d) Typical configuration Simple NBD configuration .
RAC and Network Block Device (cont’d) Installation Both client and server machines are with RHEL3 Download source from http://nbd./configure make make install .sourceforge.3.bz2 tar -xvf nbd-2.net/ As root do bunzip2 nbd-2.3 .3.tar.7.tar cd nbd-2.7.7.
01 count=300 bs=1M 300+0 records in 300+0 records out 314572800 bytes transferred in 1.683993 seconds (186801738 bytes/sec) [root@rac2 root]# Running NBD server Syntax: nbd-server <port> <filename> [root@rac2 root]# nbd-server 4101 /u01/oradata/rac/system.01 [root@rac2 root]# .RAC and Network Block Device (cont’d) Creating new empty files at NBD server [root@rac2 root]# dd if=/dev/zero of=/u01/oradata/rac/system.
rpm: V3 DSA signature: NOKEY..EL.i686.4.RAC and Network Block Device (cont’d) NBD client NBD client must be run as root (because of kernel parts of NBD).rpm warning: kernel-unsupported-2.i686. Before starting NBD client you would have to install Linux kernel NBD module Installing ndb module – RHEL3 [root@rac3 root]# rpm -Uvh kernel-unsupported-2.214.4. key ID db42a60e Preparing.EL.21-4.. ########################################### [100%] 1:kernel-unsupported ########################################### [100%] [root@rac3 root]# .
size = 307200KB bs=1024. sz=307200 [root@rac3 dev]# ..RAC and Network Block Device (cont’d) NBD client (cont’d) Loading nbd module [root@rac3 [root@rac3 [root@rac3 nbd [root@rac3 dev]# lsmod | grep nbd dev]# modprobe nbd dev]# lsmod | grep nbd 16388 0 (unused) dev]# running nbd client Syntax: nbd-client <data server> <port> /dev/nb<n> [root@rac3 dev]# nbd-client rac2 4101 /dev/nb0 Negotiation: .
Oracle Real application clusters need raw access to shared disk subsystem so mapping raw devices to block devices is needed. This could by done with standard raw command.RAC and Network Block Device (cont’d) Now block devices are configured and it is possible to access remote data. minor 0 [root@rac3 root]# . Syntax: raw /dev/raw/raw<N> /dev/<blockdev> [root@rac3 root]# raw /dev/raw/raw1 /dev/nb0 /dev/raw/raw1: bound to major 43.
html RAC and NFS Locking Caching Write through cache .com/technology/pub/articles/hunter_rac10g.Other Solutions RAC and FireWire Build Your Own Oracle RAC 10g Cluster on Linux and FireWire http://www.oracle.
HA Configuration .
Simplified RAC schema NET Database .
HA System .
Maximum Availability Architecture Application Server Application Server Dedicated Network Data Guard Primary Site Secondary Site .
Extended Distance Clusters Application Server Application Server RAC Dedicated Network Virtualization storage layer What about the Quorum Server??? Primary Site Secondary Site .
cluster_database_instances = 2 .active_instance_count = 1 *.Extended Distance Clusters (cont’d) Resolving Distance Problems Application partitioning gc_files_to_locks Wavelength Division Multiplexing Dense Wavelength Division Multiplexing DWDM Coarse Wavelength Division Multiplexing CWDM ACTIVE_INSTANCE_COUNT *.
Testing and Tuning RAC Introduction RAC testing steps Functional Application Tests RAC High Availability tests Scalability tests Digging into RAC performance problems .
Always tune single instance first! Specific RAC issues RAC aware tools .Introduction Testing isn’t trivial !!! Classical testing/tuning methods.
RAC testing steps Functional Application Tests RAC High Availability tests Be aware about the timeouts!!! Scalability tests .
RAC testing steps (cont’d) Scalability tests (cont’d) Patterns of application scalability nearly static linear exponential constrained m ofr e P r r es u e no f o Load ( # users. size of tables) .
RAC testing steps (cont’d) Scalability tests (cont’d) Good scalability 7000 6000 5000 4000 3000 2000 2 node RAC Single node na m ofr e P r 1000 0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100 # of concurrent users .
RAC testing steps (cont’d) Scalability tests (cont’d) Problem!!! 7000 6000 5000 4000 3000 2000 Possible disk bottleneck 2 node RAC Single node na m ofr e P r 1000 0 4 8 # of concurrent users 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100 .
RAC testing steps (cont’d) Scalability tests (cont’d) Problem!!! Possible interconnect bottleneck 10000 9000 8000 7000 6000 5000 4000 3000 Single node 2 node RAC na m ofr e P r 2000 1000 0 4 8 # of concurrent users 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100 .
AVG CR BLOCK RECEIVE TIME should typically be about 15 milliseconds depending -.on your system configuration and volume .Digging into RAC performance problems Interconnect and shared storage are the two most possible performance problem areas in RAC Interconnect “speed” Throughput Latency average latency of a consistent block request.
5 CPU overhead Bandwidth MB/sec low Memory Channel 3 low Myrinet Sun SCI Gigabit Hyper Fabric Infiniband Ethernet 7 to 9 low ~250 10 N/A ~ 70 100 high 100 20 low 400 < 10 low 1000 > 500 > 100 .Digging into RAC performance problems (cont’d) Interconnect types Measurement SMP BUS Latency (μs) 0.
an interconnect failure that is normally unnoticeable would instead cause an Oracle cluster failure as Oracle still attempts to access the network interface which has gone down. When you set CLUSTER_INTERCONNECTS in cluster configurations. In other words. the interconnect high availability features are not available.Digging into RAC performance problems (cont’d) cluster_interconnects parameter It provides Oracle with information about additional cluster interconnects available for use and can be used to load balance the interconnect traffic to different physical interconnects thus increasing interconnect bandwith. .
sql) .1 Script to Collect RAC Diagnostic Information (racdiag.Digging into RAC performance problems (cont’d) STATSPACK reports The STATSPACK report show statistics ONLY for the node or instance on which it was run Top 5 Timed Events Global Cache Service and Global Enqueue Service Note:135714.
47 PL/SQL lock timer 51 105 8.69 GOOD PERFORMANCE Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time --------------------------------------.92 latch free 13.969 666 51.35 db file sequential read 64.-----------.-------global cache cr request 34.23 CPU time 149 11.568 958 31.214 455 14.28 buffer busy waits 15.542 453 14.163 96 7.----------.44 buffer busy global CR 6.05 db file sequential read 25.971 295 9.513 620 20.379 159 12.Digging into RAC performance problems (cont’d) BAD PERFORMANCE Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time --------------------------------------.36 .-----------.88 buffer busy waits 10.-------latch free 10.----------.
2 11.4 0.8 51.0 GOOD 2.0 0.0 2.4 0.9 0.1 0.6 0.3 0.Digging into RAC performance problems (cont’d) Global Cache Service .0 0.2 0.7 0.7 0.8 1.0 .7 1.Workload Characteristics ----------------------------------------------Ave global cache get time (ms): Ave global cache convert time (ms): Ave build time for CR block (ms): Ave flush time for CR block (ms): Ave send time for CR block (ms): Ave time to process CR block request (ms): Ave receive time for CR block (ms): Ave pin time for current block (ms): Ave flush time for current block (ms): Ave send time for current block (ms): Ave time to process current block request (ms): Ave receive time for current block (ms): Global cache hit ratio: Ratio of current block defers: % of messages sent for buffer gets: % of remote buffer gets: Ratio of I/O for coherence: Ratio of local vs remote work: Ratio of fusion vs physical writes: BAD 11.7 1.1 0.5 0.4 9.2 0.3 7.0 0.0 6.9 1.4 0.3 4.3 33.0 0.2 0.2 0.2 0.2 4.5 0.5 3.2 0.
Digging into RAC performance problems (cont’d) Global Enqueue Service Statistics --------------------------------Ave global lock get time (ms): Ave global lock convert time (ms): Ratio of global lock gets vs global lock releases: GCS and GES Messaging statistics -------------------------------Ave message sent queue time (ms): Ave message sent queue time on ksxp (ms): Ave message received queue time (ms): Ave GCS message process time (ms): Ave GES message process time (ms): % of direct sent messages: % of indirect sent messages: % of flow controlled messages: BAD 0.1 16.6 0.4 1.0 63.3 0.9 0.2 GOOD 0.0 1.1 49.0 1.3 0.1 0.7 50.5 0.0 0.1 0.2 0.0 .4 36.5 29.0 1.7 2.
Implementing RAC in production Smooth transition from single instance to RAC Change ORACLE_HOME Relinking the RAC Option ON/OFF CLUSTER_DATABASE = TRUE/FALSE Start/Stop the second instance Start gradual movement of clients from one instance to another .
Relinking the RAC Option 1.mk ioracle . Login as the Oracle software owner and shutdown all database instances on all nodes in the cluster. 2.mk rac_on (rac_off) If this step did not fail with fatal errors then proceed to step 4. make -f ins_rdbms. make -f ins_rdbms. cd $ORACLE_HOME/rdbms/lib 3. 4.
com/technology/pub/articles/hunter_rac10g.sql) .cz/~kripac/orac-nbd/ The Oracle-on-Linux VMware Cookbook http://www.Relinking the RAC Option Metalink Note:183340.1 Frequently Asked Questions About the CLUSTER_INTERCONNECTS Parameter in 9i.html Build Your Own Oracle RAC 10g Cluster on Linux and FireWire http://www.Reference Metalink Note:211177.oracle.1 RAC Survival Kit: Rac On / Rac Off .com/technology/tech/linux/vmware/cookbook/index. http://www.oracle.muni.html Note:135714.1 Script to Collect RAC Diagnostic Information (racdiag.fi.
Thank You firstname.lastname@example.org .
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.