Professional Documents
Culture Documents
A Rough Guide To RAC: Julian Dyke Independent Consultant
A Rough Guide To RAC: Julian Dyke Independent Consultant
juliandyke.com
Agenda
juliandyke.com
Introduction
juliandyke.com
OIFCFG
VIP
CRSD
FAN EVMD ONS LMON ASM FCF GSD OCRCONFIG GES
CRS
AST
juliandyke.com
What is RAC?
Multiple instances running on separate servers (nodes) Single database on shared storage accessible to all nodes Instances exchange information over an interconnect network
Interconnect
Instance 2
Node 2
Shared Storage Local Disk
juliandyke.com
A RAC cluster includes one database one or more instances A database is a set of files Located on shared storage Contains all persistent resources An instance is a set of memory structures and processes Contain all temporal resources Can be started and stopped independently
juliandyke.com
Public Network
Instance 1
Instance 2
Instance 3
Instance 4
Node 1
Node 2
Node 3
Node 4
juliandyke.com
Located on shared storage accessible by all instances Includes Control Files Data Files Online Redo Logs Server Parameter File May optionally include Archived Redo Logs Backups Flashback Logs (Oracle 10.1 and above) Change Tracking Writer files (Oracle 10.1 and above)
juliandyke.com
Contents similar to single instance database except One redo thread per instance
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 51200K, GROUP 4 SIZE 51200K;
If using Automatic Undo Management also require one UNDO tablespace per instance
CREATE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 25600K AUTOEXTEND ON MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL;
Additional dynamic performance views (V$, GV$ but not X$) created by $ORACLE_HOME/rdbms/admin/catclust.sql
juliandyke.com
Instances communicate with each other over the interconnect (network) Information transferred between instances includes data blocks locks SCNs
Typically 1GB Ethernet UDP protocol Often teamed in pairs to avoid SPOFs Can also use Infiniband Fewer levels in stack
Other proprietary protocols are available
10
juliandyke.com
Mandatory for Database files Control files Online redo logs Server Parameter file (if used) Optional for Archived redo logs (recommended) Executables (Binaries) Password files Parameter files Network configuration files Administrative directories Alert Log Dump Files
11
juliandyke.com
Oracle supplied options Oracle Cluster File System (OCFS) Version 1 Windows and Linux Supports database and archived redo logs No executables Version 2 - August 2005 Linux, Windows and Solaris As OCFS1 plus executables Automatic Storage Management (ASM) Oracle 10.1 and above More transparent in Oracle 10.2 and above Both require underlying SAN or NAS Do not require LVM
12
juliandyke.com
Can use (continued) Network Attached Storage NFS-based Potentially lower cost - no fibre channel required Easy to administer Raw devices Difficult to administer Cannot be used with archived redo logs Third-party Cluster File System Still a popular choice with many sites Others (not supported) Firewire - maximum two nodes - recommended in 10g NBD - Network Block Devices - Solaris and Linux NFS - not supported, but might still work
13
juliandyke.com
Can install multiple copies of Oracle executables on local disks on each node Can also install Shared Oracle Home single copy of Oracle executables on shared storage Oracle 9.2 Only Oracle database software Oracle 10.1 Cluster Ready Services (CRS) Oracle database software + ASM Oracle 10.2 Oracle Clusterware (CRS) ASM Oracle database software
14
juliandyke.com
Global Resource Directory (GRD) Records current state and owner of each resource Contains convert and write queues Distributed across all instances in cluster Global Cache Services (GCS) Implements cache coherency for database Coordinates access to database blocks for instances Maintains GRD Global Enqueue Services (GES) Controls access to other resources (locks) including library cache dictionary cache
15
juliandyke.com
Background Processes
Each RAC instance has set of standard background processes e.g. PMON SMON LGWR DBWn ARCn RAC instances use additional background processes to support GCS and GES including LMON LCK0 LMDn LMSn DIAG
16
juliandyke.com
Portability
Most single-instance applications should port to RAC Some exceptions Application must scale well on single instance Can be difficult to evaluate Some features do not work e.g. DBMS_ALERT DBMS_PIPE External inputs/outputs may need modification Flat files etc Some RAC features require additional coding TAF Code may need upgrading to use RAC functionality e.g. FCF requires JDBC Implicit Connection Cache
17
juliandyke.com
Increasing availability Increasing scalability Improving maintainability Reduction in total cost of ownership
18
juliandyke.com
Realistic next step for experienced Oracle DBAs Intellectual challenge Job protection - ties organisation to Oracle technology Possible improved earnings It looks good on their CV
19
juliandyke.com
Availability
20
juliandyke.com
What is Failover?
If one node or instance fails Node detecting failure will Read redo log of failed instance from last checkpoint Apply redo to datafiles including undo segments (roll forward) Rollback uncommitted transactions Cluster is frozen during part of this process
Interconnect
Instance 1 Node 1
Instance 2 Node 2
21
juliandyke.com
Database Services are logical groups of sessions Can be configured using DBCA Enterprise Manager (10.2 and above) Can also be configured using SRVCTL (Oracle Cluster Registry only) SQL*Plus (Data Dictionary only) Text editor (Network Configuration)
In Oracle 10.1 and above, each service has Preferred Nodes (used by default) Available Nodes (used if preferred node fails)
22
juliandyke.com
Can be used with Resource Manager to control resource usage e.g. CPU Parallel execution Can be used for monitoring V$SERVICE_STATS Can be used for diagnostics DBMS_MONITOR trace statistics
23
juliandyke.com
Introduced in Oracle 10.1 (Cluster Ready Services - CRS) Renamed in Oracle 10.2 to Oracle Clusterware Cluster Manager providing Node membership services Global resource management High availability functions On Linux Configured in /etc/inittab Implemented using three daemons CRS - Cluster Ready Service CSS - Cluster Synchronization Service EVM - Event Manager In Oracle 10.2 includes High Availability framework Allows non-Oracle applications to be managed
24
juliandyke.com
Oracle Cluster Registry (OCR) Configuration information for Oracle Clusterware / CRS
25
juliandyke.com
Defines cluster resources including: Databases Instances RDBMS ASM Services Node Applications VIP ONS GSD Listener Process
26
juliandyke.com
Known as Quorum Disk / File in Oracle 9i Located on shared storage accessible to all instances Used to determine RAC instance membership In the event of node failure voting disk is used to determine which instance takes control of cluster Avoids split brain In Oracle 10.2 and above can be mirrored Odd number of copies (1, 3, 5 etc)
27
juliandyke.com
What is VIP?
Node application introduced in Oracle 10.1 Allows Virtual IP address to be defined for each node All applications connect using Virtual IP addresses If node fails Virtual IP address is automatically relocated to another node Only applies to newly connecting sessions
28
juliandyke.com
What is TAF?
TAF is Transparent Application Failover Sessions connected to a failed instance will be terminated Uncommitted transactions will be rolled back Sessions can be reconnected to another instance automatically if using TAF Can optionally re-execute in-progress SELECT statements Statement re-executed with same SCN Fetches resume at point of failure Session state is lost including Session parameters Package variables Class and ADT instantiations
29
juliandyke.com
What is TAF?
TAF is Transparent Application Failover Requires additional coding in client Requires configuration in TNSNAMES.ORA
RAC_FAILOVER = (DESCRIPTION = (ADDRESS_LIST = (FAILOVER = ON) (ADDRESS = (PROTOCOL = TCP)(HOST = node1)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = node2)(PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = RAC) (SERVER = DEDICATED) (FAILOVER_MODE =(TYPE=SELECT)(METHOD=BASIC)(RETRIES=30)(DELAY=5)) ) )
30
juliandyke.com
What is FAN?
Fast Application Notification (FAN) Introduced in Oracle 10.1 Method by which applications can be informed of changes in cluster status Handle node failures Workload balancing Applications must connect using services Can be notified using Server side callouts Fast Connection Failover (FCF) ONS API
31
juliandyke.com
What is ONS?
Oracle Notification Service (ONS) Introduced in Oracle 10.1 Allows out-of-band messages to be sent to Nodes in cluster Middle-tier application servers Clients Underlying mechanism for Fast Application Notification (FAN)
32
juliandyke.com
Depends on definition of availability May achieve less unplanned downtime May have more time to respond to failures Instance failover means any node can fail without total loss of service
Must provide have overcapacity in cluster to survive failover Additional Oracle and RAC licenses Load can be distributed over all running nodes Can use Grid to provision additional nodes
33
juliandyke.com
Can still get data corruptions Human errors / software errors Only one logical copy of data Only one logical copy of application / Oracle software Lots of possibility for human errors Power / network cabling / storage configuration Upgrades and patches are more complex Can upgrade software on subset of nodes If database is affected then still need downtime
34
juliandyke.com
Scalability
35
juliandyke.com
What is Scalability?
RAC overhead means that linear scalability is difficult to achieve Global Cache Services (blocks) Global Enqueue Services (locks) As number of instances increases, probability that instance is a resource master decreases
36
juliandyke.com
What is Scalability?
Scalability is the relationship between increments of resources and workloads Can be any resource but with RAC normally refers to adding instances Scalability can be linear - optimal but rare non-linear - suboptimal but normal
Workload
Workload
Linear
NonLinear
Resource
Resource
37
juliandyke.com
Balancing of workload across available instances Can have Client-side connection balancing Server-side connection balancing Client-side connection balancing Workload distributed randomly across nodes
RAC = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = node1)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = node2)(PORT = 1521)) (LOAD_BALANCE = ON) (FAILOVER = ON) (CONNECT_DATA = (SERVICE_NAME = RAC) (FAILOVER_MODE = (TYPE = SELECT)(METHOD = BASIC)) ) )
38
juliandyke.com
Server-side connection balancing Dependent on current workload on each node PMON monitors workload and updates listeners Depends on long or short connections In Oracle 10.1 Set PREFER_LEAST_LOADED_NODE in listener.ora OFF for long connections ON for short connections (default) In Oracle 10.2 Can specify load balancing goal for each service NONE, SERVICE_TIME or THROUGHPUT Can also specify connection load balancing goal SHORT or LONG
39
juliandyke.com
Increasing Scalability
If application scales well on a single-instance then it should scale well on RAC Eliminate contention Use sequences
Use locally partitioned tables and indexes Attempt to achieve node affinity Avoid contention for single blocks Distribute rows for hot blocks Small block size e.g. 2048 or 4096 ALTER TABLE MINIMIZE RECORDS PER BLOCK High PCTFREE / Low PCTUSED Filler columns e.g. CHAR (2000)
40
juliandyke.com
Increasing Scalability
Use larger block size for read-only objects Reduce number of GCS messages required
Minimize lock usage Eliminate unnecessary parsing Increase size of shared pool Bind variables Cursor sharing Use optimistic locking Eliminate unnecessary SELECT FOR UPDATE statements
41
juliandyke.com
Manageability
42
juliandyke.com
Introduced in Oracle 9.0.1 Must reside on shared storage Shared by all RAC instances Binary (not text) files Parameters can be changed using ALTER SYSTEM Can be backed up using the Recovery Manager (RMAN) Created using
CREATE SPFILE [ = SPFILE_NAME ] FROM PFILE [ = PFILE_NAME ];
43
juliandyke.com
Parameters
RAC uses same parameters as single-instance Some must be different on each instance Some must be same on each instance Can be global or local
[*.]<parameter_name> = <value> [<sid>]<parameter_name> = <value>
44
juliandyke.com
Parameters
45
Some parameters must be same on each instance including *: ACTIVE_INSTANCE_COUNT ARCHIVE_LAG_TARGET CLUSTER_DATABASE CONTROL_FILES DB_BLOCK_SIZE DB_DOMAIN DB_FILES DB_NAME DB_RECOVERY_FILE_DEST DB_RECOVERY_FILE_DEST_SIZE DB_UNIQUE_NAME MAX_COMMIT_PROPAGATION_DELAY TRACE_ENABLED UNDO_MANAGEMENT * Correct for Oracle 10.1
juliandyke.com
Parameters
Some parameters, if used, must be different on each instance including THREAD INSTANCE_NUMBER INSTANCE_NAME UNDO_TABLESPACE ROLLBACK_SEGMENTS DML_LOCKS must be identical on each instance if set to zero
46
juliandyke.com
DBCA
Can be used to Create RAC database and instances Create ASM instance Manage ASM instance (10.2) Add RAC instances Create RAC database templates structure only with data Create clone RAC database (10.2) Create, Manage and Drop Services Drop instances and database
47
juliandyke.com
What is SRVCTL?
Utility used to manage cluster database Configured in Oracle Cluster Registry (OCR) Controls Database Instance ASM Listener Node Applications Services Options include Start / Stop Enable / Disable Add / Delete Show current configuration Show current status
48
juliandyke.com
SRVCTL - Examples
49
juliandyke.com
Enterprise Manager
In Oracle 10.1 and above Database Control Installed by DBCA Controls single cluster Grid Control Uses separate repository Oracle 10.2 version available Requires Oracle 10.1 database
Fully supports RAC in both versions Except Oracle 10.1 cannot create / delete services Oracle 10.2 better interconnect performance monitoring
50
juliandyke.com
What is CLUVFY?
Introduced in Oracle 10.2 Supplied with Oracle Clusterware Can be downloaded from OTN (Linux and Windows) Written in Java - requires JRE (supplied)
51
juliandyke.com
CLUVFY
Stages include
-post hwos
-pre cfs -post cfs -pre crsinst -post crsinst -pre dbinst -pre dbcfg
52
juliandyke.com
CLUVFY
Components include
nodereach nodecon cfs ssa space sys clu clumgr ocr crs nodeapp admprv peer Checks reachability between nodes Checks node connectivity Checks CFS integrity Checks shared storage accessibility Checks space availability Checks minimum system requirements Checks cluster integrity Checks cluster manager integrity Checks OCR integrity Checks Oracle Clusterware (CRS) integrity Checks node applications exist Checks administrative privileges Compares properties with peers
53
juliandyke.com
CLUVFY
For example, to check configuration before installing Oracle Clusterware on node1 and node2 use:
Checks: node reachability user equivalence administrative privileges node connectivity shared stored accessibility If any checks fail append -verbose to display more information
54
juliandyke.com
Other Utilities
Additional RAC utilities and diagnostics include OCRCONFIG OCRCHECK OCRDUMP CRSCTL CRS_STAT
Additional RAC diagnostics can be obtained using ORADEBUG utility DUMP option LKDEBUG option Events
55
juliandyke.com
Advantages Fewer databases to manage Easier to monitor Easier to upgrade Easier to control resource allocation Resources can be shared between applications
Disadvantages Upgrades potentially more complex Downtime may affect more applications Requires more experienced operational staff Higher cost / harder to replace
56
juliandyke.com
57
juliandyke.com
Reduction in TCO?
Possible for sites with legacy systems Mainframes / Minicomputers Applications / Packages RAC option adds 50% to licence costs except for Users with site licences Standard edition (10.1+, max 4 CPU with ASM) Retrain existing staff or use dedicated staff Consolidation may bring economies of scale Monitoring Backups Disaster Recovery
58
juliandyke.com
Reduction in TCO?
Additional resources required Redundant hardware Nodes Network switches SAN fabric Hardware e.g. fibre channel cards
Reduction in hardware support costs May not require 24 hour support Viable to hold stock of spare components
59
juliandyke.com
Data Guard Physical Standby Introduced in Oracle 7.3.4 Stable, well proven technology Requires redundant hardware Implemented by many sites Can be used with RAC Logical Standby Introduced in Oracle 9.2 Still not widely adopted Streams Introduced in Oracle 9.2 Implemented by increasing number of sites Advanced Replication
60
juliandyke.com
Symmetric Multiprocessing (SMP) Systems Single Point of Failure Simplified configuration Eliminate RAC overhead Parallel systems For systems with deterministic input Messaging Data Warehouses Other Clustering Technologies SAN Operating System etc
61
juliandyke.com
Conclusion
Success of RAC deployments dependent on Application design and implementation Failover requirements IT infrastructure Flexibility and commitment of IT department(s) Before deploying RAC Investigate and reject alternatives Perform proof of concept Test application Evaluate benefits and costs Learn RAC concepts and administration Buy a good book :)
62
juliandyke.com
63
juliandyke.com