<Insert Picture Here>

Oracle Real Application Clusters
M.Thenappan
Oracle Advance Customer support Singapore

Agenda

• • • • •

Oracle 11gR2 Grid Infrastructure Oracle Clusterware Network Configuration Managing Oracle Clusterware Files with ASM Installation of Oracle Grid Infrastructure

Availability versus Downtime
• Customers demand high availability, what does it mean • Oracle has the tools to build it • The Maximum Availability Architecture describes it
Percentage Availability
95% 99% 99.9% 99.99% 99.999% Downtime Per Year (7x24x365)

Days
18 3 0 0 0

Hours
6 15 8 0 0

Minutes
0 36 46 53 5

99.9999%

0

0

1

Single Instance Failure

Server presents a single point of failure
Server 1 Single Instance „X‟

Database „A‟

Clusters with „cold‟ Failover

Server 1

Server 2 Instance „X‟

Database „A‟

Examples include MSCS, Unix Cluster software

Approaches to Database Clustering

Shared Nothing
Style of clustering promoted by some vendors

Shared Everything
Style of clustering delivered by Oracle Real Application Clusters.

Shared Everything
• A single database is shared among multiple “instances”
– No concept of per instance data ownership: full sharing – Utilizes inter-node cache synchronization for near linear scaleability for all types of applications: OLTP, DSS, CRM and ERP – Provides global read/write access to data: no partitioning – Allows for cluster aware installation and management

Real Application Clusters

Benefits • Highest Availability • On-demand flexible scalability • Lower computing costs • World record performance

Database

Storage

Oracle Real Application Clusters

• Unlimited scaleability
– More Computers = More users at the same speed

• Runs ALL Applications Unchanged
Application Application

Names A-Z

Names A-Z

Oracle Real Application Clusters

• Unlimited Reliability
– More Computers = More Reliable Application

Oracle Real Application Cluster

Real Application Clusters
Virtualizes server resources
SALES
HR ERP

• • • •

Runs all Oracle database applications Highly available and scalable Adapts to changes in workloads Complementary with virtual machine (VM) technologies

© 2009 Oracle Corporation

Consolidate onto the Grid
Application Servers

Database Servers Management

Storage
© 2009 Oracle Corporation – Proprietary and Confidential

Traditional HA and Disaster Tolerance
Expensive, idle redundancy

Production Server

Idle Failover Server

Idle Disaster Recovery

Solaris Cluster HP ServiceGuard IBM HACMP

BMC SQL Backtrack

Veritas Volume Manager

© 2009 Oracle Corporation – Proprietary and Confidential

Oracle Maximum Availability Architecture
Fully Utilizing Redundancy

Secure Backups Add/Remove to Cloud and Tape Nodes and CPUs
Data Guard

Active Data Guard

Add/Remove Fast Recovery Area Storage

Online Production Testing

© 2009 Oracle Corporation – Proprietary and Confidential

ASM & RAC, Understand the Architecture
public network

VIP1

VIP2

VIP3

Node1
ASM instance 1 Database instance 1 CRS
cluster interconnect

Node 2
ASM instance 2 Database instance 2 CRS
cluster interconnect

Node3
ASM instance 3 Database instance 3 CRS

...

Operating System

Operating System

Operating System

shared storage

{

Managed by ASM

{

redo logs all instances database & control files

Raw devices
Cluster filesystem

OCR & voting disk
(oracle_home)

Oracle GI

Grid Computing

•Sized for peak load •Difficult to Scale •Expensive to Manage

•Pools of shared resources •Re-distribute resources as needed •Cost efficient

Oracle Grid Infrastructure
• Oracle Grid Infrastructure (OGI) is a combined home of
– Oracle Clusterware – Oracle Automatic Storage Management (ASM)

OGI provides infrastructure software (storage management, cluster software), typically managed by System Administrators

• •

There is only one active version of the OGI on a system OGI comes in two versions:
1. Grid Infrastructure for a Cluster – Includes Oracle Clusterware, ASM 2. Grid Infrastructure for a Standalone Server – Includes Oracle Restart, ASM

Grid: Oracle Clusterware
• Major part of the Oracle Grid Infrastructure • Basis for Oracle Real Application Clusters (RAC) • Integrated with Oracle Automatic Storage Management (ASM) and is the basis for the Oracle ASM Cluster File System (ACFS) • Provides a cluster infrastructure for all kind of applications (e.g. App A & B)

Oracle Clusterware

Grid: Oracle Real Application Clusters
• Scale workloads across multiple low cost servers • Consolidate into fewer servers and databases • Runs all Oracle database applications • Built-in HA to support mission critical workloads • World record performance

RAC Oracle Clusterware

Grid: Automatic Storage Management
• Easier to manage than file systems • Reduces storage costs • Provides best performance New with • Stores all data 11.2

RAC Oracle Clusterware

Automatic Storage Management

Grid: Oracle Enterprise Manager
• Reduce operational costs
– With intelligent diagnostics and automation – Reduce database management costs by 40% – Reduce configuration management effort by 90% – Reduce patching and provisioning effort by up to 98%

Oracle Enterprise Manager

• Manage applications top-down
– From business and end-user perspective – Avoid online revenue losses up to 25% – Maximize staff productivity by 10 times or more

RAC Oracle Clusterware

• Manage entire application lifecycle
– With Application Quality Mgmt and Compliance Solutions – Reduce testing effort by up to 80% – Increase test coverage by 95%

Automatic Storage Management

Oracle Clusterware

High Availability Clusters

• High Availability Clusters are implemented primarily for the purpose of improving the availability of services which the cluster provides. • They operate by having redundant nodes, which are then used to provide service when system components fail. • HA cluster implementations attempt to manage the redundancy inherent in a cluster to eliminate single points of failure.

http://en.wikipedia.org/wiki/Computer_cluster

Benefits of a cluster

• • • •

Scaleability for applications Using lower-cost hardware Ability to fail over Ability to grow the capacity over time by adding servers, when needed

The need for Clusterware

• Clusterware…
– Joins the single nodes into a cluster of systems – Decides which process runs on which node – Automates management looking at – Starting, Stopping, Monitoring, Failover – Provides information to the applications (events) – Is capable of handling dependencies

Without clusterware, there is no cluster possible

Examples of Available Clusterware

• IBM
– HACMP

• HP
– MC-Service Guard

• Sun
– Veritas Cluster – Sun Clusterware

• Third party clusterware cost money • Oracle Clusterware working with Third Party clusterware only supported with Enterprise Edition

Oracle Clusterware

• Oracle‟s H/A and management framework for clusters and nodes can replace vendors framework
– – – – – – automates start, stop, and monitoring automates failover, relocation, restart automates events for notification and end to end HA maintains strong and optional dependencies applies policies using the global status of the system requires no external system for lights-out operation

Clusterware and Real Application Clusters
• Real Application Clusters is a database option • A supported clusterware is needed to run Real Application Clusters
• Oracle Clusterware is available on all Oracle platforms and is free when using RAC • Clusterware always needs to be installed, even when using the third party clusterware
– Depending on the platform, some functionality is handed over to the third party clusterware.

The Goals for Oracle Clusterware 11.2
• Introduce Oracle Clusterware as a cluster infrastructure software
– Tightly integrated into the Oracle RAC stack – Providing additional benefits in cluster environments

• Customer demanded improvements were made in the following areas:
– Easier Installation – Easier Management – High Availability

• No other third party cluster solution is required when using Oracle Clusterware

What is Oracle Clusterware?
• Oracle Clusterware is
• • • • • a major part of the Oracle Grid Infrastructure (OGI) integrated with Oracle Automatic Storage Management (ASM) the foundation for the Oracle ASM Cluster File System (ACFS) the foundation for Oracle Real Application Clusters (RAC) a cluster infrastructure for all kind of applications

Node 1

Node 2

Node ...
Protected App A

Node n
Protected App B

Oracle RAC

Oracle ASM / ACFS Oracle Clusterware

Consolidated Pool of Storage with Automatic Storage Management (ASM)

Oracle Clusterware requirements

• All nodes must have access common cluster aware shared storage locations
– Location(s) for Oracle Cluster Registry (OCR) – Location(s) for Oracle Cluster Voting Disk

• All nodes need an additional IP address that is not used in the system (VIP) • All nodes must have a secondary network with at least 1 Gigabit network (faster is supported)
– Same network can also be used for Oracle RAC – Should be a separate network from the public interface

• All nodes must run the same Operating System
– Rule of thumb: Must be installed using the same CD/DVD

OCR – Oracle Cluster Registry

• Cluster-wide storage of configuration data
– Raw partition or flat file on CFS or NAS

• • • • •

Looks similar to LDAP or MS Registry Hierarchical Supports links Enforces a security model w/permissions C and Java APIs, thread safe and kernel callable

Clusterware requires a Voting disk

• Determines the sanity of the cluster • Used together with the Heartbeat over the Cluster Interconnect network • Last resort if heartbeat network fails • Every node touches the voting disk every second • Minimum of 1 voting disk (=file) is needed • 1 Voting disk is single point of failure

Clusterware Voting disk

“Only the nodes that can access the majority of the voting disks will survive”
• Therefore 2 voting disks are still a single point of failure • Use 3 or more voting disks when using a single storage area • Use an odd (3,5,7 etc) number of voting disks if the disks are over multiple storage area‟s and distribute • Voting disks can be added on-line

<Insert Picture Here>

Network

Network Interconnect

• Private connection between the cluster nodes
– – – – – – – Separate network subnet – usually non-public Typically also used as RAC Interconnect Node specific IP address Not to be shared with public network Separate switch or domain Minimum 1Gb No DNS entries needed

Clusterware comes with a VIP

• VIP (Virtual IP address)
– Requires an additional IP adress that is currently not in use – Requires an IP address in the same range as the public IP address – Is used to facilitate node failure faster by the client – Is node specific (so every node needs a VIP address)

What is a VIP ?
• Each node has a Public IP • Each node has a Virtual IP • The client connects to database using the VIP
• Every times the client „needs‟ something from the server, an IP request is send
Node 1 Public = 80.80.80.101 VIP = 80.80.80.201 Node 2 Public = 80.80.80.102 VIP = 80.80.80.202

What is a VIP ?
• If the server fails, the client is not notified • Client will continue to send requests until the „tcp-timeout‟ has been reached • Tcp-timeout can be up to an 30 minutes on certain platforms
• The VIP is there to prevent the „TCP-timeout‟
Node 1 Public = 80.80.80.101 VIP = 80.80.80.201 Node 2 Public = 80.80.80.102 VIP = 80.80.80.202

What is a VIP ?
• After detection of a node failure by the cluster, cluster ware will failover the VIP to another node.
• Any IP request now made to the database through the VIP will be terminated so the client is informed of the node failure.

Node 1 Public = 80.80.80.101

• Client can now failover, invalidate a pool or give a message to the user.

Node 2 Public = 80.80.80.102 VIP = 80.80.80.202 VIP = 80.80.80.201

What is a VIP ?
• After the failed node has returned to the cluster, the VIP is returned to the node • New connections can be made again using the VIP

Node 1 Public = 80.80.80.101 VIP = 80.80.80.201

Node 2 Public = 80.80.80.102 VIP = 80.80.80.202

<Insert Picture Here>

SCAN

Single Client Access Name (SCAN)
SCAN
Back Office Front Office LOB

Free

• Used by clients to connect to any database in the cluster • Removes the requirement to change the client connection if cluster changes • Load balances across the instances providing a service • Provides failover between “moved instances”

Network Requirements for SCAN
Two options:
1. Define the SCAN in your corporate DNS (Domain Name Service) sales1-scan.example.com IN A 133.22.67.194 IN A 133.22.67.193 IN A 133.22.67.192

2. Use the Grid Naming Service (GNS) and the SCAN will be created during cluster configuration Note: For a test environment, you can install with SCAN resolving to one IP in /etc/hosts if no DNS

Single Client Access Name (SCAN)

• Allows clients to use EZConnect or simple JDBC connections • Each cluster will have 3 scan listeners, each having a scanvip defined as cluster resources on network 1 • A SCAN VIP/LISTENER will failover to another node in cluster
sqlplus system/manager@sales1-scan:1521/oltp jdbc:oracle:thin:@sales1-scan:1521/oltp

Database Settings for SCAN
• Instance registers with local listener on its node • Instance registers with SCAN listeners through “remote_listener”
– Should be SCAN:PORT (not HOST=SCAN in TNSNAMES.ORA)

• Client must be able to resolve the node VIPs for connection re-direction from SCAN
remote_listener=‘scan:1521' local_listener=‘hostname-vip:1521'

Oracle Clusterware SCAN Resources
• Created by root script during Grid Infrastructure Installation $srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521 SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521 SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521

$srvctl config scan
SCAN SCAN SCAN SCAN name: sales1-scan, Network: 1/192.87.58.0/255.255.255.0/ VIP name: scan1, IP: /sales1-scan.mycompany.com/192.87.58.143 VIP name: scan2, IP: /sales1-scan.mycompany.com/192.87.58.99 VIP name: scan3, IP: /sales1-scan.mycompany.com/192.87.58.100

Oracle Database 11g Release 2
Single Client Access Name (SCAN)

App Servers Front Office Back Office

DW

• Used by clients to connect to any database in the cluster • Removes the requirement to change the client connection if cluster changes • Load balances across the instances providing a service • Provides failover between “moved instances”

© 2009 Oracle Corporation – Proprietary and Confidential

mycluster.myco.com

RAC RAC RAC RAC RAC RAC RAC RAC RAC RAC RAC RAC

ONE A ONE B ONE C ONE D ONE E ONE F ONE G ONE H ONE I ONE J ONE K ONE L

Free

Single Client Access Name (SCAN)

• SCAN listener is the load balancer for the cluster. • Allows clients to use EZConnect or simple JDBC connections • Instance registers with local listener on its node • Database “remote_listener” registers instances with all SCAN listeners
sqlplus system/manager@ sales1-scan:1521/oltp jdbc:oracle:thin:@sales1-scan:1521/oltp

Connection Load Balancing

plication Server

Oracle RAC Database

SCAN Listeners
Clients

Local Listeners

SQL*NET and SCAN
• SQL*NET will retrieve the IP addresses from DNS or GNS, it will then load balance and failover across the IP addresses. • For MAA implementations, if client uses both primary and standby in address list, SQL*NET will retrieve all 6 IPs, it will then load balance and failover across the 6 IP addresses
sales.mycompany.com =(DESCRIPTION= (CONNECT_TIMEOUT=10)(RETRY_COUNT=3) (ADDRESS_LIST= (LOAD_BALANCE=on)(FAILOVER=ON) (ADDRESS=(PROTOCOL=tcp)(HOST=scan1)(PORT=1521)) (ADDRESS=(PROTOCOL=tcp)(HOST=scan2)(PORT=1521))) (CONNECT_DATA=
(SERVICE_NAME= sales.mycompany.com)))

SQL*NET and SCAN
• If load_balance is OFF, SQL*NET will retrieve the IP addresses for the SCAN and sequentially go through the list. • For EZConnect, Load_balance is OFF
sales.mycompany.com =(DESCRIPTION=(ADDRESS_LIST= (LOAD_BALANCE=OFF)(FAILOVER=ON) (ADDRESS=(PROTOCOL=tcp)(HOST=scan1)(PORT=1521)) (CONNECT_DATA= (SERVICE_NAME= sales.mycompany.com)))

Load Balancing Advisory

• Load Balancing Advisory is an advisory for balancing work across Oracle RAC instances. • Load balancing advice
– Is available to ALL applications that send work. – Directs work to where services are executing well and resources are available. – Adjusts distribution for different power nodes, different priority and shape workloads, changing demand. – Stops sending work to slow, hung, failed nodes early.

• Client connection pool is integrated with Oracle RAC load balancing advisory • When application does “getConnection”, the connection given is the one that will provide the best service.

<Insert Picture Here>

Managing Oracle Clusterware Files with ASM

OCR/Vote in ASM

• OCR and Voting Disks can now be stored in ASM • Grid Infrastructure Install will create the diskgroup • Best Practice is to use the same diskgroup as Database
– COMPATIBLE.ASM must be 11.2.0.0

• Disk discovery string stored in GPnP profile • Cannot stop ASM (unless stop the cluster) if OCR/Vote in ASM
– Stopping ASM will cause the database instance to get restarted, ASM will not stop

Oracle Cluster Registry in ASM

• OCR stored similar to any database file • Redundancy defined by the diskgroup (same as database files)
– Normal – 2 copies – High – 3 copies

• Only 1 OCR in a diskgroup • Best Practice is to have at least 2 OCR (one in each diskgroup)

Manage the OCR in ASM
• The OCR is managed like a datafile in ASM (new type)
– It adheres completely to the redundancy settings for the DG

Voting Disks in ASM
• Voting Disks are created on specific disks and CSS knows their location • Number of voting disks depends on the redundancy chosen for the diskgroup
– External: one (1) Voting Disk – Normal: three (3) Voting Disks – High: five (5) Voting Disks

• By default each disk in a diskgroup is in its own failure group
– Diskgroup must contain enough failure groups (disks) to create each voting disk in a separate failure group

• The „simple majority rule‟ remains:
– Each node needs to see (v/2)+1 (with v= #Voting Files) Voting Disks in order not to be rebooted (Check: every second)

What if... I have 2 Storage Arrays
• Use Quorum Failgroup to manage the 3rd Voting disk SQL> CREATE DISKGROUP TEST NORMAL REDUNDANCY FAILGROUP fg1 DISK ‘<a disk in SAN1>' FAILGROUP fg2 DISK '<a disk in SAN2>' QUORUM FAILGROUP fg3 DISK '<another disk or file>' ATTRIBUTE 'compatible.asm' = '11.2.0.0.0'; Note: “<another disk or file>” can be placed on e.g. NFS • Configuration: 1. Create any ASM diskgroup during installation 2. After the installation create a new diskgroup as shown above 3. [GRID]> crsctl replace votedisk +TEST

Backup of Clusterware Files
Fully Automatic Backups • The Voting Disks are backed up into the OCR • Any configuration change in the cluster (e.g. node addition) triggers a new backup of the Voting Files. • A single, failed Voting Disks is restored by ASM automatically within a Disk Group – no action required • Note: Do not use DD to back up the Voting Disks anymore!
• The OCR is backed up automatically every 4 hours • Manual Backups can be taken as required

• ONLY IF all Voting Disks are corrupted or failed AND (all copies of) the OCR are also corrupted or unavailable THEN manual intervention would be required

Will the Cluster Reboot if the ASM Instance dies?
• NO! • One possible Node Failure:
– If the ASM instance fails on the node that is the OCR Master, that node may reboot

Finding the Status of the Voting Disks
• • • • • > crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----------------------------- --------1. ONLINE 7c54856e98474f61bf349401e7c9fb95 (/dev/sdb1) [DATA] Located 1 voting disk(s).

OCR / Voting Disk In Oracle ASM

Next step

No RAW device support (in OUI) anymore – upgrades support RAW devices.

Manage the OCR in ASM

The OCR is managed like a datafile in ASM (new type)
It adheres completely to the redundancy settings for the DG

Manage Voting Disks – Placement in ASM
• Unlike the OCR, Voting Files are
• Stored in selected ASM disks • „crsctl‟ used to specify a disk group for Voting Disks • ASM auto creates 1/3/5 Voting Files
• • • • based on Ext/Normal/High redundancy and on Failure Groups in the Disk Group Per default there is one failure group per disk New failure group type: Quorum Failgroup

• The „simple majority rule‟ remains:
• Each node needs to see (v/2)+1 (with v= #Voting Files) Voting Disks in order not to be rebooted (Check: every secon

Backup of Clusterware Files – Fully Automatic
• Clusterware Files managed in ASM enables fully Automatic Backups: • The Voting Disks are backed up into the OCR • Any configuration change in the cluster (e.g. node addition) triggers a new backup of the Voting Files. • A single, failed Voting Disks is restored by ASM automatically within a Disk Group – no action required • Note: Do not use DD to back up the Voting Disks anymore!

• The OCR is backed up automatically every 4 hours • Manual Backups can be taken as required
• ONLY IF all Voting Disks are corrupted or failed AND (all copies of) the OCR are also corrupted or unavailable THEN manual interference would be required – the rest is automatic.

<Insert Picture Here>

Managing Your Oracle RAC 11g Release 2 Database

Two Management Styles for Oracle RAC
• Administrator Managed
– Specifically define where the database should run with a list of servers – Define where services should run within the database

• Policy Managed
– Define resource requirements of workload – Enough instances are started to support workload requirements – Goal: To remove hard coding of a service to a specific instance or node

Server Pool
• Logical division of the cluster into pools of servers • Applications (including databases) run in one or more server pools • Managed by crsctl (applications), srvctl (Oracle) • Defined by 3 attributes (min, max, importance) or a defined list of nodes
– Min- minimum number of servers (default 0) – Max – maximum number of servers (default 0 or -1) – Importance – 0 (least important) to 1000

srvctl modify serverpool –g <name> –u <max>

Server Pool Example – Instance View
public network

VIP1

VIP2

VIP3

VIP4

Scan_LSNR

Scan_LSNR

GNS Listener instance 1 ASM
Oracle Clusterware
Operating System

Scan_LSNR Listener instance 2 ASM
Oracle Clusterware
Operating System

Listener
instance 1 ASM
Oracle Clusterware
Operating System

Listener
instance 2 ASM
Oracle Clusterware
Operating System

cluster interconnect

backoffice
Min 1 Max 2Imp 3

shared storage

frontoffice
Min 1 Max 2 Imp 4 Redo / Archive logs all instances Database / Control files

Managed by ASM

OCR and Voting Disks

Cluster Managed Services with Server Pools
• One to one mapping
– IE a service can only run in one server pool

• Services are uniform (run on all instances in the pool) or singleton (runs on only one instance in the pool)

Server Assignment

Servers are assigned in the following order:
1. Generic server pool 2. User assigned server pool 3. Free

Oracle Clusterware uses importance of server pool to determine order
1. Fill all server pools in order of importance until they meet their minimum 2. Fill all server pools in order of importance until they meet their maximum 3. By Default any left over go into FREE

Server Assignment
Cluster Management via Server Pools
Back Office Min 1 Max 3 Imp 3

Front Office Min 2 Max 3 Imp 4

LOB
Min 0 Max 2 Imp 2

Free

Server Assignment
Cluster Management via Server Pools
Back Office Min 1 Max 3 Imp 3

Front Office Min 2 Max 3 Imp 4

LOB
Min 0 Max 2 Imp 2

Free

Server Assignment
Cluster Management via Server Pools
Back Office Min 1 Max 3 Imp 3

Front Office Min 2 Max 3 Imp 4

LOB
Min 0 Max 2 Imp 2

Free

Cluster Reconfiguration

If a server leaves the cluster Oracle Clusterware may move servers from one server pool to another only if a server pool falls below its min. It chooses the server to move from
– A server pool that is less important – A server from a server pool of the same importance which has more servers than its min

Oracle Clusterware will only move servers if you have non-default values for min, importance

Oracle Database 11g Release 2
Cluster Management via Server Pools
Back Office Min 3 Max 3 Imp 2

Front Office Min 1 Max 3 Imp 4

LOB
Min 0 Max 2 Imp 2

Oracle Database 11g Release 2
Cluster Management via Server Pools
Back Office Min 3 Max 3 Imp 2

Front Office Min 1 Max 3 Imp 4

LOB
Min 0 Max 2 Imp 2

Oracle Database 11g Release 2
Cluster Management via Server Pools
Back Office

Front Office

LOB
Min 0 Max 2 Imp 2

Min 1 Max 3 Imp3

Min 2 Max 3 Imp 4

Oracle Database 11g Release 2
Cluster Management via Server Pools
Back Office

Front Office

LOB
Min 0 Max 2 Imp 2

Min 1 Max 3 Imp3

Min 2 Max 3 Imp 4

Oracle Database 11g Release 2
Cluster Management via Server Pools
Back Office

Front Office

LOB
Min 0 Max 2 Imp 2

Min 1 Max 3 Imp3

Min 2 Max 3 Imp 4

GENERIC Server Pool
• Used to model databases that are “Administrator Managed”. i.e. “The OLD WAY!”
– Parent of all server pools for “Administrator Managed” databases

• • • •

Always exists but may be of size 0 Used for upgrade from 10g or 11g Release 1 Managed by srvctl managing the database Servers in Generic are named (hosting member in cluster resource)

Administrator Managed and Policy Managed
Cluster Management via Server Pools

GENERIC
Back Office 11g R1

Front Office 10g R2

LOB 11g R2
Min 0 Max 2 Imp 2

Notes for Policy-Managed Databases
• SIDs are DYNAMIC
– DBA scripts may have to be adjusted – Environment variables settings in profiles will have to check the current sid – Directories for LOG files will change over restarts

• “Pin” Nodes
– Forces Oracle Clusterware to maintain the same Node number when restarted (which maintains SID) – Automatically done on upgrades – Required when running pre-11g Release 2 versions in the cluster – On upgrade, the nodes will be “pinned”

crsctl pin css -n nodename

Installation of Grid Infrastructure

• Grid Infrastructure e works together with the Operating System • The nodes need to be „ready‟
– – – – Hardware OS components OS setup Oracle user environment setup

• Oracle Manuals provide excellent step-by-step setup of the hardware and OS

Information needed for Grid Infrastructure Install
• • • • • $CRS_HOME location Hostnames VIP hostnames SCAN Used network interfaces
– Public Network – Private Network

• Location for Cluster registry
– Single location means “External redundancy” – Two locations means “Normal redundancy”

• Location for Voting Disk
– Single location means “External redundancy” – Three locations means “Normal redundancy”

Cluster Verification Utility (cluvfy)

• Allows customers to verify cluster during various stages of its deployment from hardware setup, CRS Install, DB install, add node etc. • Extensible framework • Non-intrusive verification • Command Line only
– cluvfy comp peer -n $MYNODES | more

• Does not take any corrective action following the failure of a verification task

cluvfy Stage Checks

• RAC deployment is logically divided into several operational phases, we call them “stages” • Each stage comprises of a set of operations during RAC deployment
• Each stage has its own set of entry (pre-checks) and/or exit (post-checks) criteria

Cluster Verification Utility
-pre dbcfg
Configures RAC DB

-pre dbinst
Installs RAC

-pre crsinst
Installs

Oracle Clusterware

-pre cfs
Sets up OCFS ( OPT )

-post crsinst

-post cfs
User sets up the
Hardware,

network & storage

-post hwos

$> ./cluvfy stage

cluvfy Stage List

• $> ./cluvfy stage -list

• Valid stage options and stage names are:
-post -pre -post -pre -post -pre -pre hwos cfs cfs crsinst crsinst dbinst dbcfg : : : : : : : post-check for hardware & operating system pre-check for CFS setup post-check for CFS setup pre-check for CRS installation post-check for CRS installation pre-check for database installation pre-check for database configuration

cluvfy Component Checks

• An individual sub-system or a module of the RAC cluster is known as a “Component” in cluvfy • Availability, integrity, liveliness, sanity or any other specific behaviour of a cluster component can be verified.
• Components could be simple like a specific storage device, or complex like the CRS stack, involving a number of sub-components like CRSD, EVMD, CSSD and OCR.

cluvfy Component List

• $> ./cluvfy comp -list • Valid components are:
– – – – – – – – – – – – – nodereach nodecon cfs ssa space sys clu clumgr ocr crs nodeapp admprv peer : checks reachability between nodes : checks node connectivity : checks CFS integrity : checks shared storage accessibility : checks space availability : checks minimum system requirements : checks cluster integrity : checks cluster manager integrity : checks OCR integrity : checks CRS integrity : checks node applications existence : checks administrative privileges : compares properties with peers

Deployment of cluvfy

• Install only on local node
– Tool deploys itself on remote nodes during execution, as required

Local Node

RAC

Challenges for Oracle on Clustered Configurations
• Inter-Instance co-ordination
– ensure consistency BETWEEN instances – permit the Oracle locking to perform in the same way with multiple instances as it does with one – permit the database to survive the failure of a subset of instances or ALL instances

• • • •

Offer comparative or better performance than SMPs Offer increased scaleability Must be highly available Oracle on a cluster must not be significantly more difficult to manage than on an SMP

Cache Fusion Architecture
• Full Cache Fusion
– Cache-to-cache data shipping – Shared cache eliminates slow I/O – Enhanced IPC

Users

• Allows flexible and transparent deployment

Full Cache Fusion
• Oracle Cache Fusion increases performance and scaleability
– Data is shipped directly over high speed interconnect – Minimize disk I/O

Node A

Node B

Data Transfer
Database buffers Database buffers

Request

Database

Cache Fusion Manages Inter Instance Block Requests
• Readers and writers accessing instance A gain access to blocks in instance B‟s buffer cache All types of block access Coordination by Global Cache Service Request for Block Cache A Block Status in Cache B

• •

Read Read Write

Read Write Read

Cache Fusion Components

• Cache fusion implemented through
– Global Cache Service (GCS) – Global Enqueue Service (GES)

• Lock management is fully automatic • Enhanced Inter Process Communication
– CRS / CSS: TCP – Cache Fusion: UDP

Single system command: srvctl
• Start/stop the database and all enables instances srvctl start|stop|status database -d prod • • Start the instance PROD2 srvctl start instance -d prod -i PROD2
• Display the current status srvctl status|config database -d prod

• Start/stop VIP, GSD, TNS listener etc. srvctl start|stop|status nodeapps -n <host> You can do a lot with srvctl, see manual for more info

ADDM for Oracle RAC

Database-Level ADDM 11g
Self-Diagnostic Engine

New in 11g

Performance expert in a box Identify the most “Globally Significant” performance problems for the entire Oracle RAC database Database-wide analysis of:
– – Global cache interconnect issues Global resource contention, e.g. IO bandwidth, hot blocks Globally high-load SQL Skew in instance response times

Instance-Level ADDM

Inst 1 AWR 1

Inst 2

– –

Inst 3

AWR 2

AWR 3

Runs proactively every hour when taking AWR snapshots (default)

Q U E S T I O N S A N S W E R S

Sign up to vote on this title
UsefulNot useful