Understanding Oracle RAC Internals

1/20/2012
1
1 Copyright 2011, Oracle and/or its affiliates. All rights reserved.
Understanding Oracle RAC Internals
Markus Michalewicz (Markus.Michalewicz@oracle.com)
Senior Principal Product Manager Oracle RAC and Oracle RAC One Node
1/20/2012
2
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features
or functionality described for Oracles products remains at the sole discretion of Oracle.

Agenda
The Oracle RAC Architecture
VIPs and Networks
Listeners and SCAN
and Services
Client Connectivity
Node Membership
The Interconnect
Installation and Upgrade
1/20/2012
3
Typical hardware architecture
Interconnect
with switch
Public Lan
SAN switch
Client
Shared Storage
Network
Node 1 Node 2 Node N Node N-1
Typical software deployment 11g Release 2
Shared Storage
OS OS OS OS
Oracle Grid Infrastructure
Oracle Real Application Clusters (RAC)
1/20/2012
4
Typical software deployment 11g Release 2
Oracle ASM managed
Shared Storage
OS OS OS OS
Oracle Automatic Storage Management (ASM)
Oracle Clusterware
Oracle Real Application Clusters (RAC)
Oracle RAC 11g Release 2 per node layer
OS OS
Node
Membership
HA Framework
ASM Instance
OS
Oracle RAC
DB Instance
Oracle ASM managed
Shared Storage
Oracle Clusterware files
Voting Files (disks)
Cluster Registry (OCR)
Database Files
Redo / Archive Logs
Control Files
etc.
Cloud File System
(aka ASM Cluster File
File System - ACFS)
For all other files
11.2.0.3: For Backup Sets
1/20/2012
5
Oracle Grid Infrastructure 11g Release 2 process overview
OS OS
Node
Membership
HA Framework
ASM Instance
OS
My Oracle Support (MOS)
Note 1053147.1 - 11gR2 Clusterware
and Grid Home - What You Need to Know
Note 1050908.1 - How to Troubleshoot
Grid Infrastructure Startup Issues

Local and Cluster Resources (11.2.0.2 list)
OS OS OS OS
ora.asm
ora.dgname.dg
ora.listener.lsnr
ora.ons
ora.gsd
ora.net1.network
ora.registry.acfs
ora.asm
ora.dgname.dg
ora.listener.lsnr
ora.ons
ora.gsd
ora.net1.network
ora.registry.acfs
ora.asm
ora.dgname.dg
ora.listener.lsnr
ora.ons
ora.gsd
ora.net1.network
ora.registry.acfs
ora.asm
ora.dgname.dg
ora.listener.lsnr
ora.ons
ora.gsd
ora.net1.network
ora.registry.acfs
LISTENER_SCAN1
ora.SCAN1.VIP
ora.node1.vip

LISTENER_SCAN2
ora.SCAN2.VIP
ora.node2.vip

LISTENER_SCAN3
ora.SCAN3.VIP
ora.node3.vip

ora.oc4j
ora.cvu
ora.node4.vip

If database is deployed:
ora.Dbname.db
ora.Dbname.Srvcname.svc

Command to list:
crsctl stat res -t

1/20/2012
6
VIPs and Networks
VIPs and Networks
Oracle VIPs
SCAN and node VIPs are cluster resources
SCAN VIPs can float in the cluster
Node VIPs per default run on specific nodes

All VIPs on the public network (SCAN, node,
user or application VIPs) depend on the Oracle
network resource.
The network resource monitors
the public network interface(s).
IOCTL based checks vs. default gateway ping
Default gateway checks as backup
racgvip for backward compatibility only

OS OS

ora.listener.lsnr

ora.net1.network
LISTENER_SCAN1
ora.SCAN1.VIP
ora.rac1.vip

LISTENER_SCAN2
ora.SCAN2.VIP
ora.rac2.vip

ora.listener.lsnr

ora.net1.network
rac1 rac2
1/20/2012
7
VIPs and Networks
Node VIP Details
[GRID]> crsctl status res ora.rac1.vip -p
NAME=ora.rac1.vip
TYPE=ora.cluster_vip_net1.type
...
AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX%
...
CHECK_INTERVAL=1
CHECK_TIMEOUT=30
...
START_DEPENDENCIES=hard(ora.net1.network)
pullup(ora.net1.network)
...
STOP_DEPENDENCIES=hard(ora.net1.network)
...
USR_ORA_VIP=rac1-vip
VERSION=11.2.0.2.0
OS OS

ora.listener.lsnr

ora.net1.network

ora.rac1.vip

ora.rac2.vip

ora.listener.lsnr

ora.net1.network
rac1 rac2
1
2
3
VIPs and Networks
The network resource
OS OS

ora.listener.lsnr

ora.net1.network

ora.rac1.vip

ora.rac2.vip

ora.listener.lsnr

ora.net1.network
rac1 rac2
The Oracle network resource monitors the
public network interface(s) (e.g. eth0).
The default is network number 1 (net1)
There is one network resource per subnet:

[GRID]> srvctl config network
Network exists:
1/192.168.0.0/255.255.255.0/eth0, type static

New in 11.2.0.2:
Direct management of the network resource
srvctl verb network
Network type support (static, dhcp, mixed)

1/20/2012
8
VIPs and Networks
How to enable multiple networks in the cluster?
OS OS

ora.net1.network
ora.net2.network

ora.net1.network
ora.net2.network

[GRID]> srvctl add network h
Adds a network configuration to the Oracle Clusterware.
Usage: srvctl add network
-k <net_num> network number
(default number is 1)
-S <subnet>/<netmask>/[if1[|if2...]]
NET address spec for network
-w <network_type> The network type
(static, dhcp, mixed)
-h Print usage
-v Verbose output

For example:
srvctl add network -k 2 -S 10.1.1.0/255.255.255.0

1
VIPs and Networks
How to enable multiple networks in the cluster?
OS OS

ora.net1.network
ora.net2.network

ora.net1.network
ora.net2.network

[GRID]> srvctl add vip -h
Adds a VIP to the Oracle Clusterware.
Usage: srvctl add vip
-n <node_name> Node name
-A <vip_name|ip>/<netmask>/[if1[|if2...]]
VIP address spec for node applications
-k <net_num> network number
(default number is 1)
-h Print usage
-v Verbose output

For example:
srvctl add vip -n rac1 -k 2 -A rac1-vip2/255.255.255.0
srvctl add vip -n rac2 -k 2 -A rac2-vip2/255.255.255.0

2
ora.rac1.vip
ora.rac1-vip2.vip
ora.rac2.vip
ora.rac2-vip2.vip
1/20/2012
9
VIPs and Networks
Application / user and node VIPs
OS OS

ora.net1.network
ora.net2.network

ora.net1.network
ora.net2.network

[GRID]> crsctl stat res ora.rac1-vip2.vip -p
NAME=ora.rac1-vip2.vip
TYPE=ora.cluster_vip_net2.type
...
HOSTING_MEMBERS=rac1
...
START_DEPENDENCIES=hard(ora.net2.network)
pullup(ora.net2.network)
...
STOP_DEPENDENCIES=hard(ora.net2.network)
...
USR_ORA_VIP=rac1-vip2
VERSION=11.2.0.2.0
ora.rac1.vip
ora.rac1-vip2.vip
ora.rac2.vip
ora.rac2-vip2.vip
1
This is a node vip!

VIPs and Networks
OS OS

ora.net1.network
ora.net2.network

ora.net1.network
ora.net2.network

[GRID]> appvipcfg create
-network=1
-ip 192.168.0.66
-vipname=myVIP
-user=root

[GRID]> appvipcfg create
-network=2
-ip 10.1.1.88
-vipname=myNewVIP
-user=root
ora.rac1.vip
ora.rac1-vip2.vip
ora.rac2.vip
ora.rac2-vip2.vip
2
myVIP myNewVIP
This is an application vip!

1/20/2012
10
VIPs and Networks
OS OS

ora.net1.network
ora.net2.network

ora.net1.network
ora.net2.network

[GRID]> crsctl stat res myVIP -p
NAME=myVIP
TYPE=app.appvip_net1.type
...
HOSTING_MEMBERS=rac1
...
RESTART_ATTEMPTS=0
...
USR_ORA_VIP=192.168.0.66
VERSION=11.2.0.2.0

You can modify it as needed!
ora.rac1.vip
ora.rac1-vip2.vip
ora.rac2.vip
ora.rac2-vip2.vip
3
myVIP myNewVIP
This is an application vip!

Listeners and SCAN
1/20/2012
11
Listeners
Listeners and dependencies
In Oracle RAC 11g Release 2 there have been
three major changes regarding listeners:
1. Listeners per default now run from the
Oracle Grid Infrastructure (GRID) home
2. Listeners are managed by an Oracle agent
3. SCAN Listeners were introduced

A listener depends on a VIP
A listener depends indirectly on a network

OS OS

ora.listener.lsnr

ora.net1.network
LISTENER_SCAN1
ora.SCAN1.VIP
ora.rac1.vip

LISTENER_SCAN2
ora.SCAN2.VIP
ora.rac2.vip

ora.listener.lsnr

ora.net1.network
rac1 rac2
Listeners
The default LISTENER
[GRID]> srvctl config listener
Name: LISTENER
Network: 1, Owner: oracle Grid Software Owner
Home: <CRS home>
End points: TCP:1521

OS OS

ora.LISTENER.lsnr

ora.net1.network

ora.LISTENER.lsnr

ora.net1.network
rac1 rac2
1/20/2012
12
Listeners
The default LISTENER FAQ
OS OS

ora.LISTENER.lsnr

ora.net1.network

ora.LISTENER.lsnr

ora.net1.network
rac1 rac2
Can you define another port? - YES

Can you define another owner? YES

Can you define another home? YES, if really needed.

Can I add another listener on another port? - YES

[GRID]> srvctl modify listener -l LISTENER -p TCP:1545

Name: LISTENER
Network: 1, Owner: oracle
Home: <CRS home>

Check: <CRS HOME>/network/admin/endpoints_listener.ora before:
LISTENER_RAC1=(DESCRIPTION=(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=rac1-vip)
(PORT=1521))(ADDRESS=(PROTOCOL=TCP)
(HOST=192.168.0.51)(PORT=1521)(IP=FIRST))))
# line added by Agent

OS OS

ora.LISTENER.lsnr

ora.net1.network

ora.LISTENER.lsnr

ora.net1.network
rac1 rac2
1
1/20/2012
13
[GRID]> srvctl modify listener -l LISTENER -p TCP:1545

Name: LISTENER
Home: <CRS home>

Check: <CRS HOME>/network/admin/endpoints_listener.ora after restart:
LISTENER_RAC1=(DESCRIPTION=(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=rac1-vip)
(PORT=1545))(ADDRESS=(PROTOCOL=TCP)
(HOST=192.168.0.51)(PORT=1545)(IP=FIRST))))

OS OS

ora.LISTENER.lsnr

ora.net1.network

ora.LISTENER.lsnr

ora.net1.network
rac1 rac2
2
Can you define another owner? - YES
[GRID]> srvctl modify listener -l LISTENER -u root
You need to be root
Name: LISTENER
Network: 1, Owner: root For demonstration
purposes only
Home: <CRS home>

OS OS

ora.LISTENER.lsnr

ora.net1.network

ora.LISTENER.lsnr

ora.net1.network
rac1 rac2
1/20/2012
14
Can you define another home? - YES
[GRID]> srvctl modify listener -l LISTENER
-o /u01/app/oracle/product/11.2.0/dbhome_1

Name: LISTENER
Home: /u01/app/oracle/product/11.2.0/dbhome_1

This may not be required.
Instead, create a new listener from a separate home, if needed
and use the SRVCTL command as shown, but try using the GI home.

OS OS

ora.LISTENER.lsnr

ora.net1.network

ora.LISTENER.lsnr

ora.net1.network
rac1 rac2
Listeners
Can I add another listener on another port? - YES
Name: LISTENER
Home: <CRS home>
Name: LISTENER2011
Home: <CRS home>

MOS Note 220970.1 - RAC: Frequently Asked Questions
How to use SCAN and node listeners with different ports?
Add additional node listeners to the system using different ports

OS OS
rac1 rac2

ora.LISTENER.lsnr
ora.LISTENERK2.lsnr
ora.net1.network

ora.LISTENER.lsnr
ora.LISTENERK2.lsnr
ora.net1.network
1/20/2012
15
Listeners
Can I add another listener on another network? - YES
[GRID]> srvctl add listener -l ListenerK2 -p 1544 -k 2
Name: LISTENER
Home: <CRS home>
Name: LISTENERK2
Home: <CRS home>

MOS Note 1063571.1 - How to Configure A Second
Listener on a Separate Network in 11.2 Grid Infrastructure

OS OS
rac1 rac2

ora.LISTENER.lsnr
Ora.LISTENERK2.lsnr
ora.net1.network
ora.net2.network

ora.LISTENER.lsnr
ora.LISTENERK2.lsnr
ora.net1.network
ora.net2.network

Listeners
Remember its just another listener
Name: JUSTALISTENER
Network: 1, Owner: root
Home: <CRS home>
...
[GRID]> vi /u01/app/11.2.0/grid/network/admin/listener.ora
JUSTALISTENER=(DESCRIPTION=(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=IPC)(KEY=JUSTALISTENER))))
[GRID]> vi /u01/app/11.2.0/grid/network/admin/endpoints_listener.ora
JUSTALISTENER_RAC1=(DESCRIPTION=(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=rac1-vip)(PORT=1522))
(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.51)(PORT=1522)(IP=FIRST))))

OS OS

ora.LISTENER.lsnr

ora.net1.network
LISTENER_SCAN1
ora.SCAN1.VIP
ora.rac1.vip

LISTENER_SCAN2
ora.SCAN2.VIP
ora.rac2.vip

ora.LISTENER.lsnr

ora.net1.network
rac1 rac2
1
1/20/2012
16
Listeners
Remember its just another listener
(ADDRESS=(PROTOCOL=IPC)(RATE_LIMIT=10) For demonstration purposes only
(KEY=JUSTALISTENER)))) # line added by Agent
...
[GRID]> srvctl stop listener -l JustAListener
[GRID]> srvctl start listener -l JustAListener
...
(ADDRESS=(PROTOCOL=IPC)(RATE_LIMIT=10)
(KEY=JUSTALISTENER)))) # line added by Agent

OS OS

ora.LISTENER.lsnr

ora.net1.network
LISTENER_SCAN1
ora.SCAN1.VIP
ora.rac1.vip

LISTENER_SCAN2
ora.SCAN2.VIP
ora.rac2.vip

ora.LISTENER.lsnr

ora.net1.network
rac1 rac2
2
SCAN
The basics and overview
OS OS
rac1 rac3
LISTENER_SCAN1
ora.SCAN1.VIP

LISTENER_SCAN3
ora.SCAN3.VIP

OS
rac2
LISTENER_SCAN2
ora.SCAN2.VIP

In Oracle RAC 11g Release 2
SCAN Listeners were introduced
There is one SCAN setup per cluster

Read about SCAN:
http://www.oracle.com/technetwork/database/
clustering/overview/scan-129069.pdf
MOS Note 887522.1
11gR2 Grid Infrastructure Single Client
Access Name (SCAN) Explained

SCAN consists of a bundle containing:
1. SCAN IP
2. Associated (dependency) SCAN Listener

ora.LISTENER.lsnr

ora.net1.network

ora.LISTENER.lsnr

ora.net1.network

ora.LISTENER.lsnr

ora.net1.network
SCAN
1/20/2012
17
SCAN
The SCAN bundle
OS OS
rac1 rac3
LISTENER_SCAN1
ora.SCAN1.VIP

LISTENER_SCAN3
ora.SCAN3.VIP

OS
rac2
LISTENER_SCAN2
ora.SCAN2.VIP

3 SCANs are the default for HA and LB
Regardless of number of nodes
You can define less or more, if really needed.
SCAN listener resources run with
an active dispersion dependency
If you use more nodes in the cluster than
SCAN listeners are defined, no node should
run more than one SCAN bundle at a time
If you use less nodes in the cluster than
SCAN listeners are defined, there will be
nodes running more than one SCAN bundle
at a time.
SCAN VIP moves with the listener, if possible.

ora.LISTENER.lsnr

ora.net1.network

ora.LISTENER.lsnr

ora.net1.network

ora.LISTENER.lsnr

ora.net1.network
SCAN
1
SCAN
The SCAN bundle
SCAN
2
OS OS

ora.LISTENER.lsnr

ora.net1.network
LISTENER_SCAN1
ora.SCAN1.VIP

ora.LISTENER.lsnr

ora.net1.network
rac1 rac2
[GRID]> srvctl config scan
SCAN name: cluster1, Network: 1/192.168.0.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /cluster1.us.oracle.com/192.168.0.41

[GRID]> srvctl modify scan
Modifies the SCAN name.
Usage: srvctl modify scan
-n <scan_name> Domain name qualified SCAN name
-h Print usage

http://www.oracle.com/technetwork/database/
clustering/overview/scan-129069.pdf page 2

1/20/2012
18
SCAN
The SCAN bundle
3
SCAN
OS OS

ora.LISTENER.lsnr

ora.net1.network
LISTENER_SCAN1
ora.SCAN1.VIP

ora.LISTENER.lsnr

ora.net1.network
rac1 rac2
[GRID]> srvctl add scan -h
Adds a SCAN VIP to the Oracle Clusterware.
Usage: srvctl add scan -n <scan_name>
-n <scan_name> Domain name qualified SCAN name
-k <net_num> network number (default number is 1)
-S <subnet>/<netmask>/[if1[|if2...]]
NET address spec for network
-h Print usage

Note: SCAN can only operate on one network / in one subnet.

SCAN
The SCAN FAQ
From MOS Note 220970.1
RAC: Frequently Asked Questions
With Oracle RAC 11g Release 2 using SCAN is the default.
As with other listeners, there is no direct communication
between the node (listeners) and the SCAN listeners.
Listeners are only aware of the instances and services served,
since the instances (PMON) register themselves and the services
they host with the listeners.
The instances use the LOCAL and REMOTE Listener
parameters to know with which listeners to register.
Listeners used for a client connection to Oracle RAC should be
managed by Oracle Clusterware and should be listening on an
Oracle managed VIP.

1
SCAN
OS OS

ora.LISTENER.lsnr

ora.net1.network
LISTENER_SCAN1
ora.SCAN1.VIP

ora.LISTENER.lsnr

ora.net1.network
rac1 rac2
1/20/2012
19
SCAN
The SCAN FAQ
See MOS Note 220970.1
Use srvctl modify scan_listener p <newPort>
Can you define another owner? NO
Can you define another home? NO

Can you have more than one node listener with SCAN? YES
Can the SCAN and the node listener ports differ? - YES
See also MOS Note 972500.1 How to
Modify SCAN Setting or SCAN Listener Port after Installation

2
SCAN
OS OS

ora.LISTENER.lsnr

ora.net1.network
LISTENER_SCAN1
ora.SCAN1.VIP

ora.LISTENER.lsnr

ora.net1.network
rac1 rac2
Listeners and Instance Registration
Listener changes may require instance changes
See MOS Note 220970.1

Whenever you change an instance parameter configuration
a change to the LOCAL_LISTENER or REMOTE_LISTENER
per database (instance) may be required.
Note that using server-side TNSANMES aliases to
resolve the listener configuration is not recommended.
The use of the TNSNAMES connector string is the default:

local_listener = (DESCRIPTION= (ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.61)(PORT=2011))
))'

SCAN
OS OS
ora.LISTENER.lsnr

LISTENER_SCAN1
ora.SCAN1.VIP

ora.LISTENER.lsnr

rac1 rac2
DB Instance DB Instance
1/20/2012
20
EXAMPLE
Taken from MOS Note 220970.1 RAC: FAQ

srvctl add listener -l LISTENER2011 -p 2011
srvctl start listener -l LISTENER2011
srvctl status listener -l LISTENER2011
lsnrctl status LISTENER2011

Name: LISTENER2011
Home: <CRS home>

SCAN
OS OS
ora.LISTENER.lsnr

LISTENER_SCAN1
ora.SCAN1.VIP

ora.LISTENER.lsnr

rac1 rac2
1
EXAMPLE

New node listener does not serve any database instance
You must alter the LOCAL_LISTENER parameters
for each instance accordingly:

alter system set local_listener =
(DESCRIPTION= (ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.61)(PORT=2011))))
' scope=BOTH SID='OCRL1';
alter system register;
SCAN
OS OS
ora.LISTENER.lsnr

LISTENER_SCAN1
ora.SCAN1.VIP

ora.LISTENER.lsnr

rac1 rac2
2
1/20/2012
21
EXAMPLE

Note: Use spaces between the various segments of the command
Reason: the database agent in Oracle Clusterware currently
determines whether the local_listener or remote_listener have
been manually set by a string comparison operation.
If the string looks unaltered it will reset these parameters
with the default values that it determines on instance startup.
Slight modifications are therefore required. ER 11772838
has been filed to allow for a more convenient mechanism.
SCAN
OS OS
ora.LISTENER.lsnr

LISTENER_SCAN1
ora.SCAN1.VIP

ora.LISTENER.lsnr

rac1 rac2
3
Using listeners in different subnets / networks
SCAN can only operate in one network / subnet

If listeners in different subnets are used,
use LISTENER_NETWORKS:
http://download.oracle.com/docs/cd/E11882_01
/server.112/e17110/initparams115.htm#REFRN10313
Note: Listeners specified by the LISTENER_NETWORKS
parameter should not be used in the LOCAL_LISTENER and
REMOTE_LISTENER parameters. Otherwise, cross registration
will happen and connections will be redirected cross networks.

For more information, see also
MOS Note 1063571.1 How to Configure A Second
Listener on a Separate Network in 11.2 Grid Infrastructure

OS OS
LISTENER_SCAN1
ora.SCAN1.VIP

rac1 rac2

ora.LISTENER.lsnr
Ora.LISTENERK2.lsnr
ora.net1.network
ora.net2.network

ora.LISTENER.lsnr
ora.LISTENERK2.lsnr
ora.net1.network
ora.net2.network

1/20/2012
22
Listeners, SCAN and Services
Listeners and Service Registration
A summary
Services depend on a database
Per default, the database instance will register
all services it hosts with all listeners it registers with.

Services operate within the following boundaries:
Network (indirectly via dependency)
Server Pools (in policy-managed database environments)

Currently there is no support for service failover
Between Server Pools
Between networks
SCAN
OS OS
ora.LISTENER.lsnr

ora.Dbname.db

ora.Dbname.db

ora.LISTENER.lsnr

rac1 rac2
1/20/2012
23
Services and Dependencies
Dependency on DB and Server Pool
[GRID]> srvctl config service -d ORCL -s MyService
Service name: MyService
Service is enabled
Server pool: ORCL
Cardinality: 1
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
...
Preferred instances: ORCL_1
Available instances:

SCAN
OS OS
ora.LISTENER.lsnr

ora.ORCL.db
ora.ORCL.MyService.svc

ora.ORCL.db

ora.LISTENER.lsnr

rac1 rac2
1
Services and Dependencies
Start / Stop dependencies
[GRID]> crsctl stat res ora.orcl.myservice.svc -p
...
DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=service)
PROPERTY(SERVICE_NAME=%GEN_SERVICE_NAME%)
PROPERTY(DB_UNIQUE_NAME=CONCAT(PARSE(%NAME%, .,
2), STAT(ora.orcl.db, USR_ORA_DOMAIN), .))
ELEMENT(INSTANCE_NAME=STAT(ora.orcl.db,
GEN_USR_ORA_INST_NAME))
...
START_DEPENDENCIES=hard(ora.orcl.db,type:ora.cluster
_vip_net1.type) weak(type:ora.listener.type)
pullup(type:ora.cluster_vip_net1.type)
pullup:always(ora.orcl.db)
dispersion(type:ora.service.type)
STOP_DEPENDENCIES=hard(intermediate:ora.orcl.db,inte
rmediate:type:ora.cluster_vip_net1.type)
...
SCAN
OS OS
ora.LISTENER.lsnr

ora.Dbname.db

ora.LISTENER.lsnr

rac1 rac2
2
ora.ORCL.db
ora.ORCL.MyService.svc

1/20/2012
24
Client Connectivity
Client Connectivity
Direct or indirect connect
Production
Email
BATCH
Connect Time Load Balancing (CTLB)
Connect Time Connection Failover (CTCF)

Runtime Connection Load Balancing (RTLB)
Runtime Connection Failover (RTCF)

Connection
Pool
S
C
A
N

1/20/2012
25
Client Connectivity
Connect Time Connection Failover
jdbc:oracle:thin:@MySCAN:1521/Email
PMRAC =
(DESCRIPTION =
(FAILOVER=ON)
(ADDRESS = (PROTOCOL = TCP)(HOST = MySCAN)(PORT = 1521))
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email)))
Production
Email
BATCH
Connection
Pool
M
y
S
C
A
N

Client Connectivity
Runtime Time Connection Failover

Production
Email
BATCH
Connection
Pool
M
y
S
C
A
N

PMRAC =
(DESCRIPTION =
(FAILOVER=ON)
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email)
...))
1/20/2012
26
Client Connectivity
Runtime Time Connection Failover

Production
Email
BATCH
Connection
Pool
M
y
S
C
A
N

PMRAC =
(DESCRIPTION =
(FAILOVER=ON)
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email)
(FAILOVER_MODE= (TYPE=select)(METHOD=basic)(RETRIES=180)(DELAY=5))))
?
Client Connectivity
More information

Production
Email
BATCH
Connection
Pool
M
y
S
C
A
N

?
?
?
If problems occur, see:
Note 975457.1 How to Troubleshoot Connectivity Issues with 11gR2 SCAN Name
For more advanced configurations, see:
Note 1306927.1 Using the TNS_ADMIN variable and changing the default port
number of all Listeners in an 11.2 RAC for an 11.2, 11.1, and 10.2 Database
1/20/2012
27
Client Connectivity
Two ways to protect the client

Production
Email
BATCH
Connection
Pool
M
y
S
C
A
N

1. Transparent Application Failover (TAF)
Tries to make the client unaware of a failure
Provides means of CTCF and RTCF
Allows for pure selects (reads) to continue
Write transactions need to be re-issued
The Application needs to be TAF aware
2. Fast Application Notification (FAN)
FAN wants to inform clients ASAP
Client can react to failure asap
Expects clients to re-connect on failure (FCF)
Sends messages about changes in the cluster
?
Client Connectivity and Service Definition
Define settings on the server

Production
Email
BATCH
M
y
S
C
A
N

[GRID]> srvctl config service
-d ORCL -s MyService
Service name: MyService
...
DTP transaction: false
AQ HA notifications: false
Failover type: NONE
Failover method: NONE
TAF failover retries: 0
TAF failover delay: 0
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: BASIC
HA (and LB) settings
can be defined per service
Clients connecting to the service will
adhere to the settings considering the
client used.
1/20/2012
28
Client Connectivity
Use a FAN aware connection pool

Production
Email
BATCH
Connection Pool
M
y
S
C
A
N

If a connection pool is used
The clients (users) get a physical
connection to the connection pool
The connection pool creates a physical
connection to the database
It is a direct client to the database
Internally the pool maintains logical connections
1
Client Connectivity
Use a FAN aware connection pool

Production
Email
BATCH
Connection Pool
M
y
S
C
A
N

The connection pool
Invalidates connections to one instance
Re-establishes new logical connections
May create new physical connections
Prevent new clients to be misrouted

The application needs to handle the
transaction failure that might have occurred.
2
1/20/2012
29
Client Connectivity
The Load Balancing (LB) cases

On the Client Side
On the Server Side

Production
Email
BATCH
Connection
Pool
M
y
S
C
A
N

Client Connectivity
Connect Time Load Balancing (CTLB) on the client side
Production
Email
BATCH
Connection
Pool
M
y
S
C
A
N

PMRAC =
(DESCRIPTION =
(FAILOVER=ON)(LOAD_BALANCE=ON)
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email)))
1/20/2012
30
Client Connectivity
Connect Time Load Balancing (CTLB) on the server side
Production
Email
BATCH
Connection
Pool
M
y
S
C
A
N

?
Traditionally, PMON dynamically registers the services to the specified listeners with:
Service names for each running instance of the database and instance names for the DB
The listener is updated with the load information for every instance and node as follows:
1-Minute OS Node Load Average all 30 secs.
Number of Connections to Each Instance
Number of Connections to Each Dispatcher
Client Connectivity
Use FAN for the Load Balancing cases
Connect Time Connection Failover (CTCF)

Runtime Connection Failover (RTCF)

RAC
Database
Instance1
Instance2
Instance3
Im busy
Im very busy
Im idle
30% connections
10% connections
60% connections
1/20/2012
31
Client Connectivity
Use FAN for the Load Balancing cases
Also via AQ (Advanced Queuing) based notifications
Background is always the Load Balancing Advisory

For more information, see:
Oracle Real Application
Clusters Administration and
Deployment Guide 11g Release 2:
5 Introduction to Automatic Workload Management

RAC
Database
Instance1
Instance2
Instance3
Im busy
Im very busy
Im idle
30% connections
10% connections
60% connections
M
y
S
C
A
N

Node Membership
1/20/2012
32
Oracle Clusterware Architecture
Node Membership Processes and Basics
SAN
Network
SAN
Network
Public Lan Public Lan
CSSD CSSD CSSD
Voting
Disk
Private Lan /
Interconnect
Oracle Clusterware
Main processes involved:
CSSD (ora.cssd)
CSSDMONITOR
was: oprocd
now: ora.cssdmonitor
What does CSSD do?
Monitors nodes using 2 communication channels:
Private Interconnect Network Heartbeat
Voting Disk based communication Disk Heartbeat

Evicts (forcibly removes nodes from a
cluster) nodes dependent on heartbeat
feedback (failures)

CSSD CSSD
Ping
Ping
Oracle Clusterware
1/20/2012
33
Interconnect basics network heartbeat
CSSD CSSD
Ping
Each node in the cluster is pinged every second

Nodes must respond in css_misscount time (defaults to 30 secs.)
Reducing the css_misscount time is generally not supported

Network heartbeat failures
will lead to node evictions
CSSD-log:
[date / time] [CSSD][1111902528]
clssnmPollingThread: node mynodename
(5) at 75% heartbeat fatal, removal
in 6.770 seconds

Voting Disk basics disk heartbeat
CSSD CSSD
Ping
1
Each node in the cluster pings (r/w) the Voting Disk(s) every second

Nodes must receive a response in (long / short) diskTimeout time
IF I/O errors indicate clear accessibility problems timeout is irrelevant

Disk heartbeat failures
will lead to node evictions
CSSD-log:
[CSSD] [1115699552] >TRACE:
clssnmReadDskHeartbeat:
node(2) is down. rcfg(1) wrtcnt(1)
LATS(63436584) Disk lastSeqNo(1)
1/20/2012
34
Voting Disk basics Structure
2
Voting Disks contain dynamic and static data:
Dynamic data: disk heartbeat logging
Static data: information about the nodes in the cluster

With 11.2.0.1 Voting Disks got an identity:
E.g. Voting Disk serial number: [GRID]> crsctl query css votedisk
1. 2 1212f9d6e85c4ff7bf80cc9e3f533cc1 (/dev/sdd5) [DATA]

Voting Disks must therefore not be copied using dd or cp anymore

Node information Disk Heartbeat Logging
Voting Disk basics Simple Majority rule
CSSD CSSD
Ping
3 Oracle supports redundant Voting Disks for disk failure protection
Simple Majority Rule applies:
Each node must see the simple majority of configured Voting Disks
at all times in order not to be evicted (to remain in the cluster)

trunc(n/2+1) with n=number of voting disks
configured and n>=1
1/20/2012
35
CSSD CSSD
Same principles apply
Voting Disks are just
geographically dispersed
http://www.oracle.com/goto/rac
Using standard NFS to support
a third voting file for extended
cluster configurations (PDF)
Simple Majority rule in extended clusters

Oracle ASM auto creates 1/3/5 Voting Files
Voting Disks reside in one diskgroup only
Based on Ext/Normal/High redundancy
and on Failure Groups in the Disk Group
Per default there is one failure group per disk
ASM will enforce the required number of disks
New failure group type: Quorum Failgroup
[GRID]> crsctl query css votedisk
1. 2 1212f9d6e85c4ff7bf80cc9e3f533cc1 (/dev/sdd5) [DATA]
2. 2 aafab95f9ef84f03bf6e26adc2a3b0e8 (/dev/sde5) [DATA]
3. 2 28dd4128f4a74f73bf8653dabd88c737 (/dev/sdd6) [DATA]
Located 3 voting disk(s).
Voting Disks in Oracle ASM does not change their usage
1/20/2012
36
Oracle Cluster Registry (OCR) placement in Oracle ASM
The OCR is managed like a datafile in ASM (new type)
It adheres completely to the redundancy settings for the diskgroup (DG)
There can be more than one OCR location in more than one DG (DG:OCR 1:1)
Recommendation is 2 OCR locations, 1 in DATA, 1 in FRA for example
Backup of Clusteware files is fully automatic (11.2+)
Clusterware Files (managed in ASM) enables fully Automatic Backups:
The Voting Disks are backed up into the OCR
Any configuration change in the cluster (e.g. node
addition) triggers a new backup of the Voting Files.
A single, failed Voting Disks is restored by ASM
automatically within a Disk Group no action required
Note: Do not use DD to back up the Voting Disks anymore!

The OCR is backed up automatically every 4 hours
Manual Backups can be taken as required

ONLY IF all Voting Disks are corrupted or failed
AND (all copies of) the OCR are also corrupted or unavailable
THEN manual interference would be required the rest is automatic.
1/20/2012
37
Evicting (fencing) nodes is a preventive measure (its a good thing)!
Nodes are evicted to prevent consequences of a split brain:
Shared data must not be written by independently operating nodes
The easiest way to prevent this is to forcibly remove a node from the cluster

Fencing Basics
Why are nodes evicted?
CSSD CSSD
1 2
Fencing Basics
How are nodes evicted? STONITH
Once it is determined that a node needs to be evicted,
A kill request is sent to the respective node(s)
Using all (remaining) communication channels
A node (CSSD) is requested to kill itself STONITH like
STONITH foresees that a remote node kills the node to be evicted

CSSD CSSD
1 2
1/20/2012
38
Fencing Basics
EXAMPLE: Network heartbeat failure
CSSD CSSD
1 2
2
The network heartbeat between nodes has failed
It is determined which nodes can still talk to each other
A kill request is sent to the node(s) to be evicted
Using all (remaining) communication channels Voting Disk(s)
A node is requested to kill itself; executer: typically CSSD
Fencing Basics
What happens, if CSSD is stuck?
CSSD CSSD
1 2
2
CSSD
CSSDmonitor
A node is requested to kill itself
BUT CSSD is stuck or sick (does not execute) e.g.:
CSSD failed for some reason
CSSD is not scheduled within a certain margin
OCSSDMONITOR (was: oprocd) will take over and execute

See also: MOS note
1050693.1 -
Troubleshooting 11.2
Clusterware Node
Evictions (Reboots)

1/20/2012
39
Fencing Basics
How can nodes be evicted?
Oracle Clusterware 11.2.0.1 and later supports IPMI (optional)
Intelligent Platform Management Interface (IPMI) drivers required
IPMI allows remote-shutdown of nodes using additional hardware
A Baseboard Management Controller (BMC) per cluster node is required
CSSD CSSD
1
Fencing Basics
EXAMPLE: IPMI based eviction on heartbeat failure
CSSD
1
The network heartbeat between the nodes has failed
It is determined which nodes can still talk to each other
IPMI is used to remotely shutdown the node to be evicted
1/20/2012
40
Fencing Basics
Which node gets evicted?
CSSD CSSD
1 2
Voting Disks and heartbeat communication is used to determine the node

In a 2 node cluster, the node with the lowest node number should survive
In a n-node cluster, the biggest sub-cluster should survive (votes based)

Fencing Basics
Cluster members can escalate a kill request
Oracle RAC
DB Inst. 1
Oracle RAC
DB Inst. 2
Oracle Clusterware
Cluster members (e.g Oracle RAC instances) can request
Oracle Clusterware to kill a specific member of the cluster

Oracle Clusterware will then attempt to kill the requested member

Inst. 1:
kill inst. 2
1/20/2012
41
Fencing Basics
Oracle RAC
DB Inst. 1
Oracle RAC
DB Inst. 2
Oracle Clusterware

If the requested member kill is unsuccessful, a node eviction escalation can be issued,
which leads to the eviction of the node, on which the particular member currently resides
Inst. 1:
kill inst. 2
Fencing Basics
Oracle RAC
DB Inst. 1
Oracle RAC
DB Inst. 2
Oracle Clusterware

Inst. 1:
kill inst. 2
1/20/2012
42
Fencing Basics
Oracle RAC
DB Inst. 1
Oracle Clusterware

Re-Bootless Node Fencing
With 11.2.0.2 onwards, fencing may not mean re-boot
Oracle Clusterware
Until Oracle Clusterware 11.2.0.2, fencing meant re-boot
With Oracle Clusterware 11.2.0.2, re-boots will be seen less, because:
Re-boots affect applications that might run an a node, but are not protected
Customer requirement: prevent a reboot, just stop the cluster implemented...

CSSD CSSD
App X App Y
RAC DB
Inst. 1
RAC DB
Inst. 2
1/20/2012
43
How it works
With Oracle Clusterware 11.2.0.2, re-boots will be seen less:
Instead of fast re-booting the node, a graceful shutdown of the stack is attempted

It starts with a failure e.g. network heartbeat or interconnect failure

Oracle Clusterware
CSSD CSSD
App X App Y
RAC DB
Inst. 1
RAC DB
Inst. 2
How it works
Oracle Clusterware
CSSD CSSD
App X App Y
RAC DB
Inst. 1

Then IO issuing processes are killed; it is made sure that no IO process remains
For a RAC DB mainly the log writer and the database writer are of concern

1/20/2012
44
How it works
Oracle Clusterware
CSSD CSSD
App X App Y
RAC DB
Inst. 1

Once all IO issuing processes are killed, remaining processes are stopped
IF the check for a successful kill of the IO processes, fails reboot

How it works
Oracle Clusterware
CSSD
App X App Y
RAC DB
Inst. 1

Once all remaining processes are stopped, the stack stops itself with a restart flag

OHASD
1/20/2012
45
How it works
Oracle Clusterware
CSSD
App X App Y
RAC DB
Inst. 1

OHASD will finally attempt to restart the stack after the graceful shutdown
OHASD
EXCEPTIONS
With Oracle Clusterware 11.2.0.2, re-boots will be seen less, unless:
IF the check for a successful kill of the IO processes fails reboot
IF CSSD gets killed during the operation reboot
IF cssdmonitor (oprocd replacement) is not scheduled reboot
IF the stack cannot be shutdown in short_disk_timeout-seconds reboot
Oracle Clusterware
CSSD CSSD
App X App Y
RAC DB
Inst. 1
RAC DB
Inst. 2
1/20/2012
46
The Interconnect
The Interconnect
Heartbeat and memory channel between instances
Interconnect
with switch
Public Lan
SAN switch
Client
Network
Node 1 Node 2 Node N Node N-1
1/20/2012
47
The Interconnect
Redundant Interconnect Usage
Node 1 Node 2
HAIP
1
HAIP
2
HAIP
3
HAIP
4
Redundant Interconnect Usage can be used as a bonding alternative
It works for private networks only; the nodeVIPs use a different approach
It enables HA and Load Balancing for up to 4 NICs per server (on Linux / Unix)
It can be used by Oracle Databases 11.2.0.2 and Oracle Clusterware 11.2.0.2
It uses so called HAIPs that are assigned to the private networks on the server
The HAIPs will be used by the database and ASM instances and processes

1
The Interconnect
Redundant Interconnect Usage
Node 1 Node 2
HAIP
1
HAIP
2
HAIP
3
HAIP
4
A multiple listening endpoint approach is used
The HAIPs are taken from the link-local (Linux / Unix) IP range (169.254.0.0)
To find the communication partners, multicasting on the interconnect is required
With 11.2.0.3 Broadcast is a fallback alternative (BUG 10411721)
Multicasting is still required on the public lan for MDNS for example.
Details in My Oracle Support (MOS) Note with Doc ID 1212703.1:
11.2.0.2 Grid Infrastructure Install or Upgrade may fail due to Multicasting

2
1/20/2012
48
The Interconnect
Redundant Interconnect Usage and the HAIPs
Node 1 Node 2
HAIP
1
HAIP
2
HAIP
3
HAIP
4
If a network interface fails, the assigned HAIP is failed over to a remaining one.
Redundant Interconnect Usage allows having networks in different subnet
You can either have one subnet for all networks or a different one for each
You can also use VLANs with the interconnect. For more information see:
Note 1210883.1 - 11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip
Note 220970.1 - RAC: Frequently Asked Questions - How to use VLANs in Oracle RAC? AND
Are there any issues for the interconnect when sharing the same switch as the public network by using VLAN to separate the network?

HAIP
1
HAIP
3
Installation and Upgrade
1/20/2012
49
Installation
Patch Sets are also more than BUG fixe with 11.2.0.x
Upgrade + New Features
Upgrade + Full Installation
And (always) Out-of-place
Patch uses new Oracle Home

First patch set: 11.2.0.2
11.2.0.3 works the same
Less new features
Installation
Installation is meant to be simpler with 11g Release 2
Oracle Universal Installer (OUI)
is the main installation instrument.
Installation tips and tricks in:
MOS note 810394.1 (generic)
See:
RAC Platform Specific Starter Kits and Best Practices
Step by Step for various versions available.
See also:
RAC Platform Generic Load Testing and System Test Plan Outline

For an Oracle VM based deployment see:
Oracle RAC OVM Templates on OTN
Details in MOS note 1185244.1 Pre-Built Grid
Infrastructure and Oracle RAC Templates For Oracle VM
1/20/2012
50
Upgrade and Update
Upgrade to 11g Release 2 (11.2.0.3) recommended
The Upgrade to 11g Release 2 from
previous versions is generally recommended

Things to consider:
MOS note 1312225.1 - Things to Consider
Before Upgrading to 11.2.0.2 Grid Infrastructure
MOS note 1320966.1 - Things to Consider
Before Upgrade to 11.2.0.2 Database Performance
MOS note 1363369.1
Things to Consider Before Upgrading to 11.2.0.3
Grid Infrastructure/ASM
10
0
Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Understanding Oracle RAC Internals

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Understanding Oracle RAC Internals

Uploaded by

Copyright:

Available Formats

1/20/2012

You might also like