Les 09

9
Oracle Clusterware Administration
Copyright 2007, Oracle. All rights reserved.
Objectives
After completing this lesson, you should be able to:

Manually control the Oracle Clusterware stack
Change voting disk configuration
Back up or recover your voting disks
Manually back up OCR
Recover OCR
Replace an OCR mirror
Repair the OCR configuration
Change VIP addresses
Use the CRS framework
Prevent automatic instance restarts
9-2
Oracle Clusterware: Overview
Portable cluster infrastructure that provides HA to RAC

databases and/or other applications:
Monitors applications health
Restarts applications on failure
Can fail over applications on node failure
Protected App A
RAC DB Inst
RAC DB Inst
Listener
Listener
CRS HOME
Protected App B
ORACLE_HOME
ORACLE_HOME
Node 1
CRS HOME
Node 2
Oracle Clusterware
system files
9-3
Node 3
CRS HOME
Oracle Clusterware Run-Time View

init
oprocd
evmd
evmlogger
callout
callout
callout
9-4
ocssd
Voting
disk
crsd
racgwrap
+
racgmain
OCR
racgevtf
racgimon
racgimon
racgimon
action
action
action
Manually Control Oracle Clusterware Stack
Might be needed for planned outages:

# crsctl stop crs -wait
# crsctl start crs -wait
# crsctl disable crs
# crsctl enable crs
9-6
CRS Resources
A resource is a CRS-managed application.

Application profile attributes are stored in OCR:
Check interval
Failure policies
Action script
Privileges
Dependencies

An action script must be able to:

Start the application
Stop the application
Check the application
Life cycle of a resource:

crs_profile
crs_register
crs_start
crs_relocate
9-7
crs_stat
crs_stop
crs_unregister
RAC Resources
$ <CRS HOME>/bin/crs_stat -t
Name
Type
Target State Host
---------------------------------------------------------------ora.atlhp8.ASM1.asm
application ONLINE ONLINE atlhp8
ora.atlhp8.LISTENER_ATLHP8.lsnr application ONLINE ONLINE atlhp8
ora.atlhp8.gsd
ora.atlhp8.ons
ora.atlhp8.vip
ora.atlhp9.ASM2.asm
ora.atlhp9.LISTENER_ATLHP9.lsnr application ONLINE ONLINE atlhp9
ora.atlhp9.gsd
ora.atlhp9.ons
ora.atlhp9.vip
ora.xwkE.JF1.cs
ora.xwkE.JF1.xwkE1.srv
ora.xwkE.JF1.xwkE2.srv
ora.xwkE.db
ora.xwkE.xwkE1.inst
ora.xwkE.xwkE2.inst
9-8
Resource Attributes: Example

$ <CRS HOME>/bin/crs_stat -p ora.JFDB.JFDB1.inst
NAME=ora.JFDB.JFDB1.inst
TYPE=application
ACTION_SCRIPT=/u01/app/oracle/product/10g/bin/racgwrap
ACTIVE_PLACEMENT=0
AUTO_START=1
CHECK_INTERVAL=600
DESCRIPTION=CRS application for Instance
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=atlhp8
PLACEMENT=restricted
REQUIRED_RESOURCES=ora.atlhp8.ASM1.asm
RESTART_ATTEMPTS=5
$ <CRS HOME>/bin/crs_stat t ora.xwkE.xwkE1.inst

Name
Type
Target
State
Host
----------------------------------------------------ora....E1.inst application ONLINE
ONLINE
atlhp8
9-9
Main Voting Disk Function

Node2
Node1
Node3
CSS
CSS
We all see
CSS
Nodes can
see each other.
1&2&3
We all see
1&2&3
We all see
1&2&3
Voting disk
Split-brain
Node2
Node1
Node3 can no longer

communicate through
private interconnect.
Others no longer see
its heartbeats and
evict that node by
using the voting disk.
9 - 11
CSS
I do not see 3
I see 1&2
2 see 1&2
=>
We should
evict 3!
Node3
CSS
1&I see
CSS
1&2
I see 3
Voting disk
Ive been
evicted!
Id better stop
Important CSS Parameters
MISSCOUNT:
Represents network heartbeat timeouts

Determines disk I/O timeouts during reconfiguration
Defaults to 30 seconds
Should not be changed
DISKTIMEOUT:
Represents disk I/O timeouts outside reconfiguration
Defaults to 200 seconds
Can be temporarily changed when experiencing very long I/O
latencies to voting disks:
1. Shut down Oracle Clusterware on all nodes but one.
2. As root on available node, use: crsctl set css disktimeout
3. Reboot available node.
4. Restart all other nodes.
M+1
Can be changed ONLY under explicit guidance from Oracle

Support
9 - 13
Multiplexing Voting Disks
Voting disk is a vital resource for your cluster

availability.
Use one voting disk if it is stored on a reliable disk.
Otherwise, use multiplexed voting disks:
There is no need to rely on multipathing solutions.
Multiplexed copies should be stored on independent
devices.
Make sure that there is no I/O starvation for your voting
disks devices.
Use at least three multiplexed copies.
CSS uses a simple majority rule to decide whether

voting disk reads are consistent:
v = f*2+1
9 - 14
Change Voting Disk Configuration
Voting disk configuration can be changed dynamically.

To add a new voting disk:
# crsctl add css votedisk <new voting disk path>
To remove a voting disk:

# crsctl delete css votedisk <old voting disk path>
If Oracle Clusterware is down on all nodes, use the

force option:
# crsctl add css votedisk <new voting disk path> -force
# crsctl delete css votedisk <old voting disk path> -force
9 - 15
Back Up and Recover Your Voting Disks

Should not be needed. Instead, you should add/remove.
Recommendation is to use symbolic links.
Back up one voting disk by using the dd command.
After Oracle Clusterware installation
After node addition or deletion
Cannot be done online
$ crsctl query css votedisk
$ dd if=<voting disk path> of=<backup path> bs=4k
Recover voting disks by restoring the first one using the

dd command, and then multiplex it if necessary.
If no voting disk backup is available, reinstall Oracle
Clusterware.
9 - 16
OCR Architecture
Node1
Node2
Node3
OCR cache
OCR cache
OCR cache
CRS
process
CRS
process
CRS
process
Client
process
Shared
storage
9 - 17
OCR
primary
file
Client
process
OCR
mirror
file
OCR Contents and Organization

root
css
CRS HOME
SYSTEM
evm
crs
OCR
NODEAPPS
LOG
DATABASE
ASM
DATABASES
ONS
CRS
9 - 19
SERVICE
INSTANCE
Managing OCR Files and Locations: Overview

-export
-import
-repair ocrmirror
-upgrade
-repair ocr
-downgrade
ocrconfig
-backuploc
-showbackup
-manualbackup
-restore
-overwrite
-replace ocrmirror
ocrdump
9 - 20
-replace ocr
ocrcheck
Automatic OCR Backups

The OCR content is critical to Oracle Clusterware.
OCR is automatically backed up physically:
Every four hours: CRS keeps the last three copies.
At the end of every day: CRS keeps the last two copies.
At the end of every week: CRS keeps the last two copies.
$ cd $ORACLE_BASE/Crs/cdata/jfv_clus
$ ls -lt
-rw-r--r-- 1 root root 4784128 Jan 9
-rw-r--r-- 1 root root 4005888 Dec 30
02:54
02:54
22:54
18:54
02:54
02:54
14:54
backup00.ocr
day_.ocr
backup01.ocr
backup02.ocr
day.ocr
week_.ocr
week.ocr
Change the default automatic backup location:

# ocrconfig backuploc /shared/bak
9 - 21
Back Up OCR Manually
Daily backups of your automatic OCR backups to a

different storage device:
Use your favorite backup tool.
On demand physical backups:

# ocrconfig manualbackup
Logical backups of your OCR before and after making

significant changes:
# ocrconfig export file name
Make sure that you restore OCR backups that match

your current system configuration.
9 - 22
Recover OCR Using Physical Backups
1. Locate a physical backup:
$ ocrconfig showbackup
2. Review its contents: # ocrdump backupfile file_name

3. Stop Oracle Clusterware
# crsctl stop crs
on all nodes:
4. Restore the physical OCR backup:
# ocrconfig restore <CRS HOME>/cdata/jfv_clus/day.ocr
5. Restart Oracle Clusterware

# crsctl start crs
on all nodes:
6. Check OCR integrity: $ cluvfy comp ocr -n all
9 - 23
Recover OCR Using Logical Backups

1. Locate a logical backup created using an OCR export.
2. Stop Oracle Clusterware on all nodes:
# crsctl stop crs
3. Restore the logical OCR backup:

# ocrconfig import /shared/export/ocrback.dmp
4. Restart Oracle Clusterware on all nodes:

# crsctl start crs
5. Check OCR integrity: $ cluvfy comp ocr -n all

9 - 24
Replace an OCR Mirror: Example
# ocrcheck
Status of Oracle Cluster Registry is as follows:
Version
:
2
Total space (kbytes)
:
200692
Used space (kbytes)
:
3752
Available space (kbytes) :
196940
ID
: 495185602
Device/File Name
: /oradata/OCR1
Device/File integrity check succeeded
Device/File Name
: /oradata/OCR2
Device/File needs to be synchronized with the other device
# ocrconfig replace ocrmirror /oradata/OCR2
9 - 25
Repair OCR Configuration: Example
1. Stop Oracle Clusterware on Node2:

# crsctl stop crs
2. Add OCR mirror from Node1:

# ocrconfig replace ocrmirror /OCRMirror
3. Repair OCR mirror location on Node2:

# ocrconfig repair ocrmirror /OCRMirror
4. Start Oracle Clusterware on Node2:

# crsctl start crs
9 - 26
OCR Considerations
If using raw devices to store OCR files, make sure they

exist before add or replace operations.
You must be the root user to be able to add, replace, or
remove an OCR file while using ocrconfig.
While adding or replacing an OCR file, its mirror needs
to be online.
If you remove a primary OCR file, the mirror OCR file
becomes primary.
Never remove the last remaining OCR file.
9 - 27
Change VIP Addresses

1. Determine the interface used to support your VIP:
$ ifconfig -a
2. Stop all resources depending on the VIP:

$ srvctl stop instance -d DB -i DB1
$ srvctl stop asm -n node1
# srvctl stop nodeapps -n node1
3. Verify that the VIP is no longer running:

$ ifconfig -a
[+
$ crs_stat
4. Change IP in /etc/hosts and DNS.

9 - 28
Change VIP Addresses

5. Modify your VIP address using srvctl:
# srvctl modify nodeapps -n node1 -A
192.168.2.125/255.255.255.0/eth0
6. Start nodeapps and all resources depending on it:

# srvctl start nodeapps -n node1
7. Repeat from step 1 for the next node.
9 - 29
Change Public/Interconnect IP Subnet

Configuration: Example
Use oifcfg to add or delete network interface information
in OCR:
$ <CRS HOME>/bin/oifcfg getif
eth0 139.2.156.0 global public
eth1 192.168.0.0 global cluster_interconnect
$ oifcfg delif -global eth0

$ oifcfg setif global eth0/139.2.166.0:public
$ oifcfg delif global eth1

3
$ oifcfg setif global eth1/192.168.1.0:cluster_interconnect
$ oifcfg getif
eth0 139.2.166.0 global public
eth1 192.168.1.0 global cluster_interconnect
9 - 30
Third-Party Application Protection: Overview

High Availability framework:
Command-line tools to register applications with CRS
Calls control application agents to manage applications
OCR used to describe CRS attributes for the applications
High Availability C API:

Modify directly CRS attributes in OCR
Modify CRS attributes on the fly
Application VIPs:
Used for applications accessed by network means
NIC redundancy
NIC failover
OCFS:
Store application configuration files
Share files between cluster nodes
9 - 31
Application VIP and RAC VIP Differences
RAC VIP is mainly used in case of node down events:

VIP is failed over to a surviving node.
From there it returns NAK to clients forcing them to
reconnect.
There is no need to fail over resources associated to the
VIP.
Application VIP is mainly used in case of application

down events:
VIP is failed over to another node together with the
application(s).
From there, clients can still connect through the VIP.
Although not recommended, one VIP can serve many
applications.
9 - 32
Use CRS Framework: Overview
1. Create an application VIP, if necessary:

a.
b.
c.
d.
Create a profile: Network data + usrvip predefined script

Register the application VIP.
Set user permissions on the application VIP.
Start the application VIP by using crs_start.
2. Write an application action script that accepts three

parameters:
start: Script should start the application.
check: Script should confirm that the application is up.
stop: Script should stop the application.
9 - 33
Use CRS Framework: Overview
3. Create an application profile:
Action script location

Check interval
Failover policies
Application VIP, if necessary
4. Set permissions on your application.

5. Register the profile with Oracle Clusterware.
6. Start your application by using crs_start.
9 - 34
Use CRS Framework: Example
# crs_profile create AppVIP1 t application \

a <CRS HOME>/bin/usrvip \
o oi=eth0,ov=144.25.214.49,on=255.255.252.0
# crs_register AppVIP1
# crs_setperm AppVIP1 o root
# crs_setperm AppVIP1 u user:oracle:r-x
$ crs_start AppVIP1
9 - 35

#!/bin/sh
VIPADD=144.25.214.49
HTTDCONFLOC=/etc/httpd/conf/httpd.conf
WEBCHECK=http://$VIPADD:80/icons/apache_pb.gif
case $1 in
'start')
/usr/bin/apachectl k start f $HTTDCONFLOC
RET=$?
;;
'stop')
/usr/bin/apachectl k stop
RET=$?
;;
'check')
/usr/bin/wget q delete-after $WEBCHECK
RET=$?
;;
*)
RET=0
;;
esac
exit $RET
9 - 36
# crs_profile create myApp1 t application r AppVIP1 \

7
a myapp1.scr o ci=5,ra=2
# crs_register myApp1
# crs_setperm myApp1 o root
# crs_setperm myApp1 u user:oracle:r-x
10
$ crs_start myApp1
11
9 - 37
Summary
In this lesson, you should have learned how to:

Manually control the Oracle Clusterware stack
Change voting disk configuration
Backup and recover your voting disks
Manually back up OCR
Recover OCR
Replace an OCR mirror
Repair the OCR configuration
Change VIP addresses
Use the CRS framework
Prevent automatic instance restarts
9 - 38
Practice 9: Overview
This practice covers the following topics:

Mirroring the OCR
Backing up and restoring OCR
Multiplexing the voting disk
Using Oracle Clusterware to protect the xclock
application
9 - 39

Les 09

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Les 09

Uploaded by

Copyright:

Available Formats

9

Oracle Clusterware Administration

Copyright 2007, Oracle. All rights reserved.

After completing this lesson, you should be able to:

Copyright 2007, Oracle. All rights reserved.

Oracle Clusterware: Overview

Portable cluster infrastructure that provides HA to RAC

Copyright 2007, Oracle. All rights reserved.

Oracle Clusterware Run-Time View

Copyright 2007, Oracle. All rights reserved.

Manually Control Oracle Clusterware Stack

Might be needed for planned outages:

Copyright 2007, Oracle. All rights reserved.

A resource is a CRS-managed application.

An action script must be able to:

Life cycle of a resource:

Copyright 2007, Oracle. All rights reserved.

Copyright 2007, Oracle. All rights reserved.

Resource Attributes: Example

$ <CRS HOME>/bin/crs_stat t ora.xwkE.xwkE1.inst

Copyright 2007, Oracle. All rights reserved.

Main Voting Disk Function

Node3 can no longer

Copyright 2007, Oracle. All rights reserved.

Important CSS Parameters

Represents network heartbeat timeouts

Can be changed ONLY under explicit guidance from Oracle

Copyright 2007, Oracle. All rights reserved.

Multiplexing Voting Disks

Voting disk is a vital resource for your cluster

CSS uses a simple majority rule to decide whether

Copyright 2007, Oracle. All rights reserved.

Change Voting Disk Configuration

Voting disk configuration can be changed dynamically.

To remove a voting disk:

If Oracle Clusterware is down on all nodes, use the

Copyright 2007, Oracle. All rights reserved.

Back Up and Recover Your Voting Disks

Recover voting disks by restoring the first one using the

Copyright 2007, Oracle. All rights reserved.

Copyright 2007, Oracle. All rights reserved.

OCR Contents and Organization

Copyright 2007, Oracle. All rights reserved.

Managing OCR Files and Locations: Overview

Copyright 2007, Oracle. All rights reserved.

Automatic OCR Backups

Change the default automatic backup location:

Copyright 2007, Oracle. All rights reserved.

Back Up OCR Manually

Daily backups of your automatic OCR backups to a

On demand physical backups:

Logical backups of your OCR before and after making

Make sure that you restore OCR backups that match

Copyright 2007, Oracle. All rights reserved.

Recover OCR Using Physical Backups

1. Locate a physical backup:

2. Review its contents: # ocrdump backupfile file_name

5. Restart Oracle Clusterware

Copyright 2007, Oracle. All rights reserved.

Recover OCR Using Logical Backups

3. Restore the logical OCR backup:

4. Restart Oracle Clusterware on all nodes: