You are on page 1of 89

InfoScale Enterprise

VSE+ Level Training

InfoScale Oracle RAC


Part 1
SFRAC Basics

2 InfoScale Enterprise VSE+ training


SFRAC clusters
An SFRAC cluster enables instances of Oracle to run in parallel on different systems.

SFRAC:
• Supports a maximum of 16 nodes
• Monitors an Oracle instance running on
each system simultaneously
• Manages concurrent access to shared
storage containing database files
• Maintains cluster membership and
provides membership to Oracle
clusterware

3 InfoScale Enterprise VSE+ training


SFRAC benefits

Manageability • Single interface management of storage and file systems


• Support for standardized approach towards application and database management

• Highly scalable and available Oracle RAC instances


Availability
• Enable high-availability using PrivNIC/MultiPrivNIC agents
• Fast-failover of applications with minimum downtime

Performance • Maximize I/O performance using ODM library


• Cluster-wide resource delegation to reduce bottlenecks
• Prevent data corruption using I/O fencing

• Support for array-based thin provisioning


Efficient storage use
• Ability to reclaim unused space
• Efficient optimization of space using deduplication
4 InfoScale Enterprise VSE+ training
Typical cluster configuration
Public
network Nodes

Clients

Cluster
interconnect Fibre SAN

Switches

Storage

5 InfoScale Enterprise VSE+ training


SFRAC in the data center VCS Sybase cluster
VIOM - Management console

Clients

VCS Websphere cluster

Custom
VCS application clusters
BEA

SAP SFRAC cluster

6 InfoScale Enterprise VSE+ training


SFRAC and disaster recovery
Campus cluster DR cluster

Replication

VVR IBM HDS/HP


EMC NetApp Oracle

7 InfoScale Enterprise VSE+ training


Cluster characteristics for resiliency
Nodes + interconnect + storage = cluster
• Homogeneous nodes
– 16 nodes (maximum) in a single cluster
– Memory (RAM)
– Processor (CPU)
– Fibre Channel Host Bus Adapter (HBA)
• Fully connected private network for cluster interconnect
– Direct connection among all VCS, CVM, and CFS instances
– Assured bandwidth and latency
• Shared disks
– Access required from every node
– Dynamic multipathing (DMP) and mirroring for high
availability
8 InfoScale Enterprise VSE+ training
Storage Foundation for Oracle RAC
Oracle RAC

SF RAC
SFRAC provides the infrastructure required
Clusterware and RAC Extensions
by Oracle RAC:
• Cluster-enabled versions of the industry standard VCS
Veritas storage management platform
ODM Library
• Optimizations for Oracle Grid or Clusterware and
Oracle RAC Cluster File System
• Certification by Oracle
Cluster Volume Manager

Server

9 InfoScale Enterprise VSE+ training


SFRAC service groups
SFRAC uses parallel VCS service groups to manage:
• RAC instances and
• Shared data
In this example:
Oracle and CVM, CFS • Resources are divided into two
orasg
resources for data files groups:
- Oracle instances and data files
- SFRAC and Oracle Grid
infrastructure components
• Service group for Oracle (orasg)
depends upon cvm service group.
cvm SFRAC and Oracle
clusterware resources

Oracle multiple-instance parallel configuration example

10 InfoScale Enterprise VSE+ training


Communication infrastructure
and processes

11 InfoScale Enterprise VSE+ training


Communication infrastructure and processes
Clusterware communication
Cache fusion/locks (11g)

VCS cluster state


Oracle Oracle
clusterware clusterware
RAC Cache fusion/locks (10g) RAC

VCS core
G L L G
ODM A L Data file management L A ODM
B T T B
CFS File system metadata CFS

CVM Volume management CVM


VCS core VCS core
• Coordination among the cluster components requires an infrastructure for internode and intranode communication.
• Oracle RAC requires a scalable, low-latency transport mechanism to support reliable messaging between elements within the cluster.
12 InfoScale Enterprise VSE+ training
GAB registration
All SFRAC components register with
GAB to:
Peer group
• Provide a communication channel
Leaving cluster
between modules
– A port address enables peer-to-peer
Cluster join communication.
– Examples:
• HAD to HAD
• vxfen to vxfen
Cluster
Nodes interconnect • Notify modules of cluster membership
changes during:
– Node or module failures
– Joining and leaving events

13 InfoScale Enterprise VSE+ training


Communication channels
CVM CFS ODM Oracle
10g 11g
libodm libskgxn libskgxp libskgxp

vxconfigd
GAB vxfen HAD

kmsg
GLM GMS
VCSMM
LMX
UDP
a b h v w u y f d o
GAB

LLT

Ethernet
Other instances/cluster members
14 InfoScale Enterprise VSE+ training
I/O fencing

15 InfoScale Enterprise VSE+ training


I/O fencing overview
LAN

• Fencing:
̶ Allows access to storage by cluster vxfen GAB vxfen
members only
̶ Denies write access to non-members
• The fencing driver, vxfen: SAN

– Uses coordinator disks to act as a global Registration Coordinator disks


lock device keys

– Directs CVM to carry out actual fencing


operations at the disk group level Registration
and Data disks
reservation

16 InfoScale Enterprise VSE+ training


I/O fencing communication
Port b: vxfen
CVM • Cluster membership changes from
GAB
vxfen
• Fencing actions by CVM:
GAB b – Fences drives using SCSI-3 Persistent
Reservations (PR) commands
LLT
– Ejects all nodes for all shared disk
LLT groups during shutdown or
membership change
GAB b

vxfen

CVM

17 InfoScale Enterprise VSE+ training


Shared storage management

18 InfoScale Enterprise VSE+ training


Cluster Volume Manager (CVM) architecture
LAN
• Provides full VxVM capabilities
Master Slave
• Uses a master/slave design:
– The master handles all shared disk
group configuration changes. GAB
– Commands are shipped to the
master.
– The master role fails over if the node
crashes or leaves. SAN
– Any node has the ability to switch
roles.
– All systems can write to the SAN.

19 InfoScale Enterprise VSE+ training


CVM communication

CVM
• Port v: Kernel messaging
kmsg vxconfigd – Node membership information
– Master node selection/re-selection
GAB v w u y • Port w: vxconfigd configuration daemon messaging
LLT
VxVM shared disk group configuration changes propagate
LLT across the cluster:
GAB v w u y – vxassist
kmsg vxconfigd – vxdg
• Port u: vxconfigd commands shipped from slave to master node
• Port y: Shipping I/O when node loses access to shared storage
CVM

20 InfoScale Enterprise VSE+ training


Cluster File System (CFS) architecture
LAN

Full VxFS capabilities: Primary (/fs1) Secondary (/fs1)


Secondary (/fs2) GAB Primary (/fs2)
• Nodes perform I/O direct to the SAN
• Primary/secondary design:
Primary node: SAN
– Manages file system metadata
– Fails over if the node crashes or leaves.

21 InfoScale Enterprise VSE+ training


Cluster File System communication

Port f: GLM
CFS
• Primarily for cache coherency
GLM • GLM mastered per file system
VCSMM
GAB f • Locking modes:
LLT ̶ Shared
̶ Update
LLT
GAB f ̶ Exclusive
• Cluster-wide cache coherency
GLM
• Bypassed by RAC ODM in SFRAC
CFS

22 InfoScale Enterprise VSE+ training


VCS architecture

23 InfoScale Enterprise VSE+ training


Veritas Cluster Server (VCS) architecture
LAN
• Full VCS capabilities
• High availability daemon (HAD):
– Maintains configuration and state information HAD GAB HAD
– Manages resources using agents

• Automated startup, shutdown, and monitoring of


SFRAC components:
SAN
– CVM shared disk groups
– CFS file systems
– Oracle CRS and database instances

24 InfoScale Enterprise VSE+ training


Cluster Server communication

Agents

Agents • Port h: HAD


Resources – Resource and group configuration and status
HAD VCS core
information
h GAB
– Cluster membership changes from GAB
LLT
• Agents
LLT – Provide control and status of resources to HAD
h GAB
– Communicate only with HAD; not with agents
HAD Agents locally or remotely
Resources

Agents
VCS core
25 InfoScale Enterprise VSE+ training
Thank you!

Copyright © 2019 Veritas Technologies, LLC. All rights reserved. Veritas and the Veritas Logo are trademarks or registered trademarks of Veritas Technologies or its affiliates
in the U.S. and other countries. Other names may be trademarks of their respective owners.
This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or
implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.
InfoScale Enterprise
VSE+ Level Training

InfoScale Oracle RAC


Part 2
Oracle clusterware or RAC extensions

28 InfoScale Enterprise VSE+ training


Veritas Oracle Disk Manager (ODM)
LAN

• Extends Oracle Managed Files for a


cluster environment
ODM GAB ODM
• Handles all I/O, in conjunction with
CVM/CFS
• Bypasses caching and locks at the CFS SAN
layer
• Performs as well or better than standard
Oracle I/O to raw devices

29 InfoScale Enterprise VSE+ training


I/O path using ODM

Database instance 1
• Eliminates double buffering
System Global Area (SGA) • Avoids file system single write locks
DB buffer cache • Supports kernel asynchronous I/O

VxFS file system

Processes

Buffer cache
ODM

30 InfoScale Enterprise VSE+ training


ODM communication
ODM

libodm
Port d ODM:
GMS
• Oracle Managed Files (OMF) communication
GAB d
̶ Identifies Oracle data files
LLT
̶ Is used when creating and deleting files
LLT
GAB d • Group Message Service (GMS) adds cluster capability to
the ODM library
GMS

libodm

ODMODM

31 InfoScale Enterprise VSE+ training


VCS Membership Module (VCSMM)
LAN
• Oracle protects data integrity by enforcing
locking between instances in RAC.
• Oracle obtains cluster or VCSMM GAB VCSMM
instance membership from SFRAC:
– skgxn library
– VCSMM kernel module SAN
• When an instance goes down:
– A recovery of that instance’s redo logs is necessary.
– Locks for data blocks must be reconfigured.

32 InfoScale Enterprise VSE+ training


VCSMM communication
Oracle
libskgxn

VCSMM
• Port o: VCSMM
GAB o
LLT • skgxn

LLT
– Makes ioctl calls to VCSMM
GAB o – Is provided as a linked library with SFRAC
VCSMM

libskgxn

Oracle

33 InfoScale Enterprise VSE+ training


Oracle instance-to-instance communication
LAN
• Cross-node messaging API
• Insulates RAC from hardware and OS specifics
– Instances talk to instances, not to nodes. LMX LLT LMX
• Global Enqueue Service synchronizes access to shared
structures
• Global Cache Service internode data block transport
SAN
– Known as Cache Fusion
• Transports:
̶ Oracle 10g: LMX/LLT
̶ Oracle 11g: UDP

34 InfoScale Enterprise VSE+ training


LLT multiplexer (LMX)
Oracle
10g 11g
libskgxp libskgxp
LLT multiplexer (LMX):
• LLT multiplexer and demultiplexer
LMX
• Oracle instance-to-instance communication:
GAB – Is facilitated by skgxp: system kernel generic interface process
LLT – Makes ioctl calls to LMX
o UDP – Is provided as a linked library with SFRAC
LLT • Used with Oracle 10g RAC
GAB
• Multiple processes on each instance communicate using
LMX one LLT port per instance.

libskgxp libskgxp
10g 11g
Oracle
35 InfoScale Enterprise VSE+ training
Oracle Database concepts

36 InfoScale Enterprise VSE+ training


Database concepts
When providing a database service:
• Clients connect to a database.
• Clients do not care where the database is, as long as they can access their data.
• The database must be highly available to the clients.

Clients Instance Database

The database consists of two components:


• The instance, which consists of processes and memory
• The database, which consists of files on disk storage
37 InfoScale Enterprise VSE+ training
Oracle database components

Data Files

Redo Files

Control Files

Clients Instance Archive Files

Oracle Database

38 InfoScale Enterprise VSE+ training


Oracle instance definition
Oracle instance
System Global Area

Database Shared Redo log


buffer pool buffer
cache Processes
Dnnn

Server DBWn PMON SMON CKPT LGWR ARCn

Data files Control files Redo logs Archive logs


39 InfoScale Enterprise VSE+ training
User connections: Oracle Net Services

Direct hand-off to a dedicated server


3

Server
Client 1 Listener 2 4 Database
process

Direct hand-off to a dispatcher Shared server Shared server


process Process pool
3a
4a
Client 1 Listener 2a Dispatcher
(Dnnn) Database

40 InfoScale Enterprise VSE+ training


Mapping logical and physical structures
Logical Physical
Database

1..n Key relationships between logical and


Tablespaces Data files physical structures:
1..n • A tablespace consists of multiple data files.
Segments • Data files are standard files in the file system
or on raw volumes.

Extents
VxVM/VxFS
db_block_size blocks

41 InfoScale Enterprise VSE+ training


Oracle operations

42 InfoScale Enterprise VSE+ training


Transactions
Oracle instance
Oracle Net
System Global Area
Services
Database Shared Redo log
Listener
buffer pool buffer
cache Processes
Dnnn

Server DBWn PMON SMON CKPT LGWR ARCn

Data files Control files Redo logs Archive logs


43 InfoScale Enterprise VSE+ training
Oracle checkpoints

A checkpoint operation causes all modified blocks of data in the buffer cache to be
written to disk.
Data:
– Is written to buffers for improved performance.
– Is not guaranteed to be in the data file until a checkpoint occurs.

Checkpoint:
– Is written to the redo log buffer and flushed to disk when completed.
– Indicates the point from which Oracle recovers if the instance fails.

More frequent checkpoints minimize recovery time.

44 InfoScale Enterprise VSE+ training


Instance crash recovery

• Database recovery recovers the database to a known state after a crash.


– All committed transactions since the last checkpoint are reapplied.
– All uncommitted transactions are discarded or rolled back.

• Clients or applications are responsible for resubmitting uncommitted data.

45 InfoScale Enterprise VSE+ training


Oracle in an InfoScale Clustered
environment

46 InfoScale Enterprise VSE+ training


InfoScale Storage (SF) for Oracle

OCI client
Oracle Net Services

Listener
Data Files

Oracle
Redo Files
ODM
Control Files
VxFS /u02
Archive Files
VxVM usr2dg

Oracle Database

47 InfoScale Enterprise VSE+ training


InfoScale Enterprise (SFHA) for Oracle

OCI client
Oracle Net Services

Listener
Data Files

Oracle
Redo Files
ODM ODM
Control Files
VxFS /u02 VxFS
Archive Files
VxVM usr2dg VxVM
Oracle Database

48 InfoScale Enterprise VSE+ training


Single instance Oracle failover with SFHA

OCI client
Oracle Net Services

Listener
Data Files

Oracle
Redo Files
ODM ODM
Control Files
VxFS /u02 VxFS
Archive Files
VxVM usr2dg VxVM
Oracle Database

49 InfoScale Enterprise VSE+ training


Storage Foundation Cluster File System for Oracle

OCI client
Oracle Net Services

Listener
Data Files

Oracle
Redo Files
ODM ODM
Control Files
VxFS /u02 /u02 VxFS
Archive Files
VxVM usr2dg usr2dg VxVM
Oracle Database

50 InfoScale Enterprise VSE+ training


Single instance Oracle failover with SFCFS

OCI client
Oracle Net Services

Listener
Data Files

Oracle
Redo Files
ODM ODM
Control Files
VxFS /u02 VxFS
Archive Files
VxVM usr2dg VxVM
Oracle Database

51 InfoScale Enterprise VSE+ training


Thank you!

Copyright © 2019 Veritas Technologies, LLC. All rights reserved. Veritas and the Veritas Logo are trademarks or registered trademarks of Veritas Technologies or its affiliates
in the U.S. and other countries. Other names may be trademarks of their respective owners.
This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or
implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.
InfoScale Enterprise
VSE+ Level Training

InfoScale Oracle RAC


Part 3
Oracle RAC concepts and
terminology

54 InfoScale Enterprise VSE+ training


Oracle RAC database components

Data Files

Oracle instance 1 Redo Files

Control Files

Clients
Oracle instance 2
Archive Files

Oracle Database

55 InfoScale Enterprise VSE+ training


Conceptual view of Oracle RAC

ONS ONS
Server Server
Oracle instance 1 Oracle instance 2
System Global Area System Global Area
Database Shared Redo log Database Shared Redo log
buffer pool buffer buffer pool buffer
cache cache

Processes Processes

Redo logs instance 1


Database Redo logs instance 2

Data files Control files Redo logs Archive logs


56 InfoScale Enterprise VSE+ training
Transactions in Oracle RAC
a) Use the block in the local cache
b) Use the block from the remote cache
Update A→A’ c) Read the block from the database

Oracle instance 1 Oracle instance 2


System Global Area System Global Area
Database Shared Redo log Database Shared Redo log
buffer pool buffer buffer pool buffer
cache A cache A

Processes Processes

Redo logs instance 1


Database Redo logs instance 2
A

Data files Control files Redo logs Archive logs


57 InfoScale Enterprise VSE+ training
Transactions in Oracle RAC (continued)

Assume that the block is read from the database.

Oracle instance 1 Oracle instance 2


System Global Area System Global Area
Database Shared Redo log Database Shared Redo log
buffer pool buffer buffer pool buffer
cache A’ A→A’ cache

Processes Processes

Database
A
A→A’

Data files Control files Redo logs Archive logs


58 InfoScale Enterprise VSE+ training
Global Cache Services
Cache fusion handles data block transfers and block-level
locking between RAC instances.
• Cache fusion:
– Provides the current block directly from the memory cache of the holding instance to the memory cache
of the requesting instance for update requests
– Provides copies of blocks for read-only requests
– Is implemented as the LMS process

• When an instance must access a database block, it:


– Checks the local cache and reads locally
– Checks the cache of other instances and reads from that cache through cache fusion
– Reads the blocks from disk if they are not in the cache

59 InfoScale Enterprise VSE+ training


Global Enqueue Service

Coordinates locking for non-data block Oracle structures

• Oracle uses over 50 enqueues to serialize access to various database structures.


• GES is implemented in the Lock Manager Daemon process (LMD), which handles
remote requests for locks on the structures.

Example:
If the DBA performs change such as modifying structure of a table or index on one
node, the other RAC instances must be notified and prevented from making
concurrent changes to those structures.

60 InfoScale Enterprise VSE+ training


Other Oracle processes for RAC

Global Enqueue Service Monitor (LMON):


– Is notified if cluster membership changes
– Handles reconfiguring of global lock resources
– Is also known as Cluster Group Services (CGS)

Lock Processes (LCK):


– Manages global enqueue requests
– Balances workload using multiple LMSn processes

61 InfoScale Enterprise VSE+ training


Oracle operations

62 InfoScale Enterprise VSE+ training


Checkpoints in Oracle RAC

Checkpoint:
• Records are written to the redo log buffer of each instance when certain events occur.
• Records of instance 1 are written to Thread A of the redo log on disk.
• Records of instance 2 are written to Thread B of the redo log on disk.
• Are written at different points in time for each instance.
Threads:
• Are implemented as separate files.
• With the last checkpoint record, indicates the point from which Oracle recovers if that instance
fails.

Note: At checkpoint time, any modified blocks in the buffer caches are written to the data files on disk.

63 InfoScale Enterprise VSE+ training


Instance recovery

When an instance fails:


• The cluster membership changes to show that the instance has left the cluster.
• Oracle receives the cluster membership change notification and performs recovery.
– A remaining instance replays the redo log thread of the failed instance.
– Committed transactions are applied to a running instance that flushes the data to disk.

Note:
• Redo logs must be located on either CFS or shared raw devices.
• Archive logs must be on CFS file systems.

64 InfoScale Enterprise VSE+ training


Oracle recovery mechanism

Instance failure results in an outage


CLIENT FAN
• Redo Log must be applied
• Uncommitted transactions are rolled back
• Outage Time = Redo Apply Delay
FCF/TAF

Redesigning application for recovery INST


INST
INST
• FAN: Notifies status changes
• FCF/TAF: Creates new connections
• Client must use Oracle Libraries
Buffer Cache Buffer Cache
(In-flight transactions are still rolled back)

Shared Storage Redo Log Database Redo Log

65 InfoScale Enterprise VSE+ training


SFRAC architecture

DB VIPs DNS REPL VBS INST INST

NIC

VCS • CRS controls some resources


DR and Service Controls • OCR stores cluster info

State and Control

CRS Arb. Vote for arbitration


VCS heartbeats • Arbitration handed off to VCS
LISTNR • Grid installed after SFRAC
INTERCON
INST1 Clusterware traffic
ORACLE OCR and
bin Vote disk DBF REDO ARCH
MNT
SFRAC CRS
DISK bin bin

Local FS Cluster File System (ODM)


DMP

66 InfoScale Enterprise VSE+ training


SFRAC interactions
Cluster start-up
• VCS starts
DB VIPs INST INST
• VCS confirms CFS resources are
NIC online
• VCS starts CRS
VCS
• CRS receives cluster membership from VCS
Monitor and Control

CRS Arb.
• VCS starts instance
LISTNR • CRS starts listener
INST1 INTERCON Cluster control
MNT • Start and stop of CRS and RAC instances
DISK • Controls RAC cluster membership
• Disk infrastructure monitoring and remediation
• Monitors database instance and listener health

67 InfoScale Enterprise VSE+ training


Oracle RAC infrastructure requirements

68 InfoScale Enterprise VSE+ training


Oracle RAC infrastructure requirements

• Shared concurrent access to storage:


– Oracle Disk Manager (ODM) support
– An ODM-compliant cluster file system
– Cluster volume management
• Cluster membership management:
– Current member systems
– Joining systems
– Leaving systems
• Communication channels between systems:
– Inter-instance messaging
– Cluster state
– Cache fusion

69 InfoScale Enterprise VSE+ training


Data flow in an SFRAC cluster
LGWR LGWR

DB Instance
DB Instance
ARCn ARCn
CKPT Database CKPT
DBWn DBWn
Data files
Redo logs
ODM ODM
Archive logs
CFS Disk I/O CFS
Control files Disk I/O
CVM CVM
Shared concurrent access storage
GCS GCS

RAC Instance
RAC Instance

LMD LMD
LCK Cache fusion LCK
Lock management
LMS LMS
IPC IPC
70 InfoScale Enterprise VSE+ training
Veritas Storage Foundation for Oracle RAC
OCI client
Oracle Net Services

Listener Listener
and VIPs and VIPs

CRS/Grid CRS/Grid
Cluster interconnect
Instance Instance

ODM ODM

CFS SAN CFS


Database
CVM CVM

Data files
Redo logs
Archive logs
Control files

71 InfoScale Enterprise VSE+ training


VCS in an Oracle RAC environment

72 InfoScale Enterprise VSE+ training


VCS in an Oracle RAC environment
• VCS is used primarily to automate startup and shutdown of:
– SFRAC infrastructure
– Clusterware shared storage and daemons
– Oracle RAC instances
– Oracle database storage

• Automatic or manual management of database instances can be performed outside VCS control, if
desired.
• Private VIPs for clusterware heartbeats and Cache Fusion:
– Are managed by VCS for 10g
– Are managed by Oracle clusterware for 11g

• Clusterware manages listeners and public VIPs for Oracle.

73 InfoScale Enterprise VSE+ training


The cvm service group

• Is configured during SFRAC installation


• Is a parallel service group vxfsckd

• Starts automatically on all nodes cvm


cvm_clus
• Starts CVM ➔ Imports all shared disk
groups
cvm_vxconfigd
• Starts CFS

74 InfoScale Enterprise VSE+ training


The cvm_vxconfigd resource

Resource definition Sample value

• Starts and monitors Service group cvm


vxconfigd
Resource name cvm_vxconfigd
• Is an OnOnly resource type
Resource type CVMVxconfigd
• Is managed by the
CVMVxconfigd agent Key attributes

CVMVxconfigdArgs syslog

75 InfoScale Enterprise VSE+ training


The cvm_clus resource
Resource definition Sample value
• Starts, stops, and monitors CVM:
Service group cvm
– Sets CVM master = first node online
Resource name cvm_clus
– Uses ports u, v, w, and y
Resource type CVMCluster
– Imports shared disk groups
Key attributes
• Requires cluster name and node CVMClustName sfrac1
IDs to match CVMNodeID S1=0, S2=1
CVMTransport gab
• Is managed by the CVMCluster
agent CVMTimeout 200

The CVMClustName value determines which


shared disk groups to import during CVM startup.

76 InfoScale Enterprise VSE+ training


The vxfsckd resource

Sample
Resource definition value
• Starts, stops, and monitors CFS:
Service group cvm
– vxfsckd daemon
– GAB port f Resource name vxfsckd
Resource type CFSfsckd
• Is managed by the CFSfsckd agent Key attributes
None

77 InfoScale Enterprise VSE+ training


Resources for managing clusterware components

78 InfoScale Enterprise VSE+ training


Extending the cvm service group

cssd_resource
Resources are added to cvm to manage
Oracle clusterware:
ora_priv
• Oracle cssd daemon
ocrvote_cfsmount

ocrvote_cvmvoldg vxfsckd • Private VIP for Oracle interconnects

• Cluster file system for Oracle:


– OCR
cvm cvm_clus – Vote disks

cvm_vxconfigd

79 InfoScale Enterprise VSE+ training


The CVMVolDg agent
Resource definition Sample value
• If necessary: Service group cvm
– Imports the disk group Resource name ocrvote_cvmvoldg
– Starts all volumes in the shared disk group
– Activates disk group with the specified Resource type CVMVolDg
activation (sw) Required attributes
– Deports disk group if the CVMDiskGroup ocrvotedg
CVMDeportOnOffline attribute is set CVMVolume ocrvotevol
to 1
CVMActivation sw
– Performs periodic I/O polling on volume
sets by specifying names in Optional attributes
CVMVolumeIoTest attribute CVMDeportOnOffline 1

• Monitors disk group and volumes CVMVolumeIoTest ocrvotevol

80 InfoScale Enterprise VSE+ training


The CFSMount agent
• Mounts cluster file systems with the –o cluster option
• Monitors the file system using the fsclustadm command
• Unmounts cluster file systems
Resource definition Sample value
Service group cvm
Resource name ocrvote_cfsmount
Resource type CFSMount
Required attributes
MountPoint /ocrvote
BlockDevice /dev/vx/dsk/ocrvotedg/ocrvotevol
Optional attributes
Primary S1

81 InfoScale Enterprise VSE+ training


The PrivNIC agent
• Monitors Oracle clusterware interconnect private virtual IP address and determines:
– Which nodes are visible on which LLT links
– Whether the IP address is plumbed
Resource definition Sample value

• Fails IP address to another link if Service group cvm


link status is detected as trouble Resource name ora_priv
and value of agent attribute Resource type PrivNIC
EnableUseTroubleState is Required attributes
set to 1 Device eth1=0, eth2=1,
• Can be configured using eth3=2
installsfrac –configure Address 192.168.12.1@S1
192.168.12.2@S2
NetMask 255.255.255.0

82 InfoScale Enterprise VSE+ training


The MultiPrivNIC agent
Resource definition Sample value
Service group cvm
• Monitors LLT links for Oracle clusterware
Resource name ora_multipriv
• Enables load balancing for Oracle IPC among Resource type MultiPrivNIC
LLT links Required attributes
Device@S1 eth1=0, eth2=1
• Fails the IP address over to another LLT link if Device@S2 eth1=0, eth2=1
the link status is detected as trouble and
Address@S1 192.168.12.1=0,
value of agent attribute 192.168.2.1=0
EnableUseTroubleState is set to 1 192.168.12.2=0,
• Fails an IP address back to the preferred link Address@S2 192.168.2.2=0
• Can be configured using installsfrac – NetMask 255.255.255.0
configure Optional attributes
UseLoadBalance 1

83 InfoScale Enterprise VSE+ training


The Application agent
• Enables clusterware to be placed under VCS control:
• Monitors the ocssd process
• Requires disabling automatic startup of Oracle clusterware

Resource definition Sample value


Service group cvm
Resource name cssd_resource
Resource type Application
Required attributes
StartProgram /opt/VRTSvcs/rac/bin/cssd-online
StopProgram /opt/VRTSvcs/rac/bin/cssd-offline
MonitorProgram /opt/VRTSvcs/rac/bin/cssd-monitor
CleanProgram /opt/VRTSvcs/rac/bin/cssd-clean

84 InfoScale Enterprise VSE+ training


Oracle RAC service groups

85 InfoScale Enterprise VSE+ training


Oracle service group example

• Is a parallel service group oracle

• Relies on cvm to manage infrastructure


elements archive_cfsmnt oradata_cfsmnt

redo_cfsmnt
• Contains CVMVolDg and CFSMount resources
for managing Oracle data

orasg
• Requires setting database to manual startup
oradata_voldg

• Optionally contains an Oracle resource

86 InfoScale Enterprise VSE+ training


Service group dependencies

Online local firm dependency:


orasg • Startup behavior:
– cvm must be online
– orasg can come online only after cvm is online

• Failure behavior
– cvm faults, orasg is taken offline
– orasg faults, cvm remains online

cvm

87 InfoScale Enterprise VSE+ training


Difference between VCS single-instance Oracle and RAC service
group
Listener oracle
oracle
Critical=0 Parallel=1

IP

NIC
Mount CFSMount
Mount CFSMount
Mount CFSMount

Volume Volume Volume

Critical=1 Parallel=0 CVMVolDg

DiskGroup
Oracle RAC SG

VCS single instance All shared disk


group are imported
Oracle SG cvm

88 InfoScale Enterprise VSE+ training


Thank you!

Copyright © 2019 Veritas Technologies, LLC. All rights reserved. Veritas and the Veritas Logo are trademarks or registered trademarks of Veritas Technologies or its affiliates
in the U.S. and other countries. Other names may be trademarks of their respective owners.
This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or
implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.

You might also like