Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

s
) ha

y
m

om ฺ
c

csm Guide
e
t
@
Sun Cluster
3.2 Administration
n
b
e
u
ak Stud
y
(
in this
d
u
al use
m
o Guide
ka e tStudent
g
ban licens
a
b
ES-345 REV C
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

D61742GC20
Edition 2.0

a

Copyright © 2009, Oracle and/or its affiliates. All rights reserved.

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Disclaimer
This document contains proprietary information, is provided under a license agreement containing restrictions on use and
disclosure, and is protected by copyright and other intellectual property laws. You may copy and print this document solely for
your own use in an Oracle training course. The document may not be modified or altered in any way. Except as expressly
permitted in your license agreement or allowed by law, you may not use, share, download, upload, copy, print, display,
perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express
authorization of Oracle.
The information contained in this document is subject to change without notice. If you find any problems in the document,
please report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California 94065 USA. This
document is not warranted to be error-free.
Sun Microsystems, Inc. Disclaimer
This training manual may include references to materials, offerings, or products that were previously offered by Sun
Microsystems, Inc. Certain materials, offerings, services, or products may no longer be offered or provided.Oracle and its
affiliates cannot be held responsible for any such references should they appear in the text provided.
Restricted Rights Notice
If this documentation is delivered to the U.S. Government or anyone using the documentation on behalf of the U.S.
Government, the following notice is applicable:

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

s
) ha

U.S. GOVERNMENT RIGHTS
The U.S. Government’s rights to use, modify, reproduce, release, perform, display, or disclose these training materials are
restricted by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract.

Trademark Notice

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective
owners.
AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro
Devices. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used
under license and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark
licensed through X/Open Company, Ltd.

a

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

This page intentionally left blank.

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

s
) ha

a

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

This page intentionally left blank.

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

s
) ha

a

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Table of Contents
s
) ha

About This Course ...........................................................Preface-xxiii
Course Goals...................................................................... Preface-xxiii
Course Map.........................................................................Preface-xxiv
Topics Not Covered............................................................ Preface-xxv
How Prepared Are You?...................................................Preface-xxvi
Introductions .....................................................................Preface-xxvii
How to Use Course Materials ....................................... Preface-xxviii
Conventions ........................................................................ Preface-xxix
Icons ............................................................................ Preface-xxix

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
a Cluster
Introducing k
Sun™
o Hardware and SoftwareSTUFF ABOUT
t
g
e
NAS PG
an14FUZZY
nsPIC ZONE CLUSTER .........................................1-1
bObjectives
e
...........................................................................................
1-1
c
a
li
b
e
u
l
Relevance.............................................................................................
1-2
k
b
a
a
l y sfer Additional Resources ........................................................................ 1-3
u
d
Defining Clustering ........................................................................... 1-4
ab -tran
High-Availability (HA) Platforms.......................................... 1-4
g
n
n
o
a
n
Platforms for Scalable Applications ....................................... 1-5
ab
Sun Cluster 3.2 Hardware and Software Environment............... 1-6
Sun Cluster 3.2 Hardware Environment ........................................ 1-8
Cluster Host Systems................................................................ 1-9
Using Logical Domains (LDoms) as Cluster Host Systems 1-9
Cluster Transport Interface...................................................... 1-9
Public Network Interfaces ..................................................... 1-10
Boot Disks ................................................................................ 1-11
Administrative Workstation.................................................. 1-12
Cluster in a Box ....................................................................... 1-18
Sun Cluster 3.2 Software Support.................................................. 1-19
Software Revisions.................................................................. 1-20
Types of Applications in the Sun Cluster Software Environment....
.................................................................................................... 1-21
Cluster-Unaware (Off-the-Shelf) Applications................... 1-21
Sun Cluster 3.2 Software Data Service Support .......................... 1-26
HA and Scalable Data Service Support ............................... 1-26

vii
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision C

a

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Exploring the Sun Cluster Software HA Framework................. 1-28
Node Fault Monitoring and Cluster Membership ............. 1-28
Network Fault Monitoring .................................................... 1-28
Application Traffic Striping................................................... 1-29
Global Storage Services ................................................................... 1-31
Global Naming (DID devices)............................................... 1-31
Global Devices......................................................................... 1-32
Device Files for Global Devices ............................................ 1-34
Global File Systems................................................................. 1-35
Shared QFS Shared-Access File System............................... 1-36
Failover (Non-Global) File Systems in the Cluster ............ 1-37
Solaris 10 Zones................................................................................ 1-38
LDoms and Zones ................................................................... 1-38
Sun Cluster Distinct Zone Features...................................... 1-38
Exercise: Guided Tour of the Training Lab.................................. 1-42
Preparation............................................................................... 1-42
Task ........................................................................................... 1-42
Exercise Summary............................................................................ 1-43

s
) ha

y
m

om ฺ
c

csm Guide
e
t Console
@the eCluster
n
b
Exploring Node Console Connectivity
and
u
ak Stud
y
Software............................................................................................
2-1
(
n
s
i
i
Objectives ...........................................................................................
2-1
lud se th
a
Relevance.............................................................................................
2-2
m
u
a
o
Additional k
Resourcest ........................................................................ 2-3
g
n
se Node Consoles ............................................ 2-4
Accessing
the
Cluster
a
n
b
e
a Accessing
lic Serial Port Consoles on Traditional Nodes......... 2-4
b
e
u
l
k rabSun Terminal Concentrator (Sun NTS).................................. 2-7
a
y
Terminal Concentrators ................................................ 2-7
ul nsfe Other
d
Alternatives
to a Terminal Concentrator (TC) for Nodes with
b
a
a
r
t
g
a Serial Port Console.............................................................. 2-8
an non

ab

viii

Accessing the Node Console on Servers with Virtual Consoles
............................................................................................... 2-8
Describing Sun Cluster Console Software for an Administration
Workstation ................................................................................... 2-10
Console Software Installation ............................................... 2-10
Cluster Console Window Variations ................................... 2-10
Cluster Console Tools Look and Feel................................... 2-12
Cluster Console Common Window ..................................... 2-13
Cluster Control Panel ............................................................. 2-14
Configuring Cluster Console Tools............................................... 2-15
Configuring the /etc/clusters File .................................. 2-15
Exercise: Configuring the Administrative Console .................... 2-17
Preparation............................................................................... 2-17
Task 1 – Updating Host Name Resolution.......................... 2-18
Task 2 – Installing the Cluster Console Software............... 2-18
Task 3 – Verifying the Administrative Console Environment .

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision C

a

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

2-19
Task 4 – Configuring the /etc/clusters File................... 2-19
Task 5 – Configuring the /etc/serialports File ............ 2-20
Task 6 – Starting the cconsole Tool .................................... 2-21
Task 7 – Using the ccp Control Panel .................................. 2-21
Preparing for Installation and Understanding Quorum Devices..3-1
Objectives ........................................................................................... 3-1
Relevance............................................................................................. 3-2
Additional Resources ........................................................................ 3-3
Configuring Cluster Servers............................................................. 3-4
Boot Device Restrictions .......................................................... 3-4
Boot Device Partitioning (ZFS-root)....................................... 3-6
Configuring Cluster Storage Connections ..................................... 3-8
Cluster Topologies .................................................................... 3-8
Clustered Pairs Topology ........................................................ 3-9
Single-Node Cluster Topology ............................................. 3-15
Describing Quorum Votes and Quorum Devices ....................... 3-19
Why Have Quorum Voting at All?....................................... 3-19
Failure Fencing ........................................................................ 3-20
Amnesia Prevention ............................................................... 3-20
Quorum Mathematics and Consequences .......................... 3-21
Two-Node Cluster Quorum Devices ................................... 3-22
Clustered-Pair Quorum Disk Devices ................................. 3-23
Pair+N Quorum Disks ........................................................... 3-24
N+1 Quorum Disks................................................................. 3-25
Quorum Devices in the Scalable Storage Topology........... 3-26
Quorum Server Quorum Devices.................................................. 3-29
Preventing Cluster Amnesia With Persistent Reservations....... 3-31
SCSI-2 and SCSI-3 Reservations............................................ 3-34
SCSI-3 Persistent Group Reservation (PGR) ....................... 3-35
SCSI-3 PGR Scenario With More Than Two Nodes........... 3-35
NAS Quorum and Quorum Server Persistent Reservations ....
............................................................................................. 3-37
Intentional Reservation Delays for Partitions With Fewer
Than Half of the Nodes....................................................... 3-37
Data Fencing ..................................................................................... 3-38
Configuring a Cluster Interconnect............................................... 3-40
Point-to-Point Cluster Interconnect...................................... 3-40
Switch-based Cluster Interconnect....................................... 3-41
Cluster Transport Interface Addresses and Netmask ....... 3-41
Choosing the Cluster Transport Netmask Based on
Anticipated Nodes and Private Subnets........................... 3-42
Identifying Public Network Adapters .......................................... 3-45
Configuring Shared Physical Adapters ........................................ 3-46
Configuring the Public Network .......................................... 3-46

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

ix
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision C

a

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Allocating a Different VLAN ID for the Private Network 3-46
Exercise: Preparing for Installation ............................................... 3-47
Preparation............................................................................... 3-47
Task 1 – Verifying the Solaris OS.......................................... 3-47
Task 3 – Selecting Quorum Devices ..................................... 3-48
Exercise Summary............................................................................ 3-52
Installing and Configuring the Sun Cluster Software Framework ...
.................................................................................................. 4-1
Objectives ........................................................................................... 4-1
Relevance............................................................................................. 4-2
Additional Resources ........................................................................ 4-3
Sun Cluster Software Installation and Configuration .................. 4-4
Introduction to Sun Cluster Package Installation ................ 4-4
Sun Cluster Packaging ............................................................. 4-4
Patches for the OS and for the Sun Cluster Software .......... 4-5
Installing the Sun Cluster Packages With the Java™ ES Installer4-6
Prerequisites for Installing Sun Cluster Software ................ 4-6
Configuring the User root Environment............................ 4-13
Sun Cluster Framework Configuration ........................................ 4-14
Understanding the installmode Flag ................................ 4-14
Automatic Quorum Configuration (Two-Node Cluster Only)
........................................................................................... 4-15
Automatic Reset of installmode Without Quorum Devices
(Clusters With More Than Two Nodes Only) ................. 4-16
Configuration Information Required to Run scinstall. 4-16
Variations in Interactive scinstall.................................... 4-20
Configuring the Entire Cluster at Once ............................... 4-20
Typical Installation Compared to Custom Installation ..... 4-21
Configuring Using All-at-Once and Typical Modes: Example . 4-22
Configuring Using One-at-a-Time and Custom Modes: Example
(First Node) .................................................................................... 4-30
Configuring Additional Nodes for One-at-a-Time Method:
Example .......................................................................................... 4-43
Solaris OS Files and Settings Automatically Configured by
scinstall...................................................................................... 4-49
Changes to the /etc/hosts File........................................... 4-49
Changes to the /etc/nsswitch.conf File ......................... 4-49
Modifying /etc/hostname.xxx Files to Include IPMP .. 4-50
Modifying the /etc/vfstab File (dedicated partition for /
global/.devices/node@#)............................................... 4-51
Creating the /etc/notrouter File ...................................... 4-51
Modifying the local-mac-address? EEPROM variable 4-51
Automatic Quorum Configuration and installmode Resetting.....
...................................................................................................... 4-52
Manual Quorum Selection.............................................................. 4-53

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

x

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision C

a

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Verifying DID Devices ........................................................... 4-53
Choosing Quorum and Resetting the installmode Attribute
(Two-Node Cluster)............................................................. 4-54
Performing Post-Installation Verification .................................... 4-59
Verifying General Cluster Status .......................................... 4-59
Exercise: Installing the Sun Cluster Server Software.................. 4-64
Task 1 – Verifying the Environment .................................... 4-64
Task 2 – Updating Local Name Resolution......................... 4-65
Task 3 – Installing the Sun Cluster Packages...................... 4-65
Task 4 – Configuring a New Cluster – The All-Nodes-at-Once
Method .................................................................................. 4-66
Task 5 – Configuring a New Cluster – The
One-Node-at-a-Time Method............................................. 4-67
Task 6 – Verifying an Automatically Selected Quorum Device
(Two-Node Cluster)............................................................. 4-69
Task 7 – Configuring a Quorum Device (Three-Node Cluster
or Two-Node Cluster With No Automatic Selection) .... 4-70
Exercise Summary............................................................................ 4-72

s
) ha

ab

y
m

om ฺ
c

csm Guide
e
t
@ ......................................5-1
n
b
Performing Basic Cluster Administration
e
u
ak Stud
y
Objectives ...........................................................................................
5-1
(
n
s
i
i
Relevance.............................................................................................
5-2
th
lud ........................................................................
e
a
Additional Resources
5-3
us
am
o
Identifying
Cluster
Daemons ........................................................... 5-4
k
t
e
ngCluster
sCommands
Using
................................................................ 5-8
a
n
b
e
c
a
Commands
Relating
to
Basic Cluster Administration ........ 5-8
li
b
e
u
l
k rab Additional Commands............................................................. 5-9
a
y
Cluster Command Self-Documentation ................................ 5-9
ul nsfe Viewing
d
and Administering Cluster Global Properties............. 5-11
b
a
a
r
t
g
Renaming the Cluster............................................................. 5-11
an non

Setting Other Cluster Properties........................................... 5-12
Viewing and Administering Nodes .............................................. 5-13
Viewing Node Status and Configuration ............................ 5-13
Modifying Node Information ............................................... 5-14
Viewing Software Release Information on a Node............ 5-15
Viewing and Administering Quorum .......................................... 5-17
Viewing Quorum Status and Configuration....................... 5-17
Adding and Removing (and Replacing) Quorum Devices.......
........................................................................................ 5-18
Installing a Quorum Server (Outside the Cluster) ............. 5-18
Adding a Quorum Server Device to a Cluster.................... 5-19
Registering NAS Devices....................................................... 5-20
Adding NetApp NAS iSCSI LUN as a Quorum Device ... 5-20
Adding Sun NAS iSCSI LUN as a Quorum Device........... 5-21
Registering NAS Mounted Directories (for Data Fencing: both
NetApp and Sun NAS)........................................................ 5-22

xi
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision C

a

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Viewing and Administering Disk Paths and Settings ................ 5-23
Displaying Disk Paths ............................................................ 5-23
Displaying Disk Path Status .................................................. 5-24
Changing Disk Path Monitoring Settings............................ 5-25
Unmonitoring All Non-Shared Devices and Enabling
reboot_on_path_failure................................................ 5-26
Viewing Settings Related to SCSI-2 and SCSI-3 Disk
Reservations.......................................................................... 5-27
Modifying Properties to use SCSI-3 Reservations for Disks
With Two Paths .................................................................... 5-27
Eliminating SCSI Fencing for Particular Disk Devices...... 5-30
Viewing and Administering Interconnect Components ............ 5-33
Viewing Interconnect Status.................................................. 5-33
Adding New Private Networks ............................................ 5-33
Using the clsetup Command ....................................................... 5-35
Sun Cluster Manager ....................................................................... 5-37
Logging Into the Sun Java Web Console ............................. 5-38
Accessing Sun Cluster Manager ........................................... 5-39
Running Cluster Commands as Non-root User or Role Using Role
Based Access Control (RBAC)..................................................... 5-41
Controlling Clusters ........................................................................ 5-42
Starting and Stopping Cluster Nodes .................................. 5-42
Booting a SPARC Platform Machine With the -x Command
.................................................................................... 5-44
Maintenance Mode Example................................................. 5-48
Modifying Private Network Address and Netmask .................. 5-49
Exercise: Performing Basic Cluster Administration ................... 5-51
Preparation............................................................................... 5-51
Task 1 – Verifying Basic Cluster Configuration and Status......
...................................................................................... 5-51
Task 2 – Reassigning a Quorum Device .............................. 5-52
Task 3 – Adding a Quorum Server Quorum Device ......... 5-52
Task 5 – Changing the Cluster Private IP Address Range 5-54
Exercise Summary............................................................................ 5-57

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

Using VERITAS Volume Manager With Sun Cluster Software .... 6-1
Objectives ........................................................................................... 6-1
Relevance............................................................................................. 6-2
Additional Resources ........................................................................ 6-3
Introducing VxVM in the Sun Cluster Software Environment... 6-4
Exploring VxVM Disk Groups......................................................... 6-5
Shared Storage Disk Groups ................................................... 6-5
VERITAS Management on Local Disks (Optional in VxVM 4.x
and Above).............................................................................. 6-5
Sun Cluster Management of Disk Groups ............................ 6-7
Sun Cluster Global Devices Within a Disk Group ............... 6-7

xii

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision C

a

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

VxVM Cluster Feature Used Only for Oracle RAC ............. 6-8
Initializing a VERITAS Volume Manager Disk ............................. 6-9
Traditional Solaris OS Disks and Cross-Platform Data Sharing
(CDS) Disks ............................................................................. 6-9
Reviewing the Basic Objects in a Disk Group.............................. 6-11
Disk Names or Media Names ............................................... 6-11
Subdisk ..................................................................................... 6-11
Plex ............................................................................................ 6-11
Volume ..................................................................................... 6-12
Layered Volume...................................................................... 6-12
Exploring Volume Requirements in the Sun Cluster Environment .
................................................................................................. 6-13
Simple Mirrors......................................................................... 6-13
Mirrored Stripe (Mirror-Stripe) ............................................ 6-14
Striped Mirrors (Stripe-Mirror)............................................. 6-15
Dirty Region Logs for Volumes in the Cluster ................... 6-16
Viewing the Installation and bootdg/rootdg Requirements in the
Sun Cluster Environment ............................................................ 6-17
Requirements for bootdg/rootdg....................................... 6-17
DMP Restrictions in Sun Cluster 3.2 .................................... 6-18
Installing Supported Multipathing Software...................... 6-19
Installing VxVM in the Sun Cluster 3.2 Software Environment 6-20
Using the installer or installvm Utility........................ 6-20
Manually Using vxdiskadm to Encapsulate the OS Disk . 6-24
Configuring a Pre-Existing VxVM for Sun Cluster 3.2
Software................................................................................. 6-24
Creating Shared Disk Groups and Volumes................................ 6-26
Listing Available Disks .......................................................... 6-26
Initializing Disks and Putting Them into a New Disk Group..
......................................................................................... 6-27
Verifying Disk Groups Imported on a Node ...................... 6-27
Building a Mirrored Striped Volume (RAID 0+1).............. 6-29
Building a Striped Mirrored Volume (RAID 1+0).............. 6-30
Examining Hot Relocation.............................................................. 6-31
Registering VxVM Disk Groups .................................................... 6-32
Using the clsetup Command to Register Disk Groups . 6-33
Viewing and Controlling Registered Device Groups ........ 6-34
Managing VxVM Device Groups .................................................. 6-35
Resynchronizing Device Groups .......................................... 6-35
Making Other Changes to Device Groups .......................... 6-35
Putting a Device Group Offline and Back Online .............. 6-35
Using Global and Failover File Systems on VxVM Volumes .... 6-37
Creating File Systems ............................................................. 6-37
Mounting File Systems........................................................... 6-37
Mirroring the Boot Disk With VxVM............................................ 6-39
Exercise: Configuring Volume Management............................... 6-40

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

xiii
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision C

a

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Task 1 – Selecting Disk Drives .............................................. 6-42
Task 4 – Adding vxio on Any Non-Storage Node on Which
You Have Not Installed VxVM.......................................... 6-44
Task 5 – Rebooting All Nodes............................................... 6-44
Task 7– Configuring webdg ................................................... 6-46
Task 9– Creating a Global nfs File System ......................... 6-48
Task 10 – Creating a Global web File System ...................... 6-48
Task 11 – Testing Global File Systems ................................. 6-49
Task 12 – Managing Disk Device Groups ........................... 6-49
Task 13 (Optional) – Viewing and Managing VxVM Device
Groups Using Sun Cluster Manager ................................. 6-52
Exercise Summary............................................................................ 6-53

s
) ha

Using Solaris Volume Manager With Sun Cluster Software ........ 7-1
Objectives ........................................................................................... 7-1
Relevance............................................................................................. 7-2
Additional Resources ........................................................................ 7-3
Exploring Solaris Volume Manager Disk Space Management ... 7-4
Solaris Volume Manager Partition-Based Disk Space
Management ........................................................................... 7-4
Solaris Volume Manager Disk Space Management With Soft
Partitions ................................................................................. 7-5
Exploring Solaris Volume Manager Disksets ................................ 7-8
Solaris Volume Manager Multi-Owner Disksets (for Oracle RAC) .
.................................................................................................... 7-9
Using Solaris Volume Manager Database Replicas (metadb
replicas) .......................................................................................... 7-10
Local Replica Management ................................................... 7-10
Shared Diskset Replica Management................................... 7-11
Shared Diskset Replica Quorum Mathematics................... 7-11
Shared Diskset Mediators ...................................................... 7-12
Installing Solaris Volume Manager and Tuning the md.conf File...
.................................................................................................. 7-13
Modifying the md.conf File (Solaris 9 OS Only)................ 7-13
Initializing the Local metadb Replicas on Local Disks ............... 7-14
Using DIDs Compared to Using Traditional c#t#d#........ 7-14
Leaving Dedicated Partitions Even When Using ZFS-root ......
......................................................................................... 7-14
Adding the Local metadb Replicas to the Local Disks ...... 7-15
Repartitioning a Mirror Boot Disk and Adding metadb
Replicas.................................................................................. 7-15
Using the metadb or metadb -i Command to Verify metadb
Replicas.................................................................................. 7-15
Creating Shared Disksets and Mediators ..................................... 7-17
Automatic Repartitioning and metadb Placement on Shared
Disksets.................................................................................. 7-18

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

xiv

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision C

a

...................................................................................................... Inc................... 7-27 Mounting File Systems............................................................................................................................................................................................................... 7-29 Building Volumes for Root Partition ............. All Rights Reserved...................................... 7-22 Checking Volume Status.............................................. Revision C a ...................................... 7-28 Verifying Partitioning and Local metadbs....... 7-31 Exercise: Configuring Solaris Volume Manager .......................Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010....... 7-40 Task 9 – Viewing and Managing Solaris Volume Manager Device Groups Using Sun Cluster Manager ................... 8-4 Using ZFS with Sun Cluster: Volume Management.................................................................8-1 Objectives ........................................ 7-26 Other Changes to Device Groups ............................................. 7-30 Running the Commands.. 7-42 s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Using ZFS With Sun Cluster Software. 8-4 Using ZFS with Sun Cluster: Eliminating /etc/vfstab Entries .............................................................................. 7-36 Task 3 – Configuring Solaris Volume Manager Diskset and Volumes for Web ................. 7-32 Task 1 – Initializing the Solaris Volume Manager Local metadb Replicas .............. 7-34 Task 3 – Configuring Solaris Volume Manager Diskset and Volumes for NFS .................................................................. 7-24 Managing Solaris Volume Manager Device Groups .............. 7-20 Building Volumes in Shared Disksets With Soft Partitions of Mirrors ....................................................... 7-27 Creating File Systems ........................................... Sun Learning Services........... 7-34 Task 2 – Selecting the Solaris Volume Manager Demo Volume Disk Drives......................... 8-4 xv Copyright 2009 Sun Microsystems................ 8-1 Relevance..... 8-3 ZFS for Sun Cluster Failover Data................................................ 7-27 Using Solaris Volume Manager to Mirror the Boot Disk ............ 7-22 Managing Solaris Volume Manager Disksets and Sun Cluster Device Groups ..................................... 7-41 Exercise Summary...................................................................................................................... 7-26 Using Global and Failover File Systems on Shared Diskset Volumes ..................... 7-28 Building Volumes for Each Partition Except for Root ..... 8-2 Additional Resources ........................................... 7-26 Device Group Resynchronization.. 7-37 Task 5 – Creating a Global nfs File System ................................... 7-38 Task 6 – Creating a Global web File System ................................. 7-30 Rebooting and Attaching the Second Submirror .................... 7-21 Using Solaris Volume Manager Status Commands....... Oracle and/or its affiliatesฺ Using Shared Diskset Disk Space . 7-39 Task 8 – Managing Disk Device Groups ................................................................................................................................................................................................................................................................ 7-32 Preparation............................................................. 7-26 Putting a Device Group Offline .....................................................................................................

....................................................... 8-10 Exercise: Configuring Volume Management................. All Rights Reserved............ 9-5 ud Concepts lIPMP e a Describing General m Group usRequirements...................................................................................... Oracle and/or its affiliatesฺ Building ZFS Pools and File Systems............................................ 9-11 Network Path Failover .............................................................................. 9-16 Performing Failover and Failback Manually ................................................................................. Sun Learning Services....... 9-14 Configuring the /etc/hostname............. 9-8 a y IPMP Group With Three Members Including a Standby ul nsfe Single d 9-9 b a tra g Two IPMP Groups on Different Subnets.......... 9-7 a b c a Examining IPMP Group Examples ...... 8-6 Quotas and Reservations ... 8-15 s ) ha y m ฺ om ฺ c ฺ Managing the Public Network With IPMP ..... 9-12 Configuring IPMP........................................................................... 8-6 ZFS Snapshots .................. 8-8 ZFS Pool Automatic Failover in the Cluster ........................ 9-3 (y is S n i Introducing IPMP .................................................................. 8-9 Typical ZFS Root Configuration..........2 Administration Copyright 2009 Sun Microsystems............................................ 9-13 Putting Test Addresses on Physical or Virtual Interfaces. 8-13 Task 1 – Create a ZFS Pool and File System for Cluster Data . 8-9 ZFS Root Pool and Sun Cluster.............................................................. 9-19 Using Same Group Names on Different Nodes ....... 8-7 Zpool Ownership: In General and in Sun Cluster ........................... ................................................................. .............................xxx Files for IPMP....................... 9-5 aIPMP o Defining k t e ng ensStandby Configuring Adapters in a Group ...... 9-20 Sun™ Cluster 3... 9-15 Using IPV6 Test Address Only.............. 8-14 Task 3– Manually Migrate your ZFS Pool to Another Node.................................. 9-4 th .......................................... 9-19 Understanding Standby and Failback ..................................................................... 9-13 Examining ifconfig Options for IPMP ..............................................Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010............................................. 9-1 t @ n b Relevance........ 8-14 Task 4 – Examine ZFS Root and Mirror Your Root Pool.... 9-14 Using ifconfig Commands to Configure IPMP ................................................................ 9-10 Describing IPMP .......................................................... Revision C a ................ 9-8 li b e u l k rabSingle IPMP Group With Two Members and No Standby.............................................................................................................................2 Environment.......... 8-5 Growing a ZFS Storage Pool ................................................................ 8-13 Task 2 – Create a Snapshot and then Modify your Data......................... 9-19 Configuring IPMP Before or After Cluster Installation ................... 9-9 n n a no ab xvi Two IPMP Groups on the Same Subnet .................................... 8-8 ZFS Root in Sun Cluster..... 9-18 Configuring IPMP in the Sun Cluster 3............... 9-11 Network Path Failure Detection ........................................................................................................... sm uide 9-1 c e G Objectives ..... 9-12 Network Path Failback................................ 9-2 u ude k a t Additional Resources ... Inc...................................... ...........

............................... 10-7 am o Reviewing Components k t ngFaulteMonitor se Components ........................ ฺm 10-1 m Relevance.................. Inc........................ co eฺ 10-2 ฺ m Additional Resources .................. 10-15 The SUNW........................................................ 10-12 Describing Failover Resource Groups .................................................... 10-21 Configuring Resource and Resource Groups Through Properties..... 10-13 Resources and Resource Types ............ 10-18 Understanding Resource Dependencies and Resource Group Dependencies................. 10-4 ( S n s i i Off-the-Shelf Application. Resource Groups....................... .....................................................LogicalHostname Resource Type................................................................................................ 10-22 xvii Copyright 2009 Sun Microsystems........................................................................................SharedAddress Resource Type .....................................................................................2 Software.................................................10-1 ) ha Objectives ....................... and HA-NFSy.......... Sun Learning Services...... 9-24 Task 2 – Verifying the Adapters for the IPMP Group ..............................Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010............................................. 9-24 Preparation.................................................. Oracle and/or its affiliatesฺ Integrating IPMP Into the Sun Cluster 3... 10-14 Resource Type Versioning....... 9-27 s Introducing Data Services.......... 9-22 Exercise: Configuring and Testing IPMP ...... and the Resource Group b a a r t g Manager..................................................... 9-21 Capabilities of the pnmd Daemon in Sun Cluster 3.. 10-17 HAStoragePlus and ZFS ...................................2 Software Environment .......................................... 10-8 ul nsfe Introducing d Resources....... 10-7 a n b a Introducing lic Data Service Packaging.................................... 10-4 t @ n b Solaris 10 OS Non-GlobalkZones Act as Virtual Nodes for Data u ude a t y Services ................................................................ Revision C a ................................................................. 9-25 Task 3 – Verifying or Entering Test Addresses in the /etc/ hosts File ............ All Rights Reserved............................. 10-6 lu3............... 9-24 Task 1 – Verifying the local-mac-address? Variable .................. 10-9 n non a b Resource Groups ............................. 9-25 Task 4 – Creating /etc/hostname.....xxx Files .............. Installation......................... 10-14 Using Special Resource Types... 9-26 Task 6 – Verifying IPMP Failover and Failback ....................2d Software e a Sun Cluster usof a Data Service Agent .................................................................................. 10-3 cs Guid e Introducing Data Services in the Cluster..................................... 10-18 Generic Data Service....................................... 10-15 Guidelines for Using Global and Failover File Systems. and Registration b e u l 10-8 k rab a y Data Service Packages and Resource Types .............................. 10-15 The SUNW................... 10-4 thData Service Agents ............................................... 10-20 Resource Group Dependencies... 9-21 Summary of IPMP Cluster Integration ...... Resource Groups................... 10-10 a Resource Group Manager.........................

........................... 10-57 Exercise Summary................. Inc....................... 10-58 s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Configuring Scalable Services and Advanced Resource Group xviii Sun™ Cluster 3........... 10-38 Suspended Resource Groups ............................................................ 10-25 Some Significant Resource Group Properties .............. 10-29 Unregistering Types ................................................................................................. 10-31 Configuring a LogicalHostname or a SharedAddress Resource ............................................................................................... 10-26 Specifying Non-Global Zone Names in Place of Node Names.. 10-43 Using the clsetup Utility for Resource and Resource Group Operations.......................2 Administration Copyright 2009 Sun Microsystems... Sun Learning Services............... 10-43 Example of Status Commands for a Single Failover Application .. 10-38 Introduction to Resource State..................... Oracle and/or its affiliatesฺ Standard Resource Properties .............................. 10-22 Some Significant Standard Resource Properties ......................................................................... 10-51 Task 5 – Generating Cluster Failures and Observing Behavior of the NFS Failover ...................................................................... Revision C a ............................... 10-47 Task 2 – Registering and Configuring the Sun Cluster HA-NFS Data Services .................... 10-28 Using the clresourcetype (clrt) Command .............................................. 10-52 Task 9 – Viewing and Managing Resources and Resource Groups Using Sun Cluster Manager ............. 10-34 Displaying Resource Configuration Information...................................... 10-46 Preparation............. 10-38 Introduction to Resource Group State ......................... All Rights Reserved......................................... ......................................... 10-45 Exercise: Installing and Configuring HA-NFS ................................................................... 10-23 Resource Group Properties....... 10-30 Configuring Resource Groups Using the clresourcegroup (clrg) Command ... 10-42 Displaying Resource and Resource Group Status Using the clrg status and clrs status Commands......................................................................................................................... 10-29 Registering Resource Types .......... 10-33 Examples of Using clrslh to Add a LogicalHostname 10-33 Configuring Other Resources Using the clresource (clrs) Command........................................ 10-36 Controlling the State of Resources and Resource Groups ................................................Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.............................................................................................................................. 10-49 Task 4 – Observing Sun Cluster HA-NFS Failover Behavior ............................................. 10-46 Task 1 – Installing and Configuring the HA-NFS Agent and Server ..................... 10-31 Displaying Group Configuration Information .......................................................................................... 10-34 Complete Resource Group Example for NFS ............................................... 10-44 Using the Data Service Wizards in clsetup and Sun Cluster Manager...................... ...

................................................................................ 11-21 Task 2 – Configuring the Apache Environment...... 11-5 File Locking for Writing Data ..........11-1 Objectives ....................................................................................................................................................... 11-2 Additional Resources ..... 11-27 Exercise Summary.................................. 11-9 The Load_balancing_weights Property.............................................. 11-6 Client Affinity......................................... 11-26 Task 6 – Observing Cluster Failures ................... 11-6 Exploring Resource Groups for Scalable Services.......... 11-20 Preparation....... 11-9 The Load_balancing_policy Property ........... 11-18 Exercise: Installing and Configuring Sun Cluster Scalable Service for Apache......................................... 11-13 Fault Monitor Operations ....... 11-20 Task 1 – Preparing for Apache Data Service Configuration................................................................................. All Rights Reserved........................ 11-3 Using Scalable Services and Shared Addresses............................ 11-5 Using the SharedAddress Resource................... 11-5 File and Data Access........................................................................... 11-13 Resource Group Operations ......... 11-10 The Resource_dependencies Property .................. ................................ 11-16 Strong Positive Affinities ........................ 11-12 Controlling Scalable Resources and Resource Groups............................. 11-25 Task 5 – Verifying Apache Web Server Access ........................... 11-6 Load-Balancing Weights .... 11-17 Strong Positive Affinity With Failover Delegation ........................................ 11-15 Advanced Resource Group Relationships ..................................... 11-11 Reviewing Command Examples for a Scalable Service ..................... 11-7 Understanding Properties for Scalable Groups and Services ....... Revision C a .................................................... Sun Learning Services................................................................................................ 11-16 Weak Positive Affinities and Weak Negative Affinities ......................................................................................... 11-29 s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Performing Supplemental Exercises for xix Copyright 2009 Sun Microsystems................................................................................ Inc. 11-14 Using the clrg status and clrs status Commands for a Scalable Application ... 11-9 The Desired_primaries and Maximum_primaries Properties ..................................................................................... 11-10 Adding Auxiliary Nodes for a SharedAddress Property ............................ 11-4 Exploring Characteristics of Scalable Services................ 11-23 Task 4 – Registering and Configuring the Sun Cluster Apache Data Service ......... 11-26 Task 7 – Configuring Advanced Resource Group Relationships ............................................................................................................................................................................................. 11-1 Relevance...................................................................................Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010........... Oracle and/or its affiliatesฺ Relationships ................................. 11-21 Task 3 – Testing the Server on Each Node Before Configuring the Data Service Resources .............

......................... 12-21 Task 11 – Verifying That Oracle Runs Properly in the Cluster ................................................................................................. 12-6 Exercise 2: Integrating Oracle 11g Into Sun Cluster 3................... 12-4 Task 1 – Configuring and Installing the Zones ....................................2 Software...... 12-13 Task 3C: Create a Shared Storage File System for Oracle Software (ZFS) ............................... 12-14 Task 4 – Preparing the oracle User Environment ........................................................................................................................ 12-5 Task 2 – Booting the Zones .............. 12-22 Exercise 3: Running Oracle 11g RAC in Sun Cluster 3........................................................... .......... 12-30 Task 5................................................... 12-3 Exercise 1: Running a Clustered Scalable Application in Non-global Zones...........................2 Administration Copyright 2009 Sun Microsystems.................... 12-20 Task 10 – Creating Resources and Resource Groups for Oracle .......................................................... 12-29 Task 5.5A (If using VxVM): Creating Shared Volume Manager Framework Resource Group ...................2 Software as a Failover Application ....................................................... 12-20 Task 9 – Installing and Registering Oracle Data Service...............HAStoragePlus Type...................................... 12-33 s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a xx Sun™ Cluster 3............................... 12-15 Task 6 – Running the runInstaller Installation Script 12-15 Task 8 – Registering the SUNW.............. All Rights Reserved....................................................... 12-28 Task 3B (If Using Solaris Volume Manager) – Installing RAC Framework Packages for Oracle RAC With Solaris Volume Manager Multi-Owner Disksets ...........................2 Software ........................................................................ 12-9 Preparation.................. 12-1 Objectives ... 12-14 Task 5 – Disabling Access Control of X Server on the Admin Workstation ................................................ 12-1 Relevance.......................................................Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010............. 12-11 Task 3A – Creating a Shared Storage File System for Oracle Software (VxVM) ................................................................................... 12-23 Preparation...... Inc.......5B (If Using Solaris Volume Manager) – Creating Shared Volume Manager Framework Resource Group12-31 Task 6B – Creating Raw Volumes (Solaris Volume Manager) ..... 12-9 Task 2 – Creating oracle and dba Accounts.............. Revision C a ................................................ 12-12 Task 3B – Creating a Shared Storage File System for Oracle Software (SVM) ...................................................... 12-2 Additional Resources ...................................... Oracle and/or its affiliatesฺ Sun Cluster 3.................... Sun Learning Services.......................................................................... 12-24 Task 1 – Selecting the Nodes That Will Run Oracle RAC12-26 Task 3A (If Using VxVM) – Installing RAC Framework Packages for Oracle RAC With VxVM Cluster Volume Manager................................................................................................ 12-29 Task 4 – Installing Oracle Distributed Lock Manager ................. 12-4 Preparation..

.................. A-2 Setup Port.................................................................... A-12 Using the telnet Command to Manually Connect to a Node ....................................................... A-1 Viewing the Terminal Concentrator .......................................................................................B-2 Installing a Sun StorEdge Multipack Device ................. C-6 Role-Based Access Control ................... A-5 Specifying the Operating System Image ........................................................................ A-3 Terminal Concentrator Setup Programs.......... D-1 Brief Review of RBAC Terminology ... 12-52 Terminal Concentrator ...................B-3 Installing a Sun StorEdge D1000 Array .... Revision C a ......................................................................................................................................................... D-2 xxi Copyright 2009 Sun Microsystems................................... C-1 Quorum Server Software Installation ..................... A-13 Identifying and Resetting a Locked Port .................................................................................................... 12-39 Task 10 – Disabling Access Control on X Server of the Admin Workstation ...... A-6 Setting the Serial Port Variables..... A-6 Setting the Port Password...................................................................................... A-7 Setting a Terminal Concentrator Default Route ........................ 12-46 Task 14 – Verifying That Oracle RAC Works Properly in a Cluster.......... A-12 Using the Terminal Concentrator help Command .. Oracle and/or its affiliatesฺ Task 7 – Configuring Oracle Virtual IPs...... A-11 Troubleshooting Terminal Concentrators ........................................................................................... A-3 Setting Up the Terminal Concentrator................................ A-4 Enabling Setup Mode ............................ D-2 Authorization ................................................... 12-49 Exercise Summary.................................................... C-2 Configuring the Quorum Server...................... 12-39 Task 13 – Create Sun Cluster Resources to Control Oracle RAC Through CRS......................................................................................................................B-7 The nvramrc Editor and nvedit Keystroke Commands.......................................................................... A-12 Using the telnet Command to Abort a Node....................... All Rights Reserved..............................B-11 Quorum Server ..... B-1 Multi-Initiator Overview ................................ D-2 Role......................... Sun Learning Services....................... C-4 Clearing a Cluster From the Quorum Server................................................................................................ A-9 Using Multiple Terminal Concentrators .............................. Inc............... A-4 Setting the Terminal Concentrator Load Source ......................................................................... 12-36 Task 9A – Creating the dbca_raw_config File (VxVM) 12-38 Task 9B – Creating the dbca_raw_config File (Solaris Volume Manager) ............................................................... C-3 Starting and Stopping a Quorum Server Daemon............................................................................................... A-4 Connecting to Port 1 ............ A-13 s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Configuring Multi-Initiator SCSI ............................Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010....................................

..... D-3 Profiles ....2 Environment ................................................................. E-1 LDom Support......................... All Rights Reserved.................................................................... Oracle and/or its affiliatesฺ Assigning Authorizations............ D-3 Command Privileges .......... Revision C a .....................E-3 Guidelines for Multiple LDoms on Same Machine (in Different Clusters)............ D-4 Simplified RBAC Authorizations in the Sun™ Cluster 3.................Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.................................................................................................................... Sun Learning Services............................................................................ D-5 Assigning Sun Cluster Command Privileges to a User.......2 Administration Copyright 2009 Sun Microsystems......... D-3 The Basic Solaris™ User Profile.................................................... D-6 Assigning Sun Privileges to a Role...............................................................E-2 LDom Configuration for Sun Cluster .......E-3 Guidelines for LDom Configuration..................1 Environment (for Backward Compatibility) ..E-7 s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a xxii Sun™ Cluster 3. D-4 RBAC Relationships ....... Inc..................................................................................................................................................................................... D-5 RBAC in the Sun Cluster 3.................................. D-6 Logical Domains (LDoms) Guidelines .............

including ZFS ● Configure a failover application (Network File System (NFS)) ● Configure a load-balanced application (Apache) ● Configure applications in non-global zones ● Configure HA-Oracle and Oracle Real Application Clusters (RAC) Preface-xxiii Copyright 2009 Sun Microsystems.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. Sun Learning Services. Understand the prerequisites for y ( S into understanding with particular attention quorum devices d his t u l e a theuSun s Cluster 3. Oracle and/or its affiliatesฺ Preface About This Course Course Goals s ) ha y m ฺ omandฺ Describe the major Sun™ Cluster software components c ฺ functions csm Guide e t console software @the cluster Configure access to node consoles and n b e u akinstalling tudSun Cluster software. you should be able to: ● ● ● ● ● ● ● ● ● environment ● Use resources and resource groups to manage clustered applications ● Use storage resources to manage both global and failover file systems. Revision C a .2 b a Upon completion of this course. All Rights Reserved.2 software Install and configure m a o k t e framework Manage ngthe Sun sCluster a n b e c a liVERITAS Configure Volume Manager for Sun Cluster data b e u l k b ra ya feConfigure Solaris™ Volume Manager for Sun Cluster data l u s d ab -tran Configure ZFS for Sun Cluster data g n non a Manage public network adapters using IPMP in the Sun Cluster 3.

Revision C a . Resource Groups.2 Software Preface-xxiv Sun™ Cluster 3. Oracle and/or its affiliatesฺ Course Map The following course map enables you to see what you have accomplished and where you are going in reference to the course goals. Sun Learning Services.Course Map Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. and HA-NFS Configuring Scalable Services and Advanced Resource Group Relationships Supplemental Exercises Performing Supplemental Exercises for Sun Cluster 3. All Rights Reserved.2 Administration Copyright 2009 Sun Microsystems. Product Introduction Introducing Sun™ Cluster Hardware and Software s ) ha Installation ab y m ฺ om ฺ Exploring Node Console Installing and ฺConfiguring Preparing for Installation c mCluster Connectivity and the the s and Understanding ide c Sun u Cluster Console Software e Software Framework Quorum Devices G t @ n b ku tude a y Operation n ( this S i d lu se a m toBasic u aPerforming k e ng eAdministration sCluster a n b a lic b e u l k rab a y ul nsfe Customization d b a a r -t Using ang non Using VERITAS Volume Manager for Volume Management Managing the Public Network With IPMP Using ZFS Solaris™ Volume Manager For Volume Mangement Introducing Data Services. Inc.

Sun Learning Services. All Rights Reserved.Topics Not Covered Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ● Solaris OS administration Refer to the Sun Services catalog for specific information and registration. ● Network administration. Many of the topics listed on the overhead are described in other courses offered by Sun Services: ● Solaris™ 10 Operating System (OS) Advanced Features or Differences. Revision C Preface-xxv a . Inc. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a About This Course Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Topics Not Covered This course does not cover the following topics.

installing patches. and mirror synchronization? ● Can you perform basic Solaris OS administration tasks. such as mirroring. can you answer yes to the following questions? ● Can you explain virtual volume management terminology. such as using tar and ufsdump commands. All Rights Reserved. and safe handling practices? ecs G t @ n b ku tude a y n ( this S i d lu se a am to u k ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n no aba ● Preface-xxvi Sun™ Cluster 3. Oracle and/or its affiliatesฺ To be sure you are prepared to take this course. Inc. Sun Learning Services.2 Administration Copyright 2009 Sun Microsystems. installing the Solaris OS. creating user accounts. and adding packages? ● s ) ha Do you have prior experience with Sun hardware and the OpenBoot™ programmable read-only memory (PROM) technology? y m ฺ om ฺ c ฺ Are you familiar with general computer hardware. striping. Revision C a . volumes. using vi. concatenation. formatting disk drives. electrostatic m uide precautions.How Prepared Are You? How Prepared Are You? Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

● Name ● Company affiliation ● Title. function. All Rights Reserved. introduce yourself to each other and the instructor. and job responsibility ● Experience related to topics presented in this course ● Reasons for enrolling in this course s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a ● Expectations for this course About This Course Copyright 2009 Sun Microsystems. Revision C Preface-xxvii a . Inc. addressing the items shown in the following bullets. Sun Learning Services. Oracle and/or its affiliatesฺ Introductions Now that you have been introduced to the course.Introductions Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

his t u l a us m a k e to g ban licens a b u le k b a a l y sfer u d ab -tran g an non ● ● ab Preface-xxviii Sun™ Cluster 3. such as a process. Revision C a . ec t Gu @ n aids to convey a b dvisual Visual aids – The instructor might useuseveral e k a form. di ande video. and demonstration. This information should help you learn the knowledge and skills necessary to succeed with the activities. Inc. discussion. ● Objectives – You should be able to accomplish the objectives after completing a portion of instructional content.How to Use Course Materials Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. o an c self-check. such as m exercise. s ) ha y m ฺ Activities – The activities take on various forms. ● Lecture – The instructor will present information specific to the objective of the module. tuVisual aids commonly concept. Objectives support goals and can support other higher-level objectives. Oracle and/or its affiliatesฺ How to Use Course Materials To enable you to succeed in this course. these course materials use a learning model that is composed of the following components: ● Goals – You should be able to accomplish the goals after finishing this course and meeting all of its objectives. Sun Learning Services. animation.2 Administration Copyright 2009 Sun Microsystems. Activities ฺhelp to e facilitate ฺ m d s i mastery of an objective. in a(y visual S n contain graphics.

Conventions Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. or risk of irreversible damage to data. A caution indicates that the possibility of a hazard (as opposed to certainty) might happen. Oracle and/or its affiliatesฺ Conventions The following conventions are used in this course to represent various training elements and alternative learning resources. All Rights Reserved. Inc. Icons Additional resources – Indicates other references that provide additional information on the topics described in the module. depending on the action of the user. Sun Learning Services. About This Course Copyright 2009 Sun Microsystems. l k b a a l y sfer u d ab -tranCaution – Indicates that there is a risk of personal injury from a g n non nonelectrical hazard. or a b a the operating system. csm Guide ! e ? @ ent b u ak Stud y ( in this d u Note – Indicates additional e that can help students but is not al information s m u crucial to their understanding of the concept being described. a icExamples of notational information include keyword lminor b e u shortcuts and system adjustments. software. s ) ha y m ฺ oonmthe ฺcurrent Discussion – Indicates a small-group or class discussion c ฺ topic is recommended at this time. Students kaunderstand to the g e should be able to concept or complete the task without n ens a b this information. Revision C Preface-xxix a .

*. new words or terms. group. These are called class options. methods. example: lu sfor e a am to u 1 import java. system% You have mail.io.Courier italic is used for variables and command-line placeholders n a n that are replaced with a real name or value. Palatino italic is used for book titles. y life cycle l e f u s bd tran a g on. for example: ab Courier bold is used for characters and numbers that you type.Conventions Typographical Conventions Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. and execute rights for filename to world.servlet. programming code.awt. write.*.Dialog class contains Dialog constructor.*. k g nse njavax.servlet. s ) ha y m ฺ om ฺ c ฺ To list the files in this directory. directories. use the rm filename command. 2 import a b ce a 3 import javax. Preface-xxx Sun™ Cluster 3. Courier is also used to indicate programming constructs. Courier italic bold is used to represent variables whose values are to be entered by the student as part of an activity. type: csm Guide e # ls @ ent b u akof programming tud y ( S Courier bold is also used for each line code that is n s i i d h t referenced in a textual description. Sun Learning Services. Inc. and users.2 Administration Copyright 2009 Sun Microsystems. files. Revision C a .http. The java. for example: Use the getServletInfo method to get author information. i l b uNotice bthelejavax. for example: To delete a file. All Rights Reserved. and keywords. Oracle and/or its affiliatesฺ Courier is used for the names of commands. for example: Read Chapter 6 in the User’s Guide. and on-screen computer output. or words that you want to emphasize. such as class names. for example: Type chmod a+rwx filename to grant read.servlet interface is imported to allow access to its k a ra methods (Line 2). for example: Use ls -al to list all files.

2 software e a View the Sun Cluster s m to u katypes Describegthe eof applications in the Sun Cluster 3.2 software data service support k b a a r l y sfeExplore u the Sun Cluster 3.2 software n s a n environment ab lice b u Identify le the Sun Cluster 3.2 software high-availability (HA) d b tran framework a g onn a n Define global storage services differences ab ● ● ● ● ● ● ● ● ● Describe the different Sun Cluster 3. Oracle and/or its affiliatesฺ Module 1 Introducing Sun™ Cluster Hardware and Software s ) ha Objectives y m ฺ om ฺ Upon completion of this module. Revision C a . Inc. you should be able to: c ฺ csm Guide Define the concept of clustering e t @ andesoftware n b u Describe the Sun™ Cluster 3. All Rights Reserved.2 zone features 1-1 Copyright 2009 Sun Microsystems.2 hardware environment k tud a y Senvironment Explain the Sun Clusteri3.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services.2 n (hardware s i d h t support lu3.

Oracle and/or its affiliatesฺ Relevance ? ! Discussion – The following questions are relevant to understanding the content of this module: ● Are there similarities between the redundant hardware components in a standalone server and the redundant hardware components in a cluster? ● Why is it impossible to achieve industry standards of high availability on a standalone server? ● Why are a variety of applications that run in the Sun Cluster software environment said to be cluster-unaware? s ) ha y m ฺ om ฺ c ฺ ide for csm Guprovide What services does the Sun Cluster softwareeframework @ ent all the applications running in the cluster? b u k tud asystems y Do global devices and global file ( Shave uses other than for n s i i scalable applications in u th l thed cluster? e a am to us k ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n a no ● How do cluster-unaware applications differ from cluster-aware applications? ● ● ab 1-2 Sun™ Cluster 3. Inc. Revision C a .2 Administration Copyright 2009 Sun Microsystems. All Rights Reserved.Relevance Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services.

Sun Services.2 11/09).Additional Resources Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 1-3 a . Oracle and/or its affiliatesฺ Additional Resources Additional resources – The following references provide additional information on the topics described in this module: ● Sun Cluster System Administration Guide for Solaris OS. All Rights Reserved. part number 8207356 (for Sun Cluster 3. part number 821-0259 (for Sun Cluster 3. part number 820-7458 (for Sun Cluster 3. ● Sun Cluster Concepts Guide for Solaris OS. Inc. ● Sun Cluster Software Installation Guide for Solaris OS.2 11/09) s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems.2 11/09).

2 Administration Copyright 2009 Sun Microsystems. Revision C a .Defining Clustering Defining Clustering Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. 1-4 Sun™ Cluster 3.999 percent uptime for the application or about five minutes of downtime per year. providing paths from at least two nodes in the cluster to each physical storage device that stores data for the applications running in the cluster s ) ha y m ฺ om ฺ c ฺ csm Guide e t @ n b General goal of providing a platformu for HA and scalability for the e ak Stud applications running in the cluster y ( din e thisapplications and cluster-aware Support for a variety oflu cluster-unaware a us applications m a k e to g ban licens a High-Availability b e (HA) Platforms u l k b ya fera l u ns are generally marketed as the only way to provide highbd trClusters a a g on.availability (HA) for the applications that run on them. Many standalone servers themselves are marketed as providing higher levels of availability than our competitors (or predecessors). Clusters generally have the following characteristics: ● Separate server nodes.” This means 99. non-shared. All Rights Reserved. n a n ab ● Cluster software framework. each booting from its own. Oracle and/or its affiliatesฺ Clustering is a general term that describes a group of two or more separate servers operating as a harmonious unit. HA Standards HA standards are usually phrased with wording such as “provides 5 nines availability. copy of the OS ● Dedicated hardware interconnect. providing cluster-specific knowledge to the nodes in the cluster about the health of the hardware and the health and state of their peer nodes ● ● HA can be defined as a minimization of downtime rather than the complete elimination of downtime. providing a private transport only between the nodes of the same cluster ● Multiported storage. Most true standards of HA cannot be achieved in a standalone server environment. Sun Services. One clean server reboot often already exceeds that amount of downtime.

Platforms for Scalable Applications Clusters also provide an integrated hardware and software environment for scalable applications. to toprocessing g e example memory central unit (CPU) failure. and so on. but also planned outages. All Rights Reserved. HA Benefits for Unplanned and Planned Outages The HA benefit that cluster environments provide involves not only hardware and software failures. such as a Solaris OS kernel panic or an application failure. while fault-tolerant servers can hide any hardware g n n o a n failure. application services and data are recovered automatically (without human intervention) and quickly (faster than a server reboot). normal reboots for hardware maintenance in the cluster only affects the uptime of the applications for as much time as it takes to manually relocate the applications to different servers in the cluster. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u Fault-Tolerant Servers Are Notya ankAlternative tud to HA Clusters ( S in this d u l Many vendors provideaservers that e are marketed as fault-tolerant. Inc. In fact. As such. or that a fault-tolerant server supersedes HA in ab -tranalternative some way. These s m u servers are designed to be able tolerate any single hardware failure. Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. as previously stated. Oracle and/or its affiliatesฺ How Clusters Provide HA (Inter-Node Failover) Clusters provide an environment where. The existence of the redundant servers in the cluster and redundant server-storage paths makes this possible. Scalability is defined as the ability to increase application performance by supporting multiple instances of applications on different nodes in the cluster. Sun Services. it can also manually relocate services for planned outages. for kafailure. These instances are generally accessing the same data as each other. might already exceed maximum downtime permitted by the HA standards to which you aspire. lic b e u l k rab a y l There e is a common misconception that fault-tolerant servers are an f u s d to HA clusters.Defining Clustering Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Recovery in these circumstances on a single fault-tolerant server might still require a full OS reboot which. Revision C 1-5 a . in the case of any single hardware or software failure in the cluster. they are not designed to provide especially fast recovery in the ab case of a software failure. n ens a b without a any downtime. While a cluster can automatically relocate applications inside the cluster in the case of failures.

The following features distinguish the Sun Cluster product from competitors in the field: s ) ha y m ฺ om ฺ c ฺ csm Guide e t be physically @ Global device implementation – While data storage must n b e u connected on paths from at least two ak different tudnodes in the Sun y ( S Cluster 3.2 hardware and software all the storage in in thenvironment. 1-6 ● Cluster framework services implemented in the kernel – The Sun Cluster software is tightly integrated with the Solaris 9 OS and Solaris 10 OS kernels. More b u le about global device file naming and location can be k information b a a l y sfound er in ‘‘Global Storage Services’’ on page 1-31. k e to g s This provides to run applications on nodes that use an ctheenflexibility b a data that arelinot even physically connected to the nodes. In scalable applications. you might not need to relocate failed applications because other instances are already running on other nodes. Sun Cluster 3. is d u the cluster is logicallyaavailable l sefrom every node in the cluster using m u standard devicea semantics.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ● Off-the-shelf application support – The Sun Cluster product includes data service agents for a large variety of cluster-unaware applications.2 hardware and software environment is Sun’s latest generation clustering product. Inc. f u d ab -tran Global file system implementation – The Sun Cluster software g n non a framework provides a global file service independent of any b a ● Support for 1 to 16 nodes – Nodes can be added. transport monitoring capability.2 Hardware and Software Environment The Sun Cluster 3. ● ● particular application running in the cluster. Node monitoring capability.2 Hardware and Software Environment Clusters generally do not require a choice between availability and performance. regardless of the storage topology. Revision C a . Oracle and/or its affiliatesฺ Sun Cluster 3. You might still need to perform recovery on behalf of failed instances. and the global device and file system implementation are implemented in the kernel to provide higher reliability and performance. removed.2 Administration Copyright 2009 Sun Microsystems. HA is generally built-in to scalable applications as well as non-scalable ones. so that the same files can be accessed on every node of the cluster. and replaced from the cluster without any interruption of application service. These are tested programs and fault monitors that Sun™ Cluster 3. Sun Services. All Rights Reserved.

A full list of data service agents is in ‘‘Sun Cluster 3.2 framework is aware of Solaris 10 zones and can manage failover and scalable applications running in non-global zones. Sun Services. ● Tight integration with Solaris 10 zones – The Sun Cluster 3. All Rights Reserved. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. Clients outside the cluster see the multiple node instances of the service as a single service with a single IP address.2 Hardware and Software Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.1 Software Data Service Support’’ on page 1-26. ● Support for some off-the-shelf applications as scalable applications with built-in load balancing (global interfaces) – The scalable application feature provides a single Internet Protocol (IP) address and loadbalancing service for some applications. Revision C 1-7 a . such as Apache Web Server and Sun Java™ System Web Server.Sun Cluster 3. Inc. Oracle and/or its affiliatesฺ make applications run properly in the cluster environment.

Inc. Revision C a .2 Hardware Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems.Sun Cluster 3. or Solaris 10 OS (specific Solaris updates listed later in this module) Each node must run the same revision and same update of the OS ● Separate boot disks on each node (with a preference for mirrored boot disks) ● One or more public network interfaces per system per subnet (with a preferred minimum of at least two) s ) ha y m ฺ A redundant private cluster transport interface om ฺ c ฺ Dual-hosted.2 Hardware Environment The Sun Cluster 3. Oracle and/or its affiliatesฺ Sun Cluster 3.2 Hardware Environment Sun™ Cluster 3. Figure 1-1 demonstrates the hardware components of a typical two-node cluster: ● Cluster nodes that are running Solaris 9 OS. All Rights Reserved. mirrored disk storage csm Guide e t method) @ One terminal concentrator (or any other consoleeaccess n b u k tud Administrative workstation (ya n this S i d lu se a Administration am to u Workstation k Network ng ense a b a lic b e u l Public net b Serialak a r y port IPMP ul A nsfe Terminal d group b a Concentrator a r t g an non ● ● ● ● ab Node 1 Redundant Transport Node 2 Boot Disks Boot Disks Multihost Storage Figure 1-1 1-8 Multihost Storage Minimal Sun Cluster 3.2 hardware environment supports a maximum of sixteen nodes. Sun Services.

not on the servers themselves. Inc. Oracle and/or its affiliatesฺ Cluster Host Systems A wide range of Sun server platforms are supported for use in the clustered environment.Sun Cluster 3. All Rights Reserved. These range from small rackmounted servers (SPARC® and x86 servers) up to Sun’s largest enterprise-level servers. different types of servers used as nodes in the same cluster).2 2/08 (Update 1). The transport can be used for the following purposes: ● Cluster-wide monitoring and recovery ● Global data access (transparent to applications) ● Application-specific transport for cluster-aware applications (such as Oracle RAC) It is highly recommended to use two separate private networks that form the cluster transport. unodes am Cluster o k t e and match one or more LDoms and one or more nglegaletonsmix a It is perfectly b c using LDoms in the same cluster. Sun Services. s ) ha ab y m ฺ Using Logical Domains (LDoms) as ClusteroHost m c ฺ ฺ e Systems m d s i ec t Gu @ n with Sun Cluster b beginning e u LDoms are fully supported as clusterknodes. More private networks can provide a performance benefit in certain circumstances because global data access traffic is striped across all of the transports. Revision C 1-9 a . Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. You cannot mix SPARC and x86 servers in the same cluster.2 Hardware Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. d tu domains are supported. Both I/O domains ( n s A discussion of requirements for configuring guest di andebest hipractices t u l a LDoms for use as Sun s is included in an appendix. based on the network and storage host adapters used. ya andSguest 3. You can have more than two private networks (and you can add more later). a nodeslinot physical b e u l k rab a y ul nsfe d b a Cluster tra Transport Interface g n n a no All nodes in a cluster are linked by a private cluster transport. including Sun Fire™ 25K and M-series servers Heterogeneous environments are supported (that is.

You may configure as many IPMP groups as required.2 Administration Copyright 2009 Sun Microsystems. Sun Services. ● ab least two interfaces in each IPMP group. Inlu the environment it is supported t e a only as an interconnect us hosts.2 Hardware Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. 1-10 Sun™ Cluster 3.2 now allows you to build configurations with a single private network forming the cluster transport.2 hardware environment. Inc. Switches are optional when you have two nodes and are required for more than two nodes. A Sun Cluster server is not allowed to act as a router. Oracle and/or its affiliatesฺ Sun Cluster 3. The following types of cluster transport hardware are supported: ● Ethernet (100mb. ng ense a b a lic b e u l kNetwork Public a ab Interfaces r y l e f ns bdu trEach a a node must have public network interfaces under control of the ang non Solaris OS IP Multipathing (IPMP) software. 1gb. It is recommended to have at ● Scalable coherent interface (SCI) intended for remote shared memory (RSM) applications. This would be recommended in production only if the single private network is already redundant (using a lower level device aggregation). and 10gb adapters) ● vnet virtual ethernet adapters in Logical Domains (LDoms) s ) ha y m ฺ om ฺ c ฺ csm Guide Infiniband (Solaris 10 OS only) e @ ent b u This is a relatively new industry standard used outside d k interconnect u a t y of the Sun Cluster environment( for interconnecting a variety of hosts S n Cluster s i diSun h and storage devices. Revision C a .Sun Cluster 3. not between hosts and am between o k t storage devices. All Rights Reserved. Crossover cables are often used in a two-node cluster. The SCI interconnect is supported only in clusters with a maximum of four nodes. Not all available network interface cards (NICs) are supported in the Sun Cluster 3.

Sun Cluster 3. Many other storage configurations support more than two nodes connected to the storage. the first two listed are recommended for their simplicity: g n non ZFS (supported for boot disks starting in Solaris 10 10/08) a b a ● ● Built-in RAID controller for local disks. b e u l k rab a y l Itsisferecommended to mirror boot disks using any of the following u d ab -tranmethods.2 Hardware Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.l the boot disks connected or not visible to e s m u any other node. Sun Services. to ifit the g e Storage Area Network (SAN). VERITAS Volume Manager or Solaris Volume Manager software is typically used to mirror the storage across controllers. ZFS. Oracle and/or its affiliatesฺ Cluster Disk Storage The Sun Cluster hardware environment can use several Sun storage models. Some data storage arrays support only two physically connected nodes. would still be supported if the LUN is n ens a b not visible a to any lic other nodes. All Rights Reserved. The Sun StorEdge™ 6120 array has a single connection and must be used with a hub or a switch. That a is. You can choose not to use any Volume Manager if each node has multipathed access to HA hardware redundant array of independent disks (RAID) storage. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent Boot Disks b u ak Stud y ( The Sun Cluster 3. k For boot device was connected through a a example. Revision C 1-11 a .2 environment disks for each node be in requires d thisarethatnotboot u local to the node. on systems that have it ● Solaris Volume Manager ● Veritas Volume Manager Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. They must all accept multihost connections. Inc.

Sun Services.2 Hardware Environment Terminal Concentrator (or Other Console Access) Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 hardware akas well. All Rights Reserved. It does not affect the bd trnodes a a .cluster in any other way. The software can be a convenience in managing the multiple y fer l u s n of the cluster from a centralized location. Revision C a . This is acceptable (although to t @ n b e u administer) for Sun Cluster 3. ang non ab 1-12 Sun™ Cluster 3.A terminal concentrator (TC) is a device that provides data translation from the network to serial port interfaces.Sun Cluster 3. which can be installed on any SPARC or x86 Solaris OS u l k b a a workstation. c prefer to have only dumb-terminal console access to the cluster ฺ eฺ and m usecurity d s i keep these terminals behind locked doors requiringcstringent eless convenient G checks to open them. Each of the serial port outputs connects to a separate node in the cluster through serial port A. tud y ( S in this d u al use Administrative Workstation m ka e to g s bantheliSun enCluster Included a with software is the administration console c b e software. then you might likely have a terminal concentrator in order to provide the convenience of remote access to your node consoles.m You might o nodes. Oracle and/or its affiliatesฺ Servers supported in the Sun Cluster environment have a variety of console access mechanisms. If you are using a serial port console access mechanism (ttya). Inc.2 Administration Copyright 2009 Sun Microsystems. s ) ha y m ฺ There is always a trade-off between convenience and security.

isolated switches for each private network. In the typical case of a redundant transport. Figure 1-2 on page 1-13 shows an example of a Pair + N topology. All of the nodes in the cluster have access to the data in the shared storage through the global devices and global file system features of the Sun Cluster environment. “Preparing for Installation and Understanding Quorum Devices” for more information about these subjects.2 Hardware Environment Clusters With More Than Two Nodes Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. Sun Services. there must be separate. Oracle and/or its affiliatesฺ In clusters with more than two nodes: ● Switches are required for the transport. Figure 1-2 Pair + N Topology Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. s ) ha y m ฺ om ฺ c ฺ m uide csPublic e Gnet t @ Redundant n b Transport ku tude a y n ( this S i d lu se a am to u k ng ense a b a lic b e u l k rab a y Node Node Node ul nsfeNode d b a tra g n n no aba See Module 3. Each node is connected to each switch to form the redundant transport. ● A variety of storage topologies are supported. Revision C 1-13 a . Inc.Sun Cluster 3.

Sun Cluster nodes can use the following devices: csm Guide e @ ent b iSCSI only: ST2510 u k tud yaStorageTek ( S 5200 and 5300 Series NFS only: NetApp Filer andnSun s i i d h t RAC only): Sun StorEdge 7000 lu for Oracle iSCSI only (NFS allowed e a s am to u Series (Amber k Road) ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n no aba ● ● ● Switch Switch Node 1 Node 2 Node 3 Node 4 Network (not the cluster tranport) NAS Device (Sun or Netapp) Figure 1-3 1-14 Cluster With NAS Device as Only Shared Storage Sun™ Cluster 3. less likely. Sun Services. Revision C a . file traffic is supported on any network except those that make up the cluster interconnect. A NAS device provides file services to all of the cluster nodes through the Network File System (NFS). File and sisk services can run between the cluster nodes and the network storage server on a dedicated subnet or.Sun Cluster 3. Oracle and/or its affiliatesฺ Sun Cluster 3. s ) ha y m ฺ omNASฺ c ฺ In the current release.2 Administration Copyright 2009 Sun Microsystems. another network file system.2 Hardware Environment Cluster With Network Attached Storage (NAS) Devices Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. or a network virtual disk protocol (iSCSI). In other words.2 provides a framework for supporting network-attached storage (NAS) devices as the shared data storage available to the Sun Cluster nodes without requiring any other form of shared storage devices (Figure 1-3). on the same public subnet providing access to the clients of the cluster services. All Rights Reserved. Inc.

a a r y feCluster 3. All Rights Reserved. When Node 1 takes ownership of the data it is reading only its own copy.Sun Cluster 3. . some combination) e a us m ka of Controller-Based to Figure 1-4 ng Diagramse Data Replication a n b e a lic b e u l kThe original b release of Sun Cluster 3. The reasons for this preference can be: ● There might be an existing data replication framework in place.2 Hardware Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. In this variation data is replicated between nodes. and similarly for Node 2 (see Figure 1-4). Oracle and/or its affiliatesฺ Cluster With Controller-Based Data Replication Sun Cluster 3. Sun Cluster 3. public network.2 11/09 (Update 3) adds a a r t g support for Hitachi Universal Replicator. Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. rather than each physical storage array being dual hosted. ● You might benefit from each node having a physically local copy of the data because it can guarantee optimal data read behavior.2 supports a variation on the traditional hardware environment. Inc. Revision C 1-15 a . and using it within the cluster might be cheaper than physically crosscabling all nodes to all storage. Sun Services.2 2/08 (Update 1) added support for EMC Symmetrix ul nSun s d b Remote Data Facility (SRDF). Switch Switch Node 1 Node 2 s ) ha y m ฺ om ฺ Data Replication c ฺ csm Guide e @ ent b u ak Stud y ( Storage inReplication d hisor Medium t u l (Private fiber. n non a b a The data replication topology is ideal for wider area clusters where the data replication solution is preferred to the extra connectivity that would be involved to actually connect the storage to nodes that are far apart.2 supported only Hitachi Truecopy.

such servers could use each physical adapter as both a single transport adapter and a single public network adapter. tointerpreted g e adapters (or device On the host end. The purpose of this feature is to allow certain types of servers that can never have more than two physical network adapters.2 Hardware Environment Cluster With Shared Physical Interconnects Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Using this feature. All Rights Reserved. s ) ha Tagged Virtual Local Area Networks (VLANs) and Adapterฺmy Restrictions com ฺ ฺ e m d s i c support ua This feature makes use of network device driversethat G t @ n how packets b specifies specification called 802. Each virtual adapter a lion b e accepts and sends packets only for a particular VLAN.1q e u d k a trunked) from multiple VLANs can be passed(y (or tuover the same physical S n link by including a VLAN identifier (VLAN-ID) is in network packets. Revision C a . This di ethe thheader VLAN-ID is encapsulateda aslupart of information at the media s m u access control (MAC) by the switches and host alevel and kdrivers). u l k b a a l y sfer u d n of the supported Sun Cluster interconnect devices support asubset ab -trAtagged g VLAN. Sun Services. Both public and private networks can still be fully redundant. such as servers in a blade architecture.1q tagged VLANs.2 Administration Copyright 2009 Sun Microsystems.2 certain types of transport adapters can be used as both public network adapters and private network transport adapters simultaneously.Sun Cluster 3. to be used as Sun Cluster nodes with a fully redundant transport. Inc. Oracle and/or its affiliatesฺ In Sun Cluster 3. The list. 802. multiple virtual adapters n ens a b can be configured c top of a physical adapter. is: n n o a n ab 1-16 ● ce ● bge ● ixge ● ixgb ● e1000g ● ipge ● vnet (in LDoms) Sun™ Cluster 3. at the time of writing this course.

To public net V3 V3 Adapter 1 V1 Adapter 1 Switch 0 V1 a as Node 1 Node y 2) h ฺm m ฺco eฺ m cs GuV3id V3 e @ ent b Adapter 2 u Adapter 2 k tud a y 1 S V2 V2 n (Switch s i i d h lu se t To public net a aV1m to uPublic VLAN V3 k Private VLAN V2 Private VLAN g nse n a ab lice b u le k b a a l y Figure er 1-5 Adapters Being Shared Between Public and Private Nets f u s d ab -tran g n As shown in Figure 1-5. each switch is being used as a private network interconnect.2 Hardware Environment Illustration of Shared Private and Public Networks Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Figure 1-5 is a diagram of a two-node cluster where each node has only two physical network adapters that are capable of tagged VLANs. This requires switches that are capable of tagged VLANs.Sun Cluster 3. Inc. Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. Revision C 1-17 a . In addition. Sun Services. The isolation of traffic between the private network and public network adapters is controlled by the VLAN-ID that is assigned to the adapters themselves. the switches are interconnected to support the ban no two adapters on each node existing in the same IPMP group for public network address failover. All Rights Reserved. which pass the information to the network fabric.

2 Administration Copyright 2009 Sun Microsystems. That is. a o k t ng ense a b a lic Clusterkuinba Box e l a erab y l f u ThenSun s d b Cluster 3. at least one of the following is required. This is true for: ● ● ab ● Servers supporting hardware domains ● Servers supporting LDoms You should take as many redundancy precautions as possible. running multiple domains in different segments on Sun Fire 3x00 through 6x00 servers and segmenting across the power plane on Sun Fire 6x00 servers. Oracle and/or its affiliatesฺ The following items summarize the generally required and optional hardware redundancy features in the Sun Cluster hardware environment: ● Redundant server nodes are required.2 hardware environment supports clusters with both a a r t g an non (or all) nodes in the same physical frame. For You should locate redundant components ak as far tuapart y ( S example.Sun Cluster 3. e s m u and the redundant storage array controllers on two different I/O boards. A cluster formed of multiple LDoms in a single box is particularly susceptible to a single point of failure in the control LDom of the physical machine. the redundant public nets. on a system with multiple (I/O) boards.2 Hardware Environment Sun Cluster Hardware Redundancy Features Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. you in input/output is d h t u l should put the redundanta transport interfaces. Such a cluster is supported but does not provide true high availability. 1-18 Sun™ Cluster 3. ● HA access to data storage is required. for example. ● Mirroring across controllers for Just a Bunch Of Disks (JBOD) or for hardware RAID devices without multipathing ● Multipathing from each connected node to hardware RAID devices s ) ha y m ฺ om ฺ Redundant public network interfaces per subnet are ฺrecommended. Revision C a . Inc. ● Redundant transport is highly recommended. Sun Services. c csm Guide Mirrored boot disks are recommended. All Rights Reserved. e @ ent b u d as possible.

s ) ha y m ฺ Figure 1-6 provides a high-level overview of the softwareocomponents m c ฺ ฺ that work together to create the Sun Cluster software environment. Sun Services.2 Software Support Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. This configuration might not require a software volume manager. e m d s i ec t Gu @ b den u k ya Stu ( Cluster-Unaware n di e this Applications u l a us userland m a o k t gService nse Data nAgents a b ce a i l b u le k b a a r Cluster Framework (userland daemons) l y sfeSun u d ab -tran Sun Cluster Framework (kernel Portions) g n non a b a Solaris OS and Logical Volume Managment Figure 1-6 kernel Sun Cluster Software Layered Model Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems.2 Software Support To function as a cluster member.Sun Cluster 3. the following types of software must be installed on every Sun Cluster node: ● Solaris OS software ● Sun Cluster software ● Data service application software ● Logical volume management An exception is a configuration that uses hardware RAID. All Rights Reserved. Oracle and/or its affiliatesฺ Sun Cluster 3. Revision C 1-19 a .

2 updates earlier than Update 2 (1/09) (each node must run the same version): ● Solaris 9 OS 9/05 or 9/05 HW (Update 8 or 9) (SPARC only) ● Solaris 10 OS 11/06 through 10/08 (Updates 3 through 6) Solaris OS versions for Sun Cluster 3. Sun Services. consult Sun support personnel who have access to the latest support matrices. Revision C a .0 MP3 RP2 and RP3 (SPARC and x86) Solaris Volume Manager revisions – part of the base OS in the Solaris 9 OS and Solaris 10 OS Note – New Solaris 10 OS updates might have some lag before being officially supported in the cluster.Sun Cluster 3.1 u l k rab a y ul nsfe VxVM 5.2 Update 2 (1/09) (each node must run the same version): s ) ha ● Solaris 9 OS 9/05 or 9/05 HW (Update 8 or 9) (SPARC only) ● Solaris 10 OS 5/08 through 5/09 (Updates 5 through 7) y m ฺ m o c ฺ ฺnode Solaris OS versions for Sun Cluster 3. Inc. If there is any question.2 updates earlier t u l Veritas Volume Manager for e a am to us than Update 2k(1/09) e 9 OS/Solaris 10 OS (SPARC) ng4.2 Update 3 (11/09) ● ● VxVM 5.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ The following software revisions are supported by Sun Cluster 3.2 Update 3 (11/09) (each e m d s i must run the same version): ec t Gu @ b 8doren9) (SPARC only) Solaris 9 OS 9/05 or 9/05 HWk(Update u a (Updates tu 7 and 8) y10/09 ( S Solaris 10 OS 5/09 through n i dversions hisSun Cluster 3.0 MP3 RP1 Solaris 9 OS/Solaris 10 OS (SPARC and x86) VxVM 5. 1-20 Sun™ Cluster 3.2 Software Support Software Revisions Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Update 2 g n n a (1/09) no ● ● ● ● ● ● ● ab ● ● ● Veritas Volume Manager Versions for Sun Cluster 3.0 Solaris 9 OS/Solaris 10 OS (SPARC) d b a tra Veritas Volume Manager Versions for Sun Cluster 3.1 e sSolaris a n VxVM MP1 b a licMP2 for Solaris 10 OS (x86) b e VxVM 4. All Rights Reserved.2 software: ● ● Solaris OS versions for Sun Cluster 3.

s ) ha ab y m ฺ om ฺ c ฺ e the Sun idin Two main categories of cluster-unaware applications run csmcan u e G t Cluster 3. ● ● ● ● A data service agent for the application provides the “glue” to make it work properly in the Sun Cluster software environment. Cluster-Unaware (Off-the-Shelf) Applications A cluster-unaware application is an application that typically runs on a single server. These are described on the following pages of @ n b e u d 11.2 environment. Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. Revision C 1-21 a . Inc. This includes methods to start and stop the application appropriately in the cluster. Sun Services. Oracle and/or its affiliatesฺ Types of Applications in the Sun Cluster Software Environment The Sun Cluster software environment supports both cluster-unaware and cluster-aware applications. ang non The majority of applications that run in a cluster are in the clusterunaware category and are part of the main focus of this course.Types of Applications in the Sun Cluster Software Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. which are never started and stopped a by traditional Solaris OS boot methods. as well as fault monitors specific to that application. All Rights Reserved. The applications this module and also in detail in modules ak S10tuand y ( include the following: in this d u Failover applications al use m ka e to Scalableg applications ban licens a bcommon The e elements of all of these applications are the following: u l k b ra cluster’s resource group manager (RGM) coordinates all stopping ya feThe l u s bd tran and starting of the applications.

The client should not even be able to tell that the service is running in a cluster. All Rights Reserved.Types of Applications in the Sun Cluster Software Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. This is an IP address that always fails over from node to node along with the application. s ) ha y m ฺ om ฺ c ฺ idean IP Multiple failover applications in the same resource group share csmcan u e G address. Sun Services. The cluster provides HA by providing automatic restart on the same node or a different node of the cluster. Failover applications run on only one node of the cluster at a time. Oracle and/or its affiliatesฺ Failover Applications The failover model is the easiest to support in the cluster. Inc. ab 1-22 Sun™ Cluster 3. Failover services are usually paired with an application IP address. Revision C a . with the restriction that they must all@ fail overnto the same node t b e u together as shown in Figure 1-7. ak Stud y ( in this d u al use Application IPs m a to Applications g k e an cMounts) ns (Local File bSystem e a li Group bResource e u l k a erab y l u nsf Node 1 Node 2 d b a a r -t ang non Figure 1-7 Multiple Failover Applications in the Same Resource Group Note – Both IPV4 addresses and IPV6 addresses are supported.2 Administration Copyright 2009 Sun Microsystems. In this way. clients outside the cluster see a logical host name with no knowledge of which node a service is running on.

Inc. not every application can be made to run as a scalable application in the Sun Cluster 3. Sun Services. Applications that write data without any type of locking mechanism might work as failover applications but do not work as scalable applications.Types of Applications in the Sun Cluster Software Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 1-23 a . While scalable applications (Figure 1-8) are still off-the-shelf. Oracle and/or its affiliatesฺ Scalable Applications Scalable applications involve running multiple instances of an application in the same cluster and making it look like a single service by means of a global interface that provides a single IP address and load balancing. s ) ha y m ฺ om ฺ c ฺ Client 1 Client 2 Client 3 sm uide Requests Requestsec G t Network @ n b ku tude a y Node 1 n ( this S i d lu se Global Interface a Web xyz.2 software environment.com am to u k Page ng enseTransport a Request b aDistributionlic b Node 2 Node 3 u le k b a a l y sfer u d HTTP ab -tran HTTP HTTP g Application n n o a Application Application n ab <HTML> <HTML> <HTML> </HTML> </HTML> </HTML> Globally Available HTML Documents Figure 1-8 Scalable Application Work Flow Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. All Rights Reserved.

or by hand.2 software framework resource types. Revision C a . The following are supported in the Sun capability b on large e u l k b Cluster environment: a e3. y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( Parallel Database Applications in this d u al type seof cluster application. All Rights Reserved. This can be a highly efficient way for cluster-aware applications to share large amounts of data across the transport. This API maps data from an application instance running on one node into the address space of an instance running on another node. they can be started in their own independent scripts. They differ from cluster-unaware applications in the following ways: ● Multiple instances of the application running on different nodes are aware of each other and communicate across the private transport.2rasoftware y l f bdu transOracle 9i Real Application Clusters (RAC) a g onn Oracle 10g RAC a n ab ● ● ● Oragle 11g RAC Remote Shared Memory (RSM) Applications Applications that run on Sun Cluster 3.2 hardware. Because these applications are cluster-aware. the network connections can be monitored by cluster commands. Oracle and/or its affiliatesฺ Cluster-aware applications are applications where knowledge of the cluster is built-in to the software. Oracle RAC is the only application that is supported in the Sun Cluster 3. If they are.2 software environment that can make use of RSM. 1-24 Sun™ Cluster 3. This requires the SCI interconnect. ● s ) ha Applications are not necessarily logically grouped with external application IP addresses. handling instances of the database g e s database and even providing parallel query nsame different queries the ban on e c a li queries. Sun Services. It is also possible to monitor these cluster-aware applications with Sun Cluster 3.Types of Applications in the Sun Cluster Software Environment Cluster-Aware Applications Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. This is also limited to Oracle 9i and 10gR1 configurations running on Solaris 9 only.2 Administration Copyright 2009 Sun Microsystems. can make use of an application programming interface (API) called Remote Shared Memory (RSM). ● It is not required that the Sun Cluster software framework RGM start and stop these applications. Multiple Parallel databases areaam special u k server to cooperate in the cluster. Inc.

s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. Inc. All Rights Reserved.Types of Applications in the Sun Cluster Software Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 1-25 a . Module 12 contains procedures and an optional exercise for running Oracle 10g RAC (without RSM). Oracle and/or its affiliatesฺ Note – This course focuses on the cluster-unaware applications rather than the types presented on this page. Sun Services.

either in failover or scalable configurations.2 Software Data Service Support This section contains a list of Sun-supported data service agents that make cluster-unaware applications HA.2 Administration Copyright 2009 Sun Microsystems. HA and Scalable Data Service Support The Sun Cluster software provides preconfigured components that support the following HA data services. All Rights Reserved.Sun Cluster 3. Inc. Oracle and/or its affiliatesฺ Sun Cluster 3.2 Software Data Service Support Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. These are the components available at the time of the writing this course: s ) ha y m ฺ Sun Cluster HA for Oracle E-Business Suite (failover) om ฺc eฺ m s Sun Cluster HA for Oracle 9i Application Server (failover) c Guid e t @Serveren(failover Sun Cluster HA for Sun Java System Web or b u d k scalable) ya Stu ( n s Sun Cluster HA for SunuJava Server (failover) di System hiApplication t l e a mSuntJava usSystem Application Server EE HADB Sun Cluster HAa for o k (failover) ng se a n b e Sun a Cluster HA lic for Sun Java System Directory Server (failover) b e u l k Sun rCluster a ab HA for Sun Java System Messaging Server (failover) y l e f bdu transSun Cluster HA for Sun Java System Calendar Server (failover) a Sun Cluster HA for Sun Instant Messaging (failover) ang non ● Sun Cluster HA for Oracle Server (failover) ● ● ● ● ● ● ● ● ab 1-26 ● ● Sun Cluster HA for Sun Java System Message Queue (failover) ● Sun Cluster HA for Apache Web Server (failover or scalable) ● Sun Cluster HA for Apache Tomcat (failover or scalable) ● Sun Cluster HA for Domain Name Service (DNS) (failover) ● Sun Cluster HA for Network File System (NFS) (failover) ● Sun Cluster HA for Dynamic Host Configuration Protocol (DHCP) (failover) ● Sun Cluster HA for SAP (failover or scalable) ● Sun Cluster HA for SAP LiveCache (failover) ● Sun Cluster HA for SAP Web Application Server (failover or scalable) Sun™ Cluster 3. Revision C a .

Oracle and/or its affiliatesฺ Sun Cluster 3.2 Software Data Service Support ● Sun Cluster HA for SAP DB (failover) ● Sun Cluster HA for Sybase ASE (failover) ● Sun Cluster HA for Informix (failover) ● Sun Cluster HA for Siebel (failover) ● Sun Cluster HA for Samba (failover) ● Sun Cluster HA for BEA WebLogic Application Server (failover) ● Sun Cluster HA for IBM Websphere MQ (failover) ● Sun Cluster HA for IBM Websphere MQ Message Broker (failover) ● Sun Cluster HA for MySQL (failover) ● Sun Cluster HA for PostgreSQL (failover) ● Sun Cluster HA for SWIFTalliance Access (failover) s ) ha y m ฺ m o c ฺ ฺ Sun Cluster HA for SWIFTalliance Gateway (failover) e m d s i c Gu Sun Cluster HA for Agfa Impax (failover)e t @ n b e u d Sun Cluster HA for Sun N1 Grid akEngine tu(failover) y ( S inN1 Service Sun Cluster HA for Sun d his Provisioning System (failover) t u l e a Kerberos Sun Cluster HA for m us (failover) a o k t Sun Cluster ng eHA sfore Solaris Containers (Solaris 10 OS zones) a n b a Cluster Sun lic HA for xVM Server Sparc Guest Domains (failover x86 b e u l k virtual a ab machines and Sparc LDOM’s) r y l e f bdu trans a ang non ● ● ● ● ● ● ● ab Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. Inc. Revision C 1-27 a . Sun Services.

The Sun Cluster software adds a layer of monitoring to detect total network failure on one node and to drive the possible failover of applications to another node.2 software environment requires the use of IPMP. The Sun Cluster software framework is implemented as a series of daemons and kernel modules. Cluster Transport Monitoring The cluster transport interfaces are monitored on each node through heartbeats. it is considered as having failed. regardless of which applications are running in the cluster.2 Administration Copyright 2009 Sun Microsystems. a a r t g an non ab Public Network Management The Sun Cluster 3. to control interface failures on a node. If an active cluster transport interface on any node is determined to be inoperative. One advantage of the Sun Cluster software environment is that much of the framework resides in the kernel. All Rights Reserved. and a cluster n i is d h t u l renegotiate reconfiguration is initiatedato e cluster membership. Node Fault Monitoring and Cluster Membership s ) ha y m ฺ The cluster membership monitor (CMM) is kernel-resident on each node om c ฺ ฺ and detects major cluster status changes. s m u ka e to g n ens baMonitoring Network Fault a lic b e u l k rab a y the fepublic network interfaces and the cluster transport interfaces are ul Both s d n b monitored for potential failures. and always memory-resident. If the heartbeat from node tu is not detected within ya any ( S a defined time-out period. Oracle and/or its affiliatesฺ Exploring the Sun Cluster Software HA Framework The Sun Cluster software framework is the software layer that provides generic cluster services to the nodes in the cluster. Revision C a . 1-28 Sun™ Cluster 3. reliable. Inc. where it is fast. a standard Solaris OS feature.Exploring the Sun Cluster Software HA Framework Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. The failure is transparent to Sun Cluster applications. such as loss of communication e m d s i u kernel between one or more nodes. all nodes route interconnect traffic to functional transport interfaces. Sun Services. The CMM relies on the ectransport G t @ module to generate heartbeats across the transport n to other b dmedium e u k nodes in the cluster. Some of the services provided by the framework are described in the following sections.

16.193. Sun Services. This feature stripes IP traffic sent to the per-node logical IP addresses across all private interconnects. Revision C 1-29 a . You would only do this if you wanted to support aba cluster-aware applications inside non-global zones. No manual configuration is required. This network interface is visible with ifconfig. but only needs to be aware of a single IP address on each node configured on that node’s clprivnet0 adapter.1 172. User Datagram Protocol (UDP) traffic is striped on a per-packet basis.193. Oracle and/or its affiliatesฺ Application Traffic Striping Applications written correctly can use the transport for data transfer. Figure 1-9 shows the benefit of application traffic striping to a clusteraware application. Inc. s ) ha y m ฺ om ฺ c ฺ csm Guide e t @ enCluster-aware Cluster-aware application application b u ak Stud y ( in this d u clprivnet0 clprivnet0 al use 172. The cluster framework uses the virtual network device clprivnet0 for these transactions. Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. Transmission Control Protocol (TCP) traffic is striped on a per connection granularity.16. The application receives the benefit of striping across all of the physical private interconnects. At this time Oracle RAC is not supported in non-global zones. All Rights Reserved.Exploring the Sun Cluster Software HA Framework Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 m ka e to g ban licens a b 1-9 le Application Traffic Striping u Figure k a erab y l u nsf d b a tra Note – Non-global zones can optionally be given their own IP addresses on g n n no the clprivnet0 subnet.

Exploring the Sun Cluster Software HA Framework Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. and resource configuration Resource resource c a i l b u le k b a a l y sfer u d ab -tran g an non s ) ha The CCR structures contain the following types of information: ● ● ● ● ● ● ab 1-30 Sun™ Cluster 3. The CCR is automatically modified as the result of administrative command execution. Inc. y m ฺ m o c ฺ ฺ Cluster and node names e m d s i ec t Gu Cluster transport configuration @ b den u k Disk ID (DID) device configuration ya Stu ( n isregistered VERITAS disk Device group configuration di (including h t u l e groups or Solaris Volume a Manager s disksets) m u a o k NAS tdevices Informationgabout e n s ba type. All Rights Reserved. en group. Sun Services. The files contain generation number information that is critical to the operation of the cluster software. Oracle and/or its affiliatesฺ Cluster Configuration Repository General cluster configuration information is stored in global configuration files collectively referred to as the cluster configuration repository (CCR). The CCR must be kept consistent between all nodes and is a critical element that enables each node to be aware of its potential role as a designated backup system.2 Administration Copyright 2009 Sun Microsystems. Caution – Never attempt to modify any of the CCR-related files. Revision C a .

Different local b e c a liuse the same logical name (for example. Oracle and/or its affiliatesฺ Global Storage Services The Sun Cluster software framework provides global storage services. Revision C 1-31 a . It is important to understand the differences and relationships between the following services: ● Global naming (DID devices) s ) ha y m ฺ Global file system om ฺ c ฺ csm Guide e Global Naming (DID devices) kub@ dent ya Stu ( n i his name for every disk drive. Shared disks that might have CD-ROM drive. Inc. a node’s aroot r y l e f bdu trans a ang non ● Global devices ● ab Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. Sun Services. The DID feature provides adunique device t u l e adriveuinsthe cluster. they also provide a much more flexible environment for failover services by freeing applications to run on nodes that are not physically connected to the data. c0t0d0 for each disks b that may e u l k b disk) are each given different unique DID instance numbers. a feature which greatly distinguishes the Sun Cluster software product.Global Storage Services Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. or tape m a knameseontodifferent nodes (different controller numbers) different logical g aa ncluster-wide are given ns unique DID instance number. Not only do these features enable scalable applications to run in the cluster.

All Rights Reserved. Revision C a . 1-32 Sun™ Cluster 3. for partitions in both the /dev/did/dsk lu seand a example: /dev/did/dsk/d2s3 and /dev/did/rdsk/d2s3. This includes individual DID disk devices. as described in Module 3. Sun Services. Oracle and/or its affiliatesฺ Junction Node 1 Node 2 d1=c0t0d0 d2=c2t18d0 c0 d2=c5t18d0 d3=c0t0d0 c2 c5 c0 Node 3 d4=c0t0d0 d5=c1t6d0 d8191=/dev/rmt/0 c0 c1 rmt/0 s ) ha y m ฺ om ฺ c ฺ sm uide c e G Figure 1-10 DID devices t @ n b ku tude a y ( standard S eight Solaris OS disk Device files are created for each iof n the s i d h t /dev/did/rdsk directories. Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. as well as VERITAS Volume Manager volumes and Solaris Volume Manager volumes. “Preparing g n n o a for Installation and Understanding Quorum Devices” n t0d0 Boot Disk t18d0 Array Disk t0d0 Boot Disk t0d0 Boot Disk t6d0 CD-ROM ab DIDs are not used as components of VERITAS Volume Manager volumes. Global Devices The global devices feature of Sun Cluster 3.2 Administration Copyright 2009 Sun Microsystems. Inc.2 software provides simultaneous access to the raw (character) device associated with storage devices from all nodes. e u l k rab a y fe ul DIDs s are used as components of Solaris Volume Manager volumes and in d n b a a trchoosing cluster quorum devices.Global Storage Services Figure 1-10 demonstrates the relationship between typical Solaris OS logical path names and DID instances. am to u k ngare ejust saeglobal naming scheme and not a global access a n DIDs themselves b a lic b scheme. regardless of where the storage is physically attached. CD-ROMs and tapes.

Node 2 becomes the primary node if Node 1 fails.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 software framework manages automatic failover of the primary node for global device groups. Revision C 1-33 a . In Figure 1-11. Inc. Sun Services. all nodes have simultaneous access to the device /dev/vx/rdsk/nfsdg/nfsvol. All nodes use the same device path. s ) ha y m ฺ om ฺ c Node 1 ฺ Primary Node 2 Node 3 Nodec4sm ide u for nfsdg e G t @ n b ku tude a y n ( this S i d lu se a am to u k ng ense a b nfsdg/nfsvol a lic b e u l k rab a y ul nsfe d b a tra g n n Figure 1-11 Node Access Diagram no aba Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. All Rights Reserved. Oracle and/or its affiliatesฺ Global Storage Services The Sun Cluster 3. but only the primary node for a particular device actually talks through the storage medium to the disk device. All other nodes access the device by communicating with the primary node through the cluster transport.

beginning in Sun Cluster 3.tp 1-34 Sun™ Cluster 3. because a theyusare unwieldy. /dev/did/rdsk. Sun Cluster software creates alternate path names under the /dev/global directory that link into the global device space: proto192:/dev/md/nfsds# ls -l /dev/global lrwxrwxrwx 1 root other 34 Nov 6 13:05 /dev/global -> /global/. where nodeID is an integer representing a node in the cluster.. on a virtual partition (lofi device)..devices file systems—one for each node—are visible from each node./. the standard directories /dev/did/dsk.devices/node@2 u ak Stud y ( in this The device names under the /global/.raw proto192:/dev/md/nfsds# ls -l /dev/global/rmt/1 lrwxrwxrwx 1 root root 39 Nov 4 17:43 /dev/global/rmt/1 -> ./. the Sun Cluster 3. or./devices/pseudo/did@8191. Revision C a ../. This file system has the mount point /global/. All Rights Reserved. s ) ha y m ฺ om ฺ c ฺ 81M 5% /global/.2 1/09 (update 2)..2 m a environment provides symbolic k e to links into this namespace. but.6M /dev/lofi/126 94M 3. The global file system feature is described more on the following pages. completely dedicated to storing the device files for global devices.3s0. All of the /global/.devices/node@nodeID arena can d u l e be used directly.devices/node@1 csm Guide e @ ent b 81M 5% /global/. Inc. the Sun bsoftware e u l Cluster links the standard device access directories into the k b a a r y e fnamespace: ul global s d n b a tra ls -l /dev/vx/rdsk/nfsdg proto192:/dev/vx# g n n a root 40 Nov 25 03:57 no 1 root ablrwxrwxrwx # df -h /dev/lofi/127 94M 3.1.. g an cens b a For VERITAS Volume li Manager and Solaris Volume Manager.6M /dev/vx/rdsk/nfsdg ->/global/. Instead. Sun Services. The file system is stored on a dedicated partition on the boot disk./.2 software maintains a special file system on each node. Oracle and/or its affiliatesฺ Device Files for Global Devices The Sun Cluster 3. In other words.2 Administration Copyright 2009 Sun Microsystems../devices/pseudo/did@0:3. they are examples of global file systems.devices/node@nodeID.devices/node@1/dev/vx/rdsk/nfsdg/ For individual DID devices.devices/node@1/dev/global/ proto192:/dev/md/nfsds# ls -l /dev/global/rdsk/d3s0 lrwxrwxrwx 1 root root 39 Nov 4 17:43 /dev/global/rdsk/d3s0 -> .Global Storage Services Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. and /dev/did/rmt are not global access paths.

to the boot disks of other nodes. kub a Stud y ( The VERITAS File System (VxFS) in this d u e(hsfs) The High Sierra File al System s m u ka e to g The Sun Cluster an csoftware ns makes a file system global with a global mount b e a option. However. regardless of their physical location. but can be put on the b Thislline e u command k b of a standard mount command: a a r y fe/dev/vx/dsk/nfs-dg/vol-01 /global/nfs u-ol global s # mount d n b a tra g n n The equivalent mount entry in the /etc/vfstab file is: no aba The global file system capability is independent of the structure of the actual file system layout on disk. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent The UNIX® File System (ufs). All Rights Reserved. you can overwrite it with newfs or dd. islitypically in the /etc/vfstab file. Revision C 1-35 a .Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Global File Systems The global file system feature makes file systems simultaneously available on all nodes.2 to be file systems underlying the global file system. only one node at a time is the primary and actually communicates with the underlying file system. while you cannot mount one node’s root disk from another node. Sun Services. Oracle and/or its affiliatesฺ Global Storage Services Note – You do have raw (character) device access from one node. These are: ● ● ● /dev/vx/dsk/nfs-dg/vol-01 /dev/vx/rdsk/nfs-dg/vol-01 \ /global/nfs ufs 2 yes global The global file system works on the same principle as the global device feature. Inc. The primary node for the file system is always the same as the primary node for the device on which it is built. All other nodes use normal file semantics but actually communicate with the primary over the cluster transport. The global file system is also known by the following names: ● Cluster file system ● Proxy file system Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. It is not necessarily advisable to take advantage of this feature. through the /dev/global device paths. In other words. That is. only certain file system types are supported by Sun Cluster3.

The Shared QFS functionality is presented in the following diagram: s ) ha y m ฺ om ฺ c ฺ Node 1 csm Guide e t2 @ Node Metadata n b e u Server ak Stud y ( in this d u l e File a Data File Data s m u a o k et Metadata g n ba licens Shared QFS a b e u l k rab a y ul nsfe d b a tra g n n no aba Shared QFS has been supported in Sun Cluster 3.2 Administration Copyright 2009 Sun Microsystems. Sun Services. Shared QFS provides access to the file system data only from nodes directly connected to the storage. File data access is provided directly over the storage connections. In the near future it will be supported as a general-purpose shared file system. directories and the like) are still provided by proxy from a single node at a time for a particular filesystem. Inc. for all applications that require simultaneous access to the same data from multiple nodes. Oracle and/or its affiliatesฺ Shared QFS is a different solution to the Sun Cluster global file system that also provides simultaneous access from multiple nodes to the same file system data. rather than by proxy.1 and Sun Cluster 3. although the metadata server functionality can fail over. Unlike the Sun Cluster global file system. 1-36 Sun™ Cluster 3. File system metadata (inodes. All Rights Reserved.Global Storage Services Shared QFS Shared-Access File System Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 only as a filesystem for storing Oracle RAC data and binaries. Revision C a .

These are the supported types for failover file systems: ● UFS ● VxFS ● Sun StorageTek™ QFS software (regular. Failover file systems are available only on one node at a time.information on multiple nodes simultaneously. Sun Services. when used appropriately. There is overhead in r y l e f s global file system infrastructure of maintaining replicated state bdu tranthe a g on. n a n ab ● Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems.2 more file system types are supported as a failover file system than are supported as underlying file systems for a global file system. al uconnected s m ka e to g Failover file system access is not suitable for scalable services. Inc. ban licens a b filelesystem access.Global Storage Services Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. on a node which is running a service and has a physical connection to the storage in question. can have a u Failover k a performance ab benefit over global file system access. non-shared filesystem) s ) ha y m ฺ om ฺ c Solaris™ ZFS (Solaris 10 OS only) ฺ csm Guide e t Services @ n Failover File System Access Is Notufor Scalable b e ak Stud y ( n this for failover services that run Failover file system access d is iappropriate u only on the nodes physically e to storage devices. All Rights Reserved. Revision C 1-37 a . In Sun Cluster 3.2 software also has support in the cluster for failover file system access. Oracle and/or its affiliatesฺ Failover (Non-Global) File Systems in the Cluster Sun Cluster 3.

When you log in to a non-global zone you see the file systems belonging only to that non-global zone. Each configured nonglobal zone (the main Solaris 10 OS instance is known as the global zone) appears to be a complete OS instance isolated from every other zone and from the global zone. 1-38 Sun™ Cluster 3. Inc. Revision C a . An LDom can do anything that a Solaris instance u k a emachine ab can. A particular zone can be used with only one of the features. users and processes belonging only to that non-global zone.2 Administration Copyright 2009 Sun Microsystems. Root users logged into one nonglobal zone have superuser privileges only in that zone. You can mix and match zone features in the same cluster. including containing its own zones. All Rights Reserved. When configuring a non-global zone. as outlined in the subsections below. Each zone has its own zonepath. each running inside a physical instance of the Solaris OS. e does not have a separate kernel or al Auzone s m address space. in thinisthat it only appears to be an d u independent instance of Solaris. that is. r y physical l f bdu trans a g onn a nSun Cluster Distinct Zone Features ab Sun Cluster has three distinct ways of dealing with zones. Oracle and/or its affiliatesฺ The Solaris 10 OS zones feature is a powerful facility that allows you to create multiple lightweight virtual instances of the Solaris 10 OS. its own completely separate directory structure for all of its files starting with the root directory. Sun Services. ka e to g baa nfulllivirtual ensmachine with its own kernel and complete An LDomais c b of theleSolaris OS. its zonepath is expressed as a subdirectory somewhere inside a global zone mounted file system.Solaris 10 Zones Solaris 10 Zones Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. s ) ha y m ฺ om ฺ c ฺ csm Guide e LDoms and Zones @ ent b u ak Stud y ( A zone is a lightweight virtual machine. and so on.

Sun Services. Whichever applications happen to run b a when the zone boots will fail over as the zone itself fails or switches from node to node. dating from Sun Cluster 3. They act as virtual nodes for that part of the cluster control that launches applications. Instead.1. a zone is treated as a black box that fails over from node to node. Here. on only one node at a time. All Rights Reserved. and the zone is booted under control of Sun Cluster.Solaris 10 Zones Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. In this model. Revision C 1-39 a .2. there is a single “global cluster”. Zones run on different cluster nodes and are booted outside the control of Sun Cluster. Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. Zones as Virtual Nodes in a Single “Global Cluster” This second model was introduced in the original Sun Cluster 3. as usual. Figure 1-12 illustrates this feature: s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le HA-Containers feature k b Figure 1-12 a a l y sfer u d ab -tranAn advantage of this feature is that the zone can be treated as a g n non a completely cluster-unaware entity. and can launch applications to run in the zones. you do not think of zones as having their own cluster. Oracle and/or its affiliatesฺ Failover Zones (HA-Containers feature) This is the oldest of the zone features. zones are simply a location where applications can be launched. The cluster framework is completely aware of these zones. Inc. The zonepath for the zone lives in shared storage.

Solaris 10 Zones Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. rather than the whole zone r y l e f bdu transfailing over.2 Administration Copyright 2009 Sun Microsystems. 1-40 Sun™ Cluster 3. Revision C a . This will result in faster failover. to launch the applications in the global zone and within non-global zones. All Rights Reserved. a n ab ● ● usually without any modifications. Sun Services. Inc. a g onn The same application-specific code (Sun Cluster agent) can be used. Oracle and/or its affiliatesฺ Figure 1-8 illustrates this feature: s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u l e zones afailover Figure 1-13 Application between s m u ka e to g n ehas nsseveral advantages over the HA-containers Note that this bamodel c a li model: b e u l k a Applications ab fail from zone to zone.

2 Advanced Administration course (ES-445). Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems. All Rights Reserved. with a slightly different point of view. as illustrated by Figure 1-14: s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g  ban licens a b 1-14le Zone clusters within a “global cluster” u Figure k a erab y l u nsf d b a tra You can accomplish many of the same tasks using a zone cluster and the g n n previous feature. much like the processes in a zone are isolated from other zones. rather than in this course. Inc.Solaris 10 Zones Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 1-41 a .2 1/09 (Update 2). we think of a set of zones (across different nodes) as forming their own cluster. Sun Services. a “zone cluster” within the normal “global cluster”. the no aba configuration of the cluster applications in the zones is isolated from the the rest of the configuration. Here. Here. Zone cluster is addressed in detail in the Sun Cluster 3. introduced in Sun Cluster 3. Several zone clusters can exist within the global cluster. Oracle and/or its affiliatesฺ Zone Cluster This is the newest model for zones.

and administrative workstation. Inc. you should be able to identify Sun Cluster hardware components located in the training lab. While participating in the guided tour.2 Administration Copyright 2009 Sun Microsystems. Task s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u k tud aaccess y ( If this course is being run without local the equipment. terminal concentrator. you will identify the Sun Cluster hardware components including the cluster nodes. Oracle and/or its affiliatesฺ By the end of this exercise. take this n of tthe sSunStoCluster i i d h opportunity to review the essentials hardware and u l e a software. Sun Services.Exercise: Guided Tour of the Training Lab Exercise: Guided Tour of the Training Lab Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. am yourself o k t ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n no aba You will participate in a guided tour of the training lab. Preparation No special preparation is required for this lab. 1-42 Sun™ Cluster 3. Revision C a . or to familiarize uswith the remote lab environment.

Exercise Summary Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 1-43 a . or discoveries you had during the lab exercises. Sun Services. Oracle and/or its affiliatesฺ Exercise Summary ? ! Discussion – Take a few minutes to discuss what experiences. Inc. All Rights Reserved. issues. ● Experiences ● Interpretations ● Conclusions ● Applications s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Introducing Sun™ Cluster Hardware and Software Copyright 2009 Sun Microsystems.

Oracle and/or its affiliatesฺ Exercise Summary s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 1-44 Sun™ Cluster 3. All Rights Reserved.2 Administration Copyright 2009 Sun Microsystems.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a . Inc. Sun Services.

Sun Services. All Rights Reserved. Revision C a . you should be able to: c ฺ ide csa m u Describe the different methods for accessing console e G t @ n b Configure the Sun Cluster console deon the administration kusoftware u a t y workstation n ( this S i d Use the Cluster console lu tools e a s m u ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g an non ● ● ● ab 2-1 Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Module 2 Exploring Node Console Connectivity and the Cluster Console Software s ) ha Objectives y m ฺ om ฺ Upon completion of this module. Inc.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

Inc. Sun Services. Oracle and/or its affiliatesฺ Relevance ? ! Discussion – The following questions are relevant to understanding the content of this module: ● What is the trade-off between convenience and security in different console access methods? ● How do you reach the node console on domain-based clusters? ● What benefits does using a terminal concentrator give you? ● Is installation of the administration workstation software essential for proper cluster operation? y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha 2-2 Sun™ Cluster 3. Revision C a .Relevance Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems. All Rights Reserved.

● Sun Cluster Software Installation Guide for Solaris OS. Sun Services.Additional Resources Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 2-3 a . part number 820-4679 (for Sun Cluster 3. Inc. All Rights Reserved.2 1/09). Oracle and/or its affiliatesฺ Additional Resources Additional resources – The following references provide additional information on the topics described in this module: ● Sun Cluster System Administration Guide for Solaris OS.2 1/09). part number 8204677 (for Sun Cluster 3. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Exploring Node Console Connectivity and the Cluster Console Software Copyright 2009 Sun Microsystems.

● Must have console node access for certain emergency and informational purposes. functioning clusters. If a node is failing to boot. It is expected that a Sun Cluster 3. All Rights Reserved. ban licens a byou have u le a graphics monitor and system keyboard you are k Even if b a a l ysupposed erto redirect console access to the serial port or emulation thereof. if whatever device you have connected directly to the interfaces does not spuriously issue BREAK signals on the serial line. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( Accessing Serial Port Consoles in thon is Traditional Nodes d u al use m Traditional Sun Cluster ka 3. It is acceptable to have direct telnet. Inc. a software keyswitch position (on midrange and high-end sersvers). Sun Services.Accessing the Cluster Node Consoles Accessing the Cluster Node Consoles Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. 2-4 Sun™ Cluster 3. Most cluster operations require only that you be logged in on a cluster node as root or as a user with cluster authorizations in the Role-Based Access Control (RBAC) subsystem. f u s d ab -tran g n non a b a The rule for console connectivity is simple.2 environment administrator: ● Does not require node console access for most operations described during the duration of the course. BREAK signals on the serial port bring a cluster node to the OK prompt. Revision C a . the cluster administrator will have to access the node console to figure out why. turn the key to the third position to power the server on and disable the BREAK signal. or a file setting (on all nodes). The cluster administrator might like to observe boot messages even in normal.2 Administration Copyright 2009 Sun Microsystems. rlogin.2 node consoles.2enodes to usually use serial port ttyA as the g console. You can connect to the node ttyA interfaces any way you prefer. killing all cluster operations on that node. For those servers with a hardware keyswitch. Oracle and/or its affiliatesฺ This section describes different methods for achieving access to the Sun Cluster 3. or ssh access to the node. You can disable node recognition of a BREAK signal by a hardware keyswitch position (on some nodes).

Oracle and/or its affiliatesฺ For those servers with a software keyswitch. or by running the kbd -i command as root. When the Alternate Break sequence is in effect. only serial console devices are affected. You can use the prtconf command to figure out the name of your serial port driver. All Rights Reserved. the sequence is carriage return. and then use man serial-driver to figure out the sequence. The Alternate Break signal is defined by the particular serial port driver you happen to have on your system. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Exploring Node Console Connectivity and the Cluster Console Software Copyright 2009 Sun Microsystems.Accessing the Cluster Node Consoles Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. tilde (~). Revision C 2-5 a . Sun Services. and control-B: CR ~ CTRL-B. issue the setkeyswitch command with the secure option to power the server on and disable the BREAK signal. This setting takes effect on boot. while running Solaris OS. for the zs driver. Inc. For example. For all servers. uncomment the line KEYBOARD_ABORT=alternate in /etc/default/kbd to disable receipt of the normal BREAK signal through the serial port.

When you connect with telnet to the TC’s IP address and pass through port. 2-6 Sun™ Cluster 3. Oracle and/or its affiliatesฺ One of the popular ways of accessing traditional node consoles is through a TC.2 Administration Copyright 2009 Sun Microsystems. Inc. All Rights Reserved. s ) ha y m ฺ om ฺ c ฺ sm uide c Administrative e G Cluster Console t @ Console n b Software ku tude a y n ( this S i Network d lu se a Terminal am to u k Concentrator ng ense a Network b a Interface lic b e Serial Ports u l k b a a Serial port A y fer l 7 4 8 2 3 5 6 1 u s bd tran a g onSetup port n a n b Node 1 Node 2 a Setup Device Figure 2-1 TC Network and Serial Port Interfaces Most TCs enable you to administer TCP pass-through ports on the TC. a device which listens for connections on the network and passes through traffic (unencapsulating and reencapsulating all the TCP/IP headers) to the various serial ports. Sun Services. Figure 2-1 shows a terminal concentrator network and serial port interfaces. the TC transfers traffic directly to the appropriate serial port (perhaps with an additional password challenge). most security-conscious administrators would attach it to a private management network. A TC is also known as a Network Terminal Server (NTS). The node public network interfaces are not shown. While you can attach the TC to the public net.Accessing the Cluster Node Consoles Accessing Serial Port Node Consoles Using a Terminal Concentrator Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a .

All Rights Reserved. connected to your first cluster node. powered off. Exploring Node Console Connectivity and the Cluster Console Software Copyright 2009 Sun Microsystems. The Sun TC supports telnet and rlogin access.time that might be considered spurious. for example. ang non # telnet tc-ipname 5002 If your TC cannot meet that requirement. Sun guarantees that this TC does not spuriously issue BREAK signals when it is powered on and off. for example. you can still disable recognition of the BREAK signal or enable an alternate abort signal for your node. The TCP ports it uses as pass-through ports for telnet access are 5002 for serial port 2. and so forth. Oracle and/or its affiliatesฺ Sun Terminal Concentrator (Sun NTS) Sun rebrands a terminal concentrator. When you use telnet to connect to the Sun TC from an administrative workstation using. the following command. 5003 for serial port 3. “Terminal Concentrator. Some terminal concentrators support Secure Shell (the Sun NTS does not support Secure Shell). the Xylogics NTS. Sun Services. if you are concerned about passing TC traffic in the clear on the network.Accessing the Cluster Node Consoles Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. t @ n b ku tude a y Full instructions about installing(and configuring the Sun TC are in n this S i d Appendix A. This might influence your choice. sm uiadper-port e G password that you have to enter before going through to the serial port. Inc.” lu se a am to u k ng Concentrators se Other Terminal a n b e a lic b e u l kYou can b any type of TC as long as it does not issue BREAK signals a achoose r y l e f s the serial ports when it is powered on. reset. the Sun TC passes you directly to serial port 2. Revision C 2-7 a . or any other bdu tranon a . s ) ha ab y m ฺ om ฺ c ฺ e The Sun TC supports an extra level of passwordcchallenge.

Inc. Revision C a . s ) ha You can attach a workstation on the network exactly as you would place a TC.Accessing the Cluster Node Consoles Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. You then add lines to the /etc/remote file of the Solaris OS workstation as follows: y m ฺ om ฺ c ฺ node1:\ csm Guide e @ ent :dv=/dev/term/a:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D: b u node2:\ ak Stud y ( :dv=/dev/term/b:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D: in this d u This allows you to access econsoles by accessing the launchpad al node s m u workstation and kamanually to typing tip node1 or tip node2. Oracle and/or its affiliatesฺ Alternatives to a Terminal Concentrator (TC) for Nodes with a Serial Port Console Some possible alternatives to the terminal concentrator (TC) include the following: ● Use dumb terminals for each node console. and attach its serial ports to the node consoles. If these are in a secure physical environment. especially for a cluster with only two nodes. but least convenient. b it is easier e youto could u l k b For example. All Rights Reserved. ● Use a workstation that has two serial ports as a tip launchpad. easily disable telnet and rlogin access and a era y l f that administrators access the tip launchpad through Secure bdu transrequire a Shell. method.2 Administration Copyright 2009 Sun Microsystems. These include: 2-8 ● Hardware domain-based systems: the console access device is the system controller (SC) or system service processor (SSP) ● Servers such as Sun Fire V890: you can choose to have console access through the Remote System Control (RSC) device and software Sun™ Cluster 3. this is certainly the most secure. g e s a Solaris OS workstation instead of a TC is One advantage ban liceofnusing a that tighten the security in the Solaris OS workstation. ang non ab Accessing the Node Console on Servers with Virtual Consoles Many of the servers supported with Sun Cluster have console access through a network connection to a virtual console device. Sun Services.

s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Exploring Node Console Connectivity and the Cluster Console Software Copyright 2009 Sun Microsystems.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. Sun Services. Oracle and/or its affiliatesฺ Accessing the Cluster Node Consoles ● Modern rack-based servers: the console access device is a small onboard system controller running Advanced Lights-out Management (ALOM) ● LDoms: console access to an LDom is through a network connection to the service domain. which provides the virtual console service for the LDom. Inc. Revision C 2-9 a .

2 Administration Copyright 2009 Sun Microsystems. Revision C a .2 software environment itself has no dependency on the t @ n b e an administrative administration workstation’s console software. n ( this S i d lu se a am to u Console Software Installation k ng ense a b a licconsole software is contained in a single package. y l e f s 3. The cluster console window variations are the following: ● Cluster console (cconsole) The cconsole program accesses the node consoles through the TC or other remote console access method. you might have to respond to additional password challenges. Depending on your access method. There are internationalized packages as well (SUNWjccon. Inc. or ssh commands to connect you either directly to the nodes or to the node console access point. you might be directly connected to the console node. This software can be installed on any Solaris OS workstation running Solaris OS version 2. All Rights Reserved. as h The console software includes a common window feature that lets you issue ) y keystrokes simultaneously to all the nodes in the cluster (or even ฺtom all the om ฺ nodes in multiple clusters). b The u administrative e l k raThe b SUNWccon package is installed manually from the Sun a SUNWccon. Oracle and/or its affiliatesฺ Describing Sun Cluster Console Software for an Administration Workstation The Sun Cluster 3.SUNWkccon. and so on) but you always have to install SUNWccon first.2 software includes a small amount of software to be installed on an administrative workstation. ang non ab Cluster Console Window Variations The three variations of the cluster console tool each use a different method to access the cluster hosts.2. or you might have to login to an SSP or SC and manually connect to the 2-10 Sun™ Cluster 3.6 and above. Sun Services. rlogin. They all look and behave the same way.Describing Sun Cluster Console Software for an Administration Workstation Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. n bdu trCluster a a . ku Ittuisdjust a y convenience. c ฺ sm uide c e G The Sun Cluster 3. The console software is a convenience that displays new windows for you to the cluster nodes and issues telnet.

s ) ha y m ฺ m o c ฺ ฺ want to In certain scenarios. Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ● Cluster console (ctelnet) The ctelnet program accesses the nodes directly using the telnet command. some tools might be unusable. Sun Services.Describing Sun Cluster Console Software for an Administration Workstation node console. ● Cluster console (cssh) The cssh program accesses the nodes directly using the ssh command. but you can request that it use ssh by adding the -s option to cconsole. cconsole by default uses telnet to reach the console device. you will not be ableinto use the cconsole variation. install this software to use the tools that makeesense u t @ b den u k For example. but you might e m d s i c toGyou. if you are using dumb tuto attach to the node (yaterminals S consoles. All Rights Reserved. Oracle and/or its affiliatesฺ ● Cluster console (crlogin) The crlogin program accesses the nodes directly using the rlogin command. but you can is d h t u use the other three after e are booted. althe nodes s m u ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g an non ab Exploring Node Console Connectivity and the Cluster Console Software Copyright 2009 Sun Microsystems. Revision C 2-11 a . Inc.

Sun Services. and a small common keystroke window (Figure 2-2). Revision C a . Input directed to the common window is automatically replicated to all the other windows. All Rights Reserved. The tool automatically shows one new window for each node.Describing Sun Cluster Console Software for an Administration Workstation Cluster Console Tools Look and Feel Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. You can type in each individual window as desired. Oracle and/or its affiliatesฺ All the tools have the same general look and feel. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Figure 2-2 2-12 Cluster Console Windows Sun™ Cluster 3.2 Administration Copyright 2009 Sun Microsystems. Inc.

csm Gforubest Set the TERM environment variable to vt100 oredtterm @ ent b u ak Stud y ( Cluster Console Common in Window d his t u l e a us m a The common window. ab Figure 2-3 Cluster Console Common Window Exploring Node Console Connectivity and the Cluster Console Software Copyright 2009 Sun Microsystems. you can use the tools manually to connect to a single cluster node or to the entire cluster. the c a li b e u l khost windows b follow. # # # # # /opt/SUNWcluster/bin/cconsole node1 & /opt/SUNWcluster/bin/cconsole -s my-cluster & /opt/SUNWcluster/bin/ctelnet my-cluster & /opt/SUNWcluster/bin/crlogin node3 & /opt/SUNWcluster/bin/cssh node3 & Cluster Console Host Windows s ) ha y m ฺ om ฺ c ฺ ideoperation. The Options menu allows you to ungroup the a a r y windows. allows you to enter g commands all host at the same time. All Rights Reserved. k eshown to in Figure 2-3. f u s d ab -tran g an non Each node in the cluster has a host window. Sun Services. Revision C 2-13 a .Describing Sun Cluster Console Software for an Administration Workstation Starting the Tools Manually Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. and group them again. You can enter commands in each host window separately. Oracle and/or its affiliatesฺ As shown. when you move the common window. All of the nssystemso windows baarentotied e windows together. l e move them into a new arrangement.

Inc. Sun Services. Revision C a .Describing Sun Cluster Console Software for an Administration Workstation Cluster Control Panel Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ As shown in Figure 2-4. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u k tud Figure 2-4 Cluster Control Panel (ya n this S i d lu Panel e a s Starting the Cluster Control m u ka e to g an cControl To start thebCluster ns Panel. All Rights Reserved.2 Administration Copyright 2009 Sun Microsystems. type the following command: e a li e u l #b /opt/SUNWcluster/bin/ccp [clustername] & k b a a l y sfer u d ab -tran g n non a b a 2-14 Sun™ Cluster 3. the Cluster Control Panel provides centralized access to three variations of the cluster console tool.

no aba Configuring the /etc/clusters File You can define many different clusters in a single /etc/clusters file. Inc. Sun Services. Revision C 2-15 a .ec G t @ n b u ude k/etc/clusters a t The following is a typical entry in the file: y ( S n s i i th sc-cluster sc-node1 lud sc-node2 e a am to us k e a cluster named sc-cluster. which has two The single-line ng entry sdefines a n b e nodesanamed sc-node1 and sc-node2.2 software ul nNote s d b a tra itself requires you to define a cluster name. so you can administer several clusters from a single administrative console. The files are: ● The /etc/clusters file ● The /etc/serialports file (for cconsole variation only) When you install the Sun Cluster console software on the administrative console. All Rights Reserved. although nothing will break if it does not agree.Configuring Cluster Console Tools Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Configuring Cluster Console Tools All the information needed for the cluster administration tools that run on the administrative console is configured in two files. lic b e u l k rab a y fe– The cluster name is purely arbitrary. you must manually create the clusters and serialports files and populate them with the necessary information. Exploring Node Console Connectivity and the Cluster Console Software Copyright 2009 Sun Microsystems. s ) ha y m ฺ om ฺ c ฺ The /etc/clusters file contains the name of a cluster sm followed ide by the u names of the nodes that are part of the cluster. The Sun Cluster 3. which will likely agree with g n n this one.

Revision C a . ab 2-16 Sun™ Cluster 3. the host name or IP address of the device providing the connection as listed in /etc/hosts. Sun Services. You enter a separate line for each node in this file. hand t u l e command (typically console) a manually. Inc. you must put the correct u d an number in /etc/serialports (typically 22) and use cconsole with ab -trport g an non the -s option. m us a o k t sc-node1 node1-sc g nse22 n a sc-node2 ce 22 ab node2-sc i l b u le k b a a l yNotes–feInrorder to use ssh to the console device. The following are typical entries in the /etc/serialports file when using a terminal concentrator to pass through to the physical serial connection (ttyA) on each cluster node: sc-node1 sc-tc 5002 sc-node2 sc-tc 5003 s ) ha y m ฺ om ฺ c ฺ The following is a typical example for nodes with network sm access ideto a c u e virtual console access device. t @ n b you are connected to the console device.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Configuring the /etc/serialports File The /etc/serialports file defines the connection to the console for each node defined in the /etc/clusters file. When the cconsole command G is executed.kbut the command u ude has no idea a t console. This file uses three columns: the node name listed in /etc/hosts. You must log what you need to type to actually access y the node ( S n i is type your console access into the console access device d manually. and the port for telnet or ssh to use for the connection to or through this device. All Rights Reserved.Configuring Cluster Console Tools Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

Revision C 2-17 a . u k a ebefore ab proceeding with this exercise. r y l f s bdu tranTable a 2-1 Cluster Names and Addresses g onn a n ab System Name IP Address Administrative console TC Node 1 Node 2 Node 3 (if any) Exploring Node Console Connectivity and the Cluster Console Software Copyright 2009 Sun Microsystems. All Rights Reserved. Oracle and/or its affiliatesฺ In this exercise. you complete the following tasks: ● Task 1 – Updating Host Name Resolution ● Task 2 – Installing the Cluster Console Software ● Task 3 – Verifying the Administrative Console Environment ● Task 4 – Configuring the /etc/clusters File ● Task 5 – Configuring the /etc/serialports File ● Task 6 – Starting the cconsole Tool ● Task 7 – Using the ccp Control Panel s ) ha y m ฺ om ฺ c ฺ Preparation csm Guide e t is already installed n bOS@software This exercise assumes that the Solarisk10 e u a the following tud steps to prepare for on all of the cluster systems. Perform y ( S in this the lab: d u e assigned to your cluster. Ask your instructor name ka e to g s Cluster n_______________________________ banname: e c a b Recordletheli information in Table 2-1 about your assigned cluster 2. Inc. al for the s m u 1. Sun Services.Exercise: Configuring the Administrative Console Exercise: Configuring the Administrative Console Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

Revision C a . s ) ha y m ฺ om ฺ 1. SConsult your instructor listed first before any otherin name s i d h t such as dns should be used. Check to see if the cluster console software is already installed: (# or $) pkginfo SUNWccon If it is already installed. Verify that the /etc/nsswitch.2 Administration Copyright 2009 Sun Microsystems. Ask your instructor for the location of the Sun Cluster software. Sun Services.2 packages directory: (# or $) cd sc32_location/Solaris_{sparc_or_x86}/Products (# or $) cd sun_cluster/Solaris_10/Packages 2-18 Sun™ Cluster 3. Move to the Sun Cluster 3. it can be beneficial to resolve the names locally on the administrative console and cluster hosts. 2. the cluster console software is already installed. The cconsole program does not start unless it can first resolve the host names in the /etc/clusters file. Log in to your administrative console as user root.conf u a t y ( service. If necessary. you can skip the rest of the steps of this task. Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. All Rights Reserved. your instructor will assign you a user login. t @ n b e hosts has files dfor kufile entry 2. Oracle and/or its affiliatesฺ Software location: _______________________________ Task 1 – Updating Host Name Resolution Even though your site might use Network Information Service (NIS) or DNS to resolve host names. In this case. a m to u hosts: fileska dns ng ense a b a lic b e u l Task a 2k– Installing the Cluster Console Software b a r y ul nsfe d b a the following steps to install the cluster console software: a trPerform g n n a no Perform the following steps: ab 1. about whether any other lu nameseservice. Note – If your administrative console is in a remote lab. 3. and add the IP addresses and names of sthe idethe host c e G systems in your cluster. This can be valuable in the case of naming service failures. edit the /etc/hosts file on your administrative c ฺ mTC uand console.Exercise: Configuring the Administrative Console 3.

The entries are in the form: clustername host1name host2name host3name host4name Sample /etc/clusters File sc-cluster pnode1 pnode2 pnode3 Exploring Node Console Connectivity and the Cluster Console Software Copyright 2009 Sun Microsystems. Revision C 2-19 a . Verify that you are in the correct location: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.profile se file to verify changes: 2. Inc.profile d hishome directory if necessary. $HOME/. and t u l e add the changes. Execute a n b e a ic b (# orle$)l . SUNWccon Task 3 – Verifying the Administrative Console Environment Perform the following steps to verify the administrative console: as h 1. All Rights Reserved.Exercise: Configuring the Administrative Console 4. Install the cluster console software package: # pkgadd -d .profile u k rab a y ul nsfe d b a tra Note – You can also log out and log in again to set the new variables. Verify that the following search paths and variables are present in ) y m the . g n n a no ab Task 4 – Configuring the /etc/clusters File The /etc/clusters file has a single line entry for each cluster you intend to monitor. ma us a o k t ngthe . Oracle and/or its affiliatesฺ # ls -d SUNWccon SUNWccon 5. Sun Services.profile file in your home directory: ฺ om ฺ c PATH=$PATH:/opt/SUNWcluster/bin ฺ MANPATH=$MANPATH:/opt/SUNWcluster/man csm Guide e export PATH MANPATH @ ent b u ak Stud y ( n in your ifile Note – Create the .

Inc. ab 2-20 Note – In the remote lab environment. This example is for a three-node cluster using a Sun NTS. and add a line using the cluster and node names assigned to your system. the /etc/clusters file might already be set up for you. All Rights Reserved. Examine the file. this file might already be set up for you. a tra g n n a no The /etc/serialports file has an entry for each cluster host describing the connection path. Task 5 – Configuring the /etc/serialports File s ) ha y m ฺ om ฺ c ฺ csm Guide hostname tcname tcport e @ ent b u k tud Sample /etc/serialports File (ya S n s i i d h t pnode1 cluster-tc lu s5002 e a pnode2 cluster-tc 5003 am to u 5004 k pnode3 cluster-tc ng ense a b a the following lic to configure the /etc/serialports file: b Perform e u l k rab a y ul nsfeEdit the /etc/serialports file and add lines using the node and d b TC names assigned to your system. Sun™ Cluster 3. Revision C a . Examine the file. Note – If you are using a remote lab environment. Sun Services.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Edit the /etc/clusters file.Exercise: Configuring the Administrative Console Perform the following to configure the /etc/clusters file: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

Consult your h ) y so you instructor about whether the console access device supports ssh. and ccp tools. n se a n b e 5. Perform the following steps to start the cconsole tool: 1. Practice using the Common window Group Term Windows feature under the Options menu. If not. t (y is S n i 4. Ifathe cluster lic host systems are not booted. Sun Services. b e u l k ok bboot a a r y ul n6. You should see a response on all of the u ude k a cluster host windows. All Rights Reserved. ctelnet console tool variations. you might udis anseALOM l a need to enter an command to access the actual node consoles. Task 7 – Using the ccp Control Panel The ccp control panel can be useful if you must use the console tool variations. boot them now. 2. rearrange them. Practice using the cssh. om ฺ c ฺ sm uand idepress the cwindow. Make sure power is on for the TC and all of the cluster hosts. Revision C 2-21 a . mALOM u a o k t Consult g your instructor. and then group them together again. ask your instructor for assistance. if m ฺ can add a -s option above. 3. Start the ccp tool (ccp clustername &). log in as d b a tra user root. You can ungroup the cconsole windows. Perform the following steps to use the ccp control panel: 1. Inc. g n n o a n 7. 2. Start the cconsole tool on the administrative console: (# or $) cconsole clustername & ab as Note – Substitute the name of your cluster for clustername. Oracle and/or its affiliatesฺ This section provides a good functional verification of the TC in addition to the environment configuration. Place the cursor in the cconsole Common G t @ n b Return key several times. Exploring Node Console Connectivity and the Cluster Console Software Copyright 2009 Sun Microsystems. ctelnet. If your console device th dedicated interface. e 3.Exercise: Configuring the Administrative Console Task 6 – Starting the cconsole Tool Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.sfeAfter all cluster host systems have completed their boot. Quit the cssh.

2 Administration Copyright 2009 Sun Microsystems. All Rights Reserved. Sun Services. Revision C a .Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. Oracle and/or its affiliatesฺ Exercise: Configuring the Administrative Console s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 2-22 Sun™ Cluster 3.

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. Revision C a . you should be able to: c ฺ idehardware csm Guand List the Sun Cluster software boot disk requirements e @ ent restrictions b u k tud atopologies y Identify typical cluster storage ( n this S i d Describe quorum a votes lu andsequorum devices u reservations and cluster amnesia am quorum o Describe persistent k t ngdataefencing se a n Describe b a lic b e u l k Configure b a supported cluster interconnect system a a r y ul nsfeIdentify public network adapters d b a tra Configure shared physical adapters g n n a no ● ● ● ● ● ● ● ● 3-1 Copyright 2009 Sun Microsystems. Inc. Oracle and/or its affiliatesฺ Module 3 Preparing for Installation and Understanding Quorum Devices s ) ha Objectives ab y m ฺ om ฺ Upon completion of this module. All Rights Reserved.

All Rights Reserved. Oracle and/or its affiliatesฺ Relevance ? ! Discussion – The following questions are relevant to understanding the content of this module: ● Is cluster planning required even before installation of the Solaris OS on a node? ● Do certain cluster topologies enforce which applications are going to run on which nodes. Revision C a .2 Administration Copyright 2009 Sun Microsystems. Inc.Relevance Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. or do they just suggest this relationship? ● Why is a quorum device absolutely required in a two-node cluster? ● What is meant by a cluster amnesia problem? y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha 3-2 Sun™ Cluster 3. Sun Services.

Inc. part number 821-0259 (for Sun Cluster 3. part number 8207356 (for Sun Cluster 3. Revision C 3-3 a .2 11/09). part number 820-7458 (for Sun Cluster 3. ● Sun Cluster Software Installation Guide for Solaris OS. Oracle and/or its affiliatesฺ Additional Resources Additional resources – The following references provide additional information on the topics described in this module: ● Sun Cluster System Administration Guide for Solaris OS.2 11/09) s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems.Additional Resources Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 11/09). Sun Services. ● Sun Cluster Concepts Guide for Solaris OS. All Rights Reserved.

Configuring Cluster Servers Configuring Cluster Servers Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. a b VERITAS u le Dynamic Multipathing (DMP) is not supported. If a storage device is visible to more than one host. Inc. a SAN LUN that is masked so that it is visible to only one node can be used as a boot device. S The Solaris 9 OS 9/05 (Update (8yand and tuSolaris vary. it is shared. including the update The same version m a k example. Oracle and/or its affiliatesฺ You must plan your cluster installation before you install the Solaris OS on your cluster nodes. Sun Services.2 software release. g n non a The supported versions of VERITAS Volume Manager must b a ● You cannot use a shared storage device as a boot device. All Rights Reserved. For example. listed in Module 1) d are insupported: his t u l e a of theusSolaris OS. “Using VERITAS Volume Manager With Sun u d ab -tran Cluster Software” for more details about this restriction.2 software still does not support an actual multipathing topology under control of Volume Manager DMP. Boot Device Restrictions With the Sun Cluster 3. k b a a l y sferSee Module 6. outlined in the following subsections. there are several restrictions on boot devices including the following: s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u d 10 OS (revisions ak 9). Revision C a . If you install Solaris “blindly” on cluster nodes you are likely to miss several cluster-specific considerations and requirements. ● ● ● have the DMP device drivers enabled. to Solaris 10 11/09).2 Administration Copyright 2009 Sun Microsystems. 3-4 Sun™ Cluster 3. must be installed on all revision (for g e s nodes ban inlictheencluster. regardless of the number of physical connections to the device. but Sun Cluster 3.

ang non ● Starting in Sun Cluster 3. s ) ha ab y m ฺ om ฺ c ฺ csm Guide e t n b@ e u Note – The /globaldevices mount k point is modified during the Sun d u a t y Cluster 3. represents the u se l a number that is assigned m totoa node u when it becomes a cluster member.devices/node@nodeid file system. as discussed in Module 4. a eraThis y l f s A file system size of 512 Mbytes should suffice for even the largest bdu trancluster.2 software recommends a minimum swap partition of 750 megabytes (Mbytes). ● VERITAS Volume Manager software requires two unused partitions for encapsulation of the boot disk.devices/node@nodeid file system. have the Solaris installer create an empty filesystem mounted on /globaldevices as a placeholder.Configuring Cluster Servers Boot Device Partitioning (without ZFS root) Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ● Solaris Volume Manager software requires a 32 Mbyte partition for its meta-state databases.globaldevices. do not create a /globaldevices placeholder.(It is automatically to n twhere s Snodeid renamed i i d h /global/.cluster configurations.2 software installation. The g nse n a /globaldevices file system must have ample space and ample inode b lice a capacity b forlecreating both block-special devices and character-special u k b is especially important if a large number of disks are in the devices. You will then be able to choose the lofi option as you build the cluster using scinstall. scinstall uses 100MB as the size of the backing store when you use a lofi device. A lofi device is a virtual disk device that uses a file as the backing store for the entire contents of a file system. Oracle and/or its affiliatesฺ If you use a traditional partitioned root disk with UFS-root: ● Sun Cluster 3. While 512MB is still recommended when you use a dedicated partition. Note – scinstall will automatically provision the backing store for the lofi device in a file named /. a . If you choose this option.2 1/09 (Update 2) you must have a dedicated partition on the boot disk on each node for the /global/. Sun Services. Inc. Revision C 3-5 a . Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. If you choose this option.2 1/09 you can choose to use a lofi device rather than a dedicated partition for the /global/.devices/node@nodeid. ● Prior to Sun Cluster 3. All Rights Reserved. a k The original /globaldevices mount point is removed.

ZFS for both data and root are discussed more in Module 8.Configuring Cluster Servers Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. Inc.2 Administration Copyright 2009 Sun Microsystems. the following JumpStart software profiles represents a starting point for boot disk configuration. # example Jumpstart profile -. All Rights Reserved. and gives you the ability for checkpointing and recovery using snapshots. This greatly simplifies root disk provisioning and high availability. Revision C a .will choose lofi in scinstall d k a Stu install_type initial_install (y n system_type standalone udi his t l e a us partitioning explicit m a k e to cluster SUNWCXall g filesys c0t0d0s0 an cefree ns / b a filesys c0t0d0s1 li 1024 swap b e u l k filesys 32 a erc0t0d0s7 ab y l f bdu trans a ang non Note – The logging option is the default starting in Solaris 9 9/04 ab (Update 7). Boot Device Partitioning (ZFS-root) Solaris OS supports ZFS-root beginning in Solaris 10 10/08 (Update 6). Oracle and/or its affiliatesฺ Typical Boot Disk JumpStart™ Software Profile Examples (Without ZFS root) If you decide to configure your cluster servers using the JumpStart™ software.with dedicated /globaldevices install_type initial_install system_type standalone partitioning explicit cluster SUNWCXall filesys c0t0d0s0 free / filesys c0t0d0s1 1024 swap filesys c0t0d0s3 512 /globaldevices filesys c0t0d0s7 32 s ) ha y m ฺ om ฺ c ฺ sm uide c e # example Jumpstart profile no ZFS root but not G @ n b e u # dedicated /globaldevices --. and including all updates of Solaris 10 OS. 3-6 Sun™ Cluster 3.

devices/node@nodeid . because it is mounted globally and that is not supported by ZFS. Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. Inc. All Rights Reserved. a ang non . You might need to u l k b a a do this to support scalable applications that require global file systems or y fer l u s bd tranif you need to use SVM with Oracle RAC. Sun Services. Oracle and/or its affiliatesฺ Configuring Cluster Servers With ZFS root you will choose to use the lofi device option for the /global/. ab y m ฺ m o c ฺ ฺ # example Jumpstart profile -. rather than on g e Since the pool for n ens badisk. a file in a ZFS file system can be used as the backing store for a lofi device that contains the UFS file system required for /global/.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. But the lofi solution works nicely.ZFS with m e d s i u # space on s7 left out of the zpool for ecSVMt metadb G @ install_type initial_install b den u k cluster SUNWCXall ya Stu ( filesys c0t0d0s7 32 n di2G e this2G c0t0d0s0 u pool rpool auto l a us m a k ZFS root tois configured on a disk partition. Typical Boot Disk JumpStart™ Software Profile Example (With ZFS root) s ) ha If you decide to configure your cluster servers using the JumpStart™ software. It is not possible that /global/. Revision C 3-7 a . the following JumpStart software profile represents a starting point for boot disk configuration. an entire reserve a boot disk partition for Solaris Volume a ic can lwe b e Manager metadb’s outside the scope of the ZFS root.devices/node@nodeid itself be a ZFS file system.devices/node@nodeid file system. the two choices go together naturally.

uWhile s Sun Cluster does not require you connected to data storage m a o k t to configure a cluster using specific topologies.2 Administration Copyright 2009 Sun Microsystems. Sun Services.Configuring Cluster Storage Connections Configuring Cluster Storage Connections Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. e The following are some typical topologies: u l k b ra pairs topology ya Clustered l e f u s bd tran Pair+N topology a g onn a n N+1 topology ab ● ● ● ● 3-8 ● Multiported (more than two node) N*N scalable topology ● NAS device-only topology ● Data replication topology ● Non-storage topology ● Single-node cluster topology Sun™ Cluster 3. All Rights Reserved. Revision C a . Some storage configurations have restrictions on the total number of nodes supported. the following g bynsto eprovide ndescribed a topologies b are the vocabulary to discuss a cluster’s e c a i l connection b scheme.2 software regarding the data storage for the cluster are the following: ● Sun Cluster software never supports more than sixteen nodes. these storage devices must connect to at least m ฺ nodes. Inc. the only rules in the Sun Cluster 3. Oracle and/or its affiliatesฺ While previous versions of the Sun Cluster software had strict rules regarding how many nodes were supported in various disk topologies. However. as Shared storage devices do not need to connect to all nodes of the) h ytwo cluster. ● A shared storage device can connect to as many nodes as the storage device supports. om ฺ c ฺ csm Guide e @ ent b Cluster Topologies u ak Stud y ( din ways hiniswhich cluster nodes can be Cluster topologies describe ltypical t u e adevices.

Oracle and/or its affiliatesฺ Clustered Pairs Topology As shown in Figure 3-1. ● This configuration is well suited if you have legacy SCSI-array. or any disk array that can only be attached to two nodes. Still. You can have any even number of aba ● nodes from 2 to 16. no one node must have a significantly higher storage capacity than the others. a clustered pairs topology is two or more pairs of nodes with each pair physically connected to some storage. Revision C 3-9 a . Sun Services. ● Because each pair has its own storage. ● All nodes are part of the same cluster. ● Each pair of nodes shares storage.Configuring Cluster Storage Connections Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Storage is connected to both nodes in the pair. this does not restrict where applications can fail over to and run. ● The cost of the cluster interconnect is spread across all the nodes. but you are not restricted to this design. You are likely to design applications that run on the pair of nodes physically connected to the data storage for that application. All Rights Reserved. Because of the global device and global file system infrastructure. Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. it is likely you will configure applications to fail over in pairs of nodes attached to the same storage. ● This configuration is well suited for failover data services. Inc. Switch Switch s ) ha y m ฺ om ฺ c ฺ Node 3 Node 4 ide Node 2 Node 1 csm u e G t @ b den u k ya Stu ( n di e this u l a us m a k e to g Storage an Storage Storage Storage b licens a b 3-1 le Clustered Pairs Topology Configuration u Figure k a erab y l u nsf d b a tra The features of clustered pairs configurations are as follows: g n n no Nodes are configured in pairs.

Configuring Cluster Storage Connections Pair+N Topology Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. Sun Services. All Rights Reserved. Revision C a . a g onn a Additional cluster nodes support scalable data services or failover n ab Node 1 Node 2 Node 3 Node 4 ● ● data services with the global device and file system infrastructure.2 Administration Copyright 2009 Sun Microsystems. ● The Pair+N configuration is well suited for scalable data services ● This configuration is well suited if you have legacy SCSI-array. Switch Switch s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u Storage al use Storage m ka e to g n eTopology Figure 3-2 ba Pair+N ns c a li b e u l k a features abof the Pair+N configurations are as follows: The r y l e f bdu transAll shared storage is connected to a single pair. 3-10 Sun™ Cluster 3. the Pair+N topology includes a pair of nodes directly connected to shared storage and nodes that must use the cluster interconnect to access shared storage because they have no direct connection themselves. ● A maximum of sixteen nodes are supported. or any disk array that can only be attached to two nodes. The limitations of a Pair+N configuration is that there can be heavy data traffic on the cluster interconnects. ● There are common redundant interconnects between all nodes. Oracle and/or its affiliatesฺ As shown in Figure 3-2. You can increase bandwidth by adding more cluster transports.

or any disk array that can only be attached to two nodes. ● The backup node can take over without any performance degradation. All of the secondary paths to the storage devices are connected to the redundant or secondary system. ● This configuration is well suited if you have legacy SCSI-array. All Rights Reserved. Sun Services. which can be running a normal workload of its own. ● The backup node is more cost effective because it does not require additional data storage. ● This configuration is best suited for failover data services. enables one system to act as the storage backup for every other system in the cluster.Configuring Cluster Storage Connections Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. A limitation of the N+1 configuration is that if there is more than one primary node failure. Switch Switch s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to Storage Storage Storage g n s ba licen a b 3-3 le N+1 Topology Configuration Figure u k a erab y l f u nThe s d b a features of the N+1 configurations are as follows: a r t g n non a The secondary node is the only node in the configuration that is b a Node 1 Primary Node 2 Primary Node 3 Primary Node 4 Secondary ● physically connected to all the multihost storage. Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. shown in Figure 3-3. Oracle and/or its affiliatesฺ N+1 Topology The N+1 topology. Inc. you can overload the secondary node. Revision C 3-11 a .

ab For ordinary. more than two nodes can be physically connected to the same storage. Oracle and/or its affiliatesฺ In a scalable. 3-12 Sun™ Cluster 3. each particular disk group or diskset in the shared storage still only supports physical traffic from one node at a time.Configuring Cluster Storage Connections Scalable Storage Topology Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Switch Switch as Node 3 Node 4y) h Node 2 Node 1 ฺm m ฺco eฺ m cs Guid e @ ent b u ak Stud y ( in this d u al use m ka e to Storage Storage g n s ba licen a b u le k b a a Figure 3-4 Scalable Storage Topology y fer l u s bd tran a g on.This configuration is required for running the ORACLE Real Application n a n Clusters (RAC) across more than two nodes. Inc. or N*N topology.2 Administration Copyright 2009 Sun Microsystems. Revision C a . All Rights Reserved. Sun Services. shown in Figure 3-4. having more than two nodes physically connected to the storage adds flexibility and reliability to the cluster. However. cluster-unaware applications.

Oracle and/or its affiliatesฺ NAS Device-only Topology Sun Cluster supports the Network Appliance (NetApp) Filer NAS product. All Rights Reserved. and Sun Storage 7000 series.com.Configuring Cluster Storage Connections Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 3-13 a . and support for NAS data fencing. These concepts are described in more detail later in this module. To support the specific NetApp Filer implementation you must install a package called NTAPclnas on the cluster nodes. Sun ST2510. Inc. Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. Switch Switch s ) ha y m ฺ om ฺ c ฺ e cs3 m GuidNode Node 1 Node 2 Ne ode 4 t @ b den u k ya Stu ( Network (not the cluster tranport) n di e this u l a us m a k e to g ban licens a NAS Device (Sun or Netapp) b e u l k b ya fera l u s 3-5 NAS Device-Only Topology bd tranFigure a g onn a n ab Note – There is support for using a NAS device as a quorum device.netapp. This package is available for NetApp customers with a support contract from http://now. Sun StorageTek 5200 and 5300. Sun Services.

3-14 Sun™ Cluster 3. the supported mechanisms for performing the data replication are Hitachi TrueCopy. Hitachi Universal Replicator and EMC SRDF. This topology would be ideal with the quorum server feature introduced in Sun Cluster 3.Configuring Cluster Storage Connections Data Replication Topology Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Cluster supports a data replication topology. Oracle and/or its affiliatesฺ Starting in Sun Cluster 3. In this topology.2. shown in Figure 3-6 data storage is not physically multi-ported between nodes but is rather replicated between storage attached to the individual nodes using controller based replication. Switch s ) ha Switch y m ฺ Node 1 Node om2 ฺ c ฺ csm Guide e Data Replication @ ent b u ak Stud y ( in this d u al use m kaStorage toReplication Medium g e an cfiber. Inc.2. or some combination) b(Private e a li b e u l k a erab y l u nsf d b a 3-6 Data Replication Topology a trFigure g n n no aba At the time of the writing this course. nspublic network. Revision C a .2 Administration Copyright 2009 Sun Microsystems. The data replication topology is ideal for wider area clusters where the data replication solution is preferred to the extra connectivity that would be involved to actually connect the storage to nodes that are far apart. Sun Services. All Rights Reserved.

described later in this module. n i is d h t u al use m Sun Cluster 3.Configuring Cluster Storage Connections Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Starting with Sun Cluster 3. Prior to Sun Cluster 3. you could have an a n b e application a thatlicfails over between different non-global zones on the node. two-node clusters require shared storage because they require a quorum device. which manages a partnership of two clusters with data ab -tranreplication g across a wide area. or afor y ( S to begin a cluster with the intention of adding nodes as time goes on.2.2 provides. The cluster starts and is fully functional with just one node. Oracle and/or its affiliatesฺ Non-Storage Topology This can be suitable for an application that is purely compute-based and requires no data storage.2. Single-Node Cluster Topology In this configuration. Inc. It offers users the benefits of having application management functionality and application restart functionality. the ability to experience ka even tfor g e application n failovers. All Rights Reserved. Revision C 3-15 a . one node or domain forms the entire cluster. Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. o the first time. you could choose to use a quorum server device instead. s ) ha ab y m ฺ om ฺ c ฺ csm Guide e tto manage a cluster. This configuration allows for a single node to run as a functioning cluster. b u le k b a a l y Single ernode clusters can also be useful in the Sun Cluster Geographic f u s d Edition product. @ how Single node clusters are ideal for usersu learning n b e d k agenttudevelopment to observe cluster behavior (possibly purposes). Sun Services. Each member of such a partnership must an non be a full Sun Cluster installation. and a one-node cluster on either or both ends is acceptable. s on a single-node cluster.

All Rights Reserved. Hitachi Universal Replicator. Inc. ● Sun Cluster Geographic Edition does not currently provide an automatic failover of an application across the two clusters.0 remote mirroring (formerly SNDR)) Sun™ Cluster 3. ● Sun Cluster provides support for hardware (RAID-controller) based data replication (currently Hitachi TrueCopy. Sun Services.2 Administration Copyright 2009 Sun Microsystems. and EMC SRDF) as well as software based data replication (Available Suite 4. Revision C a . a b a ● ● 3-16 ● There is no conceptual limit to the distance between the two clusters. Application data is then replicated using data replication.Configuring Cluster Storage Connections The Sun Cluster Geographic Edition Software: A Cluster of Clusters Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Sun Cluster Geographic Edition allows you to implement a disasterrecovery scenario by forming a conceptual “cluster of clusters” across a wide area. while simultaneously performing the correct operations on the data replication framework. Instead it provides very simple commands to migrate an application (either nicely or forcefully) across a wide area. and are said to form a partnership. f u d ab -tran Exactly two clusters are involved in the relationship shown in the g n non diagram. A typical scenario using Sun Cluster Geographic Edition is illustrated in Figure 3-7: s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in tPrimary Figure 3-7 Data Replication d Between his and Secondary Clusters u l e a us m a o software has the following k e tEdition The Sun Cluster g Geographic properties: an b licens a b Cluster Sun u le Geographic Edition software is configured on top of k b a a l y sstandard er Sun Cluster software on the participating clusters.

Sun Services. as shown in figure t y ( S n Figure 3-9 di e this u l a us m a k e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Figure 3-9 Cluster Participating in Multiple Partnerships Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. Inc. All Rights Reserved. for a single u d k u a cluster to be a member of different partnerships.Configuring Cluster Storage Connections Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. s ) ha y m ฺ m ฺco eฺ Figure 3-8 Each Cluster is Primary for a Differentm Application cs Guid e t @ it isepossible n b While each partnership has only two clusters. Revision C 3-17 a . Oracle and/or its affiliatesฺ More Complex Partnerships Using Sun Cluster Geographic Edition It is possible to set up a Sun Cluster Geographic Edition partnership as a more symmetric entity as shown in Figure 3-8.

This allows single clusters to run in a wider area (campus or metro clusters) without having to connect nodes to storage that is far away). All Rights Reserved.Configuring Cluster Storage Connections Differences in Single Cluster and Sun Cluster Geographic Edition (two cluster partnership) Configurations Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Cluster Geographic Edition Protection Groups provide a framework for control of application migration and data replication between remote clusters. y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 3-18 Sun™ Cluster 3. but the actual migration/takeover is manual (an easy three-word command) s ) ha Single cluster configurations do support data replication as an alternative to full storage multi-porting between the nodes. Sun Services. Sun Cluster Geographic Edition depends on data replication to provide a disaster recovery scenario for data and applications that can be an arbitrarily wide distance apart. Oracle and/or its affiliatesฺ The following points reinforce the main concepts of Sun Cluster Geographic Edition by comparing its elements to a single cluster configuration: ● ● Sun Cluster Resource Groups control manual and automatic migration/failover of applications within a single cluster.2 Administration Copyright 2009 Sun Microsystems. Revision C a . Inc.

However. you could run as many nodes in the cluster as were able to boot at any point in time.Describing Quorum Votes and Quorum Devices Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. s ) ha ab y m ฺ om ฺ Quorum server quorum devices. Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. then a a r y feneed to have both nodes booted to run the cluster. The following are types of quorum devices: ● Directly attached.2 software framework operates on a voting system. a r t g an non ● NAS quorum devices. This defeats one of ul nyou s d b the a major goals of the cluster. All Rights Reserved. ● ● But why have quorum voting at all? If there were no quorum rules. the quorum vote and quorum devices solve the following two major problems: ● Failure fencing ● Amnesia prevention These are two distinct problems and it is actually quite clever that they are solved by the same quorum mechanism in the Sun Cluster 3. ● Certain devices can be identified as quorum devices and are assigned votes. If a two-node cluster had only node votes.x. Revision C 3-19 a . Following is an explanation of how this voting system operates: ● Each node is assigned exactly one vote. Oracle and/or its affiliatesฺ Describing Quorum Votes and Quorum Devices The cluster membership subsystem of the Sun Cluster 3. c ฺ ide votes csmofGallupossible There must be a majority (more than 50 percent e t present) to form a cluster or remainbin@ a cluster. making cluster management more complicated. Inc. n e u ak Stud y ( in this d u Why Have Quorum Voting e All? al usat m ka e to g n presented Given thearules ns in the bulleted items listed in the previous b e c a section. Disks are the traditional type of quorum device and have been supported in all versions of Sun Cluster 3. Other vendors’ cluster implementations have two distinct mechanisms for solving these problems. multiported disks.x software. Sun Services. it is clear li by looking at a simple two-node cluster why you need b quorum e u l extra k b device votes. which is to be able to survive node failure.

Imagine the following scenario: 1. Cluster configuration changes are made on Node 1. either because of a complete interconnect failure or a node crashing. Node 2 (1) Node 1 (1) s ) ha y m ฺ CCR Database CCR Database om ฺ c ฺ Quorum Quorum cs=m Guide device device = e @ ent b u ak Stud y ( in this QD(1) d u al use m ka eStorage to array g n ens baFailure Figure 3-10 a lic Fencing b e u l k rab a y e ul nsfPrevention d Amnesia b a tra g n n no If it were allowed to happen.2 Administration Copyright 2009 Sun Microsystems. 2. Sun Services. Both nodes attempt to reserve the designated quorum device. In a two-node cluster (Node 1 and Node 2). Inc. The node that fails the race to reserve the quorum device aborts the Sun Cluster software because it does not have a majority of votes. Node 2 is halted for maintenance or crashes. This is called split-brain operation. Oracle and/or its affiliatesฺ As shown in Figure 3-10. if interconnect communication between nodes ceases. 3.Describing Quorum Votes and Quorum Devices Failure Fencing Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. The first node to reserve the quorum device establishes a majority and remains as a cluster member. If allowed. Each node tries to establish a cluster by gaining another quorum vote. a cluster amnesia scenario would involve one aba Interconnect or more nodes being able to form the cluster (boot first in the cluster) with a stale copy of the cluster configuration. Node 1 is shut down. Two separate clusters cannot be allowed to exist because of the potential for data corruption. the cluster would have lost configuration changes. Revision C a . 3-20 Sun™ Cluster 3. each node must assume the other is still functional. All Rights Reserved. You try to boot Node 2 to form a new cluster. There is more information about how quorum votes and quorum devices prevent amnesia in ‘‘Preventing Cluster Amnesia With Persistent Reservations’’ on page 3-31. 4.

Revision C 3-21 a . but they are recommended for@ higher cluster t n b ku tude configured by A single quorum device can be automatically a y S using a shared disk quorum scinstall. a b e c a li devices are configured (specified) using DID device b Disk quorum e u l k a enames. Sun Services. n (cluster s i i d h device only. Oracle and/or its affiliatesฺ The general rules for quorum devices are as follows: ● A quorum device must be available to both nodes in a two-node cluster. All Rights Reserved. ● A traditional disk quorum device can also contain user data. even if all nodes are available. ab r y l f bdu trans a g onn a n Quorum Mathematics and Consequences ab ● ● ● ● When the cluster is running. y m ฺ om ฺ c ฺ m greater Quorum devices are not required in clusters swith idethan two c u e G availability. which is greater than 50 percent of the possible votes Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. nodes. it is always aware of the following: ● The total possible quorum votes (number of nodes plus the number of disk quorum votes defined in the cluster) ● The total present quorum votes (number of nodes booted in the cluster plus the number of disk quorum votes physically accessible by those nodes) ● The total needed quorum votes. Clearly. for a two-node only. s ) ha If the number of quorum devices equals or exceeds the number of nodes. Inc. the cluster cannot come up if too many quorum devices fail.Describing Quorum Votes and Quorum Devices Quorum Device Rules Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ● The maximum and optimal number of votes contributed by quorum devices should be the number of node votes minus one (N-1). this is unacceptable. lu se t a u are manually configured after the Sun am devices o All other quorum k t g nsinstallation e nsoftware Cluster is complete. ● Quorum device information is maintained locally in the Cluster Configuration Repository (CCR) database.

Inc. in this example). All Rights Reserved.Describing Quorum Votes and Quorum Devices Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services.2 Administration Copyright 2009 Sun Microsystems. this is typically a quorum disk. a single node can start clustered operation with a majority of votes (two votes. Revision C a . Oracle and/or its affiliatesฺ The consequences are quite simple: ● A node that cannot find the needed number of votes at boot time freezes waiting for other nodes to join to obtain the needed vote count. a two-node cluster requires a single quorum device. s ) ha y m ฺ om ฺ c ฺ Interconnect csm Guide e @ ent b u Node 1 (1) ak Stud Node 2 (1) y ( in this d u al use m ka e to g QD(1) ban licens a b u le k b a a Storage Array y fer l u s bd trFigure an 3-11 Two-Node Cluster Quorum Devices a g n non a b a 3-22 Sun™ Cluster 3. With the quorum disk. ● A node that is booted in the cluster but can no longer find the needed number of votes kernel panics. The total votes are three. Two-Node Cluster Quorum Devices As shown in Figure 3-11.

The nodes in each pair usually provide data service failover backup for one another. shown in Figure 3-12.Describing Quorum Votes and Quorum Devices Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. An entire pair of nodes can fail. Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. b a ● ● One quorum device from each pair was configured to allow one node from each pair together to form the cluster and to keep the cluster up. and there are still four votes out of seven. ● If Nodes 1 and 2 fail. Another possibility is to use a quorum server quorum device. A token quorum device between Nodes 2 and 3 can eliminate this problem. All Rights Reserved. The nodes wait until all are present (booted). ● A quorum is four votes. ● If both quorum devices fail. Revision C 3-23 a . a a r t g n non a There are six possible votes. Inc. k b a a r y Thefe following is true for a clustered pair configuration without the extra ul nquorum s d b device shown in Figure 3-12. there are not enough votes for Nodes 3 and 4 to continue running. always has an even number of cluster nodes. the cluster can still come up. Switch Switch Node 1 (1) Node 2 (1) Node 3 (1) Node 4 (1) s ) ha y m ฺ m o c ฺ ฺ QD(1) e m d s i ec t Gu @ QD(1) QD(1) b den u k Sun StorEdgeya ( is Stu MultiPackin System lud se th a u Figure 3-12 Clustered-Pair Devices am toQuorum k ng ense a b aare manylicpossible split-brain scenarios. Sun Services. Not all of the possible There b e u lcombinations split-brain allow the continuation of clustered operation. Oracle and/or its affiliatesฺ Clustered-Pair Quorum Disk Devices A clustered-pair configuration.

Revision C a .Describing Quorum Votes and Quorum Devices Pair+N Quorum Disks Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. a n ab ● ● ● 3-24 ● Nodes 3 and 4 do not have access to any quorum devices. a y l e f bdu transThere are seven possible votes. Oracle and/or its affiliatesฺ Figure 3-13 shows a typical quorum disk configuration in a Pair+2 configuration. and 4 or Nodes 2. Inc. Sun Services. ● Nodes 1 or 2 can start clustered operation by themselves. Switch Switch Node 1 (1) Node 2 (1) Node 3 (1) Node 4 (1) s ) ha y m ฺ QD(1) om ฺ c ฺ sm uide c e G QD(1) t @ n b ku tude a QD(1) y n ( this S i d lu se a m toDevices Figure 3-13 Pair+N a Quorum u k ng ense a b The following a is true lic for the Pair+N configuration shown in Figure 3-13: b e u l k Thereraare b three quorum disks.2 Administration Copyright 2009 Sun Microsystems. Sun™ Cluster 3. 3. and clustered operation can continue. All Rights Reserved. and 4). 3. ● Up to three nodes can fail (Nodes 1. Three quorum disks are used. a g onn A quorum is four votes.

All Rights Reserved. Sun Services. Node 3 is the failover backup for both Node 1 and Node 2. Revision C 3-25 a . n n o a n b a ● ● ● Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. u k a erab y l u nsf A quorum is three votes. Inc. Switch Switch Node 1 (1) Node 2 (1) Node 3 (1) s ) ha y m ฺ om ฺ c ฺ csm Guide e QD(1) QD(1) @ ent b u ak Stud y ( in this d u e al Devices Figure 3-14 N+1 Quorum s m u ka e to g The following ban lisictrue ensfor the N+1 configuration shown in Figure 3-14: a b Thereleare five possible votes.Describing Quorum Votes and Quorum Devices N+1 Quorum Disks Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. d b a tra g If Nodes 1 and 2 fail. Oracle and/or its affiliatesฺ The N+1 configuration shown in Figure 3-14 requires a different approach. Node 3 can continue.

as shown in Figure 3-15. Switch Switch Node 1 (1) Node 2 (1) Node 3 (1) s ) ha y m ฺ om ฺ c ฺ sm uide c e QD(2) G t @ n b ku tude a y ( is SStorage Topology Figure 3-15 Quorum Devices ininthe Scalable d lu se th a u devices shown in Figure 3-15: aformthe quorum The following is true o k t g nse The a single banquorum e device has a vote count equal to the votes of the c i l b unodes directly le attached to it minus one. If.Describing Quorum Votes and Quorum Devices Quorum Devices in the Scalable Storage Topology Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. ● The reservation is performed using a SCSI-3 Persistent Group Reservation. All Rights Reserved. Node 1 or 3 can reserve the quorum device on behalf of both of them. In all the previous examples. for example. Note – It would seem that in the same race. k b a a l y sfer u d an – This rule is universal. there were two ab -trNote g n non nodes (with one vote each) directly connected to the quorum device. The section entitled ‘‘Intentional Reservation Delays for Partitions With Fewer Than Half of the Nodes’’ on page 3-37 shows why this is unlikely. Oracle and/or its affiliatesฺ Quorum Devices in the scalable storage topology. differ significantly from any other topology. Node 2 could win and eliminate both Nodes 2 and 3.2 Administration Copyright 2009 Sun Microsystems. discussed in more detail later in this module. Sun Services. so a b a ● that the quorum device had one vote. 3-26 Sun™ Cluster 3. Revision C a . Nodes 1 and 3 can intercommunicate but Node 2 is isolated. ● The mathematics and consequences still apply.

Oracle and/or its affiliatesฺ Using a NAS Device as a Quorum Device You can use a NAS device as a quorum device.Describing Quorum Votes and Quorum Devices Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. This requires that the NTAPclnas package be installed on each node ● Sun NAS uses the native iSCSI built into Solaris 10. You can not use Sun NAS as quorum device in Solaris 9. In a two-node cluster. Sun Services. protocol required for a quorum device attached to more than two nodes. No other supported NAS devices have this restriction. Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. Its iSCSI LUNs do not fully support the SCSI-3 PGR. Inc. it is configured as a quorum device accessible to all nodes and given a number of votes that is one fewer than the number of nodes in the cluster. This architecture is illustrated in Figure 3-16. In a cluster with more than two nodes. it can be your single-vote quorum device. you can only use Sun NAS 5200 and 5300 as a 2-node quorum device. Note – At the time of writing this course. Revision C 3-27 a . Switch Switch s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tranFigure 3-16 NAS Device Used as a Quorum Device g n non a b a Node 1 (1) Node 2 (1) Node 3 (1) Node 4 (1) Network (not the cluster transport) NAS D evice Q D (3 votes) iSCSI LUN NAS Quorum Device Implementation The quorum implementations for NAS use iSCSI: ● NetApp uses a vendor-specific client implementation.

s ) ha y m ฺ om ฺ c ฺ Multiple clusters using the same NAS device can use separate sm ide iSCSI c u e G LUN’s on that device as their quorum devices. the requirements for operation as a Sun Cluster quorum device are as follows: ● You must install the iSCSI license from your NAS device vendor.Describing Quorum Votes and Quorum Devices Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. On the cluster side. ● When booting the cluster. Revision C a . the quorum acts like a SCSI-3 quorum device attached to all the nodes. t @ n b ku tude a y n ( this S i d lu se a am to u k ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n no aba ● 3-28 Sun™ Cluster 3. There should be no need for other quorum devices. in a cluster of more than two nodes. ● You must configure an iSCSI Logical Unit (LUN) for use as the quorum device. This is true because. Sun Services.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ On the NAS device side. All Rights Reserved. the requirements and restrictions are as follows: ● A cluster can use a NAS device for only a single quorum device. you must always boot the NAS device before you boot the cluster nodes. Inc.

Sun Services. Switch s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use Node 1 (1) am Node 2 (1) Node 3 (1) Node 4 (1) o k t ng ense a b c transport) a thelicluster b Network (not e u l k rab a y ul nsfe d b scqsd daemon (3 votes for this cluster) a tra g [ can be quorum for other clusters too ] n n o a n b a Switch External machine running Quorum Server Software Figure 3-17 Quorum Server Quorum Devices The characteristics of the quorum server quorum device are: ● The same quorum server daemon can be used as a quorum device for an unlimited number of clusters.2 introduced a new kind of quorum device called a quorum server quorum device.Quorum Server Quorum Devices Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. The daemon essentially takes the place of a directly connected quorum disk. Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. A quorum server daemon (scqsd) runs on this external machine. Inc. Figure 3-17 shows a cluster using the quorum server as the quorum device. The quorum server software is installed on some machine external to the cluster. ● No additional software is necessary on the cluster side. Revision C 3-29 a . Oracle and/or its affiliatesฺ Quorum Server Quorum Devices Sun Cluster 3. ● The quorum server software must be installed separately on the server (external side).

All Rights Reserved. Revision C a . you might not be able to run clustered applications unless the storage for a particular application is also available.2 Administration Copyright 2009 Sun Microsystems.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. any single node could boot into the cluster by itself. Oracle and/or its affiliatesฺ Quorum Server Quorum Devices ● A quorum server is especially useful when there is a great physical distance between quorum nodes. with a clustered pairs topology. For example. Inc. In that example. if it could access the quorum server. but those relationships can be controlled properly by the application resource dependencies that we will learn about in Module 10. It would be an ideal solution for a cluster using the data replication topology. you might prefer the simplicity of a quorum server quorum device. ● A quorum server can be used on any cluster where you prefer the logic of having a single cluster quorum device with quorum device votes automatically assigned to be one fewer than the node votes. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 3-30 Sun™ Cluster 3. Sun Services. Of course.

Meanwhile Node 1. 3. It is not able to count the c a i l b quorum device u le as a vote. In a two-node cluster (Node 1 and Node 2).Preventing Cluster Amnesia With Persistent Reservations Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. resource groups). the problem is that if you u e G t @ Node 2 at the end. running fine in the cluster. makes all sorts of cluster configuration changes (new device groups. Node 2 n e is no other copy dthere ku(because would have to use the copy that it a has u t y ( changes Sto the cluster configuration available) and you would loseinthe s i d h made in Step 2. 2. it would not have thebcorrect copy of the CCR. lu se t a am to u k e quorum involves persistent reservations that The Sun Cluster ng software sbooting a n b e prevent Node 2 from into the cluster. Node 2 waits until the other node k b a a boots to achieve the correct number of quorum votes. Earlier you reviewed the following scenario: 1. s ) ha y m ฺ 4. Therefore. All Rights Reserved. Revision C 3-31 a . You try to boot Node 2 to form a new cluster. Sun Services. om ฺ c ฺ m allowed ide to boot cswere In this simple scenario. Node 2 is halted for maintenance. y fer l u s bd tran a ang non ab Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. Now Node 1 is shut down. Oracle and/or its affiliatesฺ Preventing Cluster Amnesia With Persistent Reservations Quorum devices in the Sun Cluster software environment are used not only as a means of failure fencing but also as a means to prevent cluster amnesia. Inc.

Preventing Cluster Amnesia With Persistent Reservations Persistent Reservations and Reservation Keys Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a . Inc. this involves writing some type of information on the disk itself. You can think of the reservation e as the last node to u holder d k u a t ever manipulate the keys.2 Administration Copyright 2009 Sun Microsystems. lud se th a u am Figure 3-18 and Figure 3-19 onto page 3-33 consider two nodes booted into a k ngto aequorum se device. All Rights Reserved. The information is called a reservation key and is as follows: ● Each node is assigned a unique 64-bit reservation key value. cluster connected a n b a lic b e u l k rab a y ul nsfe d b a Node 2 tra Node 1 g n n o a n ● ab Quorum Device Node 1 key Node 2 key Figure 3-18 Two Nodes Booted and Connected to a Quorum Device 3-32 Sun™ Cluster 3. ● Every node that is physically connected to a quorum device has its reservation key physically written onto the device. s ) ha y m ฺ om ฺ c ฺ e Exactly one node’s key is recorded on the devicesm reservation idany c as the u holder. Oracle and/or its affiliatesฺ A persistent reservation means that reservation information on a quorum device: ● Survives even if all nodes connected to the device are reset ● Survives even after the quorum device itself is powered on and off Clearly. but the (y reservation S holder can later be n s i i fenced out by another registrant. Sun Services. This set of keys is called the keys registered on the device. but this node has no special priviligesegreater than other G t @ n b registrant.

and must o a n b a wait for Node 1 to boot. Sun Services. only u ns d b 1’s key remains registered on the device. All Rights Reserved. If Node 2 tries to boot first a tra Node g n n into the cluster. if Node 1 r y l e f subsequently goes down so there are no remaining cluster nodes. b unlessleitslreservation u k a Therefore. If a node’s key is preempted from the device. each node is racing to preempt the other’s key. it will not be able to count the quorum vote. A reservation key only gets added back to a quorum device by another node in the cluster whose key is already there. The reservation is persistent. so if a Now the rest g of k the equation e s cluster.Preventing Cluster Amnesia With Persistent Reservations Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ab in the scenario illustrated in the previous paragraph. Revision C 3-33 a . Inc. Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. it is fenced from the device. Node 1 Node 2 Quorum Device s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( n Cluster ithe Figure 3-19 Node 2 Leaves d his t u l e a us m a to is clear. If there is a split brain. it can detect Node 2 across the transport and add Node 2’s reservation key back to the quorum device so that everything is equal again. Oracle and/or its affiliatesฺ If Node 1 needs to fence out Node 2 for any reason it will preempt Node 2’s registered key off of the device. Node 1 key Node 2 key After Node 1 joins the cluster. a node cannot count the quorum device node is b booting an into nthe e c a i vote key is already registered on the quorum device.

Sun™ Cluster 3. In other words. any disk with physical connections from more than two nodes) must use SCSI-3. ● 3-34 The race (for example. it is a race to do a simple SCSI-2 reservation. It is not really a race to eliminate the other’s key. Instead. ● Reservation keys are written (by the Sun Cluster software.2 Administration Copyright 2009 Sun Microsystems. The next several subsections outline the differences between SCSI-2 and SCSI-3 reservations s ) ha y m ฺ om ฺ c ฺ e SCSI-2 Reservations and SCSI-2 Persistent Group Reservation csm G uid e Emulation (PGRE) @ ent b u d aaksimple tureservation SCSI-2 reservations themselves provide mechanism y ( S n out s i i (first one to reserve the deviced fences the other one) but it is not h t u l e persistent and does not involve a registered s keys.2 supports both SCSI-2 and SCSI-3 disk reservations. but are still directly writable by the Solaris OS). The default policy is called pathcount: ● Disks to which there are exactly two paths use SCSI-2. k b a a r y Cluster fe must make use of Persistent Group Reservation Emulation ul Sun s d n (PGRE) b a to implement the reservation keys. SCSI-2 is u amfencing sufficient to supportkthe goals in Sun Cluster but does not include o t g e the persistence required to implement amnesia prevention. not directly by the SCSI reservation mechanism) on private cylinders of the disk (cylinders that are not visible in the format command. in a split brain scenario) is still decided by a normal SCSI-2 disk reservation. The reservation keys have no impact on using the disk as a regular data disk. ● Disks to which there are more than two paths (for example. PGRE has the following a r t g an non characteristics: ab ● The persistent reservations are not supported directly by the SCSI-2 command set. ban licens a b e u lamnesia To implement prevention using SCSI-2 quorum devices. where you will not see the private cylinders. Module 5 describes how you can change the first of these policies so that you could use SCSI-3 even for disks with only two paths. The winner will then use PGRE to eliminate the other’s reservation key. All Rights Reserved. Oracle and/or its affiliatesฺ Sun Cluster 3. they are emulated by the Sun Cluster software. Revision C a .Preventing Cluster Amnesia With Persistent Reservations SCSI-2 and SCSI-3 Reservations Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. Sun Services.

All Rights Reserved. ● SCSI-3 reservations are generally simpler in the cluster because everything that the cluster needs to do (both fencing and persistent reservations to prevent amnesia) is done directly and simultaneously with the SCSI-3 reservation mechanism. With SCSI-3. Sun Services. a b Remember the single c quorum drive has three quorum votes. anycs time u e Guse SCSI-3. ngfourennodes se are all physically connected to a quorum drive. Oracle and/or its affiliatesฺ SCSI-3 reservations have the Persistent Group Reservations (PGR) mechanism built in. as it is in SCSI-2. Sun@ Cluster n must t b ku tude a y ( is S n i d SCSI-3 PGR Scenario More th Than Two Nodes luWith e a am to us k In Figure 3-20. a i l b u le k b a a l y sfer u d ab -tran Node 1 Node 3 Node 4 g Node 2 an non ● ab Node 1 key Node 2 key Node 3 key Node 4 key Quorum Device Figure 3-20 Four Nodes Physically Connected to a Quorum Drive Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. Disk firmware itself must be fully SCSI-3 compliant. s ) ha y m ฺ om ฺ c ฺ e two m more idthan With more than two disk paths (that is.Preventing Cluster Amnesia With Persistent Reservations SCSI-3 Persistent Group Reservation (PGR) Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. ● Removing another node’s reservation key is not a separate step from physical reservation of the disk. Revision C 3-35 a . They have the following characteristics: ● The persistent reservations are implemented directly by the SCSI-3 command set. the removal of the other node’s key is both the fencing and the amnesia prevention. nodes are connected to a device).

Preventing Cluster Amnesia With Persistent Reservations Now imagine that because of multiple transport failures there is a partitioning where Nodes 1 and 3 can see each other over the transport and Nodes 2 and 4 can see each other over the transport. The SCSI-3 protocol assures that only one pair will remain registered (the operation is atomic).2 Administration Copyright 2009 Sun Microsystems. One of those nodes can then add back reservation keys for Node 2 and Node 4. ul In n the Node 1 has successfully won the race to eliminate the s d b a a keys for Nodes 2 and 4. Node 2 and Node 4 cannot count the quorum device because their reservation key is eliminated. the node with the lower reservation key tries to eliminate the registered reservation key of the other pair. This is shown in Figure 3-21. Cluster amnesia is avoided in the same way as in a two-node quorum device. Because Nodes 2 and 4 have their reservation key r t g n n they cannot count the three votes of the quorum device. All Rights Reserved. Sun Services. If you now shut down the whole cluster. Oracle and/or its affiliatesฺ In each pair. Node 1 Node 2 Node 3 Node 4 s ) ha y m ฺ om ฺ c ฺ sm uide c e G Quorum Device t @ n b Node 1 key ku tud(3evotes) a Node 2 key y n ( this S i Node 3 key d u se Nodea 4lkey am to u k ng ense a b a OnelPair Figureb3-21 ic of Nodes Eliminating the Other e u l k rab a y e fdiagram. they will kernel panic. 3-36 Sun™ Cluster 3. Revision C a . no eliminated. Inc. aba Because they fall below the needed quorum. Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. They would have to wait for either Node 1 or Node 3 to join.

The only way it can win is if the other three nodes are really dead. Sun Services. Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. Revision C 3-37 a . In both implementations. that is. Fencing and amnesia prevention is provided in an analogous way to how they are provided with a SCSI-3 quorum device. t @ n b Figure 3-22. Inc. but three nodesu can csm e Gas shown in other while the fourth is isolated on the cluster transport. All Rights Reserved. The delay is implemented when the number of nodes that a node can see on the transport (including itself) is fewer than half the total nodes.Preventing Cluster Amnesia With Persistent Reservations Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ku tude a y n ( this S i d lu seNode 3 a m 2to u (isolated) aNode Node 1 Node 4 k g e (delay) ban licens a b u le k b a a l y sfer u d ab -tran g Quorum Device an non Node 1 key Node 2 key Node 3 key Node 4 key Figure 3-22 Node 3 Is Isolated on the Cluster Transport Is there anything to prevent the lone node from eliminating the cluster keys of the other three and making them all kernel panic? In this configuration. Intentional Reservation Delays for Partitions With ) has Fewer Than Half of the Nodes ฺmy ab om ฺ c ฺ idetalk to each Imagine the same scenario just presented. the keys are maintained in a persistent fashion on the server side. the state of the registration keys recorded with the quorum device survives rebooting of both the cluster nodes and the quorum server device. or if each is isolated and delaying the same amount. the lone node intentionally delays before racing for the quorum device. Oracle and/or its affiliatesฺ NAS Quorum and Quorum Server Persistent Reservations Both NAS quorum and quorum server provide reservation key-based persistent emulations.

which could otherwise m lead to o c ฺ ฺdata data corruption.2 Administration Copyright 2009 Sun Microsystems. s ) ha y m ฺ To eliminate this potential timing problem. data fencing is e u l k just rthe breservation and does not include any PGRE. lud for e a s SCSI-3 even if there are only two mpolicy uuse You can change a the to o k t paths (seenModule g n5). before a surviving node or nodes reconfigures the e m d s i c from and applications.Data Fencing Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. se a b e a do uselthe Ifbyou ic default SCSI-2 for a two-path device. The reason nodes are being eliminated is that there has been a communication failure with them. g n n a no ● ● ● ● ab 3-38 Data fencing is released when a fenced node is able to boot successfully into the cluster again. The node or nodes that remain in the cluster have no idea whether the nodes being eliminated from the cluster are actually still running. they will have a kernel panic (after they recognize that they have fallen beneath the required quorum votes). However. Sun™ Cluster 3. a a y feNetApp Filer NAS devices. the surviving node or nodes cannot wait for the other nodes to kernel panic before taking over the data. Oracle and/or its affiliatesฺ Data Fencing As an extra precaution. If they are running. The reason for this is to eliminate a potential timing problem. a surviving node informs the NAS ul nsFor d b a tra device to eliminate the NFS share from the eliminated node or nodes. All Rights Reserved. in the following manner: b den u k u treservation ya SCSI-2 With the default pathcount policy. is used for ( S n s i i two-path devices and SCSI-3 devices th with more than two paths. Inc. nodes that are eliminated from the cluster because of quorum problems also lose access to all shared data devices. it fences the eliminated node or e nodes uall shared G t @ data devices. Sun Services. Revision C a .

a a r y ul nsfe d b a tra g n n no aba Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems.aThe persistent for a disk with no fencing works nsas areservation bthe nsame e exactly way disk on which you are doing SCSI-2 fencing c a li b e u l k(the persistent b reservation is emulated using PGRE). s whereby races for the quorum devices can m u a be decided atomically and reliably without use of any SCSI-2 or SCSI-3 o k et g protocols. Sun Services. if fencing is disabled on the device. s ) ha y m ฺ om ฺ c ฺ sm uide c e Quroum Device on a Disk with No Fencing G t @ n b e ku ontua ddisk a Sun Cluster can support a quorum device on which it is doing y ( is S n i neither SCSI-2 nor SCSI-3 fencing. All Rights Reserved. either on a disk-by-disk basis or globally in the cluster. Sun Cluster can support SATA disks that are incapable of either SCSI-2 or SCSI-3 fencing in any cluster. We will see how this can be done in Modules 4 and 5. Oracle and/or its affiliatesฺ Optional Data Fencing Beginning with Sun Cluster 3. With the new option to disable fencing. Revision C 3-39 a . or disks incapable of SCSI-3 fencing in a cluster where more than two nodes are connected to the storage. Sun Cluster can also support access to a storage device from servers outside of the cluster.2 1/09 (Update 2) you have the option to disable fencing.Data Fencing Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc.Sun Cluster will implement a d e th u l “software” reservationa process. It is highly recommend to keep the fencing on normal SCSI-capable shared disks.

you must provide the During the Sun Cluster g n eninterfaces s names of the for each cable.2esoftware to installation. the ab -trwithout g n non installation hangs. the first node is installed f u s d an errors. Node 2 Node 1 s ) ha y m ฺ om ฺ c ฺ csm Guide e t board @ eSystem hme2 System board hme2 n b u ak Stud y ( in Interconnect Figure 3-23 Point-to-Point Cluster d his t u l e a us m a k 3. You have to correct the cluster configuration error on a b a the first node and then restart the installation on the second node. baend-point c a i l b u le provide the wrong interconnect interface names during k b Caution – If you a a l ythe initial erSun Cluster software installation.Configuring a Cluster Interconnect Configuring a Cluster Interconnect Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a .2 Administration Copyright 2009 Sun Microsystems. Sun Services. you can directly connect interconnect interfaces using crossover cables. Point-to-Point Cluster Interconnect In a two-node cluster. but when you try to manually install the second node. System board 3-40 hme1 hme1 System board Sun™ Cluster 3. Figure 3-23 shows a point-to-point interconnect configuration using 100BASE-T interfaces. Inc. All Rights Reserved. Oracle and/or its affiliatesฺ There are two variations of cluster interconnects: point-to-point and switchbased.

0. Sun Services. Switch s ) ha y m ฺ om ฺ c ฺ Switch csm Guide e @ ent b u ak Stud y ( Node 4 Node 2 Node 1 in thisNode 3 d u l e a us m a k e to g Figure 3-24 Switch-based Cluster Interconnect ban licens a b u le k b a a l y Note e–r If you specify more than two nodes during the initial portion of f u s d ab -tranthe Sun Cluster software installation. You can also use switches to join two-node cluster interconnects to prepare for the expansion of the number of nodes at a later time. If you answer yes. If necessary.0. the use of switches is assumed. g n non a b a Cluster Transport Interface Addresses and Netmask During the Sun Cluster software installation.2 software installation. Inc. you can override the default address. Oracle and/or its affiliatesฺ Switch-based Cluster Interconnect In cluster configurations with more than two nodes. A typical switch-based interconnect is shown in Figure 3-24. Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. you must join the interconnect interfaces using switches. but this is not recommended. the cluster interconnects are assigned IP addresses based on a base address of 172. Revision C 3-41 a . you must provide names for each of the switches. you are asked whether the interconnect system uses switches. Uniform addresses can be a benefit during problem isolation. During the Sun Cluster 3.16. All Rights Reserved.Configuring a Cluster Interconnect Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

This feature is r y l e f in detail in the Advanced Sun Cluster Administration bdu transdiscussed a course. All Rights Reserved. Oracle and/or its affiliatesฺ Configuring a Cluster Interconnect The netmask property associated with the entire cluster transport describes.0 and the default netmask of 255. the cluster uses this netmask to define the entire range it has access to.255.255.2.5. you can do it easily even with relatively large numbers of nodes and subnets.0. you would be dedicating an 12-bit range (255.0. similar to 192. if you used the default base address of 172.2 Administration Copyright 2009 Sun Microsystems.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.255. s ) ha y m ฺ omonฺ Choosing the Cluster Transport Netmask Based c ฺ e Anticipated Nodes and Private Subnetsecsm uid G t @ n b deby hand. Revision C a .255. ang non ● ● ● ab Note – In Sun Cluster 3. and zone clusters. Sun Services.16. Once again.0 – 172.240. the cluster kunetmask u a While you can choose the cluster transport t y n ( this S prefers instead that you specify: i d lu number e of private networks a s The maximum anticipated m u a kanticipated to number of nodes g e The maximum an cens b a li anticipated number of virtual clusters The b maximum e u l k a Virtual abclusters are another term for zone clusters.240. the entire range of addresses associated with the transport. Note – When you set 255.0 has 12 0’s at the end) to the cluster transport.0.16. if you want to restrict private network addresses with a class C-like space. This range is 172.0.16.0 as the cluster transport netmask you will not see this netmask actually applied to any of the private network adapters.15.240. together with the base address.168. 3-42 Sun™ Cluster 3. Inc. and then subdivides the range even further to cover the multiple separate networks that make up the cluster transport. For example.

y m ฺ # ifconfig -a om ฺ c ฺ lo0: flags=1000849<UP. To accurately determine the logical name of each interface on a system.BROADCAST. verify which interfaces are already up.Configuring a Cluster Interconnect Identifying Cluster Transport Interfaces Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved.9.MULTICAST.MULTICAST.IPv4> mtu u d k u a 1500 index 2 inet 129. host01:/# ce0 link: ce1 link: ce2 link: ce3 link: 2. Solaris 10 has a convenient command that lists all of your adapters.200. These are not candidates for a cluster transport interface.9. Oracle and/or its affiliatesฺ Identifying network interfaces is not an easy task.255 i lud se th ether 8:0:20:96:3:86 a kam e interface to u (at this point.RUNNING. Look for network interfaces. Inc.0.1 netmask ff000000 e @ ent b ce0: flags=1000843<UP.LOOPBACK. dladm show-dev unknown speed: unknown speed: unknown speed: unknown speed: 100 100 100 100 Mbps Mbps Mbps Mbps duplex: duplex: duplex: duplex: full full full full s ) ha With a public network address. Sun Services.IPv4> csm Guide mtu 8232 index 1 inet 127. use the following steps: 1.200.RUNNING. it might be an actual 3. b whenle u k a erab y l u nsf d b a tra g n n a no ab Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. or private annet. Choose g a candidate s ready to be set up as a secondary public net.cor none b e a notliconnected to anything at all).2 netmask ffffff00 broadcast t (y is S n 129. Revision C 3-43 a .0.

Just return to step 3 and pick a new candidate.1 n ( thupis S i d u try toseguess the “matching candidate” on 6. s ) ha y m ฺ omcheck 5. bring them down again.168.168. Cluster installation fails if your transport network interfaces are still up from testing. ku subnet u a t y # ifconfig ce1 192. you are looking at a public network adapter. Repeat again if you are configuring more than two cluster transports. 3-44 Repeat Steps 3 through 8 with transport adapter candidates for the second cluster transport. Now that you know your adapter is not on the public net. Sun Services.kChoose IP address.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ a. to c ฺ ฺ e m see if it is connected on a private net.2 bdu trans#192.168.1. Test b that lthe e u k a example: ab r y l e f ping 192.Configuring a Cluster Interconnect 4. Make sure that the interface you are looking at is not actually on the public net: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.168. for example g e 192.2 a is alive ang non ab 8. Perform Steps 4 and a 5 lto u am atocorresponding the other node. # ifconfig ce1 down unplumb 9. In another window. Do not continue.2. Inc. for lnodes 7. After you have identified the new network interfaces. ping the public broadcast address. an cens b a i can ping across each private network.1. In one window run the snoop command for the interface in question: # ifconfig ce1 plumb # snoop -d ce1 hope to see no output here b. e G address just to test out interconnectivity across Do t @ n b e not use addresses in the existing public dspace. Revision C a . # ping public_net_broadcast_address If you do see snoop output.1.1. and make sure no traffic is seen by the candidate interface. Sun™ Cluster 3. Make up some unused id subnet casprivate unetwork. All Rights Reserved.

Because you are identifying your private transport interfaces before cluster installation.* @ ent b u # ifconfig -a ak Stud y ( inpublicthnetwork is d You can verify your secondary adapter. it can be useful to identify your public network interfaces at the same time. All Rights Reserved. s ) ha y m ฺ om ฺ c ฺ csm Guide e # ls -l /etc/hostname. so as to avoid confusion. by: u l e a s u aitm Making sure is notto one of those you identified to be used as the k g nse ntransport private a b ceit can snoop public network broadcast traffic a i l Making sure b u le k b a a ifname plumb l y sfe##r ifconfig u d snoop -d ifname b tran (other window a or node)# ping -s pubnet_broadcast_addr g onn a n ab Your primary public network adapter should be the only one currently configured on the public network. In this course. if applicable. You can verify this with the following command: ● ● Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. you will do it after cluster installation in Module 9. The public network interfaces must be managed by IPMP. Inc. Oracle and/or its affiliatesฺ Identifying Public Network Adapters You will not be asked about public network configuration at the time you are installing the cluster.Identifying Public Network Adapters Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. which can be administered either before or after cluster installation. Revision C 3-45 a .

Configuring Shared Physical Adapters Configuring Shared Physical Adapters Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. The scinstall utility automatically detects a tagged VLAN-capable adapter and queries for the private VLAN ID. s is going to be used for the public network anVLAN nID3 bwhat e identity of would otherwise be ce1. Instead. All you need to do is ensure that you have a different VLAN ID for the public and private networks. Inc. When a a r y instance ce3001 is plumbed. you should perform the initial configuration using the scinstall utility. An adapter that is participating in a tagged VLAN configuration is assigned an instance number 1000*(Vlan_identifer) + physical_instance_number. Revision C a . therefore you configure the c a li b e u l adapter k instance b ce3001 by creating a file /etc/hostname. and a tagged VLAN with ID 5 as its private network personality. This procedure is documented in Module 4. y m ฺ om ฺ c ฺ csm Guide e @ ent b u Configuring the Public Network ak Stud y ( in this d u e network personality. and it is participating in a tagged VLAN with ID 3 as its public network personality. if you have a physical adapter ce1. 3-46 Sun™ Cluster 3. s ) ha For example. This allows blade architecture servers that have only two physical network adapters to be clustered and to still have redundant public and private networks. ab -trusing g n non a b a Allocating a Different VLAN ID for the Private Network You should never configure the private network ID manually. Oracle and/or its affiliatesฺ Recall that certain adapters are capable of participating in 802. All Rights Reserved. Sun Services. then it will appear as if it were two separate adapters ce3001 and ce5001. all you have To configure a shared adapter’s al public s m u to do is configure the kaadapter toinstance according to the mathematical g e formula above. the adapter driver understands that it is l e f u s d an tagged VLAN ID 3 on physical instance number 1. and can be used as both private and public network adapters assuming that the switches are also capable of tagged VLANs.2 Administration Copyright 2009 Sun Microsystems.ce3001.1q tagged VLANs.

the cconsole tool. s i i as IPaddress. Sun Services. You should be running a ZFS root. Oracle and/or its affiliatesฺ In this exercise. you verify that the boot disk is correctly partitioned on all g nodes. Type swap -l on each node and verify that it has at least 750 Mbytes of swap space. When you have ZFS-root. an non ab Perform the following steps: 1. Type the df -h command on each node. substitute appropriate for your cluster. m u a o k et g n ba licens a Taskku 1b– Verifying the Solaris OS le b a a l y sfer u d ab -tranIn this section. or clustername embedded in a d h t lu thesnames e a command string. Inc. Verify the root pool using zpool status.Exercise: Preparing for Installation Exercise: Preparing for Installation Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. you complete the following tasks: ● Task 1 – Verifying the Solaris OS ● Task 2 – Identifying a Cluster Topology ● Task 3 – Selecting Quorum Devices ● Task 4 – Verifying the Cluster Interconnect Configuration ● Task 5 – Selecting Public Network Interfaces Preparation s ) ha y m ฺ omhostsฺ through To begin this exercise. Node 1 memory: _______________ Node 2 memory: _______________ 2. Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. when you S n (node1. you must be connected to the cluster c ฺ m asuiuser de root. enclosure_name. 3. Type the /usr/sbin/prtconf command on each node and record the size of physical memory (/usr/sbin/prtconf | grep Memory). and you should be logged into csthem e G t @ n b de variable names. such kusee italicized u a t y Note – During this exercise. swap space is on a zvol. All Rights Reserved. Revision C 3-47 a .

for example). Sun Services. 2. quorum device configuration. Recable the storage arrays if necessary. Decide whether the scinstall utility will automatically be able to choose a quorum device (it can for a single quorum device for a twonode cluster). Inc. Quorum disks: ________________________________ Type Control-D to cleanly exit the format utility. All Rights Reserved. ab Verify that the storage arrays in your cluster are properly connected for your target topology.2 Administration Copyright 2009 Sun Microsystems. Revision C a . Oracle and/or its affiliatesฺ Perform the following steps: 1. Record the g number an software ns installation. Record the desired topology configuration of your cluster in Table 3-1.Exercise: Preparing for Installation Task 2 – Identifying a Cluster Topology Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. If there can be no automatic quorum configuration (for a three-node cluster. Table 3-1 Topology Configuration Number of nodes Number of storage arrays Types of storage arrays s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u Task 3 – Selecting Quorum Devices ak Stud y ( in this d u Perform the following steps: al use m ka ofequorum to devices you must configure after the 1. clusterbhost e c a li b e u Number of quorum devices: __________ l k b a a l y sfer u d an – Consult with your instructor if you are not sure about your ab -trNote g an non 2. 3-48 Sun™ Cluster 3. type the format command and record the logical path to the disks that you want to use as quorum disk drives in your storage arrays. Automatic Quorum Configuration: (yes/no)? 3.

Inc. Complete the form in Figure 3-26 on page 3-50 if your cluster uses an Ethernet-based cluster interconnect with switches. Determine the names of your cluster interconnect adapters. Record the logical names of the cluster interconnect interfaces (bge2. or your instructor does not want to tell you so that you gain experience doing it yourself. n this i d lu se a Node 2 Node 1 am to u k Primary ng Primary se a n b Interconnect Interconnect e c a i l b Interface Interface e u l k b ra ya fSecondary l e Secondary u s bd tran Interconnect Interconnect a g n Interface Interface n o a n b a Note – You can use the strategy presented on page 3-43. All Rights Reserved. if you are remote from the cluster equipment. Complete the form in Figure 3-25 ak if your tucluster y ( S Ethernet-based point-to-point interconnect configuration. Sun Services. Revision C 3-49 a . qfe1. Perform the following steps to configure a point-to-point Ethernet interconnect: 1. Oracle and/or its affiliatesฺ Task 4 – Verifying the Cluster Interconnect Configuration This task describes how to verify the cluster interconnect configuration.Exercise: Preparing for Installation Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. a. Do not use IP addresses in the existing public network space. you want to pretend you are remote. Figure 3-25 Ethernet Interconnect Point-to-Point Form Perform the following steps to configure a switch-based Ethernet interconnect: 3. Skip this section if your cluster interconnect is not point-to-point. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u d uses an 2. and so forth). Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems.

3-50 Sun™ Cluster 3. Revision C a . Inc. Note – If you have any doubt about the interconnect cabling. Node 1 Switch 1 Node 2 Primary interconnect interface Primary interconnect interface Secondary interconnect interface Secondary interconnect interface s ) ha y m ฺ om Node 3 Node 4ฺ c ฺ m Primary Primary s interconnect ide c u interconnect e G interface interface t @ n b Secondary Secondary ku tude interconnect a interconnect y interface interface n ( this S i d lu se a am to uSwitch 2 k ng ense a b a lic Interconnect With Switches Form b Figure 3-26 Ethernet e u l k rab a y fe ul 4. Do not continue this exercise until you are confident that your cluster interconnect system is cabled correctly.Exercise: Preparing for Installation Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. consult with your instructor now. you can verify that all nodes can ping each a other across the private switches by using the strategy presented on page 3-43. or your instructor does not want to tell you so that you gain experience doing it yourself. and you do not want to take your n non a b instructor’s word for it. Add or delete nodes to the diagram as appropriate. nsVerify d that each Ethernet interconnect interface is connected to the b a a r t g correct switch. All Rights Reserved. If you are remote.2 Administration Copyright 2009 Sun Microsystems. Sun Services. Oracle and/or its affiliatesฺ Note – You can use the strategy presented on page 3-43. you want to pretend you are remote. b. and that you know the names of the cluster transport adapters. if you are remote from the cluster equipment.

if you are remote from the cluster equipment. Inc. Perform the following steps to select public network interfaces: 1. you want to pretend you are remote. qfe3. Oracle and/or its affiliatesฺ Ask for help from your instructor in identifying public network interfaces on each node that can be used in IPMP groups.Exercise: Preparing for Installation Task 5 – Selecting Public Network Interfaces Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 3-51 a . Record the logical names of potential IPMP Ethernet interfaces on each node in Table 3-2. Do not use IP addresses in the existing public network range. Note – You can use the strategy presented on page 3-45. and so on). Preparing for Installation and Understanding Quorum Devices Copyright 2009 Sun Microsystems. or your instructor does not want to tell you so that you gain experience doing it yourself. Sun Services. All Rights Reserved. s ) ha y m ฺ om ฺ c ฺ ide Table 3-2 Logical Names of Potential IPMP Ethernet csm Interfaces u e G t @ n b Primary IPMP ku Backup deIPMP System u a t y interface ( S n this interface i d lu se Node 1 a am to u k Node 2 ng ense a b a3 (if any)lic Node b u le k b a a Node 4 (if y fer any) l u bd trans a g onn a n ab Note – It is important that you are sure about the logical name of each public network interface (ce1.

Inc. Revision C a . All Rights Reserved. or discoveries you had during the lab exercises. Sun Services. ● Experiences ● Interpretations ● Conclusions ● Applications s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 3-52 Sun™ Cluster 3.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Exercise Summary ? ! Discussion – Take a few minutes to discuss what experiences.Exercise Summary Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. issues.

Oracle and/or its affiliatesฺ Module 4 Installing and Configuring the Sun Cluster Software Framework s ) ha Objectives ab y m ฺ om ฺ Upon completion of this module.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. you should be able to: c ฺ m uide csconfiguration Understand the Sun Cluster installation and steps e G t @ n b e Java Enterprise Install the Sun Cluster packageskusing Sun a u Sthe tud System (Java ES) installer (y in this d u Describe the Sun a Cluster configuration l framework e s m u a installation Configure akcluster using all-at-once and typical modes to g e n s n installation using one-at-a-time and custom Configure ba liacecluster a b modesle u k a eConfigure ab additional nodes for the one-at-a-time method r y l f bdu trans Describe the Solaris OS files and settings that are automatically a configured by scinstall ang non ● ● ● ● ● ● ● ● Perform automatic quorum configuration ● Describe the manual quorum selection ● Perform post-installation configuration 4-1 Copyright 2009 Sun Microsystems. Revision C a . Inc. Sun Services.

Inc. Sun Services. Revision C a .Relevance Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Relevance ? ! Discussion – The following questions are relevant to understanding the content of this module: ● What advantages are we given by standardizing on the Java Enterprise System (Java ES) installer as the utility to lay down the cluster packages? ● Why might you not want the configuration utilities to automatically choose a quorum device for you for a two-node cluster? ● Are there pieces of the cluster configuration you might need to configure manually after running the configuration utility scinstall? y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha 4-2 Sun™ Cluster 3.

Additional Resources

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Additional Resources
Additional resources – The following references provide additional
information on the topics described in this module:

Sun Cluster System Administration Guide for Solaris OS, part number
820-7458 (for Sun Cluster 3.2 11/09).

Sun Cluster Software Installation Guide for Solaris OS, part number 8207356 (for Sun Cluster 3.2 11/09).

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

Installing and Configuring the Sun Cluster Software Framework
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

4-3

a

Sun Cluster Software Installation and Configuration

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Sun Cluster Software Installation and Configuration
You must complete two distinct steps to successfully initialize the
Sun Cluster 3.2 framework and boot into the cluster:
1.

Install the Sun Cluster 3.2 framework packages.

2.

Configure the Sun Cluster 3.2 software with scinstall.

It is required that these be done as separate steps, although both steps can
be automated as part of Solaris jumpstart.

Introduction to Sun Cluster Package Installation

s
) ha

y
m

An initial (non-upgrade) installation of the Sun Cluster packages
om isฺ
c

performed one of the following ways
csm Guide
e
t
@ epackaged
Using the Java ES installer utility that isbalways
with the
n
u
Sun Cluster framework (this can also
ud as part of a
ak be Sautomated
t
y
(
jumpstart install).
in this
d
u
e on a system where the Sun
Using a flash image that
al wasuscreated
m
Cluster packages
kahaveebeen
to installed using the Java ES installer. The
g
s been created before the cluster was actually
flash image
have
an must
n
b
e
c
a
configured with
li scinstall.
b
e
u
l
k
a erab
y
l
f
u Note
s
d
n

b
a Flash images are supported with a ZFS-root environment starting
a
r
t
g
an non with Solaris 10 10/09 (Update 8).

ab

Sun Cluster Packaging
You always install the Sun Cluster packages the same way despite the fact
that Sun Cluster 3.2 may be bundled with either:

The full collection of Java ES software (Sun Cluster, Java System Web
Server, Messaging Server, Directory Server, and so on).

A partial set of Java ES software, containing only availability-related
products. This is called the Java ES Availability Suite.

The download currently comes as a single zip file (for either SPARC or
x86 version) or a single DVD image containing both.

4-4

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Sun Cluster Software Installation and Configuration

Sun Cluster 3.2 Software Licensing

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

No license keys are required for the Sun Cluster software. You must,
however, furnish paper license numbers to obtain service.

Patches for the OS and for the Sun Cluster Software
You must consistently manage the following types of patches:

OS and hardware patches
These must be installed before the Sun Cluster framework packages
are installed. OS patches should be consistent between cluster nodes,
although when you add new ones after cluster configuration it is
almost always possible to do so in a rolling fashion, patching and
rebooting one node at a time.

s
) ha

ab

y
m

om ฺ
c

sm uide
c
e
G
Sun Cluster framework patches
t
@
n
b
deSun Cluster framework
ku aftertuthe
Available patches should be added
a
y
( ithe
SSun Cluster framework is
packages are installed but
n before
s
i
d
h
configured.
lu se t
a
u
am
o
You add these
patches
manually before you configure the Sun
k
t
g nsewith scinstall.
nframework
Cluster
a
b
ctoe add new patches after the cluster is already installed
ayou need
i
l
b
If
u
le
k
b
a
a
and
your
applications are already running, you will usually be able
y fer
l
u
s
bd tran to do so in a rolling fashion.
a
ang non

Sun Cluster Data Service Agent patches

You cannot add these until you install the particular agent being
patched.

Installing and Configuring the Sun Cluster Software Framework
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

4-5

a

Installing the Sun Cluster Packages With the Java™ ES Installer

Installing the Sun Cluster Packages With the Java™ ES
Installer
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

The Java ES installer provides both graphical and terminal-based
interfaces for installing the Sun Cluster software. Other Java System
applications can be installed along with Sun Cluster when Sun Cluster is
included in a full Java ES release.

Prerequisites for Installing Sun Cluster Software

s
) ha

Before you can use the Java ES installer to install the Sun Cluster software
framework on a cluster node, you must meet the following prerequisites:

y
m

m
1. Boot disks must be configured according to Sun Clusterostandards,
c


as defined in Module 3 of this course.
e
m
d
s
i
ec t Gu
2. The Solaris OS and OS patches must be installed.
@
b den
u
k
ya Stu
(
n
di e this
u
l
a us
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

4-6

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Installing the Sun Cluster Packages With the Java™ ES Installer

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Auxiliary Software Automatically Installed by Java ES
Installer
Sun Cluster 3.2 has several required auxiliary components. All of these
components are automatically installed or upgraded as needed when you
use the Java ES installer to install the Sun Cluster software.

Auxiliary Components Listed as Shared Components
The JES installer groups a large handful of auxiliary components as shared
components. These are some examples; there are several other Java libraries
and similar components..

s
) ha

y
m

oma Sunฺ Cluster
This is a Java application that serves as a container
for
c

e of Java
remote management interface that is implemented
sm asuaidseries
c
e
G this interface is
objects. In the standard Sun Cluster implementation,
t
@
n
b
e
used only by the Sun Cluster Manager
and by the data
dinterface
ku web
u
a
t
y
service wizards that are part
n ( ofthclsetup.
sS
i
i
d
Java Runtime Environment
lu se1.5
a
u
am toDevelopment
The Java Management
Kit (JMDK) runtime libraries
k
g
e
n
s
These
ba Javaliclibraries
en are required by the management objects.
a
b The Sun
u
le Java Web Console Application 3.0.2
k
b
a
a
r Java Web Console serves as a single point of entry and single
l y sfeSun
u
d
ab -tran sign-on (SSO) for a variety of Web-based applications, including the
g
Sun Cluster Manager used in the Sun Cluster software.
an non

The Common Agent Container (CACAO)


ab

Installing and Configuring the Sun Cluster Software Framework
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

4-7

a

Installing the Sun Cluster Packages With the Java™ ES Installer

Auxiliary Components Listed as Separate Applications

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

For the initial release of Sun Cluster 3.2, there is one component listed on
the Java ES install screens as a separate application.
This is Java DB, Sun’s implementation of the Apache pure-Java database
standard known as Derby.
Like the shared components, it is automatically selected when installing
the Sun Cluster software.

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

4-8

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Installing the Sun Cluster Packages With the Java™ ES Installer

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Running the Java ES Installer
If installing directly from CDROM or DVD medium, the Java ES installer
is run from CDROM 1 of 2 or from the DVD. If installing from spooled
software, recall that CDROM 1 of 2 and CDROM 2 of 2 must have
previously been spooled together into the same arena.
The installer runs as a graphical utility if you have a correct X-Windows
DISPLAY variable set in your environment. It can run as a terminal-based
utility if you explicitly run the installer with the -nodisplay option.

Launching the Java ES Installer

s
) ha

y
m

om ฺ
c

csm Guide
e
# DISPLAY=display-name-or-IP:display-#;
@ ent export DISPLAY
b
u
# cd cdrom-or-dvd-spooled-arena-path/Solaris_sparc
ak Stud
y
(
# ./installer
in this
d
u
e the initial screen for the graphical
al usshows
The screen shownainmFigure 4-1
k ES installer
to (the splash screen is not shown, and some
version of thegJava
e
nasslicense acceptance, are also omitted on following
other screens,
ban lsuch
e
c
a
i
b
pages).
e
u
l
k
a erab
y
l
u nsf
d
b
a
tra
g
n
n
a
no
You run the installer from the Solaris_sparc or Solaris_x86
subdirectory of CD-ROM 1 of 2 (or of the DVD or combined spooled
arena).

ab

Figure 4-1

Java ES Installer Initial Screen

Installing and Configuring the Sun Cluster Software Framework
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

4-9

a

Installing the Sun Cluster Packages With the Java™ ES Installer

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Component Selection Screen
You can choose the Java ES components that you want. The screen shown
in Figure 4-2 is stretched to show the entire component selection screen
from the Java ES Availability Suite distribution. Recall that this
distribution includes only the availability products.
In the screenshot, Sun Cluster 3.2 is selected. This includes the Sun Cluster
core product and the Sun Cluster Manager.
A check box near the bottom of the screen allows you to install
multilingual support. By default this is checked. Adding the multilingual
packages for Sun Cluster can significantly increase the time required for
your installation.

s
) ha

y
m

om ฺ
Note that you do not need to explicitly select the All Shared
c

Components. A later screen will inform you about which
smof these
ideneed to
c
u
e
G
be installed or upgraded, Note that the Java @
DB is listedt separately,
and
n
b
noted as required since we have chosen tokinstall
Sun
Cluster.
u ude
a
t
y
(
S
n
di e this
u
l
a us
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non

ab

Figure 4-2

4-10

Java ES Installer Component Selection

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Installing the Sun Cluster Packages With the Java™ ES Installer

Shared Components Screen

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

The screen shown in Figure 4-3 shows only the shared components which
the Java ES installer has determined must be installed or upgraded.
Note in this example some shared components listed on a previous page
of this module, such as JRE 1.5 and Web Console are not shown. This is
because the base OS we are running (Solaris 10 Update 8) already contains
the correct versions of these components; this determination is made
automatically, correctly, and silently by the Java ES installer.

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Figure 4-3

Java ES Installer shared components

Installing and Configuring the Sun Cluster Software Framework
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

4-11

a

Installing the Sun Cluster Packages With the Java™ ES Installer

Java ES Configuration: Configure Later

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

For some of the other Java System applications, the Java ES installer has
the ability to actually configure the applications. It does not have this
ability for the Sun Cluster framework, and you must choose the Configure
Later option, as shown in Figure 4-4.

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l yFigure
er Choose Configure Later for Sun Cluster
f
4-4
u
s
d
ab -tran
g
n non
a
b
a
Note – If you choose Configure Now you will get a screen saying that

this option is not applicable for Sun Cluster. However, you will also get an
extra screen asking if you want to enable the remote configuration utility
(discussed later). If you choose Configure Later as is shown here, the
installer assumes you do want to enable remote configuration.

Software Installation
After confirmation, the installer proceeds to install the shared components
(the auxiliary software identified on page 4-7), and then the actual
Sun Cluster framework packages.
You do not need to reboot your node before proceeding to the
configuration of the Sun Cluster framework. You will be allowing the
configuration utility, scinstall, to reboot all of your nodes.

4-12

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Installing the Sun Cluster Packages With the Java™ ES Installer

Configuring the User root Environment

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

The root login environment should include the following search path and
man page information:
PATH=$PATH:/usr/cluster/bin
MANPATH=$MANPATH:/usr/cluster/man:/usr/share/man

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

Installing and Configuring the Sun Cluster Software Framework
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

4-13

a

Sun Cluster Framework Configuration

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Sun Cluster Framework Configuration
The Sun Cluster configuration is performed using one of the following
two methods:

Using the scinstall utility interactively – This is the most common
method of configuring Sun Cluster, and the only one that is
described in detail in this module.

Using JumpStart software off a jumpstart server – The scinstall
can be run on the jumpstart server to provision the jumpstart server
so that the client (node) performs cluster configuration as part of
jumpstart finish scripts.
If you used this method, you would manually provision the
jumpstart server one of two ways:

s
) ha

y
m

om ฺ the
c

Provide clients a flash image from a cluster node
on which
de
isoftware
Java ES installer had been used to install
cluster
cthesm
u
e
G scinstall
t
@
packages. The jumpstart finish scripts
then run
the
n
b
de
configuration functionality. aku
u
t
y
n ( thifresh
s S OS (or a flash image from
i
Provide the clients ad
completely
lu packages.).
e
a Solaris without
cluster
The jumpstart finish scripts
a
s
m
u
a Java EStoinstaller to provide the cluster packages,
then run k
the
ng run
and
then
the
sescinstall configuration functionality.
a
n
b
e
a
lic
b
e
u
l
k rab
a
y
Understanding
the installmode Flag
ul nsfe
d
b
a
tra
g
n
n
a
no As you configure Sun Cluster software on cluster nodes and reboot the

ab

nodes into the cluster, a special flag called the installmode flag is set in
the cluster CCR. When this flag is set, the following happens:

The first node installed (node ID 1) has a quorum vote of one.

All other nodes have a quorum vote of zero.

This allows you to complete the rebooting of the second node into the
cluster while maintaining the quorum mathematics rules. If the second
node had a vote (making a total of two in the cluster), the first node
would kernel panic when the second node was rebooted after the cluster
software was installed because the first node would lose operational
quorum.

4-14

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

● ● You want to use a NAS device as a quorum device. u le k b a a l y sfer u d b tranYou can disable the two-node cluster automatic quorum configuration for a g on. @ b den u k tu yathe firstSdual-ported ( The quorum device chosen will be disk or LUN (the n di e this one with the lowest DIDlu number). Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. all the other nodes will kernel panic because they have zero votes out of a possible total of one. you have the option of having m theฺ scinstall ฺco edevice utility insert a script that automatically chooses your quorum ฺ as m d s i the second node boots into the cluster. a us m a kallow automatic to quorum configuration. Revision C 4-15 .one of the following reasons: n a n ab You want to choose the quorum device yourself. Post-installation steps to choose a quorum device and reset the installmode flag are unnecessary.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ● You want to use the quorum server as a quorum device. If the installation is a single-node cluster. the installmode g If you choose to e s after the quorum device is automatically ban licenreset flag isaautomatically b configured. Automatic Quorum Configuration (Two-Node Cluster a Only) as )h y m On a two-node cluster only. All Rights Reserved. If you accidentally reboot the first node. Oracle and/or its affiliatesฺ Sun Cluster Framework Configuration One important side effect of the installmode flag is that you must be careful not to reboot the first node (node ID 1) until you can choose quorum devices and eliminate (reset) the installmode flag. Sun Services. Inc. the installmode flag is not set. The defaults always u be to ec will G t accept the option.

the scinstall inserts a script to automatically reset the installmode flag. each node is assigned its proper single quorum vote. Inc. If you want a quorum device. in d e this u l a us m a k e to g n enmust s be able to resolve each other’s host name. It will not automatically configure a quorum device. All Rights Reserved. If The nodes b ina the cluster c a li is true but the names are not in each node’s b reason for some this e u l k a erabfile (the names could have been resolved through a name /etc/hosts y l f the /etc/hosts file is automatically modified to include these u server). s d n b a a trnames. Sun Services. Revision C a . s ) ha y m ฺ om ฺ c ฺ Name of the Cluster and Names of All the Nodes csm Guide e @ ent b u k by tthe The cluster name is just a name agreed a upon udnodes. it is not a name y ( S that resolves to an IP address. By resetting installmode. Configuration Information Required to Run scinstall The following information is required by scinstall and should be prepared in advance.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ In clusters with more than two nodes. you still have to do that manually after the installation.Sun Cluster Framework Configuration Automatic Reset of installmode Without Quorum Devices (Clusters With More Than Two Nodes Only) Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. g n n no aba 4-16 Sun™ Cluster 3.

A suggested netmask is then calculated for you. Inc.16.0. You should keep the default if it causes no conflict with anything else visible on any other network.0. On other nodes you will normally let c a li its auto-discovery b e u scinstall use feature to automatically determine the l k b a a l y transport er adapters. For SCI. This does not even need to match the actual topology. and are just used to match up transport connections between the various nodes. c If you need to specify a different IP address range for ฺthe ฺ you e m d s i can do so. the software really has no way of telling. Sun Services. Note – The netmask refers to the range of IP addresses that are reserved for all possible cluster transport addresses.0 and the netmask is 255. k tud a y n ( this S i d lu sand Cluster Transport Adapters e Switches a m u ka toe identify to transport adapters on at least the first g You must benprepared s a you nrun bwhich e node on scinstall. f u s d ab -tranYou can define a two-node cluster topology as using switches or just g an non using point-to-point cables. Revision C 4-17 a . the default cluster transport IP address range begins with 172. you will c Gu e be asked for the anticipated maximum number t and physical @ ofenodes n b u private networks. rather than being asked initially for a specific netmask.255.240. s ) ha ab y m ฺ m otransport. Port names for specific switch connections are arbitrary except for SCI switches. Names that you provide for switches are arbitrary.Sun Cluster Framework Configuration Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. as these addresses are not visible anywhere outside the cluster. This will not match the actual netmask that you will see configured on the transport adapters if you check using ifconfig -a. It is just the definitions in the Cluster Configuration Repository (CCR). the port name must match the switch port number to which an actual cable is connected. It is perfectly fine for multiple clusters to use the same addresses on their cluster transports. All Rights Reserved. Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. and whichever way you define it will be the way it is presented when you view the cluster configuration using command-line commands or the graphical web-based administration tool. Oracle and/or its affiliatesฺ Cluster Transport IP Network Number and Netmask As described in module 3.

Sun Services. Inc.2 Administration Copyright 2009 Sun Microsystems. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 4-18 Sun™ Cluster 3. It will automatically create a 100MB UFS file system using a file that it creates named /. the scinstall utility will ask if you want to use the lofi method. Revision C a . In this case do not have any placeholder file system. The standard convention is that you make an empty file system mounted on /globaldevices as a placeholder.Sun Cluster Framework Configuration Partition or Placeholder File System for the /global/. All Rights Reserved. ● Use a loop-back (lofi) device. Oracle and/or its affiliatesฺ You can choose one of two options for this file system: ● Have a dedicated partition on the boot disk (recommended size 512MB).globaldevices for the backing store. In the absence of any placeholder.devices/node@# File System Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

also known as secure remote procedure G call (secure t @ n b RPC) authentication. Sun Services. A reverse IP address lookup is done for a node trying to join the cluster. Inc. The reasons for considering more stringent authentication are the following: ● Nodes that are adding themselves to the cluster communicate across the public network. Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. it is allowed to add itself to the cluster. nodes are authenticated as they join the cluster using standard UNIX authentication. Oracle and/or its affiliatesฺ Using DES Authentication for Nodes to Authenticate With Each Other as They Join the Cluster By default. and if the resulting name is in the list of nodes that you typed in as nodes for this cluster. is a much stronger authentication u ude that cannot be k a spoofed by something simple like yspoofingSIPt addresses. All Rights Reserved.Sun Cluster Framework Configuration Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ( n di e this u l a us m a k e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a ● A bogus node adding itself to the cluster can add bogus information to the CCR or copy out the CCR. s ) ha y m ฺ om ฺ c ฺ sm uide c e DES authentication. Revision C 4-19 a .

you anabout nwhich If youbcare e c a lifrom the node that you want to be the highest node ID.Sun Cluster Framework Configuration Variations in Interactive scinstall Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. y l f bdu transThe Sun Cluster software packages must already be installed on all a g onnodes (by the Java ES installer). Revision C a . If you choose the option to configure the entire you n b e u scinstall on only one node. b drive should e u l k a anderlist abthe other nodes in reverse order. Sun Services.2 Administration Copyright 2009 Sun Microsystems. 4-20 Sun™ Cluster 3. you can use DES authentication as described on page 4-19. Therefore you do not need remote n a n shell access (neither rsh nor ssh) between the nodes. All Rights Reserved. If you are concerned about authentication. The remote ab ● ● ● configuration is performed using RPC installed by the Sun Cluster packages. Oracle and/or its affiliatesฺ There are the four different ways of using the interactive scinstall utility to configure a new cluster: ● ● Configure the entire cluster at once: ● Typical installation ● Custom installation Configure cluster nodes one at a time: ● Typical installation ● Custom installation s ) ha y m ฺ om ฺ Configuring the Entire Cluster at Once c ฺ csm Guide e t run @cluster. You should aware ak be S tudof the following y ( behavior: in this d u e be the last node to join the cluster The node you are driving will al from s m u because it needs kato configure to and reboot all the other nodes first. g e s node IDs are assigned to the nodes.

240.0.) g It uses the names switch1 and switch2 for the two transport an non ● ● ● switches and assumes the use of switches even for a two-node cluster. Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. de k a t y Sthe following responses: n ( assumes s i i The Typical configurationdmode h lu se t a u172. All Rights Reserved. g nse a b e you want to perform autodiscovery of cluster cthat Itaassumes i l b u transport le adapters on the other nodes with the all-at-once method. This becomes the sponsoring node for the remaining nodes.0 It uses network amaddress o k t for thencluster interconnect. k b a a r the one-node-at-a-time method. Inc. ● It assumes that you either have the dedicated partition identified by the placeholder /globaldevices file system.255. s ) ha ab y m ฺ Typical Installation Compared to Custom Installation om ฺ c ฺ e idand csmTypical u Both the all-at-once and one-at-a-time methodsehave Custom G t @ n b configuration options (to make a total of u fouruvariations). If you have more than two nodes you can run scinstall simultaneously on all but the first node. Revision C 4-21 a .0 with netmask 255.Sun Cluster Framework Configuration Configuring Cluster Nodes One at a Time Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. you run scinstall separately on each node.16. you should just run scinstall on the remaining nodes one at a time. If you care. and wait for each node to boot into the cluster before starting the next one. but it might be hard to predict which node gets assigned which node ID. Sun Services. or if it doesn’t detect that. ● It assumes that you want to use standard system authentication (not DES authentication) for new nodes configuring themselves into the cluster. it will ask if you want to use the lofi virtual partition. Oracle and/or its affiliatesฺ If you choose this method. You must complete scinstall and reboot into the cluster on the first node. it asks you if you want to use l y sfe(On u d ab -tran autodiscovery in both Typical and Custom modes.

In the example. The example is from a two-node cluster. Typical mode installation. # /usr/cluster/bin/scinstall *** Main Menu *** s ) ha y m ฺ om ฺ c ฺ e * 1) Create a new cluster or add a cluster node csm uid e G server 2) Configure a cluster to be JumpStarted from this install t @ n b 3) Manage a dual-partition upgrade ku tude a y ( is S 4) Upgrade this cluster node dinthis h 5) Print release informationlu for node tcluster e a am to us k * ?) Help with menu options ng ense * q) Quit a b a lic b e u l Option: 1 ak ab r y l e f s Cluster Node Menu *** du ranand bCluster *** New a g on-t n a n select from any one of the following options: ab Please Please select from one of the following (*) options: 1) Create a new cluster 2) Create just the first node of a new cluster on this machine 3) Add this machine as a node in an existing cluster ?) Help with menu options q) Return to the Main Menu Option: 1 4-22 Sun™ Cluster 3. scinstall is running on the node named theo.Configuring Using All-at-Once and Typical Modes: Example Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Configuring Using All-at-Once and Typical Modes: Example The following example shows the full dialog for the cluster installation that requires the least information. the all-at-once. Inc. All Rights Reserved. Revision C a . which becomes the second node (node ID 2).2 Administration Copyright 2009 Sun Microsystems. where the default is to let scinstall set up a script that automates configuration of the quorum device. Sun Services.

you must enable root access to all of the new member nodes from this node. Revision C 4-23 a . you can use Typical mode. All Rights Reserved. Typical mode and Custom. For more information about the differences between Typical and Custom modes.Configuring Using All-at-Once and Typical Modes: Example Selecting Option 1 from the scinstall installation menu presents a description of the requirements necessary for installation of the entire cluster from a single node as follows: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. you might need ab For to select the Custom mode option if not all of the Typical defaults can be applied to your cluster. However. then you must configure either the remote shell (see rsh(1)) or he secure shell (see ssh(1)) before you select this option. If rsh or ssh is used. Oracle and/or its affiliatesฺ *** Create a New Cluster *** This option creates and configures a new cluster. Inc. If the "remote configuration" option is unselected from the JES installer when you install the Sun Cluster framework on any of the new nodes. u ak Stud y ( in this d u e al uyes Do you want to continue (yes/no) [yes]? s m ka e to g an ceMode ns Typical or bCustom a li b e u l k b <<< aCustom aMode >>> Typical or r y l e f du rans b a tool -t supports two modes of operation. select the Help option from the menu. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b Press Control-d at any time to return to the Main Menu. gThis n n o a nmost clusters. Please select from one of the following options: 1) Typical 2) Custom ?) Help q) Return to the Main Menu Option [1]: 1 Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. You must use the Java Enterprise System (JES) installer to install the Sun Cluster framework software on each machine in the new cluster before you select this option.

What is the name of the cluster you want to establish? orangecat >>> Cluster Nodes <<< y m ฺ This Sun Cluster release supports a total of up to 16 nodes. Inc. n e u d k type Control-D: ya Stu ( n di e this u Node name (Control-D to finish): vincent l a ^D us m Node name (Control-D to finish): a k e to g ban licens a b This is the complete list u le of nodes: k b a a l y sfer u vincent d abtheo-tran g n non a b a s ) ha Is it correct (yes/no) [yes]? yes Attempting to contact "vincent" . Cluster Nodes..2 Administration Copyright 2009 Sun Microsystems. The name can be made up of any characters other than whitespace. done The Sun Cluster framework is able to complete the configuration process without remote shell access.. All Rights Reserved. Oracle and/or its affiliatesฺ Cluster Name.Configuring Using All-at-Once and Typical Modes: Example Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Each cluster name should be unique within the namespace of your enterprise. done Searching for a remote configuration method .. Revision C a . om ฺ c ฺ m uide csthe Please list the names of the other nodes planned for initial e G t @ cluster configuration.. Sun Services. When finished. List one node name per bline. and Remote Installation Confirmation >>> Cluster Name <<< Each cluster has a name assigned to it. Press Enter to continue: 4-24 Sun™ Cluster 3.

Sun Services. Inc. Oracle and/or its affiliatesฺ >>> Cluster Transport Adapters and Cables <<< You must identify the cluster transport adapters which attach this node to the private cluster interconnect.0 on adapter ce2>> NOT DUPLICATE .. No traffic was detected over a 10 second l e a s sample period.. All Rights Reserved. am to u k ng transport se adapter for "vincent": a n Select the second cluster b e a lic b e u l k rab a 1)yce1 u2)l ce2nsfe d b a a 3)-trce3 g n n no 4) Other aba Option:2 Option: 3 Will this be a dedicated cluster transport adapter (yes/no) [yes]? yes Plumbing network address 172.16. done Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems.0. Select the first cluster transport adapter for "vincent": 1) 2) 3) 4) ce1 ce2 ce3 Other s ) ha y m ฺ m o c ฺ ฺ yes Will this be a dedicated cluster transport adapter (yes/no) [yes]? e m d s i ec t Gu @ b den u k a Stu ytraffic ( Searching for any unexpected network n i is on "ce2" . done Plumbing network address 172. Revision C 4-25 a .Configuring Using All-at-Once and Typical Modes: Example Cluster Transport Adapters Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.0.. done d h t u Verification completed..0 on adapter ce3>> NOT DUPLICATE .16...

>>> Cluster Transport Adapters and Cables <<< You must identify the two cluster transport adapters which attach this node to the private cluster interconnect. Inc. you actually choose an adapter for the private network that is already configured (already using tagged VLAN) for the public network.. you will need to al network This adapter is used on the public s m u ka adapter configure it as a tagged VLAN to for cluster transport.. No traffic was detected over a 10 second g n sample period. done f u ns d b Verification a tra completed. Sun Services. All Rights Reserved. Select the first cluster transport adapter for "dani": y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y Option: 2 ( in this d u e also. Revision C a . Oracle and/or its affiliatesฺ Cluster Transport Adapters (Actually Using Shared Adapters) In the following variation.2 Administration Copyright 2009 Sun Microsystems. n no aba s ) ha 1) 2) 3) 4) 4-26 bge1 ce0 ce1 Other Sun™ Cluster 3.Configuring Using All-at-Once and Typical Modes: Example Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. g e an cens b a What is the cluster VLAN ID for this adapter? 5 li b transport e u l k b a any aunexpected r y l e Searching for network traffic on "ce5000" .

Oracle and/or its affiliatesฺ >>> Quorum Configuration <<< Every two-node cluster requires at least one quorum device. Revision C 4-27 a .Configuring Using All-at-Once and Typical Modes: Example Automatic Quorum Configuration Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. scinstall selects and configures a shared disk quorum device for you. such as Serial Advanced Technology Attachment (SATA) disks. n de once both nodes ku tquorum instead use clsetup(1M) to manually configure u a y S have joined the cluster for the first n ( time. you can either interrupt the process or check the log files after the cluster has been established. You have chosen to turn on the global fencing. cluster check is run on each of the new cluster nodes. This screen allows you to disable the automatic selection and configuration of a quorum device. By default. Inc. you must disable this feature. or if your shared disks do not support SCSI-2. If cluster check detects problems. Interrupt cluster creation for cluster check errors (yes/no) [no]? no Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. s ) ha y m ฺ om ฺ c ฺ sm oruifideyou cnow. If you disable automatic quorum device selection e G you must t @ intend to use a quorum device that is not abshared disk. If your shared storage devices do not support SCSI. Sun Services. s i i d h lu se t a u Do you want to disable automatic device selection (yes/no) am to quorum k [no]? no ng ense a b a lic b e u l k rab a y Verification and cluster check fe ul nInstallation s d b a ra tokay g n Is it to create the new cluster (yes/no) [yes]? yes n a no ab During the cluster creation process. All Rights Reserved.

Cannot use "/globaldevices" on "vincent". Cannot use "/globaldevices" on "theo". cluster check completed with no errors or warnings for "vincent". All Rights Reserved.. failed y m ฺ om ฺ c ฺ de csm Do you want to use a lofi device instead and continue theGui e @ ent installation (yes/no) [yes]? yes b u ak Stud y ( in this d u e al uMessages Installation and Configuration s m ka e to g Installation and Configuration ban licens a bof thelecluster transport configuration. Configuring "vincent" .. u k Starting discovery a erab y l u nsf d b The a following tra connections were discovered: g n n o ntheo:ce2 switch1 vincent:ce2 aba s ) ha /globaldevices is not a directory or file system mount point. Sun Services. Configuring "theo" . done Rebooting "theo" . Oracle and/or its affiliatesฺ Log file ...devices/node@# Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.log. 4-28 Sun™ Cluster 3. done Rebooting "vincent" . Started cluster check on "theo".. Inc..Configuring Using All-at-Once and Typical Modes: Example Letting You Use lofi Device for /global/. cluster check completed with no errors or warnings for "theo"./var/cluster/logs/install/scinstall.2 Administration Copyright 2009 Sun Microsystems.. Revision C a .... Do you want to use a lofi device instead and continue the installation (yes/no) [yes]? yes Testing for "/globaldevices" on "vincent" ...1251 Testing for "/globaldevices" on "theo" . failed /globaldevices is not a directory or file system mount point. Started cluster check on "vincent". theo:ce3 switch2 vincent:ce3 Completed discovery of the cluster transport configuration.

./var/cluster/logs/install/scinstall. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. Revision C 4-29 a . All Rights Reserved.log. Inc.. Oracle and/or its affiliatesฺ Log file .20852 Rebooting . Sun Services.Configuring Using All-at-Once and Typical Modes: Example Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

The dialog is shown for vincent. vincent:/# /usr/cluster/bin/scinstall *** Main Menu *** Please select from one of the following (*) options: y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( * ?) Help with menu options in this d * q) Quit u al use m ka e to Option: 1 g an cens b a i Menu *** *** New Cluster andbCluster lNode e u l k b raany ya from l e Pleaseuselect one of the following options: f s d n b a tra a new cluster g n 1) Create n n2)o Create just the first node of a new cluster on this machine aba s ) ha * 1) 2) 3) 4) 5) Create a new cluster or add a cluster node Configure a cluster to be JumpStarted from this install server Manage a dual-partition upgrade Upgrade this cluster node Print release information for this cluster node 3) Add this machine as a node in an existing cluster ?) Help with menu options q) Return to the Main Menu Option: 2 4-30 Sun™ Cluster 3. Sun Services. the first node in the cluster. All Rights Reserved.Configuring Using One-at-a-Time and Custom Modes: Example (First Node) Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. You cannot install other cluster nodes until this node is rebooted into the cluster and can then be the sponsor node for the other nodes. Inc.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Configuring Using One-at-a-Time and Custom Modes: Example (First Node) The following is an example of using the one-at-a-time node configuration. Revision C a .

Typical mode and Custom. b e u l k For mostaclusters. a acan be-trapplied g n n no more information about the differences between Typical and Custom For aba modes. Revision C 4-31 a . can use Typical mode. Use the Java Enterprise System (JES) installer to install Sun Cluster software. you might need ab youmode r y l e to select the Custom option if not all of the Typical defaults f u ns d b to your cluster. Inc. However. Before you select this option. Please select from one of the following options: 1) Typical 2) Custom ?) Help q) Return to the Main Menu Option [1]:2 Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. Sun Services. s ) ha y m ฺ m o c ฺ ฺ Do you want to continue (yes/no) [yes]? yes e m d s i ec t Gu @ b den u k ya Stu ( n Typical or Custom Mode di e this u l a us m a >>> Typical or Custom Mode k <<< to g e an cens b a This tool supports two li modes of operation. Press Control-d at any time to return to the Main Menu. All Rights Reserved.the Sun Cluster framework software must already be installed. Oracle and/or its affiliatesฺ *** Establish Just the First Node of a New Cluster *** This option is used to establish a new cluster using this machine as the first node in that cluster.Configuring Using One-at-a-Time and Custom Modes: Example (First Node) First Node Introduction Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. select the Help option from the menu.

sS i i d h lu(yes/no) e [yes]? no a Do you want to run cluster check s m u ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g an non >>> Check <<< ab 4-32 Sun™ Cluster 3. All Rights Reserved.Configuring Using One-at-a-Time and Custom Modes: Example (First Node) Cluster Name Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ >>> Cluster Name <<< Each cluster has a name assigned to it. Sun Services. If cluster check detects potential problems with configuring t @ n b this machine as a cluster node. What is the name of the cluster you want to establish? Option for cluster check orangecat s ) ha y m ฺ om ฺ This step allows you to run cluster check to verify that ฺc certain sm uhave basic hardware and software pre-configuration requirements idebeen c e G met. a report ofku violated dechecks is u a t y prepared and available for display on the n ( tscreen. Each cluster name should be unique within the namespace of your enterprise. Inc.The name can be made up of any characters other than whitespace. Revision C a .2 Administration Copyright 2009 Sun Microsystems.

Revision C 4-33 a . If DES authentication is selected. DES authentication is available. Do you need to use DES authentication (yes/no) [no]? no Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. publickey(4)).Configuring Using One-at-a-Time and Custom Modes: Example (First Node) Cluster Nodes Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. a This is the complete list of nodes: By default. However. This is generally considered adequate. Sun Services. Inc. All Rights Reserved. Please list the names of the other nodes planned for the initial cluster configuration. type Control-D: Node name (Control-D to finish): Node name (Control-D to finish): theo ^D s ) ha y m ฺ m o c ฺ ฺ vincent e m d s i theo ec t Gu @ b den u k Is it correct (yes/no) [yes]? yes ya Stu ( n di e this u l a DES Authenticating Nodes With m us a o k t ngto eAdd seNodes <<< >>> Authenticating Requests a n b a lic b e u l k node Once the first itself as a single node cluster. Oracle and/or its affiliatesฺ >>> Cluster Nodes <<< This Sun Cluster release supports a total of up to 16 nodes.since nodes which are not physically connected to the private cluster interconnect will never be able to actually join the cluster. List one node name per line. When finished. you must configure all necessary encryption keys before any node will be allowed to join the cluster (see keyserv(1M). You can modify this g n list by using claccess(1CL) or other tools once the cluster has been n o a n b established.other a rabestablishes y l e nodes u attempting to add themselves to the cluster configuration must f s d n b bea found ton ra the list of nodes you just provided. nodes are not securely authenticated as they attempt to add themselves to the cluster configuration.

Configuring a cluster with just one private interconnect provides less availability and will require the cluster to spend more time in automatic recovery if that private interconnect fails. ( n s i i when there are greater than two nodes. Revision C a . interconnect. Sun Services.Interactive scinstall switch for each private network in the cluster must be cabled to a node must be cabled to a requires that you identify one cluster.2 Administration Copyright 2009 Sun Microsystems. All Rights Reserved. Should this cluster use at least two private networks (yes/no) [yes]? yes s ) ha y Choosing Whether to Define Switches and Switch Names mฺm ฺco eฺ m >>> Point-to-Point Cables <<< cs Guid e b@ dent The two nodes of a two-node cluster may useku a directly-connected tu ya are Sconfigured. each adapter on a given different switch. And. this interactive form of d h t u l e a scinstall assumes that there m one switch for each s a will tobe uexactly k private network. Inc. no cluster switches However. What is the name of the first switch in the cluster [switch1]? <CR> What is the name of the second switch in the cluster [switch2]? <CR> 4-34 Sun™ Cluster 3. Oracle and/or its affiliatesฺ >>> Minimum Number of Private Networks <<< Each cluster is typically configured with at least two private networks. ng ense a b a cluster Does this two-node lic use switches (yes/no) [yes]? yes b e u l k rab a y ul nsfe d b a tra g n >>> Cluster Switches <<< n o a n b a All cluster transport adapters in this "switch". That is.Configuring Using One-at-a-Time and Custom Modes: Example (First Node) Asking If you Want Two Private Networks Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

All Rights Reserved. These are the adapters which attach to the private cluster interconnect. Oracle and/or its affiliatesฺ >>> Cluster Transport Adapters and Cables <<< You must configure the cluster transport adapters for each node in the cluster. for Ethernet and Infiniband switches. m us Adapter "ce2" is an Ethernet a o k t ng ensenetwork traffic on "ce2" . And. Sun Services. each port is assigned a name.. Revision C 4-35 a .Configuring Using One-at-a-Time and Custom Modes: Example (First Node) Transport Adapters and Connections to Switches Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. done a Searching for any unexpected b a lic No traffic was detected over a 10 second b Verification completed. ul nsfe d b aThe "dlpi" tra transport type will be set for this cluster. Use the default port name for the "ce2" connection (yes/no) [yes]? yes Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. you can choose to allow scinstall to assign a default name for you. Inc. e u l k rab a sampleyperiod. You can explicitly assign a name to each port. g n n no aba Name of the switch to which "ce2" is connected [switch1]? <CR> Each adapter is cabled to a particular port on a switch. Select the first cluster transport adapter: 1) 2) 3) 4) ce1 ce2 ce3 Other s ) ha y m ฺ m o c ฺ ฺ Option: 2 e m d s i ec t Gu @ Will this be a dedicated cluster transport adapter n [yes]? yes b (yes/no) e u d k ya Stu ( n di e this u l aadapter. Or. The default port name assignment sets the name to the node number of the node hosting the transport adapter at the other end of the cable..

Revision C a . y m ฺ m Searching for any unexpected network traffic on "ce3" . done o c ฺ ฺ Verification completed. ec t Gu @ b den u k Name of the switch to which "ce3" is connected <CR> tu ya S[switch2]? ( n di econnection his t u Use the default port name for the "ce3" (yes/no) [yes]? l a s yes am to u k ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n no aba s ) ha 4-36 Sun™ Cluster 3. Sun Services. No traffic was detected over am 10 second e d s i sample period. All Rights Reserved.Configuring Using One-at-a-Time and Custom Modes: Example (First Node) Second Transport Adapter Select the second cluster transport adapter: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010..2 Administration Copyright 2009 Sun Microsystems. Inc.. Oracle and/or its affiliatesฺ 1) 2) 3) 4) ce1 ce2 ce3 Other Option: 3 Will this be a dedicated cluster transport adapter (yes/no) [yes]?yes Adapter "ce3" is an Ethernet adapter.

and virtual m the maximum number of both nodeso and a k e tby a cluster.240.255.192 to meet anticipated future requirements of 4 cluster nodes and 4 private networks. bthis e c a liare based on the maximum number of nodes and b that private netmasks e u l k anetworks aband the virtual clusters that you anticipate for private r y l e f s this n bdu cluster. Inc.0.0. Maximum number of nodes anticipated for future growth [64]? 4 Maximum number of private networks anticipated for future growth [10]? 4 Maximum number of virtual clusters expected [12]? 2 Specify a netmask of 255. You can select another netmask.If this IP address is already in use elsewhere within your enterprise.16. ab s The default private netmask and network address result in an IP ha ) y 10 address range that supports a cluster with a maximum of 64 nodes. m ฺ private networks and 0 virtual clusters. a a r -t n o ang In n specifying the anticipated maximum number of nodes and private networks and virtual clusters for this cluster. Revision C 4-37 a . specify another address from the range of recommended private addresses (see RFC 1918 for details). om ฺ c ฺ de sm ui[yes]?yes c(yes/no) Is it okay to accept the default network address e G t @ n b u ude[yes]? no k(yes/no) Is it okay to accept the default netmask a t y ( S n di and his address will dictate t u The combination of private netmask network l e a us private networks.255. All Rights Reserved.255. Given your private clusters that can be supported g n a program ns will generate a range of recommended network address. Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. While both the private netmask and network address can be changed later.Configuring Using One-at-a-Time and Custom Modes: Example (First Node) Transport IP Address Range and Netmask Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. it is important that you give serious consideration to future growth potential. as long as it minimally masks all bits that are given in the network address. the tools for making such changes require that all nodes in the cluster be booted in noncluster mode. The default netmask is 255. Sun Services. Oracle and/or its affiliatesฺ >>> Network Address for the Cluster Transport <<< The cluster transport uses a default network address of 172.0.

Sun Services.255. b a tra g n n no you can use a loopback file (lofi).192 Plumbing network address 172. The lofi method is typically preferred. Global Devices File System system.0.16.globaldevices. fe This file system or partition should be at least uin s d n 512 MB size.0.devices/node@<nodeid>. The default is to use /globaldevices. scinstall k t ng ense a b c either an already-mounted file system or a nameliof You must supply the b e u l raw disk partition k rawhich b scinstall can use to create the global a y devices lfile system.255. with a new file aba Alternatively.2 Administration Copyright 2009 Sun Microsystems. Is it okay to use this default (yes/no) [yes]? no Is it okay to use the lofi method (yes/no) [yes]? yes 4-38 Sun™ Cluster 3. done s ) ha y m ฺ om ฺ c ฺ sm uide c e G >>> Global Devices File System <<< t @ n b u ude kfile a t y Each node in the cluster must have a local system mounted on ( S n s i i /global/. If the lofi method is used. awill o scinstall is run.Configuring Using One-at-a-Time and Custom Modes: Example (First Node) To accommodate more growth. and mount it on /global/. Inc.. Since the a "nodeID" assigned until m set uthis up for you. Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.. a new file system will be created for you. the file system must be empty.0 to support up to 8 cluster nodes and 8 private networks.255. scinstall creates a new 100 MB filesystem from a lofi device by using the file /.devices/node@<nodeID> before participate thnotsuccessfully lud siteiscan as a cluster member... If an already-mounted file system is used.255.192]? 255. done Plumbing network address 172.0 on adapter ce2>> NOT DUPLICATE . since it does not require the allocation of a dedicated disk slice. specify a netmask of 255. Oracle and/or its affiliatesฺ What netmask do you want to use [255.255. If a raw disk partition is used. All Rights Reserved.16.0 on adapter ce3>> NOT DUPLICATE . Revision C a .255.

leave fencing turned on. However. This screen allows you to turn off the global fencing. 2) You want to allow systems outside your cluster to access storage devices attached to your cluster. do not support SCSI. your n b e u starts you can still use the cluster(1CL) ak command tud to turn on global y ( S fencing. Oracle and/or its affiliatesฺ >>> Set Global Fencing <<< Fencing is a mechanism that a cluster uses to protect data integrity when the cluster interconnect between nodes is lost. in this d u al fencing se (yes/no) [no]? no m u Do you want to turn off a global k e to g ban licens a b u le k b a a l y sfer u d ab -tran g an non ab Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. turn off fencing when at least one of the following conditions is true: 1) Your shared storage devices. All Rights Reserved. 3) Sun Microsystems has not qualified the SCSI persistent group reservation (PGR) support for your shared storage devices. Most of the time.Configuring Using One-at-a-Time and Custom Modes: Example (First Node) Global Fencing Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. fencing is turned on for global fencing. Inc. Revision C 4-39 a . s ) ha y m ฺ om ฺ c ฺ csm Guide e t cluster @after If you choose to turn off global fencing now.and each disk uses the global fencing setting. such as Serial Advanced Technology Attachment (SATA) disks. By default. Sun Services.

s ) ha y m ฺ m o c ฺ ฺ If you disable automatic quorum device selection now. Inc.you instead use ec must G t @ clsetup(1M) to manually configure quorum once bboth nodes have joined n ku tude the cluster for the first time. scinstall selects and configures a shared disk quorum device for you.Configuring Using One-at-a-Time and Custom Modes: Example (First Node) Automatic Quorum Configuration (Two-Node Cluster) >>> Quorum Configuration <<< Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. By default.2 Administration Copyright 2009 Sun Microsystems. Sun Services. you must disable this feature. Oracle and/or its affiliatesฺ Every two-node cluster requires at least one quorum device. or if your shared disks do not support SCSI-2. If your shared storage devices do not support SCSI. All Rights Reserved. Revision C a . You have chosen to turn on the global fencing. such as Serial Advanced Technology Attachment (SATA) disks.or if you e intend m d s i u to use a quorum device that is not a shared disk. a y n ( this S i d Do you want to disable automatic luquorumsedevice selection (yes/no) a [no]? no am to u k ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n no aba 4-40 Sun™ Cluster 3. This screen allows you to disable the automatic selection and configuration of a quorum device.

endpoint=switch1 \ a >>> Confirmation <<< -m endpoint=:ce3.maxnodes=4.name=ce3 \ a tr-A g n -B -B type=switch.0.name=switch2 \ n o -m type=switch.255.nu a er\ab y l mvirtualclusters=2 u nsf d b a trtype=dlpi. Oracle and/or its affiliatesฺ >>> Automatic Reboot <<< Once scinstall has successfully initialized the Sun Cluster software for this machine.endpoint=switch2 \ -P task=quorum. Sun Services. Revision C 4-41 a .0.192. this machine will be established as the first node in the new cluster.authtype=sys \ a b c a -w li b e u l k netaddr=172.state=INIT Are these the options you want to use (yes/no) [yes]? yes Do you want to continue with the this configuration step (yes/no) [yes]? yes Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems.netmask=255.255.maxprivatenets=4.Configuring Using One-at-a-Time and Custom Modes: Example (First Node) Automatic Reboot Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. After the reboot. All Rights Reserved.16. Inc. Do you want scinstall to reboot for you (yes/no) [yes]? yes Option Confirmation s ) ha y m ฺ om ฺ Your responses indicate the following options to scinstall: c ฺ sm uide c e G Your responses indicate the following options to@ scinstall: t n b ku tude a y scinstall -i \ n ( this S i -C orangecat \ d lu se a -F \ am to u k -G lofi \ ng ense -T node=vincent.name=ce2 -A trtype=dlpi.node=theo. the machine must be rebooted.name=switch1 a n b endpoint=:ce2.

Sun Services. done bduthattrapower Verifying a g on.. done authentication options . done Log file . and IPMP) are described later in this module. done Unconfiguring n a n has been renamed to /etc/power.. All Rights Reserved.Configuring Using One-at-a-Time and Custom Modes: Example (First Node) Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.. Ensure that the EEPROM parameter "local-mac-address?" is set to "true" .conf file.conf. done configuration for adapter "ce2" .2 Administration Copyright 2009 Sun Microsystems..... done configuration for cable ... done configuration for switch "switch1" ...... done (id=1) csm Guide e @ ent b u Verifying that NTP is configured .conf s ) ha Power management is incompatible with the HA goals of the cluster...1065 Rebooting ... done configuration for adapter "ce3" .log..... done ( in this d u l e Updating nsswitch. nsswitch......../var/cluster/logs/install/scinstall.. Revision C a . done configuration for switch "switch2" ... hosts file..... done Ensure network routing is disabled .done r y l e f ns management is NOT configured .. 4-42 Sun™ Cluster 3.conf . done private network address options ... done ak Stud y Initializing NTP configuration . done ma s u ka e to g Adding cluster node entries /etc/inet/hosts . Inc. done y m ฺ om ฺ c ฺ Setting the node ID for "vincent" . done ns ban lto e c a i b e u l k Configuring IP amultipathing ab groups . done configuration for cable . Please do not attempt to re-configure power management. Oracle and/or its affiliatesฺ Configuration Messages Some of the post-installation steps for which scinstall is printing out messages here (NTP.power management . Initializing Initializing Initializing Initializing Initializing Initializing Initializing Initializing Initializing cluster name to "orangecat" .041309025345 ab/etc/power..

Here. Here you have no choice about the automatic quorum selection or the authentication mechanism because it was already chosen on the first node. Oracle and/or its affiliatesฺ Configuring Additional Nodes for One-at-a-Time Method: Example In the one-at-a-time method. Inc. The only difference is that the Typical mode does not ask about the global devices file system (it assumes the placeholder is /globaldevices). you can configure the remaining node or nodes. Sun Services. there is hardly any difference between the Typical and Custom modes.Configuring Additional Nodes for One-at-a-Time Method: Example Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. theo:/# scinstall s ) ha y m ฺ om ฺ c ฺ Please select from one of the following (*) options: csm Guide e @ ent b u * 1) Create a new cluster or add a cluster k node a tud this install server y ( 2) Configure a cluster to be JumpStarted from S din e this 3) Manage a dual-partitionluupgrade anode us 4) Upgrade this cluster m a k e to for this cluster node 5) Print release ginformation an cens b a * ?) Help b with menulioptions u le k b * q) Quit a a l y sfer u d b tra1n aOption: g an non *** Main Menu *** ab *** New Cluster and Cluster Node Menu *** Please select from any one of the following options: 1) Create a new cluster 2) Create just the first node of a new cluster on this machine 3) Add this machine as a node in an existing cluster ?) Help with menu options q) Return to the Main Menu Option: 3 Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. All Rights Reserved. Revision C 4-43 a . after the first node has rebooted into the cluster.

g n For most can use Typical mode. Inc. e m d s i ec t Gu @ b den u k Do you want to continue (yes/no) [yes]? y yes a Stu ( n di e this u l a us m a Typical or CustomkMode to ng ense a b a lic b e u l k raMode >>> Typical or aCustom b <<< y l e f u ns bdtool This two modes of operation.the Sun Cluster framework software must already be installed. a a rsupports tclusters. If this is a new cluster. However. Before you select this option.2 Administration Copyright 2009 Sun Microsystems. s ) ha y m ฺ m o c ฺ ฺ Press Control-d at any time to return to the Main Menu. Oracle and/or its affiliatesฺ Additional Node Configuration and Choosing Typical Instead of Custom *** Add a Node to an Existing Cluster *** This option is used to add this machine as a node in an already established cluster. Please select from one of the following options: 1) Typical 2) Custom ?) Help q) Return to the Main Menu Option [1]: 1 4-44 Sun™ Cluster 3. Typical mode and Custom. All Rights Reserved. Use the Java Enterprise System (JES) installer to install Sun Cluster software. select the Help option from the menu. For more information about the differences between Typical and Custom modes. there may only be a single node which has established itself in the new cluster. Sun Services.Configuring Additional Nodes for One-at-a-Time Method: Example Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a . you might need n o the Customyoumode a n b to select option if not all of the Typical defaults a can be applied to your cluster.

However.Configuring Additional Nodes for One-at-a-Time Method: Example Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. >>> Cluster Name <<< Each cluster has a name assigned to it.. >>> Sponsoring Node <<< For any machine to join a cluster. Sun Services. the "sponsoring" node can be any node in that cluster. you must identify the name of the cluster you are attempting to join.if the cluster is already established. s ) ha ab y m ฺ om ฺ c ฺ csm Guide e @ ent b u k tud aauthenticate y And. you may need to add it by using claccess(1CL) or other tools. A sanity check is performed to verify that the "sponsoring" node is a member of that cluster.. done Cluster name "orangecat" is correct. ng ense a b a of the What is the b name lic sponsoring node? vincent e u l k rab a y ul nsfe d b a tra Cluster Name g n n a no Already established clusters can keep a list of hosts which are able to configure themselves as new cluster members. Oracle and/or its affiliatesฺ Sponsoring Node After entering the name of the sponsoring node (first node already booted into the cluster and the cluster name).This machine should be in the join list of any cluster which it tries to join. All Rights Reserved. the authentication and the cluster name are checked with the sponsoring node. If the list does not include this machine. What is the name of the cluster you want to join? orangecat Attempting to contact "vincent" . When adding a node to the cluster. it must identify a node in that cluster willing to "sponsor" its membership in the cluster. this "sponsor" node is typically the first node used to build the new cluster. Revision C 4-45 a . if the target cluster uses DES (to new machines S nas new s i i attempting to configure themselves cluster members. the d h t u l e a necessary encryption keys m before any attempt to s uconfigured a must tbe o k join. Inc. When configuring a new cluster. Press Enter to continue: Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems.

Cannot use "/globaldevices". All Rights Reserved. Revision C a . n ( this S i d lu(yes/no) e [yes]? yes a Do you want to use autodiscovery s m u ka e to g an cens b a Probing .. Sun Services.2 Administration Copyright 2009 Sun Microsystems.. Do you want to run cluster check (yes/no) [yes]? no Cluster Transport Autodiscovery Option s ) ha y m ฺ om ฺ >>> Autodiscovery of Cluster Transport <<< c ฺ sm uide c e G If you are using Ethernet or Infiniband adapters as the tcluster @ n b transport adapters... Do you want to use a lofi device instead and continue the installation (yes/no) [yes]? yes 4-46 Sun™ Cluster 3.devices/node@# /globaldevices is not a directory or file system mount point. If cluster check detects potential problems with configuring his machine as a cluster node. a report of violated checks is prepared and available for display on the screen.. Inc.Configuring Additional Nodes for One-at-a-Time Method: Example Option for cluster check Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010... Oracle and/or its affiliatesฺ >>> Check <<< This step allows you to run cluster check to verify that certain basic hardware and software pre-configuration requirements have been met. li b e u l k a connections ab r y l e The following were discovered: f u ns d b a tra g n vincent:ce2 switch1 theo:ce2 n o a n b vincent:ce3 switch2 theo:ce3 a Is it okay to add these connections to the configuration (yes/no) [yes]? yes Letting You Use lofi Device for for /global/. autodiscovery is the best defor configuring ku method u a t y the cluster transport.

The reboot will cause this machine to join the cluster for the first time.Configuring Additional Nodes for One-at-a-Time Method: Example Automatic Reboot and Option Confirmation Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. the machine must be rebooted.name=ce2 -A trtype=dlpi.name=ce3 \ ya \ Stu ( -m endpoint=:ce2. Do you want scinstall to reboot for you (yes/no) [yes]? yes >>> Confirmation <<< Your responses indicate the following options to scinstall: s ) ha y m ฺ m scinstall -i \ o c ฺ ฺ -C orangecat \ e m d s i -N vincent \ ec t Gu @ -G lofi \ b den u k -A trtype=dlpi. Oracle and/or its affiliatesฺ >>> Automatic Reboot <<< Once scinstall has successfully initialized the Sun Cluster software for this machine. All Rights Reserved. Inc.endpoint=switch1 n di e this u -m endpoint=:ce3.endpoint=switch2 l a us m a o use (yes/no) [yes]? yes k want tto Are these the options you g e ban licens a Do you wantub to continue le with this configuration step (yes/no) [yes]? k b a a yes l y sfer u d ab -tran g an non ab Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. Revision C 4-47 a . Sun Services.

...Configuring Additional Nodes for One-at-a-Time Method: Example Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.. done Ensure network routing is disabled .conf...conf..1091 Rebooting . done adapter "ce2" to the cluster configuration ../var/cluster/logs/install/scinstall.conf .conf Power management is incompatible with the HA goals of the cluster..... done Updating file ("hosts") on node vincent .. s ) ha Verifying the major number for the "did" driver with "vincent" .. done Copying the config from "vincent" . done Copying the postconfig file from "vincent" if it exists ..041309030943 no ab/etc/power.. done a trpower Unconfiguring g n n a has been renamed to /etc/power.....log.. done Setting the node ID for "theo" .cluster") on node vincent .. done Updating file ("ntp. done (id=2) y m ฺ m o c ฺ ฺ Verifying that NTP is configured .... done ec t Gu @ b den u k Updating nsswitch.. done cable to the cluster configuration .2 Administration Copyright 2009 Sun Microsystems.done a n Configuring IP multipathing b e a lic b e u l k rab a y l femanagement is NOT configured . done cable to the cluster configuration . 4-48 Sun™ Cluster 3..... done e m d s i Initializing NTP configuration . Sun Services. done s Verifyingdu that power n b a management . Ensure that the EEPROM parameter "local-mac-address?" is set to "true" . done Log file ...... Revision C a . Oracle and/or its affiliatesฺ Configuration Message Adding Adding Adding Adding Adding node "theo" to the cluster configuration .... done ya Stu ( n is di e th.. All Rights Reserved. done adapter "ce3" to the cluster configuration ... Inc. done Please do not attempt to re-configure power management.. u Adding cluster node entries to /etc/inet/hosts done l a s am to u k nggroups se.

Inc.) ya scinstall l e f u s bd tran a ang nonChanges to the /etc/nsswitch. for b e u l k b ra to work at all. Oracle and/or its affiliatesฺ Solaris OS Files and Settings Automatically Configured by scinstall No matter which of the four scinstall variations that you use. scinstall automatically configures the following files and settings on each cluster node: ● /etc/hosts ● /etc/nsswitch. ● It adds the cluster keyword for the hosts and netmasks keywords. These names can be used by any utility or application as normal resolvable names without having to be entered in any other name service.xxx s ) ha y m ฺ /etc/notrouter om ฺ c ฺ local-mac-address? – Setting in electrically erasableide sm c u only) e programmable read-only memory (EEPROM) (SPARC G t @ b den u k ya Stu ( n Changes to the /etc/hosts di eFile his t u l a us m a kutility eautomatically to The scinstall adds all the cluster names and IP g n s a n addresses hosts file if it was not there already. through some name service.conf File ● /etc/vfstab ● ● ab The scinstall utility makes the following changes: ● It makes sure the files keyword precedes every other name service for every entry in the file. (All the b to each cenode’s aalready i l names had to be resolvable.cluster ● /etc/hostname. This keyword modifies the standard Solaris OS resolution libraries so that they can resolve the cluster transport host names and netmasks directly from the CCR. Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. Sun Services. clusternode2-priv. and so on.conf ● /etc/inet/ntp. Revision C 4-49 a .Solaris OS Files and Settings Automatically Configured by scinstall Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. The default transport host names (associated with IP addresses on the clprivnet0 adapter) are clusternode1-priv.conf. All Rights Reserved.

but not yet in IPMP groups.xxx files with IP addresses on the same subnet. scinstall detects that IPMP is already configured.qfe1 k b a a l yvincent ergroup sc_ipmp0 -failover f u s d n ab -trIfayou g happen to already for some reason have multiple an non ab /etc/hostname. You are still likely to need to modify these by hand. if used. 4-50 Sun™ Cluster 3. Oracle and/or its affiliatesฺ Creating the /etc/inet/ntp. and therefore should not need to be modified manually. to give separate node and test addresses on your primary adapter so that the node address itself can fail over. the node IP address) is u just amfollowing o k t as the test address.conf. has all nodes synchronize their time clocks against each other (and only against each other). Of course it is perfectly legal to fully configure IPMP with as much customization as you like before scinstall is run. Whatever IP address in the file (for u l e a s given the -failover flag and used example. y m ฺ om ฺ c ฺ Modifying /etc/hostname. scinstall will rewrite both files to place them in the same IPMP group.Solaris OS Files and Settings Automatically Configured by scinstall Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a . All Rights Reserved. Inc.2 Administration Copyright 2009 Sun Microsystems. For a two-node cluster it includes the lines: peer clusternode1-priv prefer peer clusternode2-priv s ) ha The entire file is ignored if there is a standard /etc/inet/ntp.xxx Files to Include de csm GuiIPMP e @ ent b u Any existing /etc/hostname. as modified by g e n s scinstall:ba en c a i l b cat vincent:/# u le /etc/hostname.cluster File This file contains a Network Time Protocol Configuration which. The shows an example file.xxx files that do dyet indicate IPMP k not u a t y group membership are rewritten so (that the adapter S in question is placed n thwas s already i i d in an IPMP group. Sun Services. This file is automatically modified to contain only lines for cluster nodes defined during scinstall.conf file at boot time. for example. and leaves it alone. We will cover this in Module 9.

such as /globaldevices.devices/node@# mount point replaces any previous placeholder. It is not supported to have a cluster node themselves k b a a l y function er as a router. Inc. Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. Revision C 4-51 a .devices/node@1 ufs 2 no global s ) ha y m ฺ om ฺ c ฺ Note – If you use a lofi device it is configured and smmounted idevia explicit c u e SMF actions automatically inserted by scinstall. This is described in more detail in Module 9. There G is no entry in t @ n b /etc/vfstab. Ethernet address for Ethernet adapters) in order to support IPMP.Solaris OS Files and Settings Automatically Configured by scinstall Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ku tude a y n ( this S i d lu se a Creating the /etc/notrouter File am to u k ng ense a b a filelicassures that cluster nodes do not accidentally turn This empty b u leinto routers. The following shows the modification to the vfstab file: vincent:/etc/inet# grep global /etc/vfstab #/dev/dsk/c0t0d0s3 /dev/rdsk/c0t0d0s3 /globaldevices ufs 2 yes /dev/did/dsk/d1s3 /dev/did/rdsk/d1s3 /global/. this EEPROM variable is set to true so that each network adapter is given a unique Media Access Control (MAC) address (that is. This ensures that each of these devices has a unique name cluster-wide. Sun Services. All Rights Reserved.devices/node@#) The vfstab file is modified so that the /global/. In addition. the DID device name is used for this file system rather than the traditional /dev/rdsk/c#t#d#. Oracle and/or its affiliatesฺ Modifying the /etc/vfstab File (dedicated partition for /global/. f u s d ab -tran g n non a b Modifying the local-mac-address? EEPROM variable a On SPARC systems.

Apr 13 03:17:39 theo cl_runtime: NOTICE: CMM: Votecount changed from 0 to 1 for node theo. y m ฺ om ฺ c ฺ In the Solaris 10 OS. Sun Services. a n b c a li b e u l k The a following ab console messages indicate the automatic quorum selection: r y l e f s cl_runtime: NOTICE: CMM: Cluster members: vincent du rantheo Apr 13 b 03:17:39 a theo. the auto-configuration of the quorum happens as part of a boot script on the last node booting into the cluster. Apr 13 03:17:39 theo cl_runtime: NOTICE: CMM: Cluster members: vincent theo. This is because k the is de aOS. If your cluster has more than two nodes. sm you idgete the u e Gthe quorum login prompt on the last node booting into the@ cluster before t n b u bootuenvironment auto-configuration runs. Apr 13 03:17:39 theo cl_runtime: NOTICE: CMM: node reconfiguration #5 complete 4-52 Sun™ Cluster 3. Revision C a . All Rights Reserved. Dotnot attempt to configure the quorum device ngthe eauto-configuration se by hand because eventually runs to completion. g on-t n a n theo cl_runtime: NOTICE: CMM: node reconfiguration #4 abApr 13 03:17:39 completed. as the last node boots into the c cluster. Inc. but the installmode flag is automatically reset as the last node boots into the cluster. no quorum device is selected automatically. Oracle and/or its affiliatesฺ Automatic Quorum Configuration and installmode Resetting On a two-node cluster on which you chose to allow automatic quorum configuration. The auto-configuration of the s quorum device does not complete m u a o until a minute or soklater.Automatic Quorum Configuration and installmode Resetting Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems. the quorum device is chosen (the lowest possible DID device number) as the second node boots into the cluster for the first time. It will be complete by the time you get the login prompt on this last node. t y controlled by the SMF of the Solaris(10 which runs boot services in S n s i i parallel and gives you the login thbefore many of the services are lud prompt e a complete. s ) ha On the Solaris 9 OS.

Revision C 4-53 a .Manual Quorum Selection Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. b e u l k a erab y l f – The local disks (single-ported) appear at the beginning and end of u nNote s d b a tra the output and cannot be chosen as quorum devices. Inc. a us m a k e to g sDID device number for the quorum device you You needa tonknow the n b e c a choose in the next li step. you need to know the DID device number the om for c ฺ ฺ quorum device or devices that you want to choose. csm Guide e t assigned to the @DID enumbers n b The cldevice (cldev) command shows the u d shows the mapping akoption tuthat disks in the cluster. The most succinct y ( S inthe corresponding is between DID numbers anddall disk paths is cldev h t u l e list -v. Oracle and/or its affiliatesฺ Manual Quorum Selection You need to choose a quorum device or quorum devices manually in the following circumstances: ● Two-node cluster where you disabled automatic quorum selection ● Any cluster of more than two nodes where a quorum device is desired Verifying DID Devices ab s If you are going to be manually choosing quorum devices that are ) ha yquorum physically attached disks or LUN’s (as opposed to a NAS device m ฺ or quorum server). Sun Services. g n n a no vincent:/# cldevice list -v DID Device Full Device Path ------------------------d1 vincent:/dev/rdsk/c0t0d0 d2 vincent:/dev/rdsk/c0t1d0 d3 vincent:/dev/rdsk/c0t6d0 d4 vincent:/dev/rdsk/c1t0d0 d4 theo:/dev/rdsk/c1t0d0 d5 vincent:/dev/rdsk/c1t1d0 d5 theo:/dev/rdsk/c1t1d0 d6 vincent:/dev/rdsk/c1t2d0 d6 theo:/dev/rdsk/c1t2d0 d7 vincent:/dev/rdsk/c1t3d0 d7 theo:/dev/rdsk/c1t3d0 d8 vincent:/dev/rdsk/c1t8d0 d8 theo:/dev/rdsk/c1t8d0 Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. You can choose any multiported disk. All Rights Reserved.

Oracle and/or its affiliatesฺ Manual Quorum Selection d9 d9 d10 d10 d11 d11 d12 d12 d13 d13 d14 d14 d15 d15 d16 d16 d17 d17 d18 d18 d19 d19 d20 d21 d22 vincent:/dev/rdsk/c1t9d0 theo:/dev/rdsk/c1t9d0 vincent:/dev/rdsk/c1t10d0 theo:/dev/rdsk/c1t10d0 vincent:/dev/rdsk/c1t11d0 theo:/dev/rdsk/c1t11d0 vincent:/dev/rdsk/c2t0d0 theo:/dev/rdsk/c2t0d0 vincent:/dev/rdsk/c2t1d0 theo:/dev/rdsk/c2t1d0 vincent:/dev/rdsk/c2t2d0 theo:/dev/rdsk/c2t2d0 vincent:/dev/rdsk/c2t3d0 theo:/dev/rdsk/c2t3d0 vincent:/dev/rdsk/c2t8d0 theo:/dev/rdsk/c2t8d0 vincent:/dev/rdsk/c2t9d0 theo:/dev/rdsk/c2t9d0 vincent:/dev/rdsk/c2t10d0 theo:/dev/rdsk/c2t10d0 vincent:/dev/rdsk/c2t11d0 theo:/dev/rdsk/c2t11d0 theo:/dev/rdsk/c0t0d0 theo:/dev/rdsk/c0t1d0 theo:/dev/rdsk/c0t6d0 s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a r l y sfeQuorum u Choosing and Resetting the installmode d b tran a (Two-Node Cluster) g Attribute nn o a n ab Before a new cluster can operate normally. the installmode will still be set on the cluster. All Rights Reserved.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. You must choose a quorum device as a prerequisite to resetting installmode. Inc. Sun Services. Revision C a . On a two-node cluster where automatic quorum selection was disabled. 4-54 Sun™ Cluster 3. you must reset the installmode attribute on all nodes.2 Administration Copyright 2009 Sun Microsystems.

However. and will not present any of its normal menus until you reset it. Revision C 4-55 a . Oracle and/or its affiliatesฺ Choosing Quorum Using the clsetup Utility The clsetup utility is a menu-driven interface which. then resetting both the quorum vote counts and the "installmode" property. Sun Services. Please k b Following are supported a a r y SunfeCluster l to refer documentation for detailed information on these u s d b tranquorum device topologies. y m ฺ om ฺ c ฺ csm Guide e @ ent b u Please do not proceed if any additional ud yet to join the aknodesSthave y ( cluster. turns into a general menu-driven alternative to low-level cluster commands. Dual-ported SCSI-2 disks may be used as quorum devices in two-node clusters. clusters with more than two nodes require that Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. # /usr/cluster/bin/clsetup >>> Initial Cluster Setup <<< s ) ha This program has detected that the cluster "installmode" attribute is still enabled. The clsetup utility recognizes if the installmode flag is still set. certain initial cluster setup steps will be performed at this time. when the installmode flag is reset. All Rights Reserved.Manual Quorum Selection Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. in this d u e yes al [yes]? s Is it okay to continue (yes/no) m u ka e to g Do you want to add bananylicquorum ens devices (yes/no) [yes]? yes a b u le Quorum Devices types in Sun Cluster. This includes adding any necessary quorum devices. As such. Inc. this involves choosing a single quorum device first. This is described in more detail in Module 5. For a two-node cluster. supported a g onn a n ab What is the type of device you want to use? 1) Directly attached shared disk 2) Network Attached Storage (NAS) from Network Appliance 3) Quorum Server q) Option: 1 >>> Add a SCSI Quorum Disk <<< A SCSI quorum device is considered to be any Sun Cluster supported attached storage which connected to two or more nodes of the cluster.

For more information on supported quorum device topologies. Inc. Is it okay to continue (yes/no) [yes]? yes Which global device do you want to use (d<N>)? d4 Is it okay to proceed with the update (yes/no) [yes]? yes s ) ha y m ฺ clquorum add d4 om ฺ c ฺ Command completed successfully. Oracle and/or its affiliatesฺ SCSI-3 PGR disks be used for all disks with more than two node-to-disk paths. All Rights Reserved. this program will Once the "installmode" banproperty e c a li each time it is run again in the future. You can use a disk containing user data or one that is a member of a device group as a quorum device. y quorum a edevices ab can always be added to the cluster using the r l f regular options. Revision C a . Resetting this property fully activates quorum s n bdu menu a a r settings and is necessary for the normal and safe operation of the t g n n cluster. csm Guide e @ ent b u ak Stud y ( Press Enter to continue: in this d u al device se (yes/no) [yes]? no Do you want to add another a quorum m u k e to g ns has been reset. Sun Services. see the Sun Cluster documentation. no aba Is it okay to reset "installmode" (yes/no) [yes]? yes clquorum reset claccess deny-all Cluster initialization is complete. Type ENTER to proceed to the main menu: 4-56 Sun™ Cluster 3.Manual Quorum Selection Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. skip "Initial uCluster Setup" b e l k However.2 Administration Copyright 2009 Sun Microsystems.

Manual Quorum Selection *** Main Menu *** Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. Revision C 4-57 a . Sun Services. Oracle and/or its affiliatesฺ Please select from one of the following options: 1) 2) 3) 4) 5) 6) 7) 8) Quorum Resource groups Data Services Cluster interconnect Device groups and volumes Private hostnames New nodes Other cluster tasks ?) Help with menu options q) Quit s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. Inc.

but the initial screens look a little different because the installmode flag is already reset. Inc. 4-58 Sun™ Cluster 3. Sun Services. # /usr/cluster/bin/clsetup *** Main Menu *** s ) ha y m ฺ om ฺ c ฺ sm uide 1) Quorum c e G 2) Resource groups t @ n b 3) Data Services ku tude a y 4) Cluster interconnect n ( this S 5) Device groups and volumes udi al use 6) Private hostnames m ka e to 7) New nodes g 8) Other clusteraproperties b n licens a ub menu leoptions ?) Helpak with b a l y sfer q)uQuit d ab -tran g n Option: on 1 a n b a Please select from one of the following options: *** Quorum Menu *** Please select from one of the following options: 1) Add a quorum device 2) Remove a quorum device ?) Help q) Return to the Main Menu Option: 1 From here. the dialog looks similar to the previous example. Revision C a . You should use clsetup to choose quorum devices. but the quorum device or devices are never automatically selected. All Rights Reserved. except that the installmode is already reset so after adding your quorum devices you just return to the main menu.Manual Quorum Selection Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Choosing a Quorum Device (Clusters With More Than Two Nodes) In a cluster of more than two nodes the installmode flag is always automatically reset.2 Administration Copyright 2009 Sun Microsystems.

such as: y m ฺ om ฺ c ฺ Quorum votes (including node and device quorum ide csm Gvotes) u e Device groups @ ent b u Resource groups and relatedyresources ak Stud ( in this Cluster interconnectustatus d al use m ka e to g Note – The command-line interface (CLI) commands are described ancluster ns b e c a in detail starting li in the next module and continuing on a per-subject basis b configure e u l k as you storage and applications into the cluster in Modules 6 b a a r y fe 11. Oracle and/or its affiliatesฺ Performing Post-Installation Verification When you have completed the Sun Cluster software installation on all nodes. Sun Services. All Rights Reserved. ul nthrough s d b a tra g n n no aba ● Nodes ● ● ● ● Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems.Performing Post-Installation Verification Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 4-59 a . verify the following information: ● General cluster status ● Cluster configuration information Verifying General Cluster Status s ) ha The status sub-command of the cluster utilities shows the current status of various cluster components. Inc.

Performing Post-Installation Verification The following two commands give identical output. and show the cluster membership and quorum vote information: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.Quorum Votes by Device ---ka ng ense a b a Device Name Present Status lic Possible b e u l ----------------------k ------b a a r y d4 1 Online ul ns1fe d b a tra g n n no The following two commands are identical. All Rights Reserved. Oracle and/or its affiliatesฺ vincent:/# cluster status -t quorum vincent:/# clquorum status Cluster Quorum === --. Inc. Sun Services.Quorum Votes Summary --Needed -----2 Present ------3 Possible -------3 s ) ha y m ฺ om ฺ c ฺ Node Name Present Possible Status sm uide c e G ---------------------------@ t n b vincent 1 1 Online ku tude a y theo 1 1 Online n ( this S i d lu se a m to u --.Quorum Votes by Node --- networks that make up the cluster interconnect (cluster transport): vincent:/# cluster status -t interconnect vincent:/# clinterconnect status Cluster Transport Paths === Endpoint1 --------vincent:ce3 vincent:ce2 4-60 Endpoint2 --------theo:ce3 theo:ce2 Status -----Path online Path online Sun™ Cluster 3. Revision C a .2 Administration Copyright 2009 Sun Microsystems. and show status of the private aba --.

list -v. The following command shows the configuration of everything. If you added a -t global at the end of the command it would list only the cluster global properties which appear in the first section of output. Inc. Revision C 4-61 a . Oracle and/or its affiliatesฺ Cluster configuration is displayed in general using the list.0 255.0.255. Sun Services. theo orangecat None sys Cluster Nodes === Node Name: Node ID: Enabled: privatehostname: reboot_on_path_failure: globalzoneshares: defaultpsetmin: quorum_vote: quorum_defaultvote: quorum_resv_key: vincent 1 yes clusternode1-priv disabled 1 1 1 1 0x49E30B7900000001 Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. show.Performing Post-Installation Verification Verifying Cluster Configuration Information Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved.192 6 4 2 480 pathcount vincent. and show -v sub-commands of the various cluster utilities.255. vincent:/# cluster show s ) ha Cluster === y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d anControl === ab Access r t Host g n non a b a Cluster Name: clusterid: installmode: heartbeat_timeout: heartbeat_quantum: private_netaddr: private_netmask: max_nodes: max_privatenets: num_zoneclusters: udp_session_timeout: global_fencing: Node List: Cluster name: Allowed hosts: Authentication Protocol: orangecat 0x49E30B79 disabled 10000 1000 172.16.

switch2@2 theo:ce3 switch2@2 Enabled Transport Switches === Transport Switch: State: Type: Port Names: Port State(1): Port State(2): switch1 Enabled switch 1 2 Enabled Enabled Transport Switch: switch2 4-62 Sun™ Cluster 3. Inc. ce3 y m ฺ om ฺ c ฺ Transport Cable: vincent:ce2. ce3 theo 2 yes clusternode2-priv disabled 1 1 1 1 0x49E30B7900000002 ce2.switch1@1 Endpoint1: vincent:ce2 csm Guide e t @ Endpoint2: switch1@1 n b e u State: ak SEnabled tud y ( s in thivincent:ce3. Sun Services.Performing Post-Installation Verification Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems.switch2@1 d Transport Cable: u l e a us Endpoint1: vincent:ce3 m a o k t Endpoint2: switch2@1 g nse n a State: Enabled ab lice b u le k b Transport Cable: theo:ce2.switch1@2 a a y fer l u Endpoint1: theo:ce2 s bd tran a Endpoint2: switch1@2 g n n State: Enabled no aba s ) ha Transport Cables === Transport Cable: Endpoint1: Endpoint2: State: theo:ce3. Oracle and/or its affiliatesฺ Transport Adapter List: Node Name: Node ID: Enabled: privatehostname: reboot_on_path_failure: globalzoneshares: defaultpsetmin: quorum_vote: quorum_defaultvote: quorum_resv_key: Transport Adapter List: ce2. Revision C a . All Rights Reserved.

All Rights Reserved. Oracle and/or its affiliatesฺ Performing Post-Installation Verification State: Type: Port Names: Port State(1): Port State(2): Enabled switch 1 2 Enabled Enabled Quorum Devices === Quorum Device Name: Enabled: Votes: Global Name: Type: Access Mode: Hosts (enabled): d2 yes 1 /dev/did/rdsk/d2s2 shared_disk scsi2 vincent. Revision C 4-63 a .Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. ak Stud y ( . sWe resource groups resources a and m u k e to chapters g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. theo s ) ha y m ฺ om ฺ c ฺ Device Groups === csm Guide e @ ent . b u . Inc. d // This continues on to show configuration u l e will cover these in later adisks. in thisof resource types.

Sun Services. you complete the following tasks: ● Task 1 – Verifying the Environment ● Task 2 – Updating Local Name Resolution ● Task 3 – Installing the Sun Cluster Packages ● Task 4 – Configuring a New Cluster – The All-Nodes-at-Once Method ● Task 5 – Configuring a New Cluster – The One-Node-at-a-Time Method s ) ha y m ฺ om ฺ c ฺ Task 7 – Configuring a Quorum Device (Three-Node eor Twom Cluster d s i c u Node Cluster With No Automatic Selection)e G t @ n b Task 8 – Verifying the Cluster Configuration de Status ku tuand a y n ( this S i d lu se a am to u k Task 1 – Verifying e ng the sEnvironment a n b e a lic b e u l Perform k the bfollowing steps on all nodes: a a r y fe that the /. the important part is that you include the path to the cluster utilities and man pages. edit the /etc/default/login file and comment out the CONSOLE=/dev/console line.profile file on each cluster node contains the ul 1. If you edit the file. more explicit path already. On all cluster nodes.2 Administration Copyright 2009 Sun Microsystems. verify that the changes are correct by logging out and in again as user root. Consult your instructor if you are unsure. PATH=$PATH:/usr/cluster/bin MANPATH=$MANPATH:/usr/cluster/man:/usr/share/man export PATH MANPATH 4-64 2. All Rights Reserved. Inc. 3.Exercise: Installing the Sun Cluster Server Software Exercise: Installing the Sun Cluster Server Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. nsVerify d b a tra following environment variables: g n n no aba ● Task 6 – Verifying an Automatically Selected Quorum Device (TwoNode Cluster) ● ● Note – The PATH and MANPATH need not be identical to those below. You may have a better. Oracle and/or its affiliatesฺ In this exercise. as highlighed. Sun™ Cluster 3. Revision C a .

In either case. if it is not already set: # env|grep DISPLAY # DISPLAY=display-name-or-IP:display-# # export DISPLAY Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. 1. set and export the DISPLAY. s ) ha y m ฺ Task 3 – Installing the Sun Cluster Packages om ฺ c ฺ m uide cssave Perform the following steps on all nodes. feOn your administrative workstation or display. Make sure the IP addresses of your administrative workstation and of all your nodes are in each /etc/hosts file. figure out the l u d b trans number of your display. or do ka installation tsome g e graphical installations on nodes and non-graphical installations on n ens a b other a nodes tolicompare the two.0: (# or $) echo $DISPLAY 2. c b e u l k rab a y 1. All Rights Reserved. Inc. Oracle and/or its affiliatesฺ Perform the following steps to update local name resolution.0.Exercise: Installing the Sun Cluster Server Software Task 2 – Updating Local Name Resolution Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. the display number is the number between the colon and the . On your cluster node. Youe will Gtime if you t @ n b perform the installation simultaneouslyuon all nodes. Note – In the Remote Lab Environment./installer -nodisplay).0 or :9. On the administrative workstation or display. k tude a y n ( this S i d luhave you Note – These instructions e do a graphical installation. Edit the /etc/hosts file on the administrative workstation and all cluster nodes. Sun Services. the machine named vnchost may be serving as the shared administrative station for all the students. Revision C 4-65 a . The output might look ang non ab similar to unix:6. Feel free to a s m u try the non-graphical o (. each student or a group might have a different display number. In the RLDC environment. enable remote Xwindows display from the nodes: (# or $) /usr/openwin/bin/xhost +nodename1 (# or $) /usr/openwin/bin/xhost +nodename2 (# or $) /usr/openwin/bin/xhost +nodename3 3.

or feel free to select it (it adds significant time to the m ฺ installation. b. Inc.cthe installer is om ฺ ฺ e pushy. say g nse ban licthat e you want to continue with the installation. s d. Run the installer: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. You will notice that Java DB is also automatically selected. 3. Sun Services. make the following choices: a. This is #5 only on the non-graphical installer. As the installation proceeds. Choose Sun Cluster 3. @ b den u k f. 2. From the Main Menu choose Option 1. Choose Configure u Later choose “Configure di (ifeyou hiaccidentally t l a Now”. skip this task and complete Task 5 instead.2 (not Sun Cluster Agents or Sun Cluster Geographic Edition). Perform the following steps to configure the entire cluster using the allnodes-at-once method: 4-66 1. Accept the license agreement. This is the node you will drive from. Oracle and/or its affiliatesฺ # cd sc32_software_location/Solaris_sparc # . Confirm upgrade of the shared components. m d s i ec t Gu e./installer a. amplease o k t configuration. say “no” when asked if you want to install the full set of products. On the non-graphical installer only. Verify that system requirements tu ya areSmet. h. From the Install Menu choose Option 1. Feel free to skip multilingual support if English-only is fine )forha y you. Revision C a . c. Sun™ Cluster 3. you will be asked if you want to enable remote usYes). ( n s g. Create a new cluster or add a cluster node.2 Administration Copyright 2009 Sun Microsystems. aConfirm b u le k b a a l y 4 –sConfiguring er f u d Task a New Cluster – The All-Nodesb tran a Method onang nat-Once ab If you prefer to use the one-node-at-a-time method. Start the scinstall utility (/usr/cluster/bin/scinstall). b. Log in as user root on the node that you want to have assigned the highest node ID in the cluster. and you cannot deselect it as long as Sun Cluster is selected. All Rights Reserved.Exercise: Installing the Sun Cluster Server Software 4.) Be persistent if you want to skip it. Create a new cluster.

All Rights Reserved. s ) ha ab y m ฺ omnodeฺ reboots. al cluster. e. answer Yes. Select the adapters that will be used for the cluster transport. h. Perform the following steps to configure the first node in your cluster (the one that will be assigned node ID 1). c 4. Revision C 4-67 a .will use link layer n(18) ip: joining multicasts banfailed e c a li b broadcasts for umulticast e l k a erab y l u nsf d b a tra g n n a no Task 5 – Configuring a New Cluster – The One-Nodeat-a-Time Method If you chose to install your cluster in “Task 4 – Configuring a New Cluster – The All-Nodes-at-Once Method” do not do this task. Choose Option 1 from the Main Menu. Oracle and/or its affiliatesฺ Exercise: Installing the Sun Cluster Server Software c. d. Accept the offer to use the lofi method for the global devices file system for each node as scinstall detects that there is no dedicated file system. ku tude a /usr/cluster/bin/scdidadm: Cannot open /etc/cluster/ccr/did_instances. Inc. From the Typical or Custom Menu choose Option 1. They will be rebooted and assigned Node ID’s in reverse order to what you type. 1. You will observe the following error messages asฺeach small normal. y in ( tadditional is S error message every time a 5. They care u e G t @ n b /usr/cluster/bin/scdidadm: Could not load DID instance list. Furnish your assigned cluster name. g. Create a new cluster or add a cluster node. Sun Services. You must wait for this node to complete and reboot into the cluster before configuring other nodes. f. Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. You will observe theufollowing d h e is also normal.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ide just the very first time. 2. As the installation proceeds. into the cluster. do not disable the automatic quorum selection (answer No). make the following choices: a. Typical. Start the scinstall utility (/usr/cluster/bin/scinstall). Enter the names of the other nodes in your cluster. s node boots into the This m u ka e to g s on clprivnet0 . For a two-node cluster. If you are asked (with a tagged VLAN-capable adapter) if it is a dedicated cluster transport adapter.

e. You willaobserve nsfollowing error messages as the node reboots. Verify the list of node names. d. you could perform these steps on all the other nodes simultaneously. This is also normal. Inc. 4-68 Choose Option 1 from the Main Menu. You will observe the following additional error message every time a n a n node boots into the cluster.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Reply yes to the automatic reboot question. Revision C a . As the installation proceeds. Furnish your assigned cluster name. For a two-node cluster. c. Select the transport adapters. do one node at a time. 2. If you want this to be predictable. as h i. g. Choose Option 2 from the Install Menu. From the Type of Installation Menu. b the very e u l k /usr/cluster/bin/scdidadm: a erab Could not load DID instance list. k e to g n ethe 3. Sun™ Cluster 3. All Rights Reserved.2 Administration Copyright 2009 Sun Microsystems. Typical. Sun Services. ฺco eฺ m cs Guid k. choose Option 1. answer Yes. They are all normal. make the following choices: a. ab ip: joining multicasts failed (18) on clprivnet0 . Allow the cluster check to run. ) ฺmy j. although then it will be hard to predict which node gets which node ID. f.will use link layer broadcasts for multicast Perform the following steps on each additional node in the cluster. d b a a r g on-t 4. h. Create just the first node of a new cluster on this machine. If you are asked (with a tagged VLAN-capable adapter) if it is a dedicated cluster transport adapter. 1. Furnish the name of the other nodes to be added later. Oracle and/or its affiliatesฺ Exercise: Installing the Sun Cluster Server Software b. Agree to use the lofi method for the global device file system. e @ ent b u l. into the cluster. If you have more than two nodes. do not disable the automaticm quorum selection (answer No). a y Accept them if they seem (appropriate. Start the scinstall utility on the new node. Examine the scinstall command options k tud for correctness. y l u nsf /usr/cluster/bin/scdidadm: Cannot open /etc/cluster/ccr/did_instances. b c a just first li time. n this S i d You must wait for luthis node a seto complete rebooting to proceed to m u the second a node.

two-node cluster: 1. Use auto-discovery for the transport adapters. In a two node cluster. Revision C 4-69 a . From the Type of Installation Menu. c. Reply yes to the automatic reboot question.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. g. All Rights Reserved. s m u ka e to g an cens b a li an Automatically Selected Quorum Task u 6b– Verifying e l k a e(Two-Node ab Device Cluster) r y l f u s bd tran a . Typical. Provide the name of the cluster that you want to join. Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. choose Option 1.Perform the following steps to verify automatic quorum selection on a ang non i. 2. Type cluster show -t global on the first node (the sponsoring node) if you have forgotten the name of the cluster. On either node. Inc. d. e. Oracle and/or its affiliatesฺ Exercise: Installing the Sun Cluster Server Software b. Sun Services. Provide the name of a sponsoring node. Add this machine as a node in an existing cluster. You will see the same normal boot error messages omonฺthe c ฺ additional nodes as on the first node. csm Guide e @ ent b u k ontthe Note – You see interconnect-related udexisting nodes beginning aerrors y ( S when a new node adds itselfin to the cluster is until the new node completes d h t u the first portion of its reboot e al operation. assuming you have allowed automatic quorum selection. Agree to use the lofi method for the global device file system. f. Examine and approve the scinstall command-line options. s ) ha ab y m ฺ j. Verify that your quorum device was chosen and that you have the expected number of quorum votes. Choose Option 3 from the Install Menu. type the clquorum status command. h. wait after the second node boots until you see console messages (on all nodes) indicating that the quorum device has been selected.

Sun™ Cluster 3. y fer l u s bd tran clquorum add d12 a May 3 22:29:13 vincent cl_runtime: NOTICE: CMM: ang non ab Cluster members: vincent theo apricot. 4. Revision C a . Examine the ak (target t y ( S standard logical path to make sure the DIDisdevice you select is a disk in a inmore d h one node. In a three node-cluster. The first few DID might n b e u disks. 2. such as the boot disk and a CD-ROM ud6). you should add a second quorum device.Exercise: Installing the Sun Cluster Server Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Reply yes to the reset installmode question (two-node cluster only). You k a a should see output similar to the following. Inc. You should see a Cluster initialization is complete message. On Node 1. wait after the third node boots until you see console messages (on all nodes) indicating that the quorum votes are being reset.2 Administration Copyright 2009 Sun Microsystems. May 3 22:29:13 proto192 cl_runtime: NOTICE: CMM: node reconfiguration #4 completed. d6. s ) ha y m ฺ Quorum disks: _______ (d4. On Node n 1. If you have three nodes in a Pair +N configuration. and record the DID devices you intend to configure as quorum disks. Navigate to the section for e s a n adding ba newlicquorum e device and supply the name of the first DID a b udevice b(global le device) that you selected in the previous step. All Rights Reserved. 5. tthan storage array and is connected to u l e a am to us k 3.g type the clsetup command. For example. 4-70 Type q to quit the clsetup utility. 6. Sun Services. you would want two quorum devices. type the cldevice list -v command. Oracle and/or its affiliatesฺ Task 7 – Configuring a Quorum Device (Three-Node Cluster or Two-Node Cluster With No Automatic Selection) Perform the following steps for quorum selection on a three-node cluster: 1. if you had three nodes in a Pair + 1 configuration. and so on) om ฺ c ฺ csm Guide e t be local @devices Note – Pay careful attention.

On any node. Inc.Exercise: Installing the Sun Cluster Server Software Task 8 – Verifying the Cluster Configuration and Status Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. type the cldevice list -v command. m cs Guidand quorum The cluster status. You should see the correct node and disk quorum votes. You should see active redundant interconnect paths between all pairs of nodes. node names. 2. On any node. type the clinterconnect status command. Sun Services. Oracle and/or its affiliatesฺ Perform the following steps on all nodes to verify the completed cluster installation: 1. type the clquorum status command. ฺm m ฺco eฺ 4. On any node. type the cluster show command. @ ent b device information should be complete. Revision C 4-71 a . u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Installing and Configuring the Sun Cluster Software Framework Copyright 2009 Sun Microsystems. 3. On any node. s Each shared (dual-ported) DID device should show a logicalypath ) ha from each cluster node. transporte configuration. All Rights Reserved.

All Rights Reserved. Oracle and/or its affiliatesฺ Exercise Summary ? ! Discussion – Take a few minutes to discuss what experiences. issues.Exercise Summary Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. Revision C a .2 Administration Copyright 2009 Sun Microsystems. Sun Services. ● Experiences ● Interpretations ● Conclusions ● Applications s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 4-72 Sun™ Cluster 3. or discoveries you had during the lab exercises.

Oracle and/or its affiliatesฺ Module 5 Performing Basic Cluster Administration Objectives s ) ha ab y m ฺ om ฺ Identify the cluster daemons c ฺ csm Guide Use cluster commands e @ ent b View and administer cluster global properties u ak Stud y ( View and administer quorum in this d u View and administer e and settings al diskuspaths m o ka e tinterconnect View and administer components g n s ba en command and the Sun Cluster Manager Use the clsetup c a i l b u le basic cluster startup and shutdown operations. including k b Perform a a r l y sfebooting nodes in non-cluster mode and placing nodes in u d n b maintenance state a tra g n n a Modify the private network settings from non-cluster mode no Upon completion of this module. Revision C a . you should be able to: ● ● ● ● ● ● ● ● ● 5-1 Copyright 2009 Sun Microsystems. Inc. All Rights Reserved. Sun Services.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

Sun Services. Oracle and/or its affiliatesฺ Relevance ? ! Discussion – The following questions are relevant to understanding the content of this module: ● What are various ways about getting more information about various Sun Cluster commands themselves? ● Why are you sometimes required to add a new quorum device before deleting an old one? ● What are some of the advantages of clsetup and Sun Cluster Manager? y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha ● 5-2 What is the value of being able to assign authorization for non-root users to do some cluster management? Sun™ Cluster 3. All Rights Reserved.2 Administration Copyright 2009 Sun Microsystems. Revision C a . Inc.Relevance Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

Sun Services. Inc.Additional Resources Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 5-3 a . Oracle and/or its affiliatesฺ Additional Resources Additional resources – The following references provide additional information on the topics described in this module: ● Sun Cluster System Administration Guide for Solaris OS.2 11/09) ● Sun Cluster Quorum Server User Guide. part number 820-7458 (for Sun Cluster 3. All Rights Reserved. part number 819-5360. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Performing Basic Cluster Administration Copyright 2009 Sun Microsystems.

regardless of which version of Solaris OS you are running. SMF itself can restart some daemons. A special cluster specific daemon monitor. All Rights Reserved. several cluster daemons are added to the traditional Solaris OS. the Solaris 10 OS manages daemons a bit differently.Identifying Cluster Daemons Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.fed root 2821 1 0 03:30:39 ? 0:00 /usr/cluster/lib/sc/scdpmd root 2845 1 0 03:30:40 ? 0:00 /usr/cluster/bin/cl_pnmd root 3057 1 0 03:30:41 ? 0:00 /usr/cluster/lib/sc/rgmd root 2805 1 0 03:30:39 ? 0:00 /usr/cluster/lib/sc/cl_ccrad 5-4 Sun™ Cluster 3. ● Solaris 9 OS – Daemons are launched by traditional Solaris OS boot scripts (and are thus guaranteed to be running by the time you get the console login prompt after a boot).pmfd. Inc. is required for restarting some daemons. rcp.cluster root 2836 1 0 03:30:39 ? 0:00 /usr/cluster/lib/sc/cl_eventd root 2824 1 0 03:30:39 ? 0:00 /usr/cluster/lib/sc/scqdmd root 2904 1 0 03:30:40 ? 0:00 /usr/cluster/lib/sc/rpc.2 Administration Copyright 2009 Sun Microsystems. However. root 1585 1 0 03:30:28 ? 0:00 /usr/cluster/lib/sc/ifconfig_proxy_serverd root 1586 1 0 03:30:28 ? 0:00 /usr/cluster/lib/sc/rtreg_proxy_serverd root 3072 1566 0 03:30:41 ? 0:01 /usr/cluster/lib/sc/rgmd -z global root 1956 1 0 03:30:34 ? 0:00 /usr/lib/inet/xntpd -c /etc/inet/ntp. y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak running The following is taken from a clusterynode tudSolaris 10 OS: ( S in this d u al use # ps -ef|grep clust m ka? e0:01 to cluster root 4 0 0 03:27:08 g an ce?ns 0:00 /usr/cluster/lib/sc/cl_execd root 1569 1568 0 03:30:27 b a li ? 0:00 /usr/cluster/lib/sc/clexecd root 1570 1 b 0 03:30:27 e u l k b root 1568 ? 0:00 /usr/cluster/lib/sc/cl_execd a1 0era03:30:27 y l f root 1546 1 s 0 03:30:27 ? 0:00 /usr/cluster/lib/sc/failfastd n bdu 1570 a root a1571 0 03:30:27 ? 0:00 /usr/cluster/lib/sc/clexecd r t g n n root 1 0 03:29:37 ? 0:00 /usr/cluster/lib/sc/qd_userd a 1513 no abroot 1566 1 0 03:30:27 ? 0:00 /usr/cluster/lib/sc/pmmd s ) ha ● Solaris 10 OS – Daemons are launched by the Solaris 10 OS Service Management Facility (SMF). behind the scenes. Sun Services. Revision C a .Therefore. Oracle and/or its affiliatesฺ Identifying Cluster Daemons When a cluster node is fully booted into a cluster. None of these daemons require any manual maintenance.conf. at boot time you might see a console login prompt before many of these daemons are launched.

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u clexecd – This is used by cluster threads ud to execute userland akkernel t y ( S commands (such as the irun_reserve n this and dofsck commands). The failfast daemon allows the kernel to panic if certain essential daemons have failed. ● ● cluster events. Oracle and/or its affiliatesฺ Identifying Cluster Daemons root 2852 1 0 03:30:40 ? 0:00 /usr/cluster/lib/sc/rpc. The daemon is automatically respawned if it is killed. you must restart it manually. The daemon is automatically respawned if it is killed. Sun Services. ● scqdmd – This daemon monitors the health of the quorum device. All Rights Reserved. The failfast driver panics the kernel if this daemon is killed and not restarted in 30 seconds. which manages the state of all cluster-unaware applications. If you kill this daemon. in 30 k b a a r l y sfecl_eventd u – This daemon registers and forwards cluster events d b tran (such as nodes a entering and leaving the cluster). ● qd_userd – This daemon serves as a proxy whenever any quorum device activity requires execution of some command in userland (for example. am to u k Thisa daemon ng eregisters se with failfastd so that the failfast device n b a willlicpanic the kernel if this daemon is killed and not restarted driver b e u lseconds. a NAS quorum device).pmfd root 2876 1 0 03:30:40 ? 0:00 /usr/cluster/lib/sc/scprivipd root 2812 1 0 03:30:39 ? 0:00 /usr/cluster/lib/sc/sc_zonesd root 2882 1 0 03:30:40 ? 0:00 /usr/cluster/lib/sc/cznetd root 2982 1 0 03:30:40 ? 0:00 /usr/cluster/lib/sc/pnm_mod_serverd root 2922 1 0 03:30:40 ? 0:00 /usr/cluster/lib/sc/cl_eventlogd ● cluster – This is a system process (created by the kernel) to encapsulate the kernel threads that make up the core kernel range of operations. There is no way to kill this process (even with a KILL signal) because it is always in the kernel. There is also a g onn protocol whereby user applications can register themselves to receive a n ab ● failfastd – This daemon is the failfast proxy server. Revision C 5-5 a . It is d also used to run cluster commands remotely (like the cluster u l e a s shutdown command). Inc. Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. ● rgmd – This is the resource group manager. Other daemons that require services of the failfast device driver register with failfastd.

lu e a s uif it is stopped. ● scprivipd – This daemon provisions IP addresses on the clprivnet0 interface. ● rpc. y m ฺ om ฺ c ฺ sm uide c e G It is automatically restarted if it is stopped. Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.pmfd – This is the process monitoring facility. on behalf of zones. They are automatically restarted if stopped. ● pnm_mod_serverd. and for most application daemons and application fault monitors (in Solaris 9 and 10 OS) The failfast driver panics the kernel if this daemon is stopped and not restarted in 30 seconds. The failfast driver panics the kernel if this daemon is stopped and not restarted in 30 seconds. e u l k b a ya Itfeis rautomatically restarted if it is stopped. ● sc_zonesd – This daemon monitors the state of Solaris 10 non-global zones so that applications designed to failover between zones can react appropriately to zone booting and failure. Sun Services.Identifying Cluster Daemons ● rpc. am It is automatically restarted o k t n–gThisedaemon se provides access from userland a n cl_ccrad b c a liapplications b management to the CCR. Sthere is no published file. so that g n non they can be reported in the output of the cldev status command. ● s ) ha cl_pnmd – This is the public network management daemon. which handles requests from rgmd to spawn methods for specific data services. which manages network status information received from the local IPMP daemon running on each node and facilitates application failovers caused by complete public network failures on nodes. It is automatically restarted if it is stopped. ifconfig_proxy_serverd – These daemons run in the global zone to provide required network services to zones in a zone cluster.2 Administration Copyright 2009 Sun Microsystems.fed – This is the fork-and-exec daemon. Revision C a . a b a ● ● ● It is automatically restarted if it is stopped. l u s d ab -tran scdpmd – This daemon monitors the status of disk paths. All Rights Reserved. 5-6 Sun™ Cluster 3. Inc. t @ n b u events de into a binary log kcluster cl_eventlogd – This daemon logs u a t y ( course. It is used as a general mechanism to initiate restarts and failure action scripts for some cluster framework daemons (in Solaris 9 OS). At the time of writing ifor n this s i d h t interface to this log. Oracle and/or its affiliatesฺ The failfast driver panics the kernel if this daemon is killed and not restarted in 30 seconds.

All Rights Reserved. and monitoring for some essential cluster processes such as rgmd that need to be aware of process status on different nodes. The failfast driver panics the kernel if this daemon is stopped and not restarted in 30 seconds.Identifying Cluster Daemons ● rtreg_proxy_serverd – This daemon runs in the global zone to provide resource type registration services to zones of a zone cluster. Revision C 5-7 a . Inc. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ It is automatically restarted if stopped. ● pmmd – This daemon provides cross-cluster starting. Sun Services. Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. restarting.

settings. and deleting quorum devices (includes node quorum information) ● clinterconnect (clintr) – Configuration. Revision C a . All Rights Reserved. Some ns at thishave bdu trcommands a an abbreviated name (long and short names are hard a g n n no linked to the same executable): aba ● 5-8 ● clnode – Configuration. or administration that is a era y l f useful point in the course. Inc.2 command-line commands have an object-oriented nature. Oracle and/or its affiliatesฺ The Sun Cluster 3. akto show tuonly give an optional last argument y ( S in this d u al use m Commands Relating ka toeBasic to Cluster Administration g an cens b a li The following commands are presented in more detail in this module b e u l k b because they relate to configuration. status. you give the name of the object as the last item on the command line. ● Every command requires a sub-command that indicates what it is you actually want to do. Every command has a full name. status. adding. and tape) ● cluster: ● Administering cluster global settings ● Showing configuration and status of everything. settings. or you give a + to indicate a wildcard. and settings for individual devices (disk. ● Every command has a common style of syntax whereby: ● When using sub-commands to operate on (or delete) a specific object. and settings for nodes ● clquorum (clq) – Configuration. s ) ha y m ฺ om ฺ c ฺ csm Guiordestatus. d a specific object. CDROM.Using Cluster Commands Using Cluster Commands Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems. That is: ● The name of the command is related to the cluster object you are trying to display or administer. and deleting private networks ● cldevice (cldev) – Configuration. The wildcard might still be limited by other command-line arguments (specific group or subtypes. status. adding. Sun Services. e When using subcommands to display configuration @ ent but you can the default is to show all objects u (ofbthat category). status. can be limited by other arguments to certain types or groups of information Sun™ Cluster 3. status. if appropriate).

clquorum <subcommand> [<options>] [+ | <devicename> .. adding. aba clquorum: ● clreslogicalhostname (clrslh) and clressharedaddress (clrssa) – Configuration. status.Using Cluster Commands Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. the usage k b If a you a r l y sfemessage always lists the possible sub-commands: u d n b a tra #g clquorum n n no (C961689) Not enough arguments. Inc. All Rights Reserved. settings. Sun Services. they are also g e While all the cluster s ban liceinnthat: self-documenting a b u lerun a command without any sub-command. adding. adding and deleting application resource groups ● clresource (clrs) – Configuration. Oracle and/or its affiliatesฺ Additional Commands The following commands relate to administration of device groups and cluster application resources. settings. No additional information is presented about them in this module: ● cldevicegroup (cldg)– Configuration. Revision C 5-9 a . adding and deleting device groups (including VxVM and Solaris Volume Manager device groups) ● clresourcegroup (clrg) – Configuration.] clquorum [<subcommand>] -? | --help clquorum -V | --version Manage cluster quorum SUBCOMMANDS: add disable enable export list remove Add a quorum device to the cluster configuration Put quorum devices into maintenance state Take quorum devices out of maintenance state Export quorum configuration List quorum devices Remove a quorum device from the cluster configuration Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. status. status. settings. which are the topics of Modules 6 through 10.. status. and deleting IP resources in application resource groups (these commands simplify tasks that can also be accomplished with clresource) ● clquorum: Usage: (C101856) Usage error. settings. and deleting individual resources in application resource groups s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d Cluster Command Self-Documentation u al use m ka commands to have excellent man pages.

you might get the wrong man page because there is a man page for a completely unrelated SQL command called cluster. Inc. clquorum: (C101856) Usage error. Sun Services. To fix that. the usage message gives you more information about the particular sub-command. -v Verbose output. you have to go to the man pages): Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems. Usage: clquorum add [<options>] <devicename> clquorum add -a [-v] clquorum add -i <configfile> [<options>] + | <devicename> . 5-10 Sun™ Cluster 3. e ahelp s -? Display this m u ka e to g -a Automatically an cens add a Shared Disk quorum device for 2b a li node cluster. a tra g n n no aba -p <name>=<value> Specify the properties... or you can type man -s1cl cluster. All Rights Reserved. Revision C a . you can put /usr/cluster/man as the first directory in your MANPATH variable. It does not give you all the information about the names of properties that you might need to set (for that. -t <type> Specify the device type. Note – If you perform the man cluster command on the Solaris 10 OS. Oracle and/or its affiliatesฺ ● # clquorum add clquorum: (C456543) You must specify the name of the quorum device to add.Using Cluster Commands reset show status Reset the quorum configuration Show quorum devices and their properties Display the status of the cluster quorum If you run a sub-command that requires arguments. s ) ha y m ฺ om ฺ c ฺ csm Guide e Add a quorum device to the cluster configuration @ ent b u ak Stud y ( OPTIONS: in this d u l text. b e u l k a erab y l f | <clconfiguration>} {u -i s d n b Specify XML configuration as input.

theo s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b the u Renaming le Cluster k b a a l y sfer u d ab -tranYou can rename the cluster using the cluster rename command.0. Inc. Sun Services. shows only the cluster global properties: # cluster show -t global Cluster === Cluster Name: clusterid: installmode: heartbeat_timeout: heartbeat_quantum: private_netaddr: private_netmask: max_nodes: max_privatenets: num_zoneclusters: udp_session_timeout: global_fencing: Node List: orangecat 0x49D36C52 disabled 10000 1000 172.0 255.240.16. Revision C 5-11 a . .16. # cluster rename -c orangecat blackcat 0x49D36C52 disabled 10000 1000 172.0 Performing Basic Cluster Administration Copyright 2009 Sun Microsystems.255.0. Oracle and/or its affiliatesฺ Viewing and Administering Cluster Global Properties The cluster show -t global command. The g cluster name is not particularly important and is not required as an n non a b argument in any other commands: a # cluster rename -c blackcat # cluster show -t global Cluster === Cluster Name: clusterid: installmode: heartbeat_timeout: heartbeat_quantum: private_netaddr: .Viewing and Administering Cluster Global Properties Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.0 64 10 12 480 pathcount vincent. All Rights Reserved. as demonstrated.

0. although it is possible to make them bigger if your nodes are very far apart. This will be covered later in the module. All Rights Reserved.60000. sm uide c e # cluster set -p heartbeat_quantum=800 G t @ n b # cluster set -p heartbeat_timeout=4000 ku tude a # cluster show -t global y n ( this S i d lu se Cluster === a am to u k Cluster Name: orangecat ng ense a b clusterid: 0x49D36C52 c a li b e u l installmode: k disabled b a a r y heartbeat_timeout: 4000 ul nsfe d heartbeat_quantum: 800 b a a r t g private_netaddr: 172.16.2 Administration Copyright 2009 Sun Microsystems. You usually do not change these values. Oracle and/or its affiliatesฺ Setting Other Cluster Properties The heartbeat_quantum controls the timing of cluster heartbeats on the private network (in milliseconds). and all nodes are booted in non-cluster mode.240. Revision C a . theo Note – Modifying the private_netaddr and private_netmask properties is a special case. Inc. 5-12 Sun™ Cluster 3. Sun Services. in that it is done only when the entire cluster is down. if for some reason you are unsatisfied with the 10-second timeout. s ) ha The cluster will enforce that heartbeat_timeout is at least five times as big as heartbeat_quantum.Viewing and Administering Cluster Global Properties Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. The heartbeat_timeout describes the number of milliseconds of missing heartbeat required by a node to declare a single interconnect dead or to declare the other node(s) dead and start reconfiguration.0 ab private_netmask: max_nodes: max_privatenets: num_zoneclusters: udp_session_timeout: global_fencing: Node List: 64 10 12 480 pathcount vincent. as illustrated in the following example: y m ฺ om ฺ # cluster set -p heartbeat_timeout=4000 c ฺ cluster: heartbeat timeout 4000 out of range 5000 .255. or smaller.0 n n o a n 255.

All Rights Reserved.Viewing and Administering Nodes Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. there is only a limited amount that you can actually change with the clnode command. Revision C 5-13 a . Viewing Node Status and Configuration The status and show sub-commands show. Oracle and/or its affiliatesฺ Viewing and Administering Nodes The clnode command can be used to show status and configuration of nodes. You could also show a single node by giving its name as the last commandline argument: s ) ha y m ฺ om ฺ c ฺ Cluster Nodes === csm Guide e @ ent b u --. Inc. ce3 theo 2 yes clusternode2-priv Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. all nodes. by default. While it shows a variety of data for each node in the cluster.Node Status --ak Stud y ( in this Status d Node Name u al use -------------m a o k t vincent Online g e n s theo Online ba licen a b u le k b # clnode show a a l y sfer u d an=== Cluster ab Nodes r t g n non a b vincent a Node Name: # clnode status Node ID: Enabled: privatehostname: reboot_on_path_failure: globalzoneshares: defaultpsetmin: quorum_vote: quorum_defaultvote: quorum_resv_key: Transport Adapter List: Node Name: Node ID: Enabled: privatehostname: 1 yes clusternode1-priv disabled 1 1 1 1 0x49D36C5200000001 ce2.

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. for example). Inc. g e n s ba licen # clnode set -p privatehostname=vinny-priv vincent a b e # clnode showkvincent u l a erab y l f uNodesn=== s d Cluster b a tra g n n o a Node nName: vincent ab Node ID: Enabled: privatehostname: reboot_on_path_failure: globalzoneshares: defaultpsetmin: quorum_vote: quorum_defaultvote: quorum_resv_key: Transport Adapter List: 1 yes vinny-priv disabled 1 1 1 1 0x49D36C5200000001 ce2. s ) ha y m ฺ om ฺ c ฺ The reboot_on_path_failure command is described sm later idinethe c u e G module.4. Oracle and/or its affiliatesฺ Viewing and Administering Nodes reboot_on_path_failure: globalzoneshares: defaultpsetmin: quorum_vote: quorum_defaultvote: quorum_resv_key: Transport Adapter List: disabled 1 1 1 1 0x49D36C5200000002 ce2.16. All Rights Reserved. ce3 Modifying Node Information Most of the information shown by clnode show cannot be modified by the clnode command.2 Administration Copyright 2009 Sun Microsystems. This name n address iIP automatically resolves to u the associated with the node’s d h t l e a clprivnet0 adapter. Sun Services. is the single private IP address whose traffic is usall amThistacross o k automatically distributed physical private networks. Some can be modified by other commands (clinterconnect for adding and deleting transport adapters. Revision C a .1 vinny-priv # clnode set -p privatehostname=clusternode1-priv vincent 5-14 Sun™ Cluster 3. t @ n b ku tude a y You could set the privatehostname ( toiswhatever S you want. ce3 # getent hosts vinny-priv 172.

0. All Rights Reserved.2.05.2.12.0.2. 125511-02 126106-37 800149-01 SUNWscrtlh: 3.22.05.REV=2006. or on all nodes. 125511-02 126106-37 800149-01 SUNWsczu: 3. Sun Services.REV=2006. 126106-37 800149-01 SUNWscmasasen: 3.05.13.2.2.09.05.12.0.REV=2006.0. 125511-02 126106-37 800149-01 SUNWsczr: 3.REV=2006.58.12.REV=2006.26.05.REV=2006.0.0.28 SUNWscdev: 3. 128556-04 800141-01 SUNWscsal: 3.0.22. 800149-01 SUNWmdmr: 3.22.2. 126106-37 800149-01 SUNWscmasar: 3.09.0.REV=2006.0.22. Inc.0.0.22.58.12.58.0.09.0.12.12. Revision C 5-15 a .2.05.22.2.12. 125508-08 126106-37 800149-01 SUNWscmautilr: 3. 125992-05 800149-01 SUNWscr: 3.12.13.22. You can use nscd -i hosts to clear this cache.0.05.2.58.REV=2006.22.REV=2006.22.0.REV=2006.58.12.58.45.REV=2006.05.05.05.2. 139921-02 800149-01 SUNWscspmr: 3.0.REV=2009.22.22.125511-02 126106-37800149-01 y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Performing Basic Cluster Administration Copyright 2009 Sun Microsystems.22.22.58. 125508-08 126106-37 800149-01 SUNWscderby: 3.12.12.58.Viewing and Administering Nodes Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.12.REV=2006.0.22.2.58.22.58.05.25.2. 126106-37 800149-01 SUNWscnmr: 3.25.REV=2006. 125511-02 126106-37 800149-01 SUNWscgds: 3.13.58.2.05.12.22.05.REV=2006.2.0.05.58.58. 126106-37 800149-01 SUNWscmasau: 3.REV=2006.0.58.12.58. Viewing Software Release Information on a Node You can use clnode show-rev -v to see installed cluster package release information on a node.05. 125992-05 800149-01 SUNWscscku: 3. 800149-01 SUNWscspmu: 3. 125511-02 126106-37 800149-01 SUNWscsckr: 3.2. it is because of the OS namecaching daemon (nscd). This is useful information to have when talking to technical support personnel. 125511-02 126106-37 800149-01 SUNWsccomzu: 3. 125511-02 126106-37 800149-01 SUNWsccomu: 3.22. 800149-01 SUNWmdmu: 3.12.58.12.2. 126106-37 800149-01 SUNWscsmf: 3.2u3 for Solaris 10 sparc SUNWscu: 3.22.05.2.0.2.REV=2006.12.2.12.0.2.2.58.05.05.05.2.22.12.22.0.05.2.0.12.58.28 SUNWscnmu: 3.58.REV=2006.REV=2006.0.0.05.22.58.22.REV=2006.12.58. 125511-02 126106-37 800149-01 SUNWscmautil: 3.2.58.REV=2006.2.REV=2006.REV=2006.REV=2006.REV=2009. Oracle and/or its affiliatesฺ Note – If it seems that the OS can resolve private hostnames that no longer exist (because you changed them).REV=2006.2.12.05.2.58.2.22.58.12.2.REV=2006.22.05.05.12.22.05.12.0.REV=2006.2.12. 800149-01 SUNWjfreechart: 3.58.58.05. 126106-37 800149-01 SUNWscvm: 3. 125514-05 800149-01 SUNWscmasa: 3.0. 126106-37 800149-01 SUNWscman: 3.22.REV=2006.22.REV=2006.0. 126106-37 800149-01 SUNWsctelemetry: 3.12. 126106-37 800149-01 SUNWscsam: 3.58.2.05. s ) ha # clnode show-rev -v Solaris Cluster 3.0.12.REV=2006.58.05.0.0.22.

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Viewing and Administering Nodes You can also directly examine the file /etc/cluster/release to get quick information about the release of the cluster software installed on a particular node: # cat /etc/cluster/release Sun Cluster 3. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 5-16 Sun™ Cluster 3. Revision C a .. All Rights Reserved. All Rights Reserved.2u3 for Solaris 10 sparc Copyright 2009 Sun Microsystems. Inc. Inc.

Quorum Votes by Device --Device Name ----------d2 Present ------1 Possible -------1 Status -----Online # clq show d2 Quorum Devices === Quorum Device Name: d2 Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Viewing and Administering Quorum The clquorum (clq) command is used to view configuration and status of quorum devices and node votes and to add and delete quorum devices. All Rights Reserved. s ) ha y m ฺ om ฺ c ฺ Cluster Quorum === csm Guide e @ ent b u --. Inc.Viewing and Administering Quorum Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.Quorum Votes Summary --ak Stud y ( in this d Needed Present Possible u al use -----------. Sun Services.m-------2 3 g ka 3 to n ense a b a lic b e u l k byraNode b ----. for example). You could reduce the amount of information by adding a type-restriction option (-t shared_disk. Viewing Quorum Status and Configuration The status. Revision C 5-17 a . or adding as the very last argument the name of a particular quorum device or node.Quorum Votes a y l e f du ransPresent bName Node Possible Status a t g n --------------------------n o a n b 1 1 Online a vincent # clq status theo 1 1 Online --. list. and show sub-options show the status and configuration of quorum devices and node-related quorum information.

y m ฺ om requires c ฺ A two-node cluster. 5-18 Sun™ Cluster 3. ฺ e m d s i you to perform repairs in that order (add and then cremove).Quorum Votes --l y by eDevice f u s d abName -tran Present g Device Possible Status n n o a n ------------------ab----------d5 1 1 Online Installing a Quorum Server (Outside the Cluster) When you install the packages SUNWscqsr and SUNWscqsu on a machine outside the cluster. All Rights Reserved. theo Adding and Removing (and Replacing) Quorum Devices s ) ha There is no specific command to replace or repair a quorum device. and clearing out data in quorum servers. Inc. uIf you have ethe G t more than two nodes you could actually perform operations in any @ n b e u order: ak Stud y ( in this # clq add d5 d u # clq remove d2 al use m # clq status -t shared_disk ka to g e ban licens Cluster Quorum === a b u le k b a a r --.2 Administration Copyright 2009 Sun Microsystems. a quorum server instance is configured automatically on the default quorum server port (9000). which absolutely requires a quorum device. You just add a new one and remove the old one. See appendix for more information about starting. This single instance can serve as the quorum device for as many clusters as you like.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. stopping. Revision C a . Sun Services. Oracle and/or its affiliatesฺ Viewing and Administering Quorum Enabled: Votes: Global Name: Type: Access Mode: Hosts (enabled): yes 1 /dev/did/rdsk/d2s2 shared_disk scsi2 vincent. examining data.

Sun Services. All Rights Reserved.Quorum l yVotessfebyr Node --u d abName-tran Present Node Possible Status g n n o a n --------------------------ab vincent theo 1 1 1 1 Online Online --. On the cluster side. Inc.Quorum Votes by Device --Device Name ----------qservydude Present ------1 Possible -------1 Status -----Online Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. qservydude).Viewing and Administering Quorum Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.Quorum Votes Summary --y ( in this d u Needed Present Possible al use m -----------ka -------to e 2 3 ng 3 ba licens a b u le k b a a --. # clq add -t quorum_server -p qshost=clustergw \ -p port=9000 qservydude # clq remove d5 s ) ha y m ฺ # clq status om ฺ c ฺ csm Guide e Cluster Quorum === @ ent b u ak Stud --. you need to specifically add a quorum server device to serve the cluster. you assign a random ascii device name to the quorum server device (in this example. Oracle and/or its affiliatesฺ Adding a Quorum Server Device to a Cluster On the cluster side. Revision C 5-19 a . This is likely to be your only quorum device (after you remove other quorum devices. if necessary) because it always has the number of votes equal to one fewer than the number of node votes.

G t @ n b delooks similar to the kucommand u a t y Registering a NAS device with the clnas n ( this S i following: d lu se a u mynetapp # clnas add -t netapp am -utoroot k Please enternpassword: g nse a b ce a i l b There command for Sun NAS devices as well.168. Inc. All Rights Reserved. Oracle and/or its affiliatesฺ Quorum Devices === Quorum Device Name: Enabled: Votes: Global Name: Type: Hosts (enabled): Quorum Server Host: Port: qservydude yes 1 qservydude quorum_server vincent. Recall that the iSCSI protocol is used on the NetApp filer NAS device only for the quorum mechanism.Viewing and Administering Quorum # clq show qservydude Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. although it uis a registration le k b a a is strictly a prerequisite only for specifying data fencing for the Sun NAS.1 9000 s ) ha y m ฺ om ฺ c ฺ Before you can use a Netapp NAS device as a quorum you must deregister sm i c u e the NAS device.101.2 Administration Copyright 2009 Sun Microsystems. y fer l u s n is no username and password for the Sun NAS device: bd trThere a a g onn a n # clnas add -t sun mysunnas ab Registering NAS Devices Adding NetApp NAS iSCSI LUN as a Quorum Device After the NetApp NAS device is registered. you can use the clsetup utility to establish one of the iSCSI LUNs on the device as a quorum. From the clsetup dialog the interaction looks similar to the following: What is the type of device you want to use? 1) Directly attached shared disk 2) Network Attached Storage (NAS) from Network Appliance 3) Quorum Server 5-20 Sun™ Cluster 3. theo 192. Revision C a . Sun Services.

which is given by the user and must be unique across all quorum devices. All Rights Reserved. and a LUN id. which defaults to 0 if not specified. s ) ha The NAS quorum device must be setup before configuring it with Sun Cluster. Sun NAS uses the standard Solaris 10 iscsiadm client to make Sun NAS iSCSI LUNs appear as if they were regular shared disk devices. Revision C 5-21 a . and installing the license and the Netapp binaries.Viewing and Administering Quorum q) Return to the quorum menu Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. see the Sun Cluster documentation. For more information on setting up Netapp NAS filer. Sun Services. Please refer to the clquorum(1M) man page and other Sun Cluster documentation for details. Unlike NetApp. Oracle and/or its affiliatesฺ Option: 2 >>> Add a Netapp NAS Quorum Device <<< A Netapp NAS device can be configured as a quorum device for Sun Cluster. creating the device. The NAS configuration data includes a device name. Inc. the filer name. In order for the node to get visibility to a Sun NAS iSCSI LUN you would need to administer it on each node: # iscsiadm modify discovery -s enable # iscsiadm list discovery Discovery: Static: enabled Send Targets: disabled Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. y m ฺ om ฺ c ฺ csm Guide e t Is it okay to continue (yes/no) [yes]? yes b@ n e u ak Stud y ( What name do you want to use for this in quorum is device? netappq d h t u l e mynetapp a[netapps]? s What is the name of the filer m u ka e to g ns [0]? 0 What is the LUN id filer baonn the e c a li b e u l k Is it okay a to eproceed ab with the update (yes/no) [yes]? yes r y l f s n bdu add a a r clquorum -t netapp_nas -p filer=mynetapp -p lun_id=0 netappq t g n n no aba Adding Sun NAS iSCSI LUN as a Quorum Device Sun NAS devices also support using an iSCSI LUN as a quorum device.

Revision C a . Oracle and/or its affiliatesฺ # iscsiadm add static-config iqn. k et g n a cens b a li NAS directories for failure fencing looks similar to Registering b thelespecific u k the a following: ab r y l e f ns add-dir -d /vol/vol_01_03 mynetapp bdu tr#aclnas a .2 Administration Copyright 2009 Sun Microsystems. from a single cluster node.# clnas add-dir -d /vol/vol_01_04 mynetapp ang non ab You can verify the configuration of the NAS device in the CCR using the clnas show command as in the following example: # clnas show Filers of type "netapp": Filer name: type: password: userid: directories: directories: 5-22 mynetapp netapp ******* root /vol/vol_01_03 /vol/vol_01_04 Sun™ Cluster 3. you could repopulate the DID device space and create a normal-looking shared disk device quorum: # cldev populate # cldev list -v # clq add d# s ) ha y m ฺ m Registering NAS Mounted Directories (for m Data ฺco eฺ d Fencing: both NetApp and Sun NAS) ecs Gui @ ent b u d atokregister tuon y Use the clnas add-dir subcommand the NAS device the ( S n s i i specific directories that are being used to thserve cluster data. The Sun ludis then e a Cluster client implementation able to perform data fencing on s m u a o these specific directories. All Rights Reserved. Sun Services. Inc.LUNName.Viewing and Administering Quorum iSNS: disabled Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.IPAddr_of_NASDevice # devfsadm -i iscsi At that point.

s ) ha y m ฺ om ฺ c ฺ csm Guide e Displaying Disk Paths @ ent b u ak Stud y ( cldev list -v is the best summary in thiofs node-to-disk paths. Disk paths are monitored by a daemon scdpmd ● Allow you to change disk monitoring settings: ● By default. . Sun Services. All Rights Reserved. ● We will see at least one reason you may like to turn off monitoring for non-shared disks. Inc. all paths are monitored. d17 theo:/dev/rdsk/c2t9d0 d18 vincent:/dev/rdsk/c2t10d0 d18 theo:/dev/rdsk/c2t10d0 d19 vincent:/dev/rdsk/c2t11d0 d19 theo:/dev/rdsk/c2t11d0 d20 theo:/dev/rdsk/c0t0d0 d21 theo:/dev/rdsk/c0t1d0 d22 theo:/dev/rdsk/c0t6d0 Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. //Omitted for brevity . and the d u corresponding DID device e al numbers: s m u k-va e to # cldev list g DID Device an cens Full Device Path b a ------------------------li b e u l k vincent:/dev/rdsk/c0t0d0 a d1 erab y l d2 vincent:/dev/rdsk/c0t1d0 f u ns d b vincent:/dev/rdsk/c0t6d0 a tra d3 g n d4 vincent:/dev/rdsk/c1t0d0 n o a n b d4 theo:/dev/rdsk/c1t0d0 a ● Allow you to change fencing properties d5 vincent:/dev/rdsk/c1t1d0 d5 theo:/dev/rdsk/c1t1d0 d6 vincent:/dev/rdsk/c1t2d0 . Oracle and/or its affiliatesฺ The cldevice (cldev) command can: ● Display disk path information (which nodes are connected to which disks) and the mapping between DID device names and c#t#d# ● Display disk path status information. Revision C 5-23 a .Viewing and Administering Disk Paths and Settings Viewing and Administering Disk Paths and Settings Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

Inc. you could yank out a disk or sever a disk path and the status might still be reported as Ok for a couple of minutes. The cldevice status command shows the status of disk paths as last recorded by the daemon. Disk path status changes are logged into /var/adm/messages with the syslogd LOG_INFO facility level. All failures are logged by using the LOG_ERR facility level. the daemon scdpmd probes disk paths periodically. Sun Services. until the next time the daemon probes the paths: s ) ha y m ฺ om ฺ c ฺ Device Instance Node sm Status ide c u e -----------------G-----t @ n b /dev/did/rdsk/d1 vincent u ude Ok k a t y ( S n /dev/did/rdsk/d10 Ok di theo his t u l e Ok a usvincent m a o k et g /dev/did/rdsk/d11 theo Ok n ba licens vincent Ok a b e u l k rab a y /dev/did/rdsk/d12 theo Ok l e f u s d vincent Ok ab -tran g n non /dev/did/rdsk/d13 a b theo Ok a # cldev status 5-24 vincent Ok /dev/did/rdsk/d14 theo vincent Ok Ok /dev/did/rdsk/d15 theo vincent Ok Ok /dev/did/rdsk/d16 theo vincent Ok Ok /dev/did/rdsk/d17 theo vincent Ok Ok /dev/did/rdsk/d18 theo Ok Sun™ Cluster 3.Viewing and Administering Disk Paths and Settings Displaying Disk Path Status Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ By default. All Rights Reserved. That is. Revision C a .2 Administration Copyright 2009 Sun Microsystems. once every few minutes.

Sun Services. You can also display only those paths that have a specific status. Inc. Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. all paths to all disks are monitored.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Viewing and Administering Disk Paths and Settings vincent Ok /dev/did/rdsk/d19 theo vincent Ok Ok /dev/did/rdsk/d2 vincent Ok /dev/did/rdsk/d20 theo Ok /dev/did/rdsk/d21 theo Ok /dev/did/rdsk/d4 theo vincent Ok Ok s ) ha y m ฺ om ฺ c ฺ /dev/did/rdsk/d6 theo csm GuiOkde e @ ent Ok vincent b u ak Stud y ( /dev/did/rdsk/d7 Ok in theo is d h t vincent Ok u l e a s am to u theo k /dev/did/rdsk/d8 Ok g nse n a vincent Ok ab lice b u le k b /dev/did/rdsk/d9 theo Ok a a y fer l u vincent Ok s bd tran a g on. All Rights Reserved.You can limit the output to a specific DID device by giving its name as an n a n ab /dev/did/rdsk/d5 theo vincent Ok Ok argument. Revision C 5-25 a . as in the following example: # cldev status -s Fail Device Instance --------------- Node ---- Status ------ Changing Disk Path Monitoring Settings By default.

and specifically the path from node theo to disk device d9: # cldev unmonitor d8 # cldev unmonitor -n theo d9 # cldev status -s Unmonitored Device Instance --------------/dev/did/rdsk/d8 Node ---theo vincent Status -----Unmonitored Unmonitored s ) ha y m ฺ om ฺ /dev/did/rdsk/d9 theo Unmonitored c ฺ sm uide c e G+ as the As is the convention for all cluster commands. Sun Services. with a single-path failure (to an entire array. n this S i d lu se a kam e to u Devices and Enabling Unmonitoring Allg Non-Shared reboot_on_path_failure ban licens a b u le k b a a l yIf yousfset erthe node property reboot_on_path_failure to enabled (the u d default an is disabled). It does not make sense to enable the reboot_on_path_failure if you are still monitoring non-shared (local) disks. you must unmonitor any local disks. so that at least any clustered applications running there can fail over to other nodes. ab This is a way to ensure HA even when there are multiple simultaneous disk path failures.2 Administration Copyright 2009 Sun Microsystems.@ you can use t n b wildcard. Oracle and/or its affiliatesฺ You can unmonitor specific paths. Typically. By default you affect the path from all connected nodes to the DID device that you mention. Inc. For example. the following commands unmonitor all paths to disk device d8. for example). Revision C a . regardless of their previous(state. then a node automatically reboots if all the ab -trmonitored g disk paths from that node are broken and there is another node n n o a n that still has working paths to at least one of the shared devices. You can limit your action to a path from a specific node by using the -n option. you might decide (by setting the property) that it is best to just have the node reboot. So. All Rights Reserved. or multi-pathed. everything would still be fine because your data would always be mirrored across multiple controllers. if you want to enable this feature. So cldev monitor + would turn de back on for all ku monitoring u a t y devices. or re-monitor them if you have previously unmonitored them. # cldev unmonitor d1 d2 d4 5-26 Sun™ Cluster 3.Viewing and Administering Disk Paths and Settings Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. But if a particular node loses access to all disk paths. or both. The whole point is to reboot if you lose contact with all multi-ported storage. but the actual feature will not reboot a node unless it detects failure on all monitored storage.

as demonstrated in the command example.Viewing and Administering Disk Paths and Settings Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. a us m a k e to # cldev show d7 g ban licens a b === DID Device Instances u le k b a a l yName:sfer u DID Device /dev/did/rdsk/d7 d n b a a r Device theo:/dev/rdsk/c1t3d0 -t Path: g Full n n o Full Device Path: vincent:/dev/rdsk/c1t3d0 a n ab Replication: default_fencing: none global Modifying Properties to use SCSI-3 Reservations for Disks With Two Paths The alternate value for the global_fencing property is prefer3. Sun Services. Revision C 5-27 a . disks with exactly two paths are fenced with SCSI-2 reservations (add SCSI-2 PGRE when used as a quorum device). is pathcount. Inc. Oracle and/or its affiliatesฺ # clnode set -p reboot_on_path_failure=enabled vincent # cldev unmonitor d20 d21 # clnode set -p reboot_on_path_failure=enabled theo Viewing Settings Related to SCSI-2 and SCSI-3 Disk Reservations A global setting controls what form of SCSI reservations. if any. are used with disks. which d hisfencing for that disk follows the u l e global property. The prefer3 global value implies that the cluster should try to use SCSI-3 fencing for every device that is still following the global policy. The default value. s ) ha y m ฺ om ฺ c ฺ # cluster show|grep global_fencing csm Guide e global_fencing: pathcount @ ent b u ak property. Nothing more is needed. which already implement the persistence needed for quorum devices. tud The default value for Each individual disk has its own(y fencing S n tthat imeans every disk is global. All Rights Reserved. When you change the global policy from pathcount to prefer3: Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. With this value. Disks with more than two paths are fenced with SCSI-3 reservations.

● If there is a quorum disk. You can see that SCSI2 behavior is left intact for an existing quorum disk. Inc. at the time the global policy is changed to prefer3. You would have to delete the quorum disk and add it back (or add a different quorum device) to get SCSI-3 behavior on the quorum disk. y m ฺ om ฺ c # cluster set -p global_fencing=prefer3 ฺ e m uidremains Warning: Device instance d4 is a quorum device . ● The cluster immediately begins to use SCSI-3 fencing for any nonquorum disk that seems to support SCSI-3.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. non-quorum disks. The cluster assumes that any device that responds appropriately to such a command can support SCSI-3. All Rights Reserved. currently configured as a SCSI-2 quorum. s ) ha The following command changes the global policy. Sun Services. There is no output or change in per-disk properties for other. t @ n b Updating shared devices on node 1 ku tude a y ( is S Updating shared devices on node 2 n i d # cldev show d4 lu se th a am to u k DID Device Instances === ng ense a b lic DID Device Name: b a /dev/did/rdsk/d4 e u l k b Full Device Path: vincent:/dev/rdsk/c1t0d0 a era y l f Full Device Path: theo:/dev/rdsk/c1t0d0 bdu trans Replication: none a g n n default_fencing: pathcount o a n b a # clq show d4 Quorum Devices === Quorum Device Name: Enabled: Votes: Global Name: Type: Access Mode: Hosts (enabled): 5-28 d4 yes 1 /dev/did/rdsk/d4s2 shared_disk scsi2 vincent.2 Administration Copyright 2009 Sun Microsystems.fencingcs protocol e G PATHCOUNT for the device. Revision C a . Oracle and/or its affiliatesฺ Viewing and Administering Disk Paths and Settings ● The cluster probes the current two-path devices by sending a sample SCSI-3 ioctl command. theo Sun™ Cluster 3. that quorum disk will not have its behavior changed to SCSI-3.

Oracle and/or its affiliatesฺ Viewing and Administering Disk Paths and Settings The per-disk policy for the existing quorum device has been set to pathcount so that it does not follow the global default of prefer3. theo Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. For other disks. you complete the following steps: 1. Add it back as a quorum device (at which point you might e m d s i c asGa uplace-holder). remove any other quorum device that is e acting t @ n b e u akthe procedure: The folowing example demonstrates tud y ( S in this d # clq add d5 u l e s # clq remove d4 ma u kadefault_fencing=global to # cldev setg-p d4 e n s a n Updating shared devices on node 1 ce devices on node 2 ab shared i l Updating b uclq add led4 k b # a a l y #sclq er remove d5 f u d ab -tran# clq show d4 g an non Quorum Devices === Quorum Device Name: Enabled: Votes: Global Name: Type: Access Mode: Hosts (enabled): d4 yes 1 /dev/did/rdsk/d4s2 shared_disk scsi3 vincent. Inc. the individual default_fencing property remains with the value global and the cluster immediately uses SCSI-3 the next time they need to be fenced. if this is a two-node cluster). ab as h ) 2. Revision C 5-29 a . m o c ฺ ฺ want to 3. Remove it as a quorum device (but you might need to add another quorum device first. Getting Quorum Device to Use SCSI-3 Policy To change the policy for an existing quorum device. All Rights Reserved. Sun Services. Change the per-disk default_fencing policy back to the value y m ฺ global.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

introduced in Sun Cluster 3.Viewing and Administering Disk Paths and Settings Eliminating SCSI Fencing for Particular Disk Devices Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 1/09 (Update 2). All Rights Reserved. Revision C a . These devices do not support SCSI fencing of any sort. Inc. allows the use of Serial Advanced Technology Attachment (SATA) devices as shared storage devices. Oracle and/or its affiliatesฺ You can turn off SCSI fencing for particular disks devices. This feature. It is not recommended in any way that you eliminate fencing for devices that support fencing.2 Administration Copyright 2009 Sun Microsystems. s ) ha The per-disk default_fencing property has values that specify that you do not want fencing for that disk: y m ฺ m existing nofencing (turns off fencing after scrubbing the diskcofoany ฺ ฺ e m reservation keys) d s i u ec anyt G nofencing-noscrub (turns off fencingbwithout scrubbing) @ n e u d k ya Softufencing for a particular ( The following example shows the elimination n di e this u disk device: l a us m a # cldev set -p default_fencing=nofencing d5 k e to g n s Updating shared devices on node 1 a cen bshared a Updating li devices on node 2 b e u l k a erab y l u nsf d b a tra g n n no aba ● ● 5-30 Sun™ Cluster 3. Sun Services.

you can use the new values of the clusterwide global_fencing property. a disk which is already a quorum device requires special manipulation: s ) ha ab y m ฺ om ฺ . Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Eliminating SCSI Fencing Globally It is likely that if none of your shared disk devices support fencing (all are SATA drives. listed previously. All Rights Reserved. As in the earlier examples of switching from scsi2 to scsi3 fencing.Viewing and Administering Disk Paths and Settings Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. you are allowed to add a disk on which SCSI fencing has been eliminated as a quorum device. The values are the same as those for the per-disk fencing property.fencing t @ n b protocol remains SCSI-3 for the device. In the unlikely case that you need to turn off fencing on all devices after Sun Cluster is already running. Inc. c ฺ vincent:/# cluster set -p global_fencing=nofencing sm uide c e Warning: Device instance d4 is a quorum device G . Sun Services. Revision C 5-31 a . for example). you would have had to turn off fencing globally during scinstall time. aku1 Stude Updating shared devices on (node y inon node Updating shared devices d his2 t u l e vincent:/# clq addad5 m tod4us aremove vincent:/# clq k vincent:/# show ngcldev se d4 a n b e a lic b e u l kDID Device b Instances === a a r y feDevice Name: ul nDID s d /dev/did/rdsk/d4 b a a r t g Full Device Path: vincent:/dev/rdsk/c1t0d0 an non Full Device Path: Replication: default_fencing: theo:/dev/rdsk/c1t0d0 none scsi3 vincent:/# cldev set -p default_fencing=global d4 Updating shared devices on node 1 Updating shared devices on node 2 vincent:/# clq add d4 vincent:/# clq remove d5 Software Quorum for Disks with no SCSI Fencing As shown in the previous section. The following example shows the elimination of all disk fencing globally.

The following command shows the verification that a quorum device is using the software quorum mechanism.Viewing and Administering Disk Paths and Settings Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems. theo y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha 5-32 Sun™ Cluster 3. Sun Services. since its SCSI fencing has been eliminated: # clq show d4 Quorum Devices === Quorum Device Name: Enabled: Votes: Global Name: Type: Access Mode: Hosts (enabled): d4 yes 1 /dev/did/rdsk/d6s2 shared_disk sq_disk vincent. Revision C a . The persistent reservations will be implemented using PGRE. All Rights Reserved. Oracle and/or its affiliatesฺ Sun Cluster will quietly implement its own software quorum mechanism to reliably and atomically simulate the SCSI-2 or SCSI-3 “race” for the quorum device. Inc. exactly as when a SCSI-2 device is used as a quorum device.

If you replace the cable. it allows you to configure new private networks and/or remove private network components without having to reboot any cluster nodes. Viewing Interconnect Status s ) ha The clintr status command shows status of all private network paths between all pairs of nodes: y m ฺ om ฺ c ฺ Cluster Transport Paths === csm Guide e @ ent b u Endpoint1 Endpoint2 Status k tud a y ( is S -------------n--------i d h vincent:ce3 theo:ce3 Path online t u l e a s vincent:ce2 Path online am to utheo:ce2 k ng ense a b a ic lsoftware Nobparticular administration is required to repair a broken e u l k b interconnect path. Sun Services. theo:ce4 and vincent:ce4) ● Define the cable between the two endpoints Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. the cluster immediately a era y l f s a path failure. for example. Oracle and/or its affiliatesฺ Viewing and Administering Interconnect Components The clinterconnect (clintr) command allows you to display the configuration and status of the private networks that make up the cluster transport. Revision C 5-33 a . All Rights Reserved. The private network definitions in the cluster configuration repository are somewhat complex. You must perform one of the following actions: ● For a private network defined without a switch (two-node cluster only): ● Define the two adapter endpoints (for example.. the path immediately goes bdu tranreports back on line. a g n n no aba # clintr status Adding New Private Networks You can cable a new private network and get it defined in the cluster without any reboots or interruption to any existing service. In addition.Viewing and Administering Interconnect Components Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. If a cable breaks.

The definitions define a switch. All Rights Reserved. the following commands define a new private network for a two-node cluster. Oracle and/or its affiliatesฺ ● For a private network defined with a switch: ● Define the adapter endpoints for each node ● Define the switch endpoint (cluster assumes any endpoint not in the form of node:adapter is a switch) ● Define cables between each adapter and the switch For example. # # # # # clintr clintr clintr clintr clintr add add add add add vincent:ce4 theo:ce4 switch3 vincent:ce4.switch3 s ) ha y m ฺ om ฺ c ฺ csm Guide # clintr status e @ ent b u Cluster Transport Paths === ak Stud y ( in this d u Endpoint1 Endpoint2 Status l e a s u -------------am t--------o k vincent:ce3 ng Path online se theo:ce3 a n vincent:ce2 theo:ce2 Path online b e c a i l b vincent:ce4 theo:ce4 Path online u le k b a a l y sfer u d ab -tran g n non a b a 5-34 Sun™ Cluster 3. Revision C a .Viewing and Administering Interconnect Components Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems. it is just a definition in the cluster. Sun Services. This does not mean that the switch needs to physically exist.switch3 theo:ce4. Inc.

The clsetup command leads you through a series of menus and dialogues.Using the clsetup Command Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. and at the end calls the lower-level administrative commands for you. # clsetup *** Main Menu *** s ) ha y m ฺ om ฺ Please select from one of the following ฺoptions: c sm uide c e 1) Quorum G t @ n b 2) Resource groups ku tude 3) Data Services (ya n this S i 4) Cluster interconnect d lu and e volumes 5) Device groups a s m u 6) Private ka ehostnames to g n Newenodes s ncluster ba7) 8) Other tasks c a li b e u l k rab a y l e ?) Help with menu options f u s d q) Quit ab -tran g an non ab Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. Revision C 5-35 a . Sun Services. Oracle and/or its affiliatesฺ Using the clsetup Command The clsetup command is a menu-driven utility meant to guide you through many common (but not all) administrative operations. Inc. so it has educational value as well. it always shows the lower-level commands as it runs them. In general. All Rights Reserved.

Nothing bad happens if you try to do things in the wrong order. or multiple operations if a switch is defined (one cable per node connected to the switch).2 Administration Copyright 2009 Sun Microsystems. you still need to know about how things are done in the Sun Cluster software environment. The clsetup command saves you from needing to remember the commands. er if you wanted to permanently delete an entire private f u s d an you need to perform the following tasks in the correct order: ab -trnetwork. g n non 1. For example. Sun Services. you see the following: *** Cluster Interconnect Menu *** Please select from one of the following options: 1) 2) 3) 4) 5) 6) 7) 8) Add a transport cable Add a transport adapter to a node Add a transport switch Remove a transport cable Remove a transport adapter from a node Remove a transport switch Enable a transport cable Disable a transport cable s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u ?) Help al use m q) Return kato the toMain Menu g e ban licens Option: a b u le k b a a l yFor example. Remove the definitions of the adapters. Remove the definition of the switch. 4. Oracle and/or its affiliatesฺ Even if you use the clsetup utility. Disable the cable or cables that define the transport. 3. Inc. if you go to the cluster interconnect submenu. Remove the definition(s) of the transport cable or cables. a b a This is a single operation for a crossover-cable definition. But this would be the same with the command-line or with clsetup. 2.Using the clsetup Command Comparing Low-level Command and clsetup Usage Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. and options. you are just informed that you missed a step. All Rights Reserved. one per node. Revision C a . 5-36 Sun™ Cluster 3. sub-commands.

Inc. Revision C 5-37 a . after you understand the nature of a task. This is frequently easier than researching and typing the syntax correctly using the lower-level commands. Sun Cluster Manager runs underneath the Sun Java Web Console. ● It highlights faulted components in real time. When you log in to the Sun Java Web Console. you might see other Web applications available besides Sun Cluster Manager.2 environment. this course does not focus on the details of using Sun Cluster Manager as the method of accomplishing the task. s ) ha y m ฺ om ฺ c ฺ csm Guide e tthe Java language to @ with You can use any modern Web-browser u enabled n b e d Manager. Sun Cluster Manager can guide you through it. a b a The Java ES installer also automatically installs or upgrades the Sun Java Web Console. and so forth. Similar to clsetup. resource groups. ● ● ● Sun Cluster Manager has additional benefits: ● It offers some graphical topology views of the Sun Cluster nodes with respect to device groups. Sun Services. However.Sun Cluster Manager Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Java Web Console is a single sign-on interface to many Solaris OS applications that are implemented as Web back-ends. Many of the exercises in the modules following this one offer you the opportunity to view and manage device and resource groups through Sun Cluster Manager. You can choose whether or not to install Sun Cluster Manager along with the Sun Cluster framework from the Java ES installer. a b It saves u leyou from having to remember the syntax of the lower-level k b a a r l y sfecommands. if needed. access the Sun Java Web Console y and akthe Sun tuCluster ( S in this d u Usage of Sun Cluster Manager al usisemuch like the usage of clsetup in that: m ka need ttoounderstand the cluster tasks before You stillgreally e s accomplishing ban licenanything. Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. u d ab -tran It accomplishes all its actions through the lower-level commands. g n non and shows the lower-level commands as it runs them. All Rights Reserved. Oracle and/or its affiliatesฺ Sun Cluster Manager The Sun Cluster Manager is an optional Web-based administrative tool for the Sun Cluster 3. The browser refreshes automatically as component status changes. regardless of whether you choose to install Sun Cluster Manager.

Alternatively. you can create a non-root user or role authorized to have full or limited access through RBAC. Revision C a . any user will be able to log in to Sun Cluster Manager through the Sun Java Web Console and have view-only access. All Rights Reserved. 5-38 Sun™ Cluster 3. https://nodename:6789 Figure 5-1 shows the Sun Java Web Console login window. By default. Cookies must be enabled on your Web browser to allow Sun Java Web Console login and usage.2 Administration Copyright 2009 Sun Microsystems.Sun Cluster Manager Logging Into the Sun Java Web Console Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Figure 5-1 Sun Java Web Console Login Window Log in with the user name of root and the root password. Oracle and/or its affiliatesฺ Use the following URL on a Web browser enabled with Java technology to access the Sun Java Web Console. Sun Services. Inc.

Sun Cluster Manager Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. You can choose to open the Sun Cluster Manager in its own browser window or in the same window. Inc. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Figure 5-2 Applications Available Underneath Sun Java Web Console Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. Sun Services. You can have other administrative applications available as well. Figure 5-2 shows the screen you access after logging into the Sun Java Web Console. Revision C 5-39 a . All Rights Reserved. Oracle and/or its affiliatesฺ Accessing Sun Cluster Manager The Sun Cluster Manager application is always available from within the Sun Java Web Console.

All Rights Reserved. Inc. Figure 5-3 shows the initial screen. with the tree-like navigation bar on the left. Topological views are available through a tab in the main window. Oracle and/or its affiliatesฺ Sun Cluster Manager navigation is simple. Revision C a . s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Figure 5-3 5-40 Sun Cluster Manager Main Window Sun™ Cluster 3. and the display corresponding to your current selection in the main frame.Sun Cluster Manager Navigating the Sun Cluster Manager Main Window Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services.2 Administration Copyright 2009 Sun Microsystems.

2 has a simplified RBAC structure that can allow you to assign cluster administrative privileges to non-root users or roles. remove and related subcommands ● ● ● Assigned directly to a role (allowed users have to assume the role.b@ online. Oracle and/or its affiliatesฺ Running Cluster Commands as Non-root User or Role Using Role Based Access Control (RBAC) Sun Cluster 3. Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. ya tu ( S n di e this u l a usRBAC authorizations of a much finer m Note – Sun Cluster 3. interconnect. list. and authorizations are syill accessible c a i l b and for back-compatibility. enable.read This authorization gives the ability to do any status. and then they would get the authorization) ● Assigned to a rights profile that is then given to a user or a role There is a review of RBAC concepts in the appendix. but they are no longer required.Running Cluster Commands as Non-root User or Role Using Role Based Access Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. the cluster authorizations can be: g n non a Assigned directly to a user b a This gives the ability to do add.admin csm Guide e t This gives the ability to do switch. Note – The Sun Java Web Console detects if the user you use for its own login has the ability to assume any roles. Sun Services. The information is passed down to the Sun Cluster Manager.cluster. There are only three authorizations that pertain to Sun Cluster: ● solaris. All Rights Reserved. show subcommands.modify s ) ha y m ฺ om ฺ c ฺ solaris.cluster. u are supported le k b a a l y sfer u d ab -tranLike any RBAC authorizations. which follows the RBAC authorizations. the Web Console gives you additional options to assume the role. These old a n b e resource groups. delete. create. e sso on).1 required a k authorizations to g granularityn (separate to manage quorum. If it does. Revision C 5-41 a . By default every user has this authorization. offline. since it is in the Basic Solaris User profile ● solaris.cluster. n e u d k disable subcommands. Inc.

s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak installation. Oracle and/or its affiliatesฺ Basic cluster control includes starting and stopping clustered operation on one or more nodes and booting nodes in non-cluster mode. Sun Services. When your cluster is in an ab -trcommand g production and running clustered applications.d/K05stoprgm: disabling failfasts Apr 9 03:59:58 vincent syseventd[413]: SIGHUP caught reloading modules Apr 9 03:59:59 vincent syseventd[413]: Daemon restarted 5-42 Sun™ Cluster 3. All Rights Reserved. Inc. you should switch resource groups to the next preferred node and then run shutdown or init on that node. configured resource groups with inwhichtto d hibe u l e a us m a k e to g ban alicCluster ens ShuttingaDown b u le k b a a r down the entire cluster with the cluster shutdown l yYou can eshut f u s d from any active cluster node. Revision C a . vincent:/# /etc/rc0. Use the cluster shutdown command to shut down all nodes in the cluster.2 Administration Copyright 2009 Sun Microsystems. environment is that you should always be able to keep at least one node running. Note – After an initial Sun Cluster software tud there are no y ( S s concerned.d/K05stoprgm: Calling clzc halt -n vincent + /etc/rc0.d/K05stoprgm: Calling scswitch -S (evacuate) /etc/rc0. Apr 9 03:59:49 vincent cl_runtime: NOTICE: CMM: Quorum device /dev/did/rdsk/d4s2: owner set to node 1. vincent:/# cluster shutdown -y -g0 Apr 9 03:59:49 vincent cl_runtime: NOTICE: CMM: node reconfiguration #20 completed. Use the standard init or shutdown command to shut down a single node. you will have a goal of n n o a n never having to do this. The whole purpose of the Sun Cluster ab Before shutting down an individual node. Starting and Stopping Cluster Nodes The Sun Cluster software starts automatically during a system boot operation.Controlling Clusters Controlling Clusters Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

startd: The system is down.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Please wait.002:0.000:0.startd: The system is coming down. Sun Services. syncing file systems. Oracle and/or its affiliatesฺ Controlling Clusters svc.. end = 6 TCP_IOC_ABORT_CONN: aborted 0 connection NOTICE: CMM: node reconfiguration #22 completed. remote = 172.000.startd: 150 system services are now being stopped. Revision C 5-43 a . WARNING: CMM: Node being shut down.000. Apr 9 04:00:05 vincent cl_eventlogd[3143]: Going down on signal 15.NOTICE: clcomm: Path vincent:ce2 theo:ce2 being drained NOTICE: clcomm: Path vincent:ce3 . Inc. TCP_IOC_ABORT_CONN: local = 000.004. svc. start = -2.theo:ce3 being drained NOTICE: CMM: Node theo (nodeid = 2) is down. Apr 9 04:00:37 vincent syslogd: going down on signal 15 Apr 9 04:00:37 rpc. All Rights Reserved. done NOTICE: CMM: Quorum device /dev/did/rdsk/d4s2: owner set to node 1.. NOTICE: CMM: Cluster members: vincent.metad: Terminated svc. Program terminated {0} ok s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Performing Basic Cluster Administration Copyright 2009 Sun Microsystems.016.

. you might want to boot a node without it joining the cluster. All Rights Reserved.ubooting d al use m {1} ok boot -x ka e to g Resetting ..2 Administration Copyright 2009 Sun Microsystems. Inc. For example. s ) ha To boot a node to non-cluster mode you supply the -x boot option. This might be to debug some sort of problem preventing a node from joining a cluster. which gets passed through to the kernel.700000/scsi@2/disk@0. Other nodes might still be up running your clustered applications. n ab Use is subject to license terms Hostname: vincent Reading ZFS config: done. or to perform maintenance. To other nodes that are still booted into the cluster. y m ฺ om ฺ c ฺ m -x ide Booting a SPARC Platform Machine With csthe u e G t @ n b Command ku tude a y in ( isthsimple: is S For a SPARC-based machine.5. Inc. Oracle and/or its affiliatesฺ Booting Nodes Into Non-Cluster Mode Occasionally. ban licens a b u le k b a a Rebooting with command: boot -x y fer l u s d /pci@1f. Sun Services.Controlling Clusters Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a .10 Version n o a Copyright 1983-2009 Sun Microsystems. It can not be reached across the cluster transport.0:a Boot device: File and args: -x n b a a r t SunOS Generic_141444-09 64-bit g Release n. All rights reserved. 5-44 Sun™ Cluster 3. Mounting ZFS filesystems: (5/5) vincent console login: root Password: # cluster status cluster: (C152734) This node is not in cluster mode. you upgrade the cluster software itself when a node is booted into non-cluster mode. a node that is booted into non-cluster node looks like it has failed completely.

Press enter to boot the selected OS. The highlighted entry will be booted automatically in 20 seconds. Sun Services. or escape to go back to the main menu. You see screen: ng ense a b a lic lower / 2095552K upper memory) b e u l GNU GRUB version 0.2 (Update 3 and above). Oracle and/or its affiliatesฺ Booting an x86 Platform Machine With the -x Command For an x86 machine. e a10 s With the normal Solaris press the e key to edit the boot m u a the following o k t parameters. Press 'b' to boot. 'o' to open a new line after ('O' for before) the selected line. y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u l OS highlighted. 'd' to remove the selected line. Inc. Revision C 5-45 a . 'c' for a command-line. Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. As the machine boots.95 (615K k rab a y ul nsfe d +----------------------------------------------------------------------+ b a (hd0.0.a) tra | root n | g n o a n | ab | kernel /platform/i86pc/multiboot | module /platform/i86pc/boot_archive | +----------------------------------------------------------------------+ Use the ^ and v keys to select which entry is highlighted. For versions of Solaris 10 OS supported with Sun Cluster 3.Controlling Clusters Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. booting is a little more complicated. All Rights Reserved. or 'c' for a command-line. 'e' to edit the commands before booting.95 (615K lower / 2095552K upper memory) s ) ha +----------------------------------------------------------------------+ | Solaris 10 11/09 s10x_u8wos_08 X86 | | Solaris failsafe | +---------------------------------------------------------------------+ Use the ^ and v keys to select which entry is highlighted. you see the following menu. 'e' to edit the selected command in the boot sequence. with the normal OS highlighted: GNU GRUB version 0. the Gnu boot menus (GRUB) are automatically used.

Now you can press b to boot. s ) ha y m ฺ om ฺ c ฺ +----------------------------------------------------------------------+ csm Guide | | root (hd0. or escape to go back to the main menu. 'c' forna 'o' to open a new line se a n b e after ('O' for before) line.2 Administration Copyright 2009 Sun Microsystems. ESC at any time exits. 'd' to remove the a the lic selected b e selected line. Revision C a .a) e @ ent b | kernel /platform/i86pc/multiboot -x | u d k u a | module /platform/i86pc/boot_archive | t (y is S n +----------------------------------------------------------------------+ i h highlighted.0. tis lud entry Use the ^ and v keys to select which e a us command in the Press 'b' to boot. For the first word. ab 1983-2009 r t Copyright g n is subject on to license terms. Inc. All Rights Reserved. u l k b a a l y5. boot sequence. 'e' to k edit amthetoselected g command-line.10 Generic Jan 2005 # cluster status cluster: (C152734) This node is not in cluster mode. Sun Services. Anywhere else TAB lists the possible completions of a device/filename. All rights reserved. ] grub edit> kernel /platform/i86pc/multiboot -x Press return to get back to the screen listing boot parameters. TAB lists possible command completions. where you will see your -x in the appropriate place. 5-46 Sun™ Cluster 3. Inc.95 (615K lower / 2095552K upper memory) Hostname: gabi gabi console login: root Password: Jul 10 00:31:50 gabi login: ROOT LOGIN /dev/console Last login: Mon Jul 10 00:19:32 on console Sun Microsystems Inc.10sVersion er Generic_141445-14 64-bit f u d SunOS Release an Sun Microsystems. then press the e again to edit that specific line and add the -x [ Minimal BASH-like line editing is supported. a Use n b a GNU GRUB version 0.Controlling Clusters Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. SunOS 5. Oracle and/or its affiliatesฺ You need to use the arrows to highlight the line that begins with kernel.

s ) ha theo:/# clq status y m ฺ Cluster Quorum === om ฺ c ฺ --. The maintenance state disables the node’s quorum vote. All Rights Reserved. You can reset the maintenance state for a node by rebooting the node into the cluster. Inc.Quorum ban Votes e c a li b e u l k Present Possible Status a NodeerName ab y l f ------------------s bdu tran--------vincent 0 0 Offline a g n n theo 1 1 Online no aba --. The node and any dual-ported quorum devices regain their votes. Oracle and/or its affiliatesฺ Placing Nodes Into Maintenance State If you anticipate that a node will be down for an extended period. Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. A typical command is as follows: theo:/# clq disable vincent The clquorum status command shows that the possible vote for theo is now set to 0. the vote count for any dual-ported quorum device physically attached to the node is also set to 0.Controlling Clusters Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.Quorum Votes by Device --Device Name ----------d4 Present ------0 Possible -------0 Status -----Offline In addition. You cannot place an active cluster member into maintenance state. Sun Services. you can place the node into maintenance state from an active cluster node. Revision C 5-47 a .Quorum Votes Summary --csm Guide e @ ent b u Needed Present akPossible tud y ( S ------------------in th1is d 1 1 u al use m ka e to g s Node --nby --.

because you would have only three of the total possible six quorum votes. Sun Services.2 Administration Copyright 2009 Sun Microsystems. There would temporarily be a total possible value of four quorum votes. “Preparing for Installation and Understanding Quorum Devices. Inc. 5-48 Sun™ Cluster 3. If you put node 4 into maintenance mode. and the quorum vote of the shared quorum device between nodes three and four. you will lose the whole cluster. This would allow you to survive the death of node 3. you would eliminate its quorum vote.” Switch Switch s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak SQD(1) tud y ( QD(1) in this d u al use m ka e Quorum to Figure 5-4 Clustered-Pair Devices g n s a n ab lice b Imagine u thereblise no way that you can add a quorum device between k a ra 3 as we described in an earlier module (maybe you have no ynodes f2eand l u s n available storage controllers). which was described in Module 3. All Rights Reserved. but you can tell that if you lose node 3. making the required number of votes 3. Revision C a . bd trmore a a g onn a n Now imagine that node 4 has died. Oracle and/or its affiliatesฺ To see the value of placing a node in maintenance state. At that point in time.Controlling Clusters Maintenance Mode Example Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. consider the following topology (shown in Figure 5-4). the cluster ab Node 1 (1) Node 2 (1) Node 3 (1) Node 4 (1) survives.

You run this from one node only. Revision C 5-49 a . and it automatically propagates to the other nodes. and you can give a different anticipated maximum number of nodes and subnets and use a different suggested netmask. No other remote shell accessbis required n between nodes. In e u d k previous revisions of Sun Cluster 3. and it guides you through this task. in this d u al use m # clsetup ka e to g ban licens a *** Main Menuub *** le k b a a l yfromsfone er of the following options: u Select d ab -tran g an non1) Change Network Addressing and Ranges for the Cluster Transport 2) Show Network Addressing and Ranges for the Cluster Transport ?) Help with menu options q) Quit Option: 1 >>> Change Network Addressing and Ranges for the Cluster Transport <<< Network addressing for the cluster transport is currently configured as follows: Private Network === Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Modifying Private Network Address and Netmask The final task presented in this module is unique in that it must be accomplished while all nodes are in multi-user.2. you have to set up rsh or ssh a Stu y ( equivalence for root manually. if you run clsetup. In this mode. s ) ha ab y m ฺ m o c ฺ ฺ same Note – In the current revision nodes can communicate using the e m d s i Remote Procedure Calls that are used use to communicate ec t Guduring @ scinstall.Modifying Private Network Address and Netmask Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. Inc. it recognizes that the only possible task is to change the private network information. non-cluster mode. You can choose a different network number. All Rights Reserved.

0 y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( Maximum number of nodes anticipated ifor n future is growth [2]? 4 d h t u e al anticipated s Maximum number of private networks for future growth m u a o k t [2]?4 ng ense a b c about number of virtual clusters a liask // Note this script does not b e u l k rab a y fe of 255. Inc.0 255.5.0 -p private_netmask=255.16.0. g n n o aba Whatn netmask do you want to use [255. Do you want to use the default (yes/no) [yes]? What network address do you want to use? no s ) ha 192. 5-50 Sun™ Cluster 3.255.255. Sun Services.0. . .0 to meet anticipated future ula netmask Specify s d n b a requirements tra of 4 cluster nodes and 4 private networks. Is it okay to proceed with the update (yes/no) [yes]? yes cluster set-netprops -p private_netaddr=192.16. Press Enter to continue: After you reboot into the cluster. .168. All Rights Reserved. For cluster-aware applications.255.0 -p max_nodes=8 -p max_privatenets=4 Command completed successfully.255.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.0 64 10 12 Do you want to change this configuration (yes/no) [yes]? yes The default network address for the cluster transport is 172.0.240. your new private network information is automatically applied by the cluster.168. clusternode1-priv.255.0]? <CR> .255.255.5. Revision C a . the same clprivnet0 private hostnames (by default. .2 Administration Copyright 2009 Sun Microsystems. . Oracle and/or its affiliatesฺ Modifying Private Network Address and Netmask private_netaddr: private_netmask: max_nodes: max_privatenets: num_zoneclusters: 172. and so on) now resolve to the new addresses.

or clustername embedded in a k command string. Inc. am to unode1.Exercise: Performing Basic Cluster Administration Exercise: Performing Basic Cluster Administration Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. you complete the following tasks: ● Task 1 – Verifying Basic Cluster Configuration and Status ● Task 2 – Reassigning a Quorum Device ● Task 3 – Adding a Quorum Server Quorum Device ● Task 4 – Preventing Cluster Amnesia ● Task 5 – Changing the Cluster Private IP Address Range ● Task 6 (Optional) – Navigating Sun Cluster Manager s ) ha y m ฺ om ฺ c ฺ e idthe csmtool Boot all nodes into the cluster and run the cconsole on u e G t @ administration workstation. Oracle and/or its affiliatesฺ In this exercise. All Rights Reserved. n s a n ab lice b u le k b a a l y 1sf–erVerifying Basic Cluster Configuration and Task u d an ab Status r t g n non a b a Preparation Perform the following steps to verify the basic status of your cluster: 1. such as lu when e a s IPaddress. Use the clquorum (clq) command to verify the current cluster membership and the quorum status. # clq status Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. g substitute e the names appropriate for your cluster. Sun Services. Verify the cluster global properties and node properties: # cluster show -t global # clnode show 2. enclosure_name. you see italicized names. n b ku tude a y n ( this S i d Note – During this exercise. Revision C 5-51 a .

Verify the revision of the currently installed Sun Cluster software on each node. Verify the interconnect status: # clintr status 6. Oracle and/or its affiliatesฺ 3. l e f bdu trans# cldev list -v a ang non 3. Quorum votes needed: _____ Quorum votes present: _____ Quorum votes possible: _____ Verify all cluster disk paths. s ) ha y m ฺ om ฺ c ฺ Task 2 – Reassigning a Quorum Device csm uide e tG @ n b e u d k Pretend that your quorum device (or one of your quorum devices) has a Stu y ( failed. Remove your old broken quorum device: # clq remove old-d-# 5. Verify the status kaof your tocurrent quorum device(s): g e an cens # clqbstatus a li d# #bcldev e status u l k a Show aallb disk paths: r y 2. # cldev status 5.2 Administration Copyright 2009 Sun Microsystems. 5-52 Sun™ Cluster 3. Sun Services. Verify the quorum status again: # clq status Task 3 – Adding a Quorum Server Quorum Device Your instructor will have the quorum server server-side software available so that you can add a quorum-server device in the cluster.Exercise: Performing Basic Cluster Administration Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a . Perform the following tasks in fromthany is one node in the cluster to reassign the quorum device:lud a use m 1. 4. Inc. All Rights Reserved. Choose a different shared disk to be the new quorum device: Add # clnode show-rev -v ab your new quorum device: # clq add new-d-# 4. Record the quorum configuration from the previous step.

If you have a cluster d u l having with non-storage a nodes. You can just add the device using the IP address. Add the quorum server quorum device. Delete your disk quorum device(s). Revision C 5-53 a . The name you give to the device on the cluster side (the last argument to the command) is unimportant. tudthe disk quorum devices y ( and remove the quorum inserverthquorum is device. (Optional): Reverse the process. a quorum server quorum device is e s m u not necessarily owhat you want. If you have a quorum server ) y m device you need no other quorum devices: ฺ om ฺ c # clq remove old-d-# [old-d#] ฺ csm Guide e # clq status @ ent b u akAddSback 3. Oracle and/or its affiliatesฺ Your instructor will provide you the quorum server IP address/name information. because it will allow the nonka exactly talive g e storagennode to s stay even if both storage nodes have failed.Exercise: Performing Basic Cluster Administration Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. # clq add -t quorum_server \ -p qshost=qserver_name_or_IP -p port=9000 qservdude # clq status as h 2. or you can use the name if it is resolvable on your cluster nodes: Perform the following steps on any one cluster node: 1. Inc. Sun Services. All Rights Reserved. a n b e #a clq add licd# [d#] b e u l k #raclq b remove qservdude a y l e f bdu trans a ang non ab Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. you can use anything you want.

You can review the information about what you will see by checking Module 3. s ) ha y m ฺ om ฺ c ฺ 4. run the procedure to change the cluster private IP address range: a. Sun™ Cluster 3. On the chosen node only. Choose (any) single node on which to perform the commands. 2. Execute the clsetup command. Shut down Node 1 with init 0. Sun Services. using persistent reservations on the quorum device. “Preparing for Installation and Understanding Quorum Devices. d. Boot Node 1. This t the quorum @ enfrom b the reservation key for Node 1 has been removed u ak Stud disk(s). If you have a three-node cluster (Pair +1). 4. Now boot Node 2. All Rights Reserved.” 1. c. b. m a o k t ng ense a b a lic the Cluster Private IP Address Task 5 –uChanging b e l k rab a y Range ul nsfe d b a a trPerform g n the following steps: n o a n b a # init 0 1. shut down the node not attached to storage with init 0. Both nodes t u l e ajoin inuclustered s startup sequence and operation. Shut down the entire cluster by typing on any one node: # cluster shutdown -y -g0 5-54 2. Shut down Node 2. Choose option 1 to change the configuration. Revision C a . Oracle and/or its affiliatesฺ Perform the following steps to demonstrate how the cluster prevents cluster amnesia. Inc. y ( in should d hiscomplete the cluster software 5. Answer no when asked if you want to accept the default base address. Boot all nodes to non-cluster mode using boot -x 3. 3. Verify (answer yes) when asked if you want to change the settings.Exercise: Performing Basic Cluster Administration Task 4 – Preventing Cluster Amnesia Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. e csm G uidis because Node 1 should hang waiting for operational e quorum.2 Administration Copyright 2009 Sun Microsystems. Verify that all nodes are active cluster members.

h. Do not accidentally conflict with the public network address space. Verify the output of cluster show e-t G t @ n b ku tude a y n ( this S i d lu se a am to u k ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n no aba a. Sun Services.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. Accept the suggested netmask (255. j. i. When the command succeeds.0). 6. Use ifconfig -a and verify the information for the physical private network adapters and for the clprivnet0 adapter. s ) ha y m ฺ om ฺ c ฺ m uide csglobal. All Rights Reserved. Answer no when asked if you want to accept the default netmask. g.255. Confirm that you want to proceed with the update. Enter a new available private address base address. b. Enter 4 for the maximum number of private networks. Revision C 5-55 a . 5. Consult your instructor if you are not sure. Enter 4 for the maximum number of nodes.255. Oracle and/or its affiliatesฺ Exercise: Performing Basic Cluster Administration e. Verify that the private network addresses have changed. f. reboot all nodes back into the cluster. Inc.

k Navigate b to the Sun Cluster Manager application. Inc. a a r y fe yourself with Sun Cluster Manager navigation and the ul 4. Ask your instructor if you need help. you might be able to use a Web browser to access Sun Cluster Manager in one of two ways: 5-56 Sun™ Cluster 3. Sun Services. All Rights Reserved.Exercise: Performing Basic Cluster Administration Task 6 (Optional) – Navigating Sun Cluster Manager Perform the following steps: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. nsFamiliarize d b a topological views. ku thetutraffic b. type the following URL: https://nodename:6789 where nodename is the name of one of your nodes currently booted into the cluster. Log inbas root with the root password. Run the browser locally anda tunnel y n ( this S i d lu se a Note – Consult your instructor u how this will be accomplished. e a ic l b e u l 3. you might need to disable or set exceptions for the proxy settings in your Web browser.2 Administration Copyright 2009 Sun Microsystems. a r g on-t n a n ab If you are running the class with remote equipment. am toabout k ntheg user se a n 2. In a Web browser window on your administrative workstation. s ) ha y m ฺ om ฺ c ฺ sm uide c e a. If you do not succeed in reaching the Sun Java Web Console. Oracle and/or its affiliatesฺ 1. Run the browser on your remote display. G t @ n b de through ssh. Revision C a .

Exercise Summary Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. ● Experiences ● Interpretations ● Conclusions ● Applications s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Performing Basic Cluster Administration Copyright 2009 Sun Microsystems. Revision C 5-57 a . Oracle and/or its affiliatesฺ Exercise Summary ? ! Discussion – Take a few minutes to discuss what experiences. issues. or discoveries you had during the lab exercises. All Rights Reserved. Inc.

Revision C a . Oracle and/or its affiliatesฺ Exercise Summary s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 5-58 Sun™ Cluster 3.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc.2 Administration Copyright 2009 Sun Microsystems. All Rights Reserved. Sun Services.

Revision C a . All Rights Reserved. Inc. Sun Services. Oracle and/or its affiliatesฺ Module 6 Using VERITAS Volume Manager With Sun Cluster Software s ) ha Objectives ab y m ฺ om ฺ Upon completion of this module. manage.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. and resynchronize VxVM disk groups as cluster device groups ● Create global and failover file systems on VxVM volumes ● Describe the procedure used for mirroring the boot drive 6-1 Copyright 2009 Sun Microsystems.2 r y l e f bdu trans software environments a Describe the general procedures and restrictions for installing and ang non ● ● ● ● ● ● administering VxVM in the cluster ● Use basic commands to put disks in disk groups and build volumes ● Describe the two flags used to mark disks under the typical hot relocation ● Register.2 @ (VxVM) and issues involved in using VxVMein n b u ak Stud software environment y ( in this and shared storage disk Differentiate between bootdg/rootdg d u al use groups m a o kVERITAS tVolume g Initialize a Manager disk e n ens a b a ic basic objects in a disk group Describe lthe b e u l k Describe a ab the types of volumes that will be created for Sun Cluster 3. you should be able to: c ฺ e csm GVolume uid Manager Describe the most important concepts of e VERITAS tthe Sun Cluster 3.

2 Administration Copyright 2009 Sun Microsystems. All Rights Reserved.Relevance Relevance Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. Oracle and/or its affiliatesฺ Discussion – The following questions are relevant to understanding the content of this module: ● Why is it so important to mirror JBOD disks with VxVM if the cluster already provides node-to-node failover? ● Are there any VxVM feature restrictions when VxVM is used in the Sun Cluster software environment? ● In what way does Sun Cluster provide high availability for disk groups? ● Can your prevent Sun Cluster from providing a global device service for your volumes? y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha 6-2 Sun™ Cluster 3. Revision C a . Inc.

part number 820-7458 (for Sun Cluster 3.2 11/09). part number 821-0259 (for Sun Cluster 3. c ฺ ฺ Symantec Corporation (available with Veritas m Storage Foundation e d s i software). Revision C 6-3 a . Inc. s ) ha y m ฺ Veritas Storage Foundation Installation Guide (version 5. part number 820-4676 (for Sun Cluster 3. Oracle and/or its affiliatesฺ Additional Resources Additional resources – The following references provide additional information on the topics described in this module: ● Sun Cluster System Administration Guide for Solaris OS. ● Sun Cluster Concepts Guide for Solaris OS.Additional Resources Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. ec t Gu @ b den u k ya Stu ( n di e this u l a us m a k e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a ● Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems.0 omfor Solaris).2 11/09)uide for Solaris OS.2 11/09). All Rights Reserved. part number 8207356 (for Sun Cluster 3. ● Sun Cluster Software Installation Guide for Solaris OS.2 1/09).

Rather.2 updates earlier than Update 2 (1/09) ● VxVM 4.2 Update m u a kMP3 toand RP3 Solaris 9 OS/Solaris 10 OS VxVM 5. Only the following versions of VxVM are supported in Sun Cluster 3.Introducing VxVM in the Sun Cluster Software Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. it briefly introduces the most important concepts of VxVM and focuses on issues involved in using VxVM in the Sun Cluster 3. Sun Services.1 MP2 for Solaris 10 OS (x86) s ) ha y m ฺ om ฺ c ฺ VxVM 5. Revision C a . Inc.2 Administration Copyright 2009 Sun Microsystems.1 MP1 Solaris 9 OS/Solaris 10 OS (SPARC) ● VxVM 4.2 environment.2 software: ● For Sun Cluster 3.0 MP3 RP1 Solarisy9aOS/Solaris ( x86) n this S i d lu 3 s(11/09) e a For Sun Cluster 3. All Rights Reserved.0 RP2 g e s (SPARC ban liand enx86) c a b u le k b a a l y sfer u d ab -tran g an non ● ● ● ● ● ab 6-4 Sun™ Cluster 3.0 Solaris 9 OS/Solaris 10 OS (SPARC) csm Guide e For Sun Cluster 3.2 Update 2 (1/09) @ ent b u k tud10 OS (SPARC and VxVM 5. Oracle and/or its affiliatesฺ Introducing VxVM in the Sun Cluster Software Environment This module does not intend to replace a full class in VxVM administration.

In that case. Disks in the same disk group act as an organizational unit. All Rights Reserved. Every disk or LUN used by VxVM must be a member of exactly one disk group. Shared Storage Disk Groups ab as h All of the data associated with applications that run in the Sun Cluster 3.x. VxVM automatically sets up a symbolic link so that whatever the actual group name is there will be a link to it called bootdg.Exploring VxVM Disk Groups Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.x manage your boot disk (that is.x and Above)ludi his t e a us m a k VxVM torequired that each node have an independent Prior to VxVM 4.two. for HA) sacrificial local disks so that you could have a rootdg. you needed at least one (preferably bdu tranmanage a .0. and you are still able to create your shared storage disk groups. Sun Services. bannamed e c a li else in VxVM until rootdg was configured. Revision C 6-5 a . This group can have any group name. You can choose not to configure any VxVM configurations on local disks. to encapsulate your boot disk). Disk groups are created using only disks oinmmultiple c ฺ ฺ e arrays that are connected to the same collection ofsnodes. m d i ec t Gu @ b den u k VERITAS Management on (Local tu (Optional in ya Disks S n VxVM 4. If you want to have VxVM 4. Inc. then you have this disk (and presumably another because there is no reason to encapsulate root if you are not going to mirror it) in a dedicated local VxVM disk group.x or 5. Oracle and/or its affiliatesฺ Exploring VxVM Disk Groups The assignment of disk drives or logical units (LUNs) into disk groups is the most important organizational concept for managing VxVM in the cluster environment. g e ns In these earlier versions you could not disk group rootdg. ang non Starting in VxVM 4. You b anything configure e u l k a needed atobhave this disk group even if you did not intend VxVM to r y l e f s your boot disk. it is optional to have any disk group at all configured for the local disks. Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems.2 ) yto at software environment must be in storage that is physically ported m ฺ least two nodes.

Revision C a . the tool that is used in the Sun Cluster environment to encapsulate the root disk (clvxvm encapsulate). which is described later in the module. All Rights Reserved.Exploring VxVM Disk Groups Typical Sun Cluster VxVM Configuration Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Figure 6-1 shows a typical organization of disks into disk groups in a simple Sun Cluster software environment. While the name of the actual disk group underlying the bootdg link can be anything. still names it rootdg. Diskgroup nfsdg Diskgroup webdg Node 1 Node 2 y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g Boot Disks Boot Disks an cens b a i (bootdg->rootdg [optional ]) (bootdg->rootdg [optional]) l b e u l k a 6-1 abTypical Disk Organization in a Sun Cluster Software r y l e Figure f Environment bdu trans a g n non a b a s ) ha 6-6 Sun™ Cluster 3. Sun Services.

Sun Services. This applies even for other nodes that are physically connected to the disks. The node that is currently importing a disk group is the one that does the following: ● It physically reads and writes data to the drives within the disk group. ( n import ito issuing all VxVM commands d hisand deport disk groups. The node t u l e adisk group currently importingm the s becomes the primary device group u a o k et server. including nodes that are not even physically connected to the disks in the disk group. VxVM commands pertaining to that diskgroup can be issued only from the node importing the disk group.Exploring VxVM Disk Groups Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. Inc. bdu trans a ang non ● Sun Cluster Global Devices Within a Disk Group While the fact that only one node imports a disk group remains true in the Sun Cluster software environment. g n a cens b a li disk groups so that the Sun Cluster software has Registering VxVM b e u l k a knowledge ab of them is described in‘‘Registering VxVM Disk Groups’’ on r y l e f page 6-32. ● It manages the disk group. Sun Cluster software’s global device infrastructure makes it appear that devices in the disk group are accessible from all nodes in the cluster. Oracle and/or its affiliatesฺ Shared Storage Disk Group Ownership While the disks in disk groups used for Sun Cluster software application data must be physically connected to at least two nodes. Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. ab as It can voluntarily give up ownership of the disk group by deporting h ) y the disk group. m ฺ om ฺ c ฺ csm Guide Sun Cluster Management of Disk Groups e @ ent b u ak Stuthed cluster is responsible for y After a disk group is managed by the cluster. or imported. disk groups are owned. All nodes that are not the current primary for a particular disk group actually access the device data through the cluster transport. Revision C 6-7 a . by only one node at a time.

this VxVM feature can be used only by the Oracle Real Application Clusters (RAC) application. you can run RAC using Solaris Volume Manager or without a volume manager. Sun Services. In the Sun Cluster 3. Revision C a . ec t Gu @ b den u k ya Stu ( n di e this u l a us m a k e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 6-8 Sun™ Cluster 3. however.2 software. The feature is not supported on x86. All Rights Reserved. directly through the storage medium.2 Administration Copyright 2009 Sun Microsystems. s ) ha y m ฺ m o c ฺ ฺ Do not confuse the usage of shared storage disk group throughout this e m d s i module with the CVM feature of VxVM. This is a separately licensed feature of VxVM that allows simultaneous access to disk group volumes by multiple nodes. Oracle and/or its affiliatesฺ You might see references throughout the VERITAS documentation to a Cluster Volume Manager (CVM) feature. Inc.Exploring VxVM Disk Groups VxVM Cluster Feature Used Only for Oracle RAC Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. and creating disk groups with a specific shared flag.

0 it is b 32MB. lu se a u am o k t As shown in Figure 6-2. For example. it must be initialized by VxVM. Sun Services. Inc.( This is a S n this disk layout specific to the i d Solaris OS. The private and public regions are used for the following different purposes: ● The private region is used to store configuration information. All Rights Reserved.enOne b u way which uses separate Solaris OS disk slices for the k tud private region (slice a y 3) and the public region (slice 4). The disk is divided into two sections called the private region and the public region. ● The public region is used for data storage. in VxVM ng ense a 5. s ) ha y Traditional Solaris OS Disks and Cross-Platform Data m ฺ m Sharing (CDS) Disks ฺco ฺ csm Guide e VxVM has two different ways of initializing t is the traditional @disks. a lic b e u l k rab Private region a y l e f bdu trans a ang non ab Configuration and Management Information Data storage Figure 6-2 Public region VxVM Disk Initialization (Traditional Solaris OS Only Disks) Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. the private region is small. Revision C 6-9 a .Initializing a VERITAS Volume Manager Disk Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Initializing a VERITAS Volume Manager Disk Before a disk can be put into a VxVM disk group.

Oracle and/or its affiliatesฺ Starting in VxVM 4. All Rights Reserved. but only using the sliced layout that is specific to the Solaris OS.0. the vxdiskadd utility can perform both steps for you. The defaults are always to use CDS disks.2 Administration Copyright 2009 Sun Microsystems. u l k b a a l y sfer u d ancannot use CDS disks for the bootdg. The EFI label is required for disks of size 1 Terabyte or greater. As shown in Figure 6-3 this layout combines the configuration and data storage into one large partition (slice 7) covering the entire disk. The VxVM configuration on these disks can be read by VxVM running on all its supported platforms. but the vxdiskadd command can guide you through setting up your disks either way Note – You cannot use the CDS layout for EFI disks. VxVM can deal with these properly. Revision C a .Initializing a VERITAS Volume Manager Disk Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. not just on the Solaris OS. the new default way to partition disks is called Cross-Platform Data Sharing (CDS) disks. Inc. Sun Services. 6-10 Sun™ Cluster 3. ab -trYou g n non a b While initializing a disk and putting it in a disk group are two separate a operations. Private CDS private and public regions both managed internally in s7 s ) ha y m ฺ om ฺ c ฺ Public csm Guide e @ ent b u ak Stud y ( in this d Figure 6-3 CDS Disk Layout u al use m ka e to you are not able to access VxVM disks g In the Sun Cluster environment s nthan ban other e from anyaservers in the same cluster. it does not matter c liyou choosethose b e which disk layout for your data disks.

Its own logical n ( tDisk s IDs i i VxVM does not use the SundCluster h u e naming scheme servesaalsimilarspurpose. The smallest possible subdisk is 1 block and g n non the largest is the whole public region of the disk. Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. Many of the objects (such as subdisks and plexes) are often automatically created for you when you use the recommended commands (such as vxassist) to create volumes. or data plex. Inc. This logical name can be anything. Oracle and/or its affiliatesฺ Reviewing the Basic Objects in a Disk Group This section reviews some of the basic naming terminology of objects in disk groups. the most reliable and the quickest method of checking status is from the command line. ● A volume with one data plex is not mirrored. Command-line status tools are easy to use in script files. striped. or RAID5. and describes the layout of the subdisks as either concatenated. A plex. Disk Names or Media Names s ) ha y m ฺ om ฺ c ฺ Each disk that you put in a diskgroup has a logical name that can m ideyou csindependent u assign. Revision C 6-11 a . ● A volume with two data plexes is mirrored. and remote logins. a b a Plex A plex is a collection of subdisk objects. andeis of its G t @ n b Solaris OS logical device name (c#t#d#). All Rights Reserved. has the following characteristics: ● A plex is a copy of the data in the volume. m u ka e to g ban licens a Subdisk b u le k b a a l y Assubdisk er is a contiguous piece of a physical disk that is used as a f u d ab -tranbuilding block for a volume. ● A volume with five data plexes is mirrored five ways. cron jobs. ku tude a y S (DIDs). Although the GUI for VxVM furnishes useful visual status information.Reviewing the Basic Objects in a Disk Group Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services.

The basic volume creation method (vxassist) lets you specify parameters for your volume. This method automatically creates the subdisks and plexes for you. csmirrored e 1+0 configurations. This striped u ude k a volume contains a single plex composed of sub-volumes rather than t y ( S n subdisks. The disk consumers may be file systems. or applications. you can create two G t @ n b and then use those as components in a larger striped volume. di e this u l a us m a While this sounds complicated.Reviewing the Basic Objects in a Disk Group Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems. Revision C a . k e to the automation involved in the g recommended anvxassist nscommand makes such configurations easy to e create. ab c li b e u l k a erab y l u nsf d b a tra g n n a no ab 6-12 Sun™ Cluster 3. Sun Services. All Rights Reserved. such as Oracle using raw volumes. Inc. For example. Only volumes are given block and character device files. Oracle and/or its affiliatesฺ Volume The volume is the actual logical storage device created and used by all disk consumers. Layered Volume s ) ha y m ฺ om ฺ c ฺ A layered volume is a technique used by VxVM that lets myouucreate de RAID ivolumes. swap space.

Inc. All Rights Reserved. Simple Mirrors s ) ha y m ฺ m arrays ostorage c Figure 6-4 demonstrates subdisks from disks in different ฺ ฺ e m d s i forming the plexes of a mirrored volume. such as hardware RAID with multipathing. With that in mind. ec t Gu @ b den u k ya Stu ( n di e this u l a us m a k evolume to nfsvol g an cens b a c1t0d0 c2t0d0 li b e u l disk01 disk02 k b disk01-01 disk02-01 a a r y ul nsfe d b a tra plex1 plex2 g n n o a n ab Figure 6-4 Forming the Plexes of a Mirrored Volume Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. Revision C 6-13 a . any of the following are acceptable volumes to hold your Sun Cluster application data.Exploring Volume Requirements in the Sun Cluster Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. is that you must mirror across controllers unless there is full redundancy provided by the controller itself. Oracle and/or its affiliatesฺ Exploring Volume Requirements in the Sun Cluster Environment The only requirement for creating a volume in shared storage disk groups for the Sun Cluster environment.

2 Administration Copyright 2009 Sun Microsystems. This is mirrored with a similar configuration to the other array. Inc.Exploring Volume Requirements in the Sun Cluster Environment Mirrored Stripe (Mirror-Stripe) Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Figure 6-5 demonstrates subdisks from disks in the same array striped together. All Rights Reserved. volume nfsvol s ) ha y m ฺ disk01-01 disk02-01 om ฺ c ฺ disk03-01 disk04-01 sm uide c e G plex1 plex2 t @ n b ku tude a y n ( this S i d lu se a am to u k ng ense c1t1d0 c2t1d0 a b c a (disk03) (disk04) i l b e u l k Figure a 6-5 abMirror-Stripe Volume r y l e f bdu trans a g onn a n ab c1t0d0 (disk01) 6-14 c2t0d0 (disk02) Sun™ Cluster 3. Revision C a . Sun Services.

and you could suffer simultaneous failures in each storage array. both mirrored across storage arrays. Oracle and/or its affiliatesฺ Figure 6-6 shows two mirrored sub-volumes. Revision C 6-15 a . n non a b a c1t0d0 disk01 Often. as demonstrated later in the module. For example. Inc. you could lose both disk01 and disk04. Sun Services. All Rights Reserved. striped together to form the final volume.Exploring Volume Requirements in the Sun Cluster Environment Striped Mirrors (Stripe-Mirror) Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. volume nfsvol subvolume s ) ha y c2t0d0 m ฺ disk01-01 disk02-01 om ฺdisk02 c ฺ sm uide c e G plex1 plex2 t @ n b subvolume ku de u a t y n ( this S i d disk03-01 lu sedisk04-01 a am to u k e plex2 ng ensplex1 c1t1d0 c2t1d0 a b disk03 c a disk04 li single striped plex b e u l k rab a y l Figure e 6-6 Stripe-Mirror Volume f u s d ab -tran g Fortunately. this is easy to create. the RAID 1+0 configuration is preferred because there might be less data to resynchronize after disk failure. and your volume will still be available.

2 Administration Copyright 2009 Sun Microsystems. DRLs have no effect whatsoever on resynchronization behavior following disk failure. in the cluster environment). The whole purpose of a sein the m u the cluster is making a everything behave as well as possible after crashes. That is.Exploring Volume Requirements in the Sun Cluster Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. 26 kilobytes (Kbytes) of DRL per 1 Gbyte of n a no ab storage. Revision C a . they might need to be resynchronized in the volume after a Solaris OS crash. The size of the ul DRLs s d n b a that the command chooses depends on the version of VxVM. but a trlogs g n might be. Oracle and/or its affiliatesฺ Dirty Region Logs for Volumes in the Cluster Dirty region log (DRL) is an optional feature you can add on a per-volume basis. That is because on recovery (on another node. s ) ha y m ฺ om ฺ c ฺ m datauimarked de in cofsthe With the DRL. 6-16 Sun™ Cluster 3. For a heavily written volume. performance is better with a DRL on a different disk than the data. just delete the DRL. a mirrored volume must be completely resynchronized after a Solaris OS crash. anda do u t y n ( this S i d luvolume You use a DRL for any large cluster. Regions n b dtoe be resynchronized. Without a DRL. Sun Services. for example. If you grow a volume. For example. VxVM knows exactly which regions e Gnot marked in t @ the DRL bitmap might need to be resynchronized. kunot need the bitmap are known to be clean. An example of this is Web data. VxVM has no way of knowing which bits of data it might have been in the middle of writing when the crash occurred. Location of DRL You can put DRL subdisks on the same disk as the data if the volume will not be heavily written. Inc. o k t ng ense a b a Size of DRL lic b e u l k rab a y are fe small. Use the vxassist command to size them. They are so small that performance on this one disk is still fine. and then use the vxassist command to add it back again so that it is sized properly. The DRL is an extra plex for the volume that does not hold data but rather holds bitmaps of which regions of the data might be dirty. you could dedicate one disk to holding all the DRLs for all the volumes in the group.

t g e storage node you do not need a license for that node. If the major number is not the same on all nodes. Oracle and/or its affiliatesฺ Viewing the Installation and bootdg/rootdg Requirements in the Sun Cluster Environment The rest of this module is dedicated to general procedures for installing and administering VxVM in the cluster. Revision C 6-17 a . Sun Services. then a non-storage t @ n b only that the vxio major number deto the kube added u a t y /etc/name_to_major file. as h VxVM must be installed on all nodes physically connected to shared ) y storage – On non-storage nodes. and more difficult manual ways for meeting the requirements described in the following list: ● The vxio major number must be identical on all nodes – This is the entry in the /etc/name_to_major file for the vxio device driver. It describes both easy automatic ways. you might be using Solaris csmVolume u e Gnode requires software to mirror root storage). All Rights Reserved. Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. the global device infrastructure cannot work on top of VxVM. om ฺ c ฺ e idManager If not (for example.Viewing the Installation and bootdg/rootdg Requirements in the Sun Cluster Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. you can install VxVM ifฺm you will use it to encapsulate and mirror the boot disk. ban licens a b u le k b a a Requirements for bootdg/rootdg l y sfer u d ab -tran g n non You can have an optional bootdg (pointing to an actual arbitrary group a b a name) on VxVM 4. Inc. n ( this S i d luon allsstorage A license is required nodes not attached to Sun e a m u StorEdge A5x00 o If you choose not to install VxVM on a nonka arrays.x and above. You will likely have this only if you want ● ● to encapsulate and mirror the OS disk.

Having multiple paths from the same node to the same storage under control of VxVM DMP is still not supported. These caveats all involve the fact that among the file systems being encapsulated on your boot disk is the /global/. All Rights Reserved. Even if you take steps a tryou g n n to disable DMP. Sun Services. for Hitachi storage ● EMC PowerPath software. You can: 6-18 ● Have no multiple paths at all from a node to the same storage ● Have multiple paths from a node under the control of: ● Sun StorEdge Traffic Manager software ● Sun Dynamic Link Manager.2 (VxVM 4. Oracle and/or its affiliatesฺ There are several cluster-specific issues with encapsulating the boot disk. you shouldeuse u ak ofSthis utility which automates the correct creation tudvolume (with different y ( in on each volume names and minor numbers d hisnode). The reason for these restrictions is that this file system is mounted as a global file system. The requirements are the following: ● Unique volume name across all nodes for the /global/.2 Administration Copyright 2009 Sun Microsystems.devices/node@# file system. Revision C a . The typical Solaris OS /etc/mnttab logic still demands that each device and major/minor combination be unique. Inc.devices/node@# volume s ) ha y m ฺ om ฺ c ฺ csm Guide e t the clvxvm @ n b If you want to encapsulate the boot disk.Viewing the Installation and bootdg/rootdg Requirements in the Sun Cluster Cluster-Specific Issues When Encapsulating the Boot Disk Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.devices/node@# volume ● Unique minor number across all nodes for the /global/.0) versions ul In n s d b a cannot permanently disable the DMP feature. e a us m a k e to g s ban licinenSun DMP Restrictions Cluster 3. This is described later t u l in this module.1 and 5. it automatically re-enables itself each time the system is o a n b a booted.2 a b e u l k rab a y fe of VxVM supported by Sun Cluster 3. for EMC storage Sun™ Cluster 3.

Viewing the Installation and bootdg/rootdg Requirements in the Sun Cluster Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. the installation and configuration order would be: 1. All Rights Reserved. Install Solaris OS and patches 2. Revision C 6-19 a . Install and configure Sun Cluster software 4. Inc. Install and configure any required multipathing software 3. Sun Services. Oracle and/or its affiliatesฺ Installing Supported Multipathing Software The best strategy is to make sure your multipathed storage systems are under control of the multipath solution supported by the cluster before you ever even install Sun Cluster software. To review. Install and configure VxVM s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems.

Sun™ Cluster 3. tofh these utilities have the lud Both e a following characteristics: am to us k g the e They can n install on multiple servers (multiple nodes) if ssoftware a n b e rsh/rcp is enabled between the nodes (or you can a or ssh/scp licit separately b e choose to run on each node). This is many more than the minimal software required to run VxVM in the cluster.0). Sun Services.2 Software Environment This section describes the following: ● Using the installer or installvm utilities to install and initialize VxVM 4.2 Software Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. These g include Web servers and Java environments to run the GUI backan non ● ● ab ends. Oracle and/or its affiliatesฺ Installing VxVM in the Sun Cluster 3. for f u d ab -tran VxVM 5. Revision C a . u l k b a a l y sThey er will install quite of few packages (30 packages for example.1 or 5. installer a y (which can install all of the VERITAS products) ( is S and installvm (which is n i specifically for volume manager). Inc.Installing VxVM in the Sun Cluster 3. All Rights Reserved.0 ● Installing the packages by hand ● Using the clvxvm utility to synchronize vxio major numbers and to encapsulate the root disk ● Addressing VxVM cluster-specific issues if you installed VxVM before the cluster was installed s ) ha y m ฺ om ฺ c ฺ Using the installer or installvm Utilitysm c Guide e @ ent b u The VERITAS software comes with two text-based k tudinstallers.2 Administration Copyright 2009 Sun Microsystems. ● 6-20 They guide you through entering licenses and initializing the software at the end of the installation. even if you choose only the required packages.

Installing VxVM in the Sun Cluster 3.0 packages are distributed on the media as . The installer and installvm utilities extract these correctly. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems.2 Software Environment Installing the Packages by Hand Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ The following are the only VxVM packages required on all the nodes: ● VRTSvxvm ● VRTSvlic The VxVM 4. All Rights Reserved. Revision C 6-21 a .gz files. If you want to install VxVM without using the utilities you copy and extract these archives manually and then install them with pkgadd on each node. Inc. You then have to manually use vxinstall to license and initialize the software. Sun Services.1 and 5.tar.

... Other names may be trademarks of their respective owners..... Symantec.... or configure rsh andause e am to u k Checking ssh communication with theo SunOS 5..S.... All rights reserved. Sun Services.?] (3) 6-22 Sun™ Cluster 3........... The initial screens look similar to the following: Veritas Volume Manager 5........ 1) 2) 3) Required Veritas Volume Manager packages ................. abAdditional s ) ha Logs for installvm are being created in /var/tmp/installvm-DTVRAi.Installing VxVM in the Sun Cluster 3.839 MB required All Veritas Volume Manager packages ... not installed c a i Checking VxVM installation installed b e u l k a erab y l f uinstalled s d VxVM can b be without optional packages to conserve disk space.....7202. Revision C a ... not b on ltheo e .. The Licensed Software and Documentation are deemed to be "commercial computer software" and "commercial computer software documentation" as defined in FAR Sections 12.212 and DFARS Section 227... the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U...10 g ns. and other countries. e n a Checking VxVM installation on vincent .2 Administration Copyright 2009 Sun Microsystems.. If installvm hangs or asks for a login ( is S n i password or hangs.. y m ฺ om ฺ c ฺ Enter the system names separated by spaces on which to install VxVM: vincent m utheo s ide c Initial system check: e G t @ n b e installvm requires that ssh commands used between systems dwithout kuexecute u a t y prompting for passwords or confirmations. n a tra g n n a nopackages are typically installed to simplify future upgrades. Oracle and/or its affiliatesฺ The installvm guides you logically through the tasks.. Inc. All Rights Reserved.863 MB required Storage Foundation Enterprise packages .... stop installvm and run it again with the ssh configured for thoption... ludthes-rsh password free logins...1051 MB required Select the packages to be installed on all systems? [1-3....q.2 Software Environment Example of installvm Screens Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010......0 Installation Program Copyright (c) 2006 Symantec Corporation..

a It installs a script to reminor the bootdg on reboot. Sun Services. Inc. Revision C 6-23 a .devices/node@# file systems on each node.2 Software Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. It @the vxio n b e u then encapsulates your root disk by performing the following tasks: k tud a y ( in aisdisk S group named rootdg. It reboots the first time using -x into non-clustered mode because the second reboot that is performed by VxVM automatically takes you back into the cluster. This is a leftover from an old problem that has been fixed. then clvxvm negotiates a vxio major number and edits the /etc/name_to_major file.Installing VxVM in the Sun Cluster 3. If it has to modify the file. including optional root encapsulation. The clvxvm checks that VxVM itself is already installed and that the VxVM general configuration (vxinstall. All Rights Reserved. Note – VxVM itself (not our utilities) still puts in a nologging option for the root file system when you encapsulate. and ndisk It encapsulates your boot i d h u se t llink. Oracle and/or its affiliatesฺ Using the clvxvm Utility The clvxvm utility automates VxVM cluster-specific configuration. Running clvxvm initialize If you choose this option. which is automated by the installer and installvm utilities) is complete. Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. then it tells you that you need to reboot. if necessary. clvxvm negotiates t major number. creates the bootdg a u o kamvolume t It gives g different names for the volumes containing the e n s /global/. ba licen a b It editslethe vfstab file for this same volume. You must remove the nologging option from /etc/vfstab manually and reboot one more time. ang non ● ● ● ● ● It reboots you into a state where VxVM on your node is fully operational. This allows VxVM encapsulation r y l f bdu trans to properly recognize this is a partition on the OS disk. s ) ha ab y m ฺ om ฺ c Running clvxvm encapsulate ฺ csm Guide e If you choose this option. replacing the DID u k a edevice ab with the original c#t#d#s#.

a b e give the root password to go into single user mode.2 Software The last possibility is that you have a potential Sun Cluster 3. The following subsections describe how to provision your VxVM to deal with the cluster-specific issues mentioned on the previous pages. VxVM cannot figure e t @VxVMenvolume edit that line after the reboot to put inuthe name.2 Administration Copyright 2009 Sun Microsystems. Make sure you do not enter any shared disk storage drives in the bootdg. Reboot the nodes for which you had to repair the minor numbers. b d k tu reboot after that ya aSsecond 4.devices/node@# file system. The result is that the volume name for the /global/. Inc. provides a unique set of minor numbers for each node. uOn those lnodes. use 50 for Node 1. k b a a l y sUse erthe following command to fix the problem: f u d ab -tran # vxdg reminor bootdg 50*nodeid g n non a For example. and so forth. 6-24 Sun™ Cluster 3. All Rights Reserved.devices/node@# file system is different on each node. Oracle and/or its affiliatesฺ Use the following steps to make boot-disk encapsulation work correctly on the cluster. 2. Do not use the default disk name on each node. put back the normal /dev/dsk/c#t#d#s# and /dev/rdsk/c#t#d#s# on the line in the /etc/vfstab file for the /global/. Reboot with boot -x. 100 for Node 2. You should use clvxvm encapsulate to automate all of this. s ) ha y m ฺ om ฺ c ฺ e csmoutGthat uidit must Without making this change.2 software node where the proper version of VxVM was already installed and initialized before the Sun Cluster environment was installed. 6. Configuring a Pre-Existing VxVM for Sun Cluster 3. however this information is given for completeness: 1. This b a 3.devices/node@# volume).2 Software Environment Manually Using vxdiskadm to Encapsulate the OS Disk Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Give a different name for the root disk on each node. They assume you will deal with these issues before you install the Sun Cluster software. Sun Services. his t l a uallsbut one of the nodes because of the m 5.Installing VxVM in the Sun Cluster 3. VxVM arranges ( n which brings you back u into di theecluster. Before you reboot manually. That second reboot fails on a k numbers tofor bootdg (the conflict is really just for g conflict innminor e ba licens the /global/. Revision C a .

Follow the same procedure as e c a i l b Step 5leon page 6-24. ab Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. Revision C 6-25 a . Fixing the /globaldevices Volume Name and Minor Number s ) ha y m ฺ om ฺ c ฺ csm Guide e If this is on a volume manager device: @ ent b u aksameSname 1.2 Software Environment Fixing the vxio Major Number Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services.devices/node@# d h u l e need to manuallyaedit the s /etc/vfstab file. if desired.Installing VxVM in the Sun Cluster 3. do the following: 1. m u a o k eoperation t 2. Performg a reminor on the rootdg if it is using the same set n s a n of b minor numbers as another node. Reencapsulate and remirror the boot disk. Manually edit the /etc/name_to_major file. Inc. 2. 4. you need to have a /globaldevices placeholder file system or at least a correctly sized device ready for cluster installation. Oracle and/or its affiliatesฺ If you need to change the vxio major number to make it agree with the other nodes. Reboot. if it is encapsulated. All Rights Reserved. Rename the volume if it has(y the tud as the /globaldevices in tvolume is of any other node. 3. You might or /global/. u k a erab y l u nsf d b a tra g n n a no If you are still at a point before cluster installation. Unmirror and unencapsulate the root disk.

2 Administration Copyright 2009 Sun Microsystems. even about disk groups not currently imported on this node. Inc. All Rights Reserved. Revision C a . 6-26 Sun™ Cluster 3. populating disk groups. Disks with the invalid state are not yet initialized by VxVM. Sun Services. In this output you can see that there is no disk group information configured on other disks other than the first disk. these are demonstrations of some simple ways of initializing disks. Oracle and/or its affiliatesฺ Creating Shared Disk Groups and Volumes The examples in this section are not trying to exhaustively cover all the possible ways of initializing disks and creating volumes.Creating Shared Disk Groups and Volumes Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. and creating the mirrored configurations described earlier in the module. Rather. Listing Available Disks s ) ha The following command lists all of the disks visible to VxVM: y m ฺ om ฺ c ฺ ide csm Ginvalid u e @ ent invalid b u invalid k tud a y invalid n ( this S i d invalid lu se a invalid am to u k invalid g e n s invalid ba licen a invalid b e u l k invalid a erab y l invalid f u ns d b invalid a tra g n invalid n o a n b invalid a vincent:/# vxdisk -o alldgs list DEVICE TYPE DISK c0t0d0s2 auto:sliced rootdg_1 c0t1d0s2 auto:none c1t0d0s2 auto:none c1t1d0s2 auto:none c1t2d0s2 auto:none c1t3d0s2 auto:none c1t8d0s2 auto:none c1t9d0s2 auto:none c1t10d0s2 auto:none c1t11d0s2 auto:none c2t0d0s2 auto:none c2t1d0s2 auto:none c2t2d0s2 auto:none c2t3d0s2 auto:none c2t8d0s2 auto:none c2t9d0s2 auto:none c2t10d0s2 auto:none - GROUP rootdg - STATUS online online online online online online online online online online online online online online online online online invalid invalid The command vxdisk list with the option -o alldgs scans every disk to see if there is any disk group information.

vincent ku tude a y n ( this S i d u contains lgroup e the name of the node on which it Note – The ID of a disk a s m u was created. # vxdiskadd c1t0d0 c1t1d0 c2t0d0 c2t1d0 .22. Sun Services.26.cds 1152725899.vincent t @ n b nfsdg enabled.Creating Shared Disk Groups and Volumes Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. c a b u le k b # vxdisk list a a y TYPEfer DEVICE ul DISK GROUP STATUS s d n b c0t0d0s2 rootdg_1 rootdg online a traauto:sliced g n c0t1d0s2 auto:none online invalid n o a n b auto:cdsdisk nfs1 nfsdg online a c1t0d0s2 The following command shows which groups are imported: c1t1d0s2 c1t2d0s2 c1t3d0s2 c1t8d0s2 c1t9d0s2 c1t10d0s2 c1t11d0s2 c2t0d0s2 c2t1d0s2 c2t2d0s2 c2t3d0s2 c2t8d0s2 c2t9d0s2 c2t10d0s2 c2t11d0s2 auto:cdsdisk auto:none auto:none auto:none auto:none auto:none auto:none auto:cdsdisk auto:cdsdisk auto:none auto:none auto:none auto:none auto:none auto:none nfs3 nfs2 nfs4 - nfsdg nfsdg nfsdg - online online online online online online online online online online online online online online online Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. Later. Inc. It guides you through initializing disks and putting them in existing or new groups. Revision C invalid invalid invalid invalid invalid invalid invalid invalid invalid invalid invalid invalid 6-27 a . Verifying Disk Groups Imported on a Node s ) ha y m ฺ om ฺ # vxdg list c ฺ NAME STATE ID sm uide c e G rootdg enabled 1152724944. kayou might to see the same disk group with the same ID g e imported a b onnalidifferent ens node. Oracle and/or its affiliatesฺ Initializing Disks and Putting Them into a New Disk Group One of the simplest ways to do this is using the dialogue presented by the vxdiskadd command.

Inc. All Rights Reserved. Sun Services. Revision C a . Oracle and/or its affiliatesฺ The following shows the status of an entire group where no volumes have been created yet: # vxprint -g nfsdg TY NAME ASSOC dg nfsdg nfsdg KSTATE - LENGTH - PLOFFS - STATE - TUTIL0 - PUTIL0 - dm dm dm dm - 71061376 71061376 71061376 71061376 - - - - nfs1 nfs2 nfs3 nfs4 c1t0d0s2 c2t0d0s2 c1t1d0s2 c2t1d0s2 s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 6-28 Sun™ Cluster 3.Creating Shared Disk Groups and Volumes Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems.

dg nfsdg default default 30000 1152725899. All Rights Reserved. Inc. Sun Services.vincent dm dm dm dm c1t0d0s2 c2t0d0s2 c1t1d0s2 c2t1d0s2 auto auto auto auto 65536 65536 65536 65536 71061376 71061376 71061376 71061376 - ENABLED ENABLED ACTIVE ACTIVE 204800 204800 SELECT STRIPE nfs1 nfs2 nfs3 nfs4 v nfsvol pl nfsvol-01 nfsvol Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems.26. Revision C fsgen 2/128 RW 6-29 a . # vxassist -g nfsdg make nfsvol 100m layout=mirror nfs1 nfs2 # vxprint -ht -g nfsdg // with -ht. vxprint prints out a long legend. with a subdisk from a disk in one controller mirrored with one from another controller. which is omitted for brevity. .Creating Shared Disk Groups and Volumes Building a Simple Mirrored Volume Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Each plex is striped between n a n two disks in the same array so mirroring is still across controllers. Oracle and/or its affiliatesฺ The following example shows building a simple mirror.The following example rebuilds the mirror. . ab dm dm dm dm nfs1 nfs2 nfs3 nfs4 c1t0d0s2 c2t0d0s2 c1t1d0s2 c2t1d0s2 auto auto auto auto 65536 65536 65536 65536 71061376 71061376 71061376 71061376 - # vxassist -g nfsdg remove volume nfsvol # vxassist -g nfsdg make nfsvol 100m layout=mirror-stripe mirror=ctlr # vxprint -ht -g nfsdg . dg nfsdg default default 30000 1152725899.26. .vincent s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u d v nfsvol ENABLED ACTIVE y204800 fsgen ak StuSELECT ( pl nfsvol-01 nfsvol ENABLED ACTIVE 204800 CONCAT RW n i is d h t u sd nfs1-01 nfsvol-01 nfs1 0l c1t0d0 ENA e204800 0 aACTIVE s m u pl nfsvol-02 nfsvol ENABLED 204800 CONCAT RW ka e0 to g sd nfs2-01 nfsvol-02 nnfs2 204800 0 c2t0d0 ENA s a n b e a lic b e u l k raaMirrored b a Building Striped Volume (RAID 0+1) y l e f u s bd tran a g on.

ub ENABLED ACTIVE 204800 SELECT nfsvol-0 fsgen e l k b pl nfsvol-03 ynfsvol RW a era ENABLED ACTIVE 204800 STRIPE 2/128 l f u sv nfsvol-S01 nfsvol-03 nfsvol-L01 1 102400 0/0 2/2 ENA s bd trnfsvol-03 an a sv nfsvol-S02 nfsvol-L02 1 102400 1/0 2/2 ENA g n n no abva nfsvol-L01 ENABLED ACTIVE 102400 SELECT fsgen . All Rights Reserved.2 Administration Copyright 2009 Sun Microsystems. As mentioned earlier in this module.vincent b u ak Stud y ( dm nfs1 c1t0d0s2 auto 65536 in 71061376 is -d h t dm nfs2 c2t0d0s2 auto 65536 71061376 u e al dm nfs3 c1t1d0s2 auto am 65536 us 71061376 k 65536 to 71061376 dm nfs4 c2t1d0s2 auto g e ban licens a v nfsvol . s ) ha # vxassist -g # vxassist -g # vxprint -ht . Inc. this layout is often preferred for faster resynchronization on disk failure and greater resiliency. Oracle and/or its affiliatesฺ Creating Shared Disk Groups and Volumes sd sd pl sd sd nfs1-01 nfs3-01 nfsvol-02 nfs2-01 nfs4-01 nfsvol-01 nfsvol-01 nfsvol nfsvol-02 nfsvol-02 nfs1 nfs3 ENABLED nfs2 nfs4 0 0 ACTIVE 0 0 102400 102400 204800 102400 102400 0/0 1/0 STRIPE 0/0 1/0 c1t0d0 c1t1d0 2/128 c2t0d0 c2t1d0 ENA ENA RW ENA ENA Building a Striped Mirrored Volume (RAID 1+0) The final example rebuilds the volume as a stripe of two mirrors.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. 6-30 Sun™ Cluster 3.26. Sun Services. dg nfsdg nfsdg remove volume nfsvol nfsdg make nfsvol 100m layout=stripe-mirror mirror=ctlr -g nfsdg pl sd pl sd nfsvol-P01 nfs1-02 nfsvol-P02 nfs2-02 nfsvol-L01 nfsvol-P01 nfsvol-L01 nfsvol-P02 ENABLED nfs1 ENABLED nfs2 ACTIVE 0 ACTIVE 0 102400 102400 102400 102400 CONCAT 0 CONCAT 0 c1t0d0 c2t0d0 v pl sd pl sd nfsvol-L02 nfsvol-P03 nfs3-02 nfsvol-P04 nfs4-02 nfsvol-L02 nfsvol-P03 nfsvol-L02 nfsvol-P04 ENABLED ENABLED nfs3 ENABLED nfs4 ACTIVE ACTIVE 0 ACTIVE 0 102400 102400 102400 102400 102400 SELECT CONCAT 0 CONCAT 0 fsgen RW c1t1d0 ENA RW c2t1d0 ENA y m ฺ om ฺ c ฺ csm Guide e @ ent default default 30000 1152725899. . Revision C RW ENA RW ENA a .

These disks can y ( S in this d NOHOTUSE u al use m Disks marked ka withe the to NOHOTUSE flag are excluded from g consideration ban licenforshot relocation. hot relocation. VxVM looks for space to substitute for the broken half of the mirror. Revision C 6-31 a . disks can be marked with one of the two following flags. Oracle and/or its affiliatesฺ Examining Hot Relocation Hot relocation is controlled by the vxrelocd daemon.Examining Hot Relocation Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. When a disk that is a component of a mirrored volume breaks. but not both: ● ● The flag settings are visible in the output of the vxdisk list and vxprint commands. VxVM can concatenate many new subdisks from other disks in the diskgroup to recreate the broken plex. All Rights Reserved. Inc. a b following u le example. disk nfs2 is set as a preferred disk for hot k In the b a a l y relocation. If one large disk breaks. The SPARE and NOHOTUSE Flags s ) ha y m ฺ om ฺ c ฺ csm Guide SPARE e t disks to use for @ n b Disks marked with the SPARE flag are the preferred e u k be used astill tudto build normal volumes. er and disk nfs1 is excluded from hot relocation usage: f u s d ab -tran# vxedit -g nfsdg set spare=on nfs2 g n non a # vxedit -g nfsdg set nohotuse=on nfs1 b a Under normal hot relocation. Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. which is started by default after you initialize the volume manager. It never uses any disk space from disks in the surviving plex. Sun Services.

cds 1152725899. you must manually register the disk group using either the clsetup utility or the cldevicegroup create command for the disk group to be managed by the cluster.2 Administration Copyright 2009 Sun Microsystems. Sun Services. Until a VxVM disk group is registered. Inc. it is referred to as a device group.Device Group Status --Device Group Name ----------------- 6-32 Primary ------- Secondary --------- Status ------ Sun™ Cluster 3. but g e The following n example ns cldg is just the short form of ba to lthe e not yet known cluster. Revision C a . u l k a erab y l # vxdg list u nsf d b NAME a ID ra tSTATE g n n rootdg 1152724944. All Rights Reserved. Such configuration changes include n b e u d k adding or removing volumes and changing u owner.26.vincent # cldevicegroup show # cldg status Cluster Device Groups === --. permissions of existing volumes. s ) ha y m ฺ om ฺ c ฺ If you create a new volume or delete a volume in a VxVM e that m diskuigroup d s c is already registered with the cluster. m u ka shows toa disk group that is known to VxVM.Registering VxVM Disk Groups Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. the cluster does not detect it. When a VxVM disk group is registered in the Sun Cluster software environment. c a i b e cldevicegroup.vincent a no enabled abnfsdg enabled. While you can build volumes and perform all normal VxVM administration tasks.in Synchronization after volumes are d h t u l the sglobal created or deleted ensuresathat e namespace is in the correct state. or (ya isthe Stgroup. Oracle and/or its affiliatesฺ Registering VxVM Disk Groups After you create a new VxVM disk group and volumes. you are not able to use the volume manager devices in the cluster.22. you must synchronize e t Gthe disk @ device group by using clsetup.

Inc. From the main menus. it only works if preferenced is also set to true.Registering VxVM Disk Groups Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. selecting Menu a menu-driven a5b(Device r y l e f Option Groups and Volumes) gives you the following submenu: u ns d b a aDevice-trGroups *** Menu *** g n n a no ● The preferenced=true/false affects whether the nodelist indicates an order of failover preference. ● Please select from one of the following options: 1) 2) 3) 4) 5) 6) 7) 8) Register a VxVM disk group as a device group Synchronize volume information for a VxVM device group Unregister a VxVM device group Add a node to a VxVM device group Remove a node from a VxVM device group Change key properties of a device group Set a VxVM disk group as a local disk group Reset a local disk group to a VxVM disk group ?) Help q) Return to the Main Menu Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. s ) ha ab y m ฺ om ฺ c ฺ e idnode csa m u The failback=true/false affects whether preferred takes e G t @ n b back its device group when it joins e The default value is u the cluster. Sun Services. Revision C 6-33 a . Oracle and/or its affiliatesฺ Using the cldg create Command to Register Disk Groups The following command registers a VxVM disk group as a cluster device group. d k u a false. # cldg create -t vxvm -n vincent. t y ( S n di e this u l a us m a Using the clsetup Command to Register Disk Groups k e to g an cens b a li can guide you through the previous options in a Thebclsetup utility e u l k environment. If it is true.theo -p preferenced=true \ -p failback=false nfsdg This command uses the following parameters: ● The -n option (node list) should list all the nodes and only the nodes physically connected to the disk group’s disks. On a two node cluster this option is only meaningful if failback is true. All Rights Reserved. You must type this command on the node currently importing the group.

Oracle and/or its affiliatesฺ Use the cldg show command to show the properties of a device group (or all device groups). Sun Services. This causes the cluster to treat the device group as failed.Registering VxVM Disk Groups Viewing and Controlling Registered Device Groups Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems. The numsecondaries=# parameter is used on the cldg command line to allow more than one secondary. use the following command syntax to control disk group ownership: # cldg switch -n node_to_switch_to nfsdg 6-34 Sun™ Cluster 3. only one node shows up as secondary. Inc. even if there are more than two nodes in the node list ab -trNote g n non for a device group. Instead. All Rights Reserved. Use cldg status to see the status. never use the vxdg import and vxdg deport commands to control ownership of the disk group. the secondary becomes primary and another (spare) node becomes secondary. theo true 1 nfsdg s ) ha y m ฺ om ฺ c ฺ csm Guide # cldg status e @ ent b u Cluster Device Groups === ak Stud y ( in this d u --. If the primary a b a fails. When VxVM disk groups are registered as Sun Cluster software device groups and have the status Online. # cldg show nfsdg Device Groups === Device Group Name: nfsdg Type: failback: Node List: preferenced: numsecondaries: diskgroup name: VxVM false vincent.Device Group Status --al use m ka Secondary to g Device Group Name Primary Status e n ens--------a b ---------------------------c a i l b nfsdg theo Online u vincent le k b a a l y sfer u d an – By default. Revision C a .

To put the device group offline all of the VxVM volumes must be unused (unmounted or otherwise not open). It is just a warning. You then issue the following command. g an non # cldg sync nfsdg ab Putting a Device Group Offline and Back Online You can take a VxVM device group out of service. for emergency repairs. For ( S example.Managing VxVM Device Groups Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. it must be resynchronized any time a new volume is created or deleted in the disk group. al use m o ka -p tfailback=false # cldg set -p preferenced=false nfsdg g e n s ba licen a b – If you u le do this command on the node that is not primary for the k Note b a a l y device er group. Inc. All Rights Reserved. This instructs the Sun Cluster software to scan the disk group and build and remove the appropriate global device files: s ) ha y m ฺ om ฺ c ฺ Making Other Changes to Device Groups csm Guide e @ ent b u The properties of existing VxVM y device ak groups tudcan also be changed. Oracle and/or its affiliatesฺ Managing VxVM Device Groups The cldevicegroup (cldg) command can be used to perform clusterspecific changes to VxVM device groups. as far as the cluster is concerned. Revision C 6-35 a . the failback and preferenced in this properties of a group can be d u modified after it is registered. you may get a warning that makes it seem like the f u s d ab -trancommand failed. Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. and the command succeeds. from any node: # cldg offline nfsdg You will rarely take a device group out of service because almost all repairs can still be done while the device group is in service. Resynchronizing Device Groups After a VxVM device group is registered with the cluster.

2 Administration Copyright 2009 Sun Microsystems. you can choose exactly which node will become the device group primary. Sun Services. One or the other will do. Revision C a . Inc. # cldg online -n theo nfsdg Note – You do not need to perform both commands.Managing VxVM Device Groups Type the following command to put the device group online on the first node in the list: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 6-36 Sun™ Cluster 3. Oracle and/or its affiliatesฺ # cldg online nfsdg Alternatively.

which must be physically connected to the storage UFS and VxFS are two file system types that can be used as either global or failover file systems. s ) ha y m ฺ m ฺco fileesystems ฺ Note – There is a convention that the mount point form global be d s i c is just somewhere underneath the /global directory. and it should be identical on all nodes who may run services which access the file system (can only be nodes physically connected to the storage): /dev/vx/dsk/nfsdg/nfsvol /dev/vx/rdsk/nfsdg/nfsvol /localnfs ufs 2 no - Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. and it should be identical on all nodes (including nodes not physically connected to the storage): /dev/vx/dsk/nfsdg/nfsvol /dev/vx/rdsk/nfsdg/nfsvol /global/nfs ufs 2 yes global A local failover file system entry looks like the following. the distinction between a global and a failover file system is made in the /etc/vfstab mount-at-boot and options columns. Inc. All Rights Reserved.2 supports running data services on the following categories of file systems: ● Global file systems – Accessible to all cluster nodes simultaneously. These examples and the exercises assume you are using UFS. Oracle and/or its affiliatesฺ Using Global and Failover File Systems on VxVM Volumes Sun Cluster 3.Using Global and Failover File Systems on VxVM Volumes Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.eThis ua convention. Revision C 6-37 a . Use newfs as normal to create a UFS file the u k a system aforb the cluster on a volume in a registered disk group: r y l e f bdu tran#snewfs /dev/vx/rdsk/nfsdg/nfsvol a g onn a n ab Mounting File Systems For UFS or VxFS. G t @ b den u k ya Stu ( n di e this u l a us Creating File Systems m a k e to g The distinction ban libetween ens a global and a failover file system is not made at c a btime oflefile system creation. Sun Services. even those not physically connected to the storage ● Failover file systems – Mounted only on the node running the failover data service. A global file system entry should look similar to the following.

All Rights Reserved. Oracle and/or its affiliatesฺ Note – The logging option is the default for all OS versions supported with Sun Cluster 3.Using Global and Failover File Systems on VxVM Volumes Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems. Sun Services. Inc. You do not need to specify it explicitly. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 6-38 Sun™ Cluster 3. Revision C a .2.

Mirroring the Boot Disk With VxVM Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. m ka e to g nsdrive correctly as shown. Initialize and add another local drive to bootdg. Verify that the system has been instructed to use these device aliases on boot: # eeprom|grep use-nvramrc use-nvramrc?=true Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. if this has not been done yet. Make sure that you do not make the disk a CDS disk (choose sliced). the second drive can also If you mirror root ban the e c a li That is because correct restricted subdisks are laid bunencapsulated.4000/scsi@3/disk@8. Oracle and/or its affiliatesฺ Mirroring the Boot Disk With VxVM If you have encapsulated your boot drive with VxVM.4000/scsi@3/disk@0. This drive must be at least as big as the boot drive. or the vxdiskadm Mirror on a c ฺ ฺ e m disk option to mirror all the volumes on the drive. if you ever need lto u a use able to. Inc.0:a devalias vx-rootmir /pci@1f. Sun Services. csboot G uid e @ ent b u Caution – Do not mirror each volume This ud withbe vxassist. The previous command also creates aliases for both the original root partition and the new mirror. You need to manually set your bootdevice variable so you can boot off both the original and the mirror. 3.0:a # eeprom boot-device="vx-rootdg_1 vx-rootmir" 4. without any reboot. akseparately tcannot y ( S leaves you in a state that the mirrored drive unencapsulated. for example. These can later be turned into regular Solaris r y l e f bdu tranOSs partitions because they are on cylinder boundaries. s ) ha y m ฺ omvolumes 2. a ang non # vxmirror -g bootdg rootdg_1 rootmir # vxdiskadd c0t8d0 ab . All Rights Reserved. mirroring the boot drive is a simple procedure that can be undertaken any time while the cluster is online. be e u l k a down onto ab the new drive. in this the second disk you will not be dunencapsulate That is. . This is done as part of the vxdiskadd dialogue. Revision C 6-39 a . Give the drive the VxVM logical name rootmir. To mirror the boot drive. # eeprom|grep vx devalias vx-rootdg_1 /pci@1f. complete the following steps: 1. Use the vxmirror command.

Inc.2 Administration Copyright 2009 Sun Microsystems. All Rights Reserved. Revision C a . you complete the following tasks: ● Task 1 – Selecting Disk Drives ● Task 2 – Using pkgadd to Install and Initialize VxVM Software (on All Storage Nodes) ● Task 3 – Using clvxvm to Verify the vxio Major Number ● Task 4 – Adding vxio on Any Non-Storage Node on Which You Have Not Installed VxVM ● Task 5 – Rebooting All Nodes s ) ha y m ฺ Task 7– Configuring webdg om ฺ c ฺ Task 8– Registering Demonstration Disk Groups csm Guide e @ ent Task 9– Creating a Global nfs File System b u akSystem tud Task 10 – Creating a Global web File y ( S inSystems d his t Task 11 – Testing Global File u l e a us m a Task 12 – Managing Disk k e toDevice Groups g n en–sViewing and Managing VxVM Device Groups Task 13 ba(Optional) a lic Manager Using SuneCluster b u l k rab a y ul nsfe d b a tra g n n a no ● Task 6 – Configuring nfsdg ● ● ● ● ● ● ● ab 6-40 Sun™ Cluster 3. Oracle and/or its affiliatesฺ In this exercise.Exercise: Configuring Volume Management Exercise: Configuring Volume Management Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services.

Node 1 Boot Disk Boot Disk bootdg bootdg Node 2 c0 s ) ha y m ฺ ccccc om ฺ c1 c2 c2 c c c1 ฺ (Encapsulated (bootdg->rootdg): Optional) csm Guide e t A A Bb@ B n e u ak Stud y ( Disk Group Quorum Disk n this and Volumesudi al use m Array A Disk ka e to Array B Disk g an cens b a Array B Disk bArray A Disk le li u k a eArray aAb r y Array B l f u s d ab -tran g n non a b a c0 nfsdg nfsvol webdg webvol Figure 6-7 Configuring Volume Management Note – During this exercise. when you see italicized names. Each data service disk group contains a single mirrored volume. All Rights Reserved. Revision C 6-41 a . enclosure_name. substitute the names appropriate for your cluster. Oracle and/or its affiliatesฺ Record the location of the VxVM software you will install during this exercise. Inc. Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. node1. The setup is shown in Figure 6-7. Encapsulating the boot disk is an optional exercise at the end.Exercise: Configuring Volume Management Preparation Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. such as IPaddress. Location: _____________________________ During this exercise. you create two data service disk groups. or clustername embedded in a command string. Sun Services.

2 Administration Copyright 2009 Sun Microsystems. It is perfectly acceptable to use a quorum disk as one of your disk group drives as well. if possible) bdu trafrom a g onn a n ab webdg disk 1 webdg disk 2 (different controller from disk 2. y m ฺ om ฺ c ฺ Table 6-1 csm Guide e @ entDID # (d#) c#t#d# b u ak Stud y ( Boot disk in this d u al use Quorum disk(s) m ka e to g an cens b a nfsdgbdisk 1 li e u l k anfsdg edisk a2b(different controller r y l f ns m disk 1. Sun Services. You may have one or more disks configured as quorum devices (or you may have a quorum server quorum device). clq status. . Perform the following step to select disk drives: s ) ha Use the cldev list -v. if possible) 6-42 Sun™ Cluster 3. you must have a clear picture of the disk drives that are used throughout this course. you must identify the boot disk and two disks in each storage array for use in the two demonstration disk groups. nfsdg and webdg. Oracle and/or its affiliatesฺ Before proceeding with this exercise. All Rights Reserved. and df -h commands to identify and record the logical addresses of disks for use during this exercise. Inc. Revision C a . In this exercise.Exercise: Configuring Volume Management Task 1 – Selecting Disk Drives Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

gz tar xf a o k t # gzcat | tar xf ngVRTSvmman. you add VxVM packages manually. If you have a non-storage node (Pair +1 cluster).tar. a a r y ul nsfe# pkgadd -d /var/tmp VRTSvlic VRTSvxvm VRTSvmman d b a tra 3. ab Spool and expand the VxVM packages. This is completely supported. you do not need to install VxVM on that node. Inc.gz xf al use|| tar m # gzcat VRTSvxvm. g n n a no 1.tar. rather than using installer or installvm.0 patches for MP3 and RP1 in that order.tar. Enter the license information as directed by your instructor b. All Rights Reserved. just to save time.tar. the MP3 patch is 122058-11 and the RP1 patch is 139352-02. c.gz /var/tmp b u ak Stud y ( # cd /var/tmp in this d u # gzcat VRTSvlic. Oracle and/or its affiliatesฺ Task 2 – Using pkgadd to Install and Initialize VxVM Software (on All Storage Nodes) In this lab. Run vxinstall to initialize VxVM: a.tar. 122058-11 139352-02 4. b Add lic VxVM packages. Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. You need to install VxVM only on nodes connected to the shared storage.tar.gz se a n b e a the new 2. At the time of writing the course. Sun Services. # cd location_of_vxvm_patches_for_course # patchadd -M .Exercise: Configuring Volume Management Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Answer yes to all questions e u l k (including b conflicts in directories). Revision C 6-43 a .gz e @ ent VRTSvmman. Perform the following steps on all cluster nodes that are connected to shared storage: s ) ha y m ฺ # cd vxvm_sw_dir/volume_manager/pkgs om ฺ c ฺ cs\m Guide # cp VRTSvlic.gz VRTSvxvm. Answer No when asked about enclosure-based naming. Add the VxVM 5. Answer No when asked about a default disk group.

s ) ha y m ฺ om ฺ c ฺ e idon Task 4 – Adding vxio on Any Non-Storage csmNode u e G t @ Which You Have Not Installed VxVM n b ku tude a y ( is S n which i If you have a non-storage nodedon hyou have not installed VxVM. If it is not. Sun Services. All Rights Reserved. Please manually verify that the vxio major number in /etc/name_to_major is the same on all cluster nodes. A non-storage node with the manually created or fixed vxio entry absolutely requires a reboot here. Be careful that you don’t conflict with any numbers already in a file (you can just choose a major number higher than that in any of the files).2 Administration Copyright 2009 Sun Microsystems. edit the file so it is the same on all nodes. Inc. picking a number higher than currently exists on any node. this fails when you have ZFS root. Oracle and/or its affiliatesฺ Run the following commands on all nodes on which you have installed VxVM: # clvxvm initialize Note – At the time of writing this course. Revision C a .Exercise: Configuring Volume Management Task 3 – Using clvxvm to Verify the vxio Major Number Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. In addition Sun Cluster manager will fail to operate properly unless you reboot here. In the real production environment. Note – Technically the storage nodes would proceed correctly even without a reboot. you would always do this one node at a time to maintain the HA of any cluster services that might already be configured. you will have to change the vxio major number on all n ab nodes. If it is. Task 5 – Rebooting All Nodes Reboot all your nodes. t u l e edit /etc/name_to_majoraand addsa line at the bottom containing the uon the storage nodes: am as tused same vxio major number o k ng ense a vxio same_major_number_as_other_nodes b a lic b e u l k rab a y e ul Note –f Be careful and assure that on the non-storage node the major s d n b a a is not already used by some other device driver on another line in trnumber g n n o a the file. 6-44 Sun™ Cluster 3.

q.n.?] (default: y) y Add disks as spare disks for nfsdg? [y. . On Node 1.q.?] (default: y) y . f u ns d b careful when answering these questions because you can destroy a tra be g n n critical data or cluster node access information. verify that the new nfsdg disk group is globally linked.n.n.q.?] (default: y) y Create the disk group as a CDS disk group? [y.devices/node@1/dev/vx/dsk/nfsdg 4.n.q.q. abtraining y l previous class. Sun Services.q.q. Oracle and/or its affiliatesฺ Perform the following steps to configure two demonstration disk groups.list. create a 500-Mbyte mirrored volume in the nfsdg disk group. In your work environment.q.n. Continue with operation? [y. Do you want to use the default layout for all disks being initialized? [y. Inc. All Rights Reserved.none.Exercise: Configuring Volume Management Task 6 – Configuring nfsdg Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. # vxassist -g nfsdg make nfsvol 500m layout=mirror Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. you should b e u l k If you are prompted about clearing old disk usage status from a a replyerno.?] (default: n) n .q. with each containing a single mirrored volume: 1. 6 2008 On Node 1.n. create the nfsdg disk group with your previously selected logical disk addresses.?] (default: none) nfsdg Create a new group named nfsdg? [y. On Node 1.n.n. o a n b a 2. # ls -l /dev/vx/dsk/nfsdg lrwxrwxrwx 1 root root 12 Dec /dev/vx/dsk/nfsdg -> /global/.?] (default: n) y Add site tag to disks? [y. Verify the status of the disk group and the names and ownership of the disks in the nfsdg disk group.?] (default: y) y s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g an cens b a Caution – If you li are prompted about encapsulating the disk. Revision C 6-45 a .?] (default: y) y Use default disk names for these disks? [y.?] (default: n) n Exclude disks from hot-relocation use? [y. you should reply yes. # vxdiskadd c#t#d# c#t#d# Which disk group [<group>. # vxdg list # vxdisk list 3.

Notice that Node 2 does 3.?] @ ent b (default: none) webdg u ak volume tud in the webdg disk 2. u e al make s m u # vxassist -g webdg webvol 500m layout=mirror a o k t ng ecommand se on both nodes. Use the a vxprint n b asee the disk not lic groups created and still imported on Node 1. create a 500-Mbyte(y mirroredS in this d group. Sun Services. 6.none. Revision C a . Issue the following command on the node that currently owns the disk group: # newfs /dev/vx/rdsk/nfsdg/nfsvol It should fail with a no such device or address error. b e u l k rab a y ul nsfe d b a tra g n n a no ab 6-46 Sun™ Cluster 3. Inc.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. On Node 1. s ) ha On Node 1. Notice that Node 2 does not see the disk groups created and still imported on Node 1.list. Answers in the dialog can be similar to those in the previous task. Use the vxprint command on both nodes. Oracle and/or its affiliatesฺ Exercise: Configuring Volume Management 5.2 Administration Copyright 2009 Sun Microsystems. Task 7– Configuring webdg 1. create the webdg disk group with your previously selected logical disk addresses. y m ฺ om ฺ c ฺ # vxdiskadd c#t#d# c#t#d# csm Guide e Which disk group [<group>. All Rights Reserved.q.

From either node. Answer the clsetup as follows: s m u a o k VxVM tdisk group you want to register? webdg Name ofgthe e n s a want Dobyou ento configure a preferred ordering (yes/no) c a i l b [yes]? e yes u l k b ra both nodes attached to all disks in this group ya feAre l u s bd tran (yes/no) [yes]? yes a ang non ab Note – Read the previous question carefully. On Node 1. Oracle and/or its affiliatesฺ Perform the following steps to register the two new disk groups with the Sun Cluster framework software: 1. use the cldg create utility to manually register the nfsdg disk group. Select option 5. # cldg create -t vxvm -n node1. verify the status of the disk groups. Sun Services. On a cluster with more than two nodes. ) y m ฺ # clsetup om ฺ c ฺ 3. From the main menu. Inc. Device groups and volumes.Exercise: Configuring Volume Management Task 8– Registering Demonstration Disk Groups Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. you will be asked if all nodes are connected to the disks. Register a VxVM doption ku select b. Which node is the preferred primary for this device group? node2 Enable "failback" for this disk device group (yes/no) [no]? yes 5. All Rights Reserved. s i i d h t luquestions e a 4. t @ n b e 1.node2 \ -p preferenced=true -p failback=true nfsdg Note – Put the local node (Node 1) first in the node list. # cldg status Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. On Node 1. Revision C 6-47 a . From the device groupsymenu. complete the followingsm ide c steps: u e G a. as h 2. In a Pair +1 configuration. u a t S disk group as a device n (group. for example. you need to answer no and respond when you are asked which nodes are connected to the disks. use the clsetup utility to register the webdg disk group.

including any non-storage nodes. tud y ( 3. On Node 1. On Node 2. 4. create a file system on webvol in the webdg disk group. On all nodes. Verify that thekfile g n ense a # mount b a /global/nfs lic #bls e u l k lost+found a ab r y l e f bdu trans a g Task n. Inc. s ) ha y m ฺ om ฺ c ฺ csm Guide e Note – Do not use the line continuation character @ (\)einntthe vfstab file. create a global mount point for the new file system. create a file system on nfsvol in the nfsdg disk group.10 – Creating a Global web File System n o a n ab /dev/vx/dsk/nfsdg/nfsvol /dev/vx/rdsk/nfsdg/nfsvol \ /global/nfs ufs 2 yes global Perform the following steps on Node 1 to create and mount a demonstration file system on the webdg disk group volume: 1. add a mount entry in the /etc/vfstab file for the new file system with the global mount option. On Node 1.Exercise: Configuring Volume Management Task 9– Creating a Global nfs File System Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. b u ak fileSsystem. On all nodes. Revision C a . # newfs /dev/vx/rdsk/nfsdg/nfsvol 2. create a global mount point for the new file system. # mkdir /global/web 6-48 Sun™ Cluster 3. # newfs /dev/vx/rdsk/webdg/webvol 2. # mkdir /global/nfs On all nodes. Oracle and/or its affiliatesฺ Perform the following steps on Node 1 to create and mount a demonstration file system on the nfsdg disk group volume: 1. All Rights Reserved.2 Administration Copyright 2009 Sun Microsystems. mount the /global/nfs in this d u # mount /global/nfs al use m a systemtois mounted and available on all nodes. Sun Services.

4. Revision C 6-49 a . 5. add a mount entry in the /etc/vfstab file for the new file system with the global mount option. All Rights Reserved. Verify that the file system is mounted and available on all nodes. Try unmounting and mounting /global/nfs from all nodes. Sun Services. In most cases. Common tasks are: ● Adding volumes to a device group ● Removing volumes from a device group Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems.kmove tothe /global/nfs file system. mount the /global/web file system. Inc. try to unmount the /global/nfs file system (umount b On Node 2. Mount the /global/nfs file system again on Node 1. e u l k a e/global/nfs). they should not be managed using VERITAS commands. Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. move out of the /global/nfs file system (cd /) and try a g onto unmount it again on Node 1. On Node 2. On Node 2. Oracle and/or its affiliatesฺ /dev/vx/dsk/webdg/webvol /dev/vx/rdsk/webdg/webvol \ /global/web ufs 2 yes global Note – Do not use the line continuation character (\) in the vfstab file. Task 12 – Managing Disk Device Groups In the Sun Cluster 3. ab You should get an error that the file system is busy. Some administrative tasks are accomplished using a combination of Sun Cluster and VERITAS commands.2 software environment. r y l f u s bd tran3.2 software environment: available file systems in lthe t u e a us am into 1. VERITAS disk groups become cluster device groups when they are registered. # mount # ls /global/web lost+found s ) ha y m ฺ om ฺ c ฺ csm Guide e Task 11 – Testing Global File Systems @ ent b u ak theSgeneral tud behavior of globally y ( Perform the following steps to confirm in Cluster dSun his3. g e # cdan ns b /global/nfs e c a li1.Exercise: Configuring Volume Management 3. On all nodes. n a n b a 4. # mount /global/web 5. On Node 1.

d b a a r t ang non 2. # cldg status 2. # vxprint -g nfsdg testvol 4. create a 100-Mbyte test volume in the nfsdg disk group. nsUnmount any file systems that are related to the volume.Exercise: Configuring Volume Management Adding a Volume to a Disk Device Group Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. s ) ha Prove that the new volume can not be used in the cluster (you will get an error message): y m ฺ # newfs /dev/vx/rdsk/nfsdg/testvol om ฺ c ฺ 5. # vxassist -g nfsdg remove volume testvol 3. remove the test volume. Sun Services. perform the following b e u l steps k on the bnode that currently has the related disk group imported: a a r y fe ul 1. On Node 1. On the node that is primary for the device group. Inc. All Rights Reserved. from the nfsdg disk ab group. Oracle and/or its affiliatesฺ Perform the following steps to add a volume to an existing device group: 1. Synchronize the changes to the nfsdg disk group configuration. Synchronize the changes to the nfsdg disk group configuration: # cldg sync nfsdg 6-50 Sun™ Cluster 3. ide and csm u e G prove that the volume is now usable: t @ n b ku tude # cldg sync nfsdg a y # newfs /dev/vx/rdsk/nfsdg/testvol n ( this S i d lu se a u am Removing a Volume Fromto a Disk Device Group k ng ense a b a a volume To remove lic from a disk device group. testvol. # vxassist -g nfsdg make testvol 100m layout=mirror 3. Revision C a .2 Administration Copyright 2009 Sun Microsystems. Verify the status of the volume testvol. Make sure the device group is online (to the Sun Cluster software).

From either node. s ) ha y m ฺ m o c ฺ ฺwith how 4. Revision C 6-51 a . ab Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. # cldg show 2. Oracle and/or its affiliatesฺ Migrating Device Groups You use the cldg show command to determine current device group configuration parameters. All Rights Reserved. Disable the device group failback feature. switch the nfsdg device group to Node 2. Inc. The nfsdg disk group should automatically migrate back to Node 1. Verify this with cldg status from any node. Perform the following steps to verify device group behavior: 1. Sun Services. It is m inconsistent e d s i u be configured the device groups work with the applications will ec that G t @ in Modules 9 and 10: b den u k tu # cldg set -p failback=false ya nfsdg ( S n # cldg set -p failback=false di e thiswebdg u l # cldg show a us m a k e to g ban licens a b u le k b a a l y sfer u d ab -tran g an non 3. Boot Node 1.Exercise: Configuring Volume Management Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. # cldg switch -n node2name nfsdg Type the init 0 command to shut down Node 1. Verify the current demonstration device group configuration.

You will see error messages about the singleton device groups c ฺ each DID. All Rights Reserved. you view and manage VxVM device groups through Sun Cluster Manager. Enter the Sun Cluster Manager Application. Inc. Use the Switch Primaries button d u l e a us group. Open up the arrow pointing to the Storage folder on the left. Sun Services. In a Web browser. ak Stud y ( in tothswitch is the primary for the device 7. Perform the following steps on your administration workstation or display station. click Device Groups. Oracle and/or its affiliatesฺ Task 13 (Optional) – Viewing and Managing VxVM Device Groups Using Sun Cluster Manager In this task. 3. log in to Sun Java Web Console on any cluster node: https://nodename:6789 2. This is normal.Exercise: Configuring Volume Management Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems. 1. Click the name of your VxVM deviceu groups n b e list). csm Guide e tthe bottom of the @ (near 6. Revision C a . s ) ha y m ฺ om ฺfor 5. 4. From the subcategories revealed for Storage. m a k e to g ban licens a b u le k b a a l y sfer u d ab -tran g an non ab 6-52 Sun™ Cluster 3.

Sun Services. or discoveries you had during the lab exercises. ● Experiences ● Interpretations ● Conclusions ● Applications s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Using VERITAS Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. Inc.Exercise Summary Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 6-53 a . Oracle and/or its affiliatesฺ Exercise Summary ? ! Discussion – Take a few minutes to discuss what experiences. issues. All Rights Reserved.

Oracle and/or its affiliatesฺ y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha a .Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

Oracle and/or its affiliatesฺ Module 7 Using Solaris Volume Manager With Sun Cluster Software s ) ha Objectives ab y m ฺ om ฺ Upon completion of this module. Sun Services. Revision C a . Inc. you should be able to: c ฺ m uide cspartitioning Describe Solaris Volume Manager traditional and soft e G t @ partitions n b ku tude a y Differentiate between shared disksets ( is Sand local disksets n i th multi-owner diskset Describe Solaris Volume lud Manager e a us(metadb) management issues and the role amdatabase o Describe volume k t ngVolume seManager mediators of Solaris a n b e c Solaris Volume Manager software a lithe b Configure e u l k rab a y l eCreate the local metadb replicas f u s d ab -tran Add disks to shared disksets g an non ● ● ● ● ● ● ● ● Build Solaris Volume Manager mirrors with soft partitions in disksets ● Use Solaris Volume Manager status commands ● Perform Sun Cluster software-level device group management ● Create global file systems ● Mirror the boot disk with Solaris Volume Manager 7-1 Copyright 2009 Sun Microsystems. All Rights Reserved.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

Oracle and/or its affiliatesฺ Relevance ? ! Discussion – The following questions are relevant to understanding the content of this module: ● Is it easier to manage Solaris Volume Manager devices with or without soft partitions? ● How many different collections of metadbs will you need to manage? ● What is the advantage of using DID devices as Solaris Volume Manager building blocks in the cluster? y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha ● 7-2 How are Solaris Volume Manager disksets registered in the cluster? Sun™ Cluster 3. Inc. Sun Services.Relevance Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. Revision C a .2 Administration Copyright 2009 Sun Microsystems.

Additional Resources Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. part number 8207356 (for Sun Cluster 3.2 11/09) s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a ● Solaris Volume Manager Administration Guide. Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Additional Resources Additional resources – The following references provide additional information on the topics described in this module: ● Sun Cluster System Administration Guide for Solaris OS. ● Sun Cluster Software Installation Guide for Solaris OS. part number 820-7458 (for Sun Cluster 3. All Rights Reserved. part number 816-4520. ● Sun Cluster Concepts Guide for Solaris OS. Revision C 7-3 a . part number 821-0259 (for Sun Cluster 3. Inc.2 11/09).2 11/09).

The original Solstice DiskSuite software (Solaris Volume Manager’s precursor product) had the restriction that Solaris OS partitions were the smallest granularity building blocks for volumes. All Rights Reserved. Revision C .2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Exploring Solaris Volume Manager Disk Space Management Solaris Volume Manager has two distinct ways of managing disk space. Inc. Sun Services. The current Solaris Volume Manager supports both the traditional way of managing space and the new method called soft partitioning. a Solaris Volume Manager Partition-Based Disk Space s a Management y) h ฺm m ฺcomanagement ฺ Figure 7-1 demonstrates Solaris Volume Manager disk space e m d s i that equates standard disk partitions with building foruvirtual cblocksG e t @ volumes.Exploring Solaris Volume Manager Disk Space Management Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. b den u k tu ya d18 submirror Volume ( S n di e this u l a us m a k e to g ban licens a b u le Volume d6 submirror k b a a r y e 7 (metadb) ul nsfSlice d b a tra g n Slice 3 n a no ab Slice 4 Volume d12 submirror Slice 6 Physical Disk Drive Figure 7-1 7-4 Solaris Volume Manager Space Management Sun™ Cluster 3.

The a g onspace does not have to be physically contiguous. Solaris Volume n a n b Manager finds whatever space it can to allow your soft partition to a Soft partitioning supports the creation of virtual partitions within Solaris OS disk partitions or within other volumes. Solaris Volume Manager Disk Space Management With Soft Partitions s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( Soft partitions add an enormous in amount is of flexibility to the capability of h t Solaris Volume Manager:lud a use m a to manage They allowkyou to volumes whose size meets your exact g e requirements ns physically repartitioning Solaris OS disks. Oracle and/or its affiliatesฺ The following are limitations with using only partitions as Solaris Volume Manager building blocks: ● The number of building blocks per disk or LUN is limited to the traditional seven (or eight. Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. You are limited in the number of soft partitions only by the size of the base component. l u bd trans They can grow if there is space left inside the parent component. All Rights Reserved.Exploring Solaris Volume Manager Disk Space Management Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ban licewithout a b They lallow e you to create more than seven volumes using space from u k b ya fearasingle large drive or LUN. Inc. ● ● ● grow within the same parent component. if you want to push it) partitions. Revision C 7-5 a . This is particularly restrictive for large LUNs in hardware RAID arrays. These virtual partitions can range in size from one block to the size of the base component. ● Disk repair is harder to manage because an old partition table on a replacement disk must be recreated or replicated manually. Sun Services.

and then use those to build your submirrors as shown in Figure 7-2. Oracle and/or its affiliatesฺ ● Make soft partitions on disk partitions. Revision C a . All Rights Reserved. Volume d10 submirror Slice 7 (metadb) soft partition d18 slice s0 s ) ha V olume d20 submirror soft partition d28 y m ฺ om ฺ Volume d30 submirror c ฺ Physical Disk Drive csm Guide e @ ent Figure 7-2 Building Submirrors From SoftbPartitions u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a soft par tition d38 7-6 Sun™ Cluster 3.2 Administration Copyright 2009 Sun Microsystems. Sun Services. Inc.Exploring Solaris Volume Manager Disk Space Management There are two ways to deal with soft partitions in Solaris Volume Manager: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

Slice 7 (metadb) Slice 7 (metadb) slice s0 slice s0 Physical disk drive submirror Physical disk drive submirror s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this volume d u al use m soft partition ka e to g ban licens a b u le Using Soft Partitions to Create Volumes k b Figure 7-3 a a l y sfer u d ab -tranAll the examples later in this module use the second strategy. Sun Services. The only disadvantage of this scheme is that you have large mirror devices even if you are only using a small amount of space.Exploring Solaris Volume Manager Disk Space Management Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. This slows down resynching of mirrors. All Rights Reserved. The g n non a advantages of this are the following: b a ● Volume management is simplified. Inc. Oracle and/or its affiliatesฺ ● Build mirrors using entire partitions. and then use soft partitions to create the volumes with the size you want as shown in Figure 7-3. Revision C 7-7 a . ● Hot spare management is consistent with how hot spare pools work. Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems.

All Rights Reserved. s ) ha y m ฺ om ฺ c ฺ e csm Grequires uid that each To create any shared disksets. athat y ( S which are described later. Figure in 7-4thshows is the contents of two shared d u disksets in a typical cluster. Revision C a . Only disks that are in the same diskset operate as a unit. ewith the local disksets on each node. The only host (node) must have a local diskset on non-shared b u k theythave requirement for these local disks is ud local diskset metadbs.2 Administration Copyright 2009 Sun Microsystems. Solaris Volume e Manager @ entdisks. Oracle and/or its affiliatesฺ Exploring Solaris Volume Manager Disksets When using Solaris Volume Manager to manage data in the Sun Cluster 3. They can be used together to build mirrored volumes. all disks that hold the data for cluster data services must be members of Solaris Volume Manager shared disksets. and primary ownership of the diskset transfers as a whole from node to node.2 environment. You should always have disks from different controllers (arrays) in the same diskset. nfsds). so that you can mirror across controllers. al along s m u ka e to g Diskset nfsds Diskset webds ban licens a b u le k b a a r 1 l y sfeNode Node 2 u d n b a tra g n n a no Shared disksets are given a name that often reflects the intended usage of the diskset (for example. ab Boot Disks (Local Diskset) Figure 7-4 7-8 Boot Disks (Local Diskset) Two Shared Disksets Sun™ Cluster 3. Inc.Exploring Solaris Volume Manager Disksets Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Only disks that are physically located in the shared storage will be members of the shared disksets. Sun Services.

s ) ha y m ฺ om ฺ c ฺ Managing multi-owner disksets is identical to managing sm uother idedisksets. In the current implementation. That is. c e except that you use a -M option when you create the tdiskset. in one of the optional exercises in Module 11. In fact. multi-owner disksets are only for use with Oracle RAC software. G @ n b ku tude a y n ( this S i d lu se a am to u k ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n no aba Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. All Rights Reserved. Revision C 7-9 a . a multi-owner diskset allows more than one node to physically access the storage. you cannot create a multi-owner diskset until you have enabled a layer of software underlying Oracle RAC known as the RAC framework. simultaneously. Oracle and/or its affiliatesฺ Solaris Volume Manager Multi-Owner Disksets (for Oracle RAC) Solaris Volume Manager has a multi-owner diskset feature that is analogous to the VERITAS Cluster Volume Manager (CVM) feature. and you will get a chance to work with it.Solaris Volume Manager Multi-Owner Disksets (for Oracle RAC) Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. This is described in more detail. Sun Services. if you choose.

In the local disksets.2 Administration Copyright 2009 Sun Microsystems.Using Solaris Volume Manager Database Replicas (metadb replicas) Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. in Figure 7-4 on page 7-8 (which shows m ฺ two nodes and two shared disksets). You might prefer to use slice 7 (s7) as a convention because the shared diskset replicas have to be on that partition. The reasons why you would never want to do this on the boot disks are described later in this module. Oracle and/or its affiliatesฺ Using Solaris Volume Manager Database Replicas (metadb replicas) Solaris Volume Manager requires that certain partitions serve as volume database replicas. storing the Solaris Volume Manager configuration and state information in a raw format (non-file system). as There are separate sets of metadb replicas for the local disksets and for) h y each shared diskset. For example. Sun Services. n s i partitions. dedicated Solaris OS partitions. You might e u d u a t do this to balance the numbers of metadb across disks or (y ireplicas S subsections. for reasons described in the following lud se th a am to u k ng ense Local ReplicaaManagement b a lic b e u l k managing When a ab local replicas: r y l e f bdu transYou must add local replicas manually. set up a dedicated partition on each disk and put three copies on each disk. For example. These are normally small. csm Guide e t n b@same You can put several copies of the metadbk on the partition. with two local disks. You can put more than one copy in the same partition to satisfy this requirement. Solaris Volume Manager logs warnings. Solaris Volume Manager allows the creation of metadbs on a large partition and then the use of the same partition as a component in a volume. a You can put local replicas on any dedicated partitions on the local ang non ● ab 7-10 ● disks. Revision C a . there are four distinctccollections om ฺ of ฺ metadb replicas. Sun™ Cluster 3. Inc. ● You must spread local replicas evenly across disks and controllers. ● If you have less than three local replicas. All Rights Reserved.

Inc. Sun Services. kulocal tmetadb’s exactly 50% (but not less than 50% a ) of u y n ( this S i d lu se a u am to Management Shared Diskset kReplica ng ense a b a the following lic Consider issues when managing shared diskset replicas: b e u l k a eThere ab are separate metadb replicas for each diskset. It simply does n lett you @ b de available. Oracle and/or its affiliatesฺ The math for local replicas is as follows: ● If fewer than 50 percent of the defined metadb replicas are available. ● If exactly 50 percent of the defined replicas for a diskset are available. and just use the metadb command to delete ones that are not available. All Rights Reserved. you cannot boot the node. the diskset still operates. a g onn They will be. but it cannot be taken or switched over. This flag is a misnomer in thatuit csm e G boot fully with to do with using SVM to mirror root.Using Solaris Volume Manager Database Replicas (metadb replicas) Local Replica Mathematics Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. you can boot to single-user mode. a n ab set md:mirrored_root_flag=1 ● ● ● You must use the metadb command to remove and add replicas if you replace a disk containing replicas. Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. and must remain. r y l f bdu trans They are automatically added to disks as you add disks to disk sets. Solaris Volume Manager ceases to operate. Revision C 7-11 a . s ) ha Note – Override the behavior of the last bullet item by inserting the line: y m ฺ om ฺ c ฺ e nothing idhas in the /etc/system file. on slice 7. ● If fewer than or equal to 50 percent of the defined metadb replicas are available at boot time. Shared Diskset Replica Quorum Mathematics The math for shared diskset replica quorums is as follows: ● If fewer than 50 percent of the defined replicas for a diskset are available. the diskset ceases to operate. However.

your diskset can keep operating. the implication of the replica mathematics described in the previous section is that if one storage array fails. because it can take a while to fix a broken array or controller. This allows you to maintain normal diskset operations with exactly 50 percent of the metadb replicas surviving. Oracle and/or its affiliatesฺ When you have nodes connected to exactly two storage arrays. and you still want to be able to gracefully survive a node failure during that period. Inc. You can also lose a node at this point (you would still have at least one golden mediator). Mediators allow you to identify the nodes themselves as tie-breaking votes in the case of failure of exactly 50 percent of the metadb replicas of a shared diskset. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 7-12 Sun™ Cluster 3. but cannot transfer primary control from node to node. Sun Services.2 software environment includes special add-ons to Solaris Volume Manager called mediators. This is unacceptable in the Sun Cluster environment. The Sun Cluster 3. indicating they count as extra votes for the shared diskset quorum mathematics (one from each node). If you lose an array. Revision C a .Using Solaris Volume Manager Database Replicas (metadb replicas) Shared Diskset Mediators Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. The mediator data is stored in the memory of a running Solaris OS process on each node. the node mediators will change to golden status.2 Administration Copyright 2009 Sun Microsystems. All Rights Reserved.

Oracle and/or its affiliatesฺ Installing Solaris Volume Manager and Tuning the md.devices/node@# file system.conf File In the Solaris 9 OS and the Solaris 10 OS. The maximum value for md_nsets is 32. plus one. The modifications are summarized in Table 7-1. Keep this file identical on all nodes of the cluster. bd tran a gmd_nsets n. Two variables might need to be updated. disksets you plan to create in your cluster. These maximums include your local disksets. c this number too high can use the a Setting i l b a lot of inodes for device u le k b a a files in your /global/. the packages associated with the standard Solaris Volume Manager functionality are part of the base OS.Installing Solaris Volume Manager and Tuning the md. The y fer l u s maximum value is 8192. Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. Revision C 7-13 a .conf. This number should be set to the number of shared a Based on your planned implementation.4 The maximum number of disksets. Inc. Solaris 10 dynamically creates device files as required.Support for shared diskset mediators is in the packages SUNWmdmr and SUNWmdmu. you might need to update Solaris Volume Manager’s kernel configuration file. These packages are automatically installed as part of the cluster framework. The absolute upper limit on number of disksets (32) and number of volumes per diskset (8192) is still the same. a n b e uses setting names of volumes as well. /kernel/drv/md. This includes the local n o a n b diskset.conf File (Solaris 9 OS Only) s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u Table 7-1 Modifications to the md. Modifying the md. Changes to this file take effect after you call devfsadm or perform a boot -r.conf File Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. Sun Services.conf File ak Stud y ( in this Default Variable Description lud Value a use m o ka e tnumber g nmd 128 The maximum Solaris Volume Manager n this s to limitofthevolumes.

7-14 Sun™ Cluster 3. the partition is s7. All Rights Reserved. recovery easier in the case that you need to access a eThis abmakes r y l these structures when booted in non-clustered mode. Sun Services. n a no ab Leaving Dedicated Partitions Even When Using ZFSroot You must have local metadb replicas on dedicated local disk partitions even when you use ZFS root. In this example. you can add any metadbฺc oropartition ฺby e m component using either its cluster DID (/dev/did/rdsk/d#s#) or d s i ec t Gu using the traditional c#t#d#. These instructions assume that you have a small partition to use for the metadb replicas on your boot disk. k et g n a cens b a li c#t#d# naming scheme for local metadb replicas and Use the b traditional e u l k devices. Make sure you initialize local metadb replicas correctly and separately on each node. if you need to use Solaris Volume Manager for your data.mthen when you add a new node or s u a o repair a node. Inc. Dedicate a small partition for metadb’s outside of your ZFS configuration.Initializing the Local metadb Replicas on Local Disks Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ( S n s node.2 Administration Copyright 2009 Sun Microsystems. You might assume to having identical controllerunumbers di e on hieach t l a be surprised this will always be true. Oracle and/or its affiliatesฺ Initializing the Local metadb Replicas on Local Disks Solaris Volume Manager management cannot be performed until you initialize the local metadb replicas on each node. Using DIDs Compared to Using Traditional c#t#d# ) has y ฺm m In the Sun Cluster environment. Omit the /dev/rdsk f u ns d b aabbreviate traditional names in all of the Solaris Volume Manager a trto g n commands. @ b den u k tu it you are restricted ya Without Use the DID naming for all shared disksets. Revision C a . as discussed in the example in Module 3.

Revision C 7-15 .Initializing the Local metadb Replicas on Local Disks Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.replica active prior to last mddb configuration change u . In that case.replica is master. you must use the -f (force) option. Sun Services. you would have to make sure you had a slice 7 of sufficient size dedicated for metada’s on each one.cf p . The following example creates three copies on each of two local disks.replica's location was patched in kernel m ./dev/rdsk/c0t8d0s0 u k tu yain place. which has m ฺco command ฺ geometry to the first.locator for this replica was read successfully c . # metadb -a -f -c 3 c0t0d0s7 Repartitioning a Mirror Boot Disk and Adding metadb a Replicas as )h y ฺmidentical This procedure assumes you have a second local disk. Oracle and/or its affiliatesฺ Adding the Local metadb Replicas to the Local Disks Use the metadb -a command to add local metadb replicas.replica does not have device relocation information o . If these disks were mirrored using ZFS root. ab # metadb -i flags first blk block count a u 16 8192 /dev/dsk/c0t0d0s7 a u 8208 8192 /dev/dsk/c0t0d0s7 a u 16400 8192 /dev/dsk/c0t0d0s7 a u 16 8192 /dev/dsk/c0t8d0s7 a u 8208 8192 /dev/dsk/c0t8d0s7 a u 16400 8192 /dev/dsk/c0t8d0s7 r . fmthard: New volume table of contents now ( S n di e this u l a us # metadb -a -c 3 c0t8d0s7 m a k e to g an cens b a Using the li or metadb -i Command to Verify b metadb e u l k metadb a eReplicas ab r y l f bdu trans a g on. The n a n -i option adds the legend information about the flag abbreviations. you can use the fmthard to e m d s i replicate the boot disk’s partition table onto the disk: u ecsecond G t @ b den # prtvtoc /dev/rdsk/c0t0d0s0|fmthard -s . As you add the first ones.The output of the metadb command includes one line for each replica.replica is up to date l . this is replica selected as input Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. All Rights Reserved. Inc.replica's location was in /etc/lvm/mddb.

2 Administration Copyright 2009 Sun Microsystems. Sun Services.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Initializing the Local metadb Replicas on Local Disks W a M D F S R - replica replica replica replica replica replica replica has device write errors is active. commits are occurring to this replica had problem with master blocks had problem with data blocks had format problems is too small to hold current data base had device read errors s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 7-16 Sun™ Cluster 3. Inc. Revision C a . All Rights Reserved.

Creating Shared Disksets and Mediators Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. You might need to run the cldev list command often to map between the two. Oracle and/or its affiliatesฺ Creating Shared Disksets and Mediators Use the metaset command to create new empty disksets and to add disk drives into the diskset. Revision C 7-17 a . Inc. # cldev list -v c1t3d0 c2t3d0 DID Device Full Device Path ------------------------d9 vincent:/dev/rdsk/c1t3d0 d9 theo:/dev/rdsk/c1t3d0 d17 vincent:/dev/rdsk/c2t3d0 d17 theo:/dev/rdsk/c2t3d0 # cldev list -v d9 d17 DID Device Full Device Path ------------------------d9 vincent:/dev/rdsk/c1t3d0 d9 theo:/dev/rdsk/c1t3d0 d17 vincent:/dev/rdsk/c2t3d0 d17 theo:/dev/rdsk/c2t3d0 s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d n -a -h vincent theo # metaset anfsds ab -t-s r g # metaset -s nfsds -a -m vincent theo n n o a n ab # metaset -s nfsds -a /dev/did/rdsk/d9 /dev/did/rdsk/d17 # metaset Set name = nfsds. The first host listed as you create a new diskset is the first one to be the owner of the diskset. Sun Services. Then you can add disks. All Rights Reserved. All the examples shown in this module of diskset operations use the DID names for disks instead of the c#t#d# names. Set number = 1 Host vincent theo Owner Yes Mediator Host(s) vincent theo Aliases Drive Dbase d9 Yes d17 Yes Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. You can add mediators with the -a -m options. You must use the -a -h options of the metaset command first to create an empty diskset.

Revision C a . k ng ense a b a lic b e u l Note Cluster 3. Inc. Solaris Volume Manager automatically detects an EFI disk. Sun Services.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ● One metadb is added to slice 7 or 6 as appropriate. it is automatically as @ erepartitioned n b u follows: ak Stud y ( in(starting A small portion of the drive d hiatscylinder 0) is mapped to slice t u l e 7 or slice 6 to be usedafor statesdatabase replicas (at least 4 Mbytes in am to u size). even slice 2 is deleted).2 Administration Copyright 2009 Sun Microsystems.2 is the first Sun Cluster version to fully support k – Sun b a a r y disks with Extensible Firmware Interface (EFI) labels. All Rights Reserved. ● The flag bits are already 01 (wu in format command). which are required l e f u s d andisks of size greater than 1 Terabyte (Tbyte). ● Slice 7 or 6 is marked with the flag bits 01 (this shows up as wu when you view the disk from the format command – writable but unmountable). Oracle and/or its affiliatesฺ Creating Shared Disksets and Mediators # metadb -s nfsds flags a u a u r r # medstat -s nfsds Mediator vincent theo first blk 16 16 Status Ok Ok block count 8192 /dev/did/dsk/d9s7 8192 /dev/did/dsk/d17s7 Golden No No Mediators become golden if exactly 50 percent of the metadb replicas fail. Sun™ Cluster 3. ● The rest of the drive is mapped to slice 0 (on a standard VTOC disk. s ) ha y Automatic Repartitioning and metadb Placement ฺon m m Shared Disksets ฺco ฺ csm Guide e t When a disk is added to a diskset. and uses ● ab slice 6 for the metadb partition. The drive is not repartitioned if the disk already has no slice 2 and slice 7 already has the following characteristics: 7-18 ● It starts at cylinder 0. These disks have no slice ab -trfor g an non 7. ● It has at least 4 Mbytes (large enough to hold a state database).

Slice 7 (metadb) s ) ha y m ฺ om ฺ c Slice 0 ฺ csm Guide e @ ent b u ak Stud y ( Physical disk drive din his t u l e a us m Figure 7-5 Automatically a k e toAdding a metadb to Slice 7 g ban licens a b – If itleis impossible to add the same number of disks from each u Note k a controller abto a diskset. you should manually run the command r y l e f s -s setname -b in order to balance the number of metadb bdu tranmetadb a g replicas across controllers.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. n n o a n ab Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. you should always add an equivalent numbers of disks or LUNs from each controller to each diskset to maintain the balance of metadb replicas in the diskset across controllers. Inc. diskset metadb replicas are added automatically to slice 7 or slice 6 as appropriate (as shown in Figure 7-5). Oracle and/or its affiliatesฺ Creating Shared Disksets and Mediators Regardless of whether the disk is repartitioned. All Rights Reserved. If you have exactly two disk controllers. Sun Services. Revision C 7-19 a .

Sun Services. ● Build sub-mirrors out of slice 0 of disks across two different controllers.2 Administration Copyright 2009 Sun Microsystems. Revision C a . All Rights Reserved. t @ of eallnnodes. ● Use soft partitioning of the large mirrors to size the volumes according to your needs. agreed-upon device name from the point of b view u ak Stud y ( in this d u e d10 (soft partition) al d11 (soft partition) s m u a o k et g n ba licens a d100 (mirror) b e u l k b ya fera l u bd trans a d101 (stripe / concat) d102 (stripe / concat) ang non ab Volumes DIDs /dev/did/rdsk/d9s0 Figure 7-6 7-20 /dev/did/rdsk/d17s0 Strategy for Building Volumes Sun™ Cluster 3. s ) ha y m ฺ Figure 7-6 demonstrates the strategy again in terms of volumem and o to(d#) c ฺ ฺ DID (d#s#). Inc. Oracle and/or its affiliatesฺ Using Shared Diskset Disk Space This section shows the commands that implement the strategy described earlier in the module. ● Build a mirror out of those sub-mirrors. always use DID numbers in the cluster toeguarantee csm Guaidunique. While Solaris Volume Manager would allow you use e c#t#d#.Using Shared Diskset Disk Space Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ● Always use slice 0 as is in the diskset (almost the entire drive or LUN).

You will see in the output of metastat on the next page how both soft partitions contain noncontiguous space. Inc. s ) ha The following is an example of the commands that are used to build the configuration described in Figure 7-6 on page 7-20: ab y m ฺ m # metainit -s nfsds d101 1 1 /dev/did/rdsk/d9s0 o c ฺ ฺ nfsds/d101: Concat/Stripe is setup e m d s i # metainit -s nfsds d102 1 1 /dev/did/rdsk/d17s0 ec t Gu @ nfsds/d102: Concat/Stripe is setup b den u k # metainit -s nfsds d100 -m d101 ya Stu ( nfsds/d100: Mirror is setup n i this # metattach -s nfsds d100 d102 alud e us nfsds/d100: submirror nfsds/d102 am istoattached k # metainit -s nfsds d10n-p 200m g d100 e s a n d10: Soft Partition is setup b e a d11 -plicd100 200m b # metainit -s nfsds u lesetup k b a a d11: Soft Partition is l y sfer u d ab -tranThe following commands show how a soft partition can be grown if there g an non is space in the parent volume. Sun Services. Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. All Rights Reserved. to a total size of 600 Mbytes. because of the order in which they are created and grown. Revision C 7-21 a . even non-contiguous space. Oracle and/or its affiliatesฺ Building Volumes in Shared Disksets With Soft Partitions of Mirrors The following are two ways to indicate which diskset you are referring to for each metainit command: ● Use -s disksetname with the command ● Use disksetname/d# for volume operands in the command This module uses the former model for all the examples.Building Volumes in Shared Disksets With Soft Partitions of Mirrors Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. # metattach -s nfsds d10 nfsds/d10: Soft Partition / metattach -s nfsds d11 nfsds/d11: Soft Partition 400m has been grown 400m has been grown Note – The volumes are being increased by 400 Mbytes.

you get output only about the local diskset on the node on which you entered the command. If you do not specify a diskset name (with -s). Revision C a .2 Administration Copyright 2009 Sun Microsystems. All Rights Reserved.Using Solaris Volume Manager Status Commands Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Checking Volume Status The following metastat command output is for the mirrored volume and soft partitions built in the previous example: s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u d count k tBlock u a y 409600 n ( this S i d 819200 lu se a am to u k nfsds/d100: Mirror ng ense Submirror 0: nfsds/d101 a b lic State: Okay b a e u l knfsds/d102 Submirror 1: a ab r y l e State: Resyncing f ns bduintrprogress: a Resync 0 % done a g n Pass: 1 n o aba Readn option: roundrobin (default) # metastat -s nfsds nfsds/d11: Soft Partition Device: nfsds/d100 State: Okay Size: 1228800 blocks (600 MB) Extent Start Block 0 409664 1 1638528 Write option: parallel (default) Size: 71118513 blocks (33 GB) nfsds/d101: Submirror of nfsds/d100 State: Okay Size: 71118513 blocks (33 GB) Stripe 0: Device Start Block Dbase d9s0 0 No State Reloc Hot Spare Okay No nfsds/d102: Submirror of nfsds/d100 State: Resyncing Size: 71118513 blocks (33 GB) Stripe 0: 7-22 Sun™ Cluster 3. Inc. Oracle and/or its affiliatesฺ Using Solaris Volume Manager Status Commands No commands display information about volumes or metadb replicas in multiple disksets at the same time. Sun Services.

Inc. Oracle and/or its affiliatesฺ Device d17s0 Start Block 0 Dbase No State Reloc Hot Spare Okay No nfsds/d10: Soft Partition Device: nfsds/d100 State: Okay Size: 1228800 blocks (600 MB) Extent Start Block 0 32 1 819296 Block count 409600 819200 Device Relocation Information: Device Reloc Device ID d17 No d9 No - s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. Revision C 7-23 a . All Rights Reserved.Using Solaris Volume Manager Status Commands Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services.

Managing Solaris Volume Manager Disksets and Sun Cluster Device Groups Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. if a mirror is in the middle of synching. Revision C a . this forces a switch anyway and restarts the synch of the mirror all over again. the commands are tightly integrated into the cluster. Oracle and/or its affiliatesฺ Managing Solaris Volume Manager Disksets and Sun Cluster Device Groups When Solaris Volume Manager is used in the cluster environment.2 Administration Copyright 2009 Sun Microsystems. In the Sun Cluster environment. All Rights Reserved. theo f u ns d b preferenced: true a tra g n numsecondaries: 1 n noname: nfsds aba diskset Device Group Name ----------------nfsds Primary ------vincent Secondary --------theo Status -----Online Addition or removal of a directly connected node to the diskset using metaset -s nfsds -a -h newnode automatically updates the cluster to add or remove the node from its list. Creation of a shared diskset automatically registers that diskset as a cluster-managed device group: # cldg status Cluster Device Groups === s ) ha --. Sun Services. Inc.Device Group Status --- y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( # cldg show nfsds in this d u al use Device Groups === m ka e to g Device Group Name: nfsds an cens b a i l Type: SVM b e u l k failback: false a erab y l Node List: vincent. # cldg switch -n theo nfsds 7-24 Sun™ Cluster 3. There is no need to use cluster commands to resynchronize the device group if volumes are added and deleted. use only the cldg switch command (rather than the metaset -[rt] commands) to change physical ownership of the diskset. As demonstrated in this example.

All Rights Reserved. Inc. Sun Services.Managing Solaris Volume Manager Disksets and Sun Cluster Device Groups Dec 10 14:06:03 vincent Cluster.Device Group Status --Device Group Name ----------------nfsds Primary ------theo Secondary --------vincent Status -----Online s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. will restart on other node] # cldg status Cluster Device Groups === --.Framework: stderr: metaset: vincent: Device busy Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ [a mirror was still synching. Revision C 7-25 a .

Device Group Resynchronization Solaris Volume Manager device groups are automatically resynchronized when new volumes are added or removed within an existing diskset. all of the Solaris Volume Manager volumes must be unused (unmounted. Then you can issue the following command: # cldg offline nfsds You rarely need to take the device group offline because almost all repairs can be done while the device group is in service. The cldg set command can be used to perform cluster-specific changes to Solaris Volume Manager device groups. As such. Oracle and/or its affiliatesฺ Managing Solaris Volume Manager Device Groups Solaris Volume Manager is cluster aware. All Rights Reserved. Revision C a .2 Administration Copyright 2009 Sun Microsystems. For example. type one of the following commands (you do not need both): # cldg online nfsds # cldg online -n node_to_switch_to nfsds 7-26 Sun™ Cluster 3. there is no need to register Solaris Volume Manager disksets with the cldg create command. Sun Services. as far r y l You f bdu trasanthes cluster is concerned. Inc. the failback property ku oftuadgroup a y modified with the following command: n ( this S i d lu nfsds # cldg set -p failback=true e a s m u ka e to g ns Offline ban lGroup e Putting a Device c a i b e u l k a canetake aba Solaris Volume Manager device group out of service.Managing Solaris Volume Manager Device Groups Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. To place the device group back online. or otherwise not open). for emergency repairs. s ) ha y m ฺ Other Changes to Device Groups om ฺ c ฺ sm uide c e G The properties of existing Solaris Volume Manager device groups can be t @ n b e could be changed. a ang non ab To put the device group offline.

m usnewfs as normal to create a UFS file a o k t system on thegvolume: e n ens ba/dev/md/nfsds/rdsk/d10 # newfs a lic b e u l k rab a y l fe uMounting s d n File Systems b a a r t g an non The distinction between a global and a failover file system is made in the /etc/vfstab mount-at-boot and options columns.Using Global and Failover File Systems on Shared Diskset Volumes Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. and it should be identical on all nodes including nodes not physically connected to the storage: /dev/md/nfsds/dsk/d10 /dev/md/nfsds/rdsk/d10 /global/nfs ufs 2 yes global A failover file system entry looks similar to the following.2 supports running data services on the following categories of file systems: ● Global file systems – These are accessible to all cluster nodes simultaneously. and it should be identical on all nodes that might run services that access the file system (they can only be nodes that are physically connected to the storage). ab as h The file system type can be UFS or VxFS regardless of whether you are ) y in m using global or failover file systems. ● Failover file systems – These are mounted only on the node running the failover data service. A global file system entry should look similar to the following. The examples and the labฺexercises om ฺ this module assumes that you are using UFS. Revision C 7-27 a . Sun Services. /dev/md/nfsds/dsk/d10 /dev/md/nfsds/rdsk/d10 /localnfs ufs 2 no - Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. Inc. c ฺ csm Guide e @ ent b Creating File Systems u ak Stud y ( dinand his file system is not made at the t The distinction between lglobal failover u e a Use time of file system creation. which must be physically connected to the storage. Oracle and/or its affiliatesฺ Using Global and Failover File Systems on Shared Diskset Volumes Sun Cluster 3. All Rights Reserved. even those not physically connected to the storage.

Verifying Partitioning and Local metadbs The bold sections of the following output emphasize the partitions that you will mirror: # df -k /dev/dsk/c0t0d0s0 /devices 7-28 33453987 7360376 25759072 0 0 0 23% 0% / /devices Sun™ Cluster 3. ku tude a y ( of ithe Sboot disk and the new The geometry and partitionin tables s d h t mirror are identical. This discussion is for those running Solaris OS prior to Solaris 10 10/08. All Rights Reserved. you might also use Solaris Volume Manager to manage and mirror the boot drive. Oracle and/or its affiliatesฺ Using Solaris Volume Manager to Mirror the Boot Disk If Solaris Volume Manager is your volume manager of choice for the shared storage.Using Solaris Volume Manager to Mirror the Boot Disk Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. That way. you can just go back to mounting the standard partitions g by editing the /etc/vfstab manually.bThis le was done previously. cluster-specific case. if you need to f u d ab -tran back out. Revision C a . The reason is that each of these file systems appear in the /etc/mnttab of each node (as global file systems) and the Solaris OS does not allow duplicate device names. om ฺ c ฺ e root All partitions on the boot drive are mirrored. The following example contains the following scenario: s ) ha y m ฺ The boot drive is mirrored after cluster installation. Copying table from one disk to the lu sthee partition a other was previously demonstrated in ‘‘Repartitioning a Mirror Boot u am o k t Disk and Adding metadb Replicas’’ on page 7-15. Sun Services. of course you will not need or be able to mirror your root drive with Solaris Volume Manager. The exampleid has sm c u e (/). swap. k a a l y sSoft erpartitions are not used on the boot disk. Note – If you already have ZFS-root.2 Administration Copyright 2009 Sun Microsystems. Inc. or still interested in using a legacy configuration in Solaris 10 10/08 and later. Gyou will be t @ n b manually creating three separate mirror devices. That is. an non ● ● ● ● ● ab ● The /global/.devices/node@1.devices/node@1 is a special. and /global/. g nse n a b bootlicdisk e and mirror have three copies of the local metadb Bothathe b ureplicas. For that one device you must use a different volume d# for the top-level mirror on each node.

Create simple sub-mirror volumes for the existing partition and the other disk partition. 4.1 16 8170064 8170064 csm Guide e @ ent b u # metadb ak Stud y ( flags first blk in block iscount d h t a m pc luo 16 8192 /dev/dsk/c0t0d0s7 u e al us8192 a pc luo 8208am /dev/dsk/c0t0d0s7 k e to 8192 a pc luo 16400 /dev/dsk/c0t0d0s7 g n s a n a m pc luo 8192 /dev/dsk/c0t8d0s7 ce ab 16 i l a pc luo 8208 8192 /dev/dsk/c0t8d0s7 b u le16400 k b a pc luo 8192 /dev/dsk/c0t8d0s7 a a y fer l u s bd tran a g onBuilding n Volumes for Each Partition Except for Root a n ab For all boot disk partitions except root. Wait to reboot until you do all partitions. 2. Make sure the d# chosen for the /global/. Edit the /etc/vfstab file manually to use the new mirror volume instead of the original partition. Sun Services. Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. Inc. All Rights Reserved. Create a mirror using only the sub-mirror mapping to the existing partition. Oracle and/or its affiliatesฺ Using Solaris Volume Manager to Mirror the Boot Disk ctfs proc mnttab swap objfs fd swap swap 0 0 0 1920832 0 0 1926208 1919440 0 0 0 0 0 0 1472 1919360 0 0 0 0 6848 1919360 80 1919360 0% 0% 0% 1% 0% 0% 1% 1% /system/contract /proc /etc/mnttab /etc/svc/volatile /system/object /dev/fd /tmp /var/run /dev/did/dsk/d11s3 491520 3735 438633 1% /global/.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 7-29 a .devices/node@# is different across your nodes. Use the -f option for the existing partition. 3. follow the same general strategy: 1.devices/node@1 /dev/did/dsk/d1s3 s ) ha y m ฺ # swap -l om ฺ c ฺ swapfile dev swaplo blocks free /dev/dsk/c0t0d0s1 32.devices/node@2 491520 3677 438691 1% /global/. Do not attach the other half of the mirror until after a reboot.

the procedure would t @ n b for the non-root slices. Create simple sub-mirror volumes for the existing partition and for the other disk partition. Revision C no - a . 3. which will automatically edit the /etc/vfstab and /etc/system files.2 Administration Copyright 2009 Sun Microsystems. s ) ha y m ฺ om ฺ c ฺ sm uide If Examples are shown for root. and /global/. All Rights Reserved.devices/node@#. follow the same general strategy: 1. Use the -f option for the existing partition. ( is S n i lud se th # metainit -f d11 1 1ac0t0d0s0 u d11: Concat/Stripe is setup am o k t # metainit d12 ng1 e1 nc0t8d0s0 se a b d12: Concat/Stripe is setup c a i l b # metainit d10 u le -m d11 k b a a d10: Mirror y fer is setup l u d10 ns bd tr#ametaroot a ang non # metainit -f d21 1 1 c0t0d0s1 Running the Commands ab d11: Concat/Stripe is setup # metainit d22 1 1 c0t8d0s1 d12: Concat/Stripe is setup # metainit d20 -m d21 d10: Mirror is setup # vi /etc/vfstab (change the correct line manually) /dev/md/dsk/d20 - - swap - # metainit -f d31 1 1 c0t0d0s3 d11: Concat/Stripe is setup 7-30 Sun™ Cluster 3. Inc.devices/node@# do you need ku tude a y to worry about making a different mirror volume name on each node. c e Gbe the same as you had any other OS disk partitions. Create a mirror using only the sub-mirror mapping to the existing partition. Only for /global/.Using Solaris Volume Manager to Mirror the Boot Disk Building Volumes for Root Partition Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. 2. Sun Services. Oracle and/or its affiliatesฺ For the root boot disk partition. Use the metaroot command. swap. Do not attach the other half of the mirror until after a reboot.

Sun Services.Using Solaris Volume Manager to Mirror the Boot Disk Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. you can attach the secondbsubmirror u k background.devices/node@1 /dev/did/dsk/d22s5 95702 /global/. The After a reboot.devices/node@2 # swap -l swapfile /dev/md/dsk/d20 0 0 0 0 6848 1919360 80 1919360 5317 80815 0% 0% 1% 1% 7% 5010 6% 81122 /system/object /dev/fd /tmp /var/run dev swaplo blocks free 85. All Rights Reserved. Revision C 7-31 a .devices/node@1 ufs 2 no global s ) ha ab y m ฺ om ฺ c Rebooting and Attaching the Second Submirror ฺ csm Guide e @ entot each volume. in this d u al use # init 6 m ka e to # df -k g s /dev/md/dsk/d10 26843000 25510407 5% / an cen1064163 b a /devices 0 0 0% /devices li 0 b e u l k ctfs 0 0 0 0% /system/contract a erab y l proc 0 0 0 0% /proc f u ns d b mnttab 0 0 0 0% /etc/mnttab a tra g n swap 1920832 1472 1919360 1% /etc/svc/volatile n a no objfs 0 fd 0 swap 1926208 swap 1919440 /dev/md/dsk/d30 95702 /global/.20 16 8170064 8170064 # metattach d10 d12 # metattach d20 d22 # metattach d30 d32 Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems.devices/node@# manually) /dev/md/dsk/d30 /dev/md/rdsk/d30 /global/. Oracle and/or its affiliatesฺ # metainit d32 1 1 c0t8d0s3 d12: Concat/Stripe is setup //following volume name (d30) must be different on each // node # metainit d30 -m d31 d10: Mirror is setup # vi /etc/vfstab (change correct line (for /global/. Inc. synchronizing of the mirrors runsyin athe tud and can take a long ( S time.

Oracle and/or its affiliatesฺ In this exercise. Revision C a .2 software f u d on Solaris 10 OS.Exercise: Configuring Solaris Volume Manager Exercise: Configuring Solaris Volume Manager Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems. Inc. 7-32 Sun™ Cluster 3. slice 7 should have been reserved ahead of time to accomodate this lab. If you happen to be using ZFS root. The examples in this exercise use slice 7 of the boot disk. an ab -trenvironment g an non s ) ha ● Task 5 – Creating a Global nfs File System ● ● ● ● ab At least one local disk on each node must have a small unused slice that you can use for the local metadb replicas. All Rights Reserved. Sun Services. you complete the following tasks: ● Task 1 – Initializing the Solaris Volume Manager Local metadb Replicas ● Task 2 – Selecting the Solaris Volume Manager Demo Volume Disk Drives ● Task 3 – Configuring Solaris Volume Manager Diskset and Volumes for NFS ● Task 4 – Configuring Solaris Volume Manager Demonstration Volumes y m ฺ om ฺ c Task 6 – Creating a Global web File System ฺ csm Guide Task 7 – Testing Global File Systems e @ ent b u Task 8 – Managing Disk Device Groups ak Stud y ( Task 9 – Viewing and Managing in Solaris is Volume Manager Device d h t u Groups Using Sun Cluster e al Manager s m u ka e to g n s Preparationaba licen b u le k b a a l yThis sexercise er assumes you are running the Sun Cluster 3.

All Rights Reserved. l u bd trans a g onn a n ab Primary Disk Mirror Disk d100 Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. such as c b IPaddress. Sun Services. u leenclosure_name. Oracle and/or its affiliatesฺ Node 1 Boot Disk Boot Disk c0 Node 2 c0 (replicas) (replicas) c1 c2 c1 c2 A B A B Quorum Disk Diskset and Volumes nfsds s ) ha y m ฺ om ฺ c ฺ webds Mirror Disk Primary Disk sm uide c e d100 G t @ n b Array A Array B ku tude a y S n ( Volume s i i Figure 7-7 Configuring Solaris Manager d h lu se t a am to u k g nse Note a – During ban lithis e exercise.Exercise: Configuring Solaris Volume Manager During this exercise. Inc. when you see italicized names. Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. you create two data service disksets that each contain a single mirrored volume as shown in Figure 7-7. substitute the names appropriate for your cluster. node1. Revision C 7-33 a . or clustername embedded in a k b a a command y fer string.

In the standard setup. Use the metadb command to increateththree is replicas on the unused boot d u l disk slice. Inc. Oracle and/or its affiliatesฺ Before you can use Solaris Volume Manager to create disksets and volumes.Exercise: Configuring Solaris Volume Manager Task 1 – Initializing the Solaris Volume Manager Local metadb Replicas Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. uid Check Caution – Your boot disk path may be different onedifferent @ ent carefully on each node.2 Administration Copyright 2009 Sun Microsystems. Sun™ Cluster 3. 3. Make sure you do not use those disks in this exercise. On allbnodes. if any. type the cldev list -v command to list all of the available DID drives and make note of the DID numbers of disks used for VxVM. type vxdisk -o alldgs list to ensure that you pick drives that do not conflict with VxVM. Perform the following steps on each node of the cluster: 1. Record the logical path and DID path numbers of four disks that you will use to create the demonstration disksets and volumes in Table 7-2. the boot disk (c#t#d# below) should be configured as the ZFS root pool. If you happen to also be using VxVM. you must initialize the state database and create one or more replicas. On Node 1. # zpool status # prtvtoc /dev/rdsk/c#t#d#s0 s ) ha y m ฺ om ฺ c ฺ e csm Gnodes. Revision C a . e c a i l b u# metadb le k b a a l y sfer u d ab -tran g n nTask on 2 – Selecting the Solaris Volume Manager Demo a b a Volume Disk Drives Perform the following steps to select the Solaris Volume Manager demo volume disk drives: 7-34 1. the replicas are configured and operational. Remember to mirror across arrays. b u ak Stud y ( 2. Sun Services. 2. Verify that the boot disk has a small slice s7 reserved for metadb’s. e a us m a # metadb -ak-f -c t3oc#t#d#s7 e ng verify sthat a n 3. All Rights Reserved.

ak oneStcluster y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. Inc. The first part is the same for all DID devices: /dev/did/rdsk. All Rights Reserved. Oracle and/or its affiliatesฺ Note – You need to record only the last portion of the DID path. Revision C 7-35 a . u dual-hosted and available to more than ud host. Table 7-2 Logical Path and DID Numbers Diskset Volumes example d100 nfsds d100 webds d100 Primary Disk Mirror Disk c2t3d0 d4 c3t18d0 d15 s ) ha y m ฺ om ฺ c ฺ csm Guide e t Note – Make sure the disks you select arebnot They must be @localendevices.Exercise: Configuring Solaris Volume Manager Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services.

di e this u # metaset -s nfsds l a us m # medstat -s a nfsds k e to # cldg status g an cens b a 5. b a submirror e u l k a # emetainit ab -s nfsds d0 1 1 /dev/did/rdsk/dxs0 r y l f bdu trans# metainit -s nfsds d1 1 1 /dev/did/rdsk/dys0 a g on. # metaset -s nfsds -a -h node1 node2 2. # metattach -s nfsds d99 d1 8.2 Administration Copyright 2009 Sun Microsystems. # metastat -s nfsds 7-36 Sun™ Cluster 3. Create a 500 Mbyte soft partition. Add the same nodes as diskset mediators to each diskset. This is the volume you will actually use for your file system data. mto the o c ฺ ฺ nfsds diskset. e m d s i ec t \Gu # metaset -s nfsds -a /dev/did/rdsk/dx @ b den /dev/did/rdsk/dy u k ya Stu ( 4. d99.6. and configure the nodes that are physically connected to it. All Rights Reserved. Revision C a . d100. using the d0 submirror. Create li on each of your disks in the nfsds diskset. Oracle and/or its affiliatesฺ Perform the following steps to create demonstration disksets and volumes for use in later exercises: 1. Verify the status of the new volume. # metainit -s nfsds d100 -p d99 500m 9. Attach the second submirror. Add the two disks chosen previously. On Node 1. # metaset -s nfsds -a -m node1 node2 s ) ha y m ฺ 3. Verify the status of the new n diskset. Inc. one from each array. d1. on top of your mirror.Exercise: Configuring Solaris Volume Manager Task 3 – Configuring Solaris Volume Manager Diskset and Volumes for NFS Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Create a mirror volume. Sun Services. n a n ab # metainit -s nfsds d99 -m d0 7. create the nfsds diskset. to the mirror volume d99.

e m d s i ec t Gu\ # metaset -s webds -a /dev/did/rdsk/dx @ b den /dev/did/rdsk/dy u k ya Stu ( 4. b Create on each of your disks in the webds diskset.6. # metaset -s webds -a -h node1 node2 2. to the mirror volume d99. n a n ab # metainit -s webds d99 -m d0 7. # metaset -s webds -a -m node1 node2 s ) ha y m ฺ 3. on top of your mirror. Attach the second submirror. Revision C 7-37 a . and configure the nodes that are physically connected to it. create the webds diskset. Oracle and/or its affiliatesฺ Task 3 – Configuring Solaris Volume Manager Diskset and Volumes for Web Perform the following steps to create demonstration disksets and volumes for use in later exercises: 1. d99. This is the volume you will actually use for your file system data. Verify the status of the new volume. # metainit -s webds d100 -p d99 500m 9. n i is # metaset -s a webds lud se th # medstat -s amwebdsto u k # cldg ngstatus se a n b e a a submirror 5. # metattach -s webds d99 d1 8. Create a mirror volume. # metastat -s webds Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. lic e u l k b a e#rametainit -s webds d0 1 1 /dev/did/rdsk/dws0 y l f u s bd tran # metainit -s webds d1 1 1 /dev/did/rdsk/dzs0 a g on. using the d0 submirror. d100. m to the c ฺ ฺ webds diskset. Inc. Add the same nodes as diskset mediators to each diskset. Create a 500 Mbyte soft partition. one from eachoarray. All Rights Reserved. On Node 1. Add the two disks chosen previously. Sun Services.Exercise: Configuring Solaris Volume Manager Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. d1. Verify the status of the new diskset.

add a mount entry in the /etc/vfstab sm fileufor idethe new c e file system with the global mount option. g e n s ba 1. create a global mount point for the new file system. g # mount an non ab 7-38 # ls /global/nfs lost+found Sun™ Cluster 3. On all nodes. On Node 1. Revision C a . you can choose a different mount point for this one.2 Administration Copyright 2009 Sun Microsystems. c a i l 4. # newfs /dev/md/nfsds/rdsk/d100 2. mount en the /global/nfs file system. G t @ n b /dev/md/nfsds/dsk/d100 /dev/md/nfsds/rdsk/d100 \ ku tude a y /global/nfs ufs 2 yes global n ( this S i d lu se a m to u aline k Note – Do not use the continuation character (\) in the vfstab file.an Verify that the file system is mounted and available on all nodes. create a file system on d100 in the nfsds diskset. On Node b u le/global/nfs k b a a # mount l y sfer u d ab -tr5. Inc. s ) ha y m ฺ # mkdir /global/nfs om ฺ c ฺ 3. Oracle and/or its affiliatesฺ Perform the following steps on Node 1 to create a global file system in the nfsds diskset: 1.Exercise: Configuring Solaris Volume Manager Task 5 – Creating a Global nfs File System Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Note – If you have already done the VxVM exercises. and already have a mounted /global/nfs file system. All Rights Reserved. Sun Services. On all nodes.

Inc. 4. All Rights Reserved.Exercise: Configuring Solaris Volume Manager Task 6 – Creating a Global web File System Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. g # mount an non # ls /global/web lost+found Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems. # newfs /dev/md/webds/rdsk/d100 2. On Node 1. Verify that the file system is mounted and available on all nodes. Revision C 7-39 a . Oracle and/or its affiliatesฺ Perform the following steps on Node 1 to create a global file system in the webds diskset: 1. G t @ n b /dev/md/webds/dsk/d100 /dev/md/webds/rdsk/d100 \ ku tude a y /global/web ufs 2 yes global n ( this S i d lu se a m to u athe k Note – Do not use line continuation character (\) in the vfstab file. On all nodes.cmount the /global/web file system. On all nodes. add a mount entry in the /etc/vfstab sm ufile idefor the new c e file system with the global mount option. and already have a mounted /global/web file system. g e n s a n b e a Nodeli1. create a global mount point for the new file system. you can choose a different mount point for this one. create a file system on d100 in the webds diskset. Note – If you have already done the VxVM exercises. s ) ha ab y m ฺ # mkdir /global/web om ฺ c ฺ 3. b On u le /global/web k b a a # mount l y sfer u d ab -tran5.

2 Administration Copyright 2009 Sun Microsystems. All Rights Reserved.Exercise: Configuring Solaris Volume Manager Task 7 – Testing Global File Systems Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. 3. Boot Node 1. Try unmounting and mounting /global/nfs from all nodes. You should get an error that the file system is busy. move into the /global/nfs file system.2 software environment: 1. 2. 5. On Node 2. try to unmount the /global/nfs file system. am groups 1. 4. 5. s ) ha y m ฺ om ฺ c ฺ sm uide c e Task 8 – Managing Disk Device Groups G t @ n b ku tude a y Perform the following steps to migrate n ( tahdisk s Sdevice group (diskset) i i d between cluster nodes: lu se a uare online (to the Sun Cluster software). Inc. move out of the /global/nfs file system (cd /) and try to unmount it again on Node 1. On Node 1. # cldg switch -n node1 nfsds 7-40 Sun™ Cluster 3. # cd /global/nfs 2. Revision C a . Both disksets should remain mastered by Node 2. On Node 2. Oracle and/or its affiliatesฺ Perform the following steps to confirm the general behavior of globally available file systems in the Sun Cluster 3. 4. Make sure thekdevice o t ng ense # cldgastatus b a lic b e u l k rab a y –fe You can bring a device group online to a selected node as follows: ul Note s d n b # cldg switch -n node_to_switch_to devgrpname a a r t g an non ab # cldg show 3. Shut down Node 1. Verify the current demonstration device group configuration. The nfsds and webds disksets should automatically migrate to Node 2 (verify with the cldg status command). Use the cldg switch command from either node to migrate the nfsds diskset to Node 1. Mount the /global/nfs file system on Node 1.

Enter the Sun Cluster Manager Application. s ) ha y m ฺ omgroups 5. ya Stu ( n 7. 3. Revision C 7-41 a . All Rights Reserved. Click the name of your Solaris Volume n device groups (near b Manager e u d k the bottom of the list). Oracle and/or its affiliatesฺ Task 9 – Viewing and Managing Solaris Volume Manager Device Groups Using Sun Cluster Manager In this task you view and manage Solaris Volume Manager device groups through Sun Cluster Manager. You will see error messages about the singleton device c ฺ ฺ for e m d s each DID. Open up the arrow pointing to the Storage folder on the left. i ec t Gu @ 6. Use the Switch Primaries di button htoisswitch the primary for the device t u l e a us group. 1. log in to Sun Java Web Console on any cluster node: https://nodename:6789 2. Sun Services. Perform the following steps on your administration workstation or display station. In a Web browser. From the subcategories revealed for Storage. Inc.Exercise: Configuring Solaris Volume Manager Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. This is normal. click Device Groups. 4. m a k e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Using Solaris Volume Manager With Sun Cluster Software Copyright 2009 Sun Microsystems.

Exercise Summary Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ● Experiences ● Interpretations ● Conclusions ● Applications s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 7-42 Sun™ Cluster 3.2 Administration Copyright 2009 Sun Microsystems. issues. All Rights Reserved. Inc. Revision C a . Sun Services. or discoveries you had during the lab exercises. Oracle and/or its affiliatesฺ Exercise Summary ? ! Discussion – Take a few minutes to discuss what experiences.

you should be able to: ● ● ● 8-1 Copyright 2009 Sun Microsystems. All Rights Reserved. Oracle and/or its affiliatesฺ Module 8 Using ZFS With Sun Cluster Software Objectives s ) ha y m ฺ m odata Understand when ZFS can be used for Sun Cluster c ฺ ฺ e m d s i Build ZFS storage pools and file systems ec u G t @ b den Use ZFS root in the Sun Cluster environment u k ya Stu ( n di e this u l a us m a k e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Upon completion of this module. Inc. Sun Services.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a .

Revision C a . All Rights Reserved. why would you use any other volume manager in the Sun Cluster environment? s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 8-2 Sun™ Cluster 3.Relevance Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. Inc. Oracle and/or its affiliatesฺ Relevance ? ! Discussion – The following questions are relevant to understanding the content of this module: ● If ZFS is so powerful and so easy to configure.2 Administration Copyright 2009 Sun Microsystems.

All Rights Reserved.2 11/09) s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a ● Solaris ZFS Administration Guide .2 11/09). part number 820-7458 (for Sun Cluster 3. part number 8207356 (for Sun Cluster 3. ● Sun Cluster Software Installation Guide for Solaris OS. part number 819-5461 Using ZFS With Sun Cluster Software Copyright 2009 Sun Microsystems. part number 821-0259 (for Sun Cluster 3. Revision C 8-3 a . Inc. Oracle and/or its affiliatesฺ Additional Resources Additional resources – The following references provide additional information on the topics described in this module: ● Sun Cluster System Administration Guide for Solaris OS. Sun Services. ● Sun Cluster Concepts Guide for Solaris OS.Additional Resources Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 11/09).

y m ฺ om ฺ c ฺ ZFS automatically manages its storage in units calledszpools. and raidz2 e of RAID 6). With ZFS.ZFS for Sun Cluster Failover Data ZFS for Sun Cluster Failover Data Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. and the added space will n theepool ns toat all babe automatically available the filesystems within. Just grow any time. 8-4 Sun™ Cluster 3. All Rights Reserved. that ZFS automatically maintains within the pool. c a li b e u l k rab a y fe ul ZFS s d n Using with Sun Cluster: Eliminating /etc/vfstab b a a r t g n nEntries on a b a The configuration database. contains all the mount information for file systems within the pool. You will be able to store data only for failover applications (not scalable applications). Oracle and/or its affiliatesฺ Beginning in the initial release of Sun Cluster 3. Inc. zpool m uThe ideraidz cof mirroring. nor will you need to perform any cluster device group management. you never m u a need to worry about the amount of space available to a particular file o k et g system. as such. You never need to create any associated vfstab entries.2 Administration Copyright 2009 Sun Microsystems. you will be able to run the applications only on nodes physically connected to the storage. Sun Services. e layer provides optional data protection in the form (a G t @ n b ZFS-specific variation of RAID 5). Using ZFS with Sun Cluster: Volume Management s ) ha When you use ZFS. Revision C a .2 ZFS became available as a failover file system for application data. you will generally not need any other software volume manager (neither VxVM nor Solaris Volume Manager). All the file u l systems in a pool share allaof the storage s in the pool. u (a variation d k u a t (y is S n i Within a zpool you can build any number d e th of file systems.

Building ZFS Pools and File Systems

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Building ZFS Pools and File Systems
In the cluster, you use traditional disk paths (not DID devices) as
components of pools. The pools can still fail over even if the device paths
have different names on different nodes (the model is precisely analogous
to how VxVM has always automatically discovered disk groups).
vincent:/# zpool create marcpool mirror c1t0d0 c2t0d0
vincent:/# zpool status
pool: marcpool
state: ONLINE
scrub: none requested
config:

s
) ha

y
m

NAME
STATE
READ WRITE CKSUM
om ฺ
c

marcpool
ONLINE
0
0
0
csm Guide
e
mirror
ONLINE
0
0
0
@ ent
b
c1t0d0 ONLINE
0
0
0
u
k tud
c2t0d0 ONLINE
0
0(ya 0
n this S
i
d
lu se
errors: No known data errors
a
am to u
k
g npool
When a ZFS
nstorage
se is created, a default root file system is created
a
b
e
for theapool. The
licdefault mount point for that file system is /poolname.
b
e
u
l
kNow we
bcan create additional filesystems within the pool. ZFS
a
a
r
y
automatically
creates mount points and mounts the file systems. You do
l
e
f
u
s
d
ab -trannot need to add /etc/vfstab entries.
g
nvincent:/#
on zfs create marcpool/myfs1
a
n
b
a
vincent:/# zfs create marcpool/myfs2

vincent:/# zfs list -r
NAME
USED
marcpool
158K
marcpool/myfs1
18K
marcpool/myfs2
18K
vincent:/# df -h
Filesystem
.
.
marcpool
marcpool/myfs1
marcpool/myfs2

marcpool
AVAIL REFER
19.6G
21K
19.6G
18K
19.6G
18K

size

used

20G
20G
20G

19K
18K
18K

MOUNTPOINT
/marcpool
/marcpool/myfs1
/marcpool/myfs2

avail capacity

20G
20G
20G

1%
1%
1%

Mounted on

/marcpool
/marcpool/myfs1
/marcpool/myfs2

Using ZFS With Sun Cluster Software
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

8-5

a

Building ZFS Pools and File Systems

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

The mount points default to /poolname/fsname, but you can change them
to whatever you want:
vincent:/# zfs set mountpoint=/oracle marcpool/myfs1
vincent:/# zfs set mountpoint=/shmoracle marcpool/myfs2
vincent:/# df -h |grep marcpool
marcpool
20G
18K
20G
1%
/marcpool
marcpool/myfs1
20G
18K
20G
1%
/oracle
marcpool/myfs2
20G
18K
20G
1%
/shmoracle

Growing a ZFS Storage Pool

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
When you grow a ZFS storage pool, y
the
ud check that you are
aksystem
twill
(
S
using the same kind of underlying
as
in RAID
isdevices already in the pool.
d
h
t
u
While it is not recommended,
you
can
specify
pools with mixed RAID
l
e
a
s
moption
uto the zpool command.
behavior by using thea-f
o
k
t
ng ense
a
b
The following
a example
lic shows adding new mirrored disks into a mirrored
b
e
pool:
u
l
k rab
a
y
fe add marcpool mirror c1t3d0 c2t3d0
ul # zpool
s
d
n
b
a
tra
g
n
n
o
nQuotas
aba
and Reservations
You can grow a ZFS storage Pool by adding new devices any time. There
is no such thing as growing or shrinking individual ZFS file systems, and
you get way more flexibility by being able to set quotas and reservations
on the file systems, as shown in the next section.

You control the amount of space associated with each individual file
system within the pool using quotas and reservations.
The default (without quotas and reservations) is that all file systems share
all the space in the pool. Any file system can hog all the space in the pool,
and any file system can be deprived of space by another file system being
a hog.
A quota sets a maximum amount of space for a file system (to prevent it
being a hog).

8-6

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Building ZFS Pools and File Systems

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

A reservation sets a guaranteed amount of space for the file system. This is
just a guaranteed amount, not translating to any particular physical file
blocks.
Quotas and reservations can be grown at any time, within the bounds of
the total size of the pool. Quotas and reservations can be shrunk at any
time, as long as the current usage of a file system is below the new
lowered values. This scheme gives way more flexibility than traditional
resizing of legacy types of file systems.
The following is an example of setting a quota and a reservation on a file
system.
# zfs set quota=20g marcpool/myfs2
# zfs set reservation=6g marcpool/myfs2

s
) ha

ab

y
m

om ฺ
c

ZFS Snapshots
csm Guide
e
@ ent
b
u
ak snapshot
ZFS has an instantaneous point-in-time
tud feature.
y
(
S
in this
d
u
e any any additional space pool. If there
Initially, snapshots do not
al consume
s
m
u
are no changes to
o file system , the parent and the snapshot
kathe parent
t
g
e
point to the
ZFS implements a copy-on-write policy for
nsameedisk
nsablocks.
bain
the parent;
this
way
ZFS
snapshot
appears to grow over time due to it
c
a
i
ldifferent
b
e
u
l
occupying
the
blocks
(the
original
blocks) from the parent after
k
b
a
a
parent
l y the
er has made modifications.
f
u
s
d
n
alist
ab zfs
r
theo:/#
t
g
USED AVAIL REFER MOUNTPOINT
anNAME non
orapool
orapool/oracle

2.70G
2.70G

30.5G
30.5G

24.5K
2.70G

/orapool
/oracle

theo:/# zfs snapshot orapool/oracle@thursday_1feb09
theo:/# zfs list
NAME
USED AVAIL REFER MOUNTPOINT
orapool
2.70G 30.5G 24.5K /orapool
orapool/oracle
2.70G 30.5G 2.70G /oracle
orapool/oracle@thursday_1feb09
299K
- 2.70G The parent file system can be rolled back. If you want to roll back to a
snapshot that is not the most recent one, you must specify a -r option to
the rollback subcommand which will automatically destroy the
snapshots taken after the one you are rolling back to.
theo:/# zfs rollback orapool/oracle@thursday_1feb09

Using ZFS With Sun Cluster Software
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

8-7

a

Zpool Ownership: In General and in Sun Cluster

Zpool Ownership: In General and in Sun Cluster
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

The commands that ZFS uses to transfer ownership of a pool from one
host to another are zpool export and zpool import. The pool is always
the unit of ownership. A new node importing a pool need never have
seen it or heard of it before; the zpool import command will
automatically scan the connected storage looking for pools:
vincent# zpool export marcpool
theo# zpool import
zpool import
pool: marcpool
id: 17715066435682450109
state: ONLINE
action: The pool can be imported using its name or numeric
identifier.
config:

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
marcpool
ONLINE
u
ak ONLINE
mirror
tud
y
(
S
c0t2d0
in thisONLINE
d
u
c1t2d0
al use ONLINE
m
ka marcpool
to
g
e
theo# zpool n
import
ba licens
a
b
u
le
k
b
a
a
r
ZFS
Failover in the Cluster
l yPool
eAutomatic
f
u
s
d
ab -tran
g
n non The Sun Cluster device group management layer does not manage ZFS
a
b
storage pools in any way.
a
Automating the failover of ZFS storage in the cluster is achieved using
application-level resources, which are covered in a later module in this
book.

8-8

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

ZFS Root in Sun Cluster.

ZFS Root in Sun Cluster.
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Solaris 10 10/08 introduced the ability to use ZFS as the root file system.
Sun Cluster 3.2 1/09 (update 2) and later support this feature. ZFS root
has little bearing on proper cluster operations. ZFS’s advantages in
simplicity and flexibility, and the ability to manage and rollback
snapshots, may make ZFS root a preferred choice even if you require
other volume managers to manage data.
Recall the /global/.devices/node@# file system must be a UFS. When
you have ZFS root it will likely reside on a lofi device using
/.globaldevices as the backing store.

s
) ha

ab

y
m

m
o
c


ZFS Root Pool and Sun Cluster
e
m
d
s
i
ec t Gu
@
nof entire disks. The
b consist
ZFS pools other than the root pool typically
e
u
d
k
a it beSbuilt
tu on top of Solaris
root pool is different and requires
ythat
(
n
partitions. This is because Solaris
di ecannot
his boot from an EFI-labeled disk.
t
u
l
a us
m
a
In typical non-cluster
k scenarios,
to the Solaris installer provisions slice 0 of
g
e
the boot disk
an tocmap
nsthe entire disk, and creates the root pool using slice
b
e
a
0. b
li
e
u
l
k
a erab
y
l
f only variation that you need to consider in Sun Cluster is the need for
u nThe
s
d
b
Volume Manager dedicated metadb partitions, if you require
a
tra Solaris
g
n
Solaris
Volume
Manager for any or all of your cluster data. Recall from
n
a
no
Module 3 that the Solaris jumpstart facility has a natural way of reserving
space for such a dedicated partition:
# example Jumpstart profile -- ZFS with
# space on s7 left out of the zpool for SVM metadb
install_type initial_install
cluster SUNWCXall
filesys c0t0d0s7 32
pool
rpool
auto
2G
2G c0t0d0s0
Currently the interactive install does not have the ability to create a
partition that is not part of the root pool. In this case, you would need
separate local disks if you required metadbs.

Using ZFS With Sun Cluster Software
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

8-9

a

ZFS Root in Sun Cluster.

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

A disk that will be used as a mirror in the root pool would likely be
configured identically. For example, if after installation you were going to
add a disk of identical geometry to your root pool, you could copy the
partitioning of the original root pool disk and then add the mirror:
# prtvtoc /dev/rdsk/c0t0d0s2|fmthard -s - /dev/rdsk/c0t1d0s2
# zpool attach rpool c0t0d0s0 c0t1d0s0
# installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk \
/dev/rdsk/c0t1d0s0
Note – You must install the ZFS boot block manually, as shown in the last
command above, when you attach a mirror to a ZFS root pool. If you
choose to mirror the root pool as you install Solaris (using the interactive
installer or Jumpstart), the boot block will automatically be placed on both
disks.

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
Typical ZFS Root Configuration
m
ka e to
g
nsis automated by the Solaris installer. There are no
Provisioning
root
baofnZFS
e
c
a
li for Sun Cluster, besides the recommendation for
brequirements
special
e
u
l
k
a eminimum
ab swap space. Note that swap space and dump space are
750MB
r
y
l
f
provisioned on ZFS volumes (zvol’s).
ns
bdu trautomatically
a
a
g status
n#n
zpool
o
a
n
ab pool: rpool
state: ONLINE
scrub: resilver completed after 0h4m with 0 errors on Fri Apr 10
03:34:06 2009
config:
NAME
rpool
mirror
c0t0d0s0
c0t1d0s0

STATE
ONLINE
ONLINE
ONLINE
ONLINE

READ WRITE CKSUM
0
0
0
0
0
0
0
0
0
0
0
0

errors: No known data errors

8-10

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

ZFS Root in Sun Cluster.

# zfs list -r rpool
NAME
rpool
rpool/ROOT
rpool/ROOT/s10s_u6wos_07b
rpool/dump
rpool/export
rpool/export/home
rpool/swap

USED AVAIL REFER MOUNTPOINT
8.84G 58.1G
94K /rpool
4.83G 58.1G
18K legacy
4.83G 58.1G 4.67G /
2.00G 58.1G 2.00G 38K
58.1G
20K /export
18K
58.1G
18K /export/home
2G 60.1G
3.56M -

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
# swap -l
in this
d
u
swapfile
dev al swaplo
e blocks free
s
m
u
/dev/zvol/dsk/rpool/swap 256,1
16
4194288 4194288
a
o
k
t
ng ense
a
b
# dumpadm
a kernellicpages
b
Dump content:
u
le
k
b
a
a
Dump
device:
/dev/zvol/dsk/rpool/dump
(dedicated)
y fer
l
u
s
Savecore
/var/crash/host01
n
bd directory:
a
a
r
t
Savecore
enabled:
yes
g onn
a
n
ab

Note – The Solaris jumpstart automatically creates the /export and
/export/home file systems. If you do not want to use them, you can just
destroy them at your leisure.

Using ZFS With Sun Cluster Software
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

8-11

a

ZFS Root in Sun Cluster.

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

ZFS Root and Snapshots
You can take snapshots of the root file system or of any other file system
within the root pool. The operation looks like any other snapshot
operation and you can give your snapshot any name you like:
# zfs snapshot rpool/ROOT/s10s_u8wos_08b@installM

There is a special procedure for rolling back snapshots of the root
filesystem:
1.

Boot or reboot the system using -F failsafe option:
ok boot -F failsafe

s
) ha

y
m

om ฺ
c

csm Guide
3. Rollback your snapshot
e
@ ent
b
4. Reboot the system
u
ak Stud
Rebooting with command: boot -F failsafe n (y
is and args: -F failsafe
di e thFile
Boot device: /pci@1f,700000/scsi@2/disk@0,0:a
u
l
a us 64-bit
SunOS Release 5.10 Version Generic_141444-09
m
a
k e toInc. All rights reserved.
Copyright 1983-2009 Sun Microsystems,
g
ns
Use is subject to license
banterms.
e
c
a
li
Hardware watchdog enabled
b
e
u
l
k
Configuring devices.
a erab
y
l
f OS instances...
Searchingdu
for installed
s
n
b
a
tra
g
n
n
ROOT/s10s_u6wos_07b
was found on rpool.
a
no
abDo
you wish to have it mounted read-write on /a? [y,n,?] y
2.

Confirm that the system mounts the root pool for you under /a
when it volunteers to do so.

mounting rpool on /a
Starting shell.
# zfs list
NAME
USED AVAIL REFER
rpool
8.84G 58.1G
94K
rpool/ROOT
4.83G 58.1G
18K
rpool/ROOT/s10s_u8wos_08b
4.83G 58.1G 4.67G
rpool/ROOT/s10s_u8wos_08b@installM 170M
- 4.64G
rpool/dump
2.00G 58.1G 2.00G
rpool/export
38K 58.1G
20K
rpool/export/home
18K 58.1G
18K
rpool/swap
2G 60.1G 3.56M
# zfs rollback rpool/ROOT/s10s_u8wos_08b@installM
# reboot

8-12

MOUNTPOINT
/a/rpool
legacy
/a
/a/export
/a/export/hom
-

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Exercise: Configuring Volume Management

Exercise: Configuring Volume Management
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

In this exercise, you complete the following tasks:

Task 1 – Create a ZFS Pool and File System for Cluster Data

Task 2 – Create a Snapshot and then Modify your Data

Task 3– Manually Migrate your ZFS Pool to Another Node

Task 4 – Examine ZFS Root and Mirror Your Root Pool

Task 1 – Create a ZFS Pool and File System for
Cluster Data

s
) ha

y
m

oa m
1. Select disk drives from the shared storage to use ฺin
new ฺZFS pool
c
m
for data. Select a disk from shared storage. Make
de you do
s sureuithat
c
e
Ggroups. Note the
not use any disks already in use in existing
device
t
@
n
b
e step 2). The following
logical device name (referred to k
asuc#t#d#din
u
a
t
y
example checks against VxVM,
Solaris
Volume
Manager, and ZFS
( is S
n
i
disks.
lud se th
a
m twith
udisks that may already be in use with
acareful
o
k
Be especially
g nManager.
nVolume
se
Solaris
The metaset command lists their DID
a
b
e
c
a
i
l
device
name.
Make
sure
you
understand the c#t#d# associated with
b
e
u
l
k
a ethose
ab DID devices and choose other disks for ZFS. Consult with your
r
y
l
f
instructor if you are unsure.
bdu trans # vxdisk -o alldgs list
a
ang non
# zpool status

ab

# metaset
# cldev list -v

2.

From one node connected to the storage, create a zpool that provides
mirroring for your two disks:
# zpool create nfspool mirror c#t#d# c#t#d#
# zpool status

3.

Create a ZFS file system within the pool:
# zfs create nfspool/nfs
# zfs list -r nfspool
# df -h

4.

Change the mount point.
# zfs set mountpoint=/testnfs nfspool/nfs
# df -h

Using ZFS With Sun Cluster Software
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

8-13

a

Exercise: Configuring Volume Management
5.

Populate your file system with some simple files

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

# cd /testnfs
# touch file1 file2 file3
# vi letter_to_mom
Hi Mom, ZFS is much better than that legacy stuff

Task 2 – Create a Snapshot and then Modify your Data
1.

Snapshot your ZFS file system:
# zfs snapshot nfspool/nfs@SAVEMYWORK
# zfs list -r nfspool

y
m

om ฺ
2. Modify the content of your file system:
c

# cd /testnfs
csm Guide
e
# touch file4
@ ent
b
u
# rm file2
ak Stud
y
(
# vi letter_to_mom
din ethan
histhat legacy stuff
Hi Mom, ZFS is muchlubetter
t
a usabout my class
Here is more information
m
a
k e to
g
an cens
b
a
li Migrate your ZFS Pool to Another
Task 3–uManually
b
e
l
k
a erab
Node
y
l
u nsf
d
b
a
tr1.a Export your pool:
g
n
n
no
aba
# cd /

s
) ha

# zpool export nfspool
# zpool status
# zfs list
2.

On the other node that is connected to storage, import the pool:
#
#
#
#
#

3.

Recover the snapshot that had migrated along with the pool:
#
#
#
#

8-14

zpool import
zpool import nfspool
zpool status
zfs list -r nfspool
df -h
cd /
zfs rollback nfspool/nfs@SAVEMYWORK
cd /testnfs
ls

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Exercise: Configuring Volume Management

Task 4 – Examine ZFS Root and Mirror Your Root Pool

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

1.

Examine your ZFS root configuration on one of your cluster nodes.
#
#
#
#

2.

zpool status
zfs list -r rpool
swap -l
dumpadm

Find another unused local disk (same controller as your root disk).
Consult your instructor if necessary.
# format

3.

Modify the partitioning of your second disk to match that of the first
disk, assuming they are of identical geometry:

s
) ha

ab

y
m

om(argument
Warning – make sure the first disk in the command below
to
c


e
m
prtvtoc) is the original root disk, and the second
one (argument
to
csthis
uid you
e
G
fmthard) is your new intended mirror. If you
get
backwards
will
t
@
n
b
have trouble!
ku tude
a
y
S
n| (fmthard
s
i
i
d
h
# prtvtoc /dev/rdsk/origdisk-c#t#d#s0
-s - /dev/rdsk/newdiskt
u
l
e
a
c#t#d#s0
am to us
k
4. Add the
to
ngmirror
seyour root pool. Make sure you use slice 0, and not
a
n
b
e
slice
2.
a
lic
b
e
u
l
k #razpool
b attach rpool origdisk-c#t#d#s0 newdisk-c#t#d#s0
a
y
#
zpool
status
l
e
f
u
s
d
ab -tran5. Install the boot block on your second disk:
g
an# installboot
-F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk \
non
/dev/rdsk/c#t#d#s0

Using ZFS With Sun Cluster Software
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

8-15

a

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Exercise: Configuring Volume Management

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

8-16

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Revision C a . Inc.xxx files ban licens a b Configure e IPMP manually with ifconfig commands u l k b ra a forced failover of an adapter in an IPMP group ya fePerform l u s bd tran Describe the integration of IPMP into the Sun™ Cluster software a environment ang non Upon completion of this module.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. you should be able to: ● ● ● ● ● ● ● ● ab 9-1 Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Module 9 Managing the Public Network With IPMP Objectives s ) ha y m ฺ om ฺ Define the purpose of IPMP c ฺ m uide Define the concepts of an IPMP group ecs t Gon a single @ n b List examples of network adaptersuin IPMP e groups ak Stud Solaris™ OS server y ( inthe in. All Rights Reserved. Sun Services.mpathd d his daemon t Describe the operation of u l e a us m a List the options to k ethetoifconfig command that support IPMP and g configure IPMP using /etc/hostname.

Sun Services.Relevance Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Relevance ? ! Discussion – The following questions are relevant to understanding the content of this module: ● Should you configure IPMP before or after you install the Sun Cluster software? ● Why is IPMP required even if you do not have redundant network adapters? ● Is the configuration of IPMP any different in the Sun Cluster environment than on a stand-alone Solaris OS? y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha ● 9-2 Is the behavior of IPMP any different in the Sun Cluster environment? Sun™ Cluster 3. Revision C a . Inc. All Rights Reserved.2 Administration Copyright 2009 Sun Microsystems.

Revision C 9-3 a . part number 8192970 ● Sun Cluster Concepts Guide for Solaris OS. Sun Services. part number 819-2971 ● Sun Cluster Software Installation Guide for Solaris OS. part number 816-4554 s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Managing the Public Network With IPMP Copyright 2009 Sun Microsystems.Additional Resources Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. part number 819-2969 ● Solaris Administration Guide: IP Services. Inc. Oracle and/or its affiliatesฺ Additional Resources Additional resources – The following references provide additional information on the topics described in this module: ● Sun Cluster System Administration Guide for Solaris OS. All Rights Reserved.

IPMP enables you to configure redundant network adapters.” provide detail a Advanced lic Resourceof Group b e about the proper configuration applications and their IP addresses in u l k b a a r software environment. and provides failover and failback of IP addresses among members of the same group. The IPMP daemon detects failures and repairs of network connectivity for adapters. s ) ha y m ฺ In the Sun Cluster 3.” Module 10. G t @ n b e ku tubydmechanisms a These application IP addresses are implemented known y ( S n s as LogicalHostname and SharedAddress The configuration of dipublic hiresources.2 Administration Copyright 2009 Sun Microsystems. on the same server (node) and on the same subnet. a Module to and g e Resource Groups. “Introducing Data Services.Introducing IPMP Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a . as an IPMP failover group. t u l these resources requires that the network adapters being used are e a us m under the control ofkIPMP. “Configuring Scalable n ens a b Services and Relationships. Existing TCP connections to the IP addresses that fail over as part of an IPMP group are interrupted for short amounts of time without data loss and without being disconnected. you must use IPMP om toฺ c ฺ manage any public network adapters on which you will e the IP mbe placing d s i c u addresses associated with applications running inethe cluster.2 software environment. and HA-NFS. 9. Sun Services. Oracle and/or its affiliatesฺ Introducing IPMP IPMP has been a standard part of the base Solaris OS since Solaris 8 OS Update 3 (01/01). All Rights Reserved. Inc. l ythe Sun eCluster f u s d ab -tran g an non ab 9-4 Sun™ Cluster 3.

This is automatically set correctly by n a n b scinstall. n b@same e if possible. b redundancy e u l k a eEach abEthernet adapter must have a unique MAC address. you cannot combine Ethernet adapters and Asynchronous Transfer Mode (ATM) adapters in the same group. All Rights Reserved. Sun Services. a s m u kamultiple You can have togroups on the same subnet.Describing General IPMP Concepts Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. The test interface enables test traffic on all the members of the IPMP group. Revision C 9-5 a . This can be any alphanumeric name. g e s You angroup with only one member. each adapter requires a dedicated test IP address. a ● ● ● ● ● ● Network adapters in the same group must be of the same type. n ( this S i d lu subnets Adapters on different e must be in different groups. This is the reason that local-mac-address?=true is required for IPMP. Defining IPMP Group Requirements You must observe the following rules when configuring an IPMP group: s ) ha ● A network adapter can be a member of only one group. Managing the Public Network With IPMP Copyright 2009 Sun Microsystems. This is an extra static IP for each group member specifically configured for the purposes of testing the health of the adapter using ping traffic. should be connectedktouphysically separate switches on d u a t y the same subnet. Inc. Oracle and/or its affiliatesฺ Describing General IPMP Concepts IPMP allows you to group together network adapters for redundancy. y m ฺ om ฺ c ฺ csm Guide e All members of a group must be on the subnet. This is r y l f bdu trans achieved by setting the local-mac-address? variable to true in the a g onOpenBoot™ PROM. t The members. ● In Solaris 9 OS. the group names are always the same (and thus need not be specified explicitly for IPv6). The group name is meaningful only inside a single Solaris OS server. when more than one adapter is in an IPMP group. For example. or test interface. ● When both IPv4 and IPv6 are configured on a physical adapter. However. there is no bacann have e c a li and there is no automatic failover. Members of the same IPMP group are identified by a common group name.

only th be used on the local subnet). k tudthat test address will a y ( is(theSaddress automatically always be the link-local IPv6naddress i d assigned to the adapter. ● s ) ha If both IPv4 and IPv6 are configured on the same adapters.2 Administration Copyright 2009 Sun Microsystems. If you configure adapters in Solaris 10 OS IPMP groups without test addresses. Note – Using IPMP without test addresses reduces network traffic and reduces the administrative strain of allocating the addresses. this author has personally experienced adapters with a valid link state but broken receive logic. Revision C a . However. the health of the adapter is determined solely by the link state of the adapter. If a failure is detected (even if you have only an IPv4 test address). uthat can l e a s IPV6 (Optional)’’ on page 9-16 to see See ‘‘Configuring m Adapters u for a o k t how to set up g IPv6nsoneyour adapters. Oracle and/or its affiliatesฺ ● Solaris 10 OS does not require test addresses. even with multiple adapters in a group. it is possible to have test addresses for both IPv4 and IPv6. Sun Services. y m ฺ om ฺ c ฺ csm Guide e @ ent b u If you do choose to have an IPv6 test address.Describing General IPMP Concepts Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. but it is not necessary. n a ab lice b u le k b a a l y sfer u d ab -tran g an non ab 9-6 Sun™ Cluster 3. The remaining examples in the module focus on creating IPMP configurations with test addresses. all IPv4 and IPv6 addresses (except the test address) fail over to the other physical adapter. Such an adapter would be properly faulted using test addresses. Inc. the testing is less robust. For example.

● The standby adapter is preferred as a failover target if another member of the group fails. le k b valid a a l y sfer u d ab -tran g n non a b a ● You must have at least one member of the group that is not a standby adapter. setting b u option. Standby adapters have the following properties: ● You are allowed (and must) configure only the test interface on the adapter.Describing General IPMP Concepts Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Managing the Public Network With IPMP Copyright 2009 Sun Microsystems. am members o k t g nse IPMP group in the Sun Cluster software If you had anthree-member a b ce up one of the members as a standby would still be a a i l environment. you can configure an adapter as a standby adapter for the group. All Rights Reserved. Revision C 9-7 a . Any attempt to manually configure any other addresses fails. Oracle and/or its affiliatesฺ Configuring Standby Adapters in a Group In an IPMP group with two or more members. Sun Services. Inc. This allows the n (additional s i i d h Sun Cluster software to balance IP addresses associated with u se t l a applications across both u of the group. s ) ha y m ฺ om ฺ c ฺ csm Guide e t n b@groups e u Note – The examples of two-memberkIPMP in d the Sun Cluster u a t y software environment will not use any standby S adapters. ● Additional IP addresses are added to the adapter only as a result of a failure of another member of the group.

Single IPMP Group With Two Members and No Standby Figure 9-1 shows a server with two member adapters in a single IPMP group. These two adapters must be on the same subnet and provide failover for each other. Sun Services. s ) ha y m ฺ om ฺ c ฺ Group: csm Guide therapy e @ ent b u qfe0 qfe4 ak tud y ( S in this d u e al failover s mutual m u ka e to g ban licens a Solaris™ OS Server b e u l k b ya fera l u bd trans a g on. Inc.Examining IPMP Group Examples Examining IPMP Group Examples Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a .Figure 9-1 Server With Two Member Adapters in a Single IPMP Group n a n ab 9-8 Sun™ Cluster 3. Oracle and/or its affiliatesฺ The following sections describe and illustrate examples of network adapters in IPMP groups on a single Solaris OS server.2 Administration Copyright 2009 Sun Microsystems. All Rights Reserved.

Inc. All Rights Reserved. Revision C 9-9 a . Oracle and/or its affiliatesฺ Single IPMP Group With Three Members Including a Standby Figure 9-2 shows a server with three members of a single IPMP group. one of the adapters in the group is configured as a standby.Examining IPMP Group Examples Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. In the example. Sun Services. Group: therapy qfe0 qfe4 qfe8 (standby) s ) ha y m ฺ om ฺ failover c ฺ csm Guide e Solaris™ Operating System @Server ent b u ak Stud y ( din Members his of a Single IPMP Group Figure 9-2 Server WithluThree t e a us m a k e to g s Different Subnets Two IPMPbGroups an cenon a li b e u l k a Figure a9-3bshows how different IPMP groups must be used for adapters r y l e f bdu tranons different subnets. a g onn a n ab Group: therapy qfe0 qfe4 qfe1 Group: mentality qfe5 Solaris™ OS Server Figure 9-3 Two IPMP Groups on Different Subnets Managing the Public Network With IPMP Copyright 2009 Sun Microsystems.

2 Administration Copyright 2009 Sun Microsystems.Examining IPMP Group Examples Two IPMP Groups on the Same Subnet Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. Inc. Oracle and/or its affiliatesฺ Figure 9-4 shows two different IPMP groups configured on the same subnet. All Rights Reserved. Revision C a . Group: therapy qfe0 qfe4 Group: mentality qfe1 qfe5 s ) ha y m ฺ Solaris™ OS Server om ฺ c ฺ csm Guide e @ ent b u k Subnet aSame tud Figure 9-4 Two IPMP Groups on (the y S in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 9-10 Sun™ Cluster 3. Failover still occurs only within each particular group.

or cables and still have a valid link state. even in Solaris 10. s ) ha ab y m ฺ om ฺ c ฺ de The following paragraphs describe the functionality csmofGtheuiin. is more robust. b u without test addresses. default If no targets are discovered through the routing table. network c. the targets are discovered by a ping command to the 224.mpathd daemon automatically chooses targets in the following order: 1. host b. Managing the Public Network With IPMP Copyright 2009 Sun Microsystems. All Rights Reserved. Inc.mpathd e daemon in a configuration using test addresses.mpathd daemon starts automatically when an adapter is made a member of an IPMP group through the ifconfig command.mpathd daemon sends Internet alused. @ Inenthet Solaris 10 OS.Note – Using test addresses.mpathd daemon controls the behavior of IPMP. Sun Services. You can ang non Network Path Failure Detection have broken adapters. The in.0.Describing IPMP Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. 2.uthe s When test addresses are m a o kProtocol t(ICMP) Control Message echo probes (pings) to the targets g e n s a n connected the link all adapters that belong to a group to detect ce Theontest abandtorepair. Targets are chosen from the routing table in memory in the following order: a. l e f u s bd tran a . Revision C 9-11 a .0. adapter failure akdetection tudand repair is based solely y ( S on the link state of the adapter. Oracle and/or its affiliatesฺ Describing IPMP The in. i l failures address is used as the source address of these b e u l k b ra ya pings. This behavior can be summarized as a three-part scheme: ● Network path failure detection ● Network path failover ● Network path failback The in. in this d u ein. ports.1 (all-hosts) multicast IP address.

d his The Solaris Administration You can put these commands in u l e a us listed in the resources section at the Guide: IP Services fromadocs. For example. These will then be chosen first as m ฺ targets.com m k suggests to making a boot script beginning of thisgmodule e /etc/rc2.168. If there are no replies to five consecutive probes.1.168. If you have not configured a standby adapter.mpathd daemon chooses that for failover of IP addresses and multicast memberships.mpathd daemon considers the adapter as having failed.d/S70ipmp. The probing rate depends on the failure detection time (FDT).168. You might need to manipulate the choice of ping targets. if you wanted to use 192. you might have default routers that are explicitly configured not to answer pings. Revision C a .5 ak Stud-static y ( ianboot tscript.5 192.mpathd chooses the adapter with the least number of IP addresses. 9-12 Sun™ Cluster 3.39 and om ฺ c ฺ 192.Describing IPMP Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. The new adapter assumes all of the IP addresses of the failed adapter.2 Administration Copyright 2009 Sun Microsystems. you could run these commands: csm Guide e # route add -host 192.mpathd daemon probes all the targets separately through all the adapters in the multipathing group.168.39 t @ e-static n b u # route add -host 192. ub le k b a a l y sfer u d an Path Failover abNetwork r t g n non a b a After a failure is detected.168. failover of all network access occurs from the failed adapter to another functional adapter in the group. the in. For example. This works on both Solaris 9 and Solaris ban licens a 10. The default value for failure detection time is 10 seconds. For a failure detection time of 10 seconds. Sun Services. At this time the IP addresses moved during the failover will be returned to their original path. assuming failback is enabled.targets. Oracle and/or its affiliatesฺ To ensure that each adapter in the group functions properly. If you have configured a standby adapter. Inc. as h One strategy is to enter specific static host routes to IP addresses that are ) y the on the same subnet as your adapters.1.sun.39 192. the in. All Rights Reserved. except for the test address.1.1. the probing rate is approximately one probe every two seconds. Network Path Failback The in.5 as the targets. using each adapter’s test address.168. the in.1. in.1.mpathd daemon detects if the failed path has been repaired.

The maximum number of virtual interfaces per physical interface is 8192. Examining ifconfig Options for IPMP ab as h Several new ifconfig options have been added for use with IPMP. Any IP address marked with this option is not r y l e f used as the source IP address for any client connections initiated bdu trans from a this machine.) They y are the following: m ฺ om ฺ in a c ฺ group groupname – Adapters on the same subnet are placed m group idename. Inc. After these are correct. while not required. different group names across different n ( the s i i d h t lu option e -failover – Useathis to demarcate the test address for each s minterface u is the only interface on an adapter that atest adapter. The failover group by configuring them withe the cssame u Gonly within a t @ group name can be any name and must be unique n b e ku tifud single Solaris OS (node). Oracle and/or its affiliatesฺ Configuring IPMP This section outlines how IPMP is configured through options to the ifconfig command. turns that adapter into a standby-only adapter. It is irrelevant you have the same or a y S nodes of a cluster. You will specify the correct IPMP-specific ifconfig options in the /etc/hostname. The o k t ngfaileover seto another adapter when a failure occurs. is typically used on deprecated b e u l k the a abtest interfaces. ● addif – Use this option to create the next available virtual interface for the specified adapter. Managing the Public Network With IPMP Copyright 2009 Sun Microsystems. doesanot n b a lic – This option.Configuring IPMP Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ang non ● ● ● ● standby – This option. by just giving the IP address assigned to the interface. All Rights Reserved. when used with the physical adapter (-failover deprecated standby). you never have to change them again.xxx files. You do not need to know the virtual interface number. Revision C 9-13 a . No other virtual interfaces can be configured on that adapter until a failure occurs on another adapter within the group. ● removeif– Use this option to remove a virtual interface. Sun Services.

it is a good practice to put names in /etc/hosts to keep track of the addresses.xxx files once and never n ab The convention changed due to a bug relating to having a deprecated flag on the physical adapter.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Putting Test Addresses on Physical or Virtual Interfaces IPMP was designed so that the test address for each member of an IPMP group can be either on the physical interface (for example. Inc. qfe0) or on a virtual interface (for example.4. toof placing the test interface on a virtual g e interface. s ) ha y m ฺ om ฺ c ฺ csm Guide e t specifically @applications.20. Sun Services. created with the ifconfig xxx addif command). is therefore. # cat /etc/hosts . While it is not required to have host names for these.194 vincent-qfe1-test 9-14 Sun™ Cluster 3. There were at one time many existing IPMP documents written with the test IP on the physical interface because it looks cleanest to have only the virtual interfaces failing over to other adapters and the IP on the physical interface always staying where it is.xxx files still need to exist for subsequent reboots. qfe0:1. The /etc/hostname. This /etc/hosts fragment shows the dedicated IP address for the test interface for each adapter. Note – The bug concerns failure of certainuRPC n b e ak S when the deprecated flag is on the physical adapter. tud You should always y ( use the deprecated flag with test inaddresses. All Rights Reserved.20. Refer to bug #4710499 for more information. have to worry about IPMP configuration again. whenngiven ba licens a b u le k b a a l y ifconfig er f Using Commands to Configure IPMP u s d b tran a g onn a While you will configure your /etc/hostname. these examples show using the ifconfig command directly with the IPMP-specific options.192 vincent # test addresses for vincent (node 1) 172. u the test addresses are not a onl the physical The examples in this e s m u module follow the convention kathe choice. Revision C a . you should ensure d thadapter. #physical host address 172.Configuring IPMP Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.4.

4.IPv4.20. Revision C 9-15 a .20.MULTICAST.4.xxx Files for IPMP u Configuring d ab -tran g n n# catno/etc/hostname.4. Oracle and/or its affiliatesฺ # ifconfig qfe1 vincent group therapy netmask + broadcast + up # ifconfig qfe1 addif vincent-qfe1-test -failover deprecated \ netmask + broadcast + up # ifconfig qfe2 plumb # ifconfig qfe2 vincent-qfe2-test group therapy -failover deprecated \ netmask + broadcast + up # ifconfig -a .MULTICAST.192 netmask ffffff00 broadcast 172. Inc.4.RUNNING.20.qfe2 vincent-qfe2-test group therapy -failover deprecated \ netmask + broadcast + up Managing the Public Network With IPMP Copyright 2009 Sun Microsystems.195 netmask ffffff00 broadcast 172.NOFAILO VER> mtu 1500 index 3 inet 172.BROADCAST. Sun Services. qfe1: flags=1000843<UP.194 netmask ffffff00 broadcast 172.255 groupname therapy ether 8:0:20:f1:2b:e s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer the /etc/hostname.RUNNING.195 vincent-qfe2-test Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.20.20.MULTICAST.Configuring IPMP 172.IPv4> mtu 1500 index 2 inet 172. All Rights Reserved.20.BROADCAST.255 qfe2:flags=9040843<UP.255 groupname therapy ether 8:0:20:f1:2b:d qfe1:1:flags=9040843<UP.DEPRECATED.4.DEPRECATED.4.NOFAI LOVER> mtu 1500 index 2 inet 172.IPv4.qfe1 a b a vincent group therapy netmask + broadcast + up addif vincent-qfe1-test -failover deprecated netmask + broadcast + up # cat /etc/hostname.4.RUNNING.20.BROADCAST.

create empty file names for the IPV6 interfaces. so you do not need to also have an IPV6 test address. # touch /etc/hostname6. Sun Services.qfe2 b a up lic b -failover u le k b a # init 6 a l y sfer u d ab -tran g n nUsing on IPV6 Test Address Only a b a It is possible to configure IPV6 test addresses only. configuring IPV4 without any test addresses and then configuring IPV6 with test addresses. you must manually configure your adapters for IPv6. without a test address. You can choose to configure IPV6 with or without an IPV6 test address. de sm Theuionly c e address that can serve this purpose is the link-local address. G which is t @ n b automatically assigned to the interface and can be used only on the local u ude k a t subnet. you can still have(very y simple S IPV6 adapter n s i i configuration files: lud se th a # vi /etc/hostname6. Inc. The advantage is do not have to use any space in your IPV4 public network address range for test addresses.Configuring IPMP Configuring Adapters for IPV6 (Optional) Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a . IPV6 is enabled on the interfaces with the same group name as the IPV4 interfaces. that is.qfe1 # touch /etc/hostname6.qfe1 am to u -failover up g k n ense a # vi /etc/hostname6. If you do not want an IPV6 test address.2 Administration Copyright 2009 Sun Microsystems. you can choose to use an IPV6 test address. Oracle and/or its affiliatesฺ If you want to support application-specific IP addresses that use IPv6.qfe2 # init 6 s ) ha y m ฺ om ฺ c ฺ Alternatively. therefore. The disadvantage is you would have to have some external machines outside the cluster also configured with IPV6 to answer the ping. 9-16 Sun™ Cluster 3. You already have an IPV4 test address. All Rights Reserved. as in the example in the previous section. Upon reboot.

If the response comes back correctly. # FAILURE_DETECTION_TIME=10000 # # Failback is enabled by default. # The minimum time that can be specified is 100 ms. the in. Managing the Public Network With IPMP Copyright 2009 Sun Microsystems. Note u le you do not need to edit the default k b a a /etc/default/mpathd configuration file. y fer l u s bd tran a g on. the in. It also prints the time that it can meet currently. If the load on the network is too great. This file contains the following default settings and information: # # Time taken by mpathd to detect a NIC failure in ms. Then the in.mpathd Configuration File The in.mpathd daemon prints a message on the console.mpathd daemon does not fail back the addresses if the FAILBACK option is set to no. Inc. Turn off this # option to track all network interfaces on the system # TRACK_INTERFACES_ONLY_WITH_GROUPS=yes s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b – Generally.Configuring IPMP Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Changes to this file are read by the in. Sun Services. indicating that the time cannot be met. Revision C 9-17 a . failbacks take place when the failed adapter is repaired. you can set this to no so that in.The three settings you can alter in this file are: n a n ab ● FAILURE_DETECTION_TIME – You can adjust the value of this parameter. However.mpathd daemon uses the settings in the /etc/default/mpathd configuration file to invoke multipathing. All Rights Reserved. To disable failback # turn off this option # FAILBACK=yes # # By default only interfaces configured as part of # multipathing groups are tracked.mpathd daemon at startup and on a SIGHUP.mpathd monitors traffic even on adapters not in groups. Oracle and/or its affiliatesฺ The in.mpathd daemon meets the failure detection time provided in this file. In the cluster environment make sure you leave the value yes so that private transport adapters are not probed. the system cannot meet the failure detection time value. ● TRACK_INTERFACES_ONLY_WITH_GROUPS – In stand-alone servers. ● FAILBACK – After a failover.

Revision C a . are failed over to another member of the group. except the test interface. use: s ) ha y m ฺ om ฺ c ฺ This command allows the adapter to take back the IPsaddresses idtoewhich cis m e it was originally assigned. All IP addresses. Oracle and/or its affiliatesฺ You can do a forced failover of an adapter in an IPMP group.Performing Failover and Failback Manually Performing Failover and Failback Manually Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a # if_mpadm -r qfe1 9-18 Sun™ Cluster 3.2 Administration Copyright 2009 Sun Microsystems. The adapter is marked down and will not be accessed by in. Inc. The following is an example: # if_mpadm -d qfe1 This command causes IPMP to behave as if qfe1 had failed. Sun Services. To reenable an adapter after this operation. assuming FAILBACK=yes set G in u the @ ent b /etc/default/mpathd file.mpathd until reenabled.

xxx files without IPMP groups. your configuration. Sun Services. or g youncan econfigure IPMP before scinstall. Inc. scinstall automatically rewrites the files such that each adapter is a member of an IPMP group. IPMP is configured on each node exactly as in a nonclustered environment. Revision C 9-19 a . but kuaddress) the data address (for example. It is a helpful convention to use the same group names on all nodes that are connected to the same subnet. the node u a t y ( iTos customize S you will want a different test n address. by copying in the correct a y fer l u /etc/hostname. Managing the Public Network With IPMP Copyright 2009 Sun Microsystems.2 environment.2 Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. IPMP itself is aware only of what is going on within the local Solaris OS. as you are doing in this u after e albe s module (where youm will modifying some files rewritten by u a o k t scinstall). as in the following example: s ) ha ab y m ฺ om ฺ c ฺ ide by hand You still need to correctly customize multi-adapter IPMP u groups csm e Gscinstall uses t to use the appropriate test addresses.qfe1 vincent group sc_ipmp0 -failover Using Same Group Names on Different Nodes It makes no difference to IPMP whether you use the same group names or different group names for IPMP across nodes.Configuring IPMP in the Sun Cluster 3. Oracle and/or its affiliatesฺ Configuring IPMP in the Sun Cluster 3. s bd tran a ang non vincent:/# cat /etc/hostname. Thebfile created by @ n deas the test address.2 Environment There are no special cluster-related tools for configuring IPMP in the Sun Cluster 3. n s a ceto make IPMP configuration part of your Solaris OS ab choose i l b You might e u lsoftware k b a JumpStart installation. All Rights Reserved. Configuring IPMP Before or After Cluster Installation If you have /etc/hostname. i d h t you can choose to wait until scinstall.xxx files as part of a JumpStart™ software finish script.

Sun Services. If you do. Oracle and/or its affiliatesฺ It is unlikely that you want to use a standby adapter in a two-member IPMP group in the cluster environment. It makes sense that you want them to rebalance when a repair is detected. the Sun Cluster software will be unable to load-balance additional application-related IP addresses across the members of the group.Configuring IPMP in the Sun Cluster 3. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 9-20 Sun™ Cluster 3. Inc.2 Environment Understanding Standby and Failback Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved.2 Administration Copyright 2009 Sun Microsystems. Revision C a . Keep FAILBACK=yes in the /etc/default/mpathd file. Sun Cluster software is automatically trying to load-balance additional IP addresses across the members of the group.

Revision C 9-21 a . “Introducing Data Services. IPMP itself is unaware of anykof d requirements. “Configuring Scalable Services and Advanced Resource Group Relationships. the Sun Cluster 3. and HA-NFS. u a t y Instead. it consults a file named /var/cluster/run/pnm_callbacks. In the cluster environment.2 Software Environment There is nothing special about IPMP itself within the cluster. and manages failovers and failbacks between the adapters in a group. needs additional capability wrapped around IPMP to do the following: ● Make the status of IPMP groups on each node available throughout the cluster s ) ha y m ฺ om ฺ c ฺ csm Guide e t @ cluster n bthese e u Clearly. (There is more information about this in Module 9. however. but the corresponding group on the same subnet on another node has a healthy adapter ● IPMP groups ● Facilitate application failover When pnmd detects that all members of a local IPMP group have failed. Inc.2 Capabilities pnmd a n b e a lic Software b e u l k rab a y fe cluster environment.mpathd is concerned with probing network adapters only on a single Solaris OS.2 software uses a cluster-specific public ( is S n i network management daemon thto perform this cluster integration.2 Software Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. the pnmd daemon has the following ul nInsthe d b a tra capabilities: g n n no aba Communicate with pnmd on other nodes to transmit the status of ● Facilitate application failover in the case where all members of an IPMP group on one node have failed. This file contains entries that would have been created by the activation of LogicalHostname and SharedAddress resources. All Rights Reserved. Sun Services. as in a noncluster environment. The Sun Cluster software environment.Integrating IPMP Into the Sun Cluster 3. Oracle and/or its affiliatesฺ Integrating IPMP Into the Sun Cluster 3. lud s(pnmd) e a am to u k nofgthe se Daemon in Sun Cluster 3.” and Module 10. Resource Groups. in.”) Managing the Public Network With IPMP Copyright 2009 Sun Microsystems.

# cat /var/cluster/run/pnm_callbacks therapy orangecat-nfs. All Rights Reserved.2 Administration Copyright 2009 Sun Microsystems.mpathd tud y ( S in pnmdthis d pnmd u al use m ka e to g hafoip_ipmp_callback n ens baresources a failover lic b e u l k rab a y e f9-5 ul Figure s d IPMP Cluster Integration n b a a r t g onn a n ab 9-22 Sun™ Cluster 3. Inc.mon \ /usr/cluster/lib/rgm/rt/hafoip/hafoip_ipmp_callback mon nfs-rg orangecatnfs Summary of IPMP Cluster Integration Figure 9-5 summarizes the IPMP and pnmd elements of public network management in the cluster. in the following example. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u in. Revision C a . Sun Services.Integrating IPMP Into the Sun Cluster 3. Oracle and/or its affiliatesฺ It is the job of the hafoip_ipmp_callback. to decide whether to migrate resources to another node.2 Software Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.mpathd akin.

Node IPMP Group Status --Node Name --------vincent vincent theo theo Group Name ---------therapy therapy therapy therapy Status -----Online Online Online Online Adapter ------qfe2 qfe1 qfe2 qfe1 Status -----Online Online Online Online s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Managing the Public Network With IPMP Copyright 2009 Sun Microsystems.Integrating IPMP Into the Sun Cluster 3. All Rights Reserved.2 software environment. Sun Services. Revision C 9-23 a . # clnode status -m --. Oracle and/or its affiliatesฺ Viewing Cluster-wide IPMP Information With the clnode status -m Command In the Sun Cluster 3.2 Software Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. running the clnode status -m command from any node shows the status of IPMP group members on all the nodes. Inc.

It is assumed that the Sun Cluster software is installed and operational.Exercise: Configuring and Testing IPMP Exercise: Configuring and Testing IPMP Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. and that the only IPMP configuration is the single-adapter group called sc_ipmp0 automatically created by scinstall. Revision C a . k ng ense a b b a le licthe local-mac-address? Variable Task 1k–uVerifying a erab y l u nsf d b Perform the following steps on each node in the cluster: a a r t g n non 1. Oracle and/or its affiliatesฺ Perform the following tasks on all cluster nodes. you complete the following tasks: ● Task 1 – Verifying the local-mac-address? Variable ● Task 2 – Verifying the Adapters for the IPMP Group ● Task 3 – Verifying or Entering Test Addresses in the /etc/hosts File s ) ha y m ฺ Task 5 – Rebooting and Verifying That IPMP Is Configured om ฺ c ฺ e Task 6 – Verifying IPMP Failover and Failbackcsm uid e G t @ n b ku tude a y Preparation n ( this S i d lu se a No preparation is required am fortothisu exercise. If it needs to be changed to true. Sun Services. In this exercise.2 Administration Copyright 2009 Sun Microsystems.xxx Files ● ● true: # eeprom "local-mac-address?" The variable is set to true by the scinstall utility at cluster installation time. All Rights Reserved. so the only reason it should be false now is if it were changed manually. do so now. Verify that the EEPROM local-mac-address? variable is set to a b a ● Task 4 – Creating /etc/hostname. Inc. 9-24 Sun™ Cluster 3.

Verify with your instructor which IP addresses should be used for the test interfaces for each adapter on each node. Revision C 9-25 a . it is advisable to have all test addresses in the hosts file for all nodes to indicate that these addresses are reserved and what they are reserved for. While one node does not need to know anything about g n n a no 3. to have test IP addresses listed in d b a tra the hosts file. Oracle and/or its affiliatesฺ Perform the following steps on each node of the cluster: 1. Making sure it is not configured as a private transport om ฺ c ฺ b. Make sure you know which are the redundant adapters on the public network. All Rights Reserved.” 2.* # ifconfig -a s ) ha ab y m ฺ a. You can verify your secondary public network adapter by: another node’s test addresses. Inc.Exercise: Configuring and Testing IPMP Task 2 – Verifying the Adapters for the IPMP Group Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Making sure it can snoop public network idetraffic: csmbroadcast u e G # ifconfig ifname plumb b@ t n ku tude # snoop -d ifname (ya S npingth-sis pubnet_broadcast_addr i (other window or node)# d lu se a am to u k ng enorseEntering Test Addresses in the Task 3 – Verifying a b a File lic b /etc/hosts e u l k rab a y ul nItsisfea good idea. Perform the following steps: 1. You might have already written this down in the exercises for Module 3. You can verify this with: # ls -l /etc/hostname. Managing the Public Network With IPMP Copyright 2009 Sun Microsystems. “Preparing for Installation and Understanding Quorum Devices. Sun Services. although not required. Your primary public network adapter should be the only one currently configured on the public net.

qfe1 vincent group therapy netmask + broadcast + up addif vincent-qfe1-test -failover deprecated netmask + broadcast + up # vi /etc/hostname.194 vincent-qfe1-test 172. The following ak that tuared configured on your are just examples. The group nameuisbunimportant. Revision C a . While scinstall places the -failover flag on an non ab the node address.197 theo-qfe1-test 172. Oracle and/or its affiliatesฺ 2. Sun Services.4. All Rights Reserved.20.xxx Files s ) ha y m ฺ Perform the following task: om ฺ c ฺ sm uide c e On each node.4. you will be using it only with dedicated test addresses.20.qfe2 vincent-qfe2-test group therapy -failover deprecated \ netmask + broadcast + up 9-26 Sun™ Cluster 3.4. # vi /etc/hostname.195 vincent-qfe2-test 172.2 Administration Copyright 2009 Sun Microsystems. You will be completely f u s d this file to place the adapter in a real.20.Exercise: Configuring and Testing IPMP Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. so use the adapter(ynames S in this public network. Inc. It really does not matter if you use the same names as another group working on another cluster if the IP addresses are different: # IPMP TEST ADDRESSES 172. create the appropriate /etc/hostname. as in the examples.xxx a b u le was used to configure your cluster.20. multi-adapter IPMP an ab -troverwriting g group. d u al use m ka utility toshould have already modified the g e Note – The scinstall ns(one of the two below) that already existed at the ban licefile /etc/hostname. Your adapter was time scinstall k b a a in l yplaced eran IPMP group called sc_ipmp0.xxxG to place @ ent files adapters in IPMP groups.4. Enter the IP addresses in /etc/hosts on each node if they are not already there. The following is only an example.198 theo-qfe2-test Task 4 – Creating /etc/hostname.

Exercise: Configuring and Testing IPMP

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Task 5 – Rebooting and Verifying That IPMP Is
Configured
The following steps can be performed on one node at a time, so that the
cluster stays active the whole time:
1.

Reboot the node.

2.

Verify the new IPMP configuration with ifconfig -a.

3.

Verify IPMP cluster-wide status with clnode status -m.

Task 6 – Verifying IPMP Failover and Failback

s
) ha

y
m

Perform the following steps on at least one of your nodes: m
ฺco eand
ฺ keep it
m
1. From outside of the cluster, launch ping -ssnodename,
d
i
ec t Gu
running.
@
n
e
ubcluster
d
k
2. If you have physical access to a
your
hardware,
unplug the
u
t
y
(
S
Ethernet cable from the inetwork
n thadapter
is that currently has the node
d
physical interface on
it.
u
al use
m
a physical
If you havekno
to access, you can sabotage your adapter with:
g
e
# b
ifconfig
modinsert ldterm@2
an ceadapter_name
ns
a
i
l
b Observe
3.
u
le the node messages (on the console or in the
k
b
a
a
/var/adm/messages
file).
y fer
l
u
s
bd tran4. Observe the output of the clnode status -m command.
a
g onn
a
5. Observe the behavior of your command from step 1 (keep it
n
ab
running).

If you have physical access, reattach the broken cable. If you have no
physical access, use the following to repair your sabotage:
# ifconfig adapter_name modremove ldterm@2
6.

Observe the messages and the behavior of your command from
step 1.

7.

Observe the output of the clnode status -m command.

Managing the Public Network With IPMP
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

9-27

a

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Exercise: Configuring and Testing IPMP

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

9-28

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Module 10

Introducing Data Services, Resource
Groups, and HA-NFS
s
) ha

Objectives

ab

y
m

om ฺ
Upon completion of this module, you should be able to:
c

iindea cluster to
csmservice
u
Describe how data service agents enable aedata
G
t
@
operate properly
n
b
ku tude
a
y
List the components of a data
service
( is Sagent
n
i
d e thinstallation, and registration
Describe data service
lupackaging,
a
m topurpose
us of resource groups
aprimary
Describe the
k
ng ebetween
se failover and scalable data services
a
n
Differentiate
b
a
lic
b
e
u
l
k Describe
b how to use special resource types
a
a
r
y
ul nsfeList the components of a resource group
d
b
a
tra
Differentiate between standard, extension, and resource group
g
n
n
a
no







properties

Register resource types

Configure resource groups and resources

Perform manual control and switching of resources and resource
groups

10-1
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Relevance

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Relevance

?

!

Discussion – The following questions are relevant to understanding the
content of this module:

What is a data service agent used for in the Sun™ Cluster 3.2
software?

What is involved in fault monitoring a data service?

What is the implication of having multiple applications in the same
failover resource group?

How do you distinguish the resources representing different
instances of the same application?

y
m

om
What are the specific requirements for setting up NFS
in
the Sun
c


e
m
Cluster 3.2 software environment?
d
s
i
ec t Gu
@
b den
u
k
ya Stu
(
n
di e this
u
l
a us
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

s
) ha

10-2

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Additional Resources

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Additional Resources
Additional resources – The following references provide additional
information on the topics described in this module:

Sun Cluster System Administration Guide for Solaris OS, part number
820-7458 (for Sun Cluster 3.2 11/09).

Sun Cluster Software Installation Guide for Solaris OS, part number 8207356 (for Sun Cluster 3.2 11/09).

Sun Cluster Concepts Guide for Solaris OS, part number 821-0259 (for
Sun Cluster 3.2 11/09)

s
) ha

y
m

om ฺ
c

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

Introducing Data Services, Resource Groups, and HA-NFS
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

10-3

a

Introducing Data Services in the Cluster

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Introducing Data Services in the Cluster
The Sun Cluster 3.2 software framework makes applications HA,
minimizing application interruptions after any single failure in the cluster.
In addition, some applications, such as Apache web server software, are
supported not only with HA features, but also in a scalable configuration.
This configuration allows the service to run on multiple nodes of the
cluster simultaneously while providing a single IP address to the client.

Solaris 10 OS Non-Global Zones Act as Virtual Nodes
a
s
a
for Data Services
)h

y
m

m and
omodule
On the Solaris 10 OS, the entire framework described in this
c


e
m
d
s
the next module treats every non-global zone as a virtual
node.
That
is,
i
c applications.
u
e
G
non-global zones can host clustered failover and
scalable
@ ent
b
u
k tud
yathe
(
S
The intention of this feature is ton
allow
flexibility
of configuring nons
i
i
d
h
global zones on multiple nodes,
t your application fail over (or
lu andsehaving
a
be load balanced) between
non-global
zones
on different nodes. You
m to u
aapplications
k
might prefer to have
running
in
non-global zones for a
g
e
n
s
a including
variety of reasons,
n the security sandbox that they provide (you
b
e
c
a
i
l
could b
give application
the root password to a non-global
eare onlyadministrators
u
l
k
b
zone,
and
they
super
within
that
zone).
a era
y
l
f
ns it is not strictly the intention of this feature, you could even have
bdu trWhile
a
a
ang non applications fail over between non-global zones on the same physical

ab

node. With the zone integration feature, a data-service developer with
access to only a one-node cluster can experiment with real application
failover between non-global zones on the node.

Some applications, such as NFS, are not supported in non-global zones.
This is typically an application or OS restriction, not a cluster restriction.

Off-the-Shelf Application
For most applications supported in the Sun Cluster software environment,
software customization is not required to enable the application to run
correctly in the cluster. A Sun Cluster 3.2 software agent is provided to
enable the data service to run correctly in the cluster environment.

10-4

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

Introducing Data Services in the Cluster

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Application Requirements
You should identify requirements for all of the data services before you
begin Solaris OS and Sun Cluster software installation. Failure to do so
might result in installation errors that require you to completely reinstall
the Solaris OS and Sun Cluster software.

Determining the Location of the Application Binaries
You can install the application software and application configuration files
on one of the following locations:

The local disks of each cluster node

s
) ha

y
m

om ฺ
c

csm Guide
e
t you might
@ enzones,
b
If you are running applications inunon-global
k local tstorage
ud visible to the noninstall the application only inya
the
(
S
global zones.
in this
d
u
e file system using shared storage
al or failover
s
A global file system
m
u
ka e to binaries on a shared file system, you have
g
If you n
put
the application
ntosmaintain and manage. However, you must shut
baone lcopy
e
only
c
a
b downlethe idata service in the entire cluster to upgrade the application
u
k
a esoftware.
ab If you can spare a small amount of downtime for upgrades,
r
y
l
f
bdu trans place a single copy of the application and configuration files on a
a
shared global or failover file system.
ang non
Placing the software and configuration files on the individual cluster
nodes lets you upgrade application software later without shutting
down the service. The disadvantage is that you have several copies
of the software and configuration files to maintain and administer.

ab

Introducing Data Services, Resource Groups, and HA-NFS
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

10-5

a

Introducing Data Services in the Cluster

Sun Cluster 3.2 Software Data Service Agents

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

The word agent is an informal name for a set of components, written
specifically for Sun Cluster 3.2 software, that enable a data service in a
cluster to operate properly. Figure 10-1 shows the relationship between a
standard application and the data service agent.

Data Service
Agent
Standard
Application

start
stop

y
m

om ฺ
c

fault
sm uide
monitorsec
G
t
@
n
b
ku tude
a
y
n ( this S
i
d
e Data Service Agent
alu usand
Figure 10-1 Standard m
Application
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

s
) ha

10-6

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Reviewing Components of a Data Service Agent

Reviewing Components of a Data Service Agent
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Typical components of a data service agent include the following:

Methods to start and stop the service in the cluster

Fault monitors for the service

Methods to start and stop the fault monitoring

Methods to validate configuration of the service in the cluster

A registration information file that allows the Sun Cluster software
to store all the information about the methods into the CCR
You only need to reference a resource type to refer to all the
components of the agent.

s
) ha

ab

y
m

m
ofrom
c
Sun Cluster software provides an API that can be called
shell


e
m
d
s
i
programs, and an API that can be called from C or
programs.
Most of
u
ec C++
G
the program components of data service b
methods
that
are
supported
by
t
@
n
e
u
Sun products are actually compiledaCkand C++
tudprograms.
y
(
S
in this
d
u
al use
Fault Monitor Components
m
ka e to
g
ns
ban licecomponents
Fault a
monitoring
specific to data services in Sun Cluster 3.2
b
e
software
are
run
on
the
local
node
only. This is the same node that is
u
l
k
b
a
a
the data service. Fault monitoring components are intended to
l y running
erapplication
f
u
s
d
detect
failure and can suggest either application restarts or
b tranfailovers in the cluster.
a
ang non
The actual capabilities of these fault monitors are application-specific and
often poorly documented, the general strategy for fault monitors in the
Sun Cluster 3.2 environment is to monitor the health of the following:

The daemons, by placing them under control of the process monitoring
facility (rpc.pmfd). This facility calls action scripts if data service
daemons stop unexpectedly.

The service, by using client commands.

Note – Data service fault monitors do not need to monitor the health of
the public network itself because this is already done by the combination
of in.mpathd and pnmd, as described in Module 8, “Managing the Public
Network With IPMP.”

Introducing Data Services, Resource Groups, and HA-NFS
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

10-7

a

Introducing Data Service Packaging, Installation, and Registration

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Introducing Data Service Packaging, Installation, and
Registration
Agent packages released along with Sun Cluster are part of the Java™ ES
medium. You can install them using the Java ES installer at the same time
that you install the Sun Cluster framework, you can invoke the Java ES
installer at a later time to install them, or you can just use the pkgadd
command.
Some agents are supplied with the application software, rather than with
the cluster software.

as
h
If you intend to run certain clustered applications only in Solaris 10 non)
y
m
global zones, you might install the agents only in these non-global
zones.

oinmthe global
Alternatively, you might choose to install the agents always
c

ฺ to all
e
m
zone, without the -G option, and have them automatically
propagate
d
s
i
ec t Gu
existing and future non-global zones.
@
b den
u
k
ya Stu
(
n
Data Service Packages u
and
di Resource
his Types
t
l
e
a us
m
a
k encapsulates
to
Each data servicegagent
all the information about the agent
e
n
s
a
n
as a resource
type.
ab lice
b
u this resource
le type is registered with the cluster software, you do not
k
b
a
When
a
r
l yneedstofeknow
u
the location or names of the components of the agent. You
d
n need to reference
b tronly
a
a
the application’s resource type to determine all the
ang non correct information about methods and fault monitors for that component.

ab

Note – The package that you add by using the pkgadd command to install
an agent, and the corresponding resource type might have different
names. For example, when you install the SUNWschtt package, a resource
type called SUNW.iws becomes available.

10-8

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Introducing Resources, Resource Groups, and the Resource Group Manager

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Introducing Resources, Resource Groups, and the
Resource Group Manager
Data services are placed under the control of the cluster by configuring
the services as resources within resource groups. The rgmd daemon is the
resource group manager, which controls all activity having to do with
resources and resource groups, as shown in Figure 10-2. That is, the rgmd
daemon controls all data service activity within the cluster.

Resource
Group
Manager

s
) ha

y
m

om ฺ
controls
c

sm uide
c
e
G
Resource
t
@
n
b
Group
ku tude
a
y
n ( this Scontains
i
contains
d
lu se
a
am to u
k
Resource
Resource
g nse
n
a
ab lice
b
u
is a
leis a
k
b
a
a
l y sfer
u
d
Resource Type
Resource Type
ab -tran
g
n
n
no
aba
Figure 10-2 Resource Group Manager

Introducing Data Services, Resource Groups, and HA-NFS
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

10-9

a

Introducing Resources, Resource Groups, and the Resource Group Manager

Resources

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

In the context of clusters, the word resource refers to any element above
the layer of the cluster framework that can be turned on or off, and can be
monitored within the cluster.
An example of a resource would be an instance of a running data service.
For example, an Apache web server with a single httpd.conf file, counts
as one resource.
Each resource is an instance of a specific resource type. For example, each
Apache web server is an instance of SUNW.apache resource type.

s
) ha

y
m

om ฺ
c

csm Guide
A particular resource is identified by the following:
e
@ ent
b
u
Its resource type
ak Stud
y
(
A unique name, which is used
in Sun Cluster utilites
in as identification
is
d
h
t
u
e
al are
A set of properties,
which
parameters
that define the
s
m
u
a
o
k a particular
configuration of
t resource
g
e
n
s
a cen
b
a
limultiple resources of the same type in the cluster, either
You can
b configure
e
u
l
k
in
athe same
abor different resource groups.
r
y
l
e
f
s
nexample,
bdu trFor
a
a
you might want to run two failover Apache web server
g on- application services
n
that reside, by default, on different nodes in the
a
n
ab
cluster, but could fail over to the same node if there were only one node
Other types of resources represent IP addresses and data storage that are
required by the application service.



available. In this case, you have two resources, both of type SUNW.apache,
in two different resource groups.

Resource Groups
Resource groups are collections of resources. Resource groups are either
failover or scalable.

10-10

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Oracle and/or its affiliatesฺ Failover Resource Groups For failover applications in the cluster. Sun Services. and simultaneously fail over or be switched over to another node or nonglobal zone. Revision C 10-11 a . This is described further in Module 11. Resource Groups. and the Resource Group Manager Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. the resource group becomes the unit of failover. All Rights Reserved. Introducing Data Services.Introducing Resources. Scalable Resource Groups Scalable resource groups describe the collection of services that run simultaneously on one or more nodes or zones. Inc. the resource group is the collection of services that always run together on one node of the cluster at a given time. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a A scalable application in a scalable resource group also refers to an application making use of the load-balancing services built into Sun Cluster software. and HA-NFS Copyright 2009 Sun Microsystems. Resource Groups. Applications which are in scalable groups but do not make use of the built-in loadbalancing service are referred to as multimaster applications. That is.

and the Resource Group Manager Number of Data Services in a Resource Group Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. you can run two Sun Java System Web Server application services. All Rights Reserved. and is a failover. and a Sybase database as four separate resources in the same failover resource group. The services could still run on the same node. The rgmd ab 10-12 daemon launches all methods that start resources in resource groups and performs all failovers. For example. Oracle and/or its affiliatesฺ Multiple data services can be configured to run in the same resource group. because more n s i i d h predictable behavior. Some customers prefer services this ud to deployprovides ak this tconfiguration y ( S way. Resource Groups. in a single resource group. and always simultaneously migrate to another node. These applications always run on the same node at the same time.Introducing Resources. Or you can put all four of the preceding services in separate resource groups. a Sun Java System Directory Server software identity management service. Revision C a . you can predict that the new m to u asame k node will have the performance. assuming equivalent servers. the node that is not currently t the resource group @ hosting n b e u is in a pure backup mode. but could fail over and switch over independently. s ) ha y ฺm Benefits of Putting All Services in a Single Resourcem Group ฺco eฺ m s If you put all data services in the same failover c resource group uid of a e G two-node cluster. Inc.2 Administration Copyright 2009 Sun Microsystems. Sun™ Cluster 3. If the node currently running all the data services is t luthere e a s performing optimally. g e n s ba licen a b u le Manager k b Resource Group a a l y sfer u d ab -tranThe rgmd daemon maintains all the information about data services that g an non are known to the cluster as resources in resource groups.

Resource Groups.HAStoragePlus l a us m a k e to g ns Host Resource ban liceLogical a b u le k b a a Resource Type: y fer l u SUNW. Introducing Data Services. they move together. Configured resources are added to the resource group. Figure 10-3 shows a typical failover resource group. By placing these resources into the same failover resource group. not merely unique within a particular resource group.LogicalHostname s bd tran a g onn a n ab Failover Resource Group Resource Type: SUNW. Oracle and/or its affiliatesฺ Describing Failover Resource Groups The following example is provided to help you understand the general concept of resources as part of specific resource groups. All Rights Reserved. Inc. s ) ha Application Resource y m ฺ om ฺ c ฺ csm Guide e b@ dent Data Storage Resource ku ya Stu ( n Resource iType: d e this u SUNW. Sun Services. A resource name must be globally unique.nfs Figure 10-3 Failover Resource Group This failover resource group would be defined with a unique name in the cluster.Describing Failover Resource Groups Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 10-13 a . and HA-NFS Copyright 2009 Sun Microsystems.

such as an instance of the ORACLE database.nfs:3. For example.Describing Failover Resource Groups Resources and Resource Types Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.nfs ● nfs:3. For that matter. Revision C a . There is no such thing as a disembodied resource that is not a member of a resource group. Some applications that are typically considered to be a single entity. s ) ha y m ฺ om ฺ c ฺ csm Guide e ta few other @services. actually require two different resources with different types: the ORACLE server and the ORACLE listener. Oracle and/or its affiliatesฺ Each instance of a data service under cluster control is represented by a resource within a resource group. old and new versions of data service agents r y l resource f s nco-exist bdu trcan as separate types.2 ● nfs Sun™ Cluster 3. Each resource has a resource type that describes the type of resource it is (for example. d h t u al use m ka e to g Resource TypenVersioning ba licens a b 3.2.nfs:3. the version suffix can be dropped if there is no ambiguity. At least one defined resource type exists for each supported service in the cluster. Sun Services. All Rights Reserved. SUNW.2lesoftware gives you the ability to use different versions of u Sun Cluster k a etypes. all of the following names can be used to refer to the nfs resource type: 10-14 ● SUNW. ab For example. Inc. y ( S types are described in ‘‘Using Special in Resource is Types’’ on page 10-15. When using a resource type name.2 software. ab Officially.2 ● SUNW. Resources exist only inside resource groups. For example. when you initially install Sun Cluster 3. the official name of the NFS resource type is SUNW. In addition to resource types that relate tou data n b e d These resource special resource types relate to IP addresses ak and tustorage. the vendor prefix can also be dropped. the version number is appended to the resource type name.2 Administration Copyright 2009 Sun Microsystems.nfs for an NFS resource). Individual resources of an original resource a a g an non type can be upgraded to a new type on a resource-by-resource basis.

Resource Groups. This IP address is configured on the public net on only one node or zone with failover capability. c ฺ ฺ IP address is automatically configured in the appropriate m zone. Inc.SharedAddress represent a special type of IP an non address that is required by scalable services. Oracle and/or its affiliatesฺ Using Special Resource Types The following sections describe special resource types that you include in resource groups to complete the configuration. Sun Services. The SUNW. instance of a data of which physical node or zone is o k g e actually running service. each IP address described by a SUNW. This subject is described in greater detail in Module 11. along with the services for that group. ideThe clients cscluster. but provides a load-balanced IP address that supports scalable applications that run on multiple nodes or zones simultaneously.SharedAddress er f u s d ab -tran g Resources of type SUNW. The n ( along s i i client is that IP addresses migrate with d h t lu logical e clients always use the a same IP address to contact a particular s m tregardless u aservice. ban licthe ens a b u le k b a a The Resource Type l y SUNW. Introducing Data Services.LogicalHostname resource ) y If the migrates from node to node.Using Special Resource Types Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ab as h That is. u use these IP addresses to access the services inethe G t @ n b de transparent to the ku relatively u a t A large part of what makes cluster failover y S services of the group.LogicalHostname Resource Type Resources of type SUNW. All Rights Reserved. Revision C 10-15 a . m ฺ group is mastered by a non-global zone rather than a physical the om node.LogicalHostname represent one or more IP addresses on a particular subnet that will become the logical IP addresses used to access the services in that resource group. and HA-NFS Copyright 2009 Sun Microsystems.

which include failing over from a failed node. Revision C a . ● ● ● The SUNW. It is the resource methods that provide the failover of the file system.HAStoragePlus resource type manages the following types of storage: ● Global raw devices ● Global file systems ● Traditional (non-ZFS) failover file systems ● ZFS zpools. ab Global raw devices and global file system management. c a i l b u Usually. All Rights Reserved. are part of the Sun Cluster software framework. but serves the following di euseful hispurposes: t u l The START methodmofaSUNW. managing global storage withyaaSUNW. Oracle and/or its affiliatesฺ The SUNW. So it might seem redundant to provide a resource type that also manages these global devices and file systems. including all the datasets within Global Raw Devices and Global File Systems s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u k tud In fact.HAStoragePlus resource. but can fail over only to nodes physically connected to the storage.Using Special Resource Types The SUNW. you le use the Resource_dependencies standard property to k b a a l y sconfigure er a dependency so that the application resources depend on f u d SUNW.HAStoragePlus to manage a failover file system.2 Administration Copyright 2009 Sun Microsystems.HAStoragePlus resource ( S n is optional.HAStoragePlus is used to check if the us a o k t global devices g or nfilesesystems in question are accessible from the n a node where b the eresource group is going online. 10-16 Sun™ Cluster 3. Failover File Systems You must configure a SUNW. the resource group ab -tran the g manager does not try to start services if the storage the services n n o a n depend on is not available. Inc.HAStoragePlus resource type can also be used to attempt co-location of resource groups and device groups onto the same node. Sun Services. thus enhancing the performance of disk-intensive data services. A failover file system must fail over together as the service fails over. In this way.HAStoragePlus Resource Type Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

HAStoragePlus methods distinguish between global and failover file systems using the /etc/vfstab entry. especially for services that are file system intensive. ● ● ● ● ab When to Use a Failover File System Use a failover file system if all of the following are true: ● The file system is for a failover service only. Oracle and/or its affiliatesฺ Using Special Resource Types The configuration of a global or failover file system looks very similar. and HA-NFS Copyright 2009 Sun Microsystems. Both use the FilesystemMountPoints property to identify the file system or file systems in question. Revision C 10-17 a .Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. The SUNW. Sun Services. as a ang non performance benefit. that is. Guidelines for Using Global and Failover File Systems These are some general guidelines about how to choose between global and failover file systems. ● Only services in a single resource group are using the file system. If these conditions are true. All Rights Reserved. Introducing Data Services. Resource Groups. a failover file system provides a higher level of performance than a global file system. When to Use a Global File System s ) ha y m ฺ Use a global file system if you want to support any of the following: om c ฺ ฺ e m A scalable service d s i c Gu e t physically @ A failover service that must fail over to a node not n b e u connected to the storage ak Stud y ( s for different failover services incontains A single file system that d hidata t u l e a on udifferent that may be running nodes s m a o k dataefrom t a node that is not currently running the Access to the g n s service ba licen a b u le k b a a If your storage is physically connected to all possible nodes for the y fer l u s group containing the HAStoragePlus resource. you can migrate bd tranresource a the underlying ownership of the storage along with the service. Inc. if the Nodelist for the resource groups is the same as the Nodelist for the device group. ● The Nodelist property for the resource group contains only nodes that are physically connected to the storage.

10-18 Sun™ Cluster 3. Inc. For ZFS. an HAStoragePlus instance represents one or more zpools and all the file systems within. but not both. HAStoragePlus and ZFS The same HAStoragePlus resource that provides failover of traditional filesystems can also provide ZFS failover.ZFS technology is zone-aware. The GDS composes a fully functional Sun Cluster Resource Type complete with callback methods and a Resource Type Registration file. s ) ha y m ฺ m have to oyou To manage failover of ZFS filesystems within the cluster. an cenorsmultiple ZFS zpools. Oracle and/or its affiliatesฺ HAStoragePlus Resources in Zones When a traditional (non-ZFS) file system that is controlled by an HAStoragePlus resource is contained in a resource group that is mastered by a non-global zone. When you b a use thebZpools property the values of the properties for traditional li e u l k filesystems and AffinityOn) are ignored.Using Special Resource Types Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ฺall c ฺ e m d s do is configure an HAStoragePlus instance with the value of the Zpools i c over. Revision C a . Generic Data Service The Generic Data Service (GDS) is a resource type for making simple applications highly available or scalable using the resource type SUNW. Sun Services. They ab are not visible in the global zone.gds by plugging them into the Sun Cluster RGM framework. the file system is always mounted in the appropriate global zone. e u need to configure any /etc/vfstab entries d k tuconfiguration database.2 Administration Copyright 2009 Sun Microsystems. and then made available to the non-global zone through a loopback mount. and ZFS zpools that are configured into a ang non non-global zone via HAStoragePlus are only exposed in that zone. b a era(FilesystemMountPoints y l f bdu trans a . yathe zpool mount information is self-contained(in S n di e this u l a us m a Note – A single HAStoragePlus k e to instance can refer to multiple traditional g (non-ZFS) filesystems. All Rights Reserved.GYou u do not efail property indicating one or more pools that should t @ n All of the ZFS bwhatsoever.

and how to stop this application are just properties that can be set differently for each instance of SUNW.gds. DHCP. what is being launched. All Rights Reserved. y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Introducing Data Services. Sun Services. and many more) do not actually supply new resource types. Resource Groups. Instead. Revision C 10-19 a . Samba. how to probe this application.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. and HA-NFS Copyright 2009 Sun Microsystems. s ) ha In the lab exercises for this module. they supply configuration scripts that configure resources to represent the applications as instances of SUNW. Inc. Oracle and/or its affiliatesฺ Using Special Resource Types Basically.gds. All you have to do is have a resource type where what is being launched is not implied by the resource type itself.gds type to put your own customized application under control of the Sun Cluster software. Many Sun-supported data services (for example. you will get an opportunity to use an instance of the SUNW. Rather. the concept is that many different applications can share the same resource type.

10-20 Sun™ Cluster 3. then there is one very hard omandฺ fast c rule (known henceforth as the “prime directive of regular ฺdependencies”): sm uide c e Resource A cannot be brought online unless resource GB is already t @ n b online. Revision C a . ku tude a Note this has some implications: y n ( this S i d The cluster must serialize lu sestarting A and B (waiting to start A a until B is online) u where otherwise they could be started am intcases o k in parallel. Inc. Regular Resource Dependencies s ) ha y m ฺ If Resource A depends on resource B. If B fails to start. This is somewhat counterintuitive. Weak Dependencies (Weaker Than Regular Dependencies) If resource A has a weak dependency on resource B. All Rights Reserved. even if n o a n b resource A is still running. the rgmd will still try to start A. but does not enforce the dependency in other ways. but because a ● ● ● you are explicitly issuing the command (clrs disable) to disable the dependee. you can do it. rgmd preserves the order of the stops and starts as if it were a regular dependency. You can also configure group dependencies that take into account only the state of a group rather than the state of its resources. disable resource B. d b a a trYou g n are allowed to specifically. does not explicitly forbid you from stopping the resources in the “wrong order”. Note the prime directive (the first bullet item above). manually.2 Administration Copyright 2009 Sun Microsystems.Understanding Resource Dependencies and Resource Group Dependencies Understanding Resource Dependencies and Resource Group Dependencies Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ng ense a b aThe cluster lic must serialize stopping A and B (waiting to stop B b e u l k runtil b A is stopped) in cases where otherwise they could be a a y ul nsfe stopped in parallel. Sun Services. There are four levels of resource dependencies. You can manually disable either resource or its group freely. Relationships between individual resources can be between resources in the same or different resource groups. Oracle and/or its affiliatesฺ You can declare dependency relationships between resources.

ku until therefore. This type of ab -tranGroup g dependency considers the state of the group only rather than the an non resources inside it. while leaving resource A running. therefore. n b deB actually restarts. f u s d H cannot be brought offline unless group G is offline. Oracle and/or its affiliatesฺ Restart Dependencies (Stronger Than Regular Dependencies) Restart dependencies have all the attributes of regular dependencies. Similar to the case of regular dependencies. Note – Resource group dependencies are somewhat limited. Resource Groups. and HA-NFS Copyright 2009 Sun Microsystems. group G cannot be brought online unless group H is online. In this slight variation. the restart dependency begins. All Rights Reserved. They are enforced between groups running on different nodes when you manually try to start or stop resource groups in the wrong dependency order. However. Inc. regardless of which nodes the groups are mastered. when RGM is informed thatm the de issues a s theuiRGM coffline. Resource A explicitly depends on resource B. and the RGM restarts resource A Offline-Restart Dependencies (Slight Variation of Restart Dependencies) y) has ab ฺm m o ฺ ฺcdependee. resource B. resource group dependencies are made obsolete by the various flavors of resource dependencies. with the additional attribute that if the RGM is told to restart resource B. you are allowed to manually. Sun Services. resource B. the RGM will automatically restart resource A. is offline. when you manually reenable resource B.Understanding Resource Dependencies and Resource Group Dependencies Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. They consider only the group state and not individual resource state. resource A’s start will be blocked u a t y n ( this S i d lu se a m to u Resource Group aDependencies k ng ense a b a grouplicdependencies imply an ordering relationship between Resource b u leIf resource group G has a group dependency on resource two groups: k b a a l y group erH. or when it is put explicitly e G t @ restart on resource A. Revision C 10-21 a . specifically disable the dependee. now that they can be between resources in different groups. or is informed that some agent did the restart of resource B itself. Introducing Data Services.

because it is cluster-unaware software. All Rights Reserved. Scripts specifically used to run the service in the cluster can read these property values and use them to locate configuration files. a erab y l u nsf d b a tra Resource Properties g n n Standard no aba The names of standard resource properties can be used with any type of resource. Properties consist of a set of name=value pairs. or can use them to pass command-line parameters to the actual services. Fortunately. You can access a full list of standard resource properties and their general meanings by typing: # man r_properties Of the dozens of properties listed. Inc.2 Administration Copyright 2009 Sun Microsystems. most administrators b e c a li environments never need to deal with the of SunbCluster software e u l k properties. There is no way they can be accessed directly by the application software itself. only a handful are critical for a particular resource type. Revision C a . y m ฺ om ฺ c ฺ sm uide c e G Other properties are essential only for data service fault tmonitoring. These properties are used by the data service agent and by rgmd.Configuring Resource and Resource Groups Through Properties Configuring Resource and Resource Groups Through Properties Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. n ( this S i d lu dozens e of properties. @ n b Misconfiguring these properties might leave de running fine. Sun Services. Other properties can be ignored. Oracle and/or its affiliatesฺ You configure specific resources within the cluster by defining values for resource properties and resource group properties. 10-22 Sun™ Cluster 3. Therefore. but ku thetuservice a y cause fault monitoring to fail. many a s Each resource can literally have m u kafor particular to types of resources are automatically important properties g e provided with anreasonable nsdefault values. s ) ha Some properties are essential for running the service in the cluster. or can have default values that are unlikely to be changed.

an ab -trResource_dependencies_offline_restart g Resource_dependencies_restart=resB an non Resource_dependencies_offline_restart=resB Introducing Data Services. but @ b den dependency. Inc. Revision C 10-23 a . set the Resource_dependencies_weak propertyidon m s u ec nottaGreal resource A. (Recall that this implies an ordering.Configuring Resource and Resource Groups Through Properties Some Significant Standard Resource Properties Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. which imply ordering. set the u To k set b a a or l yResource_dependencies_restart er f u s d property on resource A. they can have meaningful values for many different types of resources.) u k ya Stu ( Resource_dependencies_weak=resB n di e this u l a us m a The Resource_dependencies_restart and k e to g Resource_dependencies_offline_restart Properties ban licens a bthe twolkinds e of restart dependencies. that is. are configured using standard properties. so that resource A has a weak odependency c ฺ ฺ e on resource B. Resource Groups. The Resource_dependencies Property Resource dependencies. All Rights Reserved. and HA-NFS Copyright 2009 Sun Microsystems. Resource_dependencies=nfs-stor The Resource_dependencies_weak Property s ) ha ab y m ฺ m To set up a weak dependency. Sun Services. Oracle and/or its affiliatesฺ The following properties are standard resource properties.

u l k b to another ya fera l u RESTART_ONLY No Yes bd transOther resources in The a the same resource STOP_FAILED g n non a group can still start flag is set on b a NONE LOG_ONLY Other resources in the same resource group can still start (if nondependent). g anresource HARD The whole Yes Yes ns The node b e c a i l group b is switched enode. Inc. The STOP_FAILED flag is set on the resource. it must be manually cleared by using the clrs clear command before the service can start again. e aflag m u ka ethetoresource. Oracle and/or its affiliatesฺ The Failover_mode Property The Failover_mode property describes what should happen to a resource if the resource fails to start up or shut down properly. reboots.2 Administration Copyright 2009 Sun Microsystems. Sun Services. (if nondependent).Configuring Resource and Resource Groups Through Properties Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. # clrs clear -n node -f STOP_FAILED resourcename 10-24 Sun™ Cluster 3. the resource. Other resources in the same resource group can still start (if non-dependent). If the STOP_FAILED flag is set. All Rights Reserved. Table 10-1 describes how values of the Failover_mode property work. Revision C a . The STOP_FAILED flag is set on the resource. Table 10-1 The Failover_mode Value Operation Value of the Failover_mode Property Failure to Start Failure to Stop Can Fault Monitor Cause RGM to Fail RG Over? Can Fault Monitor Cause RGM to Restart the Resource? Yes Yes No No s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak YesStud Yes SOFT The whole resource The y ( group is switched STOP_FAILED in this d u l is setson to another node.

There is no hard-and-fast rule about extension properties. Revision C 10-25 a . if you are setting up the Apache web server. Resource Group Properties Certain properties apply to an entire resource group. The y fer l u s property (Zpools) is used only for specifying pool names for a bd tranthird a failover ZFS resource. Oracle and/or its affiliatesฺ Extension Properties Names of extension properties are specific to resources of a particular type. For example: # man SUNW. Resource Groups. Confdir_list=/global/dnsconflocation Many have other ways of identifying their configuration and data. s ) ha ab y m ฺ m o c ฺ ฺ FilesystemMountPoints=list_of_storage_mount_points e m d s i AffinityOn=True/False ec t Gu @ -------------------b den u k Zpools=list_of_zpools ya Stu ( n di properties his to identify which storage t u l Use the first of these extension e a The m ussecond extension property is a parameter resource is being described. and HA-NFS Copyright 2009 Sun Microsystems.apache # man SUNW. in which case the first two properties are not used. which points to the directory containing the apachectl script that you want to use to start the web server. You can get information about extension properties from the man page for a specific resource type. You can get information about these properties by typing: # man rg_properties Introducing Data Services. All Rights Reserved. The HAStoragePlus type has the following important extension properties regarding filesystems. Inc. a o k t g nframework that tells the cluster to switch physical control of the storage se the service. annode bthe e groupato running Switching control to the node that c i l b isurunning lthe e service optimizes performance when services in a single k b a a failover resource group are the only services accessing the storage. Sun Services.HAStoragePlus For example.Configuring Resource and Resource Groups Through Properties Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ang non Many resource types (including LDAP and DNS) have an extension property called Confdir_list that points to the configuration. you must create a value for the Bin_dir property.

but not necessarily on the same node. g the e n s a n it is possible for the same groups’s node list to include multiple zones on b lice a the same b physical e node. Sun Services. Inc. In other words. 10-26 Sun™ Cluster 3. no services can start if the logical IP addresses cannot be brought online. luA non-global e a s from the point of view of a resource group node list. then resource group A cannot be brought online unless resource group B is online. Oracle and/or its affiliatesฺ The following properties are associated with a resource group. t @ n b ku tude a y (an ordered S list. If resource group A has the property RG_dependencies=rgB. Resource group B must be brought online somewhere. This property makes all the services in the resource group dependent on all the LogicalHostname and SharedAddress resources. As mentioned before. All Rights Reserved. rather than an individual resource.2 Administration Copyright 2009 Sun Microsystems.Configuring Resource and Resource Groups Through Properties Some Significant Resource Group Properties Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a . the resource group automatically switches back when a preferred node or zone (earlier in the node list) joins the cluster. m to u a k while it is not strictly intention of the cluster zone integration feature. The Implicit_network_dependencies Property The Implicit_network_dependencies property is TRUE by default. from most preferred The value of this property is always n s i i d h t zone acts as a virtual node node to least preferred node. The RG_dependencies Property The RG_dependencies property will be set on a resource group to describe its dependency on another group. s ) ha y m ฺ om ฺ c ฺ ide or The Nodelist property for a resource group describes what u nodes csm e G zones the resource group can run on. u l k b ya fera l u nsFailback Property bd trThe a a ang non The Nodelist Property ab If the Failback property is TRUE (not the default). You can set the Implicit_Network_Dependencies property to FALSE if you have a service that does not depend on logical IP addresses.

which uses the property to find its dfstab file. NFS needs the aba dfstab file so that it knows what to share. NFS also uses the same arena to store file lock state information. the resource checks that the Pathprefix property is set on its resource group. then the RGM does not consider that node or zone a candidate for the group failover. u ude k a t from causing endless ping-ponging (y between S nodes or zones.nfs/dfstab. Oracle and/or its affiliatesฺ This property controls resource group behavior in a couple of different ways: ● If a resource group fails to start twice on the same particular node or zone (failure of START methods of same or different resources in the group) within the interval (expressed in seconds). the RGM rejects the second failover if it is within the interval. Sun Services. s ) ha y m ฺ om ฺ c ฺ idefaulty start csm u e Note – The Pingpong_interval property is meant to prohibit G t @ n b scripts or properties and faulty fault monitors. and HA-NFS Copyright 2009 Sun Microsystems. the only data service that must have the Pathprefix property g n n no set is NFS. Introducing Data Services. and that the dfstab file exists. File lock state information is typically stored in the /etc directory on a stand-alone NFS server.resource_name When you create an NFS resource. Inc. or problem applications. ● If the fault monitor for one particular resource requests that a group be failed off a particular node or zone.Configuring Resource and Resource Groups Through Properties The Pingpong_interval Property Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 10-27 a . but it must be in the shared storage in the cluster. An NFS resource must have its dfstab file named: value_of_Pathprefix_for_RG/SUNW. and then the fault monitor for the same resource requests another failover that would bring the group back to the original node or zone. Resource Groups. All Rights Reserved. n s i i lud se th a u am o The Pathprefix Property k t ng ense a b a ThebPathprefix licproperty points to a directory in a shared-storage file e u l ksystemrathat b is used for the administrative purposes of services in the a y fe group. ul nresource s d b a tra Currently.

. All Rights Reserved.. n b desame or different kunodes u a t You can easily give zones on different the y S n ( tothbeisdifferent i zone names. In u d ab -tran other words. Oracle and/or its affiliatesฺ You may specify non-global zone names in the place of node names in the following contexts: ● When specifying or modifying the node list for a resource type ● When specifying or modifying the node list for a resource group ● When switching a resource group to a particular node ● When enabling or disabling a resource or resource flag on a particular node s ) ha y m ฺ om ฺ c ฺ idae The form nodename:zonename is used to specify a zoneuon csm e G t @ particular node. Revision C a ...node2. Sun Services. They are considered virtual nodes by the d u l e cluster either way.. ab lice b uThe syntax le-n node1.node2:zonename. but there are some general rules in using zone names: ● ● ● ● 10-28 Sun™ Cluster 3.2 Administration Copyright 2009 Sun Microsystems. it is a shorthand you when specifying multiple zones g n non on different nodes that have the same zone name.. a m to us a k The syntax g -n nodename e -z zonename is identical to n s a n -n nodename:zonename.Specifying Non-Global Zone Names in Place of Node Names Specifying Non-Global Zone Names in Place of Node Names Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. a b a The commands to perform these operations are presented in the next few sections. Inc. -z zonename k b a a l y sisfea rshorthand for -n node1:zonename.

The only property of a registered resource type that you can modify with the clrt command is the node list. you are allowing that type to be used on all nodes and non-global zones. All Rights Reserved. a a r y fe register SUNW. you are restricting the resource type to certain nodes or zones. Inc. and HA-NFS Copyright 2009 Sun Microsystems. Yo Note – You can specify a zone with the syntax nodename:zonename. The second c a li b e u l kexample blimits the node list to only the nodes mentioned. In other words.Using the clresourcetype (clrt) Command Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 10-29 a . The assumption is that you will have the data service agent installed on all nodes. the default is to place no restriction on where resources of this registered type could run. Oracle and/or its affiliatesฺ Using the clresourcetype (clrt) Command You use clresourcetype (clrt) to register or unregister a resource type.apache g n n no aba If you do specify a node list.nfs ul n#sclrt d b a tra # clrt register -n vincent. even ones that are added and created in the future. The cluster then allows you to specify only those nodes or zones for the resource groups that contain resources of that type. If you register a type and do not specify a node list. Sun Services. The first an examples nnode bspecifies e example a list of all current and future nodes. Introducing Data Services.theo SUNW. Registering a resource type makes it known to the CCR and available to be used for new resources. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u Registering Resource Types al use m ka e to g The following s show registration of resource types. Resource Groups.

apache:4.SharedAddress:2 SUNW.Using the clresourcetype (clrt) Command Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a .2 SUNW. Sun Services. Inc. All Rights Reserved. Global_zone (False) means that methods for the resource type will run in a nonglobal zone when the type is configured in a non-global zone. it can be failover or scalable).1 s m API_version: 2 ka e to /opt/SUNWscapc/bin g RT_basedir: ban licens Single_instance: False a b Proxy: False u le k b a a Init_nodes: All potential masters y fer l u s Installed_nodes: vincent theo bd tran a Failover: False g onn a SUNWscapc n ab Pkglist: RT_system: Global_zone: False False Note – The Failover (False) means that the resource type is not limited to being failover only (that is. Oracle and/or its affiliatesฺ Viewing Registered Resource Types clrt list -v shows the list of registered resource types and their node lists.2 Administration Copyright 2009 Sun Microsystems.HAStoragePlus:8 SUNW. Unregistering Types You can unregister only types for which there are no remaining resources: # clrt unregister nfs 10-30 Sun™ Cluster 3.nfs:3.1 Node List --------<All> <All> <All> <All> vincent theo s ) ha y m ฺ vincent:/# clrt show apache om ฺ c ฺ csm Guide Registered Resource Types === e @ ent b u Resource Type: SUNW. clrt show can display more about a specific registered type or all registered types: vincent:/# clrt list -v Resource Type ------------SUNW.LogicalHostname:3 SUNW.apache:4.1 ak Stud y ( RT_description: s Server on Sun Cluster inApache iWeb d h t u RT_version: e al u4.

]] RG_name clrg delete RG_name clrg set [-p property [….Configuring Resource Groups Using the clresourcegroup (clrg) Command Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Configuring Resource Groups Using the clresourcegroup (clrg) Command You must create a resource group before creating resources in it.. you can only delete a resource group that contains no resources.theo:frozone ng en-zsefrozone ora-rgora-rg a b # clrg create -n vincent.. Inc. All Rights Reserved.theo a lic b e u l k rab a y fe– If you omit the node list when creating a group. it defaults to all ul nNote s d b a tra physical nodes of the cluster. in anticipation of placing an NFS resource in the group: s ) ha y m ฺ om ฺ c ฺ # clrg create -n vincent. The Pathprefix property is set.. In addition. and none of the non-global zones. the following command creates a resource group nfs-rg that runs on two physical nodes (not on non-global zones). Use the following command syntax: clrg create [-n nodelist] [-p property […. Sun Services.theo -p Pathprefix=/global/nfs/admin de csm Guinfs-rg e t @Both especify n b u The following two commands are identical. a new resource k tud a y group that specifies two non-global zones on two different nodes as the ( is S n i potential masters. Some examples follow. You can duse theesecond th syntax because the non-global lusame a zones happen to have the name: am to us k # clrg create -n vincent:frozone. Resource Groups.]] RG_name For example.. and HA-NFS Copyright 2009 Sun Microsystems. Revision C 10-31 a . # clrg show nfs-rg Resource Groups and Resources === Resource Group: RG_description: RG_mode: RG_state: Failback: nfs-rg <NULL> Failover Unmanaged False Introducing Data Services. g n n a no ab Displaying Group Configuration Information You can display resource group information with clrg list or clrg show.

Oracle and/or its affiliatesฺ Resource Groups and Resources === Resource Group: RG_description: RG_mode: RG_state: Failback: Nodelist: ora-rg <NULL> Failover Unmanaged False vincent:frozone theo:frozone # clrg show -p pathprefix nfs-rg s ) ha y m ฺ om ฺ c ฺ Resource Group: nfs-rg Pathprefix: /global/nfs/admin csm Guide e @ ent b u # clrg show -v nfs-rg ak Stud y ( in this d Resource Groups and Resources === u al use m ka e to Resource Group: nfs-rg g n s a n RG_description: <NULL> ab lice RG_mode: Failover b e u l k b RG_state: ya Unmanaged ra l e f u RG_project_name: default s bd tran a RG_affinities: <NULL> g n n RG_SLM_type: manual no aba Auto_start_on_new_cluster: True Resource Groups and Resources === Failback: Nodelist: Maximum_primaries: Desired_primaries: RG_dependencies: Implicit_network_dependencies: Global_resources_used: Pingpong_interval: Pathprefix: RG_System: Suspend_automatic_recovery: 10-32 False vincent theo 1 1 <NULL> True <All> 3600 /global/nfs/admin False False Sun™ Cluster 3. Sun Services.2 Administration Copyright 2009 Sun Microsystems. All Rights Reserved.Configuring Resource Groups Using the clresourcegroup (clrg) Command Nodelist: # clrg show ora-rg vincent theo Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. Revision C a .

ane IPV6 ฺ m d s i address. e u d k tuof Hostnamelist. respectively. m a k e toNetIfList. Oracle and/or its affiliatesฺ Configuring a LogicalHostname or a SharedAddress Resource The commands clreslogicalhostname (clrslh) and clressharedaddress (clrssa) are special commands for creating and modifying LogicalHostname and SharedAddress resources. or both. all the defaults are used. that the resource name is also yathe value ( S n is to use on each node.myother-ip \ -N therapy@vincent.ipmp_grp@node_id. in the dofi which hadapters NetIfList: Indication t u l e a us format: ipmp_grp@node_id. these commands deal with two special properties of these IP resources and can simplify the administration of these properties: ● Hostnamelist: List of IPs (on a single subnet) corresponding to this resource as h Multiple logical IP resources on different subnets must be separate ) y resources.therapy@theo orangecat-nfs Introducing Data Services. Inc. although they can still be in the same resourceฺm group.. Sun Services. om c ฺ Each name in Hostnamelist can be an IPV4 address.Configuring a LogicalHostname or a SharedAddress Resource Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. e u l k a ederived. the value of NetIfList is automatically b Hostnamelist. The HostnameList value is the same as the name of the resource. All Rights Reserved. ang non ● ab Examples of Using clrslh to Add a LogicalHostname In the first example. Revision C 10-33 a . ec t Gu @ ncommands assume b these If you do not specify the Hostnamelist. the commands try to figure out if g If you n do not specify nsIPMP group per node on the subnet indicated by bais only e there one c a li If so. The command has shorthand options (-h and -N) for supplying the values: # clrslh create -g nfs-rg -h orangecat-nfs. and HA-NFS Copyright 2009 Sun Microsystems. ab When you use the -N shorthand of the clrslh command r y l f bdu trans you can use the node names. Specifically. but the actual value of the property will a contain the node id’s.. # clrslh create -g nfs-rg orangecat-nfs The second example provides values for the two special properties. Resource Groups.

2 nfs-rg abnfs-res # clrs show nfs-res Resources === Resource: Type: Type_version: Group: R_description: Resource_project_name: Enabled{vincent}: Enabled{theo}: Monitored{vincent}: Monitored{theo}: 10-34 nfs-res SUNW. clrs create -t resource_type_name -g RG_name [-p property […]] res-name clrs set [-p property […]] res-name clrs delete res-name For example.HAStoragePlus:8 g orangecat-nfs SUNW. the following adds an HAStoragePlus and an nfs resource: # clrs create -t HAStoragePlus -g nfs-rg -p AffinityOn=true \ -p FilesystemMountPoints=/global/nfs nfs-stor s ) ha y m ฺ omnfs-res # clrs create -t nfs -g nfs-rg -p Resource_dependencies=nfs-stor c ฺ ฺ e m d s i ec t Gu @ b den u k tu Displaying Resource Configuration Information ya S ( n di e this u l a clrsusshow to display the configuration of You can use clrs list and m a k theeIPtoresources): any resource (including g ban licens # clrs list -v a b le Type Resource Name ku Resource Resource Group b a a r y -------------------------------------l e f u s d nfs-stor nfs-rg ab -tran SUNW.LogicalHostname:3 nfs-rg n n o a n SUNW. and change properties of resources. Revision C a .nfs:3.2 3.2 Administration Copyright 2009 Sun Microsystems. Inc.2 nfs-rg default True True True True Sun™ Cluster 3. Oracle and/or its affiliatesฺ Configuring Other Resources Using the clresource (clrs) Command Use the following command syntax to create. delete. All Rights Reserved. Sun Services.Configuring Other Resources Using the clresource (clrs) Command Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.nfs:3.

Configuring Other Resources Using the clresource (clrs) Command # clrs show -p resource_dependencies nfs-res Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C 10-35 a . Sun Services. All Rights Reserved. Oracle and/or its affiliatesฺ Resources === Resource: Resource_dependencies: nfs-res nfs-stor --. and HA-NFS Copyright 2009 Sun Microsystems. Resource Groups. Inc.Standard and extension properties --- s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Introducing Data Services.

Bring the resource group online. Add the logical host name resource: 6. Dependency on the a lic property b e u storage is expressed by using the standard property.nfs # clrt register SUNW. 1.nfs of the directory referenced by the group’s Pathprefix property. Sun Services.HAStoragePlus 2. l y sthe eLogicalHostname f u d ab -tran # clrs create -t nfs -g nfs-rg \ g -p Resource_dependencies=nfs-stor nfs-res an non ab 3. Oracle and/or its affiliatesฺ Complete Resource Group Example for NFS The following example is based on the assumption that the NFS agent has been added.theo \ -p Pathprefix=/global/nfs/admin nfs-rg s ) ha y m ฺ om ฺ c ฺ # clrslh create -g nfs-rg orangecat-nfs m s ide c u e G 4. Dependency on l k b a a r resource is implied. Add (register) the resource types: # clrt register SUNW.HAStoragePlus resource: @ t n b # clrs create -t HAStoragePlus de \ ku-g nfs-rg u a t y S -p AffinityOn=true \ n ( s i i d h -p FilesystemMountPoints=/global/nfs nfs-stor lu se t a u many services have properties that 5.Complete Resource Group Example for NFS Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. You should also assume that the dfstab file has been configured in the subdirectory SUNW. Revision C a . am Although o k t point to configuration ng ensefiles or binaries.2 Administration Copyright 2009 Sun Microsystems. these state manipulations are covered in the next section: # clrg online -M nfs-rg 10-36 Sun™ Cluster 3. Create the resource group: # clrg create -n vincent. All Rights Reserved. Add the SUNW. NFS does not because it uses a b the Pathprefix of the resource group. Add the NFS service. Inc.

● when_disabled – You can change with the clrs set command if the resource is disabled. Sun Services.Complete Resource Group Example for NFS Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Each property for a particular type has a tunability characteristic that can be one of the following: ● at_creation – You can only set the property as you do the clrs create command to add the resource. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Introducing Data Services. Oracle and/or its affiliatesฺ Modifying Properties With clrs set -p .. Resource Groups. ● anytime – You can change anytime with the clrs set command. All Rights Reserved.2 software allows the developer of a resource type to place restrictions on when particular properties can be set or changed. Sun Cluster 3. and HA-NFS Copyright 2009 Sun Microsystems. Revision C 10-37 a .. Inc.

s ) ha y m ฺ A resource group that is mangaged but offline can be thought oofmas ฺ c ฺ econtrol “currently not online on any nodes. A a etransitions abis disabled r y l resource that when its group is unmanaged or offline will still f u ns d b a disabled (and will not run) when the group goes online. or if the cluster reconfigures u ude because any node k a fails or joins. and survives cluster reboots. If a resource is enabled. Introduction to Resource Group State When a resource group is first created it is in an unmanaged state. Oracle and/or its affiliatesฺ Controlling the State of Resources and Resource Groups The following examples demonstrate how to use the clrg and clrs commands to control the state of resource groups. The managed/unmanaged state is a “persistent” state that survives cluster reboots.” That is.2 Administration Copyright 2009 Sun Microsystems. Sun Services. 10-38 Sun™ Cluster 3. resources. If you change a trbe g n the enabled/disabled state of a resource while the group is online. t y ( S n di e this u l a us m a Introduction to Resource k e toState g an cens b a li have a persistent disabled/enabled state flag that Individual b resources e u l k survives of its groups state. it affects what will happen to the resource when the group goes online. and data service fault monitors. it turns n a no ab on and off the resource. the cluster will still manageeitcand bring if G t @ n b the entire cluster reboots. then another similar state flag can disable and enable just the fault monitoring for the resource. Once it is brought into a managed state. but still subject to automatic sm iitdonline u and recovery. regardless of the state of its resource group.Controlling the State of Resources and Resource Groups Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a . Inc. All Rights Reserved. the cluster provides automatic control and failover for its resources. If you do so while the group is not online. When a resource is created using clrs create or clrslh create or clrssa create it is automatically put into the enabled state.

With -e. or offline. s ) ha y m ฺ om ฺ c ฺ csm Guide e t n b@nfs-rg # clrg switch [-M] [-e] -nknode e u d a The tugroup y ( S Remaster a failover resource group. With -e. the group must already be in the managed state. Without -e. Resource Groups. # clrg online [-M] [-e] nfs-rg # clrg online [-M] [-e] -n node nfs-rg ● Switch a failover resource group to a specific node. Revision C 10-39 a . Inc. the enabled/disabled state of each resource is preserved. All Rights Reserved. Without -M.Controlling the State of Resources and Resource Groups Resource Group Operations Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. all disabled resources are enabled. (It might be online already on another node. it will bring the group into the managed state first if it is unmanaged. the group must already be in the managed state. all disabled resources are enabled. it will bring the group into the managed state first if it is unmanaged. Sun Services. With -M. so Offline a n b e itais still subject lic to automatic recovery: b e u l k #raclrg b offline nfs-rg a y l e f bdu trans Restart a resource group: a g onn # clrg restart nfs-rg a n b a ● ● ● Resource Operations Use these commands to do the following: ● Disable a resource and its fault monitor: # clrs disable nfs-res ● Enable a resource and its fault monitor: # clrs enable nfs-res ● Clear the STOP_FAILED flag: # clrs clear -n node -f STOP_FAILED resourcename Introducing Data Services. a u # clrg remaster am nfs-rg o k t nagresource segroup. The group remains in the managed state. Without -e. even luif it issealready running on another node.) With -M. and HA-NFS Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Use these commands to do the following: ● Online an offline resource group onto its preferred node or a specific node. switches to its n this i d preferred node. the enabled/disabled state of each resource is preserved. Without -M.

2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ ● Disable the fault monitor for a resource: # clrs unmonitor nfs-res ● Enable the fault monitor for a resource: # clrs monitor nfs-res s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 10-40 Sun™ Cluster 3. Revision C a .Controlling the State of Resources and Resource Groups Fault Monitor Operations Use these commands to do the following: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. Inc. Sun Services.

all transitions preserve the state of the enabled/disabled flags (for the service and its fault monitoring). Sun Services.Controlling the State of Resources and Resource Groups Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. Revision C 10-41 a . and HA-NFS Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Summary of Resource Group and Resource Transitions The diagram in Figure 10-4 summarizes the resource and resource group transitions. Note the following: ● The -M switch for clrg switch or clrg online manages a group and then brings it online. Inc. ● By default. s ) ha y m ฺ m Group online o c ฺ ฺ resources running e m d s if enabled i res1: disabled/enabled ec t Gu res2: disabled/enabled @ b den u k tu clrg switch -n node rg ya Sres ( clrs disable/enable n di e this u l ma to us clrg online -n node rg clrg offlinekrg a (enabled/disabled clrg online rg (brings it on preferred node) g nse n state preserved) (enabled/disabled state of resource preserved) a b lice a b u le k Group: Offline b a a r y No resources running res1: disabled/enabled l e f u s d res2: disabled/enabled ab -tran Resources may be g an non enabled or disabled ab clrg unmanage rg (must disable each resource first) clrg online -M rg clrg online -M -n node rg clrs disable/enable res (affects whether they will run when group switched on) clrg manage rg res1: disabled/enabled res2: disabled/enabled Group: Unmanaged clrs disable/enable res Figure 10-4 Resource Group and Resource Transitions Introducing Data Services. You can force all resources to enabled by adding a -e option to clrg switch/online. Resource Groups.

b e c a i l b u le k b a a you put a resource group offline (but it is still l yFor example. you disable any automatic recovery. Oracle and/or its affiliatesฺ Suspended Resource Groups When you suspend a resource group. To suspend a group. start udsome applications akand Sstop t y ( manually. using the commands presented on the previous pages. is currently suspended. failover. or restarting of the resource group or the resources within. nor will resource groups automatically fail over. ● You can still enable/disable any individual resources.Suspended Resource Groups Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. ang non ● ab Resources will not automatically be restarted by fault monitors. That is: ● You can still transition the resource group any way you like using the commands presented on the previous pages. s ) ha y m ฺ om ฺ c ฺ csm Guide e t group is to @ resource The reason you might want to suspend an online n b e u perform maintenance on it—that is. type: # clrg resume grpname To see whether a group. the resources will go on and off accordingly. Revision C a . erandwhen f u s d managed not suspended). 10-42 Sun™ Cluster 3. a node failure still causes the group to go b tran a . Sun Services. All Rights Reserved. type: # clrg suspend grpname To remove the suspension of a group. u l e a s am to u k g suspend The reason younmight an offline resource group is so that it does sewhen a n not go online automatically you did not intend it to do so. Inc. so that dependencies still honored correctly.online. even if an entire node fails. but while preserving the online status of the group and other in can isbe d h t components. use clrg status .2 Administration Copyright 2009 Sun Microsystems. as demonstrated on the next page. ● The fault monitors for resources will still be running. If the resource group is online.

Revision C 10-43 a . Resource Groups.LogicalHostname online. Introducing Data Services. All Rights Reserved. and HA-NFS Copyright 2009 Sun Microsystems.b-----u nfs-rg vincent No Offline ud ak S t y ( theo Noin is Online d h t u al use # clrs status -g nfs-rg m ka e to g Cluster Resources === ban licens a b u le State k b Resource Name Node Name Status Message a a r y f--------l e u -----------------------------s bd tran vincent a nfs-stor Offline Offline g n n theo Online Online no aba orangecat-nfs vincent theo Offline Online Offline Online . Inc. Sun Services.Service is online. When viewing resources.Displaying Resource and Resource Group Status Using the clrg status and clrs Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Example of Status Commands for a Single Failover Application s ) ha # clrg status y m ฺ Cluster Resource Groups === om ฺ c ฺ Group Name Node Name Suspended Status csm Guide e @ ent -------------------------. nfs-res vincent theo Offline Online Offline Online . Oracle and/or its affiliatesฺ Displaying Resource and Resource Group Status Using the clrg status and clrs status Commands There are separate commands to show the status of resource groups and resources. you can still use -g grpname to limit the output to a single group.

clrs. and clrssa commands that have been described in previous sections of this module. Oracle and/or its affiliatesฺ The clsetup utility has an extensive set of menus pertaining to resource and resource group management.2 Administration Copyright 2009 Sun Microsystems. Revision C a . as h The Resource Group menu for the clsetup utility looks similar to the ) y m following: ฺ m o c ฺ ฺ *** Resource Group Menu *** e m d s i ec t Gu @ Please select from one of the following options: b den u k ya Stu ( 1) Create a resource group n s i higroup t 2) Add a network resource tolu a dresource e a tousa resource group m 3) Add a data service resource a k e to 4) Resource type registration g 5) Online/Offline a resource group banorliSwitchover ens c a b 6) Suspend/Resume recovery for a resource group e u l k b 7) Enable/Disable ya fera a resource l u 8) Change d ransproperties of a resource group b9) a properties of a resource -t g 10)onChange n Remove a resource from a resource group a n ab 11) Remove a resource group 12) Clear the stop_failed error flag from a resource ?) Help s) Show current status q) Return to the main menu Option: 10-44 Sun™ Cluster 3. These menus are accessed by choosing Option 2 (Resource Group) from the main menu of the clsetup utility. clrslh. Inc. The clsetup utility is an intuitive. All Rights Reserved. Sun Services. The clsetup utility calls the clrg.Using the clsetup Utility for Resource and Resource Group Operations Using the clsetup Utility for Resource and Resource Group Operations Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. menu-driven interface that guides you through the options without having to remember the exact command-line syntax.

without even mentioning the terminology of resources and resource groups. These wizards guide you through the entire task of integrating some of the popular applications into the cluster.Using the Data Service Wizards in clsetup and Sun Cluster Manager Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. The NFS wizard. Resource Groups. and provision the proper share command for you. Sun Services. new to Sun Cluster 3.2. All Rights Reserved. can create and online the NFS resource group and resources. and HA-NFS Copyright 2009 Sun Microsystems. are yet another way to configure application resource groups and resources. They are available through the Tasks item in Sun Cluster Manager and the Data Services submenu of clsetup: *** Data Services Menu *** s ) ha ab y m ฺ om ฺ c ฺ 1) Apache Web Server 2) Oracle csm Guide e 3) NFS @ ent b u 4) Oracle Real Application Clusters akServer tud y ( S 5) SAP Web Application in Storage d his t 6) Highly Available u l e aHostname 7) Logical m us a o k t 8) g Shared Address n se a n b e a ?) Help lic b e u l k rabq) Return to the Main Menu a y ul nsfe d b Option: a tra g n n a no Please select from one of the following options: The wizards are intended as an alternative to the Resource Group menu items presented on the previous page. Introducing Data Services. The idea is that it can guide you through the task in a wizard-like fashion. Inc. Revision C 10-45 a . just like the Resource Group menu items could. for example. It can also create the vfstab entry for your storage (it will not actually build your volume or file sytstem). Oracle and/or its affiliatesฺ Using the Data Service Wizards in clsetup and Sun Cluster Manager The data service wizards.

Sun Services. Oracle and/or its affiliatesฺ In this exercise. All Rights Reserved. or clustername.2 Administration Copyright 2009 Sun Microsystems. enclosure_name. Revision C a . when you see italicized terms. such as IPaddress. node1. embedded in a command string. 10-46 Sun™ Cluster 3. you complete the following tasks: ● Task 1 – Installing and Configuring the HA-NFS Agent and Server ● Task 2 – Registering and Configuring the Sun Cluster HA-NFS Data Services ● Task 3 – Verifying Access by NFS Clients ● Task 4 – Observing Sun Cluster HA-NFS Failover Behavior ● Task 5 – Generating Cluster Failures and Observing Behavior of the NFS Failover s ) ha y m ฺ m Task 7 – Putting Failover Application Data in a ZFS FileoSystem c ฺ ฺ e m Task 8 – Making a Customized Application Fail Over With a Generic d s i ec t Gu Data Service Resource @ b den u k Task 9 – Viewing and Managing Resources a Stuand Resource Groups y ( Using Sun Cluster Managerin d e this u l a us m a o Preparation ng k se t ba licen a b e are explained in this section: u ltasks The following k b a a l y sPreparing er to register and configure the Sun Cluster HA for NFS data f u d ab -tran service g an non ● Task 6 – Configuring NFS to Use a Failover File System ● ● ● ● ab ● Registering and configuring the Sun Cluster HA for NFS data service ● Verifying access by NFS clients ● Observing Sun Cluster HA for NFS failover behavior Note – During this exercise.Exercise: Installing and Configuring HA-NFS Exercise: Installing and Configuring HA-NFS Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. substitute the names appropriate for your cluster. Inc.

Inc. you will be using NFS y l f s 4 by default. All Rights Reserved. Install the Sun Cluster HA-NFS data service agent on all nodes: # cd sc32_location/Solaris_sparc/Product # cd sun_cluster_agents/Solaris_10/Packages # pkgadd -d . If your u Note k a clienterand abserver are both running Solaris 10 OS. This is a stateful protocol which will intentionally bdu tranVersion a . Oracle and/or its affiliatesฺ Task 1 – Installing and Configuring the HA-NFS Agent and Server In earlier exercises in Module 6. SUNWscnfs s ) ha ab y m ฺ om ฺ c ฺ sm uide c e 2. “Using VERITAS Volume Manager With Sun Cluster Software.” or Module 7. Verify that the /global/nfs file system is mounted and ready for use. Revision C 10-47 a . Resource Groups. Change G the line that t @ n b currently reads: ku tude a y GRACE_PERIOD=90 n ( this S i d luperiodsto and lower the grace e 10: a m u GRACE_PERIOD=10 ka e to g ban licens a b – Thisleshould speed up HA-NFS switchovers and failovers.” you created the global file system for NFS. Confirm that this file system is available and ready to configure for Sun Cluster HA for NFS.Exercise: Installing and Configuring HA-NFS Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. The GRACE_PERIOD controls the length of the delay. and HA-NFS Copyright 2009 Sun Microsystems. Sun Services. Modify the /etc/default/nfs file on all nodes.delay resumption of NFS activity so that clients have a chance to reclaim ang non their state any time the server is recovering (which will include any cluster switchover or failover). 3. Perform the following steps: 1. “Using Solaris Volume Manager With Sun Cluster Software. # df -h Introducing Data Services.

Add an entry to the /etc/hosts file on each cluster node and on the administrative workstation for the logical host name resource clustername-nfs.nfs li e u l k b a # eviradfstab. Inc. Revision C a . Substitute the IP address supplied by your instructor. # # # # cd /global/nfs mkdir /global/nfs/data chmod 777 /global/nfs/data touch /global/nfs/data/sample.nfs in this d u 6. Create the administrative directory that contains the dfstab. Create the dfstab. n ab 5. Verify that the entry for your cluster exists on the vnchost. In practice.nfs-res file.nfsres file for the NFS resource. Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.nfs directory. All Rights Reserved.nfs-res file. Add the entry to share the ka e to g /global/nfs/data an censdirectory. 10-48 Sun™ Cluster 3. Create the directory specified in the dfstab. Oracle and/or its affiliatesฺ IP_address clustername-nfs Note – In the RLDC.nfs-res y l f bdu transshare -F nfs -o rw /global/nfs/data a g onn a 7. Sun Services. b a #bcd SUNW. Perform the remaining steps on just one node of the cluster. s ) ha y m ฺ om ฺ c ฺ # cd /global/nfs csm Guide e @ ent # mkdir admin b u # cd admin ak Stud y ( # mkdir SUNW. and use the same IP address to create the entry on your cluster nodes.nfs-res ein the al ufile s m /global/nfs/admin/SUNW. the /etc/hosts file on the vnchost already contains the appropriate entry for each cluster.2 Administration Copyright 2009 Sun Microsystems. you would be more specific about the share options in the dfstab.Exercise: Installing and Configuring HA-NFS 4.file Note – You are changing the mode of the data directory only for the purposes of this lab.

you should connected to the storage t u l e a us to true.HAStoragePlus # clrt list -v 2. a # clrs create -t nfs -g nfs-rg \ ang non # clrg create -n node1.HAStoragePlus resource types. All Rights Reserved.nfs # clrt register SUNW. for example).nfs resource.Exercise: Installing and Configuring HA-NFS Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Task 2 – Registering and Configuring the Sun Cluster HA-NFS Data Services Perform the following steps on one node only to configure HA-NFS in the cluster: 1. Verify that the data service is online. node. Inc. # clrg online -M nfs-rg Note – The very first time you do this you may see a console warning message concerning “lockd: cannot contact statd”. Bring the resource group to a managed state and then online. # clrt register SUNW. nonstorage set the value ofm AffinityOn a k esettothe value of AffinityOn to false. # clrs list -v # clrs status # clrg status Introducing Data Services. you should g an create ns-t HAStoragePlus -g nfs-rg \ e #ab clrs c li b -p AffinityOn=[true|false] \ e u l k b a a -p FilesystemMountPoints=/global/nfs nfs-stor y fer l u s bd tran5. e G t @ b den # clrslh create -g nfs-rg clustername-nfs u k tu If all of your nodes are ya resource. If you have a third. Create the SUNW. Note that on a three-node cluster the node list (specified with -n) can include nodes not physically connected to the storage if you are using a global file system. and HA-NFS Copyright 2009 Sun Microsystems. Register the NFS and SUNW. Create the failover resource group. ( S 4. Both daemons are restarted (successfully) and you can ignore the message.node2. 7.HAStoragePlus n di (two-node hiscluster. Resource Groups. s ) ha y m ฺ om ฺ c ฺ m uide csresource 3. Create the logical host name resource to the group. Revision C 10-49 a .[node3] \ -p Pathprefix=/global/nfs/admin nfs-rg ab -p Resource_dependencies=nfs-stor nfs-res 6. Sun Services. Create the SUNW.

nfs file from the lab files location into your home directory.file 2.nfs script and verify that the logical host name and NFS file system names are correct. The script displays e m how idlong cstimes the file to standard output (stdout). On the administration workstation. s ) ha y m ฺ When this script is running. Sun Services. e u d k ya Stu ( n di e this u l a us m a k e to g ban licens a b u le k b a a l y sfer u d ab -tran g an non ab 10-50 Sun™ Cluster 3.2 Administration Copyright 2009 Sun Microsystems. This script the u e G t @ NFS data service is interrupted during b switchovers n and takeovers. Edit the $HOME/test. copy the test. verify that you can access the cluster file system.Exercise: Installing and Configuring HA-NFS Task 3 – Verifying Access by NFS Clients Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Perform the following steps to verify that NFS clients can access the file system of the Sun Cluster HA for NFS software package: 1. (# or $) ls /net/clustername-nfs/global/nfs/data sample. it creates and writes a file containing a m oalso c ฺ ฺ timestamp to your NFS-mounted file system. Inc. Revision C a . 3. All Rights Reserved. On the administration workstation.

# ifconfig -a 8. and HA-NFS Copyright 2009 Sun Microsystems. Inc. transfer control of the NFS service from one node to another. All Rights Reserved. test its HA operation by performing the following steps: 1. Oracle and/or its affiliatesฺ Task 4 – Observing Sun Cluster HA-NFS Failover Behavior Now that the Sun Cluster HA for NFS environment is working properly.7. om ฺ c ฺ e 4. start the $HOME/test. s ) ha y m ฺ Substitute the name of your offline node for dest-node. On one node of the cluster. g whichn file systems a cens b a li b __________________________________________________ e u l k a e__________________________________________________ ab r y l f bdu trans __________________________________________________ a . How long was the data service interrupted n b e u from one physical host to another? ak Stud y ( in this __________________________________________________ d u al and seshare commands on all nodes to verify u 6. On the administration or display station. On one node of the cluster. Use the ifconfig command on all nodes to observe the additional ang non # clrg switch -n dest-node nfs-rg ab IP address associated with the Logical Hostname resource configured as a virtual interface on one of the adapters in your IPMP group. Revision C 10-51 a . Sun Services.nfs script. # clrg switch -n dest-node nfs-rg Introducing Data Services. Observe the messages displayed by the test. determine the name of the node currently primary for the nfs-rg resource group. Use the cldgam status k ethetonodes are now mounting and exporting. use the clrg switch command to transfer control of the NFS service back to its preferred host. 2. 3. On one node of the cluster. uid e t the switchover @ during 5.Exercise: Installing and Configuring HA-NFS Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.nfs csm Gscript. Resource Groups.

Revision C a . you can still bring a node to the ok prompt using a break signal through your terminal concentrator. Repeat some tests after suspending the resource in this d u # clrg suspend nfs-rg al use m ka e to g s your results. Inc. All Rights Reserved.a ) or sabotage the adapter using ifconfig modinsert as youฺm didyin Module 8) om ฺ c ฺ Multiple public network failure on a single node csm Guide e @ ent b u Try your tests while the resource groupais nonsuspended k in its tnormal. Try to generate the following failures: ● Node failure (power down a node or bring it to the ok prompt) s Single public network interface failure (pull a public network cable h . If you do not have physical access to the cluster.2 Administration Copyright 2009 Sun Microsystems.Exercise: Installing and Configuring HA-NFS Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. If you have physical access to the cluster. you can physically pull out network cables or power down a node. Oracle and/or its affiliatesฺ Task 5 – Generating Cluster Failures and Observing Behavior of the NFS Failover Generate failures in your cluster to observe the recovery features of Sun Cluster 3. ud group: y ( S state. remove the suspension: When you are nwith bansatisfied e c a li nfs-rg #bclrg resume e u l k a erab y l u nsf d b a tra g n n a no ● ● ab 10-52 Sun™ Cluster 3.2 software.

Make sure you still have seven fields on no aba that line. Introducing Data Services. it is all just a convention. and HA-NFS Copyright 2009 Sun Microsystems. All Rights Reserved. # # # # # # 5. The file system should be mounted only on the node running the NFS service and should fail over appropriately. just by convention. and keeping the same mount point simplifies the flow of these exercises. y l u nsf d b b.rabChange yes to no in the mount-at-boot column. d his t u l e a us # umount /global/nfs m a k editethe to/etc/vfstab file to make /global/nfs a 4. Perform the following steps: 1. you can either local bafile e c a liedit. n system ns(if you have a non-storage node. Replace it a tra g n n with a minus sign (-). Sun Services. # clrg online nfs-rg 6. Observe the file system behavior. If you have a nonstorage third node. However. Revision C 10-53 a . Disable the NFS resource group. clrg clrs clrs clrs clrs clrs remove-node -n nonstorage-node-name nfs-rg disable nfs-res disable nfs-stor set -p AffinityOn=TRUE nfs-stor enable nfs-stor enable nfs-res Restart the NFS resource group. Resource Groups. Oracle and/or its affiliatesฺ Task 6 – Configuring NFS to Use a Failover File System In this task. Unmount the global NFS infile system. delete the nonstorage node from the group node list. you wouldn’t have a failover file system mounted under a subdirectory of /global. Remove the word global from the mount options. Note – Ordinarily. On eachgnode. and set the affinity: s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( 3. or just remove the line completely): b makelthis e u k a ea. you configure NFS to use a failover file system rather than a global file system. # clrg offline nfs-rg 2. Inc.Exercise: Installing and Configuring HA-NFS Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

Oracle and/or its affiliatesฺ In this task.2 Administration Copyright 2009 Sun Microsystems. verify that the nfspool/nfs dataset am int(this o is mounted g at k /testnfs will match the destination for the cpio e n s a n done in ce ab stepl4). From the node identified in step 1. Move l e f u s bd tran to the node identified in step 1: a g onn a # clrg switch -n node_importing_the_zpool nfs-rg n ab Peform the following steps only on the single node that is currently the primary for the NFS resource group: 4. remove it from the group using clrg remove-node -n nonstorage-node-name. b ku tude a y # zpool status n ( this S i d lu se a 2. -print|cpio -pdmuv /testnfs cd / Sun™ Cluster 3. This should be the case if you have completed Task 6. Note – This task assumes that there are no longer any non-storage nodes included in your NFS resource group. you will migrate your server-side data to a failover ZFS filesystem. s ) ha y m ฺ om ฺ c ฺ m uide csZFS 1. All Rights Reserved. If you have a non-storage node in your resource group (and have not done Task 6). On the node identified step u 1. Sun Services. Revision C a . i b u# df -hble| grep nfspool/nfs k a rayour NFS failover resources (and current failover filesystem) y3. Inc. disable your NFS service and migrate the data to your ZFS file system: # # # # # 10-54 clrs disable nfs-res clrs delete nfs-res cd /global/nfs find . Identify the node that is currently importingethe storage pool G t @ n (nfspool) that you created in the ZFS module.Exercise: Installing and Configuring HA-NFS Task 7 – Putting Failover Application Data in a ZFS File System Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

Resource Groups. Revision C 10-55 a . clrs disable nfs-stor clrs delete nfs-stor umount /global/nfs zfs set mountpoint=/global/nfs nfspool/nfs df -k Recreate your NFS resources using the ZFS storage: # clrs create -g nfs-rg -t HAStoragePlus \ -p Zpools=nfspool nfs-stor # clrs create -g nfs-rg -t nfs \ -p Resource_dependencies=nfs-stor nfs-res # clrg status # clrs status # df -k s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u d file system from your aklegacy tuNFS y 7. Remove or comment out the old ( S in this /etc/vfstab on bothdnodes. al andufailover s m 8. All Rights Reserved. and it will be unmounted by the clrs disable (so the umount is there just in case you had not done task 6). and remount the ZFS under /global/nfs. and HA-NFS Copyright 2009 Sun Microsystems. Inc. # # # # # 6.Exercise: Installing and Configuring HA-NFS Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. u e behavior of the NFS application. If you have already done task 6 then the traditional /global/nfs is already a failover file system. Disable your old (non-ZFS) NFS resource. Observe switchover a know to the zpool containing your ZFS file system. whichn will include g e ba licens a b u le k b a a l y sfer u d ab -tran g n non a b a Introducing Data Services. Oracle and/or its affiliatesฺ 5. Sun Services.

Inc. Perform the following steps on the nodes indicated: 1.[node3] to g e # clrg create myapp-rg n ens a b a one node. create a u l e a us application: m a k -n node1. All Rights Reserved.node2. Repeat step b a few times. the group switches over to the other node. Wait a little while and note that it restarts on the same node. From licregister the Generic Data Service resource type: b e u l k # clrt a abregister SUNW. Verify that you can manually switch the group from node to node. Sun Services. 10-56 Verify the behavior of your customized application. c.Exercise: Installing and Configuring HA-NFS Task 8 – Making a Customized Application Fail Over With a Generic Data Service Resource Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. you can see how easy it is to get any daemon to fail over in the cluster. Revision C a . a.gds \ a n b a -p Start_Command=/var/tmp/myappdaemon \ -p Probe_Command=/bin/true -p Network_aware=false \ myapp-res # clrg online -M myapp-rg 6.ansFrom one node. Eventually. create the new resource and enable the group: a g onn # clrs create -g myapp-rg -t SUNW. Sun™ Cluster 3. create a daemon that represents your customized application: # vi /var/tmp/myappdaemon #!/bin/ksh while : do sleep 10 done s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u k nodes.gds r y l e f bdu tr5. On all nodes (or do it on one node and copy the file to other nodes in the same location). 4. Wait until clrs status shows that the resource is fully online again. Oracle and/or its affiliatesฺ In this task. by using the Generic Data Service (so that you do not have to invent your own resource type). Kill the daemon. Make sure the file is executable(y on S n failover inew d his resource group for your t 3.2 Administration Copyright 2009 Sun Microsystems. aall tud 2. From any one node. b.

Select the check box next to a resource group. Log in as root and enter the Sun Cluster Manager Application. Resource Groups. and HA-NFS Copyright 2009 Sun Microsystems. Revision C 10-57 a . m and use the e Switch d s i c u Primary button to switch the primary node group.Exercise: Installing and Configuring HA-NFS Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. log in to Sun Java Web Console on any cluster node: https://nodename:6789 2. Oracle and/or its affiliatesฺ Task 9 – Viewing and Managing Resources and Resource Groups Using Sun Cluster Manager Perform the following steps on your administration workstation: 1. In a Web browser. Click the Resource Groups folder on the left. 3. Go back to the Status Panel for Resource Groups. Inc. Introducing Data Services. All Rights Reserved. Investigate the status information and graphical topology information that you can see regarding resource groups. e for at resource G @ n b ku tude a y n ( this S i d lu se a am to u k ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n no aba 5. Sun Services. 4. s ) ha y m ฺ om ฺ c ฺ 6.

Inc. Sun Services. Revision C a .Exercise Summary Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Exercise Summary ? ! Discussion – Take a few minutes to discuss what experiences. ● Experiences ● Interpretations ● Conclusions ● Applications s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 10-58 Sun™ Cluster 3. or discoveries you had during the lab exercises. issues. All Rights Reserved.

Revision C a . Inc. All Rights Reserved. you should be able to: c ฺ csm Guide Describe the characteristics of scalable services e @ ent b u Describe the function of the load-balancer ak Stud y ( Create the failover resource in group isfor the SharedAddress resource d h t u egroup for the scalable service alresource Create the scalable s m u kathe eSharedAddress to Describeghow resource works with scalable n s a n services ab lice b u Addbauxiliary le nodes k a a r y ul nsfeCreate these resource groups d b a tra Control scalable resources and resource groups g n n a no ● ● ● ● ● ● ● ● ● View scalable resource and group status ● Configure and run Apache as a scalable service in the cluster 11-1 Copyright 2009 Sun Microsystems. Sun Services.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Module 11 Configuring Scalable Services and Advanced Resource Group Relationships s ) ha Objectives ab y m ฺ om ฺ Upon completion of this module.

Inc. But will every application within a web server necesarily behave properly in a scalable environment? ● Why might you want a “client affinity” for load balancing? ● What reason might you have to have separate resource groups with a requirement that group A run only on a node also running group B? Why not just combine them into one group? y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha 11-2 Sun™ Cluster 3. Oracle and/or its affiliatesฺ Relevance ? ! Discussion – The following questions are relevant to understanding the content of this module: ● Web servers are supported as scalable applications. All Rights Reserved.2 Administration Copyright 2009 Sun Microsystems. Revision C a .Relevance Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services.

All Rights Reserved. Inc.Additional Resources Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. part number 8207356 (for Sun Cluster 3. part number 821-0259 (for Sun Cluster 3.2 11/09).2 11/09). Oracle and/or its affiliatesฺ Additional Resources ● Sun Cluster System Administration Guide for Solaris OS. Revision C 11-3 a . Sun Services. part number 820-7458 (for Sun Cluster 3.2 11/09) s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Configuring Scalable Services and Advanced Resource Group Relationships Copyright 2009 Sun Microsystems. ● Sun Cluster Software Installation Guide for Solaris OS. ● Sun Cluster Concepts Guide for Solaris OS.

while appearing to the client to be running on a single server. nor do they care. Inc. y m ฺ om ฺ c ฺ csm Guide e Client 1 Client 2 @Client e3 nt b u Requests Requests ak Stud y Network ( in this d u Node 1 al use m a to Global Interface g k e Web xyz.coman b licens Page a bRequest le Transport u k a Distribution ab r y Node 2 Node 3 l e f u s d ab -tran g n non HTTP a b HTTP HTTP Application a s ) ha Application Application <HTML> <HTML> <HTML> </HTML> </HTML> </HTML> Globally Available HTML Documents Figure 11-1 Scalable Services. Clients connect to such a service using a single IP address called a shared address.2 Administration Copyright 2009 Sun Microsystems. Sun Services. such as Apache web server. Shared Addresses Architecture 11-4 Sun™ Cluster 3. Revision C a . clients do not know which node or zone they connect to. to run simultaneously on multiple nodes or non-global zones. All Rights Reserved. Typically. Figure 11-1 shows the architecture of scalable services and shared addresses. Oracle and/or its affiliatesฺ The scalable service architecture allows some services.Using Scalable Services and Shared Addresses Using Scalable Services and Shared Addresses Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

Oracle and/or its affiliatesฺ Exploring Characteristics of Scalable Services Similar to the failover services described in the previous modules. The application binaries running on the various nodes are unaware of each other. they already have the locking in place to make them ideal to work as scalable services in Sun Cluster 3. a file system-oriented scalable service sm can idstill cresource u e the global file system. While Sun Cluster 3. all the data for the scalable service must h ) y in the shared storage. All Rights Reserved.2 SharedAddress mechanism. are usually written in such a way that multiple instances can be launched simultaneously even on a stand-alone server. File and Data Access abes Similar to the failover services.HAStoragePlus be G t @ n b configured to manage dependencies between eservice and the storage. Configuring Scalable Services and Advanced Resource Group Relationships Copyright 2009 Sun Microsystems. although not written specifically for a scalable platform. m ฺ om ฺ c ฺ e must use Unlike the failover service.2 software. it does not automatically call any file-locking primitives for you. k b writtenain such a l y sfer u d ab -tranIn Sun Cluster 3. the applications used as scalable services in the Sun Cluster 3. Web servers that do file writing in common gateway interface (CGI) scripts. u uthe d k a t (y is S n i th ludData File Locking for Writing e a am to us k ngbarriers seto taking just any application and turning it into a One of the big a n b e a servicelicis that a service that is cluster-unaware may have been scalable b u le a way as to ignore file-locking issues. This works because of the Sun Cluster 3.2 software provides the global data access methods. Revision C 11-5 a .2 software. Web servers that perform file writing using Java™ servlets must be examined much more closely. Therefore. A SUNW. Sun Services. A servlet might use thread synchronization rather than file locking to enforce serial write access to a critical resource.2 environment are generally off-the-shelf applications that are not specifically compiled or released to run in the cluster. Inc. an application that performs data g n non modification without any type of locking or file synchronization a b a mechanism generally cannot be used as a scalable service.Exploring Characteristics of Scalable Services Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. This does not translate properly into a scalable service.

2 Administration Copyright 2009 Sun Microsystems. You could have multiple SharedAddress resources that provide load balancing for the same application using IP addresses on different subnets. Inc. Client Affinity s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m A single SharedAddress ka resource to can provide standard (per connection g e load balancing) thentype s of sticky load balancing required by banAand e shoppingacarts. rather than a per connection basis.2 software is the SharedAddress resource. Revision C a . Note – You could have a single SharedAddress resource that provides load balancing for the same application using multiple IP addresses on the same subnet. where the state of the client’s shopping cart is recorded only in the memory of the particular node where the cart was created. That is. a y ul nsfe d b a Load-Balancing tra g Weights n n o a n ab Certain types of applications require that load balancing in a scalable service be on a per client basis. The load balancing properties discussed on this page are set per scalable application. they require that the same client IP always have its requests forwarded to the same node or zone. a better node or zone can be made to handle a greater percentage of requests. not per address.Using the SharedAddress Resource Using the SharedAddress Resource Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. This resource provides not only a single IP address that makes the scalable service look similar to a single server from the point of view of the client. The prototypical example of an application requiring such client affinity is a shopping cart application. Sun Cluster 3. but through the use of properties described in the following sections. scalable service’s agent registers which type of load c li b e u l balancing k rais brequired. depending on the stickiness) per node or zone. but also provides the load balancing of requests to all the nodes or zones on which a scalable service is active. Oracle and/or its affiliatesฺ The glue that holds a scalable service together in Sun Cluster 3. The default weighting is equal (connections or clients. based on a property of the data service. 11-6 Sun™ Cluster 3. Sun Services.2 software also lets you control the weighting that should be applied for load balancing.

A failover resource group contains the SharedAddress resource.apache Resource g n non a b a Figure 11-2 Scalable and Failover Resource Groups Configuring Scalable Services and Advanced Resource Group Relationships Copyright 2009 Sun Microsystems. Sun Services. or mastered. All Rights Reserved. It is online. s ) ha y m ฺ m oGroup c Scalable Resource Group Failover Resource ฺ ฺ e m d s i ec t Gu @ n b SUNW. Figure 11-2 is a block diagram showing the relationship between scalable and failover resource groups. A scalable resource group contains an HAStoragePlus resource and the actual service resource. Revision C 11-7 a .HAStoragePlus d k tu ya SResource ( Resource n di e this u l a us m a k e to g ban licens a bResource_dependencies u le k b a a l y sfer u d ab -tran SUNW. by only one node at a time. The node that masters the SharedAddress resource is the only one that receives incoming packets for this address from the public network. The HAStoragePlus resource is added to guarantee that the storage is accessible on each node before the service is started on that node. Oracle and/or its affiliatesฺ Exploring Resource Groups for Scalable Services A scalable service requires the creation of two resource groups.SharedAddress e u SUNW.Exploring Resource Groups for Scalable Services Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc.

SharedAddress HostnameList=apache-lh e m d s i Netiflist=therapy@1.1@2 Scalable=TRUE Port_list=80/tcp Resource_dependencies=web-stor. Inc. apache-lh Sun™ Cluster 3.] ● Resource group contents: See Table 11-1..therapy@2 ec t Gu @ b den u k ya Stu Resource group name: apache-rg ( n di e this Mode=Scalable u Properties: [Nodelist=vincent.theo Mode=Failover Failback=False. Table 11-1 sa-rg Resource Group Contents s ) ha y m ฺ m o c ฺ ฺ apache-lh SUNW.. Sun Services.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Resources and Their Properties in the Resource Groups Table 11-1 and Table 11-2 demonstrate the properties and contents of the two resource groups required to implement a scalable service: ● Resource group name: sa-rg ● Properties:[Nodelist=vincent. Revision C a .Exploring Resource Groups for Scalable Services Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved.HAStoragePlus FilesystemMountPoints=/global/web b a Resource Name Resource Type Properties ● ● ● AffinityOn=False apache-res 11-8 SUNW.apache Bin_dir=/global/web/bin Load_balancing_policy=LB_WEIGHTED Load_balancing_weights=3@1.theo l a Maximum_primaries=2] m us Desired_primaries=2 a o k t g contents: ngroup se See Table 11-2. Resource a n b e a lic b e u l Table 11-2 apache-rg k Resource b Group Contents a a r y fe ul d Resource Name ansResource Type Properties b a r t g n non a web-stor SUNW.

the lu sgroup e a u Desired_primaries properties are both amand tMaximum_primaries o k automatically ngset etonthe senumber of nodes in the nodelist for the group. the Mode=Scalable e G t @ automatically set. Sun Services. n b ku tude a y n ( this S i d Note – If you create a resource using clrg create -S. Revision C 11-9 a . ● Lb_sticky_wild – Connections from the same client to any server port go to the same node. Oracle and/or its affiliatesฺ Understanding Properties for Scalable Groups and Services Certain resource group properties and certain resource properties are of particular importance for scalable services. but you can manually bring it online to more nodes to using clrg online. It has one of the following no aba values: ● Lb_weighted – Client connections are all load balanced. Repeated connections from the same client might be serviced by different nodes. up to Maximum_primaries. ● Lb_sticky – Connections from the same client IP to the same server port all go to the same node. a b a lic b e u l k rab a y l Load_balancing_policy fe Property uThe s d n b a tra g n n This is a property of the data service resource. This is good when port numbers are generated dynamically and not known in advance. The rgmd tries to run the service on Desired_primaries nodes. s ) ha y m ฺ om ฺ c ฺ ideis csm property u If these values are greater than 1. The Desired_primaries and Maximum_primaries Properties These are group properties that indicate how many nodes the service should run on. This is only for the ports listed in the Port_list property.Understanding Properties for Scalable Groups and Services Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Configuring Scalable Services and Advanced Resource Group Relationships Copyright 2009 Sun Microsystems. This is the default. Inc. Load balancing is only for different clients. All Rights Reserved.

Its value must include the SharedAddress resource (in the other resource group). All Rights Reserved. Inc.Understanding Properties for Scalable Groups and Services Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services.3@2 indicates node 1 will handle five requests (or clients. Oracle and/or its affiliatesฺ The Load_balancing_weights Property This property controls the weighting of the load balancer. if using sticky policies) for every three handled by node 2.2 Administration Copyright 2009 Sun Microsystems. A value such as 5@1. Revision C a . This is used by the data service agent to register the data service with the load balancer associated with the SharedAddress resource. s ) ha y m ฺ om ฺ c ฺ The same Resource_dependencies property of the scalable e also m uservice d s i c typically includes the HAStoragePlus resource inethe same Ggroup. The Resource_dependencies Property You must set this property on a scalable resource. The default is that all nodes handle the same volume. t @ n b ku tude a y n ( this S i d lu se a am to u k ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n a no ab 11-10 Sun™ Cluster 3.

Oracle and/or its affiliatesฺ Adding Auxiliary Nodes for a SharedAddress Property The SharedAddress logic includes a routine whereby the actual IP address associated with the resource is configured on the public network adapter (IPMP groups) on the primary node and is configured as a virtual address on the loopback network on all other nodes in its nodelist. ya Stu ( n di hasea tspecial his auxiliary nodes property that u l The SharedAddress resource a us m allows you to augment node list of its group. you want only nodes vincent and theo to be f u d primaries for the SharedAddress (to actually host it on the public net ab -tranthe g and do the load balancing). All Rights Reserved. This is set with the -X option s a n b e to clrssa a create. lic b e u l k rab a y l Insthe e following example. Revision C 11-11 a . Sun Services. theo. Inc. s ) ha ab y m ฺ If the node list for the SharedAddress resource group is o am superset of the c ฺ ฺ But node list of every scalable service that might run onm it. and apricot: # clrg create -n vincent. while still allowing a u larger nlist for the scalable b node e d k services dependent on it.theo sa-rg # clrssa create -g sa-rg -X apricot apache-lh Configuring Scalable Services and Advanced Resource Group Relationships Copyright 2009 Sun Microsystems. This enables scalable services that are running on nodes other than the primary SharedAddress node to still bind to the IP address used for the SharedAddress. But you might be supporting scalable services an non that run on vincent. just for the purposes of a knodesethe toscalable g supporting n more for services. However. you are fine. e d s i c the primary u for the you might want to restrict which nodes mightebe G t @ SharedAddress.Adding Auxiliary Nodes for a SharedAddress Property Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. the SharedAddress resource must know about all possible nodes on which any scalable service associated with it might run.

Oracle and/or its affiliatesฺ Reviewing Command Examples for a Scalable Service The following examples assume that the Apache web server agent was added from the Sun Cluster 3.an Create the Apache service: g an non # clrs create -t SUNW.2 Administration Copyright 2009 Sun Microsystems.HAStoragePlus to g e # clrs create -g apache-rg \ n ens a b -pa FilesystemMountPoints=/global/web \ lic b e u -p AffinityOn=false \ l k rab a y l eweb-stor f u s d ab -tr7. Inc. 1. Register the resource types: # clrt register SUNW.Reviewing Command Examples for a Scalable Service Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.theo sa-rg 3. Sun Services. All Rights Reserved. Online the resource group: # clrg online apache-rg 11-12 Sun™ Cluster 3. Create the SharedAddress resource: y m ฺ om ฺ 4. Bring the resource group online: c ฺ # clrg online -M sa-rg csm Guide e 5. Revision C a . Create the HAStoragePlus ka-t SUNW.apache -g apache-rg \ s ) ha # clrssa create -g sa-rg apache-lh ab -p Resource_dependencies=web-stor. Create the scalable resource group: ub@ ent d ak S y # clrg create -S -n vincent.apache-lh \ -p Bin_dir=/global/web/bin \ -p Port_list=80/tcp \ -p Scalable=TRUE \ apache-res 8.theo \ tu ( in this apache-rg d u e al resource: s m u 6. Create the failover resource group for the SharedAddress by typing: # clrg create -n vincent.HAStoragePlus 2.apache # clrt register SUNW.2 Data Services CD using the Java ES installer or the pkgadd command.

Oracle and/or its affiliatesฺ Controlling Scalable Resources and Resource Groups The following examples demonstrate how to use the clrg and clrs commands to control the state of resource groups. it is switched off. Any nodes not mentioned are not affected (if the group is running on other nodes.Controlling Scalable Resources and Resource Groups Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Resource Group Operations Use the following commands to: ● Online an offline resource group onto its preferred nodes. in this d u l [-e] e-n node..node. and data service fault monitors. le k b a a previously running on more nodes or on nodes that were less y fer l u s bd tran preferred..node. it just stays running there). ● ● ● Offline a resource group (it goes off of all nodes): # clrg offline apache-rg ● Offline a resource group on specific nodes. On any other y ( nodes.. Sun Services. Revision C 11-13 a .node [-M] t @[-e]eapache-rg n b u akto theSspecified Switch a scalable resource group tud nodes. apache-rg a[-M] # clrg switch s m u a kscalable to Remaster a resource group. The group switches to its most g e n s a n preferred b linodes e (the number of nodes is controlled by c a b It may go offline on some nodes if it was u Desired_primaries). Inc. # clrg offline -n node. apache-rg Configuring Scalable Services and Advanced Resource Group Relationships Copyright 2009 Sun Microsystems.node apache-rg ● Restart a resource group: # clrg restart apache-rg ● Restart a resource group on a particular node or nodes: # clrg restart -n node. # clrg online [-M] [-e] apache-rg s ) ha ab y m ฺ om ฺ c ฺ csm Guide e # clrg online -n node. resources. a # clrg remaster apache-rg ang non ● Online a resource group on specific nodes.. Any nodes not mentioned are not affected (if the group is running on other nodes. All Rights Reserved.. it just stays running there).

.node.. apache-res ● Enable a resource and its fault monitor on all nodes: # clrs enable apache-res ● s ) ha Enable a resource and its fault monitor on specified nodes: y m ฺ om ฺ c ฺ Fault Monitor Operations csm Guide e @ ent b u Use the following commands to: ak Stud y ( in on allthnodes: is d Disable just the fault monitor u l e aapache-res m us # clrs unmonitor a o k t Disable a just ngthe efault semonitor on specified nodes: n b a unmonitor lic -n node.2 Administration Copyright 2009 Sun Microsystems. Inc. Oracle and/or its affiliatesฺ ● Disable a resource and its fault monitor on all nodes: # clrs disable apache-res ● Disable a resource and its fault monitor on specified nodes: # clrs disable -n node.. apache-res ● ● ● ● # clrs monitor -n node.node.node.. apache-res #bclrs e u l k Enable a aba fault monitor on all nodes: r y l e f bdu trans# clrs monitor apache-res a g onn Enable a fault monitor on specified nodes: a n ab # clrs enable -n node.Controlling Scalable Resources and Resource Groups Resource Operations Use the following commands to: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a ...... All Rights Reserved.node.. Sun Services.. apache-res 11-14 Sun™ Cluster 3..

Revision C 11-15 a . All Rights Reserved. Sun Services. a vincent ab r y l e theo Offline Offline f u ns d b a tra vincent g n web-stor Online Online n o a n b theo Online Online a apache-res vincent theo Online Online Online .Using the clrg status and clrs status Commands for a Scalable Application Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.SharedAddress online.Service is online Online . Configuring Scalable Services and Advanced Resource Group Relationships Copyright 2009 Sun Microsystems.web-rg in this d u al use Cluster Resources === m ka e to g Resource Name NodebName an censState Status Message a li -----------------------------b--------e u l k orangecat-web Online Online . Oracle and/or its affiliatesฺ Using the clrg status and clrs status Commands for a Scalable Application Use the status subcommands as follows to show the state of resource groups and resources for a scalable application: # clrg status Cluster Resource Groups === Group Name ---------sa-rg Node Name --------vincent theo Suspended --------No No Status -----Online Offline s ) ha y m ฺ om ฺ c web-rg vincent No Online ฺ theo No Online csm Guide e @ ent b u ak Stud y ( # clrs status -g sa-rg. Inc.Service is online.

not a requirement. explicitly. but this affinity is not satisfied by the current state of the two resource groups 11-16 Sun™ Cluster 3. banoflthe e on the location source c a i b e u l k a ea rweak ab positive affinity says that the source group prefers to switch y Setting l f nsnode already running the target group. Sun Services. Weak positive and weak negative group affinities are denoted by a single plus or single minus sign. g The first two kinds e ns group. explicitly. In this example. that is fine. All Rights Reserved. In this section.not running at all. # clrg set -p RG_affinities=+rg1 rg2 WARNING: resource group rg2 declares a weak positive affinity for resource group rg1. if any. The source is the group for which the value of RG_affinities is set. place either on any node that you want. ab Setting a weak negative affinity means that the source group prefers any other node besides the target. Resource group affinities provide a mechanism for specifying either a preference (weak affinities) or a requirement (strong affinities) that certain resource groups either run on the same node (positive affinities) or do not run on the same node (negative affinities). This does not affect the current location of either group. in the following example: s ) ha y m ฺ om ฺ # clrg set -p RG_affinities=++rg1 rg2 c ฺ csm Guide e rg2 is referred to as the source and rg1 as the @ target. t n b e u ak Stud y ( din Weak hisNegative Affinities Weak Positive Affinitiesluand t e a us m a kof affinities toplace only a preference. Once again. You can freely switch online and offline ang non either group. You can freely. and freely. the words source and target are used to refer to the resource groups with an affinities relationships. and the target is the group referred to by the value of the property. group rg2 is given a weak positive affinity for rg1. Inc. respectively. So. place either group on whatever node you want. Oracle and/or its affiliatesฺ Advanced Resource Group Relationships Sun Cluster 3. Revision C a .2 software offers a series of advanced resource group relationships called resource group affinities. If the target group is bdu trtoathe a .Advanced Resource Group Relationships Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems. it is just a preference.

when the source group needs to fail over it will fail over to the node running the target group. This example sets a strong positive affinity: # clrg set -p RG_affinities=++rg1 rg2 The following applies: ● The only node or nodes on which the source can be online are nodes on which the target is online. when a source group goes online and you do not specify a specific node. weak affinities are not enforced when you manually bring or switch a group onto a specific node. Configuring Scalable Services and Advanced Resource Group Relationships Copyright 2009 Sun Microsystems. Strong Positive Affinities Strong positive affinities (indicated with a ++ before the value of the target) place a requirement that the source run on the same node as the target. The following command will succeed. it will go onto the same node as the target. g onn a n ab However. a source can have more than one target. and you switch the target to another node. Inc. ngweak se aand a n b e the source prefers to choose node satisfying the greatest possible c a i l b number of lweak e affinities. even if that node is not a preferred node on the source group’s node list. even if the source group has a weak affinity for a target running on a different node. a source a o k t can have both positive weak negative affinities. ● If the source and target are currently running on one node. Sun Services. In addition. it drags the source with it. Revision C 11-17 a . For example. All Rights Reserved. ● Putting the resource group online onto a nonspecified node: # clrg online source-grp Similary. even if that node is not a preferred node on the source group’s node list.Advanced Resource Group Relationships The following will be affected by a weak positive affinity (if the target group is actually online): Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. In s m u other words. Oracle and/or its affiliatesฺ ● Failover of the source group If the target is online. In these cases. select a node that satisfies two u k b a a weak positive affinities and two weak negative affinities rather than a y fer l u s that satisfies three weak positive affinities and no weak negative bd trannode a affinities. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u # clrg switch -n specific-node aksrc-grp tud y ( S in this d u l There can be multiple a resource groups e as the value of the property.

Inc. if a resource in the source group fails and its fault monitor suggests a failover. G t @ n b e resources in the u to put For some reason. and then the source is dragged correctly. That is. y m ฺ om ฺ c ฺ You might want to be able to offline the source group leave e the m but d s i c u target group running.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. you are saying the failover group must run on some node already running the scalable group. All Rights Reserved. ● An attempt to switch the source to a node where the target is not running will fail. Oracle and/or its affiliatesฺ Advanced Resource Group Relationships ● If you offline the target group. it might make sense to make them just one group.# clrg set -p RG_affinities=+++rg1 rg2 n a n ab ● ● The only difference between the +++ and the ++ is that here (with +++). (See the next section for the solution to this example. What happens is that the RGM moves the target group over to where the source wants to fail over to.) The source and target are closely tied together. Sun Services. you might not be k able dsome u a t y ( check same group (the resources might and S reject you if you try to n s i i d h put it in the same group as another resource). 11-18 Sun™ Cluster 3. ● If a resource in the source group fails. which works with thiserelationship. it will offline the source as well. k ng ense a b a Affinity lic Strong Positive With Failover Delegation b e u l k b ya fera l u ns variation on strong positive affinity is set with the +++ syntax: bd trAaslight a g on. the failover can succeed.2 Administration Copyright 2009 Sun Microsystems. Revision C a . the source group still cannot fail over to a node where the target is not running. So why does strong positive affinity exist? ● s ) ha The relationship can be between a failover group (source) and a scalable group (target). If you have two failover resource groups with a strong positive affinity relationship. But you still want u se t l a them all to be running am ontotheu same node or nodes.

Advanced Resource Group Relationships Strong Negative Affinity Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u You need to be careful with strong negative d because the HA of k taffinities u a y the source group can be compromised. and one node crashes (whichever it is). If there are no more nodes. the source switches offline. Configuring Scalable Services and Advanced Resource Group Relationships Copyright 2009 Sun Microsystems. n ( this S i d lu se a am to u k ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n no aba For example. Oracle and/or its affiliatesฺ A strong negative affinity is set using the following syntax: # clrg set -p RG_affinities=--rg1 rg2 Here. if you have a two-node cluster with both the source and target groups online (on different nodes). categorically refuses to run on the same node. it chases the source out of the way and the source switches to a different node. Revision C 11-19 a . All Rights Reserved. Inc. It absolutely refuses to switch to any node where the target is running. the source cannot run on the same node as the target. if any. Sun Services. only the target group can remain running because the source group absolutely. If you switch the target to the node where the source is running.

11-20 Sun™ Cluster 3. Sun Services. enclosure_name. when you see italicized names. node1.2 Administration Copyright 2009 Sun Microsystems. such as IPaddress. you complete the following tasks: ● Task 1 – Preparing for Apache Data Service Configuration ● Task 2 – Configuring the Apache Environment ● Task 3 – Testing the Server on Each Node Before Configuring the Data Service Resources ● Task 4 – Registering and Configuring the Sun Cluster Apache Data Service s ) ha y m ฺ Task 6 – Observing Cluster Failures om ฺ c ฺ m uide Task 7 – Configuring Advanced Resource Group csRelationships e G t @ n b ku tude a y Preparation n ( this S i d lu se a The following tasks are uin this section: amexplained o k t g SunnCluster nfor se HA for Apache registration and Preparing a b e a configuration lic b e u l k Registering a ab and configuring the Sun Cluster HA for Apache data r y l e f bdu transservice a g onVerifying Apache web server access and scalable capability n a n b a ● Task 5 – Verifying Apache Web Server Access ● ● ● ● ● Note – During this exercise.Exercise: Installing and Configuring Sun Cluster Scalable Service for Apache Exercise: Installing and Configuring Sun Cluster Scalable Service for Apache Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. substitute the names appropriate for your cluster. Oracle and/or its affiliatesฺ In this exercise. Revision C a . Inc. or clustername embedded in a command string.

Revision C 11-21 a . This directory is deleted every time you reboot. Verify that the entry for your cluster smexistsuion c e G on your cluster vnchost. the vnchost already contains theฺc appropriate dethe for each cluster.Exercise: Installing and Configuring Sun Cluster Scalable Service for Apache Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Letting this script create it resolves the problem. create /etc/hosts entry on your de ku the u a t y administrative workstation as well. and use the same IP address to create the entry t @ n b nodes. Install the Sun Cluster Apache data service software package: # cd sc32_location/Solaris_sparc/Product # cd sun_cluster_agents/Solaris_10/Packages # pkgadd -d . In a non-RLDC environment. All Rights Reserved.sfeMake a resource-specific copy of the /usr/apache2/bin/apachectl d b a tra script and edit it: g n n o a n # mkdir /global/web/bin # cp /usr/apache2/bin/apachectl /global/web/bin # vi /global/web/bin/apachectl a. Oracle and/or its affiliatesฺ Task 1 – Preparing for Apache Data Service Configuration Perform the following steps on each node of the cluster: 1.conf' Configuring Scalable Services and Advanced Resource Group Relationships Copyright 2009 Sun Microsystems. Add a line to make an Apache runtime directory. Inc. perform the following steps: a a r y ul n1. n ( this S i d lu se a am to u k Task 2 – Configuring ng ensethe Apache Environment a b a lic b e u l kOn (any) bone node of the cluster. SUNWscapc 2. Create an entry in /etc/hosts for the shared address you will be configuring with the Apache web server: IP_address clustername-web s ) ha ab y m ฺ om ฺ entry Note – In the RLDC. Sun Services. Locate the line: HTTPD='/usr/apache2/bin/httpd' And change it to: HTTPD='/usr/apache2/bin/httpd -f /global/web/conf/httpd. Just add the line in bold as the second line of the file: #!/bin/sh mkdir -p /var/run/apache2 b.

Inc.cgi from the classroom server to /global/web/cgi-bin. sCreate er directories for the Hypertext Markup Language (HTML) f u d ab -tran and CGI files and populate with the sample files. Copy the file called test-apache.1 ServerName clustername-web s c Guid e @ ent DocumentRoot "/var/apache2/htdocs" DocumentRoot b "/global/web/htdocs" u ak"/global/web/htdocs"> tud y <Directory "/var/apache2/htdocs"> <Directory ( S in this d u ScriptAlias /cgi-bin/ e /cgi-bin/ al ScriptAlias s m u "/var/apache2/cgi-bin/" [one kline] "/global/web/cgi-bin/" [one line] a to g e <Directory "/var/apache2/cgi-bin"> ban licens <Directory "/global/web/cgi-bin"> a b u le k b a a l y4. Oracle and/or its affiliatesฺ # mkdir /global/web/conf # cp /etc/apache2/httpd. Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.conf. The changes are shown in their order in the file. All Rights Reserved.0.conf-example to /global/web/conf/httpd.conf-example /global/web/conf/httpd. so you can search for the first place to change. # chmod 755 /global/web/cgi-bin/test-apache. Edit the /global/web/conf/httpd.conf File Old Entry New Entry KeepAlive On KeepAlive Off s ) ha y m ฺ Listen 80 Listen clustername-web:80 om ฺc eฺ m ServerName 127. Revision C a . g # cp -rp /var/apache2/htdocs /global/web n non a b # cp -rp /var/apache2/cgi-bin /global/web a 5.0. Table 11-3 Entries in the /global/web/conf/httpd. and so on.Exercise: Installing and Configuring Sun Cluster Scalable Service for Apache 2.conf file. Sun Services. Make sure that test-apache. as shown in Table 11-3.conf 3.2 Administration Copyright 2009 Sun Microsystems. then search for the next. You use this file to test the scalable service. Copy the sample /etc/apache2/httpd. change it.cgi 11-22 Sun™ Cluster 3. and change the following entries.cgi can be executed by all users.

conf -k webservd 4602 4601 0 10:20:05 -f /global/web/conf/httpd.conf -k webservd 4605 4601 0 10:20:05 -f /global/web/conf/httpd.conf -k root 4601 1 0 10:20:04 -f /global/web/conf/httpd. Verify that the server is running. on one node at a time.conf -k webservd 4606 4601 0 10:20:05 -f /global/web/conf/httpd. # ifconfig pubnet_adapter addif clustername-web netmask + broadcast + up 2. 1. steps 1-7. Temporarily configure the logical shared address (on one node). as h ? 0:00 /usr/apache2/bin/httpd ) y start m ฺ ? 0:00 /usr/apache2/bin/httpd om ฺ c ฺ start csm Guide e ? 0:00 /usr/apache2/bin/httpd @ ent b start u d ak 0:01 ? tu/usr/apache2/bin/httpd y ( S start in this d u al? use 0:00 /usr/apache2/bin/httpd m ka e ?start to g 0:00 /usr/apache2/bin/httpd n ens a b start a lic b e u l k rab a y ul nsfe d b a tra g n n no aba vincent:/# ps -ef|grep apache2 webservd 4604 4601 0 10:20:05 -f /global/web/conf/httpd. All Rights Reserved. Revision C 11-23 a . Sun Services. Oracle and/or its affiliatesฺ Task 3 – Testing the Server on Each Node Before Configuring the Data Service Resources Perform the entire task. Inc. # /global/web/bin/apachectl start 3.conf -k Configuring Scalable Services and Advanced Resource Group Relationships Copyright 2009 Sun Microsystems.Exercise: Installing and Configuring Sun Cluster Scalable Service for Apache Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.conf -k webservd 4603 4601 0 10:20:05 -f /global/web/conf/httpd. Start the server (on that node).

e u l k a erab y l f the Apache web server. # ifconfig pubnet_adapter removeif clustername-web 11-24 Sun™ Cluster 3. ns you might need to disable proxies or set a Note – If you bacannot c a i bexception proxy inl your web browser. Sun Services. Verify that the server has stopped. Use http://clustername-web (Figure 11-3). 0:00 grep apache2 Take down the logical IP address. u 5.Exercise: Installing and Configuring Sun Cluster Scalable Service for Apache Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. nsStop d b a tra g #n /global/web/bin/apachectl stop n o a n ab 6. Connect to the server from the web browser on your administration or display station.2 Administration Copyright 2009 Sun Microsystems. Revision C a . s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( dinPagee this Figure 11-3 Apache Serverlu Test a us m a k e to g n econnect. All Rights Reserved. # ps -ef | grep apache2 root 8394 8393 0 17:11:14 pts/6 7. Oracle and/or its affiliatesฺ 4.

# clrg create -n node1. Inc.node2 sa-rg 3.apache -p Bin_dir=/global/web/bin \ -p Scalable=TRUE -p Resource_dependencies=clustername-web.apache 2. # clrg status # clrs status Configuring Scalable Services and Advanced Resource Group Relationships Copyright 2009 Sun Microsystems. Bring the scalable resource group online: # clrg online -M web-rg 9.HAStoragePlus \ # clrs create -gbweb-rg u le k -p FilesystemMountPoints=/global/web -p AffinityOn=false web-stor b a a r y l e f bdu tran7. Register the resource type required for the Apache data service. ton ya to run 5.web-stor \ apache-res 8.) (The example assumes dtwo his t u l e a-S -n unode1. Revision C 11-25 a .Exercise: Installing and Configuring Sun Cluster Scalable Service for Apache Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. s ) ha Add the SharedAddress logical host name resource to the resource group. Oracle and/or its affiliatesฺ Task 4 – Registering and Configuring the Sun Cluster Apache Data Service Perform the following steps only on (any) one node of the cluster: 1. ab y m ฺ # clrssa create -g sa-rg clustername-web com ฺ ฺ e m d s i 4. Create a failover resource group for the shared address resource. Verify that the data service is online. If you have more than two nodes. c e t Gu @ # clrg online -M sa-rg b den u k u all nodes of the cluster. Create a scalable resource group ( S n i nodes. a b e a -tlicSUNW. All Rights Reserved. you can include more than two nodes after the -n.s Add an application resource to the scalable resource group. Bring the failover resource group online.node2 s # clrg create web-rg m a o k t g nresource 6. Sun Services. # clrt register SUNW. Add anstorage se to the scalable group. a ang non # clrs create -g web-rg -t SUNW.

Over time. n a n ab 3. a g on. Inc. y l e f bdu transSharedAddress resource is online. however. All Rights Reserved. 4. Repeatedly press the Refresh or Reload button on the browser. s ) ha y m ฺ om ฺ c ฺ e Task 6 – Observing Cluster Failures ecsm uid G t @ n b u at autime. 11-26 Fail (or bring to the OK prompt) one of the nodes running the scalable service.2 Administration Copyright 2009 Sun Microsystems. Revision C a .2. so you might see several consecutive requests serviced by the same node.cgi. de and fix each one k(one a Cause as many of the following failures t y ( can. The test-apache. Connect to the web server from a browser anywhere else besides a cluster node using http://clustername-web/cgi-bin/testapache. Reboot one of the nodes running the scalable service. you should see the load balancing be 50-50. Oracle and/or its affiliatesฺ Perform the following steps to verify the Apache web server access and scalable utility: 1.isgiven S your physical access to n you before going on to the next one) as i d h lu se t the cluster.Exercise: Installing and Configuring Sun Cluster Scalable Service for Apache Task 5 – Verifying Apache Web Server Access Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. The load balancing is not performed on a round-robin basis. a am to u k Perform the following ng ensteps se to observe the behavior of the scalable a b service: a lic b e u l k Fail raasingle b public network interface on the node where the a 1. Sun Services.cgi script shows the name of the cluster node that is currently servicing the request. 2. Sun™ Cluster 3. Fail all public network interfaces on that node.

you observe the effects of configuring variations of the RG_affinities property. Put your myapp-rg resource group offline. a n b e a Observe what lic happens to Apache. Revision C 11-27 a . 2. Sun Services. Oracle and/or its affiliatesฺ Task 7 – Configuring Advanced Resource Group Relationships In this task. Does it come back online on the b e u l k first bnode? a a r y ul nsfe# clrg switch -n othernode myapp-rg d b # clrg status a tra g n n a 6. complete Task 8 in Module 10. web-rg Set a weak positive affinity so that myapp-rg (source) always prefers to run on the same node as nfs-rg (target). Switch the myapp-rg offline. # clrg offline myapp-rg 3. If you have not yet done so. y m ฺ om ฺ c ฺ # clrg set -p RG_affinities=--myapp-rg smweb-rg ide c u e G is running. Switch your nfs-rg so that it is not on the preferred node of myapprg: # clrg switch -n nonpreferrednode nfs-rg Configuring Scalable Services and Advanced Resource Group Relationships Copyright 2009 Sun Microsystems. Perform the following steps: 1. Print out the value of the Nodelist for myapp-rg. All Rights Reserved. # clrg show -p Nodelist myapp-rg 9. This means that Apache categorically refuses to run on any node where myapp-rg is running. # clrg offline myapp-rg # clrg status # clrg set -p RG_affinities="" 7. dethe console messages as kuObserve u a t y well.Exercise: Installing and Configuring Sun Cluster Scalable Service for Apache Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. n ( this S i d # clrg switch -nlusomenode myapp-rg e a s # clrg status am to u k g myapp-rg 5. Switch the myapp-rg onto some node@ where Apache t n b Observe what happens to Apache. s ) ha Create a strong negative affinity with the Apache resource group as the source and myapp-rg as the target. Inc. Can Apache come back online on more no ab nodes? Now remove the relationship. Switch nthe se to another node where Apache is running. 4. # clrg set -p RG_affinities=+nfs-rg myapp-rg 8.

Bring myapp-rg online without specifying the node. Switch the myapp-rgaso luit is noselonger on the same node as nfs-rg. u Can you do it? am o k t e # clrg aswitch myapp-rg ng e-n sothernode n b # clrg status c a li b e u l k b 15. All Rights Reserved. Switch the myapp-rg offline and now change the affinity to a strong positive affinity with delegation. if you switch the target group nfs-rg? a What ahappens r y l e f switch -n othernode nfs-rg bdu trans## clrg a clrg status g onn a n ab # clrg offline myapp-rg # clrg set -p RG_affinities=+++nfs-rg myapp-rg 16. idedoes it csm Where u e end up? G t @ n b # clrg online myapp-rg ku tude a y # clrg status n ( this S i d 14.Exercise: Installing and Configuring Sun Cluster Scalable Service for Apache 10. Can you do it? Is a weak affinity a preference or a requirement? # clrg switch -n othernode myapp-rg # clrg status 12. What is the difference between ++ and +++ (use +++)? The answer lies a few steps further on. Where does it end up? Why does it not end up on the first node of its own node list? Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C a . Inc. Oracle and/or its affiliatesฺ # clrg online myapp-rg # clrg status 11. Switch the myapp-rg so it is no longer on the same node as nfs-rg. 11-28 Sun™ Cluster 3. Sun Services. s ) ha y m ฺ om ฺ c ฺ 13. Bring myapp-rg online without specifying the node. What happens if RGM wants to failover the source myapp-rg because its fault monitor indicates application failure? Kill myappdaemon on whichever node it is running a few times (be patient with restarts) and observe.2 Administration Copyright 2009 Sun Microsystems.

issues. or discoveries you had during the lab exercises.Exercise Summary Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. Oracle and/or its affiliatesฺ Exercise Summary ? ! Discussion – Take a few minutes to discuss what experiences. Revision C 11-29 a . Inc. All Rights Reserved. ● Experiences ● Interpretations ● Conclusions ● Applications s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Configuring Scalable Services and Advanced Resource Group Relationships Copyright 2009 Sun Microsystems.

Oracle and/or its affiliatesฺ y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha a .Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

you should be able to: c ฺ ide csm u Configure a scalable application in non-global zones e G t @ n b Configure HA-Oracle in a Sun Cluster software de environment as a ku 3. All Rights Reserved. Sun Services. Oracle and/or its affiliatesฺ Module 12 Performing Supplemental Exercises for Sun Cluster 3.2 u a t y failover application n ( this S i d Configure Oracle a RAC lu 11gsinea Sun Cluster 3.2 software environment am to u k ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n a no ● ● ● ab 12-1 Copyright 2009 Sun Microsystems.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc.2 Software s ) ha Objectives y m ฺ om ฺ Upon completion of this module. Revision C a .

Relevance Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. Inc. Oracle and/or its affiliatesฺ Relevance ? ! Discussion – The following questions are relevant to understanding the content this module ● Why is it more difficult to install Oracle software for a standard Oracle HA database on the local disks of each node rather than installing it in the shared storage? ● Is there any advantage to installing software on the local disks of each node? ● How is managing Oracle RAC different from managing the other types of failover and scalable services presented in this course? y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha 12-2 Sun™ Cluster 3. Revision C a . Sun Services.2 Administration Copyright 2009 Sun Microsystems.

Revision C 12-3 a . All Rights Reserved. Oracle and/or its affiliatesฺ Additional Resources Additional resources – The following references provide additional information on the topics described in this module: s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Performing Supplemental Exercises for Sun Cluster 3.2 Software Copyright 2009 Sun Microsystems. Inc. Sun Services.Additional Resources Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

The r y Many l f ns group manager (RGM) is fully knowledgeable about zones and bdu trresource a a g on. ab ● Task 4 – Verify That the Apache Application Is Running Correctly in the Non-global Zones If you already have an application running across the physical nodes in the cluster (global zone). All Rights Reserved. The same HAStoragePlus resource that is used to control global or failover storage in global zones can be configured into a resource group that runs across non-global zones.2 Administration Copyright 2009 Sun Microsystems. it is a relatively simple operation to move it into a non-global zone. Revision C a . Sun Services. m a k e to g ns Clustered Applications in Zones ba–nRunning e Background c a li b e u l k a ofethe abclustered applications are supported in non-global zones. HAStoragePlus methods automatically arrange for non-global zones to gain access to your failover or global file system through simple loopback mounts. Oracle and/or its affiliatesฺ This exercise demonstrates running a clustered application in non-global zones. Inc.treats them as if they were virtual nodes as far as configuring resource n a n groups. In this exercise. you complete the following tasks: ● Task 1 – Configuring and Installing the Zones ● Task 2 – Booting the Zones ● Task 3 – Modifying Your Resource Groups So Apache Runs in the Non-global Zones s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b Preparation u ak Stud y ( s background information in din eread hithe Before continuing with this lexercise.Exercise 1: Running a Clustered Scalable Application in Non-global Zones Exercise 1: Running a Clustered Scalable Application in Non-global Zones Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. t u a us this section. 12-4 Sun™ Cluster 3.

The file </myzone/root/var/sadm/system/logs/install_log> contains a log of the zone installation. All Rights Reserved.2 Software Copyright 2009 Sun Microsystems. Configure the zone. Make sure you use a different IP for the zone on each node. Performing Supplemental Exercises for Sun Cluster 3. so get them up to where they are running the installation stage (step 3) simultaneously.y.w a tra g n zonecfg:myzone:net> set physical=node_pubnet_adapter n o a n b zonecfg:myzone:net> end a Note – Only scalable load-balanced applications require a dedicated nonglobal zone IP address. so as not to waste time.Exercise 1: Running a Clustered Scalable Application in Non-global Zones Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. # zoneadm -z myzone install Preparing to install zone <myzone>. Sun Services. Use 'create' g zonecfg:myzone> an cens create b a set zonepath=/myzone li b zonecfg:myzone> e u l k set autoboot=true a ezonecfg:myzone> ab r y l zonecfg:myzone> add net f u ns d b zonecfg:myzone:net> set address=x. u ak Stud y ( in this d # zonecfg -z myzone u e alzoneuconfigured s myzone: No such m ka toe begin to configuring a new zone. zonecfg:myzone> commit zonecfg:myzone> exit 3. . Allocate a dedicated public net IP address for the zone. . Inc. Revision C 12-5 a . Install the zone. It can take a while to build your zones.z. Make sure you use an IP that is not going to conflict with any other node or previously used application IP or IPMP test address. . Creating list of files to copy from the global zone. 1. You will identify the address in step 2. You could run failover applications in non-global zones without dedicating public network IP addresses for the zones. You are not required to put it in any /etc/hosts file. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b 2. Oracle and/or its affiliatesฺ Task 1 – Configuring and Installing the Zones Perform the following steps on all nodes.

it will c a i l reboot You can stay connected to the zone console. e u l k b a /etc/hosts file in the zone (on each node) and add the IP rthe y3. b automatically. or you may have trouble with the rest of the configuration screens.Exercise 1: Running a Clustered Scalable Application in Non-global Zones Task 2 – Booting the Zones On each node. It will look similar to a standard Solaris™ OS that is booting after a sysunconfig: # zlogin -C myzone [Connected to zone 'myzone' console] s ) ha Wait until the SMF services are all loaded. vt100 usually works. Revision C a . y m ฺ om ฺ c ฺ csm Guide e t n b@ e u Note – While the zone on each node willkhave the same d zone name u a t y (SolarisishostS name in the configuration (myzone). Do not choose sun. 12-6 Disconnect from the zone console using ~. Edit the /etc/default/login file in the zone to allow root logins: # vi /etc/default/login [Comment out the CONSOLE=/dev/console line] # exit 5. Oracle and/or its affiliatesฺ 1. Sun Services.2 Administration Copyright 2009 Sun Microsystems. Use the same name and IP address a g onthat you have in the /etc/hosts file in the global zone. and navigate through the configuration screens. Get your terminal type correct.lu so you cantdistinguish between them (and so e a s that the Apache proof ofm concept CGI program will make sense). give each zone a different n i d h screens (myzone-nodename). # zoneadm -z myzone boot 2. u a o k t g nse nhave a Whenb you efinished the system configuration in the zone. Sun™ Cluster 3. All Rights Reserved. Boot the zone.a Edit l e f u s bd tran address for clustername-web. Connect to the zone console and configure the zone. perform the following steps: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. n a n ab 4. Inc.

Once the resource group is running in the zones. This is normal. unmanage m a k toeproperly to reconfigure the shared address in all the This is required g zones.node3:myzone] web-rg d u l e a us and then remanage all the groups. Performing Supplemental Exercises for Sun Cluster 3. On all nodes (still in the global zone). All Rights Reserved.node3:myzone] y ( in this # clrg set -n node1:myzone. From any one node. manually do a loopback mount of the /global/web filesystem into the zone.web-rg + k b a a l y sfe#r clrg unmanage sa-rg web-rg u d b tran # clrg online -Me sa-rg web-rg a ang non Note – You may see console messages about the scalable service being registered with the physical node name. From any one node. s ) ha ab y m ฺ om ฺ c # mkdir -p /myzone/root/global/web ฺ # mount -F lofs /global/web /myzone/root/global/web csm Guide e @ ent 3. 4. Sun Services.2 Software Copyright 2009 Sun Microsystems.node2:myzone[. the HAStoragePlus methods will automatically be doing the mounts. although the application will really be running in the zone. Oracle and/or its affiliatesฺ Task 3 – Modifying Your Resource Groups So Apache Runs in the Non-global Zones Perform the following in the global zone on the nodes indicated: 1. Revision C 12-7 a . From any one node.node2:myzone[. banandlicstart ensthe load-balanced service: a b # clrs u le disable -g sa-rg.Exercise 1: Running a Clustered Scalable Application in Non-global Zones Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. modify the node lists: b u ak Stud sa-rg # clrg set -n node1:myzone. Inc. offline both the scalable and shared address resource groups: # clrg offline web-rg sa-rg 2. This will allow the apache resource to validate correctly when the node list of its group is changed to the zones.

Revision C a . and to demonstrate that the load balancing is working when the scalable application is running across non-global zones.cgi.Exercise 1: Running a Clustered Scalable Application in Non-global Zones Task 4 – Verify That the Apache Application Is Running Correctly in the Non-global Zones Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. On each node. log in to the non-global zone and verify that the Apache application is running within the zone: s ) ha y m ฺ om ฺ c ฺ (in the non-global zone) sm uide c e G # ps -ef|grep apache2 t @ n b # exit ku tude a y S point a Web browser 3.ble k a y fera l u bd trans a g onn a n ab # zlogin myzone 12-8 Sun™ Cluster 3. i d to http://clustername-web/cgi-bin/test-apache. Inc. might get a response from the same zone several times c a i l b uin a row. from the global zone. nyou a b e therefore. On any node. Oracle and/or its affiliatesฺ Perform the following steps to verify the correct operation of Apache.2 Administration Copyright 2009 Sun Microsystems. working across e It is not round-robin load balancing. verify the cluster application status: # clrg status # clrs status 2. On the administration workstation n ( thorisdisplay. 1. All Rights Reserved. Sun Services. Press lu se a m totimes u to demonstrate that load balancing is reload or refreshaseveral k g thenszones.

HAStoragePlus Type r y l f bdu trans Task 9 – Installing and Registering Oracle Data Service a g onn Task 10 – Creating Resources and Resource Groups for Oracle a n ab ● Task 3B – Creating a Shared Storage File System for Oracle Software (SVM) ● ● ● ● ● ● ● ● ● Task 11 – Verifying That Oracle Runs Properly in the Cluster Preparation Before continuing with this exercise.2 Software as a Failover Application In this exercise you integrate Oracle 11g into Sun Cluster 3. Performing Supplemental Exercises for Sun Cluster 3. Revision C 12-9 a . Inc.2 Software as a Failover Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. you complete the following tasks: ● Task 1 – Creating a Logical IP Entry in the /etc/hosts File ● Task 2 – Creating oracle and dba Accounts ● Task 3A – Creating a Shared Storage File System for Oracle Software (VxVM) s ) ha y m ฺ om ฺ c ฺ e m Oracle idSoftware Task 3C: Create a Shared Storage File System csfor u e G (ZFS) t @ n b de ku Environment u a Task 4 – Preparing the oracle User t y ( is S nControl i d Task 5 – Disabling Access th of X Server on the Admin u l e a Workstation m us a o k t Task 6n–g Running sethe runInstaller Installation Script a n b e a 7 – Preparing Task an Oracle Instance for Cluster Integration lic b e u l k a eTask ab8 – Registering the SUNW. read the background information in this section.2 software as a failover application.Exercise 2: Integrating Oracle 11g Into Sun Cluster 3. Oracle and/or its affiliatesฺ Exercise 2: Integrating Oracle 11g Into Sun Cluster 3. In this exercise. Sun Services.2 Software Copyright 2009 Sun Microsystems. All Rights Reserved.

12-10 Sun™ Cluster 3. you cannot use a failover file system if you want your application to fail over to a nonstorage node.toseofinstall. You will not be able to use a nonstorage node (just pretend it does not exist). In this exercise there are options to use VxVM. the application runs faster after it is up and running). or ZFS. Installing an Application on Local Storage (Each Node) or Shared Storage s ) ha For ease of installation and management.2 environment. b u le k b a a l y sfer u d n of File System and Storage b trChoice a a g onn a n A failover file system is optimal for performance (although actual failover ab time is longer. Sun Services. All Rights Reserved. While it is more difficult you get the advantage of m to u arolling k being able to perform maintenance (including patching) of the g e n s a node software onbone while n keeping your database alive on the other e c a i l node. SVM. 3B.” You will be installing and configuring the software exactly as you would on a stand-alone system and then manipulating the configuration so that it listens on a failover logical IP address. the Oracle software is “clusterunaware. this lab takes the strategy of installing the Oracle binaries themselves directly in shared storage. The disadvantage is that there would be no way to perform maintenance or patching on the software while keeping your application available on another node.Exercise 2: Integrating Oracle 11g Into Sun Cluster 3. or 3C depending on your choice of storage management. Revision C a . y m ฺ om ฺ c ฺ csm Guide e @ ent b u d tucopies yakseparate ( S In production. you only need to perform the Oracle installation once. a storage). Inc.2 Administration Copyright 2009 Sun Microsystems. you might want toninstall of the software s i i d h locally on each node (with the course. Therefore. However. Choose only one of tasks. being in the shared t lu data. but it is unlikely anyone would ever use a global file system or a non-storage node for HA-Oracle. This exercise uses a failover file system. Like the majority of failover applications.2 Software as a Failover Background Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. 3A. Oracle and/or its affiliatesฺ It is relatively straightforward to configure the Oracle 11g Database software as a failover application in the Sun Cluster 3.

x.2 Software Copyright 2009 Sun Microsystems. k a erab y l u nsf # passwd oracle d b New password: oracle a tra g n Re-enter new Password: oracle n a no ab Performing Supplemental Exercises for Sun Cluster 3. Oracle and/or its affiliatesฺ Task 1 – Creating a Logical IP Entry in the /etc/hosts File Perform the following task: Create an entry for a logical IP address for Oracle in the /etc/hosts file on all storage nodes as follows: # vi /etc/hosts .y. All Rights Reserved. Create the oracle user. the d u l e a ituwill home directorym because be located in shared storage that is not s a o yet created. perform omthe ฺfollowing c ฺ steps on all storage nodes: csm Guide e 1.. Sun Services.z. Inc.w ora-lh Task 2 – Creating oracle and dba Accounts s ) ha y m ฺ To create the oracle user and dba group accounts..2 Software as a Failover Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Create the dba group account: @ ent b u # groupadd -g 8888 dba yak tud ( S n arethspecifying. but not creating. iYou is 2.Exercise 2: Integrating Oracle 11g Into Sun Cluster 3. Revision C 12-11 a .k t g e n s n 8888 -d /oracle oracle # useradd -s /bin/ksh ba-g dba e-u c a i l b Createlethe oracle user password: u 3.

The g n n no nodelist property contains all nodes physically connected to the aba storage. Revision C a . All Rights Reserved. On the node on which you created the storage group. Only step 6 is performed on all storage nodes.node2 oradg 5. we will create an unmirrored volume. # cldg create -t vxvm -n node1. Oracle and/or its affiliatesฺ Task 3A – Creating a Shared Storage File System for Oracle Software (VxVM) Perform the following steps on any node physically connected to the storage.2 Administration Copyright 2009 Sun Microsystems.2 Software as a Failover Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. s ) ha y m ฺ om ฺ c ฺ sm uide c e # vxdisk -o alldgs list G t @ n b # zpool status ku tude a # metaset y n ( this S i # cldev list -v d lu thissdisk e a 2.a Register the new disk group (and its volume) with the cluster. You can always add another disk and mirror your volume later at your leisure. Note – Just to speed up the lab. The following example checks against VxVM. Make sure that you do not use any disks already in use in existing device groups. or pretend that you were going to. Create a mount point and an entry in /etc/vfstab on all storage nodes: # mkdir /oracle # vi /etc/vfstab /dev/vx/dsk/oradg/oravol /dev/vx/rdsk/oradg/oravol /oracle ufs 2 no - 7. Inc. Solaris Volume Manager. mount the file system and change its ownership: # mount /oracle # chown oracle:dba /oracle 12-12 Sun™ Cluster 3. Sun Services. Create a disk group using m u ka e to -i c#t#d# g # /etc/vx/bin/vxdisksetup n oradg ns ora1=c#t#d# bainit e # vxdg c a li b a volume e u l 3. Create to hold the Oracle binaries and data. and ZFS disks.Exercise 2: Integrating Oracle 11g Into Sun Cluster 3. Select a disk from shared storage. Create a UFS file system on the volume: # newfs /dev/vx/rdsk/oradg/oravol 6. k b a a r y ul ns#fevxassist -g oradg make oravol 6g d b a tr4. Note the logical device name (referred to as c#d#t# in step 2). 1.

Exercise 2: Integrating Oracle 11g Into Sun Cluster 3.2 Software as a Failover

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Task 3B – Creating a Shared Storage File System for
Oracle Software (SVM)
Perform the following steps on any node physically connected to the
storage. Only step 5 is performed on all storage nodes. This lab assumes
that local metadb’s have been created already.
Note – Just to speed up the lab, we will create unmirrored volumes. You
can always add another disk and mirror the parent volume later at your
leisure, or pretend that you were going to.
1.

s
) ha

Select a disk from shared storage.. Make sure that you do not use any
disks already in use in existing device groups. Note the DID device
name (referred to as d# in step 2). The following example checks
against VxVM, Solaris Volume Manager, and ZFS disks.

y
m

om ฺ
c

sm uide
c
e
# vxdisk -o alldgs list
G
t
@
n
b
# zpool status
ku tude
a
# metaset
y
( is S
# cldev list -v din
u se th
l
a
2. Create a diskset
u disk.
amusingtothis
k
# metaset
orads
se -a -h node1 node2
ang ce-s
#ab
metaset
-snorads -a /dev/did/rdsk/d#
li
b Createlea soft
u
3.
partition (d100) of a mirror volume for Oracle. (SVM
k
b
a
a
r
y
ul nsfestill calls it a mirror, although we only have one submirror. This
d
b
gives us flexibility to add the other half of the mirror at our leisure
a
a
r
t
g
later.)
n non
a
b
a
# metainit -s orads d11 1 1 /dev/did/rdsk/d#s0
# metainit -s orads d10 -m d11
# metainit -s orads d100 -p d10 6g
4.

Create a file system:
# newfs /dev/md/orads/rdsk/d100

5.

Create a mount point and an entry in /etc/vfstab on all storage
nodes:

# mkdir /oracle
# vi /etc/vfstab
/dev/md/orads/dsk/d100 /dev/md/orads/rdsk/d100 /oracle ufs 2 no -

6.

On the node on which you created the storage, mount the file system
and modify its permission:
# mount /oracle
# chown oracle:dba /oracle

Performing Supplemental Exercises for Sun Cluster 3.2 Software
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

12-13

a

Exercise 2: Integrating Oracle 11g Into Sun Cluster 3.2 Software as a Failover

Task 3C: Create a Shared Storage File System for
Oracle Software (ZFS)
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Perform the following steps on any one node physically connected to the
storage.
Note – Just to simplify, we will create a ZFS storage pool with one disk.
You can always add another disk and mirror the existing disk later.
1.

Select a disk from shared storage.. Make sure that you do not use any
disks already in use in existing device groups. Note the device name
(referred to as c#t#d# in step 2). The following example checks
against VxVM, Solaris Volume Manager, and ZFS disks.

s
) ha

y
m

# vxdisk -o alldgs list
om ฺ
c

# zpool status
csm Guide
e
# metaset
@ ent
b
# cldev list -v
u
aksystem
tuford Oracle. Verify that
y
(
S
2. Create a ZFS storage pool and
a
file
inand change
d
his its permissions:
your file system is mounted
t
u
l
e
a uc#t#d#
s
m
# zpool create
orapool
a
o
k
t
# zfs create
orapool/orafs
e
ng e-onsmountpoint=/oracle
a
b
# df
-h
a oracle:dba
lic
#bchown le
/oracle
u
k
b
a
a
l y sfer
u
d
a4n– Preparing the oracle User Environment
abTask
r
t
g
n non
a
b
a

To configure environment variables required for the Oracle software, run
the following commands from the cluster node which owns the Oracle
storage:
1.

Switch to the oracle user:
# su - oracle

2.

Edit the .profile file as shown. Be sure to substitute the proper
value for your DISPLAY variable.
$ vi .profile
ORACLE_BASE=/oracle
ORACLE_HOME=$ORACLE_BASE/product/11.1.0/db_1
ORACLE_SID=MYORA
PATH=$PATH:$ORACLE_HOME/bin
DISPLAY=display-name-or-IP:#
export ORACLE_BASE ORACLE_HOME ORACLE_SID PATH DISPLAY

12-14

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Exercise 2: Integrating Oracle 11g Into Sun Cluster 3.2 Software as a Failover

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

$ exit

Task 5 – Disabling Access Control of X Server on the
Admin Workstation
Perform the following task:
To allow client GUIs to be displayed, run the following command on
the admin workstation:
(# or $) /usr/openwin/bin/xhost +

s
) ha

Task 6 – Running the runInstaller Installation Script
y

ฺm
m
co ethe
ฺperform

To install the Oracle software on the local file system,
m
d
s
i
u
following steps on the node that owns the Oracle
ec storage:
G
t
@
b den
u
1. Switch to the oracle user:
k
ya Stu
(
# su - oracle
n
i
is
d
h
t
u
l
e
2. Change directoryato the location
s of the Oracle software:
m
u
a
o
k et
$ cd ORACLE-11gR1-db-software-location
g
n
ns
3. Run
script:
bathe runInstaller
e
c
a
i
l
b
u $ ./runInstaller
le
k
b
a
a
r
l y 4.sfeRespond
to the dialogs by using Table 12-1.
u
d
b tran
a
g 12-1
anTable
non The runInstaller Script Dialog Answers

ab

Dialog

Action

Select Installation Method

Click Advanced Installation (near the bottom).
[The basic installation makes some wrong choices
for Sun Cluster.]
Click Next.
Click OK when you get the error about the
previous inventory location (there obviously was
none).

Performing Supplemental Exercises for Sun Cluster 3.2 Software
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

12-15

a

Exercise 2: Integrating Oracle 11g Into Sun Cluster 3.2 Software as a Failover

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Table 12-1 The runInstaller Script Dialog Answers (Continued)
Dialog

Action

Specify Inventory Directory and
Credentials

Change the full path of the inventory directory to:
/oracle/oraInventory .
Verify the group name (dba) and click Next.

Select Installation Type

Select the Custom radio button, and click Next.

Install Location

Verify and click Next.
[The path is taken from the $ORACLE_HOME that
you set in your profile, so if it looks wrong, you
need to quit and start again.]

s
) ha

y
m

othemOracle
A popup will warn you against using
c


e
m
d
s
Base as the home directory. Click
Yes.
i
ec t Gu
@
n
Product Specific Prerequisite
The tests will either u
allbsucceed...or..
e
d
k
Checks
you will be warned
swap space
tu
ya about
(
S
n
requirements.
di e this
u
l
a us
m
Click
Next,
and if you get a pop-up warning
a
k window,
toclick
g
e
Yes to proceed.
n ens
a
b
a
lic Deselect the following (so our install goes faster):
Available Product Components
b
e
u
l
k rab
a
Enterprise Edition Options
y
l
e
f
u
s
bd tran
Oracle Programmer 11.1.0.6.0
a
g
n
n
a
no

ab

Click Next.

Privileged Operating System
Groups

Verify that they all say dba, and click Next.

Create Database

Verify that the Create a Database radio button is
selected, and click Next.

Summary

Verify, and click install.

Oracle Net Configuration Assistant
Welcome

Verify that the Perform Typical Configuration
check box is not selected, and click Next.

Listener Configuration, Listener
Name

Verify that the Listener name is LISTENER, and
click Next.

Select Protocols

Verify that TCP is among the Selected Protocols,
and click Next.

12-16

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Exercise 2: Integrating Oracle 11g Into Sun Cluster 3.2 Software as a Failover

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Table 12-1 The runInstaller Script Dialog Answers (Continued)
Dialog

Action

TCP/IP Protocol Configuration

Verify that the Use the Standard Port Number of
1521 radio button is selected, and click Next.

More Listeners

Verify that the No radio button is selected, and
click Next.

Listener Configuration Done

Click Next.

Naming Methods Configuration

Verify that the No, I Do Not Want to Configure
Additional Naming Methods radio button is
selected, and click Next.

s
) ha

y
m

om ฺ
DBCA Step 1: Database Templates
Select the “General Purpose or Transaction
c

Processing” radio button, and
Next.
ide
csmclick
u
e
G Name text
t
@
Step 2: Database Identification
Type MYORA in the b
Global
Database
n
de in the SID text
ku it istechoed
field, observeathat
u
y
field, and
n (clickthNext.
sS
i
i
d
lu s“Configure
e
a
Step 3: Management Options
Uncheck
Enterprise Manager” and
m
u
a
o
k click
t Next.
g
e
n
s
Step 4: Database Credentials
ba licen Select the Use the Same Administrative Password
a
b
for All Accounts radio button.
u
le
k
b
a
a
l y sfer
u
Enter cangetin as the password, and click Next.
d
b tran
a
gStep 5:oStorage
n- Options
Verify that File System is selected, and click Next.
n
a
n
b
a
Done

Click Finish.

Step 6: Database File Locations

Select or verify the “Use Database File Locations
from Template” radio button and click Next.

Step 7: Recovery Configuration

Uncheck all the boxes, and click Next.

Step 8: Database Content

Uncheck Sample Schemas, and click Next.

Step 9: Initialization Parameters

On the Memory tab, select the Typical radio
button.
Make the memory size as small as they will let
you (you can tell using the slider next to Memory
Size box). Click Next.

Step 10: Security Settings

Leave the radio button at “Keep the enhanced....”
and click Next.

Performing Supplemental Exercises for Sun Cluster 3.2 Software
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

12-17

a

Exercise 2: Integrating Oracle 11g Into Sun Cluster 3.2 Software as a Failover

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Table 12-1 The runInstaller Script Dialog Answers (Continued)
Dialog

Action

Step 11: Automatic Maintenance
Tasks

Uncheck the checkbox (we’re not hanging around
too long after these labs) and click Next.

Step 12: Database Storage

Click Next.

Step 13: Creation Options

Verify that Create Database is checked, and click
Finish.

Confirmation

Verify summary of database creation options,
parameters, character sets, and data files, and
click OK.

s
) ha

y
m

Note – This might be a good time to takem
o a break.
c

The database configuration assistant
takes e
15ฺ to
m
d
s
i
20 minutes to complete. ec
u
G
t
@
b den
u
k
Database Configuration Assistant
Click Exit.
ya Stu
(
n
i
Script (setup privileges)
Open a d
terminal
window
his as user root on the
t
u
l
e
installer
a node
s and run:
m
u
a
o
k et
g
/oracle/oraInventory/orainstRoot.sh
n
a
ns
b
e
c
a
li Now run
b
e
u
l
k
a erab
/oracle/product/11.1.0/db_1/root.sh
y
l
f
u
s
bd tran
a
Accept the default path name for the local bin
ang non
directory.

ab

Click OK in the script prompt window.
End of Installation

12-18

Click Exit and confirm.

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Exercise 2: Integrating Oracle 11g Into Sun Cluster 3.2 Software as a Failover

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Task 7 – Preparing an Oracle Instance for Cluster
Integration
On the same node from which the Oracle software was installed, perform
the following steps as indicated to prepare the Oracle instance for Sun
Cluster 3.x software integration:
1.

Configure an Oracle user for the fault monitor (as user oracle):

$ sqlplus /nolog
SQL> connect / as sysdba
SQL> create user sc_fm identified by sc_fm;
SQL> grant create session, create table to sc_fm;
SQL> grant select on v_$sysstat to sc_fm;
SQL> alter user sc_fm default tablespace users quota 1m on users;
SQL> quit

s
) ha

ab

y
m

om ฺ
c

e (as user
2. Test the new Oracle user account used by thesm
c faultGmonitor
uid
e
oracle):
@ ent
b
u
$ sqlplus sc_fm/sc_fm
ak Stud
y
SQL> select * from sys.v_$sysstat;
(
in this
SQL> quit
d
u
e oracle). You can be creative about
al (asuuser
s
m
3. Create a sample
table
ka e to
your data
values:
g
an cens
$ sqlplus /nolog ab
li
b sysdba
e
l
SQL> connect k
/ uas
atableemytable
ab (Name VARCHAR(20), age NUMBER(4));
r
y
SQL> create
l
f
s mytable values ('vincent', 14);
du rainto
SQL>binsert
n
a
-t into mytable values ('theo', 14);
g insert
n
o
anSQL>
n
SQL> commit;
SQL> select * from mytable;
SQL> quit

Note – This sample table is created only for a quick verification that the
database is running properly.
4.

Shut down the Oracle instance:
$ sqlplus /nolog
SQL> connect / as sysdba
SQL> shutdown
SQL> quit

5.

Stop the Oracle listener:
$ lsnrctl stop

Performing Supplemental Exercises for Sun Cluster 3.2 Software
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

12-19

a

Exercise 2: Integrating Oracle 11g Into Sun Cluster 3.2 Software as a Failover
6.

Configure the Oracle listener (as user oracle):
$ vi $ORACLE_HOME/network/admin/listener.ora

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

(a) Modify the HOST variable to match logical host name ora-lh.
(b) Add the following lines at the top:
SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(SID_NAME = MYORA)
(ORACLE_HOME = /oracle/product/11.1.0/db_1)
(GLOBALDBNAME = MYORA)
)
)

s
) ha

y
m

$ vi $ORACLE_HOME/network/admin/tnsnames.ora om
ฺc eฺ
m
Modify the HOST variable to match logical host
name
s
c Gora-lh
uid
e
@ ent
b
u
ak Stud
y
Task 8 – Registering the SUNW.HAStoragePlus
Type
(
n this
i
d
lu se
a
m to u
Perform the followingatask:
k
ng(if not
se in a previous exercise), register the
If necessary
done
a
n
b
e
a
SUNW.HAStoragePlus
type from any one node in the cluster:
lic
b
e
u
l
k
a # eclrt
abregister SUNW.HAStoragePlus
r
y
l
f
bdu trans
a
on- 9 – Installing and Registering Oracle Data
ang nTask
7.

ab

Configure the tnsnames.ora file (as user oracle):

Service

Perform the following steps as indicated to install and register the Oracle
data service:
1.

Install the data service agent on all storage nodes:
# cd sc32_location/Solaris_sparc/Product
# cd sun_cluster_agents/Solaris_10/Packages
# pkgadd -d . SUNWscor

2.

Register the SUNW.oracle_server and SUNW.oracle_listener
resource types from one cluster node:
# clrt register SUNW.oracle_server
# clrt register SUNW.oracle_listener

12-20

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

Exercise 2: Integrating Oracle 11g Into Sun Cluster 3.2 Software as a Failover

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Task 10 – Creating Resources and Resource Groups
for Oracle
Perform the following steps from the node which owns the Oracle storage
to create the resource group and resources necessary for the Oracle data
service:
1.

Create an empty resource group:
# clrg create -n node1,node2 ora-rg

2.

Add a LogicalHostname resource:
# clrslh create -g ora-rg ora-lh

3.

Add an HAStoragePlus resource.
For VxVM or SVM:

s
) ha

ab

y
m

# clrs create -g ora-rg -t HAStoragePlus c\om


e
-p FilesystemMountPoints=/oracle \ sm
d
i
ec t Gu
-p AffinityOn=true ora-stor
@
b den
u
k
For ZFS:
ya Stu
(
n
i -tthHAStoragePlus
# clrs create -g ora-rg
\
is
dora-stor
u
l
-p Zpools=orapool
e
a us
m
a
k e to
g
4. Putbthe
anresource
nsgroup online (required by the dependency
e
c
a
li in the next step):
b specification
e
u
l
k
b online -M ora-rg
a e#raclrg
y
l
f
bdu tran5.s Create an oracle_server resource by creating and running a script.
a
(the command is too long to type interactively in some shells):
ang non
a.

Write a script containing the clrs command:

# vi /var/tmp/setup-ora-res.sh
#!/bin/sh
/usr/cluster/bin/clrs create -g ora-rg \
-t oracle_server \
-p Resource_dependencies=ora-stor \
-p ORACLE_sid=MYORA -p ORACLE_home=/oracle/product/11.1.0/db_1 \
-p Alert_log_file=/oracle/diag/rdbms/myora/MYORA/trace/alert_MYORA.log \

-p Connect_string=sc_fm/sc_fm \
ora-server-res
b.

Run the script:

# sh /var/tmp/setup-ora-res.sh

Performing Supplemental Exercises for Sun Cluster 3.2 Software
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

12-21

a

Exercise 2: Integrating Oracle 11g Into Sun Cluster 3.2 Software as a Failover

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

6.

Create an oracle_listener resource:

# clrs create -g ora-rg -t oracle_listener \
-p ORACLE_home=/oracle/product/11.1.0/db_1 \
-p Listener_name=LISTENER -p Resource_dependencies=ora-stor \
ora-listener-res
7.

Verify the application:
# clrg status
# clrs status
# ps -ef | grep oracle

Task 11 – Verifying That Oracle Runs Properly in the
a
s
a
Cluster
y) h

ฺm
m
co software
Perform the following steps as indicated to verify that the ฺOracle

e
m
d
s
i
properly runs in the cluster:
ec t Gu
@
n running the
b nodedcurrently
1. Switch user to oracle user on the cluster
e
u
k
ora-rg resource group:
ya Stu
(
n
di e this
# su - oracle
u
l
a theucluster
s node currently running the oram
2. Query the database
from o
a
k
t
rg resource
nggroup:
se
a
n
b
e
a
$ sqlplus
SYS/cangetin@MYORA
as sysdba
lic
b
e
u
l
* from mytable;
k SQL>raselect
b
a
y
SQL>
quit
ul nsfe
d
b
a
tr3.a As user root (from any cluster node), switch the resource group to
g
n
n
another node:
a
no

ab

# clrg switch -n other-node ora-rg
4.

Query the database from the cluster node currently running the orarg resource group (as user oracle):
$ sqlplus SYS/cangetin@MYORA as sysdba
SQL> select * from mytable;
SQL> quit

5.

12-22

Verify that the database fails over properly under various failure
conditions:
a.

Complete failure of the node on which the resource group is
running (bring it to the OK prompt).

b.

Complete public network failure on the node on which the
resource group is running.

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

Exercise 3: Running Oracle 11g RAC in Sun Cluster 3.2 Software

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Exercise 3: Running Oracle 11g RAC in Sun Cluster 3.2
Software
In this exercise, you run Oracle 11g RAC in Sun Cluster 3.2 software.
In this exercise, you complete the following tasks:

Task 1 – Selecting the Nodes That Will Run Oracle RAC

Task 1.5 – Unconfigure any Nonstorage Node From the Cluster

Task 2 – Creating User and Group Accounts

Task 3A (If Using VxVM) – Installing RAC Framework Packages for
Oracle RAC With VxVM Cluster Volume Manager

s
) ha

y
m

om ฺ
c

csm Guide
e
t
@
n
b
Task 4 – Installing Oracle Distributed
Lock Manager
e
u
k tud
athe
y
Task 5 – Creating and Enabling
(
S Framework Resource Group
n thisRAC
i
d
lu sCreating
Task 6A (If using a
VxVM):
Shared Volume Manager
e
m
u
Framework Resource
Group
ka e to
g
n Using
Taska6B(If
nsSolaris Volume Manager) – Creating Shared Volume
b
e
c
a
i
Resource Group
lFramework
b Manager
e
u
l
k
a eTask
ab7A – Creating Raw Volumes (VxVM)
r
y
l
f
bdu trans Task 7B – Creating Raw Volumes (Solaris Volume Manager)
a
g onn
Task 8 – Configuring Oracle Virtual IPs
a
n
ab

Task 3B (If Using Solaris Volume Manager) – Installing RAC
Framework Packages for Oracle RAC With Solaris Volume Manager
Multi-Owner Disksets






Task 9 – Configuring the oracle User Environment

Task 10A – Creating the dbca_raw_config File (VxVM)

Task 10B – Creating the dbca_raw_config File (Solaris Volume
Manager)

Task 11 – Disabling Access Control on X Server of the Admin
Workstation

Task 12 – Installing Oracle CRS Software

Task 13 – Installing Oracle Database Software

Task 14 – Create Sun Cluster Resources to Control Oracle RAC
Through CRS

Task 15 – Verifying That Oracle RAC Works Properly in a Cluster

Performing Supplemental Exercises for Sun Cluster 3.2 Software
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

12-23

a

Exercise 3: Running Oracle 11g RAC in Sun Cluster 3.2 Software

Preparation

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010, Oracle and/or its affiliatesฺ

Before continuing with this exercise, read the background information in
this section.

Background
Oracle 11g RAC software in the Sun Cluster environment encompasses
several layers of software, as follows:

RAC Framework

s
) ha

This layer sits just above the Sun Cluster framework. It encompasses
the UNIX® distributed lock manager (udlm) and a RAC-specific
cluster membership monitor (cmm). In the Solaris 10 OS, you must
create a resource group rac-framework-rg to control this layer (in
the Solaris 9 OS, it is optional).

y
m

om ฺ
c

csm Guide
e
Shared Volume Manager framework
@ ent
b
u
k thetRAC
ud framework starting
afrom
This framework has been split off
y
(
S
in Sun Cluster 3.2 11/09 (Update
in 3).
d
hItisis required specifically and
t
u
l
only for shared volume
manager
support.
e
a us
m
a
kReadyeServices
to (CRS)
Oracle Cluster
g
s
an cenOracle’s
CRSaisbessentially
own implementation of a resource group
i
l
b
umanager.
leThat is, for Oracle RAC database instances and their
k
b
a
a
and related resources, CRS takes the place of the
l y sassociated
erClusterlisteners
f
u
d
Sun
resource
group manager.
b tran
a
g onOracle Database
n
a
n
b
a

The actual Oracle RAC database instances run on top of CRS. The
database software must be installed separately (it is a different
product) after CRS is installed and enabled. The database product
has hooks that recognize that it is being installed in a CRS
environment.

Sun Cluster control of RAC
The Sun Cluster 3.2 environment features new proxy resource types
that allow you to monitor and control Oracle RAC database
instances using standard Sun Cluster commands. The Sun Cluster
instances issue commands to CRS to achieve the underlying control
of the database.

12-24

Sun™ Cluster 3.2 Administration
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C

a

The performance detriment this might cause might eliminate the advantages of using RAC in the first place. Oracle and/or its affiliatesฺ Oracle RAC DB Sun Cluster RAC proxy resource lets you use Sun Cluster commands to control database by "proxying" through CRS Oracle CRS Shared VM Framework RAC Framework s ) ha y m ฺ m o c ฺ ฺ RAC Database Storage e m d s i ec t Gu @ In the Sun Cluster environment. you have n choices of where b the following e u d k to store your actual data for RACydatabases: a Stu ( n i Raw devices using the Cluster dVxVM his Volume Manager (CVM) t u l e a us feature m a k usingethe toSolaris Volume Manager multi-owner diskset g Raw devices ban licens feature a b Raw devices u le using Oracle’s Automatic Storage Managment (ASM) k b a a r l y sfeRaw u devices using no volume manager (assumes hardware RAID) d b tran a g onShared QFS file system on raw devices or Solaris Volume Manager n a n b multi-owner devices a Figure 12-1 Oracle RAC Software Layers ● ● ● ● ● ● On a supported NAS device Note – Use of global devices (using normal device groups) or a global file system is not supported. Sun Services. Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Performing Supplemental Exercises for Sun Cluster 3. Revision C 12-25 a . your cluster transport would now be used for both the application-specific RAC traffic and for the underlying device traffic. The rationale is that if you used global devices or a global file system.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. All Rights Reserved.2 Software The various RAC layers are illustrated in Figure 12-1. Inc.2 Software Copyright 2009 Sun Microsystems.

Revision C a . Task 1 – Selecting the Nodes That Will Run Oracle RAC s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u Note – All the cluster command examples d the rest of the RAC k shown anodes. The variations in the lab are indicated by tasks that have an A or B suffix. you can choose to use either VxVM or Solaris Volume Manager. tufor y ( S exercise assume you have two storage You can make the n s i i d h t than two storage nodes on appropriate substitutions if lyou more u have e a s which you choose to install RAC. When you have built your storage devices. All Rights Reserved. Sun Services. In the lab. you create all the volumes using the same pair of disks. there are very few variations in the tasks that actually install and test the Oracle software. ab 12-26 Sun™ Cluster 3. e this will be noted further in the sections n s a n where you b ce storage volumes. The nodes you select must all be attached to shared storage. Inc. a create lyour i b u le k b a a l y sfer u d ab -tran g an non Select the nodes on which the Oracle 11g RAC software will be installed and configured. You will always choose only one such task.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. This lab does not actually address careful planning of your raw device storage.2 Software In this lab.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. There m to u are some storage volumes that are a k created on a per-node g basis.

2 Software Copyright 2009 Sun Microsystems. Inc. Reboot the nonstorage node to noncluster mode. edit /etc/vfstab and comment out lines containing a global filesystem except the line for /global/. module. is very nasty and kernel-panicky if you still have nonstorage nodes in the cluster. Sun Services.5 – Unconfigure any Nonstorage Node From the Cluster VxVM’s CVM feature.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. even if you do not intend to use them for Oracle RAC. as h 3. in particular. On bthe en node. and are also attempting tuyou y ( the node lists for the apache in application d his like this: t u l e a -nus\ # clrg remove-node m a k e to name_of_non_storage_node:myzone sa-rg web-rg g n s a nonstorage 4.devices/node@# (you can leave that one). All Rights Reserved.2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Task 1. 2. type the following to remove itself from the c a i l b cluster:le u k b a e#raclnode y remove -n name_of_any_other_node_remaining l f u s d ab -tran g an non ab Performing Supplemental Exercises for Sun Cluster 3. Revision C 12-27 a . Perform the following steps on the indicated nodes if you have a nonstorage node: 1. On the nonstorage node. type the following to ) y m allow the nonstorage node to remove itself: ฺ om ฺ c # claccess allow -h name_of_non_storage_node ฺ # clrg remove-node -n name_of_non_storage_node csm Guide+ e t lab in this @ theenzones b If for some reason you already completed u d will need to clean up akthisSlab. On one of the nodes remaining in the cluster.

Sun Services.2 Software Task 2 – Creating User and Group Accounts Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Provide a password for the oracle user account: # pkgadd -d . SUNWscucm SUNWudlm SUNWudlmr SUNWcvm SUNWcvmr SUNWscor 2. Your instructor will tell you from where you can paste a CVM license.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. Verify that you now have a second license for the CVM_FULL feature: # vxlicrep 4. Create the dba group account: # groupadd -g 7777 dba 2. 1. then delete the oracle user and dba groups from all nodes before proceeding. # vxlicinst 3. Refresh the VxVM licenses that are in memory: # vxdctl license init 12-28 Sun™ Cluster 3. Oracle and/or its affiliatesฺ Perform the following steps on all selected cluster nodes: Note – If you have already completed the HA-Oracle lab. All Rights Reserved. Enter a license for the CVM feature of VxVM. Revision C a . Create the oracle user account: # useradd -g dba -u 7777 -d /oracle -m -s /bin/ksh oracle s ) ha y m ฺ om ฺ # passwd oracle c ฺ New Password: oracle sm uide c e Re-enter new Password: oracle G t @ n b ku tude a y n ( this SRAC Framework Task 3A (If Using VxVM)u–diInstalling e VxVM Cluster Volume al With s Packages for Oracle RAC m u ka e to g Manager ban licens a b the following u le Perform steps on all selected cluster nodes k b a a r y fe the appropriate packages from the software location: ul 1. nsInstall d b a tra g #n cd sc32_location/Solaris_sparc/Product n no ab#a cd sun_cluster_agents/Solaris_10/Packages 3.2 Administration Copyright 2009 Sun Microsystems. Inc.

Install the appropriate packages from the software location: # cd sc32_location/Solaris_sparc/Product # cd sun_cluster_agents/Solaris_10/Packages # pkgadd -d . The pkgadd command prompts you for the group that is to be DBA for the database. List the local metadbs: 2. Add metadbs on the root drive.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. 1. ORCLudlm 2. a b e a thelicORCLudlm package: 1.tar.2 Software Copyright 2009 Sun Microsystems. For Solaris Volume Manager installations. Revision C 12-29 a .Z /var/tmp u d ab -tran # cd /var/tmp g n non # zcat ORCLudlm. Inc. All Rights Reserved.2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. no reboot is required. Oracle and/or its affiliatesฺ Task 3B (If Using Solaris Volume Manager) – Installing RAC Framework Packages for Oracle RAC With Solaris Volume Manager Multi-Owner Disksets Perform the following steps on all selected cluster nodes.Z|tar xf a b a # pkgadd -d . SUNWscucm SUNWudlm SUNWudlmr SUNWscor s ) ha y m ฺ # metadb om ฺ c ฺ e 3. Sun Services. only if they s idexist: c domnotGyet u e # metadb -a -f -c 3 c#t#d#s7 @ ent b u ak Stud y ( inDistributed d Task 4 – Installing Oracle Lock Manager his t u l e a am to us k Perform thenfollowing g nssteps e on all selected cluster nodes as user root.tar. Respond using dba: Please enter the group which should be able to act as the DBA of the database (dba): [?] dba Performing Supplemental Exercises for Sun Cluster 3. b Install e u lORACLE_clusterware_location/udlm k b a # cd a l y sfe#r cp ORCLudlm.

Revision C a . you are in good shape. If your nodes are still alive (not kernel panic).node2 shvm-framework-rg a g on-t n a n -g shvm-framework-rg -t vucmm_framework vucmm-framework-res ab# clrs create # clrg online -M rac-framework-rg # clrs create -g shvm-framework-rg -t vucmm_cvm\ -p Resource_dependencies=vucmm-framework-res vucmm-cvm-res # clrg online -M shvm-framework-rg Following that there will be about 100 lines of console output as CVM initializes. you will see a message on the console: ฺco ฺ e m d s i u Unix DLM version(2) and SUN Unix DLM Library Version ec (1):compatible G t @ b den u k ya Stu ( n Task 6A (If using VxVM): Creating di e thisShared Volume u l a us Group Manager FrameworkmResource a k e to g n ens nodes.2 Administration Copyright 2009 Sun Microsystems.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. Oracle and/or its affiliatesฺ Task 5 – Creating and Enabling the RAC Framework Resource Group On any one of the selected nodes. Inc. All Rights Reserved.2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. 12-30 Sun™ Cluster 3. run the following commands as root: On any oneb ofathe selected a lic b e u l # clrt registerak vucmm_framework ab r y l e # clrt register vucmm_cvm f s du r-S n # clrgab create -n node1. run the following commands as root: # clrt register rac_framework # clrt register rac_udlm # clrg create -S -n node1. Sun Services.node2 rac-framework-rg # clrs create -g rac-framework-rg -t rac_framework rac-framework-res # clrs create -g rac-framework-rg -t rac_udlm \ -p Resource_dependencies=rac-framework-res rac-udlm-res s ) ha y m ฺ m If all goes well.

Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. use only those nodes on which you have selected to run RAC (the ones connected to the physical storage): # clrt register vucmm_framework # clrt register vucmm_svm # clrg create -S -n node1.node2 shvm-framework-rg as h # clrs create -g shvm-framework-rg -t vucmm_framework vucmm-framework-res ) y m ฺ om ฺ # clrs create -g shvm-framework-rg -t vucmm_svm \ c ฺ -p Resource_dependencies=vucmm-framework-res vucmm-svm-res csm Guide e @ ent b # clrg online -M shvm-framework-rg u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g an non ab Performing Supplemental Exercises for Sun Cluster 3. Oracle and/or its affiliatesฺ Task 6B(If Using Solaris Volume Manager) – Creating Shared Volume Manager Framework Resource Group On any one of the selected nodes.2 Software Copyright 2009 Sun Microsystems. All Rights Reserved. Revision C 12-31 a . run the following commands as root to create the RAC framework resource group. Sun Services.2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. As you create the group. Inc.

Make sure you do not use any disks already in use in existing device groups. This script is reproduced here. You can use the provided a g onn a create_rac_vxvm_volumes script. Inc.2 Software Task 7A – Creating Raw Volumes (VxVM) Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Create a licshared disk group using this disk: b e u l k # /etc/vx/bin/vxdisksetup c#t#d# a ab-s init ora-rac-dg -i r y l e # vxdg ora1=cAtAdA f bdu tr4.2 Administration Copyright 2009 Sun Microsystems. 1. All Rights Reserved. Sun Services. Select a single disk from shared storage for a new disk group. You d u l can always add another disk and mirror all volumes later at your leisure. just in case you are running any combination. if n ab # # # # vxdisk -o alldgs list zpool status metaset cldev list -v you want to type it in yourself rather than use the script: vxassist vxassist vxassist vxassist vxassist vxassist vxassist vxassist vxassist vxassist vxassist vxassist vxassist vxassist 12-32 -g -g -g -g -g -g -g -g -g -g -g -g -g -g ora-rac-dg ora-rac-dg ora-rac-dg ora-rac-dg ora-rac-dg ora-rac-dg ora-rac-dg ora-rac-dg ora-rac-dg ora-rac-dg ora-rac-dg ora-rac-dg ora-rac-dg ora-rac-dg make make make make make make make make make make make make make make raw_system 750m raw_spfile 100m raw_users 120m raw_temp 100m raw_undotbs1 312m raw_undotbs2 312m raw_sysaux 550m raw_control1 110m raw_control2 110m raw_redo11 120m raw_redo12 120m raw_redo21 120m raw_redo22 120m raw_ocr 256m Sun™ Cluster 3. Revision C a .Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. Oracle and/or its affiliatesฺ To create the raw volumes for the Oracle 11g RAC database. Determine which node is the CVM Master by running the following command on all nodes: # vxdctl -c mode 2. The following example checks against VxVM. perform the following steps from the single node that is the CVM Master. e a us m a going or pretend that youkwere to to. g e n ens baaCVM 3. iwe is unmirrored volumes. ZFS and Solaris Volume Manager disks. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( n willthcreate Note – Just to speed up the lab.ansCreate the volumes for the database.

Sun Services. ab To create the raw volumes for the Oracle 11g RAC database. You will probably get errors indicating that the end of the device does not align exactly on a 1kk (1MB) boundary.2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Step 6 is the only step that is run on all selected nodes. Oracle and/or its affiliatesฺ vxassist -g ora-rac-dg make raw_css_voting_disk 256m Note – You need to create 3 additional volumes for each additional node (more than 2) that you may be using.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. perform steps 1–5 from any one of the selected cluster nodes. All Rights Reserved. Zero out the configuration and voting devices. For example. ZFS. just in case you are running any combination. Make sure you do not use any disk already in use in existing device groups. This eliminates problems that might arise from data left over from a previous class: # dd if=/dev/zero of=/dev/vx/rdsk/ora-rac-dg/raw_ocr bs=1kk # dd if=/dev/zero of=/dev/vx/rdsk/ora-rac-dg/raw_css_voting_disk bs=1kk s ) ha y m ฺ om ฺ c ฺ csm Guide e @ withenthet newly created b 6. Note the device identifier (DID) number of your selected disk (referred to as d# in step 2). raw_redo13 (120m). Select a single disk from shared storage for a new disk set. Revision C 12-33 a .2 Software Copyright 2009 Sun Microsystems. and Solaris Volume Manager disks. and raw_redo23 (120m) 5. Inc. Change the owner and group associated u ak here): tud volumes (assumes sh or ksh syntax y ( S in this d # cd /dev/vx/rdsk/ora-rac-dg u l e s # for vol inm *a u a to > do g k e s > b vxedit set user=oracle group=dba $vol an c-genora-rac-dg a i l > done b u le k b a a l y sfer u d an7B – Creating Raw Volumes (Solaris Volume ab Task r t g an nonManager) Note – Do not be concerned with error messages that appear at the end of running the above commands. The following example checks against VxVM. # # # # vxdisk -o alldgs list zpool status metaset cldev list -v Performing Supplemental Exercises for Sun Cluster 3. for a third node create the volumes raw_undotbs3 (size 312m). 1.

Create the volumes for the database. # metaset -s ora-rac-ds -a -M -h node1 node2 # metaset -s ora-rac-ds -a /dev/did/rdsk/d# 3. You can always add another disk and mirror the parent volume later at your leisure. us m a k d100 to(750mb) g e # systemnvolume ba -slicora-rac-ds ens metainit d100 -p d10 750m a b e u l k # spfile a ab volume d101 (100mb) r y l e f bdu transmetainit -s ora-rac-ds d101 -p d10 100m a ang non # users volume d102 (120mb) ab metainit -s ora-rac-ds d102 -p d10 120m # temp volume d103 (100mb) metainit -s ora-rac-ds d103 -p d10 100m # undo volume #1 d104 (312mb) metainit -s ora-rac-ds d104 -p d10 312m # undo volume #2 d105 (312mb) metainit -s ora-rac-ds d105 -p d10 312m # sysaux volume d106 (550mb) metainit -s ora-rac-ds d106 -p d10 550m # control volume #1 d107 (100mb) metainit -s ora-rac-ds d107 -p d10 110m 12-34 Sun™ Cluster 3.2 Administration Copyright 2009 Sun Microsystems. Sun Services. then you could easily just add the second half of the mirror later at your leisure) s ) ha y m ฺ om ฺ # metainit -s ora-rac-ds d11 1 1 /dev/did/rdsk/d#s0 c ฺ # metainit -s ora-rac-ds d10 -m d11 csm Guide e t provided 4. we will create unmirrored volumes. Create a multi-owner diskset using this disk. u l e a rather than use the script. You can usenthe @ b e u k script create_rac_svm_volumes script. Revision C a . Oracle and/or its affiliatesฺ Note – Just to speed up the lab.aThis here.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. 2.2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. or pretend that you were going to. tudis reproduced y ( S including the comments indicating which volumes you are creating. in if tyou d hiswant to type them in yourself The commands are highlighted. Inc. Create a big mirror from which to partition all the necessary data volumes (still call it a mirror although it has only one submirror. All Rights Reserved.

Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. Inc. Oracle and/or its affiliatesฺ # control volume #2 d108 (100mb) metainit -s ora-rac-ds d108 -p d10 110m # redo volume #1. On all selected nodes. Sun Services. From either node. This eliminates problems that might arise from data left over from a previous class: # dd if=/dev/zero of=/dev/md/ora-rac-ds/rdsk/d113 bs=1kk # dd if=/dev/zero of=/dev/md/ora-rac-ds/rdsk/d114 bs=1kk Note – Do not be concerned with error messages that appear at the end of running the above commands.1 d109 (120m) metainit -s ora-rac-ds d109 -p d10 120m # redo volume #1. 6. zero out the configuration and voting devices. d116 (120m) for redo #1.3. for a third node create b lice a d115 b (312mb) e for the undo #3 volume . For example.2 d112(120m) metainit -s ora-rac-ds d112 -p d10 120m s ) ha ab y m ฺ om ฺ c ฺ # OCR Configuration volume d113 (256mb) sm uide metainit -s ora-rac-ds d113 -p d10 c256m e G t @ n b ku tude # CSS voting disk d114 (256mb) a y metainit -s ora-rac-ds -pSd10 256m n (d114 s i i d h lu se t a u am o k t Note – You need to create 3 additional volumes for each additional node g nse n a (more than 2) that you may be using. You will probably get errors indicating that the end of the device does not align exactly on a 1kk (1MB) boundary. Revision C 12-35 a . All Rights Reserved.2 Software Copyright 2009 Sun Microsystems.2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. and u l k b (120mb) ra for redo #2.2 d110 (120m) metainit -s ora-rac-ds d110 -p d10 120m # redo volume #2.3. ya d117 l e f u s bd tran a ang non 5. change the owner and group associated with the newly created volumes. # chown oracle:dba /dev/md/ora-rac-ds/dsk/* # chown oracle:dba /dev/md/ora-rac-ds/rdsk/* Performing Supplemental Exercises for Sun Cluster 3.1 d111 (120m) metainit -s ora-rac-ds d111 -p d10 120m # redo volume #2.

failing over both of them to a surviving node if one of the nodes crashes.z. Revision C a .w nodename1-vip u al use x.w+1 nodename2-vip m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g an non ab 12-36 Sun™ Cluster 3. then the configuration assistant in the next task automatically guesses the second IP when you type in the first. You will have one IP per node that you are using with Oracle RAC. they get a “connection refused” indication.z. Instead. Oracle and/or its affiliatesฺ Task 8 – Configuring Oracle Virtual IPs Configure virtual IPs for use by the Oracle RAC database. Perform the following task on all selected nodes as root (you can edit the hosts file on one node and copy it over or paste in your entries): s ) ha Edit the /etc/hosts file and create public network entries for new virtual IPs. If you use consecutive IP addresses.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. Oracle clients do not successfully contact the database using that IP. Oracle CRS controls these IPs. and have their client software set to automatically try the other IP.2 Administration Copyright 2009 Sun Microsystems.2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( # vi /etc/hosts in this d x. Make sure you do not conflict with any other physical or logical IPs that you or any other students have been using during the week. All Rights Reserved.y. When one of these IPs fails over. Sun Services.y. Inc.

/. $ echo + >/oracle/.0/crs TNS_ADMIN=$ORACLE_HOME/network/admin DBCA_RAW_CONFIG=$ORACLE_BASE/dbca_raw_config #SRVM_SHARED_CONFIG=/dev/md/ora-rac-ds/rdsk/d101 SRVM_SHARED_CONFIG=/dev/vx/rdsk/ora-rac-dg/raw_spfile DISPLAY=display-station-name-or-IP:display# if [ `/usr/sbin/clinfo -n` -eq 1 ]. .RAC provided) to include environment variables for Oracle 11g RAC: $ vi .1. Make sure your actual X-Windows display is set correctly on the line that begins with DISPLAY=. Edit . Enable rsh for the oracle user.0/db_1 CRS_HOME=$ORACLE_BASE/product/11.2 Software Task 9 – Configuring the oracle User Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.1. Revision C 12-37 a .profile file and verify the environment.profile ORACLE_BASE=/oracle ORACLE_HOME=$ORACLE_BASE/product/11. then ORACLE_SID=sun1 fi if [ `/usr/sbin/clinfo -n` = 2 ]. 5.profile (or start by copying the profile. You might need to make a modification in the file. Inc.oracle 2. Read in the contents of your new . then ORACLE_SID=sun2 fi PATH=/usr/ccs/bin:$ORACLE_HOME/bin:$CRS_HOME/bin:/usr/bin:/usr/sbin export ORACLE_BASE ORACLE_HOME TNS_ADMIN DBCA_RAW_CONFIG CRS_HOME export SRVM_SHARED_CONFIG ORACLE_SID PATH DISPLAY s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran3.2 Software Copyright 2009 Sun Microsystems. All Rights Reserved. 4. Make sure the line n non a b beginning SRVM_SHARED_CONFIG lists the correct option and that the a other choice is deleted or commented out. Sun Services. depending on g your choice of VxVM or Solaris Volume Manager.rhosts Performing Supplemental Exercises for Sun Cluster 3. Oracle and/or its affiliatesฺ Perform the following steps on all selected cluster nodes: 1. Switch to the oracle user: # su . $ .profile $ env 6.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3.

Sun Services.2 Software Task 10A – Creating the dbca_raw_config File (VxVM) Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Perform the following task: The dbca_raw_config file defines the locations of the raw devices of the Oracle 11g RAC database. You must either copy the existing dbca_raw_config. redo1_3. ab -trnode. Revision C a . $ vi /oracle/dbca_raw_config system=/dev/vx/rdsk/ora-rac-dg/raw_system spfile=/dev/vx/rdsk/ora-rac-dg/raw_spfile users=/dev/vx/rdsk/ora-rac-dg/raw_users temp=/dev/vx/rdsk/ora-rac-dg/raw_temp undotbs1=/dev/vx/rdsk/ora-rac-dg/raw_undotbs1 undotbs2=/dev/vx/rdsk/ora-rac-dg/raw_undotbs2 sysaux=/dev/vx/rdsk/ora-rac-dg/raw_sysaux control1=/dev/vx/rdsk/ora-rac-dg/raw_control1 control2=/dev/vx/rdsk/ora-rac-dg/raw_control2 redo1_1=/dev/vx/rdsk/ora-rac-dg/raw_redo11 redo1_2=/dev/vx/rdsk/ora-rac-dg/raw_redo12 redo2_1=/dev/vx/rdsk/ora-rac-dg/raw_redo21 redo2_2=/dev/vx/rdsk/ora-rac-dg/raw_redo22 s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a bAdd entries u le for undotbs3. g n non a b a 12-38 Sun™ Cluster 3.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. Inc. All Rights Reserved. and redo2_3 using the Note – k b a a l yappropriate er additional volumes names you created if you have a third f u s d an Extrapolate appropriately for each additional node.vxvm file that your instructor has prepared for you or manually create the file as user oracle on all selected cluster nodes.2 Administration Copyright 2009 Sun Microsystems.

Sun Services.2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.svm file that your instructor has prepared for you or manually create the file as user oracle on all selected cluster nodes. Inc. run the following command on the admin workstation or display station: (# or $) /usr/openwin/bin/xhost + Performing Supplemental Exercises for Sun Cluster 3. Revision C 12-39 a . redo1_3. You must either copy the existing dbca_raw_config. Oracle and/or its affiliatesฺ Task 10B – Creating the dbca_raw_config File (Solaris Volume Manager) Perform the following task: The dbca_raw_config file defines the locations of the raw devices of the Oracle 11g RAC database. and redo2_3 using the f u s d ab -tranappropriate additional volumes names you created if you have a third g n non node. a b a Task 11 – Disabling Access Control on X Server of the Admin Workstation Perform the following task: To allow client GUIs to be displayed. Extrapolate appropriately for each additional node.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. $ vi /oracle/dbca_raw_config system=/dev/md/ora-rac-ds/rdsk/d100 spfile=/dev/md/ora-rac-ds/rdsk/d101 users=/dev/md/ora-rac-ds/rdsk/d102 temp=/dev/md/ora-rac-ds/rdsk/d103 undotbs1=/dev/md/ora-rac-ds/rdsk/d104 undotbs2=/dev/md/ora-rac-ds/rdsk/d105 sysaux=/dev/md/ora-rac-ds/rdsk/d106 control1=/dev/md/ora-rac-ds/rdsk/d107 control2=/dev/md/ora-rac-ds/rdsk/d108 redo1_1=/dev/md/ora-rac-ds/rdsk/d109 redo1_2=/dev/md/ora-rac-ds/rdsk/d110 redo2_1=/dev/md/ora-rac-ds/rdsk/d111 redo2_2=/dev/md/ora-rac-ds/rdsk/d112 s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y Note e–r Add entries for undotbs3.2 Software Copyright 2009 Sun Microsystems. All Rights Reserved.

Exercise 3: Running Oracle 11g RAC in Sun Cluster 3.1. Click Next.0/crs a 3. 12-40 Sun™ Cluster 3. n ens a b a lic the path to /oracle/oraInventory. Click Next. Verify the Specify Inventory directory Change b e u l and credentialsyak ab group name (dba) and click Next. The installer automatically copies over the binaries to the other nodes using rsh at the end of the installation. Respond to the dialog boxes by using Table 12-2. Change directory to the Oracle 11g CRS 11.2 Administration Copyright 2009 Sun Microsystems. Run the runInstaller installation program by: $ . r l e f s du Details n Specify Change the Path to: bHome a a r t g n non a b /oracle/product/11. Revision C a . and if you get a pop-up warning window. Inc.1. Sun Services. din his t u l e a us m a Click OK get the error about the previous k ewhen to you(there g inventory location obviously was none). All Rights Reserved. Perform the following steps on the first selected cluster node as the oracle user: 1. Product Specific Prerequisite Checks Most likely. If you happen to get a warning.0./runInstaller s ) ha y m ฺ om ฺ c ฺ Table 12-2 Oracle CRS Installation Dialog Actions csm Guide e @ ent b u Dialog Action ak Stud y ( Welcome Click Next.6 software location: $ cd ORACLE11gR1-CRS-software-location 2. these checks will all succeed.2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. click Next. Oracle and/or its affiliatesฺ Task 12 – Installing Oracle CRS Software Install the Oracle CRS software from the first selected cluster node. click Yes to proceed.

ng To se anitadapter a n highlight and click Then choose the appropriate b e cradio button and clickEdit. d b tran a ang non Mark only the clprivnet0 interface as private. Click Next.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. For each node listed. including actual private network adapters. Sun Services. as Do Not Use. VxVM: Enter /dev/vx/rdsk/ora-rac-dg/raw_ocr SVM: Enter /dev/md/ora-rac-ds/rdsk/d113 Click Next. Inc. verify the private host name (clusternode1-priv. All Rights Reserved.2 Software Copyright 2009 Sun Microsystems. Specify Oracle Cluster Registry (OCR) Location Choose the External Redundancy radio button. etc). verify that the Virtual Host Name column contains nodename-vip similar to what you entered in the /etc/hosts file. a i l b OK u le k b a a l y sfer u Mark your actual public adapters as public. ab Mark all other adapters. For each node listed. s ) ha y m ฺ om ฺ c ฺ e be If you get some error concerning s am node that idcannot c u e G clustered (null). n i th section! Specify Network Interface Be very careful this lud with e a Usage am to us k mark in the instructions in this box. These should have been determined automatically. Revision C 12-41 a .2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.rhosts file. You must have one even on u ude k a t the node on which (y youisareSrunning the installer. Click Next. it is probably because you do not have t @ n b an /oracle/. Performing Supplemental Exercises for Sun Cluster 3. Oracle and/or its affiliatesฺ Table 12-2 Oracle CRS Installation Dialog Actions (Continued) Dialog Action Specify Cluster Configuration Enter the name of the cluster (this is actually unimportant).

a tra g n n a no On all selected nodes.. you may have to wait the whole 600 k b a a l y sfer seconds for the verification to complete (have a snack u d n b courtesy of your training center). Sun Services. one at a time. NO KEYS WERE Written. the messages: EXISTING Configuration detected. 12-42 Sun™ Cluster 3. Click OK when root. End of Installation Click Exit and confirm. Configuration Assistants These should all succeed and pass you to the final screen. Execute Configuration Scripts IMPORTANT: You must wait for the scripts below to complete on one node before starting another.1.. starting with the node on which you are running the installer. VxVM: Enter /dev/vx/rdsk/ora-rac-dg/raw_css_voting_disk SVM: Enter /dev/md/ora-rac-ds/rdsk/d114 Click Next. ban inlic/etc/inittab ens a b u le On the first node.sh finished on the last node. As the root. various CRS daemon resources will be configured and started on all nodes.sh ak Stud y /oracle/product/11.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3.. Summary Verify.0/crs/root... open a terminal window as user root and run the scripts: ab On all but the first node. Oracle and/or its affiliatesฺ Table 12-2 Oracle CRS Installation Dialog Actions (Continued) Dialog Action Specify Voting Disk Location Choose the External Redundancy radio button.2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010..2 Administration Copyright 2009 Sun Microsystems. All Rights Reserved.sh script runs on the last node.sh ( in this d u e the voting device and al scriptusformats The second m ka thee CRS enables to daemons on each node. Inc. Entries are put g so that the daemons run at boot time. and click Install. are correct and expected. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u /oracle/oraInventory/orainstRoot. Revision C a .

and if you get a pop-up warning window. When you see the warning about the Oracle ab -tran g Base.0/db_1 k b a a l y sfer u d Click Next. Inc. u am thetoCustom k Select Installation Type Select radio button. This is in a different directory than the CRS software location: $ cd ORACLE11gR1-db-location $ . g nse n a e especially that the destination path is Install Location ab licVerify.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3.ฺco ฺ e m d s i ec t Gu Table 12-3 Install Oracle Database Software Dialog Actions@ b den u k ya Stu Dialog Action ( n i is lud se th Welcome Click a Next. Put check marks next to all of your selected cluster nodes. Click Next. as the user oracle. n n o a n ab Specify Hardware Cluster Installation Mode Verify that the Cluster Installation radio button is selected. click Yes to proceed. and click Next. 1. Sun Services. Revision C 12-43 a ./runInstaller s ) ha y m ฺ m 3. All Rights Reserved. On all selected nodes. Respond to the dialog boxes by using Table 12-3. Click Next. Product Specific Prerequisite Checks You may get a warning about swap space. run the the database installer. The installer automatically copies over the binaries to the other nodes using rsh at the end of the installation.2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Software Copyright 2009 Sun Microsystems.1. click Yes. become root and change the /oracle permissions back: # chown -R oracle:dba /oracle 2. Oracle and/or its affiliatesฺ Task 13 – Installing Oracle Database Software Install the Oracle database software. b u le /oracle/product/11. On the node from which you installed CRS. Performing Supplemental Exercises for Sun Cluster 3.

d k Assistant Welcome box is not selected. Done Click Finish. Revision C a . and click Next. and click Next. e c a li b e u l k TCP/IP Protocola that the Use the Standard Port Number of 1521 ab Verify r y l e Configuration radio button is selected.0 Click Next. 12-44 Sun™ Cluster 3. s ) ha y m ฺ Create Database Verify that Create a Database is selected. it takes a few seconds to n ab move on) Listener Configuration Done Click Next. and click Next. Verify the Listener di name hisisLISTENER. and click Next. and click Next. and g Select Protocols Verify ns ban click Next.2 Administration Copyright 2009 Sun Microsystems.2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Table 12-3 Install Oracle Database Software Dialog Actions (Continued) Dialog Action Available Product Components Deselect the following components (to speed up the installation): ● Enterprise Edition Options ● Oracle Programmer 11. I Do Not Want to Configure Additional Naming Methods radio button is selected. Privileged Operating System Groups Verify that dba is listed in all entries. Inc.6. c ฺ ฺ e m Summary Verify.1. and click Next. Naming Methods Configuration Verify that the No. and click Install d s i ec t Gu @ Oracle Net Configuration Verify that the Perform Typical Configuration check n e ubNext. DBCA Step 2: Database Identification Type sun in the Global Database Name text field (notice that your keystrokes are echoed in the SID Prefix text field). and click om Next. DBCA Step 1: Database Templates Select the General Purpose radio button. andaclick tu y ( S n Listener Configuration.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. and click g n n o a Next (be patient with this one. Sun Services. All Rights Reserved. f u ns d b aListeners-tra More Verify that the No radio button is selected. t u l e a us Listener Name m a k thateTCP to is among the Selected Protocols.0.

Sun Services. DBCA Step 7: Database Uncheck Sample Schemas. s ) ha y m ฺ om ฺ c ฺ DBCA Step 6: Recovery Leave the boxes unchecked. no aba Click Next. All Rights Reserved. DBCA Step 5: Storage Options Select the Raw Devices radio button. Click Next. Oracle and/or its affiliatesฺ Table 12-3 Install Oracle Database Software Dialog Actions (Continued) Dialog Action DBCA Step 3: Management Options Uncheck “Configure Enterprise Manager” and click Next. and click ide csm Next. Inc. DBCA Step 8: Initialization Keep a the s m u Parameters ka e to g Use the slider to choose the lowest possible Memory ban licSize ens a b u le k b a a Click Next. DBCA Step 12: Creation Options Verify that the Create Database check box is selected. Summary Verify and click OK. These locations are determined by the contents of the /oracle/dbca_raw_config file that you prepared in a previous task. DBCA Step 11: Database Storage Verify that the database storage locations are correct by clicking leaves in the file tree in the left pane and examining the values shown in the right pane.2 Software Copyright 2009 Sun Microsystems. Performing Supplemental Exercises for Sun Cluster 3.. and click Finish.” g n n Settings and click Next.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. and click Next. Verify that the check box for Raw Devices Mapping File is selected. u Configuration e G t @ n b e Next. DBCA Step 4: Database Credentials Select the Use the Same Password for All Accounts radio button. Enter cangetin as the password. Revision C 12-45 a . l y sfer u d n ab Step-t9:raSecurity DBCA Verify the radio button is at “Keep the enhanced.. and that the value is /oracle/dbca_raw_config.2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. DBA Step 10 Automatic Maintenance Tasks Uncheck the checkbox (we aren’t staying around for maintenance) and click Next. dclick ku and u a t y Content n ( this S i d luTypical eradio button selected..

2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Create a group to hold the resource to monitor the shared storage: # clrg create -S -n node1. nsRegister d b a tra control: g n n a no ab# clrt register crs_framework # clrt register ScalDeviceGroup # clrt register scalable_rac_server_proxy 2. Revision C a . Inc.2 Administration Copyright 2009 Sun Microsystems.node2 \ -p RG_affinities=++rac-framework-rg \ rac-storage-rg 12-46 Sun™ Cluster 3. All Rights Reserved.0/db_1/root. open a terminal window as root.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. Execute Configuration Scripts On each of your selected nodes. Wait a while (anywhere from a few seconds to a few minutes: be patient) and you will get a pop-up window informing you that Oracle is starting the RAC instances.sh s Accept the default path name for the local bin directory. Oracle and/or its affiliatesฺ Table 12-3 Install Oracle Database Software Dialog Actions (Continued) Dialog Action Database Configuration Assistant Click Exit. Create a CRS framework resource (the purpose of this resource is to try to cleanly shut down CRS if you are evacuating the node using cluster commands): # clrs create -g rac-framework-rg -t crs_framework \ -p Resource_dependencies=rac-framework-res \ crs-framework-res 3. Sun Services. ha ) y m ฺ Click OK in the script prompt window. and run the script: /oracle/product/11. csm Guide e @ ent b u ak Stud to Control y ( Task 14 – Create Sun Cluster Resources in this d u Oracle RAC Through CRS al use m ka e to g s on only one of your selected nodes to create Perform the following an censteps b a Sun Cluster resources li that monitor your RAC storage. and that allow you bSun Cluster e u l to use k b to control your RAC instances through CRS: a a r y fe the types required for RAC storage and RAC instance ul 1.1. om ฺ c ฺ End of Installation Click Exit and confirm.

Create the resource to monitor the shared storage. Bring the resource group that monitors the shared storage online. will d be warned about the u a t y strong affinities. do this one and skip the next one: # clrs create -g rac-storage-rg -t ScalDeviceGroup \ -p DiskGroupName=ora-rac-dg \ -p Resource_dependencies=rac-cvm-res{local_node} \ rac-storage-res If you are using Solaris Volume Manager. Oracle and/or its affiliatesฺ If you are using VxVM. Everyone should continue here. but the command S succeed. Inc. Revision C 12-47 a . Create a group and a resource to allow@ you to run t cluster commands n b You e u that control the database throughkCRS. Do one of the following commands: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.++rac-storage-rg \ a o k t rac-control-rg ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n a no 5.2 Software Copyright 2009 Sun Microsystems.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. All Rights Reserved.node2 a \ lu e s m u -p RG_affinities=++rac-framework-rg. Sun Services.2 Software 4. perform this one: # clrs create -g rac-storage-rg -t ScalDeviceGroup \ -p DiskGroupName=ora-rac-ds \ -p Resource_dependencies=rac-svm-res{local_node} \ rac-storage-res s ) ha y m ฺ om ฺ c ฺ # clrg online -M rac-storage-rg csm Guide e 6. n ( thisshould i d # clrg create -S -n node1. ab Performing Supplemental Exercises for Sun Cluster 3.

Revision C a . You can use clinfo -n to verify the node id on each node. Inc. All Rights Reserved.1. Oracle and/or its affiliatesฺ Exercise 3: Running Oracle 11g RAC in Sun Cluster 3.0/crs \ -p Resource_dependencies_offline_restart=rac-storage-res{local_node} \ -p Resource_dependencies=rac-framework-res \ rac-control-res y m ฺ om ฺ c ฺ csm Guide e # sh /var/tmp/cr_rac_control @ ent b u ak Stud y ( # clrg online -M rac-control-rg in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha 12-48 Sun™ Cluster 3.0/db_1 \ -p Crs_home=/oracle/product/11.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.1.2 Software Note – For the following command. make sure you understand which node has node id 1.2 Administration Copyright 2009 Sun Microsystems. Sun Services. # vi /var/tmp/cr_rac_control clrs create -g rac-control-rg -t scalable_rac_server_proxy \ -p DB_NAME=sun \ -p ORACLE_SID{name_of_node1}=sun1 \ -p ORACLE_SID{name_of_node2}=sun2 \ -p ORACLE_home=/oracle/product/11. so that you match correctly with the names of the database sub-instances. and so forth. which has node id 2.

query the other database sub-instance and lother b e u l k verify b that the data is there: a a r y ul nsfe$ sqlplus SYS@sun2 as sysdba d b Enter password: cangetin a tra g n n a no 2. a Stu y14). verify as the oracle user that the instance is unavailable: $ crs_stat -t $ sqlplus SYS@sun2 as sysdba Enter password: cangetin Performing Supplemental Exercises for Sun Cluster 3. Switch to the oracle user (all selected cluster nodes): # su . 14). Inc.2 Software Copyright 2009 Sun Microsystems. From a ic node. Oracle and/or its affiliatesฺ Task 15 – Verifying That Oracle RAC Works Properly in a Cluster Run the following commands as indicated to verify that the Oracle software properly runs in the cluster: Note – So that the flow of the following task makes sense. # clrs disable -n node2 rac-control-res 5. Revision C 12-49 a . make sure you understand which node has node id 1. On the node for which you shutdown Oracle. take advantage of the Sun Cluster control resource to use Sun Cluster commands to shut down the database on one of the nodes (the resource accomplishes the actual control through CRS). and so forth. connect to database sub-instance sun1 and create a table. n di e this u SQL> commit. ( SQL> insert into mytable values ('theo'. On either node. l a us SQL> select * from mytable.oracle s ) ha y m ฺ $ sqlplus SYS@sun1 as sysdba m o c ฺ ฺ Enter password: cangetin e m d s i ec t Gu @ SQL> create table mytable (Name VARCHAR(20). SQL> quit 4. e u d k SQL> insert into mytable values ('vincent'. Sun Services. am k e to SQL> quit g n ens bathe 3. ab On node 1. which has node id 2. 1. All Rights Reserved. SQL> select * from mytable. age n b NUMBER(4)).Exercise 3: Running Oracle 11g RAC in Sun Cluster 3.2 Software Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

Oracle and/or its affiliatesฺ # clrs enable -n node2 rac-control-res 7.node1. 12. After the node boots. 2005.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. If you choose the latter. All Rights Reserved. On either node. the node will panic and reboot.0. Run the following as user oracle: s ) ha y m ฺ om ฺ c ฺ m uinode2 ora.1. S This virtual IP fails over failover listener controlled by Oracle n s i d h merely so a client quickly gets a TCP t disconnect without having to luClient e a s wait for a long time-out.node2. reenable the instance through the Sun Cluster resource: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. you should be able to repeat step 5. the CRS-controlled virtual IP for the crashed node migrate to the surviving node.0 .6. Boot the node that you had halted. Cause a crash of node 1: # <Control-]> telnet> send break 9. It might take a few attempts before the database is initialized and you can successfully access your data. Sun Services. and the database instance on the surviving node: $ crs_stat -t $ /usr/cluster/bin/clrs status It can take several minutes for the full recovery. monitor the automatic recovery of the virtual IP.Production on Tue Mar 25 k b a a 10:56:18 2009 y fer l u s bd tran Copyright (c) 1982. On the surviving node. Keep repeating the steps. All rights reserved. n s a n $ sqlplus as sysdba ce ab SYS@sun1 i l b uSQL*Plus: le Release 11.2 Administration Copyright 2009 Sun Microsystems. Inc.vip application ONLINE sONLINE de ora. a ang non $ crs_stat -t|grep vip ab Enter password: ERROR: ORA-12541: TNS:no listener Enter user-name: ^D 11.vip application ONLINEec ONLINE node2 G t @ n b u that 10. 8.kverify dethere is actually no u a t y ( iCRS. the listener. by typing boot or go at the OK prompt in the console. 12-50 Sun™ Cluster 3. you should see (after 45 seconds or so). Revision C a . On that (affected node).2 Software 6. software then has a client-side option m to u a k to fail over g to the other e instance. While this virtual IP has failed over. ORACLE.

SQL> quit s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Performing Supplemental Exercises for Sun Cluster 3.2 Software 13. Verify the proper operation of the Oracle database by contacting the various sub-instances as the user oracle on the various nodes: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. Revision C 12-51 a . All Rights Reserved.Exercise 3: Running Oracle 11g RAC in Sun Cluster 3. Inc.2 Software Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ $ sqlplus SYS@sun1 as sysdba Enter password: cangetin SQL> select * from mytable.

2 Administration Copyright 2009 Sun Microsystems. All Rights Reserved. ● Experiences ● Interpretations ● Conclusions ● Applications s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a 12-52 Sun™ Cluster 3. Inc. or discoveries you had during the lab exercises. Oracle and/or its affiliatesฺ Exercise Summary ? ! Discussion – Take a few minutes to discuss what experiences. Revision C a . Sun Services.Exercise Summary Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. issues.

Sun Services.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a A-1 Copyright 2009 Sun Microsystems. Revision C a . All Rights Reserved. Inc. Oracle and/or its affiliatesฺ Appendix A Terminal Concentrator This appendix describes the configuration of the Sun™ Terminal Concentrator (TC) as a remote connection mechanism to the serial consoles of the nodes in the Sun Cluster software environment.

Viewing the Terminal Concentrator Viewing the Terminal Concentrator Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.52. it must not send an abort signal to the attached host systems when it is powered on. Sun Services. Inc. Figure A-1 shows that the TC is a self-contained unit with its own operating system.enet ku tude a y Serial Ports n7 ( 8this S i 5 6 1 4 d 2 3 lu se a Serial port A am to u k ng ense Setup port a b a Node Node 2 lic1 b e u l k rab a y ul nsfe d b a traSetup Device g n n no aba Memory Network Interface Figure A-1 Terminal Concentrator Functional Diagram Note – If the programmable read-only memory (PROM) operating system is older than version 52. Note – If any other TC is substituted. you must upgrade it. Oracle and/or its affiliatesฺ The Sun Terminal Concentrator (Annex NTS) has its own internal operating system and resident administration programs. Revision C a . All Rights Reserved.2 Administration Copyright 2009 Sun Microsystems. A-2 Sun™ Cluster 3. s ) ha y m ฺ Self -Load Power-On om ฺ c ฺ PROM sm uide c e Operating System G t @ n b oper.

It is used primarily to set u up the IP e address and load d k u a sequence of the TC. Placing the operating system on an external server can decrease the reliability of the terminal server. boot path. It loads a resident PROM-based operating system into the TC memory. f u ns d b a tra Use the following resident programs to specify this information: g n n a addr no ● ● seq ● image ● admin Terminal Concentrator Copyright 2009 Sun Microsystems. di e this u l a us m a k e toSetup Programs g Terminal Concentrator ban licens a bmust configure u le k You the TC nonvolatile random access memory (NVRAM) b a a r y l e with the appropriate IP address. s ) ha ab y m ฺ Setup Port om ฺ c ฺ e smis used idonly cthat u Serial port 1 on the TC is a special purpose port during e G t @ n b initial setup. It runs a PROM-based self-test and displays error codes. it performs the following steps: 1. Inc.Viewing the Terminal Concentrator Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Operating System Load You can set up the TC to load its operating system either internally from the resident PROM or externally from a server. In the cluster application. Revision C A-3 a . 2. When power is first applied to the TC. it is always set to load internally. Sun Services. You can access port 1 from either a tip connection or t y ( S n from a locally connected terminal. and serial port information. All Rights Reserved.

Oracle and/or its affiliatesฺ The TC must be configured for proper operation. You can also connect an American Standard Code for Information Interchange (ASCII) terminal to the setup port. A-4 Sun™ Cluster 3. and image commands to complete the configuration. press the TC Test button shown in Figure A-3 until u To nenable s d b a TC power indicator begins to blink rapidly. Figure A-2 shows a tip hardwire connection from the administrative console. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u e1 al touPort s m Figure A-2 Setup Connection ka e to g ban licens a bSetupleMode Enabling u k a erab y l f Setup mode. Sun Services. You can use the default values for many of the prompts. a monitor:: prompt should appear on the setup device. they can be confusing and it is easy to make a mistake. Inc.2 Administration Copyright 2009 Sun Microsystems. Although the TC setup menus seem simple. then release the Test a trthe g n n no button and press it again briefly. Revision C a .Setting Up the Terminal Concentrator Setting Up the Terminal Concentrator Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. seq. Use the addr. you must connect to the TC setup port. Connecting to Port 1 To perform basic TC setup. aba 1 2 3 4 5 6 7 8 STATUS POWER UNIT NET Power indicator Figure A-3 ATTN LOAD ACTIVE 1 2 3 4 5 6 7 8 Test button Terminal Concentrator Test Button After you have enabled Setup mode. All Rights Reserved.

182.22.0.182. but you must still define these parameters during the initial setup sequence.150.98]:: 129.255.150. separated by commas or n ab spaces. Enter them in the g onn a order they should be tried.98]:: 129.9. All Rights Reserved.0]:: Enter Preferred load host Internet address [192.9.52.Setting Up the Terminal Concentrator Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.255. monitor:: addr Enter Internet address [192.22. Terminal Concentrator Copyright 2009 Sun Microsystems. Possible interfaces are: Ethernet: net SELF: self Enter interface sequence [self]:: The self response configures the TC to load its operating system internally from the PROM when you turn on the power.9. Usually this is set correctly when your cluster arrives.enet. Sun Services.0]:: 129. Inc. Revision C A-5 a .22.150.100 Enter Broadcast address [0. Enabling the self-load feature negates other setup parameters that refer to an external load host and dump host.98]:: 129. Oracle and/or its affiliatesฺ Setting the Terminal Concentrator IP Address The following example shows how to use the addr program to set the IP address of the TC.255 Enter Preferred dump address [192. The PROM image is currently called oper.100 Enter Subnet mask [255.0.150.100 Select type of IP packet encapsulation (ieee802/ethernet) [<ethernet>]:: Type of IP packet encapsulation: <ethernet> s ) ha y m ฺ om ฺ c ฺ csm Guide e Load Broadcast Y/N [Y]:: y ub@ ent ak Stud y ( in this d u l Setting the TerminalaConcentrator Load Source e s m u ka e to g The following an example ns shows how to use the seq program to specify the b e c a type li mechanism to be used: b of loading e u l k a emonitor:: ab seq r y l f u s a list of 1 to 4 interfaces to attempt to use for bd tran Enter a downloading code or upline dumping.182. but you should always verify that it is correct.182.

k rab a y ul nsfe d b a tra g n n o the Serial Port Variables nSetting aba Enter Image name ["oper. This includes the type and mode port settings. Port 1 requires different type and mode settings.enet"]:: Enter TFTP Load Directory ["9. this introduces an additional layer of complexity that is prone to failure.Setting Up the Terminal Concentrator Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems.7/"]:: Enter TFTP Dump path/filename ["dump. Specifying the Operating System Image Even though the self-load mode of operation negates the use of an external load and dump device. Sun Services. You should verify the port settings before installing the cluster host software.2.150. Oracle and/or its affiliatesฺ Note – Although you can load the TC’s operating system from an external server.. Inc.. annex: su A-6 Sun™ Cluster 3.52. If you nspress Control-C b n you e make a mistake. Escape character is '^]'.129. All Rights Reserved. Inc.182. The following is an example of the entire procedure: admin-ws# telnet terminal_concentrator_name Trying terminal concentrator IP address . you should still verify the operating system image name as shown by the following: monitor:: image s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud monitor:: y ( in this d u al use m Note – Do not define kaa dump toor load address that is on another network g e because you a receive additional questions about a gateway address. can to abort the setup and start c a li b e u l again. Revision C a . Connected to terminal concentrator IP address.100"]:: The TC port settings must be correct for proper cluster operation. Rotaries Defined: cli Enter Annex port name or number: cli Annex Command Line Interpreter * Copyright 1991 Xylogics.

0. Revision C A-7 a . csm Guide e @ ent b u ak Stud y ( in this Setting the Port Password d u al use m a krecommended to security feature is to set per port An optional and g e ns an extra password challenge as you access a passwords. . Connected to terminal concentrator IP address.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. use c ฺ public network access. Sun Services. provides banThese e c a li mode remotely through the telnet command. g n n a no If you ever forget a port password but know the TC root password. 8 ports Terminal Concentrator Copyright 2009 Sun Microsystems.1. You must u nYou s d b a tra set the enable_security parameter to Y to enable all the passwords.0. . you can just reset the passwords to whatever you want.1. admin-ws# telnet terminal_concentrator_name Trying terminal concentrator IP address . Escape character is '^]'. Inc. Inc. Rotaries Defined: cli Enter Annex port name or number: cli Annex Command Line Interpreter * Copyright 1991 Xylogics. annex: su Password: type the password annex# admin Annex administration MICRO-XL-UX R7. 8 ports admin: show port=1 type mode Port 1: type: hardwired mode: cli admin:set port=1 type hardwired mode cli admin:set port=2-8 type dial_in mode slave admin:set port=1-8 imask_7bits Y admin: quit annex# boot bootfile: <CR> warning: <CR> s ) ha ab y m ฺ Note – Do not perform this procedure through the special m osetupฺport. Oracle and/or its affiliatesฺ Setting Up the Terminal Concentrator Password: type the password annex# admin Annex administration MICRO-XL-UX R7. b port line slave serial u k a erab y l f can set different (or the same) port passwords on each port. All Rights Reserved.

Inc.2 Administration Copyright 2009 Sun Microsystems.Setting Up the Terminal Concentrator Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ admin: admin: admin: admin: admin: set port=2 port_password homer set port=3 port_password marge reset 2-3 set annex enable_security Y reset annex security s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a A-8 Sun™ Cluster 3. All Rights Reserved. Sun Services. Revision C a .

2 129.1.23 129. Sun Services.17 Terminal Concentrator Copyright 2009 Sun Microsystems.1.50.12 y m ฺ om ฺ c Network ฺ sm uide c129.annex y n ( this S i d %gateway lu129.annex configuration file. If the TC routing table overflows.23 e metric 1 active a net default gateway s m u ka e to g ban licens a b Nodele1 u Node 2 Node 3 Node 4 k b a a r y ul nsfe d b a tra Figure A-4 Terminal Concentrator Routing g n n no aba 129. Administrative Console Router hme0 hme1 s ) ha 129. Oracle and/or its affiliatesฺ Setting a Terminal Concentrator Default Route If you access a TC from a host that is on a different network.50. Figure A-4 shows that you can correct this problem by setting a default route within the TC config.1 e G t @ Terminal Concentrator (129.50. All Rights Reserved. the TC’s internal routing table can overflow.50.Setting a Terminal Concentrator Default Route Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.1.50. Inc. Revision C A-9 a .2.50.50.2. network connections can be intermittent or lost completely.35) n b ku tude a config.

the node console connections are not available.central Trying 129. Oracle and/or its affiliatesฺ To create a default route for the TC.annex.23 metric 1 active ^W annex# admin set annex routed n You may need to reset the appropriate port. you must edit an electrically erasable programmable read-only memory (EEPROM) file in the TC named config. Escape character is ’^]’.. Inc. annex# boot s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non Note – You must enter an IP routing address appropriate for your site.2 Administration Copyright 2009 Sun Microsystems.35 .1. Sun Services. A-10 Sun™ Cluster 3.Setting a Terminal Concentrator Default Route Creating a Terminal Concentrator Default Route Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. The following is a summary of the general process: admin-ws# telnet tc1.. Connected to 129.50.1.35.. a b a While the TC is rebooting.1. annex: su Password: root_password annex# edit config.. Annex subsystem or reboot the Annex for changes to take effect.annex (Editor starts) Ctrl-W:save and exit Ctrl-X:exit Ctrl-F:page down Ctrl-B:page up %gateway net default gateway 129. [Return] [Return] Enter Annex port name or number: cli . You must also disable the TC routing function.50. Revision C a .50. All Rights Reserved.

the normal node connection to port 1 is replaced and the TC rebooted. Inc. and the TC placed into setup mode. You might need a dedicated TC for each node in a cluster. All Rights Reserved. After configuration is complete. it might be necessary to have cluster host systems separated by more than the serial port cable limit. The maximum length for a TC serial port cable is approximately 348 feet. the node connection to port 1 must be switched with a serial connection to the configuration device. Oracle and/or its affiliatesฺ Using Multiple Terminal Concentrators A single TC can provide serial port service to a maximum of eight nodes. As shown in Figure A-5. Revision C A-11 a . s ) ha y m ฺ om ฺ c ฺ Router csm Guide e @ ent b u ak Network tud y ( S in this d Network u al use m ka e to TC g n s a n ab lice TC b u le k b a a l y sfer u d ab -tran Node 2 g n n o a n ab Node 1 Administrative Console Figure A-5 Multiple Terminal Concentrators Terminal Concentrator Copyright 2009 Sun Microsystems. Sun Services.Using Multiple Terminal Concentrators Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. If it is necessary to reconfigure the TC.

Revision C a . Sun Services. you must breaksthe telnet e m d i c thenGtype u quit. After o you c ฺ ฺ finished and logged out of the node. All Rights Reserved. Inc. such as the cconsole tool. the serial port remains locked u and e d k applications. When you have the telnet prompt. you can either use the telnet e u l k command connect directly to the node and use the Control-] keyboard a eraortobyou y l sequence. If connection with the Control-] keyboard sequencee and t @ nbe used by other b cannot you do not.Troubleshooting Terminal Concentrators Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. The commands to do this are not well documented in the cluster manuals. Oracle and/or its affiliatesฺ Troubleshooting Terminal Concentrators Occasionally. it is useful to be able to manually manipulate the TC.2 Administration Copyright 2009 Sun Microsystems. you can use the telnet command to connect to a specific serial port as follows: # telnet tc_name 5002 s ) ha y m ฺ m have You can then log in to the node attached to port 5002. ya Stu ( n di e this u l a us m Using the telnet Command a k e to to Abort a Node g an cens b a If you b have to abort li a cluster node. Using the telnet Command to Manually Connect to a Node If the cconsole tool is not using the TC serial ports. you can abort the a trconsole g n node with the following command: n a no ab telnet > send brk ok Note – You might have to repeat the command multiple times. A-12 Sun™ Cluster 3. can use the Control-] keyboard sequence in a cluster f u ns d b a window.

Inc. # telnet IPaddress Trying 129. Inc.146.Troubleshooting Terminal Concentrators Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. and then use the admin program to reset the locked port.241. and then use the resident command-line interface (CLI) to perform status and administration procedures. 8 ports admin : reset 6 admin : quit annex# hangup Terminal Concentrator Copyright 2009 Sun Microsystems.. Connected to 129.135. Revision C A-13 a .135 Escape character is '^]'. You can use the who command to identify which port is locked. Sun Services. All Rights Reserved. it can leave a telnet session active that effectively locks the port from further use.241.. you can get online t After you connect directly into u l e a us help as follows: am k e to g s annex: nhangup ban help e c a annex: help li b e u l k a erab y l f uIdentifying s d n and Resetting a Locked Port b a a r t g an non If a node crashes.0.146. annex: s ) ha ab y m ฺ om ฺ c ฺ csm Guide e t @ eCommand n b Using the Terminal Concentrator help u ak Stud y ( ina terminal d his concentrator. The command sequence is as follows: annex: who annex: su Password: annex# admin Annex administration MICRO-XL-UX R7.1. Enter Annex port name or number: cli Annex Command Line Interpreter * Copyright 1991 Xylogics. Oracle and/or its affiliatesฺ Connecting to the Terminal Concentrator Command-Line Interpreter You can use the telnet command to connect directly to the TC.

Oracle and/or its affiliatesฺ Using the TC erase command can be dangerous. the erase command is available only through the port 1 interface. the password will be set to this IP. For security reasons. G t @ n b ku tude a y n ( this S i d lu se a am to u k ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n no aba Enter 1 or 2 :: 1 A-14 Sun™ Cluster 3. Sun Services. it destroys idethe TC csm u e boot PROM-resident operating system.e. A typical procedure is as follows: monitor :: erase 1) EEPROM(i. Configuration information) 2) FLASH(i. All Rights Reserved. Self boot image) s ) ha y m ฺ om ฺ c ฺ Caution – Do not use option 2 of the erase command. It returns all settings to their default values.e.Troubleshooting Terminal Concentrators Erasing Terminal Concentrator Settings Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.2 Administration Copyright 2009 Sun Microsystems. Inc. Use it only when you have forgotten the superuser password. Revision C a . When the addr command is then run to give the TC its IP address.

All Rights Reserved. Sun Services. Revision C a .Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Appendix B Configuring Multi-Initiator SCSI This appendix contains information that can be used to configure multi-initiator Small Computer System Interface (SCSI) storage devices. including the Sun StorEdge™ MultiPack desktop array and the Sun StorEdge D1000 array. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a B-1 Copyright 2009 Sun Microsystems. Inc.

and set the other host adapters to n ens a b unused SCSI Proper planning dictates that these “unused” a addresses.) n b e d k The default hardware configuration in tu environment yaa multi-initiator ( S results in a conflict because all SCSI host adapters default to 7. The default SCSI address of SCSI host adapters in Sun™ systems is 7. This SCSI host adapter circuit is referred to as the SCSI initiator. As this terminology implies. All Rights Reserved. Cluster configurations share storage between multiple server nodes. b tran a . (The host adapter is u also a device on the SCSI bus. lic both b e SCSI addresses include currently and eventually unused addresses. Sun Services. u l k b a a l yAn example er of addresses unused in the future is the addition of storage f u s d by installing new drives into empty drive slots. on each SCSI bus leave one of the SCSI host ka address toof g e adapters with the SCSI 7. In a stand-alone server. You can set this property globally for a node or on a per host adapter basis. more than one SCSI initiator exists on the SCSI bus. n i is d h t u al use m To resolve this conflict. Sun™ Cluster 3.2 Administration Copyright 2009 Sun Microsystems. In most configurations. Inc. the server node controls the SCSI bus activities using the SCSI host adapter circuit connecting this server to a particular SCSI bus. the configuration is referred to as multi-initiator SCSI. Revision C a . When the cluster storage consists of singled-ended or differential SCSI devices. Instructions for setting a unique scsi-initiator-id for each SCSI host adapter are included in the chapter for each disk enclosure in Sun™ Cluster Hardware Guides. ang non ab B-2 You can change the selected SCSI addresses for these host adapters by setting the scsi-initiator-id OpenBoot™ PROM property.Multi-Initiator Overview Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. s ) ha y m ฺ om ฺ c ฺ m uide ca sSCSI The SCSI specification requires that each device on have a e Gbus t @ unique SCSI address.the available SCSI address for a second host adapter is 6. This circuit initiates all bus activities for this SCSI bus. Oracle and/or its affiliatesฺ Multi-Initiator Overview This section applies only to SCSI storage devices and not to Fibre Channel storage used for the multihost disks.

3. Connect the cables to the device. This procedure refers to the host adapter you choose for SCSI address 7 as the host adapter on the second node. 1. if you install k b a a l y sfer u d more information. The default SCSI address for host adapters is 7. you g e should avoid setting scsi-initiator-id for the first node to the SCSI n ens a b address a for thatlicdisk slot. Use this procedure to install a Sun StorEdge MultiPack device in a cluster prior to installing the Solaris™ Operating System and Sun™ Cluster software. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m Caution – Even k though a a tslot o in the device might not be in use.2 Installation Guide and your server hardware manual. This procedure refers to the host adapter with an available SCSI address as the host adapter on the first node. in step 7 you change the scsi-initiator-id of the remaining host adapter in the SCSI chain to an available SCSI address. Revision C B-3 a . Depending on the device and configuration settings of the device. as shown in Figure B-1 on page B-4. n n o a n ab 2. Sun Services. either SCSI address 6 or 8 is usually available. To avoid conflicts. All Rights Reserved. Perform this procedure with the procedures in Sun Cluster 3. Oracle and/or its affiliatesฺ Installing a Sun StorEdge Multipack Device This section provides the procedure for an initial installation of a Sun StorEdge MultiPack device. For the procedure on installing host adapters. Install the host adapters in the nodes that will be connected to the device. refer to OpenBoot™ 3. Reserve SCSI address 7 for one host adapter in the SCSI chain. Inc.x Command Reference Manual ab -tranFor g and the labels inside the storage device. refer to the documentation that shipped with your nodes. This precaution minimizes future complications b u leadditional disk drives. Ensure that each device in the SCSI chain has a unique SCSI address.Installing a Sun StorEdge Multipack Device Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Configuring Multi-Initiator SCSI Copyright 2009 Sun Microsystems.

power on the first node.Installing a Sun StorEdge Multipack Device Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Edit the nvramrc script to set the scsi-initiator-id for the host adapter on the first node. abort the system to continue with OpenBoot PROM b a OUT OUT Monitor tasks. Inc. Connect the b a a l y spower er source. If g n non a necessary. f u d ab -tr5. see the ‘‘The nvramrc Editor and nvedit Keystroke Commands’’ on page B-11. 6.2 Administration Copyright 2009 Sun Microsystems. and record these paths. Find the paths to the host adapters. For a list of nvramrc editor and nvedit keystroke commands. Refer to the documentation that shipped with the device for other restrictions regarding SCSI operation. and host adapter. Do not include the trailing /disk or /sd entries in the device paths. Oracle and/or its affiliatesฺ Make sure that the entire SCSI bus length to each device is less than 6 meters. This measurement includes the cables to both nodes. Revision C a . 7. Node 1 Host Adapter A Host Adapter B Node 2 Host adapter B Host Adapter A SCSI IN 9-14 1-6 SCSI IN IN 9-14 1-6 IN s ) ha y m ฺ m o c ฺ ฺ SCSI OUT SCSI OUT e m d s i ec2 t Gu Enclosure 1 Enclosure @ b den u k a Stu SCSI (y n cables di e this u l a us m a Figure B-1 Example of a Sun MultiPack Desktop Array k Mirrored toStorEdge g e Enclosure Pair ban licens a b u le AC power cord for each device of the pair to a different k 4. node. as well as the bus length internal to each device. Sun Services. B-4 Sun™ Cluster 3. Use this information to change the SCSI addresses of these controllers in the nvramrc script. All Rights Reserved. ok show-disks Identify and record the two controllers that will be connected to the storage devices.an Without allowing the node to boot.

The OpenBoot PROM Monitor prints the line numbers (0:. eAfter n s a n If you are not sure about the changes.4000/scsi@4 2: 6 encode-int " scsi-initiator-id" property 3: device-end 4: cd /pci@1f. All Rights Reserved. y S n ( ththeisnvedit i d The changes you make through command are done on a u se l a temporary copy script. discard them. type: k b a a l y sfer ok nvquit u d ok ab -tran g n non a To store the changes.1 6 encode-int " scsi-initiator-id" property device-end install-console banner ok Configuring Multi-Initiator SCSI Copyright 2009 Sun Microsystems. b lice a b To u lediscard the changes.4000/scsi@4 6 encode-int " scsi-initiator-id" property device-end cd /pci@1f. save the changes. 1:. type: b a ● ● ok nvstore ok 9. ok printenv nvramrc nvramrc = probe-all cd /pci@1f.1 5: 6 encode-int " scsi-initiator-id" property 6: device-end 7: install-console 8: banner <Control-C> ok s ) ha y m ฺ om ฺ c ฺ Note – Insert exactly one space after the first double idemark and csmquotation u e before scsi-initiator-id. Store the changes.Installing a Sun StorEdge Multipack Device Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. G t @ n b ku tude a 8. Sun Services. Revision C B-5 a . Verify the contents of the nvramrc script you created in step 7. You can continue to edit this mof thetonvramrc u a k copy without you complete your edits.4000/scsi@4. Oracle and/or its affiliatesฺ The following example sets the scsi-initiator-id to 6.4000/scsi@4. and so on). g risk. nvedit 0: probe-all 1: cd /pci@1f.

Installing a Sun StorEdge Multipack Device 10. Sun Services. Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Continue with the Solaris OS. abort the system to continue with OpenBoot PROM Monitor tasks.4000/scsi@4 ok . power on the second node. Sun Cluster software.2 Administration Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ ok setenv use-nvramrc? true use-nvramrc? = true ok 11.. Instruct the OpenBoot PROM Monitor to use the nvramrc script. All Rights Reserved. Inc. If necessary.. e m d s i ec t Gu @ b den u k ya Stu ( n di e this u l a us m a k e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha B-6 Sun™ Cluster 3. Verify that the scsi-initiator-id for the host adapter on the second node is set to 7. y m ฺ 13. ok cd /pci@1f. 12. Revision C a .properties scsi-initiator-id 00000007 . and volume m o c ฺ ฺ management software installation tasks. Without allowing the node to boot.

refer to the documentation that shipped with your nodes. Revision C B-7 a . Install the host adapters in the node that will be connected to the array. Use this procedure to install a Sun StorEdge D1000 array in a cluster prior to installing the Solaris OS and Sun Cluster software. you should aslot s m u Note – Even though a in the a kscsi-initiator-id to avoid setting the for the first node to the SCSI g e nthat disk s This precaution a n b address for slot. Sun Services. Connect the cables to the arrays. Reserve SCSI address 7 for one host adapter in the SCSI chain. This procedure refers to the host adapter with an available SCSI address as the host adapter on the first node. Ensure that each device in the SCSI chain has a unique SCSI address. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u l edevice might not be in use. in step 7 you change the scsi-initiator-id of the remaining host adapter in the SCSI chain to an available SCSI address.Installing a Sun StorEdge D1000 Array Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. 1. refer to OpenBoot™ 3. Perform this procedure with the procedures in Sun™ Cluster 3. To avoid conflicts.x Command Reference Manual b tranand the labels a inside the storage device. Inc. Oracle and/or its affiliatesฺ Installing a Sun StorEdge D1000 Array This section provides the procedure for an initial installation of a Sun StorEdge D1000 array. minimizes future complications e c a i l b ifuyou install leadditional disk drives. The default SCSI address for host adapters is 7. For the procedure on installing host adapters. as shown in Figure B-2. Configuring Multi-Initiator SCSI Copyright 2009 Sun Microsystems. SCSI address 6 is usually available. k b a a r l y For emore f u s d information. 3. This procedure refers to the host adapter you choose for SCSI address 7 as the host adapter on the second node. g onn a n ab 2. All Rights Reserved.1 Installation Guide and your server hardware manual.

an cens b a 5. Inc. 6. Use this information to Disk array 2 change the SCSI addresses of these controllers in the nvramrc script. r y l e f bdu transok show-disks a g onn Identify and record the two controllers that will be connected to the a n ab storage devices and record these paths. node. For a list of nvramrc editor and nvedit keystroke commands.2 Administration Copyright 2009 Sun Microsystems. 7. see ‘‘The nvramrc Editor and nvedit Keystroke Commands’’ on page B-11. All Rights Reserved.Installing a Sun StorEdge D1000 Array Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Power b onlethelifirst node and the arrays. Sun Services. Connect the ACa power k e to g power source. Node 1 Host Adapter A Host Adapter B Node 2 Host Adapter B Host Adapter A Disk array 1 s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ud Mirrored Pair akD1000 tArray y ( S Figure B-2 Example of a Sun StorEdge in this d u al cordufor seeach array of the pair to a different m 4. This measurement includes the cables to both nodes. Oracle and/or its affiliatesฺ Make sure that the entire bus length connected to each array is less than 25 meters. u k a Find athebpaths to the host adapters. B-8 Sun™ Cluster 3. and the host adapter. as well as the bus length internal to each array. Edit the nvramrc script to change the scsi-initiator-id for the host adapter on the first node. Do not include the /disk or /sd trailing parts in the device paths. Revision C a .

Oracle and/or its affiliatesฺ The following example sets the scsi-initiator-id to 6. save g thenchanges. n s a discard them. The OpenBoot PROM Monitor prints the line numbers (0:.Installing a Sun StorEdge D1000 Array Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Revision C B-9 a . ab lice b u To lestore the changes. Sun Services. 1:. and so on). Inc. ok printenv nvramrc nvramrc = probe-all cd /pci@1f. e If you are not sure about the changes. You u se a can continue tom edit this u copy without risk. nvedit 0: probe-all 1: cd /pci@1f. After you complete your a o k t edits. All Rights Reserved.1 5: 6 encode-int " scsi-initiator-id" property 6: device-end 7: install-console 8: banner <Control-C> ok s ) ha y m ฺ om ฺ c ฺ Note – Insert exactly one space after the first double idemark and csmquotation u e before scsi-initiator-id.4000/scsi@4 6 encode-int " scsi-initiator-id" property device-end cd /pci@1f. type: b a ● ● ok nvquit ok 9. Verify the contents of the nvramrc script you created in step 7. y S n ( thiscopy i d The edits are done lon a temporary of the nvramrc script. type: k b a a l y sfer ok nvstore u d ok ab -tran g n non a To discard the changes.4000/scsi@4.4000/scsi@4.4000/scsi@4 2: 6 encode-int " scsi-initiator-id" property 3: device-end 4: cd /pci@1f. G t @ n b ku tude a 8. Store or discard the changes.1 6 encode-int " scsi-initiator-id" property device-end install-console banner ok Configuring Multi-Initiator SCSI Copyright 2009 Sun Microsystems.

1 Installation For software installation procedures. Instruct the OpenBoot PROM Monitor to use the nvramrc script..properties scsi-initiator-id 00000007 differential isp-fcode 1. All Rights Reserved.4000/scsi@4 ok . Oracle and/or its affiliatesฺ ok setenv use-nvramrc? true use-nvramrc? = true ok 11. y ( n this S i d lu serefer to Sun™ Cluster 3. Without allowing the node to boot. If necessary. Revision C a . Verify that the scsi-initiator-id for each host adapter on the second node is set to 7. Inc. abort the system to continue with OpenBoot PROM Monitor tasks. ok cd /pci@1f. 13. Sun Services. Sun Cluster n b e u k tud management software installationatasks. Continue with the Solaris OS. 12. Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010..2 Administration Copyright 2009 Sun Microsystems. power on the second node. a Guide.21 95/05/18 device_type scsi . s ) ha y m ฺ om ฺ c ฺ csm Guide e t and volume @software.Installing a Sun StorEdge D1000 Array 10. am to u k ng ense a b a lic b e u l k rab a y ul nsfe d b a tra g n n a no ab B-10 Sun™ Cluster 3.

An nvramrc script contains a series of OpenBoot PROM commands that are executed during the boot sequence. Revision C B-11 a . s ) ha y m ฺ Table B-1 and Table B-2 on page B-12 list useful nvramrcoeditor m and c ฺ ฺnvedit nvedit keystroke commands. The OpenBoot PROM Monitor has a set of default aliases for the commonly occurring devices in the system. i b e u l resumes editing previous contents. respectively. add the entries to the end of the script. To edit an nvramrc script or merge new lines in an nvramrc script. All Rights Reserved. The procedures in this guide assume that this script is empty. and discards the contents of the temporary buffer. Prompts for confirmation. b den u k ya Stu ( Table B-1 The nvramrc Editor Commands n di e this u l a us Command Description m a k e to g nvedit an Enters nsthe nvramc editor. This command fails if nvedit is executed between the time the content of nvramrc was lost and the time the content of the nvramrc was executed. Configuring Multi-Initiator SCSI Copyright 2009 Sun Microsystems. you must use nvedit editor and nvedit keystroke commands. and you can a a r t g save the contents of this buffer by using the nvstore an non ab command. This command works on a buffer. nvstore Copies the contents of the temporary buffer to nvramrc.x G t @ Command Reference Manual. nvrun Executes the contents of the temporary buffer. If the data remains in the b ltemporary e c a buffer from a previous nvedit session. nvrecover Attempts to recover the content of the nvramrc if the content was lost as a result of the execution of set-defaults. Inc. nvquit Discards the contents of the temporary buffer.The nvramrc Editor and nvedit Keystroke Commands Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ The nvramrc Editor and nvedit Keystroke Commands The OpenBoot PROM Monitor builds its own device tree based on the devices attached to the system when the boot sequence is invoked. reads the k b a a r y contents of nvramrc into the temporary buffer and begin ul nsfe d b editing it. without writing it to nvramrc. Sun Services. For an m entire list of e d s i c OpenBoot™ editor and nvedit keystroke commands. Otherwise. If your nvramrc script contains data. refereto u 3. then enters the nvramrc editors as with nvedit.

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Oracle and/or its affiliatesฺ Table B-2 The nvedit Keystroke Commands Keystroke Description ^A Moves to the beginning of the line ^B Moves backward one character ^C Exits the script editor ^F Moves forward one character ^K Deletes until end of line ^L Lists all lines s ) ha y m ฺ ^N Moves to the next line of the nvramrc editing buffer om ฺ c ฺ m stays ^O Inserts a new line at the cursor positionsand idonethe c u e G current line t @ n b de editing buffer kofuthe nvramrc ^P Moves to the previous y line u a t ( is S n line i ^R Replaces the current d lu se th a m tocharacter u Delete Deletes aprevious k ng eannew se line at the cursor position and advances to Return baInserts a thelnext ic line b e u l k rab a y ul nsfe d b a tra g n n a no ab B-12 Sun™ Cluster 3.The nvramrc Editor and nvedit Keystroke Commands Table B-2 lists more useful nvedit commands. All Rights Reserved.2 Administration Copyright 2009 Sun Microsystems. Inc. Revision C a . Sun Services.

s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a C-1 Copyright 2009 Sun Microsystems. Inc. Revision C a . Oracle and/or its affiliatesฺ Appendix C Quorum Server This appendix describes basic operation of the quorum server software.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. All Rights Reserved.

query. without having to reboot. s ) ha y m ฺ om ฺ c The actual quorum server daemon.time after installation. Inc.Quorum Server Software Installation Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. This includes starting the daemon the first bd trstop a a . All Rights Reserved. ang non The package installation will install: ● ● ● ● ab ● An entry in /etc/services for the default port: scqsd C-2 9000/tcp # Sun Cluster quorum server daemon Sun™ Cluster 3. you obviously need to get the server software installed and rolling. /etc/scqsd/scqsd. stop. The names of the packages are SUNWscqsr and SUNWscqsu.conf ( in this d u An rc script (Solaris 9) SMF service 10) to automatically alordaemon seat boot(Solaris m u start the quorumaserver time. though you may prefer to just add the two packages by hand. See page 4. Revision C a . Oracle and/or its affiliatesฺ Quorum Server Software Installation Before configuring a quorum-server quorum device into the cluster. you k b a a always use the y fer clquorumserver (clqs) command if you need to start and l u nsthe daemon manually. The quorum server packages are not part of the cluster software core framework itself.2 Administration Copyright 2009 Sun Microsystems. /usr/cluster/lib/sc/scqsd ฺ e csm Gserver. Sun Services. and clear theequorum tclqs) @ name n /usr/cluster/bin/clquorumserver (short b e u ak Stud y A single configuration file. The quorum server software is one of the choices in the Java™ Enterprise System (Java ES) installer. uid The utility to start. k e to g ban licens a Noteu–bRegardless le of the OS that the quorum server is running on.

Quorum Server Copyright 2009 Sun Microsystems.conf . . Sun Services. # Lines beginning with # are treated as comments and ignored. # # To configure more instances. each must have a different port and directory. There isn’t a particular practical reason to run multiple daemons. All Rights Reserved.Configuring the Quorum Server Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. If you do run multiple daemons. Oracle and/or its affiliatesฺ Configuring the Quorum Server The configuration file is installed with a single quorum server instance running on port 9000: # cat /etc/scqsd/scqsd. # /usr/cluster/lib/sc/scqsd -d /var/scqsd -p 9000 s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a You can change the port number or data directory for the single instance. or you could add more lines representing more daemons. add commands to this file. If there is no text identifier (and there is none in the default line already in the configuration file). the server instance will be identified by its port number. Inc. You can optionally associate each instance with a text identifier (using -i textid). since the single instance can handle as many clusters as you like. Revision C C-3 a .

since the ) y m boot/shutdown scripts will do the job for you. Inc. or you can use the wildcard + to indicate all server daemons in the configuration file: # clqs start + # ps -ef|grep scqsd root 13108 13107 0 22:40:22 ? -d /var/scqsd -p 9000 root 13107 1 0 22:40:22 ? -d /var/scqsd -p 9000 0:00 /usr/cluster/lib/sc/scqsd 0:00 /usr/cluster/lib/sc/scqsd as h Normally.Starting and Stopping a Quorum Server Daemon Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved.Cluster aorangecat k rab(id 0x4480DAF7) Reservation --y dul ransfe Nodeab ID: 2 t g 0x4480daf700000002 n Reservation on key: a n b a --- Cluster orangecat (id 0x4480DAF7) Registrations --- Node ID: Registration key: 1 0x4480daf700000001 Node ID: Registration key: 2 0x4480daf700000002 --- Cluster chicos (id 0x448DE82F) Reservation --- Node ID: Reservation key: --- C-4 1 0x448de82f00000001 Cluster chicos (id 0x448DE82F) Registrations --- Sun™ Cluster 3. Once you n (again. Revision C a . s i i d h t as you like: the quorum server from as many lu clusters e a s am to u # clqs show k g === n9000 se === Quorum Server on port a n b e a lic b e u l --. You either give a single daemon identifier (port number or text identifier). Oracle and/or its affiliatesฺ Starting and Stopping a Quorum Server Daemon Use clqs start and clqs stop. ฺ om ฺ c ฺ m uide csSide) Displaying Quorum Server Data (on the Server e G t @ n b u the uquorum de server machine) k(on The following is an example of querying a t y S can happily add access to the state of the quorum server.2 Administration Copyright 2009 Sun Microsystems. Sun Services. you will not need to run these commands very much.

Inc. Revision C C-5 a . The currently not fenced out of the cluster) and theereservation t to ever perform @ reservation holder would be equivalent to the last node n b e u any quorum fencing in the cluster. the command displays the registrants (nodesethat m d s i c Gu holder. Sun Services. in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g an non ab Quorum Server Copyright 2009 Sun Microsystems.yit aisk really tuthedregistrants that are more ( S interesting. All Rights Reserved. Oracle and/or its affiliatesฺ --- Cluster food (id 0x448E2AE6) Reservation --- Node ID: Reservation key: --- 1 0x448de82f00000001 1 0x448e2ae600000001 Cluster food (id 0x448E2AE6) Registrations --- Node ID: Registration key: 1 0x448e2ae600000001 Node ID: Registration key: 2 0x448e2ae600000002 s ) ha y m ฺ m o c ฺ ฺ are For each cluster.Starting and Stopping a Quorum Server Daemon Node ID: Registration key: Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

Clearing a valid quorum server could compromise the cluster quorum. In the previous output. Oracle and/or its affiliatesฺ Clearing a Cluster From the Quorum Server If you remove a quorum server object cleanly from the cluster side. d u l e using the quorum server as a a quorum s device will not find out until it is m u a too late! o k et g n ba licens a b u le k b a a l y sfer u d ab -tran g an non ab C-6 Sun™ Cluster 3. If you rebuilt a cluster without ever deleting old quorum information. its data will be removed cleanly from the quorum server state. and run: s ) ha # clqs clear -c chicos -I 0x448DE82F 9000 The quorum server to be cleared must have been removed from the cluster. Do you want to continue? (yes or no): yes y m ฺ om ฺ c ฺ csm Guide e @ ent that is actually b Note – Be careful not to clear quorum server information u ud out a JBOD disk ak Styanking still in use.Clearing a Cluster From the Quorum Server Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. Revision C a . In order to clear out the cluster information. I have since actually rebuilt the cluster chicos and I am no longer using the quorum server on behalf of that cluster. which is a quorum disk device. the cluster itself in tInhiother device. you may have to clean up the quorum server manually. All Rights Reserved. It is exactly equivalent to(y physically s words. Inc.2 Administration Copyright 2009 Sun Microsystems. obtain the cluster name and ID from the output of clqs show.

Oracle and/or its affiliatesฺ Appendix D Role-Based Access Control This appendix provides a brief review of role-based access control (RBAC). s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a D-1 Copyright 2009 Sun Microsystems.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Inc. All Rights Reserved. Sun Services. Revision C a .

cluster. privileged lic but this is not a requirement. having a particular authorization means nothing unless the software you are running is specifically “instrumented” to check for your authorization and grant you a particular right to do a task. Role A role is like a user. lu se a uand shadow databases just like regular athempasswd o Roles have entries in k t ngroles seassigned one of the “pf-shells” so they can run users.cluster. but authorizations are interpreted only by RBAC-aware applications. the following sections provide a review of basic RBAC concepts to help you understand its integration into the cluster. Inc. Sun Services.” dthe smwhere i c u e intention is that multiple administrators assume this identity G when they t @ n b need to do privileged tasks. Oracle and/or its affiliatesฺ Brief Review of RBAC Terminology Because RBAC can be confusing. Authorizations have names like solaris. D-2 Sun™ Cluster 3.cluster. except for the following: ● You can not log in to it directly. The names form a natural hierarchy. Authorizations are meant to convey certain rights to users or roles.read. All Rights Reserved. s ) ha y m ฺ om ฺ c ฺ e A role is meant to replace an old-style “privileged user.modify and solaris.2 Administration Copyright 2009 Sun Microsystems. The advantage is that you must log in as a u ude k a t regular user and also know the password for the role and be allowed to n (y this S i assume the role to assume the d identity. b e u l k rab a y ul nsfe d Authorization b a tra g n n a no ● ab Only certain users are allowed to switch to the role. You must log in as a user and then use the su command to switch to the role.* has the meaning that you expect. Most often.Brief Review of RBAC Terminology Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. are a n b e acommands. Revision C a . In other words. so that assigning solaris.

or they can be part of “rights profiles. smor both. Authorizations can be assigned directly to a user or role or through inclusion in a profile assigned to the user or role. a user can be u kgiven rthe a abprivilege to run a specific command and have root-like ability to y l e f s all files regardless of their permissions. idThis c u e G bit. n ( this S i d lu se a Starting With Solaris am 10toOSu k ng enthe seSolaris 9 command privileges feature to support baaugments Solarisa10 lic b fine grainedle privileges over a specific command.” which are collections of authorizations and command privileges.privilege to write all files. Role-Based Access Control Copyright 2009 Sun Microsystems. Oracle and/or its affiliatesฺ Assigning Authorizations Authorizations can be assigned to users or roles. The rights profiles could then be assigned to users or roles. All Rights Reserved.Brief Review of RBAC Terminology Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Command privileges can be assigned only to a user or role through its inclusion in a profile. Command Privileges Command privliges have been enhanced in the Solaris 10 OS: Solaris 9 OS and Earlier s ) ha y m ฺ om ฺ with a Command privileges allow users or roles to run certainฺccommands e is different real user ID (uid). Inc. rather than the individual authorizations. effective user ID (euid). Revision C D-3 a . but still be denied the bdu tranread a . this RBAC similar to turning the setuid bit on. For example. ang non ab Profiles Profiles are collections of authorizations and command privileges. But unlike the setuid t @ n b mechanism lets you restrict which users deare allowed to obtain ku or roles u a t y command privileges. Sun Services.

Inc. Sun Services. users. ub@ ent ak Stud y ( in su this users roles d u l e a am to us k ng ense a b profiles a lic b e u l k rab a y ul nsfe d b a auths command privs tra g n n o a n s ) ha RBAC Relationships ab Figure 12-2 RBAC Relationships D-4 Sun™ Cluster 3. and roles.Brief Review of RBAC Terminology Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. y m ฺ om ฺ c ฺ ide csmprivileges u Figure 12-2 shows the relationships between command e G authorizations. profiles.2 Administration Copyright 2009 Sun Microsystems. and there is no way to remove it. All Rights Reserved. the solution is to do the following: ● Remove the offending authorizations from the Basic Solaris User profile. Revision C a . ● Create a custom profile that includes only the offending authorizations. If there are authorizations included in the Basic Solaris User profile that you want to remove from some users. and assign that to the users to whom you want to give those authorizations. Oracle and/or its affiliatesฺ The Basic Solaris™ User Profile The Basic Solaris User profile is a “magical” profile in that it is automatically assigned to every user.

Sun Services.{read.admin.Simplified RBAC Authorizations in the Sun™ Cluster 3.device.cluster.{read.appinstall ● solaris. It is still recognized (for backward compatibility) e u ak Stud y ( solaris.cluster. s m u ka e to solaris. All Rights Reserved. Revision C D-5 a .admin.system.cluster.resource. Inc.2 Environment All the software is fully instrumented to understand RBAC authorizations.cluster. just for easeaofl notation.modify a a l y sfer u d b tran solaris.1 environment had a finer granularity t but not required: @ n b system.admin ● solaris.quorum.cluster.cluster. in this admin.{read.admin.{read.modify} a solaris. modify} ● solaris.system.1 Environment (for y ฺm m Backward Compatibility) o ab ฺc eฺ m s c Gauthorization uid e The Sun Cluster 3. modify} d u e denotes three separate authorizations: This.read g ban licens solaris.cluster.{read.{read.cluster.2 Environment Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.cluster.system.admin.admin.cluster.admin a b u le k b solaris.read (part of Basic Solaris User) ● solaris.cluster.cluster.cluster.cluster.modify} ● solaris. Oracle and/or its affiliatesฺ Simplified RBAC Authorizations in the Sun™ Cluster 3.admin.modify} ● solaris.gui Role-Based Access Control Copyright 2009 Sun Microsystems.modify} ang non ● ● ● ● solaris. The only authorizations you need (discussed in Module 5) are: ● solaris.network.modify} ● solaris.modify s ) ha RBAC in the Sun Cluster 3.{read.install ● solaris.system.node.cluster.transport.cluster.

Oracle and/or its affiliatesฺ The following example assigns Sun Cluster admin privileges (but not modify privileges) directly to a user (rather than through a profile).Simplified RBAC Authorizations in the Sun™ Cluster 3. When they assume the role. they have the authorizations implied by the role. All Rights Reserved.cluster. Inc.cluster. Various users can be given permission to assume the role.2 Administration Copyright 2009 Sun Microsystems.admin clusoper # usermod -R clusoper youzer # usermod -R clusoper shmouzer D-6 Sun™ Cluster 3. y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha # roleadd -u 9301 -d /export/home/clusoper \ -A solaris.admin youzer Assigning Sun Privileges to a Role The following example assigns the Sun Cluster admin privileges to a new role. Revision C a .2 Environment Assigning Sun Cluster Command Privileges to a User Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. Sun Services. # usermod -A solaris.

Revision C a . Sun Services.Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. All Rights Reserved. Inc. Oracle and/or its affiliatesฺ Appendix E Logical Domains (LDoms) Guidelines This appendix describes basic guidelines for configuring LDoms for usage with Sun Cluster. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a E-1 Copyright 2009 Sun Microsystems.

Inc.2 Administration Copyright 2009 Sun Microsystems. Sun Services.2 2/08 (Update 1). y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a s ) ha E-2 Sun™ Cluster 3. The following qualifications apply: ● Sun Cluster is supported in guest LDoms and IO (service) LDoms ● You can mix and match LDoms and physical machines (or hardware domains) in the same cluster. Revision C a . ● Multiple LDoms on the same physical server can be members of different clusters ● Multiple LDoms on the same physical server can be members of the same cluster. The entire cluster consists of LDoms on the same physical machine. Oracle and/or its affiliatesฺ LDoms are supported as Sun Cluster nodes. All Rights Reserved. beginning with Sun Cluster 3. This is supported but cannot provide true high availability.LDom Support LDom Support Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

the discussion centers on the correct implementation of LDoms for use of cluster nodes. Oracle and/or its affiliatesฺ LDom Configuration for Sun Cluster This discussion revolves around the typical deployment of a guest LDom as a cluster node. All Rights Reserved. Inc. Sun Services. as illustrated in Figure E-1. s ) ha y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( in this d u al use m ka e to g ban licens a b u le k b a a l y sfer u d ab -tran g n non a b a Figure E-1 Guest LDom as a cluster node Guidelines for LDom Configuration Here are the rules and guidelines for the various elements: Logical Domains (LDoms) Guidelines Copyright 2009 Sun Microsystems. The fact that a peer node in a cluster may also be an LDom or may be a physical machine has no bearing whatsoever on the correct deployment of one particular guest LDom. Revision C E-3 a . It is not important that both are all nodes of the cluster are LDoms.LDom Configuration for Sun Cluster Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010.

An LDom that is a cluster node cannot hide behind routing or NAT provided by its I/O domain. because it needs to be in the same subnet space as another node (whether that node be another LDom or a physical node) so that IP addresses can properly fail over on the public network. IPMP Groups s ) ha IPMP groups must be configured in the guest LDom that is serving as the cluster node.SP 4 4G 0. A virtual switch is added and then a virtual adapter is ab provided for the switch for the guest domain. b e priority handling u l k a erab y l f u Thenfollowing s d example is run in the control domain. regardless of whether groups are configured in the I/O domain. The guest domain’s IPMP group is independent of any that you configure in an I/O domain. a There is aicspecial flag mode=sc for these switches to indicate lof traffic. Oracle and/or its affiliatesฺ You must configure the standard “bridged networking” in the I/O domain. IP address failover (LogicalHostname) works only when IPMP is configured on the public network of the cluster node. Inc. # ldm list NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME primary active -n-cv. in this configuration the I/O domain uses a vnet adapter to bridge the guest LDom (cluster node) out to the same subnet. y m ฺ om ฺ c ฺ csm Guide e @ ent b u ak Stud y ( Cluster Interconnect in this d u al use m You must configurektwo domain physical adapters as virtual a separate to I/O g e switches bridging the Sun Cluster node (guest LDom) to the private n ens a b network.LDom Configuration for Sun Cluster Public Network Unauthorized reproduction or distribution prohibitedฺ Copyright© 2010. E-4 Sun™ Cluster 3.4% 12h 40m myLDom1 active -n---.