<Insert Picture Here>

Everything you ever wanted to know

about the Cluster Health Monitor (CHM)
Markus Michalewicz
Principal Product Manager Oracle RAC & Oracle Clusterware
The preceding is intended to outline our general
product direction. It is intended for information
purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decisions.
The development, release, and timing of any
features or functionality described for Oracles
products remains at the sole discretion of Oracle.
<Insert Picture Here>
Agenda
Introduction
What is? Why use? Where to get?
Cluster Health Monitor (CHM)
Installation
Of the Tool
Of the GUI
CHM in Action
Administration
FAQ & More Information
OTN Migration
What is the Cluster Health Monitor (CHM)?
Introduction
The Cluster Health Monitor (CHM)
(formerly a.k.a. Instantaneous Problem Detector for Clusters or IPD/OS)
is designed to
detect and to analyze operating system (OS)
and cluster resource related degradation and failures
in order to bring more explanatory power to many issues that occur in clusters,
in which Oracle Clusterware and / or Oracle RAC are used, e.g. node evictions.
It is Oracle Clusterware and Oracle RAC independent in the current release.
Why should you use CHM?
Because there is a Monday morning for example
Assume the following scenario:
Leaving the office Friday night
Getting an email that one node in the cluster rebooted on Sunday morning
Getting a question from your manager why that node rebooted on Monday
Typical way of addressing this question:
Gather and analyze Oracle Clusterware and operating system logs
(e.g. following MOS doc 330358.1 - CRS 10gR2/ 11gR1/ 11gR2 Diagnostic Collection Guide)
Open a Service Request with Oracle Support
Possible outcomes:
Oracle Support finds the answer in one of the logs
Oracle Support needs more node specific information to answer the question
For the latter: This why you need Cluster Health Monitor (CHM) for example
Why should you use CHM?
Because you want to prevent another incident
Based on the previous scenario:
It is determined that the reboot was caused by
an abnormally high CPU load in conjunction with extreme IO waits.
Your manager asks you:
What caused the high CPU load? What can we do to prevent this in future?
For the latter: CHM provides a historical view on collected data for analyzes
>crfgui -d "00:05:00" -m 192.168.2.8
Cluster Health Analyzer V1.10 Look for Loggerd via node 192.168.2.8
...reading 300 sec from the past
Connected to Loggerd on rac1
Note: Node rac1 is now up
Cluster 'MyCluster', 2 nodes. Ext time=2010-08-18 23:22:30
Where can I get CHM?
Free Download
Direct download link:
http://www.oracle.com/technetwork/database/clustering/downloads/ipd-download-
homepage-087212.html
But better go to:
http://www.oracle.com/goto/rac
On this page, follow this link:
Cluster Health Monitor Download
Reason: OTN Migration (later)
<Insert Picture Here>
Installation
How to Install CHM?
Use the documentation
Overview of Cluster Heath Monitor (CHM)
(http://www.oracle.com/technetwork/database/enterprise-edition/ipd-overview-130032.pdf)
Summary of installation steps:
1. Download the software
2. Unzip the downloaded file
Do not install from a shared file system
3. Set up an OS-user for CHM
The user must have passwordless SSH access to all nodes
The user can be the same as the Oracle Grid Infrastructure-owner
4. Install the software
$CHM_install_DIR/install/crfinst.pl i {node1,node2} b /BDBdirectory
Do not use a shared destination for the location of the BDBdirectory
The software is distributed across all nodes specified under i automatically
Define one of the nodes as the master node
Run crfinst.pl -f -b /BDBdirectory as root on all nodes to enable the tool
How to Install CHM?
Tips and tricks part 1
Set up an OS-user for CHM
The user must have passwordless SSH access to all nodes
The user can be the same as the Oracle Grid Infrastructure-owner
The passwordless SSH setup basically follows the configuration that
you would use for the Oracle Grid Infrastructure (Oracle Clusterware
11g Release 2) setup, which allows setting up the passwordless SSH
automatically.
If you plan on deploying Oracle Grid
Infrastructure on this system, you
might want to do it first and then install
Cluster Health Monitor.
How to Install CHM?
Tips and tricks part 2
Install the software
$CHM_install_DIR/install/crfinst.pl i {node1,node2} b /BDBdirectory
Do not use a shared destination for the location of the BDBdirectory
The software is distributed across all nodes specified under i automatically
For the BDBdirectory (Berkeley Database directory) some rules apply:
Do not use a shared destination. Local storage is preferred.
IF you want to use a shared location, make sure that you have a different
folder / directory for the BDB for each node in the cluster (you can specify
this in the final crfinst.pl -f -b /orachmbdbrac/ run as root).
This directory must not be in the root FS
It must be of sufficient size per node (2GB* # of nodes in the cluster):
ERROR: Enough space not available on /u01/orachmbdb.
Space available 4002196 KB, but required 4194304 KB
Please rerun with a valid storage location for BDB.
The location should be a path on a volume with at least
2GB per node space available and writable by root only.
It is recommended to not create it on root filesystem.
After the Installation
Enable the stack
The tool is installed under: /usr/lib/oracrf/ on Linux
After the stack is installed, you need to enable it on each node:
[root@rac1 ~]# /etc/init.d/init.crfd
Usage: /etc/init.d/init.crfd {start|stop|restart|status|disable|enable}
[root@rac1 ~]# /etc/init.d/init.crfd enable
[root@rac1 ~]# ssh rac2
Last login: Tue Aug 17 21:21:52 2010 from rac1
[root@rac2 ~]# /etc/init.d/init.crfd enable
What you will find:
[root@rac2 ~]# ps -ef |grep oracrf
root 28025 27949 0 21:26 ? 00:00:00 /bin/sh/usr/lib/oracrf/bin/crfcheck
root 28028 27949 0 21:26 ? 00:00:00 /usr/lib/oracrf/bin/osysmond
oracle 28089 1 0 21:26 ? 00:00:00 /usr/lib/oracrf/bin/oproxyd
root 28127 1 0 21:26 ? 00:00:00 /usr/lib/oracrf/bin/ologgerd -m rac1 -r -d /u01/orachmbdb/
After the Installation
What was installed on a 2-node cluster?
osysmond
ologgerd
oproxyd
osysmond
ologgerd
oproxyd
3 daemons are installed
osysmond is the monitoring and OS metric collection daemon on every node
ologgerd follows a master / standby paradigm if more than 1 node in the cluster;
the master manages the OS metric database in BDB
and interacts with the standby to manage a replica of the master metrics
oproxyd is a proxy on all nodes which handles connection to the public interface
(default port: 61027, not configurable in current release / as of 11.2.0.1)
The GUI needs to be installed separately.
It is recommended to install the GUI on a separate (client) machine
The GUI can be installed on (one) node(s) of the cluster, if it has to
If your client is a Windows client, download the Windows version of the tool
Unzip and install the GUI using:
Usage: crfinst.pl -a [<nodelist>]
-c [<nodelist>]
-d
-f [-b <bdb loc>]
-g <ui install dir>
-h
-i <nodelist> -b <bdb loc> [-m <master>]
-N ClusterName.
Note that the installation is performed using a perl script
An Oracle client installation typically includes a perl version you can use
How to install the GUI?
Use the documentation + tips & tricks
<Insert Picture Here>
Cluster Health Monitor
in Action
Cluster Health Monitor in Action
Get started or Overview
D:\chmgui\bin>crfgui -m 192.168.2.8
Cluster Health Analyzer V1.10
Look for Loggerd via node 192.168.2.8
...Connected to Loggerd on rac1
Note: Node rac1 is now up
Cluster 'MyCluster', 2 nodes. Ext time=2010-08-19 01:01:25
Making Window: IPD Cluster Monitor V1.10 on mmichale-lap, Logger V1.04.20091223,
Cluster "MyCluster" (View 0), Refresh rate: 1 sec
Cluster Health Monitor in Action
Overview in detail
Cluster Health Monitor in Action
Details as required
Cluster Health Monitor in Action
Alerts based on pre-defined thresholds
Cluster Health Monitor in Action
Alerts and detailed views on the situation
Cluster Health Monitor in Action
Information on processes, CPU usage, network, IO, etc.
Cluster Health Monitor in Action
Even more details for IO and network
Back to: Why should you use CHM?
Because of the Monday morning
Assume its Monday and you are:
getting a question from your manager what caused the high CPU load on
Current view (assume Monday morning) no issues:
Lets go back in time (crfgui -d "00:35:00" -m 192.168.2.8):
<Insert Picture Here>
Administration
Administration part 1
The main administration tool for CHM: oclumon
[oracle@rac1 ~]$ oclumon -h
For help from command line : oclumon <verb> -h
For help in interactive mode : <verb> -h
Currently supported verbs are :
showtrail, showobjects, dumpnodeview, manage, version, debug, quit and help
[oracle@rac1 ~]$ oclumon version
Instantaneous Problem Detection - OS Tool, Version 1.04.20091223 - Production
Copyright 2009 Oracle. All rights reserved.
Administration part 2
How long can I go back in time?
Reviewing historical data is limited by the size of the Berkeley DB
By default the database retains the node views from all the nodes
for the last 24 hours in a circular manner.
This limit can be increased to 72 hours by using the following oclumon command:
'oclumon manage -bdb resize 259200'.
resize is set in seconds
In the current release (as of 11.2.0.1) you cannot query the current retention time
You can, however, set it to the time that you think is appropriate / reasonable
Whenever time is specified in the format HH:MM:SS, it refers to the
amount of time that you want to go back (in hours, minutes, seconds).
This command: crfgui -d "00:35:00" -m 192.168.2.8
Views the data 35 minutes ago from now.
Administration part 3
Get me information on the command line
> oclumon dumpnodeview -v -n rac1 -last "00:00:03 3 seconds
----------------------------------------
Node: rac1 Clock: '08-19-10 03.53.53 UTC' SerialNo:63193
----------------------------------------
SYSTEM:
#cpus: 2 cpu: 4.5 cpuq: 1 physmemfree: 13896 mcache: 959952 swapfree: 1900208 ior: 0 iow: 297 ios: 17 netr: 57.9 netw: 43.56 procs: 187 rtprocs: 11 #fds: 2658 #sysfdlimit:
6815744 #disks: 7 #nics: 4 nicErrors: 0
TOP CONSUMERS:
topcpu: 'osysmond(13446) 0.66' topprivmem: 'ologgerd(13532) 102260' topshm: 'ologgerd(13532) 46680' topfd: 'crsd.bin(10754) 102' topthread: 'crsd.bin(10754) 58'
PROCESSES:
name: 'osysmond' pid: 13446 #procfdlimit: 1024 cpuusage: 0.66 memusage: 78912 shm: 41196 #fd: 22 #threads: 9 priority: 139
name: 'orarootagent.bi' pid: 10890 #procfdlimit: 65536 cpuusage: 0.66 memusage: 6420 shm: 10032 #fd: 7 #threads: 34 priority: 19
name: 'ologgerd' pid: 13532 #procfdlimit: 1024 cpuusage: 0.0 memusage: 102260 shm: 46680 #fd: 19 #threads: 9 priority: 139

DEVICES:
sdf ior: 0.0 iow: 0.0 ios: 0 qlen: 0 wait: 0 type: SYS
sdf1 ior: 0.0 iow: 0.0 ios: 0 qlen: - wait: - type: SYS
sde ior: 0.0 iow: 0.0 ios: 0 qlen: 0 wait: 0 type: SYS
sde1 ior: 0.0 iow: 0.0 ios: 0 qlen: - wait: - type: SYS
sdd ior: 0.0 iow: 0.0 ios: 0 qlen: 0 wait: 0 type: SYS

NICS:
lo netrr: 21.3 netwr: 21.3 neteff: 42.7 nicerrors: 0 pktsin: 7 pktsout: 7 errsin: 0 errsout: 0 indiscarded: 0 outdiscarded: 0 inunicast: 7 innonunicast: 0 type:
PUBLIC
eth0 netrr: 25.65 netwr: 15.94 neteff: 41.60 nicerrors: 0 pktsin: 13 pktsout: 13 errsin: 0 errsout: 0 indiscarded: 0 outdiscarded: 0 inunicast: 13 innonunicast:
0 type: PRIVATE latency: <1
eth1 netrr: 10.27 netwr: 6.58 neteff: 16.85 nicerrors: 0 pktsin: 30 pktsout: 22 errsin: 0 errsout: 0 indiscarded: 0 outdiscarded: 0 inunicast: 30 innonunicast: 0
type: PRIVATE latency: <1
eth2 netrr: 0.12 netwr: 0.0 neteff: 0.12 nicerrors: 0 pktsin: 0 pktsout: 0 errsin: 0 errsout: 0 indiscarded: 0 outdiscarded: 0 inunicast: 0 innonunicast: 0
type: PUBLIC latency: <1
PROTOCOL ERRORS:
IPHdrErr: 0 IPAddrErr: 0 IPUnkProto: 0 IPReasFail: 0 IPFragFail: 0 TCPFailedConn: 50 TCPEstRst: 13 TCPRetraSeg: 69 UDPUnkPort: 41 UDPRcvErr: 0
End of data
Administration part 4
Time is crucial the clock
> oclumon dumpnodeview -n rac1 -s "2010-08-19 02.00.01" -e "2010-08-19 02.00.03"
----------------------------------------
Node: rac1 Clock: '08-19-10 02.00.01 UTC' SerialNo:58695
----------------------------------------
SYSTEM:
#cpus: 2 cpu: 4.20 cpuq: 4 physmemfree: 17728 mcache: 953248 swapfree: 1900208 ior: 0 iow: 103
ios: 7 netr: 46.36 netw: 39.29 procs: 187 rtprocs: 11 #fds: 2658 #sysfdlimit: 6815744
#disks: 7 #nics: 4 nicErrors: 0
TOP CONSUMERS:
topcpu: 'osysmond(13446) 1.31' topprivmem: 'ologgerd(13532) 102260' topshm: 'ologgerd(13532)
46680' topfd: 'crsd.bin(10754) 102' topthread: 'crsd.bin(10754) 58'
End of data
Alternative:
oclumon dumpnodeview -allnodes -s "2010-08-19 02.00.01" -e "2010-08-19 02.00.03
The "Clock:" in the oclumon output is printed in the
timezone which the master daemon is running with.
Administration part 5
Sampling data and refresh rate
Two independent rates to distinguish:
1. The sampling rate of the tool
2. The refresh rate of the GUI
The sampling rate of the tool depends on the currently active processes
and the devices on the system. Up to a total of 1000 active processes and
disks with ideal system, the sampling interval is approximately 1 second.
The refresh rate of the GUI is 1 second per default, but a higher refresh
rate can be specified using the r parameter followed by the time in secs.
Example: crfgui -r 5 -m 192.168.2.8
<Insert Picture Here>
Frequently Asked
Questions (FAQ)
The most common FAQs
Are answered in the tool readme
Direct download link:
http://www.oracle.com/technetwork/database/clustering/downloads/ipd-download-
homepage-087212.html
But better go to:
http://www.oracle.com/goto/rac
On this page, follow this link:
Cluster Health Monitor Download
and then Readme
FAQ #1
Is CHM a CVU (Cluster Verification Utility) replacement?
NO
CVU is a separate tool with
a completely different purpose.
CVU does not gather nor
provide the same data that
CHM provides.
For more information on CVU
got to:
http://www.oracle.com/goto/rac
On this page, follow this link:
Cluster Verification Utility - Download
FAQ #2
Can CHM be used as an OS Watcher replacement?
YES
OS Watcher (OSW) is a collection of UNIX shell scripts(*) intended to
collect and archive operating system and network metrics
to aid support in diagnosing performance issues.
OSW is provided by and made available through Oracle Support.
For more information see MOS doc ID 301137.1 - OS Watcher User Guide
Note: OS Watcher may have some specific environments, in which it
provides additional information (e.g. Version 3.0 OS of Watcher adds
additional collections for Exadata, as per the MOS note mentioned.)
(*) on Unix there is also a Windows version of OS Watcher
FAQ #3
Is CHM the standard tool to be used?
YES
Oracle RAC Development recommends using CHM whenever possible:
When using Oracle Clusterware, Oracle Grid Infrastructure, or Oracle RAC
The current release is available on Linux and Windows both 32 and 64bit.
CHM will be the standard tool moving forward
Therefore, more OSs will be supported in future
<Insert Picture Here>
More Information
Future Development of CHM
What you will find in Oracle Grid Infrastructure 11.2.0.2
Cluster Health Monitor is planned to be integrated with
Oracle Grid Infrastructure starting with 11.2.0.2 as follows:
The data gathering part of the tool will be part of the standard installation
CHM will therefore be installed into the Oracle Grid Infrastructure home
The Berkeley DB will be installed in the Oracle Grid Infrastructure home (default)
The GUI remains as a separately downloadable item
Changes in some parts of the architecture are possible, but the principles remain
The tool will provide more configuration options on the command line for example
The tool will be enabled per default with a default retention time (adjustable)
Going forward, all OS supported for Oracle Grid Infrastructure
will be supported for Cluster Health Monitor.
More Operating Systems are planned to be supported for CHM as 11.2.0.2
becomes available on these Operating Systems (last planned for 11.2.0.3)
http://www.oracle.com/goto/rac
Download link: Cluster Health Monitor - Download
http://www.oracle.com/goto/clusterware
Technical White Paper
Oracle Clusterware 11g Release 2 Technical Overview
For OS Watcher
My Oracle Support doc ID 301137.1 - OS Watcher User Guide
More Information
OTN Migration
A migration with some impact
Note that Oracle Technology Network (also known as OTN) was migrated
URLs containing http://otn.oracle.com/ are moved
Individual items (e.g. papers) are migrated to a new Content Management System
Direct links using the old URL to those items may therefore not work anymore
Some links to main pages should be redirected to some new pages e.g.:
http://otn.oracle.com/rac (might go away over time)
http://www.oracle.com/technetwork/database/clustering/overview/index.html
New short links available:
www.oracle.com/goto/rac
www.oracle.com/goto/clusterware
www.oracle.com/goto/asm
Items are linked on the main pages to the new URLs
Tip: follow the links on the main pages until migration is complete