P. 1


|Views: 5|Likes:
Published by Thota Mahesh

More info:

Published by: Thota Mahesh on Aug 29, 2013
Copyright:Attribution Non-commercial


Read on Scribd mobile: iPhone, iPad and Android.
download as PPT, PDF, TXT or read online from Scribd
See more
See less





Looking at RAC, GI/Clusterware Diagnostic Tools
Leighton L. Nelson Oracle DBA Team Lead (10 yrs experience, 6 years with RAC) RAC SIG US Events Chair and IOUG Liaison

Session# 373

Clusterware & RAC is Complex! .

Where do I begin? .

ASM & RAC Diagnostics • • • • • • Diagcollection Cluster Verification Utility (cluvfy) Cluster Health Monitor (CHM) Remote Diagnostics Agent (RDA) ADRCI/Support Workbench OS Utilities .Clusterware.

pl --collect --crshome $ORA_CRS_HOME (10gR2) $GRID_HOME/bin/diagcollection. traces plus OS logs and core files* $ORA_CRS_HOME/bin/diagcollection.Diagcollection • • Gathers and packages Clusterware logs. Limited information collected if not run as root In 11.pl --collect --core|crs|all (11gR2) Logs can be filtered by date/time with --adr --beforetime --aftertime Allocate enough space in current directory for diagnostic files Needs to be run on all nodes in the cluster.2 diagcollection enhanced to collect ADR and CHM data • • • • • • .

0/grid/bin/diagcollection. Note: core files will be packaged only with the --core option.gz -> logs from Operating System Collecting crs data .tar.tar.sh --collect Production Copyright 2004. traces and cores from CRS home. ocrcheck etc coreData_oelgrid02_20120225_1723. All rights reserved Cluster Ready Services (CRS) diagnostic collection tool The following CRS diagnostic archives will be created in the local directory: crsData_oelgrid02_20120225_1723.gz -> ocrdump.gz -> contents of CRS core files in text format osData_oelgrid02_20120225_1723. Oracle. 2010. ocrData_oelgrid02_20120225_1723.tar.gz -> logs.tar.diagcollection example [root@oelgrid02 u02]# /u01/app/11.2.

0.cvu “cluvfy comp –list” displays components that can be checked For standalone cluvfy set CV_HOME CV_JDKHOME and CV_DESTLOC .0 .2.2.Cluster Verification Utility • • • • • Cluvfy runs in stage mode or component mode Can be executed from the Grid Infrastructure Home in 11gR2 or from installation media New resource in 11.ora.

3.0.Cluster Verification Utility • • • • • • Use stage mode during installation/upgrade Use component mode to diagnose components after Clusterware installation Doesn’t diagnose all components e. HAIP $GRID_HOME/bin/cluvfy $INSTALL_DISK/runcluvfy.0 : cluvfy comp healthcheck .g.2.sh New in 11.

Cluster Verification Utility cluvfy comp –list output .

0.2+ Collects metrics at 1 sec interval in 11.3 Command Line Interface $GRID_HOME/bin/oclumon Collects CHM data using diagcollection.pl --collect --chmos .2.Cluster Health Monitor (CHM) • Cluster Health Monitor (CHM) monitors and collect OS and clusterware metrics in real-time • • • • Installed by default in 11.2 and 5 sec interval in

Uses OS API to collect metrics reducing overhead Clusterware resource called ora.2. instance evictions.2.Cluster Health Monitor (CHM) • • • • • Useful for troubleshooting root cause analysis .node reboots/hangs.2 version are incompatible. If you have 11.0.crf CHM doesn’t require RAC or Clusterware .0.2 then you cannot install OTN version. performance degradations etc OTN version of CHM and 11.

vmstat) • • • • Useful for diagnosing OS resource and performance problems.0 OS has been renamed to OS Watcher (OSWbb) UNIX shell scripts for monitoring the OS (ps. node reboots Should run on all nodes in a cluster Setup private interconnect monitoring Execute startOSWbb. mpstat.• • Watcher Black Box Black Box OS Watcher v4.sh arg1 arg2 where arg1=collection frequency and arg2=retention time . iostat. netstat. top.

OS Watcher Black Box • Bundled with OS Watcher Black Box Analyzer (OSWbba) Requires Java 1.4. disk Use CLI option to script profile generation for troubleshooting • • • • .2 or greater Correlate OS statistics using the analyzer profile Generates graphs and reports for memory. cpu.

OS Watcher Black Box .

OS Watcher Black Box OSWbb Free Memory Graph .

RACcheck – RAC Configuration Audit Tool • RACCHECK OUTPUT .

Clusterware and ASM Useful for pre-upgrade and post-upgrade system verification Uses “Best Practices” to report configuration problems – PASS/WARNING/FAIL/INFO • Generates detailed and summary reports with scorecard .RACcheck – RAC Configuration Audit Tool • • • Assess the configuration of RAC.

Remote Diagnostics Assistant • • • • The diagnostics tool recommended by MOS Collects a wealth of information based on configuration – OS/Clusterware/Database logs Runs AWR/Statspack report for Performance problems Generates reports in HTML format .

• • • • Procwatcher Debug Oracle & Clusterware processes using oradebug short_stack or OS debugger (e. instance evictions .g. pstack) Run as Oracle process owner to debug database or as root for clusterware processes Can be deployed as a Clusterware resource Useful for troubleshooting session hangs. gdb. severe performance problems.

1 Wed Feb 25 02:30:26 CDT 2012: if you have any comments./prw. suggestions.sh start all Wed Feb 25 02:30:26 CDT 2012: Starting Procwatcher Wed Feb 25 02:30:26 CDT 2012: Thank you for using Procwatcher. :-) Wed Feb 25 02:30:26 CDT 2012: Please add a comment to Oracle Support Note 459694. Wed Feb 25 02:30:26 CDT 2012: Started Procwatcher . or issues with this tool.Procwatcher grid@node1[+ASM1]-/u02 >.

ADRCI/Support Workbench • Automatic Diagnostic Repository (ADR) stores database diagnostic information • • • Package diagnostics files using ADRCI or Support Workbench Manages incidents and problems from alert logs Enterprise Manager provides GUI interface to ADR called Support Workbench .

ADRCI/Support Workbench .

RACDIAG.SQL • • • • Gathers debug information for RAC Session Hangs One-time data capture Performs hanganalyze dumps Certain types of hangs will prevent it from running .

OS Utilities • • • • truss/strace – trace system calls and signals pstack – dump stack trace for process pmap/procmap – maps process memory nmon/nmon analyzer – collects and analyzes OS stats collectl /collectl utils – collects and analyzes OS stats • .

Summary Tool/Utility diagcollection cluvfy Instance Evictions ✓ ✗ Node reboots ✓ ✗ Clusterware Problems ✓ ✓ RAC Performance ✗ ✗ CHM OSWbb/OSWbb a RDA RACcheck Procwatcher ADRCI/SW ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✗ ✓ ✓ .

1] Script to Collect RAC Diagnostic Information (racdiag.1] Data Gathering for Troubleshooting RAC Issues [ID 556679.1] Introducing Cluster Health Monitor (IPD/OS) [ID 736752.RAC Configuration Audit Tool [ID 1268927.1] OS Watcher Black Box Analyzer User Guide [ID 461053.1] CRS 10gR2/ 11gR1/ 11gR2 Diagnostic Collection Guide [ID 330358.1] Diagnosability for Oracle Clusterware (CRS or Grid Infrastructure) Component and Resource [ID 357808.sql) [ID 135714.1] Data Gathering for Troubleshooting Oracle Clusterware (CRS or GI) Issues [ID 289690.1] RACcheck .1] .MOS Notes • • • • • • • • • • • OS Watcher Black Box User Guide [ID 301137.1] Cluster Health Monitor (CHM) FAQ [ID 1328466.1] Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware Processes [ID 459694.

griddba.com LinkedIn – Leighton Nelson Twitter .@leight0nn Email: leighton.Contact Information • • • • Website .net .blogs.nelson@mercy.

You're Reading a Free Preview

/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->