RAC Troubleshooting This is the one section what will be updated frequently as my experience with RAC grows, as RAC

has been around for a while most problems can be resolve with a simple google lookup, but a basic understanding on where to look for the problem is required. In this section I will point you where to look for problems, every instance in the cluster has its own alert logs, which is where you would start to look. Alert logs contain startup and shutdown information, nodes joining and leaving the cluster, etc. Here is my complete alert log file of my two node RAC starting up. The cluster itself has a number of log files that can be examined to gain any insight of occurring problems, the table below describes the information that you may need of the CRS components $ORA_CRS_HOME/crs/log contains trace files for the CRS resources $ORA_CRS_HOME/crs/init contains trace files for the CRS daemon during startup, a good place to start $ORA_CRS_HOME/css/log

contains cluster reconfigurations, missed check-ins, connects and disconnects Look here to obtain when reboots occur

$ORA_CRS_HOME/css/init contains core dumps from the cluster synchronization service daemon (OCSd $ORA_CRS_HOME/evm/lo log files for the event volume manager and eventlogger daemon g $ORA_CRS_HOME/evm/ini pid and lock files for EVM t $ORA_CRS_HOME/srvm/lo log files for Oracle Cluster Registry (OCR) g $ORA_CRS_HOME/log

log files for Oracle clusterware which contains diagnostic messages at the Ora

As in a normal Oracle single instance environment, a RAC environment contains the standard RDBMS log files, these files are located by the parameter background_dest_dump. The most important of these are $ORACLE_BASE/admin/ud contains any trace file generated by a user process ump $ORACLE_BASE/admin/cd contains core files that are generated due to a core dump in a user process ump Now lets look at a two node startup and the sequence of events First you must check that the RAC environment is using the connect interconnect, this can be done by either of the following logfile ## The location of my alert log, yours may be different

picked_ksxpia. as seen in the logfile it is a seven step process (see below for more details on the seven step process). if the heartbeat information is missing for x amount of time. taken Reconfiguration started (old inc 2. 0 closed Set master node info . Messages are sent across the interconnect an amount of time then a communication failure is assumed by default UDP is used a an eye on the logs if too many reconfigurations happen for reason 3. invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Mar 20 11:35:53 2010 LMS 0: 0 GCS shadows cancelled. the dead and the Instance Membership Recovery (IMR) process initiates reconfiguration 3 means communication failure of a node/s. pub_ksxpia. The LMON trace file also has details about reconfigurations it also details the reason for the event reconfiguation reason description 1 means that the NM initiated the reconfiguration event. typical when a node joins or le means that an instance has died 2 How does the RAC detect an instance death. Remember when a node joins or leaves the cluster the GRD undergoes a reconfiguration event. List of nodes: 01 Global Resource Directory frozen * allocate domain 0. SQL> oradebug setmypid SQL> oradebug ipc oradebug Note: check the trace file which can be located by the parameter user_dump_d cluster_interconnects system parameter Note: used to specify which address to use When the instance starts up the Lock Monitor's (LMON) job is to register with the Node Monitor (NM) (see below table). every instance updates the control file w checkpoint (CKPT)./u01/app/oracle/admin/racdb/bdump/alert_racdb1.log ifcfg command table check oifcfg getif select inst_id. ip_ksxpia from x$ksxpia. Example of a Sat Mar 20 11:35:53 2010 reconfiguration. new inc 4) from the alert log.

the system will have to wait until all nodes have seen the commit SCN. a broadcast method is used after a commit operation. the log file will state the following cluster mode Sat Mar 20 11:36:02 2010 Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE) Completed: ALTER DATABASE MOUNT Staring with 10g the SCN is broadcast across all nodes. this feature can only be used in a Unix environment (no windows option). Lamport Algorithm The lamport algorithm generates SCNs in parallel and they are assigned to transaction on a first come first served basis. this is different than a single instance environment.Submitted all remote-enqueue requests Dwn-cvts replayed. this method is more CPU intensive as it has to broadcast the SCN for every commit. When set to less than 100 the broadcast on commit algorithm is used. Disable/Enable Oracle RAC There are times when you may wish to disable RAC. VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Sat Mar 20 11:35:53 2010 LMS 0: 0 GCS shadows traversed. 3291 replayed Sat Mar 20 11:35:53 2010 Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Note: when a reconfiguration happens the GRD is frozen until the reconfiguration is Confirm that the database has been started in cluster mode. You can change the board cast method using the system parameter _lgwr_async_broadcasts. but he other nodes can see the committed SCN immediately. Disable Oracle RAC (Unix only) Log in as Oracle in all nodes shutdown all instances using either normal or immediate option change to the working directory $ORACLE_HOME/lib . by default it is 7 seconds. The initialization parameter max_commit_propagation_delay limits the maximum delay allow for SCN propagation.

the trace files will be very large. .mk ioracle Enable Oracle RAC (Unix only) Log in as Oracle in all nodes shutdown all instances using either normal or immediate option change to the working directory $ORACLE_HOME/lib run the below make command to relink the Oracle binaries without the RAC option (should take a few mi make -f ins_rdbms. it is unable to automatically detect and resolve the deadlock. When this event occurs you must obtain dumps from each of the instances (3 dumps per instance in regular times). thus hanging the database. # using oradebug SQL> select * from dual.mk rac_off Now relink the Oracle binaries make -f ins_rdbms. however if the situation occurs with the internal kernel-level resources (latches or pins). capture information ## Using alter session SQL> alter session set max_dump_file_size = unlimited.run the below make command to relink the Oracle binaries without the RAC option (should take a few mi make -f ins_rdbms. usually Oracle will detect the deadlock and rollback one of the processes. SQL> alter session set events 'immediate trace name systemstate level 10'.mk ioracle Performance Issues Oracle can suffer a number of different performance problems and can be categorized by the following • • • • Hung Database Hung Session(s) Overall instance/database performance Query Performance A hung database is basically an internal deadlock between to processes.mk rac_on Now relink the Oracle binaries make -f ins_rdbms.

a systemstate dump is normally used to analyze this problem. SQL> oradebug hanganalyze <level> ## Another way using oradebug SQL> setmypid SQL> setinst all SQL> oradebug -g def hanganalyze <level> Note: you will be told where the output will be dumped to hanganalyze levels 1-2 3 oradebug only hanganalyze output. it also has a number of limitations • • • • Reads the SGA in a dirty manner. no process dump at all. sql method alter session set events 'immediate trace hanganalyze level <level>'.problems connecting ## If you get problems connecting with SQLPLUS use the command below $ sqlplus -prelim Enter user-name: / as sysdba A severe performance problem can be mistaken for a hang. To overcome these limitations a new utility command was released with 8i called hanganalyze which provides clusterwide information in a RAC environment on a single shot. SQL> oradebug setmypid SQL> unlimit SQL> oradebug -g all dump systemstate 10 Note: the select statement above is to avoid problems on pre 8 Oracle SQLPlus .SQL> oradebug setmypid SQL> unlimit SQL> oradebug dump systemstate 10 # using oradebug from another instance SQL> select * from dual. this usually happen because of contention problems. click here for an example level 1 du Level 2 + Dump only processes thought to be in a hang (IN_HANG state) . however a systemstate dump taken a long time to complete. so it may be inconsistent Usually dumps a lot of information does not identify interesting processes on which to perform additional dumps can be a very expensive operation if you have a large SGA.

thus intra-instance coordination is not required. When a communication failure happens the heartbeat information in the control cannot happen. this generally happens when you have communication problems. the instance that obtains the lock tallies the votes from all members. IMR will remove any nodes from the cluster that it deems as a problem. the group membership must conform to the decided (voted) membership before allowing the GCS/GES reconfiguration to proceed. I have already discussed the voting disk in my architecture section. The Node Monitor (NM) provides information about nodes and their health by registering and communicating with the Cluster Manager (CM). To understand eviction problems you need to now the basics of node membership and instance membership recovery (IMR) works. this works at the cluster level and can work with 3rd party software (Sun Cluster. Node membership is represented as a bitmap in the GRD. for example if a node joins or leaves the cluster. as stated above memberships is held in a bitmap in the GRD. LMON will let other nodes know of any changes in membership. You can see the controlfile records using the gv$controlfile_record_section view. For eviction node problems look for ora-29740 errors in the alert log file and LMON trace files. systemdump is better but if you over whelmed try hanganalyze first.instance id 1 (internal mem no 0) One thing to remember is that all nodes must be able to read from and write to the controlfile. LMON handles many of the CGS functionalities. the CKPT process updates the controlfile every 3 seconds in an operation known as a heartbeat. It writes into a single block that is unique for each instance. IGN_DMP Level 4 + Dump all processes involved in wait chains (NLEAF state) Dump all processes (IGN state) The hanganalyze command uses internal kernel calls to determine whether a session is waiting for a resource and reports the relationship between blockers and waiters. the bitmap is rebuilt and communicated to all nodes. it uses a voting mechanism to check the validity of each member. Veritas Cluster). CGS makes sure that members are valid. this block is called the checkpoint progress record.4 5 10 Level 3 + Dump leaf nodes (blockers) in wait chains (LEAF. LEAF_NW. IMR will ensure that the larger part of the cluster will survive and kills any remaining nodes. IMR is part of the service offered by Cluster Group Services (CGS). A cluster reconfiguration is performed using 7 steps Name service is frozen. Node registering (alert log) lmon registered with NM . the controlfile vote result is stored in the same block as the heartbeat in the control file checkpoint progress record. thus data corruption can happen. the CGS contains an internal database of all the . Debugging Node Eviction A node is evicted from the cluster after it kills itself because it is not able to service the application. all members attempt to obtain a lock on the controlfile record for updating.

mgmt. instance name and uniqueness verification.ENABLED=true -DTRACING...LEVEL=2 the string should look like this exec $JRE -DTRACING.... GSD.members/instances in the cluster with all their configurations and servicing details. GCS must synchronize the cluster to be sure that all members get the reconfiguration event and that they all see the same bitmap. GSDCTL or SRVCONFIG.... this prevents processes from obtaining locks on resources that were mastered by the departing/dead instance Determination of membership and validation and IMR Bitmap rebuild takes place.sh/srvctl/srvconfig file in the $ORACLE_HOME/bin directory At the end of the file look for the below line exec $JRE -classpath $CLASSPATH oracle. Delete all dead instance entries and republish all names newly configured Unfreeze and release name service for use Hand over reconfiguration to GES/GCS Debugging CRS and GSD Oracle server management configuration tools include a diagnostic and tracing facility for verbose output for SRVCTL.daemon. set it to blank to remove the debugging Enable tracing Disable tracing $ export SRVM_TRACE=true $ export SRVM_TRACE="" .ops.. Lock database (IDLM) is frozen. In Oracle database 10g setting the below variable accomplishes the same thing.. To capture diagnose following the below use vi to edit the gsd.ENABLED=true -DTRACING.OPSMDaemon $MY_OHOME Add the following just before the -classpath in the exec $JRE line -DTRACING.LEVEL=2 -classpath.

Sign up to vote on this title
UsefulNot useful