This action might not be possible to undo. Are you sure you want to continue?
Many times Oracle DBA’s are in a situation when the database is hung and does not seem to be responding. In some scenarios, state is such that you cannot even connect to the sqlplus session. Majority people restart the database (Sometimes I wonder if this is due to the fact that most of us started working on Computer’s on Microsoft Windows ) and then log a ticket with Oracle support . They inturn happily inform us that “They do not have any diagnostic information to diagnose and resolve the issue and we need to wait for next occurrence to collect some diagnostic information) Based on my experiences , I am writing this article to assist my fellow Oracle DBA’s to diagnose the problem and collect the required information. So Let’s Start. 1) First of all we need to ensure that this is really a database hung situation and not a slow database condition. This can be done by asking some questions to users. a) Is a particular user complaining of database hang or its the condition for all the users. If one or few user are reporting then are these users executing a batch job? b)Are you able to make new connections to database? c)Also check if any initialisation parameter has been changed recently? d)Check if any resource manager plan is in effect. One more way to establish if database is hung is t0 try to query v$session_wait view to find events being waited on select sid,event,seq#,p1,p2,p3 from V$session_wait where wait_time=0 and event not like ‘%message%’; This will give the events for all the waiting session. In case you see something like ‘log file switch (archiving required)’ then this problem is caused by archiving issue. See if there is free space in archiving destination. Suppose this gives events like row cache enqueue or latches, then we need to gather Hanganalyze andSystemstate for the support. Else it could be that you are experiencing a slow database. In this case use AWR or statspack to diagnose the issue. Look out for top timed events. In case you see Library Latch or shared pool latch consuming lot of time, then look at the Hard parses per sec section in Load profile. 2)Look at database alert log and see if any messages are present. In case you are facing Latching or Enqueueissues, then you might see errors like below PMON failed to acquire latch, see PMON dump Errors in file /u01/BDUMP/test10_pmon_12864.trc: In this case you are required to upload tracefile (reported in alert log) to Oracle support. Note : -Make sure that max_dump_file_size is set to unlimited so that the tracefile contains the complete data. Coming back to Hanganalyze and Systemstate. Find details for them below
A)Hanganalyze HANGANALYZE is used to determine if a session is waiting for a resource, and reports the relationships between blockers and waiters. Use following syntax and take hanganalyze from two sessions at interval of 1 min SQL>sqlplus “/ as sysdba” SQL> oradebug setmypid SQL> oradebug unlimit; SQL> oradebug hanganalyze 3 SQL>oradebug tracefile_name Last command will report the tracefile name which has to be uploaded to Oracle support. Alternatively you can use SQL>ALTER SESSION SET EVENTS ‘IMMEDIATE TRACE NAME HANGANALYZE LEVEL 3′; In case you wish to understand how to interpret hanganalyze file, then use Metalink Note:215858.1: Interpreting HANGANALYZE trace files to diagnose hanging and performance problems B)Systemstate Systemstate is used to dump the process information which is useful for Oracle support to diagnose why the sessions are waiting. For 22.214.171.124 and above gather systemstate as below SQL> oradebug setmypid SQL> oradebug unlimit; SQL> oradebug dump systemstate 266 SQL>oradebug tracefile_name Last command will report the tracefile name which has to be uploaded to Oracle support. Perform this 2-3 times at interval of 1 min. Again you can use ALTER SESSION SET EVENTS ‘IMMEDIATE TRACE NAME SYSTEMSTATE LEVEL 266′; For Oracle 126.96.36.199 and less use level 10 instead of 266 ALTER SESSION SET EVENTS ‘IMMEDIATE TRACE NAME SYSTEMSTATE LEVEL 266′; Level 266 includes short stacks (Oracle function calls) which are useful for Oracle Developers to determine which Oracle function’s are causing the problem. This is also helpful in matching existing bugs. In case you are unable to connect to database then capture systemstate using below note Note 121779.1-Taking a SYSTEMSTATE dump when you cannot CONNECT to Oracle. Apart from this, following information can also be captured
a)Database alert log b)AWR report /statspack report for 30-60 min during database hang c)Output of OS tools to ensure that everything is fine at OS level. E.g $vmstat 2 20 This will capture 20 snapshots at 2 seconds interval. Look for CPU contention or swapping issues In Addition to above , you can use utility called LTOM which has got some predefined rules based on which it determines that database is in Hung situation and takes systemstate and hanganalyze dump automatically . Please refer to following Metalink note for more details Note:352363.1: LTOM – The On-Board Monitor User Guide In case you are able to narrow down to a blocking session manually, then you can very well take errorstack for the blocking process as below connect / as sysdba oradebug setospid 1234 oradebug unlimit oradebug dump errorstack 3 wait 1 min oradebug dump errorstack 3 wait 1 min oradebug dump errorstack 3 oradebug tracefile_name * In case the ospid for blocking session is 1234 Last command will report the tracefile name which has to be uploaded to Oracle support. In case you are able to capture the above information, you stand 99% chance of getting solution . I have kept 1 % for the cases when Oracle Support will ask for setting up some events and waiting for Next Hang Occurence for getting more information.
RAC Hang Analysis and/or System State Dumps
First, login using the -prelim option of SQL*Plus as "/ as sysdba". Then run the following:
alter session set tracefile_identifier=’sysstatedump’; oradebug setmypid oradebug unlimit prompt Issuing command: oradebug -g all dump systemstate 10 oradebug -g all dump systemstate 10 prompt Sleeping for 1 minute !sleep 60 oradebug -g all dump systemstate 10 prompt We may not be able to do dump of locks prompt Issuing command: oradebug -g all dump locks 8 prompt If the sqlplus session hangs here for more than several minutes prompt kill it by hitting control-c oradebug -g all dump locks 8 prompt Sleeping for 1 minute !sleep 60 oradebug -g all dump locks 8
Getting a list of all of the background processes is also handy if you are a DBA and as a RAC DBA, you will come across some additional processes, which are available in your RAC environment. SQL> select name, description from v$bgprocess where PADDR <> '00'; PMON process cleanup DIAG diagnosibility process LMON global enqueue service monitor LMD0 global enqueue service daemon 0 LMS0 global cache service process 0 LMS1 global cache service process 1 MMAN Memory Manager DBW0 db writer process 0 LGWR Redo etc. LCK0 Lock Process 0 CKPT checkpoint SMON System Monitor Process RECO distributed recovery . . . . .
The additional RAC centric processes are DIAG, LCK, LMON, LMDn, and LMSn processes. We will give a brief description of each and discuss how they interact in a RAC environment next. DIAG: This is a diagnostic daemon. It constantly monitors the health of the instances across the RAC and possible failures on the RAC. There is one per instance. LCK: This lock process manages requests that are not cache-fusion requests. Requests like row cache requests and library cache requests. Only a single LCK process is allowed for each instance. LMD: The Lock Manager Daemon. This is also sometimes referred to as the GES (Global Enqueue Service) daemon since its job is to manage the global enqueue and global resource access. It also detects deadlocks and monitors lock conversion timeouts. LMON: The Lock Monitor Process. It is the GES monitor. It reconfigures the lock resources adding or removing nodes. LMON will generate a trace file every time a node reconfiguration takes place. It also monitors the RAC cluster wide and detects a node’s demise and trigger a quick reconfiguration. LMS: This is the Lock Manager Server Process or the LMS process, sometimes also called the GCS (Global Cache Services) process. Its primary job is to transport blocks across the nodes for cache-fusion requests. If there is a consistent-read request, the LMS process rolls back the block, makes a Consistent-Read image of the block and then ship this block across the HSI (High Speed Interconnect) to the process requesting from a remote node. LMS must also check constantly with the LMD background process (or our GES process) to get the lock requests placed by the LMD process. Up to 10 such processes can be generated dynamically.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.