25/4/23, 15:25 Document 2159643.

1
  Pedro (Not Available) (0) Contact Us Help
PoLast Login: April 25, 2023 12:31 PM CLST Switch to Cloud Support

 
Dashboard Knowledge Service Requests Patches & Updates

Give Feedback...
Copyright (c) 2023, Oracle. All rights reserved. Oracle Confidential.

Alert - Solaris: Process spins/ASM and DB Crash if RAC Instance Is Up For > 248 Days by LMHB with ORA-29770 To Bottom
(Doc ID 2159643.1)

In this Document   Was this document helpful?  

Description Yes
No
Occurrence    

Symptoms
  Document Details  
Workaround
Patches
Type: ALERT
History Status: PUBLISHED
References Last Major May 4, 2021
Update: Jun 4, 2022
Last Update:
English
Language:
APPLIES TO:    

Oracle Database Cloud Exadata Service - Version N/A and later   Related Products  
Oracle Database Cloud Service - Version N/A and later
Oracle Database Cloud
Gen 1 Exadata Cloud at Customer (Oracle Exadata Database Cloud Machine) - Version N/A and later Exadata Service
Oracle Cloud Infrastructure - Database Service - Version N/A and later Oracle Database Cloud Service
Oracle Database - Enterprise Edition - Version 10.2.0.4 to 12.1.0.2 [Release 10.2 to 12.1] Gen 1 Exadata Cloud at
Oracle Solaris on SPARC (64-bit) Customer (Oracle Exadata
Database Cloud Machine)
Oracle Cloud Infrastructure -
DESCRIPTION Database Service
Oracle Database - Enterprise
Edition
ASM or DB processes may start spinning in a RAC environment after the instance has been running continuously for more than Show More
248 days. This issue only affects Solaris platforms and is due to a faulty C compiler optimization.
   

  Information Centers  
The same problem can also affect non-RAC/ASM sessions (in particular if SQLNET.EXPIRE_TIME is used but this is not a
requirement to hit the problem) Information Center: Oracle
Cloud Infrastructure (OCI) &
Platform as a Service (PaaS)
OCCURRENCE Overview [2048297.2]

Index of Oracle Database

Only affects Solaris SPARC with Oracle RAC/non-RAC with ASM Information Centers
[1568043.2]

SYMPTOMS Information Center: OCI

Networking - Email Delivery
[2886428.2]
ASM or DB processes may start spinning in a RAC environment after the instance has been running continuously for more than
Information Center: OCI Cloud
248 days. This issue only affects Solaris platforms and is due to a faulty C compiler optimization. The same problem can also Guard [2765346.2]
affect non RAC / ASM sessions (in particular if SQLNET.EXPIRE_TIME is used but this is not a requirement to hit the problem).
Information Center: OCI
The RAC instance (DB or ASM) has been up for more than 248 days continuously with no shutdown. Container Instances
[2922704.2]
The spinning processes show stacks similar to: sslssalck <- sskgxp_alarm_set <- skgxp_setalarm() <- sslsstehdlr() <-
__sighndlr() <- call_user_handler() <- __pollsys() <- _pollsys() ... Show More
Instance crashes can occur following process/es starting to spin due to various timeouts and blocked resources and so
   

the symptoms of a crash due to this issue can vary.

  Document References  
On Non-RAC/ASM User sessions are seen to spin with a stack like above especially if SQLNET.EXPIRE_TIME is set in the
server side SQLNET.ORA ( Note: It is NOT a requirement to have EXPIRE_TIME set to encounter this bug issue, but if that Bug 18740837 - Solaris:
Dispatcher (or other) process
is set then the issue may be more visible). can hang/spin - superseded on
some versions [18740837.8]
Example traces from 11gR2 on a two node SPARC (64-bit) cluster.
Bug 10194190 - Solaris:
LMHB terminated one of the instances. Process spin and/or ASM and
DB crash if RAC instance up
for > 248 days [10194190.8]
The following errors were reported in the Alert log:
Bug 22901797 - Solaris:
Errors in file <ORACLE_BASE>/diag/rdbms/<dbname>/<sid>/trace/<sid>_lmhb_29680.trc LMHB: Terminating the
instance due to error ORA-
(incident=144129):
29770 [22901797.8]
ORA-29770: global enqueue process LMON (OSID 29660) is hung for more than 70 seconds
ERROR: Some process(s) is not making progress.    

LMHB (ospid: 29680) is terminating the instance.

Please check LMHB trace file for more details.   Recently Viewed  
Please also check the CPU load, I/O load and other system properties for anomalous behavior
Bug 30336742 - Hit ORA-
ERROR: Some process(s) is not making progress. 29770: global enqueue
LMHB (ospid: 29680): terminating the instance due to error 29770 process LMON is hung When
Opening PDBs [30336742.8]

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=221131923352674&id=2159643.1&_afrWindowMode=0&_adf.ctrl-state=127drml… 1/3
25/4/23, 15:25 Document 2159643.1
 For each LM BG processes (LMON, LMD0, LMS0, LMS1 and LCK0 in this case), the trace file shows information similar to the Bug 27254851 - INSTANCE
following: CRASHED WITH ORA-29770
DUE TO LMON HAS NOT
HEARTBEAT [27254851.8]
*** 2012-06-17 02:06:31.607 ORA-29770: global enqueue
============================== process LMS0 (OSID 11912)
LMON (ospid: 29660) has not moved for 30 sec (1339891590.1339891560) is hung for more than 70
: waiting for event 'rdbms ipc message' for 25 secs with wait_id 3381334669. seconds [2217764.1]
===[ Wait Chain ]=== OERR: ORA-29770 "global
Wait chain is empty. enqueue process %s (OSID
%s) is hung for more than
*** 2012-06-17 02:06:36.617 %s seconds" Reference Note
============================== [2010718.1]
LMD0 (ospid: 29662) has not moved for 32 sec (1339891595.1339891563) Alert - Solaris: Process
: waiting for event 'ges remote message' for 32 secs with wait_id 2766541183. spins/ASM and DB Crash if
===[ Wait Chain ]=== RAC Instance Is Up For >
248 Days by LMHB with
Wait chain is empty.
ORA-29770 [2159643.1]
Show More
   

Note that in the trace output, the wait_id related to each of the BG processes is not changing throughout the LMHB trace file .

Hence in this example, all LMON 'waiting for event' reports in the trace file reflect the same wait_id (3381334669 in this
example)

Non-RAC ASM instance crashed after the following error message 

Sat Dec 28 15:53:27 2013

NOTE: ASM client db0:db died unexpectedly.
NOTE: Process state recorded in trace file
<ORACLE_BASE>/diag/asm/+asm/+ASM0/trace/+ASM0_ora_295.trc
Sat Dec 28 15:54:06 2013
Errors in file <ORACLE_BASE>/diag/asm/+asm/+ASM0/trace/+ASM0_pmon_28911.trc:
ORA-00490: PSP process terminated with error
PMON (ospid: 28911): terminating the instance due to error 490
Sat Dec 28 15:54:09 2013
ORA-1092 : opitsk aborting process
Sat Dec 28 15:54:09 2013
License high water mark = 6
Instance terminated by PMON, pid = 28911
USER (ospid: 3728): terminating the instance
Instance terminated by USER, pid = 3728
Sat Dec 28 15:54:15 2013
Starting ORACLE instance (normal)

 Instance up for about 248 days(last startup time 2013-04-24 and crashed time 2013-12-28).

Wed Apr 24 02:36:39 2013

Starting ORACLE instance (normal)
....

Sat Dec 28 15:54:06 2013

Errors in file <oracle_base>/diag/asm/+asm/+ASM0/trace/+ASM0_pmon_28911.trc:
ORA-00490: PSP process terminated with error
PMON (ospid: 28911): terminating the instance due to error 490

WORKAROUND

As a workaround restart the instance every less than 248 days or whenever you hit this problem.

PATCHES

The patch 18740837 has the fix for the Bug 10194190 and is a more complete fix, so applying the patch 18740837 is
preferred. However, note that applying the patch 10194190 would still resolve most of the issues associated with the instance
running more than 248 days on 12.1.0.1 and earlier versions.

For 12.1.0.2, the fix for Bug 22901797 is required. Apply the 12.1.0.2.181016 (Oct 2018) Database Patch Set
Update (DB PSU) or 12.1.0.2.160419 (Apr 2016) Database Proactive Bundle Patch (DBBP) or later DBPSU/DBBP
where the bug has been fixed or request/apply one-off patch for Bug 22901797.

NOTE: The fix for Bug 18740837 included with 12.1.0.2 is incomplete and does not fix the issue properly, therefore
Bug 22901797 fix is required.

For 12.1.0.1 and earlier versions, request/apply patch for Bug 18740837.  

For 11.2.0.4 it is recommended to apply the latest Database PSU and apply one-off Patch 31309788 on top of it
(up until 11.2.0.4.210420 Apr 2021 PSU where bug 31309788 is fixed). Bug fix 22901797 was first included in the

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=221131923352674&id=2159643.1&_afrWindowMode=0&_adf.ctrl-state=127drml… 2/3
25/4/23, 15:25 Document 2159643.1
Database PSU 11.2.0.4.181016 (Oct 2018) and later. However, the patch files for bug fix 18740837 are missing starting in the
Database PSU 11.2.0.4.181016 (Oct 2018) due to internal Bug 31309788 (bug fix 31309788 changes are first included in
Database PSU 11.2.0.4.210420 (Apr 2021) and later PSU's).
If applying 11.2.0.4.210420 (Apr 2021) or later, the one-off Patch 31309788 is already fixed so the one-off would no longer be
required.

NOTE: 11.2.0.4.210119 and later PSU's require Market Driven Support (MDS) license to download, for further on MDS see
Doc ID 2728619.1.

HISTORY

07/11/2016 - note created

08/05/2016 - first published

10/19/2017 - Updated the complete fix 22901797

05/17/2018 - Updated to clarify that fix 22901797 is for 12.1 only because 12.1 is missing the fix for the patch 18740837 that
was intended to be in 12.1.  For 11.2, need to apply the patch 18740837

05/18/2018 - Updated the title to include alert to reflect that this MOS note is an alert type MOS note

10/10/2019 - Updated to clarify the fix 18740837 is superceded by 22901797, so fix 22901797 is required on 12.1.0.2 versions

01/29/2021 - Added section for 11.2.0.4 patching

04/23/2021 - Updated 11.2.0.4 to accurately reflect the missing fix 18740837 in 11.2.0.4 and the requirements to fix this issue

REFERENCES

NOTE:18740837.8 - Bug 18740837 - Solaris: Dispatcher (or other) process can hang/spin - superseded on some versions

NOTE:10194190.8 - Bug 10194190 - Solaris: Process spin and/or ASM and DB crash if RAC instance up for > 248 days
NOTE:22901797.8 - Bug 22901797 - Solaris: LMHB: Terminating the instance due to error ORA-29770
BUG:10194190 - DB AND ASM INSTANCE CRASH - ASM PROCESSES SPINNING
Didn't find what you are looking for? Ask in Community...

Related
Products
Oracle Cloud > Oracle Platform Cloud > Oracle Database Cloud Exadata Service > Oracle Database Cloud Exadata Service
Oracle Cloud > Oracle Platform Cloud > Oracle Database Cloud Service > Oracle Database Cloud Service
Oracle Cloud > Oracle Infrastructure Cloud > Oracle Cloud at Customer > Gen 1 Exadata Cloud at Customer (Oracle Exadata Database Cloud Machine)
Oracle Cloud > Oracle Platform Cloud > Oracle Cloud Infrastructure - Database Service > Oracle Cloud Infrastructure - Database Service
Oracle Database Products > Oracle Database Suite > Oracle Database > Oracle Database - Enterprise Edition > RDBMS > ORA-00445, (PMON, SMON, VKTM ...etc), Process
Termination, Process Spawn
Oracle Database Products > Oracle Database Suite > Oracle Database > Oracle Database - Enterprise Edition > Real Application Cluster > RAC DB Crash (ORA-29740, Instance
Eviction)
Oracle Cloud > Oracle Platform Cloud > Oracle Database Cloud Service > Oracle Database Cloud Schema Service
Oracle Cloud > Oracle Platform Cloud > Oracle Database Cloud Service > Oracle Database Exadata Express Cloud Service
Oracle Cloud > Oracle Platform Cloud > Oracle Database Backup Service > Oracle Database Backup Service

Keywords
BUG; PATCH; SOLARIS

Errors
ORA-00490; ORA-29770
Translations
English Source Japanese 日本語

Back to Top
 
Copyright (c) 2023, Oracle. All rights reserved. Legal Notices and Terms of Use Privacy Statement
   

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=221131923352674&id=2159643.1&_afrWindowMode=0&_adf.ctrl-state=127drml… 3/3