8/12/22, 11:09 AM 19c - Oracle process memory leak : 네이버 블로그

Bug Report

19c - Oracle process memory leak

채니아빠 이웃추가
2020. 9. 11. 17:16

Oracle 19c ExadataX8oracle linux 7.7 / oracle 19.6.0​* 오라클 backgroup Process중 mmon_slave Process에 대한 메모리 과다 사용
에 대한 현상 ​Huge Page 영역과 Non-Huge Page영역의 모든 Instance에서 rss 증가됨.특히 Non-Huge Page영역의 DB는 1주일만
에 m00~m006 합계 15G 이상 증가됨.​=> Non-Huge  23:00~01:30 사이 ora_m00의 특정 프로세스에 대한 rss 값이 500M 이상 랜
덤하게 급증

--아래는 약 2~3개월간 약 35G 이상 RSS 값이 증가됨

root@exa8db:/root# ps -eo user,pid,ppid,rss,size,vsize,pmem,pcpu,time,cmd --sort -rss | head -n 21

USER        PID   PPID   RSS  SIZE    VSZ %MEM %CPU     TIME CMD

oracle    78869      1 7896616 2056100 11504328  2.0 0.1 01:46:05 ora_m000_TEXADB2           --------->

oracle   191453      1 7213468 1466680 10914524  1.8 0.1 01:22:59 ora_m003_TEXADB2

oracle    78579      1 6197248 40344 9495028  1.5 0.2 04:21:20 ora_mmon_TEXADB2

oracle    92493      1 6110912 844084 10276328  1.5 0.1 00:53:59 ora_m004_TEXADB2

oracle    78513      1 5484288 167620 9751148  1.3 0.2 04:29:46 ora_lms1_TEXADB2

oracle    78511      1 5446816 135220 9456748  1.3 0.2 04:19:15 ora_lms0_TEXADB2

oracle    78534      1 5283644 100856 9529028  1.3 0.0 00:12:38 ora_dbw1_TEXADB2

oracle    78532      1 5063776 92160 9258636  1.2 0.0 00:11:31 ora_dbw0_TEXADB2...... 생략

os 명령어

/bin/ps -eo user,pid,ppid,rss,size,vsize,pcpu,pmem,time,cmd --sort -rss | grep ora_m

1 0 각자도생-시즌2

https://m.blog.naver.com/k65fac/222086804310 1/14
8/12/22, 11:09 AM 19c - Oracle process memory leak : 네이버 블로그

pga 사용량

SELECT TO_CHAR(SYSDATE,'YYYY/MM/DD HH24:MI:SS') REG_DT,(SELECT INSTANCE_NAME FROM GV$INSTANCE WHERE INST_ID

=P.INST_ID AND ROWNUM=1) INST_NAME,

P.INST_ID, P.SPID, P.PID, S.SID, S.SERIAL#, P.PGA_ALLOC_MEM/1024/1024 PGA_ALLOC_MB, P.PGA_USED_MEM/1024/1024 PGA_USE

D_MB, S.PROGRAM

FROM GV$SESSION S, GV$PROCESS P

WHERE S.INST_ID=P.INST_ID

AND S.PADDR=P.ADDR

AND S.PROGRAM LIKE '%(M%'

AND P.BACKGROUND IS NOT NULL

분석을 위한 Dump 수행

SQL> oradebug setospid <m002 의 os pid>

SQL> oradebug dump pga_summary 1

m002 의 rss 가 급증한 이후 한번 더 dump

SQL> oradebug dump pga_summary 1

SQL> oradebug tracefile_name

===== Linux =====

Let's take pid : 18822 (ora_m001_boa2 ) as example to see their RSS usage:

1. it is clear that RSS increased from 1516828 to 2047340, total increased = 2047340 - 1516828 = 530512

F S RUSER PID PPID C PSR PRI NI ADDR RSS SZ WIDE-WCHAN-COLUMN STIME TT TIME CMD

ps_eo_20200902_2300.log:0 S oracle 18822 1 0 19 19 0 - 1516828 2304060 do_semtimedop Aug31 ? 00:03:54 ora_m001_boa2

ps_eo_20200903_0100.log:0
1 0 각자도생-시즌2
S oracle 18822 1 0 2 19 0 - 2047340 2306108 do_semtimedop Aug31 ? 00:04:30 ora_m001_boa2

https://m.blog.naver.com/k65fac/222086804310 2/14
8/12/22, 11:09 AM 19c - Oracle process memory leak : 네이버 블로그

2. By further checking, most of the RSS increased is caused by "shmid=0x170001e " (a shared memory segment),

$ cat pmap_xp_18822_20200902_2300.log | head

18822: ora_m001_boa2

Address Kbytes RSS Dirty Mode Mapping

0000000000400000 373008 110400 0 r-x-- /u01/app/oracle/product/19.0.0.0/dbhome_1/bin/oracle

0000000017243000 1444 816 8 r---- /u01/app/oracle/product/19.0.0.0/dbhome_1/bin/oracle

00000000173ac000 384 276 24 rw--- /u01/app/oracle/product/19.0.0.0/dbhome_1/bin/oracle

000000001740c000 328 208 208 rw--- [ anon ]

0000000017adc000 1092 996 996 rw--- [ anon ]

0000000060000000 4 0 0 r--s- [ shmid=0x16f801d ]

0000000060001000 8708 5264 5264 rw-s- [ shmid=0x16f801d ]

0000000061000000 8241152 1293104 1293076 rw-s- [ shmid=0x170001e ] <<<<<===== RSS 1293104

$ cat pmap_xp_18822_20200903_0100.log | head

18822: ora_m001_boa2

Address Kbytes RSS Dirty Mode Mapping

0000000000400000 373008 114048 0 r-x-- /u01/app/oracle/product/19.0.0.0/dbhome_1/bin/oracle

0000000017243000 1444 816 8 r---- /u01/app/oracle/product/19.0.0.0/dbhome_1/bin/oracle

00000000173ac000 384 276 24 rw--- /u01/app/oracle/product/19.0.0.0/dbhome_1/bin/oracle

000000001740c000 328 208 208 rw--- [ anon ]

0000000017adc000 1092 996 996 rw--- [ anon ]

0000000060000000 4 0 0 r--s- [ shmid=0x16f801d ]

0000000060001000 8708 5264 5264 rw-s- [ shmid=0x16f801d ]

0000000061000000 8241152 1803940 1803888 rw-s- [ shmid=0x170001e ] <<<<<===== RSS 1803940

total increased = 2047340 - 1516828 = 530512

shmid=0x170001e increased = 1803940 - 1293104 = 510836

So the 각자도생-시즌2
1 shared0 memory is the reason that caused most increasing RSS

https://m.blog.naver.com/k65fac/222086804310 3/14
8/12/22, 11:09 AM 19c - Oracle process memory leak : 네이버 블로그

===================

참고로, rss 는 아시다시피 다음을 의미합니다.

RSS resident set size, the non-swapped physical memory that a task has used (in kiloBytes). (alias rssize, rsz).​

=> OS상 특이사항 없음(shared memory 충분하며, 이슈사항 없음)

--------- Oracle Process에 대한 Memory Leak Bug -----------

p31840094_196000DBRU_Linux-x86-64.zip

참고로 해당 패치는 다음의 이슈들에 대한 merge 패치입니다.

++ BUG 31425761 - HIGH PGA CONSUMPTION/MEMORY LEAK BY MMON SLAVE

++ BUG 31695062 - PGA LEAK IN ADVISOR FRAMEWORK LAYER (KEA)

또한, Linux 엔지니어의 업데이트도 있습니다.

Here, the shared memory segment for shmid 24117278 is 8438939648 bytes ~ 7.85G

From the currently available data, we cannot see any OS issues.​

1 0 각자도생-시즌2

https://m.blog.naver.com/k65fac/222086804310 4/14
8/12/22, 11:09 AM 19c - Oracle process memory leak : 네이버 블로그

=> 패치후 ora_m 프로세스가 재기동 되면서 메모리를 반환함

*정말 중요한 내용은 RSS 값은 물리 메모리 할당값으로 생각하면 안됨.

*top, ps 등은 추이 분석에 적합, 실제 메모리 사용량을 보여주는 명령어는 없음.
pmap -x 명령을 사용해서 별도로 분석해야 되며,

RSS 이 값에는 커널 VM 추적 구조, 페이지 테이블의 물리적 페이지에 대한 유효한 매핑이있는 프로세스 주소 공간의 모든 가상 메
모리 페이지가 포함됩니다. 즉, RAM에도있는 프로세스 주소 공간에 매핑 된 모든 페이지입니다. / OS에서 반환을 요청하지 않아서
이미 할당된 영역도 포함됨
 - 아래 참조

실제 오라클 Process의 Private한 영역(일반적으로 오라클 Process가 사용하는 메모리 usage) 영역은 가장 pmax -x 결과의 anon 영
역으로 판단됨.
=> v$sysstat.pga 영역, v$process.pga_ 컬럼을 참고 

RSS 값에 대한 증가는 Instance 구성환경에 따라 Small page와 Large Page간 차이가 있다

결론 : RSS 증가에 대한 SR 진행은 Memory Leak 관련 Bug 확인되었고, 패치 진행후 더 이상 DB 내 PGA 메모리 증가 현상은 없어
졌으나.
1 각자도생-시즌2
OS상 0RSS 값에 대한 증가 자체가 문제가 될수 없다는 설명이 부족해서 아쉽다.

https://m.blog.naver.com/k65fac/222086804310 5/14
8/12/22, 11:09 AM 19c - Oracle process memory leak : 네이버 블로그

--------- 참고 ------------------------------------------------------------
Process size가 비정상적으로 보이는 이유

Oracle process가 사용하고 있는 memory가 이상하게 나타나는 두 가지 원인은 다음과 같

다 : a) non-private(shared) memory의 size가 total에 포함되어 있고, b) O/S가 free

memory를 신속하게 reclaim하지 않기 때문이다. 다음 절에서 이에 대한 자세한 설명을 하도

록 한다.a) Non-private memory size를 total size에 포함시킨다한 Oracle process의 memory는 다음과 같은 여러 요소들로 구성된
다:

 shared memory (SGA)

 shared libraries (public 및 private)  

 private data (DATA 또는 heap이라 칭한다)   

 the executable (TEXT라 칭한다) 

이 중 SGA와 TEXT는 모든 Oracle process들이 공유하는 영역이다. 이 영역들은 memory에

단 한 번 mapping된다 (process별로 mapping되는 것이 아니다!). 따라서 새로운 Oracle

process가 생성되더라도 SGA와 TEXT 영역은 새롭게 할당되는 것이 아니다! 그러나 'ps' 또

는 'top' 같은 utility들은 private memory 산정시 TEXT 영역을 포함시킨다. 또한 SGA 영역

까지 private memory 계산에 포함되는 경우가 있어, 한 server process가 차지하는 memory

가 비상식적인 수치로 표현된다. 일부 O/S는 보다 정확한 process size를 보여주는 툴을 제

공하기도 한다. 예를 들어 Solaris의 /usr/proc/bin/pmap, HP의 glance (process의

virtual memory map을 보여준다) 또는 kernel debugger q4 등이 있다. -----------------------------------------------------------------

--------------------

http://tech.e2sn.com/oracle/performance/unix-performance-tools/process-memory-matrix

Using Process Memory Matrix script for understanding Oracle process memory usage - tech.E2SN
E2SN Tech Corner
tech.e2sn.com

1 0 각자도생-시즌2

https://m.blog.naver.com/k65fac/222086804310 6/14
8/12/22, 11:09 AM 19c - Oracle process memory leak : 네이버 블로그

Using Process Memory Matrix script for understanding Oracle process memory usage
This tool currently works only on Solaris. I will write support for (newer) Linux versions soon and possibly also for HP-UX.

When working on a problem I wrote a script which helps to present the output of Solaris pmap in a better way. If you don't know what pmap is, it's
which displays you the breakdown of each processes address space - virtual memory mappings. This is way better than just relying on ps or top com

My script gives a better overview of how much memory an Oracle process is really using. Historically this has been problematic due various differenc
large amount of data returned from pmap command.

My script procmm.sh (Process Memory Matrix) will simply run pmap on the processes specified and aggregate the output into a matrix.

Example output is here:

oracle@solaris02:~/research/memory$ ./procmm.sh 12755

-- procmm.sh: Process Memory Matrix v1.01 by Tanel Poder ( http://tech.e2sn.com )

-- All numbers are shown in kilobytes

PID            SEGMENT_TYPE      VIRTUAL          RSS         ANON       LOCKED    SWAP_RSVD

------ -------------------- ------------ ------------ ------------ ------------ ------------

12755                   lib        22932        22872          900            0         1048

12755                oracle        95824        95820          232            0         2468

12755        ism_shmid=0x1d       409608       409608            0       409608            0

12755                  anon         5728         5512         5508            0         5724

12755                 stack          156          156          156            0          156

12755                  heap         1924          868          868            0         1924

------ -------------------- ------------ ------------ ------------ ------------ ------------

12755             TOTAL(kB)       536172       534836         7664       409608        11320

On the Horizontal axis you see the various memory sizes pmap reports (like VIRTUAL size and SWAP_RSVD - swap space reservation) and on vertica
dress these memory figures are shown (for example, Oracle binary, heap memory (think malloc()), libraries and process private memory allocations w

I don't do any of my own computation magic here, I just show output from couple pmap commands in a better aggregated and understandable ma
ust run man pmap.

1 0 각자도생-시즌2

https://m.blog.naver.com/k65fac/222086804310 7/14
8/12/22, 11:09 AM 19c - Oracle process memory leak : 네이버 블로그

I will explain the columns shortly here too:

 SEGMENT_TYPE  Shows what uses that amount of memory reported.

lib - both Unix system and and application (Oracle) libraries

oracle - the oracle executable (or any other executable if examiing some other proc
ess)
ism_shmid=0x1d - shmid shows that we are dealing with the good old System V sh
ared memory segment. The 0x1d here corresponds to shared memory segment's SH
M ID seen from ipcs -ma output (if you convert the number to decimal).

ism prefix shows that we are dealing with Solaris-specific Intimate Shared Memory (I
SM). dism stands for dynamic intimate shared memory (DISM).

anon - anonymous memory deserves its own page, but shortly said this is process p
rivate memory. For example since version 9i, Oracle can allocate PGA/UGA/CGA me
mory by mapping /dev/zero into its process address space using mmap() system call
instead of calling malloc()/brk() which just extends the process data segment (heap).
The reason this memory segment type is called anonymous is that this memory doe
sn't correspond to any real (named) file on disk. When you load an executable, librar
y or some datafile into memory, then their in-memory pages correspond to their on
-filesystem files, thus are not anonymous. Process private allocations don't correspon
d to any files, thus are called anonymous pages.
stack - this is the relativaly small memory area (few kilobytes or few tens of kB) pres
ent in each thread where some function-local scope variables are kept (as determine
d by compiler) and also the return path to parent functions is remembered there. Th
is information is used for resuming the parent function once the child functions it ca
lls finish their work and return. Stack tracing is extremely useful for troulbleshooting
and will have its own page soon.
heap - this is the memory "application scratch area", called data segment, which is u
sed when you allocate memory with conventional malloc() calls. However Oracle doe
s not use the heap for PGA/UGA's anymore thanks to the realfree heap allocation w
hich was introduced in 9i (and enabled by default in 10g).

 VIRTUAL  This shows the total virtual address space (reservation) size in kB. Reserving virtual
address space does not actually allocate corresponding physical memory from anyw
here (OS'es are lazy in that sense, they only will allocate physical pages of RAM for y
ou if you actually start touching these virtual pages. This is when the page fault mec
hanism will find a physical page of RAM and map it together with given process'es v
1 0 irtual memory page.
각자도생-시즌2

https://m.blog.naver.com/k65fac/222086804310 8/14
8/12/22, 11:09 AM 19c - Oracle process memory leak : 네이버 블로그

Note that some OS'es (Solaris, HP-UX) do automatically reserve swap space for such
virtual memory segments which may need paging out in the future (writable segme
nts). AIX and Linux (by default) will not try to reserve the swap space during virtual
memory segment set-up, but that raises a risk that when running out of memory th
en some victim process will get an error signal or exception in random location in th
e code and will not be able to gracefully handle this situation.

It is important to remember that VIRTUAL memory does not equal to physical mem
ory. Physical pages are allocated and mapped to the virtual segment only when the
process actually tries to use (touch) these memory pages.

 RSS  This shows process Resident Set Size in kB. This value includes all virtual memory p
ages in the process address space which have a valid mapping to a physical page in
kernel VM tracking stuctures, pagetables. In other words, all pages mapped to the  p
rocess address space which also happen to be in RAM.

This is why with Oracle processes you can sometimes see that when a process starts
up and attaches to SGA, it's virtual size is let say 10G (assuming ~10GB SGA), but th
e RSS is only a few hundred megabytes. This is because that process has not yet tou
ched all the SGA pages (which are in physical RAM), thus all the virtual-physical pag
e mappings for that process haven't been done and won't show up in RSS.

This is specific to your instance configuration btw (large vs small pages, ISM or not
etc).

 ANON  This is the key metric in procmm output. It shows in kB how much actual memory
has been allocated for that process (not just reserved, but actually used, allocated).
Note that this figure counts two things:

Memory allocated for that process and currently in RAM (RSS)

Memory allocated for that process and currently not in RAM (paged out)

Now you might ask, why do we see cases in above output where RSS is much larger
than ANON? For example the "oracle" file mapping. The answer is that RSS shows b
oth the resident named-file (corresponding to the file "oracle") contents and also th
e private anonymous memory pages which don't correspond to any named file (thus
are "anonymoys"), but just to the process.

1 0 각자도생-시즌2

https://m.blog.naver.com/k65fac/222086804310 9/14
8/12/22, 11:09 AM 19c - Oracle process memory leak : 네이버 블로그

So, ANON is the figure which allows to answer the question of "How much memory
are the Oracle processes really using?".

Note that there are many subtleties involved here and neither pmap nor my procm
m.sh script are perfect, but on Solaris reading ANON is a quick and decent way for
understanding Oracle memory usage.

 LOCKED  This shows (in kB) how much memory in the process address space is also locked t
o physical RAM. Intimate Shared Memory pages are always automatically locked to
memory, thus can't ever be paged out (non-pageability makes implementation of sh
ared pagetables easier). Other types of pages (Dynamic ISM for example) can be loc
ked to memory using mlock() system calls - if the program has root access. This is w
hy the oradism process exists in $ORACLE_HOME/bin

 SWAP_RSVD  On Solaris this shows how much memory (in kB) has been reserved in swapfs (whic
h is the on-disk swap area + virtual swap cache in RAM). Reserving swap does not
mean it's actually used and filled with some data, it merely means that Solaris adjust
s its "free swap space left" counters so that it would be quaranteed that this memor
y could be swapped out in case there's need for that.

In the above example I examined only a single process. Below I pass all processes of an instance as a parameter and procmm walks through them. T

The -t option below means "Total", the script doesn't show individual PID memory breakdown but sum of all PIDs passed to it.

oracle@solaris02:~/research/memory$ ./procmm.sh -t `pgrep -f ora_.*SOL102`

-- procmm.sh: Process Memory Matrix v1.01 by Tanel Poder ( http://tech.e2sn.com )

-- All numbers are shown in kilobytes

Total PIDs 17, working: .................

PID            SEGMENT_TYPE      VIRTUAL          RSS         ANON       LOCKED    SWAP_RSVD

------ -------------------- ------------ ------------ ------------ ------------ ------------

0                      
1 0 각자도생-시즌2
lib       389844       388796        13180            0        17816

https://m.blog.naver.com/k65fac/222086804310 10/14
8/12/22, 11:09 AM 19c - Oracle process memory leak : 네이버 블로그

0                    oracle      1629064      1628908         3336            0        42012

0            ism_shmid=0x1d      6963336      6963336            0      6963336            0

0             hc_SOL102.dat           48           48            0            0            0

0                      anon        32936        15936        15452            0        32868

0                     stack         1660         1628         1592            0         1660

0                      heap        37004        18016        16844            0        37004

------ -------------------- ------------ ------------ ------------ ------------ ------------

0                 TOTAL(kB)      9053892      9016668        50404      6963336       131360

-- Note that in Total (-t) calculation mode it makes sense to look into ANON and SWAP_RSVD

-- totals only as other numbers may be heavily "doublecounted" due overlaps of shared mappings

The ANON figure reports roughly 50404 kB as the Oracle instance processes actual memory usage (I should be more precise and say memory alloca

However the total 50404 kB also includes 13180 kB of ANON memory allocated by various libraries (in addition to Oracle libraries also multiple OS l
ritable) ANON memory has been allocated "in" oracle binary. This is because the BSS section in the binary which holds various static global variables
er object modules go to BSS section).

Every new Oracle process reuses the shared Oracle binary pages (only one copy of Oracle binary is in memory), but when the process tries to write
cal page and maps the new page to process address space as a writable page. That's called copy on write.

To Be Continued... 

#oracle #bug #memory #leak #exadata #19c

첨부파일

MEM_LOG 쉘.TXT

1 0 각자도생-시즌2

https://m.blog.naver.com/k65fac/222086804310 11/14
8/12/22, 11:09 AM 19c - Oracle process memory leak : 네이버 블로그

1 0

채니아빠
IT·컴퓨터

따뜻한 말 잘하자

이웃추가

이 블로그 Bug Report 카테고리 글

19.13 GI RU / Cluster Nodes will not Start Automatically After a Node Eviction (Doc ID 2821641.1)
2021. 11. 28.

19c-DBRU 19.13 undefined reference to `jox_eujs_nowait_'

2021. 11. 25.

jdbc 11.2.0.3 이하버전 에서 11.2.0.4 업그레이드 시 date format 이슈

2021. 11. 24.

[12c] The value for NEXT_CHANGE# column in V$LOG is changed in 12.2

2021. 11. 23.

19c - Oracle process memory leak 각자도생-시즌2

1 0
2020. 9. 11.
https://m.blog.naver.com/k65fac/222086804310 12/14
8/12/22, 11:09 AM 19c - Oracle process memory leak : 네이버 블로그
1
0

이 블로그의 #oracle 다른 글

19c - 33123985: DBW PROCESS GENERATE HUGE TRACES WITH DUMPING DBWR PROCESS STATE AFTER DBRU 19.11
2022. 3. 31.
0

Oracle ASM Storage Limits

2022. 3. 14.
0

set_column_stats
2022. 3. 13.
0

Oracle XML DB 11g에서 복잡한 XML 데이터 관리하기

2022. 8. 6.
0

Bug 1291810780 Force Disabling Internal Heatmap Feature각자도생-시즌2

Even When Heatmap Parameter is Off
https://m.blog.naver.com/k65fac/222086804310 13/14
8/12/22, 11:09 AM 19c - Oracle process memory leak : 네이버 블로그
2022. 8. 5.
0

맨 위로

PC버전으로 보기

1 0

https://m.blog.naver.com/k65fac/222086804310 14/14