Identifying memory pressure - Part 2

Continue from part 1....

OK, ALL THIS STUFF LOOKS GREAT, BUT HOW COULD I ESTIMATE THE SIZE OF A PROCESS IN MEMORY?

Taking into account what have been told, the Resident Set Size is the closest to the size in memory of a process, but when checking it by means of top, ps, or another Unix tool we find a problem. The output of those commands takes into account the shared memory segments and the shared libraries, so the results are oversized for double counting the sizes.

In Unix we have mainly two sources to retrieve the information about the memory map of a process, the proc filesystem and the command pmap.

With regards to Oracle processes, we'll find those kinds of memory segments: (Thanks Tanel Poder for this explanations!)

lib - ld : shared libraries, dynamic linker/loader. Unix System and Oracle libraries

oracle : the Oracle executable

shmid : System V shared memory segments (if using memory target parameter, we will see a lot of segments instead, because memory target uses POSIX implementation. Will comment later)

anon: anonymous memory. shortly said, process private memory. This memory does not correspond to any real file on disk. This is the figure we should consider to estimate the size ofthe process.

stack: a small area where some function-local scope variables are kept  and also the return of some path to parent functions is remembered there.

heap: this is the memory "application scratch area", called data segment, which is used when you allocate memory with conventional malloc() calls

Text :The actual program instructions. This section's contents are never modified during execution.

Data: Constant data that is not modified during execution. E.g. program constants, string literals etc. etc.

BSS : Statically allocated program variables etc. that are modified during program execution.

So, my approach has been this:

To get the estimate size in memory of a process, I will retrieve all the memory segments RSS in /proc/<pid>/smaps that are not shared and are writable.

Here I assume that, as they are writable, is less likely that they are shared and are particular to each process.

Here is the command I use:

 for i in `ps -fu oracle | grep -v grep | grep -v PID | awk '{print $2}'`; do cat /proc/$i/smaps |sed '1 i\1' | while read line; do  awk '$2~/w/ && $2!~/s/ {x=NR+2;next}(NR<=x){print}'; done | grep Rss | awk '{v+=$2} END{print v}'; done | awk '{total+=$1} END{print total}'

I observed that the first line of the output of smap did not appear. It looked like as if the cat command inside the loop read the first line as if it were not a new line, so does not count it. To overcome this issue, I added the sed '1 i\1', that adds a new line at the beginning of the output.

To retrieve the size of the loaded libraries I use:

for i in `ps -fu oracle | grep -v grep | grep -v PID | awk '{print $2}'`; do cat /proc/$i/smaps |sed '1 i\1' | while read line; do  awk '($6~/ld/ || /lib/ || /oracle/) && ($2!~/w/) {x=NR+2;next}(NR<=x){print}'; done | grep Rss | awk '{v+=$2} END{print v}'; done | sort -n

Theorically, those libraries should be loaded in memory just once, and all the processes should open pointers to the memory adresses where those segments are. So, what I do is to take only one value from  the previous output, the biggest one. (asn an option, could take the average or the median).

For the shared memory, if do not use memory_target, then I take the output from /proc/sysvpc/shm

cat /proc/sysvipc/shm | grep `id -u oracle` | awk '{v+=$15} END{print v/1024}'

When the memory_target parameter is set, we can use the contents from /dev/shm filesystem, which contains a lot of files with this format:

ora_<SID>_14680074_143

The "reserved space" in the filesystem for each of those files correspond to the granule size of the Oracle instance they belong to. (output of ls -l )

The actual space (du -h) can differ. If the output os ls -l is zero, that means that is only the file descriptor allocated, but the segment empty.

Let's see an example (the system is a real one, but the names have been changed):

[TST][servidor].oracle:shm > pwd

/dev/shm

[TST][servidor].oracle:shm >ls -l

:

:

-rw-r----- 1 oracle asmadmin 33554432 Aug 22 12:26 ora_ORCL_14680074_300

-rw-r----- 1 oracle asmadmin 33554432 Aug 22 12:26 ora_ORCL_14680074_305

-rw-r----- 1 oracle asmadmin 33554432 Aug 22 12:26 ora_ORCL_14680074_291

-rw-r----- 1 oracle asmadmin 33554432 Aug 22 12:26 ora_ORCL_14680074_317

-rw-r----- 1 oracle asmadmin 33554432 Aug 22 12:26 ora_ORCL_14680074_315

Here we'll see that some segments show 0 in the fifth field (size)

[TST][servidor].oracle:shm > ls -lrt *ORCL* | wc -l

321

[TST][servidor].oracle:shm > ls -lrt *ORCL* | wc -l

[TST][servidor].oracle:shm > ls -lrt *ORCL* | wc -l

321

[TST][servidor].oracle:shm >

Let' connect to the instance:

ORCL SQL> select (321*33554432)/1024/1024/1024||' GB' from dual;

(321*335544

-----------

10.03125 GB

ORCL SQL>  show parameter memory_target

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

memory_target                        big integer 10G

ORCL SQL>

So, here we see how the contents of /dev/shm correspond to the memory target of the instance.

The reasons of the difference between using memory target and do not use it are explained in note 731658.1 of Oracle Support. Basically, when Oracle on Linux is configured to use Auto Memory Management (AMM), it utilizes the POSIX implementation of SHM on Linux. When AMM is disabled, Oracle falls back to using the System V SHM implementation as seen in earlier versions of Oracle for Linux.