Anatomy of Linux Server Memory Utilization

Ajith Narayanan
Dell
Bangalore, India

Introduction
Oracle cloud services customers were facing a long time issue with production server memory utilization
always being on the higher side (> 90% Memory usage). At some point that seemed to be something
unusual and not realistic because if the memory usage is at that high level business would never be as
usual for them, but customers were only worried about the memory graph that stays at 95% most of the
time even when their business was as usual. My friend Neeraj Bhatia did some research on this issue and
found out a Linux bug (From Oracle's perspective) and below is the explanation of the issue found.

Problem Statement
Oracle Enterprise Manager (OEM grid control) pulls memory usage information from SAR output saved
in the Linux servers, Which used to mislead the Memory utilization for a while, Not sure about if Oracle
EM 12C Which is the current production monitoring tool) has rectified the below mentioned Linux issue.
If not, the memory usage is still misleading us.

Below is the explanation for the memory usage misleading issue.

## Here "kbmemfree","kbmemused","%memused","kbbuffers","kbcached" is shown where %memused is

##merely calculated by formula Memory utilization = kbmemused/sum(kbmemused, kbmemfree)).
## In order to get actual memory utilization, subtract kbbuffers and kbcached from kbmemused, and
##When we used the above formula.
## In this case, Memory utilization = (kbmemused-kbbuffers-kbcached)/sum(kbmemused,kbmemfree)

Kbmemused kbmemfree kbbuffers kbcached %memused

33180580 16252192 1591888 22660668 Incorrect Formula 0.671226368 = 67%
Correct Formula 0.180609414 = 18%

The reason behind this is Linux treats unused memory as a wasted resource and so uses as much RAM as
it can to cache process/kernel information.
Here Buffers= amount of physical memory used as buffers for disk writes Cached = amount of physical
memory used as cache for disk reads

Even if you use top utility to check the memory utilization, situation seems dreadful. It appears that
almost all memory is being used. This is because of the way top display memory utilization information.

in KB in MB in GB
Total Physical Memory 49432772 48274.19141 47.14277
Memory Used 33167868 32390.49609 31.63134
Memory Free 16264904 15883.69531 15.51142

The top command reports that total 2065628k bytes (2017MB) of memory is configured on the system,
1195636k (1167MB) is used and 869992k (849MB) is free. Some of the used memory that the top
command reports is actually being used for cache. In other words, it is available to be reassigned if
another process requires it.

To get a better picture of memory utilization, you should use free command. free is basically a c-program
that reads output from /proc virtual file system.

The free command reports output in 1024-byte blocks. I used -m switch to report memory utilization in
megabytes.

We can interpret free command output as follows. In the first line, it reports that this system has
48274MB memory configured, out of which 32581MB is used and 15693 is free. The shared memory
column is obsolete and should be ignored. 1554MB reported under buffers column, being used as buffers
for disk writes and 22106MB used as cache for disk read operations. It should be note that memory
reported under buffers and cached, is available for use. Hence in second line, they are added to free
memory. Thus in actual, system has 39353 MB free for use (~ =15693+1554+22106 MB). Thus it is
important to note that, we should concentrate on second line “-/+ buffers/cache:” to get free memory in
the system.

The third line reports swap utilization information. Here system has 1984MB of swap configured, swap is
not being used and total swap is free for use. We can also get current memory utilization using vmstat
(virtual memory statistics tool).

Here you can download a quick tool(It shows the actual host memory utilization%) that was written with
expectation of OEM or Grid control to realize this Linux bug at least in future releases(EM 12c is out
now!!) SARex https://www.dropbox.com/s/rapqdlpb5afji6n/SARex.zipx
 About Author:

Ajith Narayanan has a total of 9 Years of work experience primarily on Oracle database
administration and Oracle Apps administration, Capacity planning & Performance tuning on various
Oracle environments.
His work experience includes –
TCS - (Oracle/Oracle APPS DBA),
Oracle India - (Oracle Apps DBA/Capacity Planning),
J.P.MorganChase & Co (Oracle DBA) &
Dell IT as Software Development Advisor (Oracle Apps R12 projects in Dell Inc.).
Serves on the board of RACSIG as website chair and speaks at Oracle user community events like
SANGAM-12, NZOUG-13, AIOUG-TechDay etc.

Contact address:

Ajith Narayanan
Dell IT
Bagmane Tech Park Byra Sundra,
C V Raman Nagar Bagmane Parin Katha
Bangalore South Taluk
Bangalore INDIA-560093

Phone: +91 9008488882

Email ajith_narayanan@dell.com
Internet: http://oracledbascriptsfromajith.blogspot.com