Configuring HugePages for Oracle on Linux (x86-64)

 Introduction
 Configuring HugePages
 Force Oracle to use HugePages (USE_LARGE_PAGES)
 Disabling Transparent HugePages (RHEL6/OL6 and RHEL7/OL7)
 Configuring 1G Hugepagesize

Introduction
For large SGA sizes, HugePages can give substantial benefits in virtual memory management.
Without HugePages, the memory of the SGA is divided into 4K pages, which have to be
managed by the Linux kernel. Using HugePages, the page size is increased to 2MB (configurable
to 1G if supported by the hardware), thereby reducing the total number of pages to be managed
by the kernel and therefore reducing the amount of memory required to hold the page table in
memory. In addition to these changes, the memory associated with HugePages can not be
swapped out, which forces the SGA to stay memory resident. The savings in memory and the
effort of page management make HugePages pretty much mandatory for Oracle 11g systems
running on x86-64 architectures.

Just because you have a large SGA, it doesn't automatically mean you will have a problem if you
don't use HugePages. It is typically the combination of a large SGA and lots database
connections that leads to problems. To determine how much memory you are currently using to
support the page table, run the following command at a time when the server is under
normal/heavy load.

# grep PageTables /proc/meminfo

PageTables: 1244880 kB

Automatic Memory Management (AMM) is not compatible with Linux HugePages, so apart
from ASM instances and small unimportant databases, you will probably have no need for AMM
on a real database running on Linux. Instead, Automatic Shared Memory Management and
Automatic PGA Management should be used as they are compatible with HugePages.

Configuring HugePages
Run the following command to determine the current HugePage usage. The default HugePage
size is 2MB on Oracle Linux 5.x and as you can see from the output below, by default no
HugePages are defined.
$ grep Huge /proc/meminfo

AnonHugePages: 0 kB

HugePages_Total: 0

HugePages_Free: 0

HugePages_Rsvd: 0

HugePages_Surp: 0

Hugepagesize: 2048 kB

Depending on the size of your SGA, you may wish to increase the value of Hugepagesize to 1G.

Create a file called "hugepages_setting.sh" with the following contents.

#!/bin/bash

# hugepages_settings.sh

# Linux bash script to compute values for the

# recommended HugePages/HugeTLB configuration

# Note: This script does calculation for all shared memory

# segments available when the script is run, no matter it

# is an Oracle RDBMS shared memory segment or not.

# Check for the kernel version

KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`

# Find out the HugePage size

HPG_SZ=`grep Hugepagesize /proc/meminfo | awk {'print $2'}`

# Start from 1 pages to be on the safe side and guarantee 1 free HugePage
NUM_PG=1

# Cumulative number of pages required to handle the running shared memory

segments

for SEG_BYTES in `ipcs -m | awk {'print $5'} | grep "[0-9][0-9]*"`

do

MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`

if [ $MIN_PG -gt 0 ]; then

NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`

fi

done

# Finish with results

case $KERN in

'2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;

echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;

'2.6' | '3.8' | '3.10' | '4.1' | '4.14' ) echo "Recommended setting:

vm.nr_hugepages = $NUM_PG" ;;

*) echo "Unrecognized kernel version $KERN. Exiting." ;;

esac

# End

Thanks to Bjoern Rost for pointing out the issue when using the script against UEK3 and the
suggested fix. I've subsequently added support for 3.10 and 4.1. There is a newer version of this
script available from MOS (Doc ID 401749.1) which includes these kernel versions also.
Make the file executable.

$ chmod u+x hugepages_setting.sh

Make sure all the Oracle services are running as normal on the server, then run the script and
make a note of the recommended "vm.nr_hugepages" value.

$ ./hugepages_setting.sh
Recommended setting: vm.nr_hugepages = 305

Edit the "/etc/sysctl.conf" file as the "root" user, adding the following entry, adjusted based on
your output from the script. You should set the value greater than or equal to the value displayed
by the script. You only need 1 or 2 spare pages.

vm.nr_hugepages=306

One person reported also needing the hugetlb_shm_group setting on Oracle Linux 6.5. I did not
and it is listed as a requirement for SUSE only. If you want to set it, get the ID of the dba group.

# fgrep dba /etc/group

dba:x:54322:oracle

Use the resulting group ID in the "/etc/sysctl.conf" file.

vm.hugetlb_shm_group=54322

Run the following command as the "root" user.

# sysctl -p

Alternatively, edit the "/etc/grub.conf" file, adding "hugepages=306" to the end of the kernel line
for the default kernel and reboot.

You can now see the HugePages have been created, but are currently not being used.

$ grep Huge /proc/meminfo

AnonHugePages: 0 kB

HugePages_Total: 306

HugePages_Free: 306

HugePages_Rsvd: 0

HugePages_Surp: 0
Hugepagesize: 2048 kB

Add the following entries into the "/etc/security/limits.conf" script or "/etc/security/limits.d/99-

grid-oracle-limits.conf" script, where the setting is at least the size of the HugePages allocation
in KB (HugePages * Hugepagesize). In this case the value is 306*2048=626688.

* soft memlock 626688

* hard memlock 626688

If you prefer, you can set these parameters to a value just below the size of physical memory of
the server. This way you can forget about it, unless you add more physical memory.
Check the MEMORY_TARGET parameters are not set for the database and SGA_TARGET and
PGA_AGGREGATE_TARGET parameters are being used instead.

SQL> show parameter target

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

archive_lag_target integer 0

db_flashback_retention_target integer 1440

fast_start_io_target integer 0

fast_start_mttr_target integer 0

memory_max_target big integer 0

memory_target big integer 0

parallel_servers_target integer 16

pga_aggregate_target big integer 200M

sga_target big integer 600M

SQL>

Restart the server and restart the database services as required.

Check the HugePages information again.

$ grep Huge /proc/meminfo

AnonHugePages: 0 kB

HugePages_Total: 306

HugePages_Free: 98

HugePages_Rsvd: 93

HugePages_Surp: 0

Hugepagesize: 2048 kB

You can see the HugePages are now being used.

Remember, if you increase your memory allocation or add new instances, you need to retest the
required number of HugePages, or risk Oracle running without them.

Force Oracle to use HugePages (USE_LARGE_PAGES)

Sizing the number of HugePages correctly is important because prior to 11.2.0.3, if the whole
SGA doesn't fit into the available HugePages, the instance will start up without using any. From
11.2.0.3 onward, the SGA can run partly in HugePages and partly not, so the impact of this issue
is not so great. Incorrect sizing may not be obvious to spot. Later releases of the database display
a "Large Pages Information" section in the alert log during startup.

****************** Large Pages Information *****************

Total Shared Global Region in Large Pages = 602 MB (100%)

Large Pages used by this instance: 301 (602 MB)

Large Pages unused system wide = 5 (10 MB) (alloc incr 4096 KB)

Large Pages configured system wide = 306 (612 MB)

Large Page size = 2048 KB

***********************************************************
If you are running Oracle 11.2.0.2 or later, you can set the USE_LARGE_PAGES initialization
parameter to "only" so the database fails to start if it is not backed by hugepages. You can read
more about this here.

ALTER SYSTEM SET use_large_pages=only SCOPE=SPFILE;

SHUTDOWN IMMEDIATE;

STARTUP;

On startup the "Large Page Information" in the alert log reflects the use of this parameter.

****************** Large Pages Information *****************

Parameter use_large_pages = ONLY

Total Shared Global Region in Large Pages = 602 MB (100%)

Large Pages used by this instance: 301 (602 MB)

Large Pages unused system wide = 5 (10 MB) (alloc incr 4096 KB)

Large Pages configured system wide = 306 (612 MB)

Large Page size = 2048 KB

***********************************************************

Attempting to start the database when there aren't enough HugePages to hold the SGA will now
return the following error.

SQL> STARTUP

ORA-27137: unable to allocate large pages to create a shared memory segment

Linux-x86_64 Error: 12: Cannot allocate memory

SQL>

The "Large Pages Information" section of the alert log output describes the startup failure and the
appropriate action to take.
****************** Large Pages Information *****************

Parameter use_large_pages = ONLY

Large Pages unused system wide = 0 (0 KB) (alloc incr 4096 KB)

Large Pages configured system wide = 0 (0 KB)

Large Page size = 2048 KB

ERROR:

Failed to allocate shared global region with large pages, unix errno = 12.

Aborting Instance startup.

ORA-27137: unable to allocate Large Pages to create a shared memory segment

ACTION:

Total Shared Global Region size is 608 MB. Increase the number of

unused large pages to atleast 304 (608 MB) to allocate 100% Shared Global

Region with Large Pages.

***********************************************************

Disabling Transparent HugePages (RHEL6/OL6 and RHEL7/OL7)

Starting from RHEL6/OL6, Transparent HugePages are implemented and enabled by default.
They are meant to improve memory management by allowing HugePages to be allocated
dynamically by the "khugepaged" kernel thread, rather than at boot time like conventional
HugePages. That sounds like a good idea, but unfortunately Transparent HugePages don't play
well with Oracle databases and are associated with node reboots in RAC installations and
performance problems on both single instance and RAC installations. As a result Oracle
recommends disabling Transparent HugePages on all servers running Oracle databases, as
described in this MOS note.

 ALERT: Disable Transparent HugePages on SLES11, RHEL6, RHEL7, OL6, OL7 and
UEK2 Kernels (Doc ID 1557478.1)
The following examples use the base path of "/sys/kernel/mm/transparent_hugepage/" which is
used by OL6/OL7. For RHEL6/RHEL7 use "/sys/kernel/mm/redhat_transparent_hugepage/" as
the base path.
You can check the current setting using the following command, which is displaying the default
value of "enabled=[always]".

# cat /sys/kernel/mm/transparent_hugepage/enabled

[always] madvise never

For Oracle Linux 6 the preferred method to disable Transparent HugePages is to add
"transparent_hugepage=never" to the kernel boot line in the "/boot/grub/grub.conf" file.

title Oracle Linux Server (2.6.39-400.24.1.el6uek.x86_64)

root (hd0,0)

kernel /vmlinuz-2.6.39-400.24.1.el6uek.x86_64 ro
root=/dev/mapper/vg_ol6112-lv_root rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=uk

LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_NO_DM

rd_LVM_LV=vg_ol6112/lv_swap rd_LVM_LV=vg_ol6112/lv_root rhgb quiet numa=off

transparent_hugepage=never

initrd /initramfs-2.6.39-400.24.1.el6uek.x86_64.img

Oracle Linux 7 is similar, but uses GRUB2 so you need to edit the "/boot/grub2/grub.cfg" file
using the grubby command.

# grubby --default-kernel

/boot/vmlinuz-4.1.12-61.1.6.el7uek.x86_64

# grubby --args="transparent_hugepage=never" --update-kernel /boot/vmlinuz-

4.1.12-61.1.6.el7uek.x86_64

# grubby --info /boot/vmlinuz-4.1.12-61.1.6.el7uek.x86_64

index=2
kernel=/boot/vmlinuz-4.1.12-61.1.6.el7uek.x86_64

args="ro vconsole.font=latarcyrheb-sun16 rd.lvm.lv=ol/swap rd.lvm.lv=ol/root

crashkernel=auto vconsole.keymap=uk rhgb quiet LANG=en_GB.UTF-8
transparent_hugepage=never"

root=/dev/mapper/ol-root

initrd=/boot/initramfs-4.1.12-61.1.6.el7uek.x86_64.img

title=Oracle Linux Server 7.2, with Unbreakable Enterprise Kernel 4.1.12-

61.1.6.el7uek.x86_64

The server must be rebooted for this to take effect.

Alternatively, add the following lines into the "/etc/rc.local" file and reboot the server.

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then

echo never > /sys/kernel/mm/transparent_hugepage/enabled

fi

if test -f /sys/kernel/mm/transparent_hugepage/defrag; then

echo never > /sys/kernel/mm/transparent_hugepage/defrag

fi

Whichever method you choose, remember to check the change has work after reboot.

# cat /sys/kernel/mm/transparent_hugepage/enabled

always madvise [never]

In OL7/RHEL7 you also need to consider the "tuned profile". The following script shows how to
create and enable an amended version of the currently active tuned profile.

# # Check the active profile

# tuned-adm active

Current active profile: virtual-guest

#

# # Create directory to hold revised profile.

# mkdir /etc/tuned/virtual-guest-nothp

# # Create new profile based on the curren active profile.

# cat <<EOF >> /etc/tuned/virtual-guest-nothp/tuned.conf

[main]

include= virtual-guest

[vm]

transparent_hugepages=never

EOF

# # Make the script executable.

# chmod +x /etc/tuned/virtual-guest-nothp/tuned.conf

# # Enable the new profile.

# tuned-adm profile virtual-guest-nothp

Thanks to Mor for pointing this out and directing me to the notes here and here.
With Transparent HugePages disabled, you should proceed to configure conventional
HugePages, as described above.

Configuring 1G Hugepagesize
As mentioned by Eugene in the comments, Oracle currently don't recommend using 1G
Hugepagesize. You can read more about this in MOS Doc ID 1607545.1. With that in mind, the
rest of this section should probably be considered more of an academic exercise.
Check if your current hardware can support a Hugepagesize of 1G. If the following command
produces any output, it can.
# cat /proc/cpuinfo | grep pdpe1gb

Thanks to Kevin Closson for pointing out the hardware support requirement.

Edit the "/etc/grub.conf" file, adding the following entries on to the kernel line of the default
grub entry. Adjust the "hugepages" entry to the desired number of 1G pages. Notice this includes
the disabling of Transparent HugePages, which is not mandatory, but a good idea.

transparent_hugepage=never hugepagesz=1G hugepages=1 default_hugepagesz=1G

Check the current HugePages setup.

# grep Huge /proc/meminfo

HugePages_Total: 0

HugePages_Free: 0

HugePages_Rsvd: 0

HugePages_Surp: 0

Hugepagesize: 2048 kB

Reboot and check the HugePages setup again.

# grep Huge /proc/meminfo

HugePages_Total: 1

HugePages_Free: 1

HugePages_Rsvd: 0

HugePages_Surp: 0

Hugepagesize: 1048576 kB

For more information see:

 Overview of HugePages
 Configuring Linux Hugepages for Oracle Database Is Just Too Difficult! Isn’t It? Part – I.
  Huge Pages and Transparent Huge Pages
 HugePages on Oracle Linux 64-bit [ID 361468.1]
 HugePages on Linux: What It Is... and What It Is Not... [ID 361323.1]
 ALERT: Disable Transparent HugePages on SLES11, RHEL6, OEL6 and UEK2 Kernels
[ID 1557478.1]
 USE_LARGE_PAGES
 Oracle Linux: Shell Script to Calculate Values Recommended Linux HugePages /
HugeTLB Configuration (Doc ID 401749.1)
 ALERT: Disable Transparent HugePages on SLES11, RHEL6, RHEL7, OL6, OL7 and
UEK2 Kernels (Doc ID 1557478.1)

Hope this helps. Regards Tim...