Exadata Patch Staging and Prechecks: Solution

Exadata Patch Staging and Prechecks
Describe how to perform staging for an Exadata Platinum Patching event and
execute prechecks.
SOLUTION
Part 1 - Staging
Preparation
Before running the staging script you must download the required patches into a
directory (default is $HOME/patches) on the gateway (or to the patch base
(default /u01/patches) on the first node of the customer system). This directory
must conform to the Platinum Patching directory structure.
The top level directory maps to Patch Base and contains the Patch Utils zip file and
the following subdirectories:
BP - This directory contains the database bundle patch(es) to be applied
ONEOFF - This directory contains any database oneoff patches to be applied
OPATCH - This directory contains the latest opatch zip files
CELL - This directory contains the Exadata cell patch.
YUM - This directory contains the Exadata compute node patch(es) as well as the
dbserver orchestrator patch
Execution
Unzip patch 20629335 to the /tmp directory
unzip p20629335_121212_Linux-x86-64.zip -d /tmp
cd /tmp/20629335
./stage.sh -V
stage.sh, version 17.11.08.18.03
./stage.sh -h
Usage:
./stage.sh [-dbsgroup=<dbs_group-file>] [-patchbase=<patch-
base-direcory>]
The staging script has 3 phases. You have the option to skip or run each step
independently.
Simply answer yes or no to the prompt at each phase.
 Phase 1: Copy patches from gateway

o answer no to this step if you have already manually staged patches to
first node of the customer system
o You will be asked if you want to clean the patch base on all nodes.
This is recommended. All files will be deleted from the patch base on
all nodes.
o The script will check if there is at least 15G of space on all nodes. If
this step fails you will have a chance to continue anyway. This is only
advised if you did not clean patch base and are recopying files.
o You will need your username and password for the gateway
o You will need to know the directory location containing the patches as
described above
 Phase 2: Stage patches to cluster nodes
o This step will copy all of the zip files found in the patch base
(default /u01/patches) of the current node to the corresponding patch
base on all the other nodes.
o Note that the CELL subdirectory will not be copied to the remote nodes
because it is only needed from node 1.
o Another space check will occur before copying. Again, you will have the
chance to continue anyway. This is only advised if you are re-copying
files.
 Phase 3: Unzip patches
o This step will unzip all of the patches that are required to be unzipped
on all nodes. Note that not all patches are required to be unzipped.
Once completed, move over to the PRECHECKS directory to proceed with prechecks.
Sample run of staging script
# ./stage.sh -dbsgroup=dbs_group -patchbase=/u02/patches
2017-Nov-08 22:02:25 UTC [INFO] Verify contents of

/tmp/platinum/dbs_group
dmiscdb01vm02
dmiscdb02vm02

Are these the correct nodes? (Yes/No) y
Copy patches from the gateway? (Yes/No): y
2017-Nov-08 22:02:35 UTC [INFO] The script will remove ALL

FILES from the /u02/patches directory on ALL NODES
2017-Nov-08 22:02:35 UTC [INFO] The following files are
present under /u02/patches on the LOCAL NODE
-rw-r--r-- 1 root root 1113 Oct 18 22:35 patch.conf.sample

-rwxr--r-- 1 root root 170927 Oct 18 22:35 dbhomeupdate.sh
-rwxr-xr-x 1 root root 100 Nov 4 03:10 reboot_cells.sh
-rw-r--r-- 1 root root 720 Nov 4 03:19 readme.txt
drwxr-xr-x 2 root root 4096 Nov 8 22:04 OPATCH
drwxr-xr-x 3 root root 4096 Nov 8 22:07 CELL
-rw-r--r-- 1 root root 37594297 Nov 8 23:13
p20629338_121212_Linux-x86-64.zip
drwxr-xr-x 5 root root 4096 Nov 8 23:13 ONEOFF
drwxr-xr-x 3 root root 4096 Nov 8 23:14 BP
drwxr-xr-x 2 root root 4096 Nov 8 23:14 POSTCHECKS
drwxr-xr-x 2 root root 4096 Nov 8 23:14 DBSCRIPTS
drwxr-xr-x 2 root root 4096 Nov 8 23:14 PRECHECKS
drwxr-xr-x 4 root root 4096 Nov 8 23:14 VNC
drwxr-xr-x 3 root root 4096 Nov 8 23:15 YUM
Proceed with removing ALL FILES from /u02/patches directory on

ALL NODES? (Yes/No) y
2017-Nov-08 22:02:40 UTC [SUCCESS] All files have been removed

from /u02/patches on all nodes
2017-Nov-08 22:02:40 UTC [PASS] dmiscdb01vm02: Free space

on /u02/patches. Required: 15728640K Available: 17150632K

Please enter your GUID on gateway machine: root
Please enter the gateway IP address or hostname: dmiscdb02
Please enter the directory containing patches on the gateway
(Default: /home/root/patches): /EXAVMIMAGES/ft/
2017-Nov-08 22:03:18 UTC [INFO] Make sure the password is

reset on cta for 'root' user using 'sudo passwd root'
Proceed with copying patches? (Yes/No): y

2017-Nov-08 22:03:25 UTC [SUCCESS] Ping successful to
dmiscdb02
2017-Nov-08 22:03:25 UTC [INFO] Copying patches from

dmiscdb02...
==========================================
EEST and Platinum Exadata OVM test environment
==========================================
Contact Pratap Unnava (pratap.unnava@oracle.com) for any

queries or config changes
VM List
-------
dmiscdb0[12]vm01 - EEST Lab 1 - 12.2.0.1.170814 Grid and RDBMS
dmiscdb0[12]vm02 - Platinum Lab 1 - 12.2.0.1.170814 Grid and
RDBMS
RDBMS
dmiscdb0[12]vm05 - SE Lab - SE-clu
RDBMS
** DOM0 - DO NOT REBOOT OR PERFORM ANY ACTION WHICH CAN BREAK

THE SYSTEM **

root@dmiscdb02's password:
p26610322_121020_Linux-x86-64.zip 100% 2144MB 85.8MB/s 00:25
p23727148_121020_Generic.zip 100% 982KB 981.8KB/s 00:00
p26526726_12102170814ProactiveBP_Linux-x86-64.zip 100% 870KB
870.4KB/s 00:00
p26762083_12102170718ProactiveBP_Linux-x86-64.zip 100% 1718KB
1.7MB/s 00:00
2017-Nov-08 22:04:31 UTC [SUCCESS] Patches copied to
dmiscdb02vm02
Stage patches to the rest of the cluster? (Yes/No): y

2017-Nov-08 22:04:38 UTC [INFO] Checking required space for
copying patches
2017-Nov-08 22:04:38 UTC [SUCCESS]
/u02/patches/ONEOFF/p23727148_121020_Generic.zip copied to all
nodes
2017-Nov-08 22:04:38 UTC [SUCCESS]
/u02/patches/ONEOFF/p26526726_12102170814ProactiveBP_Linux-
x86-64.zip copied to all nodes
2017-Nov-08 22:04:38 UTC [SUCCESS]
x86-64.zip copied to all nodes
2017-Nov-08 22:04:58 UTC [SUCCESS]
/u02/patches/BP/p26610322_121020_Linux-x86-64.zip copied to
all nodes
2017-Nov-08 22:04:59 UTC [SUCCESS]
/u02/patches/OPATCH/p6880880_121010_Linux-x86-64.zip copied to
all nodes
2017-Nov-08 22:05:00 UTC [SUCCESS]
/u02/patches/p20629338_121212_Linux-x86-64.zip copied to all
nodes
2017-Nov-08 22:05:13 UTC [SUCCESS]
/u02/patches/YUM/p26151721_122110_Linux-x86-64.zip copied to
all nodes
2017-Nov-08 22:05:14 UTC [SUCCESS]
/u02/patches/YUM/p21634633_181100_Linux-x86-64.zip copied to
all nodes
Unzip patches? (Yes/No): y

2017-Nov-08 22:05:17 UTC [INFO] Unzipping

/u02/patches/ONEOFF/p23727148_121020_Generic.zip on all DB
Nodes
2017-Nov-08 22:05:17 UTC [SUCCESS] Patch
p23727148_121020_Generic.zip unzipped successfully on all
nodes

x86-64.zip on all DB Nodes
p26526726_12102170814ProactiveBP_Linux-x86-64.zip unzipped
successfully on all nodes

x86-64.zip on all DB Nodes
p26762083_12102170718ProactiveBP_Linux-x86-64.zip unzipped
successfully on all nodes

/u02/patches/BP/p26610322_121020_Linux-x86-64.zip on all DB
Nodes
p26610322_121020_Linux-x86-64.zip unzipped successfully on all
nodes

/u02/patches/p20629338_121212_Linux-x86-64.zip on all DB Nodes
nodes

/u02/patches/YUM/p21634633_181100_Linux-x86-64.zip on all DB
Nodes
nodes

2017-Nov-08 22:06:41 UTC [SUCCESS] Unzipping completed

successfully
2017-Nov-08 22:06:41 UTC [INFO] Next steps:

2017-Nov-08 22:06:41 UTC [INFO] Create or modify
/u02/patches/patch.conf according to the Precheck Automation
tab in the patch plan
2017-Nov-08 22:06:41 UTC [INFO] cd /u02/patches/PRECHECKS
2017-Nov-08 22:06:41 UTC [INFO] Execute the precheck.sh script
2017-Nov-08 22:06:41 UTC [INFO] Refer to MOS note 2325909.1
for more infomation
Part 2 - Prechecks
Preparation
Before running the precheck.sh script you must do the following:
a. create a patch.conf file in the patch base directory. The required contents for
this file can be found on the "Precheck Automation" tab of the patch plan.
b. locate or create the required group files containing one node per line. The
group files are needed for
o compute nodes (dbs_group - default is
/opt/oracle.SupportTools/onecommand/dbs_group)
o cell nodes (cell_group - default is
/opt/oracle.SupportTools/onecommand/cell_group)
o IB switches (ibswitches.lst - default is ibswitches.lst in the cell patch
directory)
Execution
Change to the PRECHECKS directory under the patch base and execute the
precheck.sh script. In most cases you should not need to supply any arguments. Use
the -help flag to list all options.
# ./precheck.sh -V
precheck.sh, version 17.10.31.18.54
# ./precheck.sh -h
./precheck.sh [options]
Options:
-patchbase=<dir> Default:/u01/patches
-dbsgroup=<file>
Default:/opt/oracle.SupportTools/onecommand/dbs_group
-cellgroup=<file>
Default:/opt/oracle.SupportTools/onecommand/cell_group
-switchgroup=<file>
Default:EXADATA_CELL_DIRECTORY/ibswitches.lst
-recheck_all Discards pass list and rechecks all
applicable checks
-skip_dbsrv Skips compute node checks
-skip_cells Skips cell checks
-skip_database Skips database checks
-skip_ib Skips infiniband checks
-skip_em Skips EM checks
-skip_data_collection Skips data collection
-event Discards pass list and rechecks all
applicable checks
except YUM, Cell, IB, and dbhomeupdate
prechecks
-critical Discards pass list and rechecks only
critical checks
-d|--debug Displays verbose and debugging output

-V|--version Prints the version number and exits
-h|-help|--help Displays this help message and exits
You may execute the script as many times as needed. The script will remember the
last run for up to 24 hours. Upon re-execution, the script will, by default, only
recheck checks that have not already passed. If you want to redo all checks then
supply the -recheck_all flag.
A failure summary is displayed at the end of each run.
Once completed, upload the resulting zip file the the patch event SR.
Sample run of precheck script
# ./precheck.sh -dbsgroup=dbs_group -patchbase=/u02/patches
2017-Nov-08 23:29:53 UTC [INFO] Server Type: DomU
2017-Nov-08 23:29:53 UTC [INFO] Using /u02/patches/patch.conf
2017-Nov-08 23:29:53 UTC [INFO] ga_pass_list cleared

2017-Nov-08 23:29:53 UTC [INFO] Command executed:
./precheck.sh -dbsgroup=dbs_group -patchbase=/u02/patches
2017-Nov-08 23:29:53 UTC [INFO] Checking ssh equivalence to DB
nodes
2017-Nov-08 23:29:53 UTC [PASS] Ssh equivalence configured
from dmiscdb02vm02 to dmiscdb01vm02
2017-Nov-08 23:29:53 UTC [PASS] Ssh equivalence configured
from dmiscdb02vm02 to dmiscdb02vm02
2017-Nov-08 23:29:53 UTC [INFO] Checking dcli utility

2017-Nov-08 23:29:53 UTC [PASS] dcli utility found
2017-Nov-08 23:29:53 UTC [INFO]

**************************************************************
********
2017-Nov-08 23:29:53 UTC [INFO] PERFORMING COMPUTE NODE CHECKS
2017-Nov-08 23:29:53 UTC [INFO]
**************************************************************
********
2017-Nov-08 23:29:53 UTC [INFO] Checking free space in LV's

found in VG
found in VG
2017-Nov-08 23:29:54 UTC [INFO] Checking free space on / for

db nodes
2017-Nov-08 23:29:54 UTC [PASS] dmiscdb01vm02: Free space on /
filesystem
2017-Nov-08 23:29:54 UTC [FAIL] dmiscdb02vm02: Free space on /
filesystem. Required: 3145728K Available: 3051520K
2017-Nov-08 23:29:54 UTC [INFO] Checking for active snapshots

2017-Nov-08 23:29:54 UTC [PASS] dmiscdb01vm02: No active
snapshots found
2017-Nov-08 23:29:54 UTC [PASS] dmiscdb02vm02: No active
snapshots found
2017-Nov-08 23:29:54 UTC [INFO] Checking free space on /u01

for db nodes
on /u01 filesystem
on /u01 filesystem
2017-Nov-08 23:29:54 UTC [INFO] Checking dbnodeupdate.sh

version
2017-Nov-08 23:29:56 UTC [INFO] dmiscdb01vm02: dbnodeupdate.sh
version 5.171104 is 5 days old
2017-Nov-08 23:29:56 UTC [PASS] dmiscdb01vm02: dbnodeupdate
version OK
2017-Nov-08 23:29:58 UTC [INFO] dmiscdb02vm02: dbnodeupdate.sh
version 5.171104 is 5 days old
2017-Nov-08 23:29:58 UTC [PASS] dmiscdb02vm02: dbnodeupdate
version OK
2017-Nov-08 23:29:58 UTC [PASS] Dbnodeupdate.sh version
matches across all nodes
2017-Nov-08 23:29:58 UTC [INFO] Checking for duplicate labels

2017-Nov-08 23:29:58 UTC [PASS] dmiscdb01vm02: No Duplicate
filesystem labels
2017-Nov-08 23:29:59 UTC [PASS] dmiscdb02vm02: No Duplicate
filesystem labels
2017-Nov-08 23:29:59 UTC [INFO] Checking label consistency

2017-Nov-08 23:29:59 UTC [PASS] dmiscdb01vm02: BOOT configured
in /etc/fstab
2017-Nov-08 23:29:59 UTC [PASS] dmiscdb01vm02: BOOT current
mountpoint matches configured mountpoint
2017-Nov-08 23:29:59 UTC [PASS] dmiscdb01vm02: DBSYS device
found in fstab
2017-Nov-08 23:30:00 UTC [PASS] dmiscdb02vm02: BOOT configured
in /etc/fstab
2017-Nov-08 23:30:00 UTC [PASS] dmiscdb02vm02: BOOT current
mountpoint matches configured mountpoint
2017-Nov-08 23:30:00 UTC [PASS] dmiscdb02vm02: DBSYS device
found in fstab
2017-Nov-08 23:30:00 UTC [INFO] Checking kernel files

2017-Nov-08 23:30:00 UTC [PASS] dmiscdb01vm02: Running kernel
matches default kernel in grub.conf
2017-Nov-08 23:30:00 UTC [PASS] dmiscdb01vm02: Default kernel
file exists
2017-Nov-08 23:30:00 UTC [PASS] dmiscdb01vm02: Running initial
ramdisk matches default initial ramdisk in grub.conf
2017-Nov-08 23:30:00 UTC [PASS] dmiscdb01vm02: Default initial
ramdisk file exists
2017-Nov-08 23:30:01 UTC [PASS] dmiscdb02vm02: Running kernel
matches default kernel in grub.conf
2017-Nov-08 23:30:01 UTC [PASS] dmiscdb02vm02: Default kernel
file exists
2017-Nov-08 23:30:01 UTC [PASS] dmiscdb02vm02: Running initial
ramdisk matches default initial ramdisk in grub.conf
2017-Nov-08 23:30:01 UTC [PASS] dmiscdb02vm02: Default initial
ramdisk file exists
2017-Nov-08 23:30:01 UTC [INFO] Performing HugePage Checks

2017-Nov-08 23:30:01 UTC [PASS] dmiscdb01vm02: HugePages < 80%
of physical memory
2017-Nov-08 23:30:01 UTC [PASS] dmiscdb02vm02: HugePages < 80%
of physical memory
2017-Nov-08 23:30:01 UTC [INFO] Performing GRUB Stage1 boot
area check
2017-Nov-08 23:30:02 UTC [PASS] dmiscdb01vm02: GRUB Stage1
boot area looks good
2017-Nov-08 23:30:02 UTC [PASS] dmiscdb02vm02: GRUB Stage1
boot area looks good
2017-Nov-08 23:30:02 UTC [INFO] Checking inode usage

2017-Nov-08 23:30:02 UTC [PASS] dmiscdb01vm02: Inode usage
(59812) is within normal range
2017-Nov-08 23:30:02 UTC [PASS] dmiscdb02vm02: Inode usage
(53511) is within normal range
2017-Nov-08 23:30:02 UTC [INFO] Checking image status on db

nodes
2017-Nov-08 23:30:03 UTC [PASS] dmiscdb01vm02: Image status is
success
2017-Nov-08 23:30:05 UTC [PASS] dmiscdb02vm02: Image status is
success
2017-Nov-08 23:30:05 UTC [INFO] Checking IB Interface state is

down or not
2017-Nov-08 23:30:05 UTC [PASS] dmiscdb01vm02: IB Interfaces
are functional
2017-Nov-08 23:30:05 UTC [PASS] dmiscdb02vm02: IB Interfaces
are functional
2017-Nov-08 23:30:05 UTC [INFO] Checks if IB mode equals

connected
2017-Nov-08 23:30:05 UTC [PASS] dmiscdb01vm02: IB mode is
connected
2017-Nov-08 23:30:05 UTC [PASS] dmiscdb02vm02: IB mode is
connected
2017-Nov-08 23:30:05 UTC [INFO] Checking whether cellip.ora

file is consistent across DB nodes
2017-Nov-08 23:30:05 UTC [PASS] cellip.ora file is consistent
across DB nodes
2017-Nov-08 23:30:05 UTC [INFO] Verify image version matches

across db nodes
2017-Nov-08 23:30:06 UTC [PASS] dmiscdb01vm02: version is
12.1.2.3.4.170111
2017-Nov-08 23:30:08 UTC [PASS] dmiscdb02vm02: version is
12.1.2.3.4.170111
2017-Nov-08 23:30:08 UTC [INFO] Checking image status

2017-Nov-08 23:30:09 UTC [PASS] dmiscdb01vm02: Image status
correct
2017-Nov-08 23:30:10 UTC [PASS] dmiscdb02vm02: Image status
correct
2017-Nov-08 23:30:10 UTC [INFO] Verifying if the current
version of Exadata is less than target version
2017-Nov-08 23:30:12 UTC [FAIL] dmiscdb01vm02: The current
version of Exadata is NOT less than target version
2017-Nov-08 23:30:13 UTC [INFO] Verifying if the current

Exadata version's date code is less than that of target
version
Exadata version's date code is NOT less than that of target
version
version
2017-Nov-08 23:30:15 UTC [INFO] Checking whether the SSH

inactive timeout is atleast 3 hours
2017-Nov-08 23:30:16 UTC [PASS] dmiscdb01vm02: Inactive SSH
timeout is greater than 3 hours
2017-Nov-08 23:30:16 UTC [PASS] dmiscdb02vm02: Inactive SSH
timeout is greater than 3 hours
2017-Nov-08 23:30:16 UTC [INFO] Checking for interactive user

shells
2017-Nov-08 23:30:26 UTC [PASS] dmiscdb01vm02: User shells are
non-interactive.
2017-Nov-08 23:30:36 UTC [PASS] dmiscdb02vm02: User shells are
non-interactive.
2017-Nov-08 23:30:36 UTC [INFO] Checking for ASM instance

availability
2017-Nov-08 23:30:36 UTC [PASS] dmiscdb01vm02: ASM is up and
accessible.
2017-Nov-08 23:30:37 UTC [PASS] dmiscdb02vm02: ASM is up and
accessible.
2017-Nov-08 23:30:37 UTC [INFO] Checking ASM Disk Repair Time

2017-Nov-08 23:30:39 UTC [FAIL] dmiscdb01vm02: ASM attribute
disk_repair_time is NOT valid.
2017-Nov-08 23:30:40 UTC [INFO] Checking for ASM failgroup

repair time.
failgroup_repair_time is NOT valid.
2017-Nov-08 23:30:42 UTC [INFO] Checking IB error counters
2017-Nov-08 23:30:42 UTC [PASS] dmiscdb01vm02: IB Fabric has
no errors
2017-Nov-08 23:30:42 UTC [PASS] dmiscdb02vm02: IB Fabric has
no errors
2017-Nov-08 23:30:42 UTC [INFO] Checking IB Port Rate

2017-Nov-08 23:30:42 UTC [PASS] dmiscdb01vm02: IB ports have
data transfer rate of 40GB/s
2017-Nov-08 23:30:42 UTC [PASS] dmiscdb02vm02: IB ports have
data transfer rate of 40GB/s
2017-Nov-08 23:30:42 UTC [INFO] Checking IB link status

2017-Nov-08 23:30:42 UTC [PASS] dmiscdb01vm02: IBLink is up
and Active
2017-Nov-08 23:30:42 UTC [PASS] dmiscdb02vm02: IBLink is up
and Active
2017-Nov-08 23:30:42 UTC [INFO] Checking BIOS version can be

captured
2017-Nov-08 23:30:42 UTC [PASS] dmiscdb01vm02: BIOS version
can be captured
2017-Nov-08 23:30:42 UTC [PASS] dmiscdb02vm02: BIOS version
can be captured
2017-Nov-08 23:30:42 UTC [INFO] Checking kdump configuration

KDUMP_KERNEL_NAME
2017-Nov-08 23:30:42 UTC [PASS] dmiscdb01vm02: kdump
configuration KDUMP_KERNEL_NAME is correct
configuration KDUMP_KERNEL_NAME is correct

KDUMP_INITRD_NAME
configuration KDUMP_INITRD_NAME is correct
configuration KDUMP_INITRD_NAME is correct

KDUMP_KERNELVER
configuration KDUMP_KERNELVER is correct
configuration KDUMP_KERNELVER is correct
2017-Nov-08 23:30:43 UTC [INFO] This is an OVM user domain.

Skipping MegaCli64 existence check.
2017-Nov-08 23:30:43 UTC [INFO] Checking that localhost is

127.0.0.1 in /etc/hosts
2017-Nov-08 23:30:43 UTC [PASS] dmiscdb01vm02: 127.0.0.1 is
localhost in /etc/hosts.
2017-Nov-08 23:30:43 UTC [PASS] dmiscdb02vm02: 127.0.0.1 is
localhost in /etc/hosts.
2017-Nov-08 23:30:43 UTC [INFO] Checking eth0 speed is

1000Mb/s
2017-Nov-08 23:30:43 UTC [PASS] dmiscdb01vm02: eth0 is meeting
standard rate - 1000MB/s.
2017-Nov-08 23:30:43 UTC [PASS] dmiscdb02vm02: eth0 is meeting
standard rate - 1000MB/s.
2017-Nov-08 23:30:43 UTC [INFO] Checking if all DBHOME and

GRID entries in /etc/oratab are valid.
2017-Nov-08 23:30:43 UTC [PASS] dmiscdb01vm02: All DBHOME and
GRID /etc/oratab entries are valid
2017-Nov-08 23:30:43 UTC [PASS] dmiscdb02vm02: All DBHOME and
GRID /etc/oratab entries are valid
2017-Nov-08 23:30:43 UTC [FAIL] EXADATA_DBSRV_VERSION not set

in patch.conf. Skipping YUM prechecks.
2017-Nov-08 23:30:43 UTC [INFO]

**************************************************************
********
2017-Nov-08 23:30:43 UTC [INFO] SKIPPING CELL NODE CHECKS
2017-Nov-08 23:30:43 UTC [INFO]
**************************************************************
********
2017-Nov-08 23:30:43 UTC [INFO]

**************************************************************
********
2017-Nov-08 23:30:43 UTC [INFO] PERFORMING DB HOME CHECKS
2017-Nov-08 23:30:43 UTC [INFO]
**************************************************************
********
2017-Nov-08 23:30:43 UTC [INFO] Checking accessibility of

patch base (/u02/patches)
2017-Nov-08 23:30:43 UTC [PASS] dmiscdb01vm02: Permissions
on /u02/patches are OK
2017-Nov-08 23:30:44 UTC [PASS] dmiscdb02vm02: Permissions
on /u02/patches are OK
2017-Nov-08 23:30:44 UTC [INFO] Checking that patches can be

rolled back
2017-Nov-08 23:30:44 UTC [INFO] dmiscdb01vm02: DBHOME_0:
/u01/app/12.2.0.1/grid
2017-Nov-08 23:30:48 UTC [PASS] dmiscdb01vm02: DBHOME_0: All
patches can be rolled back
/u01/app/12.2.0.1/grid
2017-Nov-08 23:30:52 UTC [PASS] dmiscdb02vm02: DBHOME_0: All
patches can be rolled back
2017-Nov-08 23:30:52 UTC [INFO] Checking grid lock status

2017-Nov-08 23:30:52 UTC [PASS] dmiscdb01vm02: Grid home is
locked
2017-Nov-08 23:30:52 UTC [PASS] dmiscdb02vm02: Grid home is
locked
2017-Nov-08 23:30:52 UTC [INFO] Verify oracle md5sum matches

across nodes

b3b771aedfe6fa838ba688bdeb58846f
/u01/app/12.2.0.1/grid/bin/oracle
b3b771aedfe6fa838ba688bdeb58846f
/u01/app/12.2.0.1/grid/bin/oracle
2017-Nov-08 23:30:55 UTC [PASS] DBHOME_0: oracle md5sum
2017-Nov-08 23:30:55 UTC [INFO] Checking dbhomeupdate.sh

version
2017-Nov-08 23:30:55 UTC [INFO] dmiscdb01vm02: dbhomeupdate.sh
version 17.09.28.20.30 is 0 days old
2017-Nov-08 23:30:55 UTC [INFO] dmiscdb02vm02: dbhomeupdate.sh
version 17.09.28.20.30 is 0 days old
2017-Nov-08 23:30:55 UTC [PASS] Dbhomeupdate.sh version
2017-Nov-08 23:30:55 UTC [INFO] Checking that Database Node

Death Detection (DFNDD) is enabled and GI < 12.1.0.2.0
2017-Nov-08 23:31:01 UTC [FAIL] dmiscdb01vm02: Database Node
Death Detection (DFNDD) failed
2017-Nov-08 23:31:07 UTC [INFO] Running dbhomeupdate.sh

precheck
2017-Nov-08 23:31:20 UTC [FAIL] dmiscdb01vm02: DBHOME_0:
Database patch precheck failed. Check log file
dmiscdb01vm02_dbhome_0_prechecks_171109045953.log
2017-Nov-08 23:31:33 UTC [INFO]

**************************************************************
********
2017-Nov-08 23:31:33 UTC [INFO] PERFORMING EM RELATED CHECKS
2017-Nov-08 23:31:33 UTC [INFO]
**************************************************************
********
2017-Nov-08 23:31:33 UTC [INFO] Checking orarom agent

connectivity
2017-Nov-08 23:31:38 UTC [PASS] OMS is reachable from
dmiscdb01vm02
2017-Nov-08 23:31:44 UTC [PASS] OMS is reachable from
dmiscdb02vm02
2017-Nov-08 23:31:44 UTC [INFO] Checking orarom agent is

configured for autostart
2017-Nov-08 23:31:44 UTC [INFO] dmiscdb01vm02 : orarom Agent
home path is /opt/OracleHomes/core/12.1.0.5.0
2017-Nov-08 23:31:44 UTC [PASS] dmiscdb01vm02 : orarom agent
is configured for autostart
2017-Nov-08 23:31:44 UTC [INFO] dmiscdb02vm02 : orarom Agent
home path is /opt/OracleHomes/core/12.1.0.5.0
2017-Nov-08 23:31:44 UTC [PASS] dmiscdb02vm02 : orarom agent
is configured for autostart
2017-Nov-08 23:31:44 UTC [INFO] Checks if OMS route added to

network files
2017-Nov-08 23:31:44 UTC [FAIL] dmiscdb01vm02: Unable to
determine Platinum Gateway IP address from /etc/hosts file to
check EM Agent route to OMS
2017-Nov-08 23:31:44 UTC [INFO] Checking orarom agent version

2017-Nov-08 23:31:46 UTC [PASS] dmiscdb01vm02: Agent version
(12.1.0.5.0) meets minimum requirements.
2017-Nov-08 23:31:48 UTC [PASS] dmiscdb02vm02: Agent version
(12.1.0.5.0) meets minimum requirements.
2017-Nov-08 23:31:48 UTC [INFO]

**************************************************************
********
2017-Nov-08 23:31:48 UTC [INFO] PERFORMING DATA COLLECTION
2017-Nov-08 23:31:48 UTC [INFO]
**************************************************************
********
2017-Nov-08 23:31:48 UTC [INFO] Performing data collection on

database nodes
2017-Nov-08 23:31:48 UTC [INFO] Collecting data on node
dmiscdb01vm02
2017-Nov-08 23:31:53 UTC [INFO] Logfile:
/u02/patches/PRECHECKS/LOGS/171109045953/dmiscdb01vm02_data_co
llection_171109045953.log
2017-Nov-08 23:31:53 UTC [INFO] Zipfile:
llection_171109045953.zip
2017-Nov-08 23:31:53 UTC [INFO] Collecting data on node
dmiscdb02vm02
2017-Nov-08 23:31:58 UTC [INFO] Logfile:
llection_171109045953.log
2017-Nov-08 23:31:58 UTC [INFO] Zipfile:
llection_171109045953.zip
2017-Nov-08 23:31:58 UTC [INFO] ++++++++++++++++END OF

PRECHECKS++++++++++++++++
2017-Nov-08 23:31:58 UTC [INFO] gv_warning_count=0
2017-Nov-08 23:31:58 UTC [INFO] gv_failure_count=16
2017-Nov-08 23:31:58 UTC [FAIL] Prechecks completed with the

following failures:
2017-Nov-08 23:31:58 UTC [INFO]
**************************************************************
********
2017-Nov-08 23:31:58 UTC [INFO] FAILURE SUMMARY
2017-Nov-08 23:31:58 UTC [INFO]
**************************************************************
********
2017-Nov-08 23:29:54 UTC [FAIL] dmiscdb02vm02: Free space on /
filesystem. Required: 3145728K Available: 3051520K
version
version
2017-Nov-08 23:30:43 UTC [FAIL] EXADATA_DBSRV_VERSION not set
in patch.conf. Skipping YUM prechecks.
2017-Nov-08 23:31:58 UTC [FAIL] Prechecks completed with the
following failures:
2017-Nov-08 23:31:58 UTC [INFO] Log files located in directory
/u02/patches/PRECHECKS/LOGS/171109045953
2017-Nov-08 23:31:58 UTC [INFO] Logs zipped to
/u02/patches/PRECHECKS/LOGS/prechecks_dmiscdb02vm02.in.oracle.
com_171109045953.zip. Please upload this file to parent SR.

Exadata Patch Staging and Prechecks: Solution

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Exadata Patch Staging and Prechecks: Solution

Uploaded by

Copyright:

Available Formats

Exadata Patch Staging and Prechecks

BP - This directory contains the database bundle patch(es) to be applied

ONEOFF - This directory contains any database oneoff patches to be applied

OPATCH - This directory contains the latest opatch zip files

CELL - This directory contains the Exadata cell patch.

Unzip patch 20629335 to the /tmp directory

unzip p20629335_121212_Linux-x86-64.zip -d /tmp

stage.sh, version 17.11.08.18.03

Simply answer yes or no to the prompt at each phase.

 Phase 1: Copy patches from gateway

Sample run of staging script

# ./stage.sh -dbsgroup=dbs_group -patchbase=/u02/patches

2017-Nov-08 22:02:25 UTC [INFO] Verify contents of

Are these the correct nodes? (Yes/No) y

Copy patches from the gateway? (Yes/No): y

2017-Nov-08 22:02:35 UTC [INFO] The script will remove ALL

-rw-r--r-- 1 root root 1113 Oct 18 22:35 patch.conf.sample

Proceed with removing ALL FILES from /u02/patches directory on

2017-Nov-08 22:02:40 UTC [SUCCESS] All files have been removed

2017-Nov-08 22:02:40 UTC [PASS] dmiscdb01vm02: Free space

2017-Nov-08 22:03:18 UTC [INFO] Make sure the password is

Proceed with copying patches? (Yes/No): y

2017-Nov-08 22:03:25 UTC [INFO] Copying patches from

Contact Pratap Unnava (pratap.unnava@oracle.com) for any

** DOM0 - DO NOT REBOOT OR PERFORM ANY ACTION WHICH CAN BREAK

Stage patches to the rest of the cluster? (Yes/No): y

Unzip patches? (Yes/No): y

2017-Nov-08 22:05:17 UTC [INFO] Unzipping

2017-Nov-08 22:05:17 UTC [INFO] Unzipping

2017-Nov-08 22:05:17 UTC [INFO] Unzipping

2017-Nov-08 22:05:17 UTC [INFO] Unzipping

2017-Nov-08 22:06:16 UTC [INFO] Unzipping

2017-Nov-08 22:06:38 UTC [INFO] Unzipping

2017-Nov-08 22:06:41 UTC [SUCCESS] Unzipping completed

2017-Nov-08 22:06:41 UTC [INFO] Next steps:

Before running the precheck.sh script you must do the following:

precheck.sh, version 17.10.31.18.54

-d|--debug Displays verbose and debugging output

A failure summary is displayed at the end of each run.

Sample run of precheck script

# ./precheck.sh -dbsgroup=dbs_group -patchbase=/u02/patches

2017-Nov-08 23:29:53 UTC [INFO] Server Type: DomU

2017-Nov-08 23:29:53 UTC [INFO] Using /u02/patches/patch.conf

2017-Nov-08 23:29:53 UTC [INFO] ga_pass_list cleared

2017-Nov-08 23:29:53 UTC [INFO] Checking dcli utility

2017-Nov-08 23:29:53 UTC [INFO]

2017-Nov-08 23:29:53 UTC [INFO] Checking free space in LV's

2017-Nov-08 23:29:54 UTC [INFO] Checking free space on / for

2017-Nov-08 23:29:54 UTC [INFO] Checking for active snapshots

2017-Nov-08 23:29:54 UTC [INFO] Checking free space on /u01

2017-Nov-08 23:29:54 UTC [INFO] Checking dbnodeupdate.sh

2017-Nov-08 23:29:58 UTC [INFO] Checking for duplicate labels

2017-Nov-08 23:29:59 UTC [INFO] Checking label consistency

2017-Nov-08 23:30:00 UTC [INFO] Checking kernel files

2017-Nov-08 23:30:01 UTC [INFO] Performing HugePage Checks

2017-Nov-08 23:30:02 UTC [INFO] Checking inode usage

2017-Nov-08 23:30:02 UTC [INFO] Checking image status on db

2017-Nov-08 23:30:05 UTC [INFO] Checking IB Interface state is

2017-Nov-08 23:30:05 UTC [INFO] Checks if IB mode equals

2017-Nov-08 23:30:05 UTC [INFO] Checking whether cellip.ora

2017-Nov-08 23:30:05 UTC [INFO] Verify image version matches

2017-Nov-08 23:30:08 UTC [INFO] Checking image status