ADOP Patching Gotchas!

What can and will go wrong when patching EBS

12.2.X and how to fix it!
Prepared by: Oti Ometie – Oracle Applications DBA
@OhOmetie

Session ID: 10175

Agenda
• Introduction
• Overview of ADOP
• Patching Multiple Nodes
• Patching a single node in a multi-node configuration
• Troubleshoot and recover from a failure.
• Perform cleanup operations after a failed patching cycle
• Dropping Old Editions
• Impact of DR
• Extra stuff
• Summary
• Q&A
April 2-6, 2017 in Las Vegas, NV USA #C17LV 2
About me
• 17+ years of experience with Oracle products
• 15 years experience as an Oracle DBA
• Oracle Certified Expert, OCP 9i,10G,11G, EBS R12
• Worked as a DBA on 3 continents in a variety of
industries (Banking, Consulting, Construction,
Manufacturing)
• Martial artist and crazy about Obstacle Course Races –
Tough Mudder, Mud hero, Spartan…

April 2-6, 2017 in Las Vegas, NV USA #C17LV 3

About New Flyer Industries

• The largest transit bus and motor coach

manufacturer and parts distributor in North America
• Fabrication, manufacturing, distribution and service
centers across Canada and the United States
• Employs approximately 5,000 team members
• Multi Billion dollar revenue
• Completed a 9 month upgrade from 11i to 12.2.4 in
September 2015
April 2-6, 2017 in Las Vegas, NV USA #C17LV 4
Overview of ADOP
Overview of ADOP

• AD Online Patching is the ONLY way to apply patches in

EBS from Version 12.2
• Patching is carried out while users are online and the
application is in use.
• ADOP phases: prepare, apply, finalize, cutover, cleanup.
• Small downtime required for when the flip of the run and
patch file systems to occur

April 2-6, 2017 in Las Vegas, NV USA #C17LV 6

ADOP Patching Cycle

April 2-6, 2017 in Las Vegas, NV USA #C17LV 7

Simple Commands
adop phase=prepare
adop phase=apply patches=123456
adop phase=finalize
adop phase=cutover
adop phase=cleanup

April 2-6, 2017 in Las Vegas, NV USA #C17LV 8

More commands
adop phase=prepare
adop phase=apply workers=24 patches=123456, 98765
adop phase=finalize
adop phase=cutover cm_wait=2
adop phase=cleanup

April 2-6, 2017 in Las Vegas, NV USA #C17LV 9

Still more …
adop phase=prepare
adop phase=apply workers=24 patches=123456, 98765
adop phase=finalize,cutover cm_wait=1
adop phase=cleanup

April 2-6, 2017 in Las Vegas, NV USA #C17LV 10

Patching Multiple Nodes
Patching Multiple Nodes

• Ensure that the patch(es) is/are available on the

$PATCH_TOP of all nodes.
• Run the commands you would run if you were patching a
single node. The commands will be run sequentially on
all nodes
• Patches are applied on the primary node before being
applied on other nodes.

April 2-6, 2017 in Las Vegas, NV USA #C17LV 12

Patching Multiple Nodes

• Patching is non interactive i.e. you need to specify the

patch(es)
Which of the following is right?
• adop phase=apply patches=123456
• adop phase=apply

April 2-6, 2017 in Las Vegas, NV USA #C17LV 13

Patching a single node in a
multi-node configuration
Patching a single node in a multi-node
configuration

• Run adop phase=prepare on the primary node

• Apply the patch to the node using:
• PRIMARY NODE- adop phase=apply patch=1234 allnodes=no
action=db
• SECONDARY NODES- adop phase=apply patch=1234
allnodes=no action=nodb
• Continue as usual – finalize, cutover

April 2-6, 2017 in Las Vegas, NV USA #C17LV 15

Patching a single node in a multi-node
configuration

• Commands I’ve tried out and work for different

situations
adop phase=prepare allnodes=no action=db
adop phase=cutover allnodes=no action=db cm_wait=1

April 2-6, 2017 in Las Vegas, NV USA #C17LV 16

Patching a single node in a multi-node
configuration
• You cannot run a prepare phase on a non-admin node.
adop phase=prepare allnodes=no action=nodb
The above command will fail with the error below:

April 2-6, 2017 in Las Vegas, NV USA #C17LV 17

Troubleshoot and recover from
a failure
Troubleshooting the error
adop –status –detail

adop phase=apply patches=xyz analytics=yes

April 2-6, 2017 in Las Vegas, NV USA #C17LV 19

Recover from a failure
• Examples
adop phase=apply patches=123456 abandon=no restart=yes
flags=autoskip
adop phase=apply patches=21473943 allnodes=no action=nodb
abandon=no restart=yes flags=autoskip

April 2-6, 2017 in Las Vegas, NV USA #C17LV 20

If all else fails…
• ABORT
• An abort is always my last resort, it means pulling the
plug on all the patches that have been applied.
• Command:
• adop phase=abort

April 2-6, 2017 in Las Vegas, NV USA #C17LV 21

Aborting a patch cycle

April 2-6, 2017 in Las Vegas, NV USA #C17LV 22

Aborting a patch cycle
• The abort command terminates the patching cycle.
• All actions being performed are discarded.
• After an abort is performed a cleanup is required and will
be performed during the next adop prepare phase if not
initiated by the DBA.
• Command:
• adop phase=abort, cleanup
• adop phase=abort,cleanup cleanup_mode=full
• adop phase=fs_clone

April 2-6, 2017 in Las Vegas, NV USA #C17LV 23

Perform cleanup operations
after a failed patching cycle
Perform cleanup operations after a failed
patching cycle

• A cleanup is required after every failed patching session.

• Failure to explicitly run the cleanup command means
that your next patching cycle will be delayed while the
prepare phase kicks off a cleanup
• Commands:
• adop phase=cleanup
• adop phase=cleanup cleanup_mode=full

April 2-6, 2017 in Las Vegas, NV USA #C17LV 25

Dropping Old Editions
Dropping Old editions
• After each online patching cycle completes, the database
stores the previous edition.
• After a certain point this begins to impact performance
• Oracle recommends that when the number of editions
reaches 25 it’s time to purge/drop old editions

April 2-6, 2017 in Las Vegas, NV USA #C17LV 27

Steps to Drop Old Editions
• Commands:
adop phase=prepare
adop phase=actualize_all
adop phase=finalize finalize_mode=full
adop phase=cutover
adop phase=cleanup cleanup_mode=full

April 2-6, 2017 in Las Vegas, NV USA #C17LV 28

Impact of DR – Dataguard
Standby Databases
Impact of DR – Dataguard Standby Databases

[WARNING] adop has detected a configured disaster recovery site.

[WARNING] Follow the instructions in the section "Oracle E-Business Suite
[WARNING] Maintenance with Standby Database" of Business Continuity for
[WARNING] Oracle E-Business Suite Release 12.2 depending on the database version
used.
Do you want to continue with the prepare phase [Y/N] ?

Note: The error above has been fixed see Doc ID 2131833.1

April 2-6, 2017 in Las Vegas, NV USA #C17LV 30

Impact of DR – Dataguard Standby Databases

• If the standby database is available while a patch is being

applied, the changes on the primary will be automatically
propagated to the standby.

April 2-6, 2017 in Las Vegas, NV USA #C17LV 31

Impact of DR – Dataguard Standby Databases

• Doc ID 1491855.1: Section 7 goes into details

• If the standby is running during a patch application, changes
will be automatically replicated on the standby.
• Oracle recommends that before cutover:
1. Stop recovery on the standby database
SQL>alter database recover managed standby database cancel;

2. Stopping the shipping of redo on the Primary Database

SQL>alter system set log_archive_dest_state_<n>=defer;

April 2-6, 2017 in Las Vegas, NV USA #C17LV 32

Extra stuff
Patch Wizard

April 2-6, 2017 in Las Vegas, NV USA #C17LV 34

Abandoning a node
• When a node in a multi-node architecture is inaccessible
during an online patching cycle, it will be abandoned.
• If an adop patch fails on a secondary node. if you choose
to proceed with the cutover, node will be marked as
abandoned.
Do you want adop to continue with other completed nodes [y/n]
• Select “n”, resolve the issue and run something similar
to:
adop phase=apply patches=123456 abandon=no restart=yes
flags=autoskip allnodes=no action=nodb

April 2-6, 2017 in Las Vegas, NV USA #C17LV 35

Examples
• If you need help and are unable to access online
resources use examples.

adop -examples

April 2-6, 2017 in Las Vegas, NV USA #C17LV 36

Summary
• AD Online Patching is the ONLY way to apply patches in
EBS from Version 12.2
• Patching is carried out while users are online and the
application is in use.
• When patching multiple nodes ensure that the patch is
copied to the $PATCH_TOP of all nodes.
• Patching multiple nodes is non-interactive
• You indicate you want to patch a single node using the
“allnodes=no” argument and specify if it’s the primary or
secondary node using action=db/nodb
April 2-6, 2017 in Las Vegas, NV USA #C17LV 37
Summary cont.
• You can troubleshoot by using the analytics=yes
argument and –status –details argument
• Use the autoskip argument to recover from pesky errors
and continue with the session on the node with the error
using the allnodes=no argument.
• If all else fails – ABORT
• Running a cleanup before your next patch cycle saves
time
• Drop old editions to improve performance – 25 is a good
number
• Shutdown/defer Dataguard standby before cutover
April 2-6, 2017 in Las Vegas, NV USA #C17LV 38
References
• Oracle E-Business Suite Maintenance Guide Release 12.2
– Patching procedures
• Business Continuity for Oracle E-Business Suite Release
12.2 Using Oracle 11g (11gR2) Physical Standby Database
(Doc ID 1491855.1)
• Steven Chan’s blog:
https://blogs.oracle.com/stevenChan/entry/op_faq2

April 2-6, 2017 in Las Vegas, NV USA #C17LV 39

Please Complete Your
Session Evaluation
Evaluate this session in your COLLABORATE app.
Pull up this session and tap "Session Evaluation"
to complete the survey.

Session ID: 10175

Oti Ometie
LinkedIn: linkedin.com/in/otiometie
Email: ometie@gmail.com