MOS Patching Best Practices

Oracle Database Patching Best
Practices
MOS
David Marshall
Principal Technical Support Engineer
Database Proactive Support
April 21, 2020
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 3

Program Agenda
1. Types of Patches
2. Patch Delivery Methods
3. Accessing Patches
4. Conflict Resolution
5. Testing Patches
6. Applying Patches
7. Other Patching Related News

Types of Patches

Types of Patches
One Off / Interim Patch
• Fixes a single bug: easy to verify if problem has been fixed, quickly available
• Released with full component level regression testing at Oracle
• Accessible by everyone with a support license
• Low risk of introducing breakages
• Vast majority can be installed with zero or minimal downtime
• Drawback:
– Easy availability leads to tendency to customize environments by combining interim
patches with other patches, causing supportability & maintainability problems
– Not fully tested outside the component or with other fixes
Types of Patches
Security Patch Update (SPU or CPU) for 11.2
• Cumulative Quarterly Patch to fix security vulnerabilities
• Extensively tested at Oracle
• Some fixes can be quite involved and may require post installation steps or
configuration changes
• Historically has been very low risk
• Drawbacks:
– Business wide coordination effort may need to be in place to manage fast roll-out.
– Because the SPU does not contain fixes for high impact non-security bugs
encountered by customers, a mission critical system will almost certainly need to
combine the SPU with one-offs, increasing overall risk.
– Not available from 12c forward
Types of Patches
Database Patch Set Update (DB PSU) for 11.2 or 12.1
• Cumulative Quarterly Patch to fix most recent high impact bugs
• Contains Security Content
• Strict content inclusion criteria:
– No optimizer changes, must be RAC rolling installable, DG Standby First Installable,
fixes already tested by customers
– Low level volume of content
• Low risk
• Drawback:
– Strict content inclusion restrictions can sometimes lead to need to customize
environments with one-off patches as well.
Types of Patches
Grid Infrastructure Patch Set Update (GI PSU) for 11.2 or 12.1
• Contains Everything the GI Needs

1. DB PSU
2. OCW (Oracle ClusterWare) PSU
3. ACFS (ASM Cluster File System) PSU
4. DBWLM (DataBase WorkLoad Management) PSU
• 1 and 2 Updates Both DB Homes and Grid Home
• 3 and 4 Updates Only Grid Home
• Cumulative

Types of Patches
Proactive Bundle Patch (BP) for 12.1
• Previously known as "Database Patch for Engineered Systems and Database
In-Memory"
• Cumulative Quarterly Patch to fix high impact bugs for a given configuration
(e.g. Exadata and installations using in-memory)
– Contains PSU Content
• Recommended over PSU
• Content inclusion criteria to address stabilization needs of majority of
customers running this configuration:
– Also optimizer changes, must be RAC rolling installable, DG Standby First Installable.
– Some fixes are getting released for the first time within the BP
Types of Patches
Proactive Bundle Patch (BP) for 12.1
• A little higher risk than other patches
• Drawbacks:
– Less restriction on content leads to higher volumes of fixes; Optimizer changes and
potential “first time” patches.
– More testing needed

New Oracle Database Release and Update Model in 12.2
Summary
• Annual feature release of Oracle Database software
– Improve quality by reducing the number of software changes released at one time
• Quarterly Release Updates (Updates) and Release Update Revisions (Revisions)
– Updates are Proactive, highly tested bundles of critical fixes which enable customer to avoid known
issues.
• Contains both security-related fixes and high-priority non-security fixes to each Annual feature release
– Revisions contain security and regression fixes to an Update
• Extends an Update with fixes for known regressions plus includes the latest security vulnerability fixes
– Updates and Revisions ship on the same January, April, July, October proactive patch schedule as today
– Can switch back and forth between Updates and Revisions
• Release Numbering changes; 12.2.0.2 becomes 18

Version Number Timeline for Release 19 Updates and
Revisions
Production Q2 Q3 Q4 Q1 Q2
19.1.0 19.2.0 Update
19.2.1 Revision #1
19.2.2 Revision #2
Note: No revisions planned for initial production release

Revisions
19.1.0 19.2.0 19.3.0 Update
19.2.1 19.3.1 Revision #1
19.2.2 19.3.2 Revision #2

Revisions
19.1.0 19.2.0 19.3.0 19.4.0
19.2.1 19.3.1 19.4.1
19.2.2 19.3.2 19.4.2

15
Oracle recommends Updates
• minimize the chance of encountering known bugs and security
vulnerabilities.
• Less need for separate interim one-off patches
– Numerous interim one-off patches can lead to unique software baselines and the
potential for ongoing costly patch maintenance.
19.1.0 19.2.0 19.3.0 19.4.0 19.5.0 19.6.0
19.2.1 19.3.1 19.4.1 19.5.1
19.2.2 19.3.2 19.4.2

Alternative: Test/Dev 3-6 Months Ahead of Prod
– If you have reached a stable state, and want to prioritize security updates over
functional fixes. In such cases, consider Revisions.
• With Revision Release.Update.1 - lag with the Update content by 3 months.
• With Revision Release.Update.2 - lag with the Update content by 6 months.
• Pros / Cons
– a more conservative approach to Database software maintenance
– risk the chance of hitting known issues fixed in the most recent Update.
19.1.0 19.2.0 19.3.0 19.4.0 19.5.0 19.6.0
19.2.1 19.3.1 19.4.1 19.5.1
19.2.2 19.3.2 19.4.2

Types of Patches
QFSD
• Cumulative Quarterly Full Stack download for Exadata BP

– Everything Exadata Needs
• Cumulative Quarterly Full Stack download for SuperCluster BP
– Everything SuperCluster Needs

Common to All Types of Patches
Rollback
• All patches can be rolled back
• Rollback may be a 2 step process
1. Rollback the changes to the ORACLE_HOME
a) Run opatch(n)rollback OR
b) Restoring the ORACLE_HOME from backup
2. Rollback the changes from all databases associated to the ORACLE_HOME
a) Execute Scripts which are provided to undo SQL patch changes within patches
b) Run datapatch for 12c, 18c, 19c and beyond

Rollback
• Can the patch be rolled back from the database without rolling back the
ORACLE_HOME? - Yes, but not easily
• Why not just run the rollback scripts?
1. The SQL being executed is the SQL that was installed by the patch
Therefore, nothing will be rolled back as the same SQL for the apply is being run for the rollback
2. The binaries being executed are the binaries that were installed by the patch
Therefore, the binaries being used when the db is reopened are the binaries that were installed by the
patch which means that the binaries have the code changes from the patch

• How can the patch be rolled back from the database without rolling it back
from the ORACLE_HOME?
• Scenario:
– 19c ORACLE_HOME (OH1) has the DB RU installed
– OH1 has 2 database associated to it (DB1 and DB2)
– The RU needs to be rolled back from DB1 only
• Steps necessary
1. Clone OH1 to a new home (OH2)
2. Associate DB1 to the new home (OH2)
3. Perform the complete rollback on OH2 and DB1
4. Leave DB1 on OH2 until it is desired to reapply the patch (e.g. the Release Update)

Patch Delivery Methods

• Oracle Database - Overview of Database Patch Delivery Methods (Doc ID
1962125.1 and 2337415.1 )
– Which Patching Method to Use?
Oracle makes the following recommendation for which patch method to use for Database related
installations:
• Every customer should at least install PSUs. Minimal testing required.
• 12.1 Customers wanting a more comprehensive set of fixes should install the Database Proactive
Bundle patch. This requires a bit more testing than a Patch Set Update (PSU), but delivers a larger set
of fixes.
• 12.2, 18c, 19c and beyond – the new Release Update method

1962125.1 and 2337415.1 )

1962125.1 and 2337415.1 )

1962125.1 and 2337415.1 )

1962125.1 and 2337415.1 )

• Footnote 1
The "Database Proactive Bundle Patch" requires a bit more testing than a Patch Set
Update (PSU) as it delivers a larger set of fixes.
• October 2016
– Patch Set Update – 418 fixes
– Database Proactive Bundle Patch - 1405 fixes
• In all cases, also need to install the relevant OJVM Release Update
• Install interim patches only for specific issues
– Keep the number of interim patches installed to a minimum.

Accessing Patches

Downloading patches with My Oracle Support (MOS)
• Find the patch(es) by performing one of the following
– a Simple search for a patch or a group of patches
– an Advanced search
– a Saved search
– a Recent search
– can often be downloaded directly from a MOS NOTE

MOS Patch Search Screen





Patchset can be downloaded directly from a MOS NOTE
– for example, from the primary document that is issued each cycle, called the Patch
Set Update and Critical Patch Update [month year] Availability Document or PAD.

Conflict Resolution

Patch Conflicts
• Patches conflict because they are updating one or more common objects
or files
• OPatch checks every patch being applied for conflicts
• Example of OPatch output when conflict occurs
– Following patches have conflicts: [ 9482399 9654983 ]
• Both update
– $ORACLE_HOME/lib/libserver11.a (kdlx.o)

How are Conflicts Resolved
• Merge patch
– The fixes in all conflicting patches are merged into a single patch
– Merge patch replaces new and existing patch
• Overlay patch
– Create a new version of the patch
– Only the actual code that is in conflict is merged
– Installed on top of the existing patch

Conflict Resolution in MOS 14.2 (Prior version)
1
2 4
3
5
10

MOS Conflict Checker
• A conflict resolution tool is available from the patch Search and Detail
screen
• Self Service tool that doesn’t require an SR to be logged
• Upload your OPatch inventory to resolve conflicts
• Resolution patches that are available are provided immediately
• Resolution requests are automatically filed if they do not exist
• Document 1091294.1 "How to use the My Oracle Support Conflict Checker Tool"

Patch’s Detail

MOS Patch Conflict Checker
1
2 3












Testing Patches

Categorize Your Tests
The Major Categories
• Functional Regression Tests

– Purpose is to check if application flows and administrator activities behave as
expected.
– These tests should be repeatable
– Catalog all activities to test for and create tests to represent these.
– Save expected results.
– Run tests against the newly patched version. Compare new results against expected
results.
– Divide tests into groups based on functional area so you can target runs depending on
patch.

Categorize Your Tests
The Major Categories
• Load Tests
– Check how the system or application behaves under production load.
– You may also want to simulate load beyond limits of normal operation for mission
critical systems.
– Key outcome is that availability should not be affected.
• Performance Tests
– Define key performance indicators to measure against for application flows. Examples
are responsiveness and throughput.
– Measuring against these, no material negative differences should be seen.

Applying Patches

Best Practices for Applying Patches
• Aim for end to end automation of the patch application process. Write scripts to do this
if necessary.
• Ensure you have all prerequisites in place for patch application to complete without
error
• If you have multiple patches to apply at a time, apply them in one downtime
• The vast majority of patches (>98%) can be installed in a highly available manner. Check
the patch README for what is possible
– RAC Rolling
– Dataguard Standby First
– Online patches: patch a running Database
– All patches can be applied Out of Place : patch a Cloned ORACLE_HOME.
• Afterwards check that patch has installed correctly

Prerequisite Checking
1. Check if there is enough system space to install the patch
opatch prereq CheckSystemSpace
2. Check the OPatch version against the required version for the patch.
opatch prereq CheckMinimumOPatchVersion
3. Check if the patch will conflict with already installed patches

opatch prereq CheckConflictAgainstOHWithDetail
4. Check if multiple patches will conflict with each other

opatch prereq CheckConflictAmongPatches
5. If applicable, run cvu and exachk where applicable

README - Installation commands based on Patch Packaging
• System Patches
• ‘opatchauto’ automatically installs “System Patches”, and handles post-install actions too
– Install with Command : opatchauto apply
• Used for Exadata and GI bundles
• Depending on command line options specified, one invocation of opatchauto can patch the GI
home, Oracle RAC database homes, or both GI and Oracle RAC database homes
• Singleton Patches
• Used for singleton / interim patches
– Install with Command : opatch apply
– Other options :
• -silent : parameters are passed via the response file ‘-ocmrf’

Optimizing patch application
• Apply several patches during the same downtime
– Unzip the patches in the same patch location
– Execute opatch napply <patch_location> -skip_subset -skip_duplicate
• Don’t mix bundle patches and overlays (ordering issue). Should apply BP
first, then overlays. Will be fixed in a later OPatch release.
• In 12c, 18c, 19c, and beyond, datapatch takes care of applying the post
SQL

Running ‘datapatch’ with Database 12c
• Ensures data/sql related changes are patched
• Takes care of installing/rolling back data changes
• Steps
• After ‘OPatch apply’, Connect to the DB
SQL> Connect / as sysdba
SQL> startup
SQL> alter pluggable database all open; [only for Multitenant DB]
SQL> quit
% $ORACLE_HOME/OPatch/datapatch [-verbose]

Running ‘datapatch - verbose’ with Database 12c
$ datapatch -verbose
SQL Patching tool version 12.1.0.2.0 Production on Tue May 23 17:11:54 2017
Copyright (c) 2012, 2016, Oracle. All rights reserved.
Log file for this invocation:

$ORACLE_BASE/cfgtoollogs/sqlpatch/sqlpatch_572_2017_05_23_17_11_54/sqlpatch_inv
ocation.log
Connecting to database...OK
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...

Running ‘datapatch - verbose’ with Database 12c
Bootstrapping registry and package to current versions...done
Determining current state...done
Current state of SQL patches:

Patch 25437695 (Database PSU 12.1.0.2.170418, Oracle JavaVM Component (APR2017)):
Installed in the binary registry only
Bundle series PSU:
ID 170418 in the binary registry and not installed in the SQL registry
Adding patches to installation queue and performing prereq checks...

Installation queue:
Nothing to roll back
The following patches will be applied:
25437695 (Database PSU 12.1.0.2.170418, Oracle JavaVM Component (APR2017))
25171037 (DATABASE PATCH SET UPDATE 12.1.0.2.170418)
Installing patches...
Patch installation complete. Total patches installed: 2

Verifying Binary Installation with OPatch
• OPatch Commands
– opatch lsinventory or opatch lsinv
• List the patches installed in the oracle home
• Other Options
– -all_nodes: Report the patches installed on given Oracle Home in all nodes of RAC system
– -detail(s): Display the components and the list of patches with their associated files
– -xml : Generate xml formatted output
– opatch lspatches
• List the installed patches and their description
• Other options:
– -bugs : Lists bug fixed by each patch
– -verify : Verifies if specified patch is installed in the oracle home

Checking Installed Registry Patches via PL/SQL
Database 12c, 18c, 19c, and beyond: Queryable Patch Inventory

DBMS_QOPATCH subprograms
• dbms_qopatch provides access to the OPatch
inventory information from within the database
GET_OPATCH_LIST
– PLSQL/SQL interface to view
• list of patches applied GET_OPATCH_LSINVENTORY
• check if a particular patch is applied IS_PATCH_INSTALLED
• patch inventory across RAC nodes

GET_SQLPATCH_STATUS
• SQL patch status
• Check datapatch entry in registry table : ‘select *
from dba_registry_sqlpatch;’

Other Patching Related News

Multitenant for Simplified Patching
Patching the Container Database Results in Patching all of its Pluggable
Databases
3) Startup &
Apply Post
Install on all
Databases
2) Update the
1) Shutdown ORACLE_HOME
the database

Large-Scale Enterprise Quarterly Patching Options
1. DB Cloning using EM12c Provisioning
DB Provisioning - Deploy Gold Images to the Cloud
• Mass Deployment of Oracle Software (Database, Real Application Clusters)
• http://download.oracle.com/technology/products/oem/screenwatches/em12/12g_Provi
sioning_GI_and_RAC/12g_prov_rac_viewlet_swf.html
2. EM12c Patch Management Solution

End to End Patch Automation Solution for Oracle Databases
• Oracle Database Advisor Webcast Schedule and Archive recordings (Doc ID
1456176.1)
• Look for the 56 minute video entitled “Automation of patch conflict resolution and
deployment through EM 12c”

3. EM Database Lifecycle Management Pack
Cloud Control plug-in solution that automates the time-consuming tasks related to discovery,
initial provisioning and cloning, patching, configuration management, ongoing change
management, and compliance management.
http://www.oracle.com/technetwork/oem/lifecycle-mgmt-
495331.html
4. Oracle Rapid Home Provisioning

Automating Patching for Cloud
• Eliminate the need to patch individual Databases
• Update any number of Databases with a single command
• http://www.oracle.com/technetwork/database/database-cloud/private/rapid-home-
provisioning-2405191.pdf

5. Automation of minimal downtime patching in a Dataguard

environment.
Introduced in Database 12.1.0.2, uses the dbms_rolling package to patch a “Leading
Group” (aka a new Primary Database) and then switch over applications from the “Trailing
Group” (aka the actual Primary Database) to point to the “Leading Group”.
Reference: Data Guard Concepts and Administration
Chapter 14 Using DBMS_ROLLING to Perform a Rolling Upgrade

Additional References
• Oracle Database - Overview of Database Patch Delivery Methods (Doc ID 1962125.1 and
2337415.1)
• Composite Patches for Oracle Products [Video] (Doc ID 1376691.1)
• Database Patch Conflict Resolution (Doc ID 1321267.1)
• My Oracle Support Patch Conflict Checker Overview [VIDEO] (Doc ID 1941934.1)
• How to Use the My Oracle Support Conflict Checker Tool for Patches Installed with
OPatch [Video] (Doc ID 1091294.1)
• Datapatch: Database 12c Post Patch SQL Automation (Doc ID 1585822.1)
• Release Update Introduction and FAQ (Doc ID 2285040.1)
• Latest PAD (as of this presentation) (Doc ID 2534806.1)


MOS Patching Best Practices

Uploaded by

Copyright:

Available Formats

You might also like

MOS Patching Best Practices

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

MOS Patching Best Practices

Uploaded by

Copyright:

Available Formats

Oracle Database Patching Best

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 3

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 4

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 5

• Contains Everything the GI Needs

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 9

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 11

• Release Numbering changes; 12.2.0.2 becomes 18

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 12

Note: No revisions planned for initial production release

19.2.2 19.3.2 Revision #2

Note: No revisions planned for initial production release

Note: No revisions planned for initial production release

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 16

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 17

• Cumulative Quarterly Full Stack download for Exadata BP

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 18

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 19

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 20

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 21

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 22

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 23

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 24

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 25

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 26

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 27

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 28

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 29

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 30

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 31

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 32

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 33

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 34

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 35

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 36

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 37

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 38

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 39

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 41

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 42

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 44

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 45

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 46

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 47

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 48

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 49

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 50

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 51

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 52

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 53

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 54

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 55

• Functional Regression Tests

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 56

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | 57