Deep Dive Into: Parallel Database Upgrades

Deep Dive into
Parallel Database Upgrades

With or Without Oracle Multitenant
Mike Dietrich
Master Product Manager
Database Upgrade
Oracle Corporation
Updated: 8-DEC-2015
Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Deep Dive into Parallel Database Upgrades 2
$> whoami
6 years
RDBMS Core & Mission Critical Support
5.5 years
Technology Presales for DataGuard, Upgrades
>7.5 years
Mike Dietrich ST Upgrade Development Team
Master Product Manager
Database Upgrades & Migrations
MikeDietrichDE
50% 50% + x%
Reference Workshops Development
http://blogs.oracle.com/UPGRADE Projects Worldwide Work
Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Why you need to upgrade NOW to Oracle Database 3
12c!
Download Slides from the Database Upgrade Blog
 http://blogs.oracle.com/UPGRADE
Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Why you need to upgrade NOW to Oracle Database 12c! 4
Deep Dive into Parallel Database Upgrades
1 Database Upgrade News
2 Oracle Multitenant Overview
3 Multitenant Upgrade
4 Inside catctl.pl – and Options
5 Performance Figures
6 Wrap Up
Upgrade SQL Automation
New Pre-Upgrade Script
 preupgrd.sql
 Executes pre-upgrade checks
 Runs in source environment
 Generates fixup scripts
 Get the newest version:
MOS Note:884522.1
preupgrd.sql: MOS Note:884522.1
preupgrd.sql
preupgrd.sql => utluppkg.sql
TEXT
upgrade.xml
preupgrade.log
preupgrade_fixups.sql
postupgrade_fixups.sql
DBUA
$ORACLE_BASE/cfgtoollogs/sid/preupgrade
40%
Faster Upgrades in
Oracle Database 12c
Faster Upgrade – Less Downtime
New Parallel Upgrade catctl.pl
 catctl.pl
script1.sql script2.sql script3.sql
 Runs database upgrade in parallel
– Only for ORACLE SERVER
script4.sql script7.sql script9.sql
 Used and proven by selected Oracle

Database 11g global customers script5.sql script8.sql script10.sql
– Telco billing
script6.sql script11.sql
– >100 SAP systems

– Large DWH script6.sql
“The new parallel upgrade script promises to drastically
reduce downtime due to planned maintenance. We saw a
37% improvement over the previous upgrade process in our
environment.”
Harald Stefan
Leiter Datenbanken
Payback GmbH
New Parallel Upgrade

$> $ORACLE_HOME/perl/bin/perl catctl.pl catupgrd.sql
6 Wrap Up
Three possible database deployments in Oracle 12c
 Stand Alone  Single Tenant  Multitenant
– Same as in previous releases – No extra license required – Requires Multitenant license
– Possible with SE and EE – Requires Enterprise Edition
PDB$
SEED
PDB1
PDB$
SEED
PDB1 PDB1 ...
DB CDB$ROOT CDB$ROOT
Why you will have to play with Oracle Single/Multitenant
Oracle Multitenant – Concepts
PDB$
SEED
PDB1 PDB2 PDB3
CDB1_MUL
PDB$
SEED
PDB1 PDB2
CDB2_MUL
Oracle Multitenant – Sharing Resources
spfile Redo Control Flashback
PDB$
SEED
PDB1 PDB2 PDB3
CDB
CDB-PDB: Who's who?
 Data dictionary and objects
OBJ$ TAB$ SOURCE$ OBJ$ TAB$ SOURCE$

EMP DEPT PDB2 EMP DEPT
PDB1
PDB$SEED
OBJ$ TAB$ SOURCE$
CDB$ROOT
Why does a PDB require an upgrade?
 Each PDB has its own Data Dictionary
– The documentation states:
 Comparing OBJ$ in CDB$ROOT
versus PDB
6 Wrap Up
Upgrade: Everything at once vs Unplug/Plug
 Everything at Once  Unplug/Plug
PDB2 PDB3
Upgrade: Everything at once
PDB$
SEED
PDB1 PDB2 PDB3 PDB4
preupgrd.sql
CDB1 – Oracle 12.1.0.1
UPGRADE PDB$ UPGRADE

UPGRADE
PDB1 PDB2 PDB3 PDB4
SEED
UPGRADE UPGRADE
UPGRADE CDB1 – Oracle 12.1.0.2
Upgrade: Everything at once – Step by Step – 1/2
 Preupgrade.sql: PDB$
SEED
PDB1 PDB2 PDB3 PDB4
 Copy new preupgrd.sql and utluppkg.sql preupgrd.sql
preupgrade_fixups.sql
into CBD1's $ORACLE_HOME/rdbms/admin
 SQL> ALTER PLUGGABLE DATABASE ALL OPEN; CDB – Oracle 12.1.0.1
 $ORACLE_HOME/perl/bin/perl
$ORACLE_HOME/rdbms/admin/catcon.pl -n 1
-d $ORACLE_HOME/rdbms/admin
-l /home/oracle/mike -b preupgrd preupgrd.sql
 In CDB – Oracle 12.1.0.1:

-d $ORACLE_HOME/cfgtoollogs/cdbupgr/preupgrade
-l /home/oracle/mike -b preupgrade_fixups preupgrade_fixups.sql
preupgrd.sql in Everything at Once
 preupgrd.sql runs in every container PDB$
PDB1 PDB2 PDB3 PDB4
– CDB$ROOT SEED
preupgrd.sql
– PDB$SEED
– All PDBs CDB – Oracle 12.1.0.1
 Executed with catcon.pl
– catcon.pl –n 1 –e –b preupgrade –d '''.''' preupgrd.sql
 Results get logged to preupgrade0.log
 Specific changes for each container
Upgrade: Everything at once – Step by Step – 2/2
 In CDB – Oracle 12.1.0.2:
 SQL> STARTUP UPGRADE
 SQL> ALTER PLUGGABLE DATABASE ALL OPEN UPGRADE;
 cd $ORACLE_HOME/rdbms/admin
catctl.pl -d $ORACLE_HOME/rdbms/admin -n 16 –M
-l /home/oracle/mike catupgrd.sql
 See $ORACLE_HOME/cfgtoollogs/<SID>/upgrade/upg_summary.log
UPGRADE PDB$ UPGRADE
PDB2 PDB3 PDB4 UPGRADE
PDB1
 SQL> STARTUP UPGRADE SEED
UPGRADE
 SQL> ALTER PLUGGABLE DATABASE ALL OPEN; postupgrade_fixups.sql
UPGRADE Oracle 12.1.0.2 CDB –
-d $ORACLE_HOME/cfgtoollogs/cdbupgr/preupgrade
-l /home/oracle/mike
-b postupgrade_fixups postupgrade_fixups.sql
 $ORACLE_HOME/perl/bin/perl catcon.pl -n 1 -e -b utlrp -d '''.''' utlrp.sql
Upgrade: One (or many) at a time
PDB$
preupgrd.sql
SEED
PDB1 PDB2 PDB3
PDB1
xml
PDB$
UPGRADE PDB1
SEED
Upgrade: One (or many) at a time – Step by Step – 1/2
 In CDB1:
Copy new preupgrd.sql and utluppkg.sql into preupgrade_fixups.sql
preupgrd.sql
PDB$
PDB1 PDB2 PDB3
SEED
CBD1's $ORACLE_HOME/rdbms/admin
 SQL> alter session set container=PDB1;
 SQL> @?/rdbms/admin/preupgrd.sql CDB1 – Oracle 12.1.0.1
 SQL> @/u01/app/oracle/cfgtoollogs/CDB1/
preupgrade/preupgrade_fixups.sql
 SQL> alter sesstion set container=CDB$ROOT; PDB1
 SQL> alter pluggable database PDB1 close; xml
 SQL> alter pluggable database PDB1 unplug
into '/stage/pdb1.xml';
 SQL> exit
Plugin Compatibility Check
 Not always as helpful as intended
– COMPATIBLE  NO ?!?
SET SERVEROUTPUT ON
PDB1 DECLARE
compatible CONSTANT VARCHAR2(3) :=
CASE DBMS_PDB.CHECK_PLUG_COMPATIBILITY(
PDB1
pdb_descr_file => '/data/pdb1.xml',
xml pdb_name => PDB1')
WHEN TRUE THEN 'YES' ELSE 'NO'
PDB$ END;
SEED
PDB1
BEGIN
DBMS_OUTPUT.PUT_LINE(compatible);
CDB END;
/
Upgrade: One (or many) at a time – Step by Step – 2/2
 In CDB2:
In SQL*Plus:
– SQL> alter session set container=CDB$ROOT;
– SQL> create pluggable database PDB1 using
'/stage/pdb1.xml' file_name_convert=(
'/oradata/CDB1/pdb1', '/oradata/CDB2/pdb1');
– SQL> alter pluggable database PDB1 open upgrade; PDB1
– SQL> exit PDB1
On the command prompt: xml
– $> cd $ORACLE_HOME/rdbms/admin
PDB$
– $> $ORACLE_HOME/perl/bin/perl catctl.pl UPGRADE
SEED
PDB1
–c "PDB1" catupgrd.sql
Back in SQL*Plus:
– SQL> alter session set container=PDB1; Oracle 12.1.0.2 CDB2 –
– SQL> startup
– SQL> @/u01/app/oracle/cfgtoollogs/CDB1/preupgrade/postupgrade_fixups.sql
– SQL> @?/rdbms/admin/utlrp.sql
PDB_PLUG_IN_VIOLATIONS
 If you plug in and don't upgrade your PDB can be opened RESTRICTED only
 Check PDB_PLUG_IN_VIOLATIONS for required actions
6 Wrap Up
catctl.pl
 Performed in phases
 Determined at run time by
parsing catctl tags within the SQL files
 Each phase builds dependencies for the
next phase
 Each phase must complete before moving
onto the next phase
 Run in serial or in parallel
One Interface Used By All
 Since Oracle 12.1.0.2 we broke out catctl.pl

catctl.pl into a common catcon.pl
shared library called catcon.pm sqlpatch.pl
 catcon.pm used in
– catctl.pl
– datapatch (sqlpatch.pl) catcon.pm
– catcon.pl
 catctl.pl calls packages within
catcon.pm to perform the
upgrade DATABASE
How Upgrade Works in a Multitenant Database
 Upgrade CDB$ROOT first

CATCTL.PL
– Exit if there is an error
 Upgrade PDBs next
UPGRADE EXIT IF
CDB$ROOT ERROR
UPGRADE UPGRADE
PDB1 PDB2
How Upgrade Works in a Multitenant Database
 Main catctl.pl process MAIN PROCESS

– Starts child processes CATCTL.PL
– Waits for child processes to complete
 Child processes upgrade PDBs
– Independent from main
CHILD CHILD
PROCESS 1 PROCESS 2
 Process order by CON_IDs CATCTL.PL CATCTL.PL
PDB1 PDB2
Multiple PDB Upgrade
 catctl.pl -n $ORACLE_HOME/perl/bin/perl catctl.pl -n 24 catupgrd.sql

– Multiple PDB upgrades controlled by -n
– Number of PDBS upgraded in parallel
 Divided by 2 – rounded down
-n PDB1 PDB2 PDB3 TO PDBn
– Default: cpu_count/2 Upgrade Upgrade Upgrade Upgrade
– Examples:
 -n 24 ==> Upgrade 12 PDBs in parallel
 -n 31 ==> Upgrade 15 PDBs in parallel
Parallel Processing Within PDB Upgrades
 catctl.pl -N $ORACLE_HOME/perl/bin/perl catctl.pl –N 2 catupgrd.sql

– Parallel processing within each PDB
upgrade controlled by -N SQL PROC 1 SQL PROC 3 SQL PROC 5 SQL PROC N-1
SQL PROC 2 SQL PROC 4 SQL PROC 6 SQL PROC N
– No. of SQL processes to spawn per PDB
– Default: 2
 Example:
– cpu_count=32 -N PDB1 PDB2 PDB3 TO PDBn/2
– 16 PDBs per cycle
– Each PDB 2 parallel workers
– Maximum: 8
– Minimum: 1
Putting It All Together
CATCTL.PL
 catctl.pl –n 8 CATCON.PM
– Upgrade CDB$ROOT first

Upgrade first: EXIT IF
– Upgrade PDBs CDB$ROOT ERROR
 Startup catctl child processes
 4 PDBs upgraded at a times START CHILD CATCTL.PL PROCESSES
 2 SQL processes per PDB
CATCTL.PL CATCTL.PL CATCTL.PL CATCTL.PL
– As soon as one PDB is upgraded then
the next PDB can be processed until all CATCON.PM CATCON.PM CATCON.PM CATCON.PM
the PDBs have been upgraded
SQL PROC 1&2 SQL PROC 3&4 SQL PROC 5&6 SQL PROC 7&8
PDB$SEED PDB1 PDB2 PDB3
Default: Switch CDB$ROOT to Normal Mode
 catctl.pl default behavior

Upgrade
– Switch CDB$ROOT to NORMAL mode
before PDBs get upgraded Restart: NORMAL mode CDB$ROOT
Post upgrade
– Advantage:
 Each PDB becomes available after upgrade
– Disadvantage: Upgrade
 Slower overall due to active processes
Restart: RESTRICT mode
PDB
Post1upgrade
PDB2
Shutdown
Option: Keep CDB$ROOT in Upgrade Mode
 catctl.pl with –M option

Upgrade
– Keep CDB$ROOT in UPGRADE mode
while PDBs are upgraded Restart: UPGRADE mode CDB$ROOT
Post upgrade
– Advantage:
 Faster overall completion
– Disadvantage: Upgrade
 All PDBs need to be upgraded to become
Restart: RESTRICT mode
available again
PDB
Post1upgrade
PDB2
Shutdown
Logging, Prefixing and Directories
 catctl.pl –l – catctl.pl –l /log/upg

– Directory to spool log files into
– Default: current working directory
 catctl.pl –i
– catctl.pl –i _ORCL_
– Prefix for logfiles
– Default: none
 catctl.pl –d – catctl.pl –d
– Directory containing files to execute $ORACLE_HOME/rdbms/admin
– Default: current working directory
Logs and Verification
 upg_summary.log Find Errors in upg_summary.log

– Summary report and identify the PDB Name
– Most important log file
– Different location than other logs:
$ORACLE_HOME/cfgtoollogs/<SID>/ Check
upgrade/upg_summary.log
catupgrd<pdb_name><proc#>.log
– Information included as well in:
catupgrd0.log
– PDB logs naming:
 catupgrd<PDB NAME><proc#>.log Search for error noted in summary
– CDB$ROOT logs naming: report
 catupgrd<proc#>.log
Example upg_summary.log
Upgrade Success/Failure?
 REGISTRY$ERROR SQL> describe REGISTRY$ERROR;

– Contains all errors
Name Null? Type
– ORACLE SERVER component ------------------------------------ -------- -------------------------
determines success/failure of upgrade USERNAME VARCHAR2(256)
as a whole TIMESTAMP TIMESTAMP(6)
SCRIPT VARCHAR2(1024)
– Other components have their own IDENTIFIER VARCHAR2(256)
validation routines MESSAGE CLOB
STATEMENT CLOB
Container Inclusion/Exclusion
 catctl.pl –c
–
–
Include container
Define order of upgrades
 CDB$ROOT
Upgrade
– Example:
 -c 'CDB$ROOT PDB$SEED PDB3 PDB2 PDB1'
 catctl.pl –C  PDB$SEED
Upgrade
PDB3
Upgrade
– Exclude container
– Example:
 -C 'PDB7 PDB21'
 PDB2
Upgrade
PDB1
Upgrade
Postpone and Restart Phases
 catctl.pl –x
– Postpone datapatch/catuppst.sql
– Not recommended
 catctl.pl –p
– Begin phase
 catctl.pl –P
End phase
–
 Example: -c 'PDB7' –p 68 -P 73
o Upgrade log file identified each phase:
PHASE_TIME___START 68
PHASE_TIME___END 73
Debugging and Tracing
 catctl.pl –z catupgrd.sql CATCON.PM

catconInit: base for log and spool file names
– Debugs catcon.pm = catupgrd
running catconInit(User = 0,
 catctl.pl –Z catupgrd.sql InternalUser = ,
SrcDir = 0,
– Debugs catctl.pl LogDir = 0,
 Done using PERL to generate trace files in format LogBase = catupgrd,
catctl_YYYYMMDDHHMNSC_pid_trace.log CATCTL.PL
Package catctl.pl.
269: my @gArgs;
270: foreach $argnum (0 .. $#ARGV)
271: {
272: push (@gArgs, $ARGV[$argnum]);
6 Wrap Up
Performance Example 1: Exadata V1 25 PDBs: -n 26
 Exadata V1  Upgrade using just one node:

– 2 CPUs – 4 cores each – 64 GB RAM – catctl.pl –M –n 26 –N 2
– 8GB SGA  3 cycles
– Cycle 1: CDB$ROOT
– 25 PDBs
– Cycle 2: PDB$SEED, PDB1-PDB12
 25 GB in size each – Cycle 3: PDB13-PDB25
 Swingbench schemas
– TOTAL: 2:18:57 hrs
 CDB$ROOT: 20:54 mins
 Median PDB upgrade: 59:00 mins
 Mapped to single databases: 5:07 mins/db
Performance Example 2: Linux Server 252 PDBs: -n 24
 24 core Server  Upgrade:

– 2 CPUs – 12 cores each – sga_target=40GB
– 252 PDBs  Flush Shared Pool operations removed
– 70 GB RAM – catctl.pl –n 24 –N 2
 22 cycles
– ...
– Cycle 21: PDB239-PDB250
– TOTAL: 15 hrs 10m 56s
 Median PDB upgrade: 38m 31s
 Mapped to single databases: 3m 36s per DB
Performance Example 3: Linux Server 252 PDBs -n 32

– 252 GB RAM – catctl.pl –n 32 –N 2
 17 cycles
– ...
– TOTAL: 13h 27m

Performance Example 4: Linux Server 252 PDBs –n 32 -M

– 252 GB RAM – catctl.pl –n 32 –N 2 -M
 17 cycles
33% Faster with – ...
the –M option – Cycle 17: PDB239-PDB252

– TOTAL: 9h 6m 18s

Performance Example 5: Linux Server 252 PDBs –n 64 -M

– 252 GB RAM – catctl.pl –n 64 –N 2 -M
 8 cycles
Too heavy – ...
contention slows – Cycle 8: PDB224-PDB252

– TOTAL: 15h 14m 34s
down upgrade  Median PDB upgrade: 99m 30s

Conclusion
 Using the –M option decreases total upgrade time at the expense of
individual PDB availability
– The –M option gives more relative benefit with larger numbers of PDBs per cycle
 More PDBs per cycle is generally better than more processes per PDB
– This will decrease overall upgrade time
– But, individual PDB upgrades will take longer
– At some point contention (dictionary, shared pool) will increase dramatically
 Keep no. of cycles as low as possible…to a point
– There is no magic formula
6 Wrap Up
Slides and More
 MikeDietrichDE
 http://blogs.oracle.com/UPGRADE

Deep Dive Into: Parallel Database Upgrades

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Deep Dive Into: Parallel Database Upgrades

Uploaded by

Copyright:

Available Formats

Deep Dive into

Parallel Database Upgrades

New Pre-Upgrade Script

New Parallel Upgrade catctl.pl

 Used and proven by selected Oracle

– >100 SAP systems

New Parallel Upgrade

spfile Redo Control Flashback

OBJ$ TAB$ SOURCE$ OBJ$ TAB$ SOURCE$

OBJ$ TAB$ SOURCE$

 Comparing OBJ$ in CDB$ROOT

 Everything at Once  Unplug/Plug

CDB1 – Oracle 12.1.0.1

UPGRADE PDB$ UPGRADE

UPGRADE CDB1 – Oracle 12.1.0.2

 In CDB – Oracle 12.1.0.1:

CDB1 – Oracle 12.1.0.1

CDB2 – Oracle 12.1.0.2

 Since Oracle 12.1.0.2 we broke out catctl.pl

 Upgrade CDB$ROOT first

 Main catctl.pl process MAIN PROCESS

 catctl.pl -n $ORACLE_HOME/perl/bin/perl catctl.pl -n 24 catupgrd.sql

 catctl.pl -N $ORACLE_HOME/perl/bin/perl catctl.pl –N 2 catupgrd.sql

– Upgrade CDB$ROOT first

PDB$SEED PDB1 PDB2 PDB3

 catctl.pl default behavior

 catctl.pl with –M option

 catctl.pl –l – catctl.pl –l /log/upg

 upg_summary.log Find Errors in upg_summary.log

 REGISTRY$ERROR SQL> describe REGISTRY$ERROR;

 catctl.pl –z catupgrd.sql CATCON.PM

 Exadata V1  Upgrade using just one node:

 Mapped to single databases: 5:07 mins/db

 24 core Server  Upgrade:

 32 core Server  Upgrade:

 Mapped to single databases: 3m 11s per DB

 32 core Server  Upgrade:

the –M option – Cycle 17: PDB239-PDB252

 Mapped to single databases: 2m 10s per DB

 32 core Server  Upgrade:

contention slows – Cycle 8: PDB224-PDB252

 Mapped to single databases: 3m 37s per DB

You might also like