5.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming GomsOfProgramming Upgrade Oracle RAC GRID + Database from 11.2,0.3.0 to 11.2.0.4.0...Or..”Let’s Run The Gauntlet!” One tries to evade the inevitable. but there will always be one day when you just have to do it...upgrade the databi and or grid software. Since the support of 11.2.0.3.0 is ending, see below: However we want to take things SLOW...and in small steps..The idea is not to go for the big bang and seramble for the restore procedures, In short: ~ Install a new oracle 11.2.4.0 grid home (the Oracle software). — Upgrade the GRID/ASM infrastructure. = Install a new oracle 11.2.4.0 database home (the Oracle software). — Upgrade the database. Some questions answered in advance (for downtime/impact planning): Can an Oracle 11.2. .3.0 database run on an ASM home/instance of version 11.2.0.4.0? -> Yes, This is not an issue. The reason for this question is: in our situation we have one RAC which services sbase. Thus the 11.2.0.3.x database is required to run for a while on the upgraded ASM version. So far this is running for 3 weeks without any hitehes. three databases, but there was only downtime available for the ASM and one da — When and how much do the databases/ASM need to bounce? -> The update of ASM needs a bounce, and renders access to all the databases is services inaccessible. The upgrade of the database can be done per instance, but still in effect the whole database on all the nodes where not accessible, although node 2 was on-line. This can be important for SLA fine-print, however from us point of view: no access to the database, means downtime. — What can be expected in regards to prerequisites? -> When upgrading from 11.2.0.3.0 vanilla version: no specific need to do extra patching, However when applying the October PSU to the 11.2.0.4.0 software, the OPatch version needs to be at least version 11.2.0.3.6 and downloaded separately. NOTE: in this post an out-of-place upgrade will be done, since this is Oracle recommended, and with an inline some components will not work as advertised according to Oracle documentation. tps:lgemsofrogramming werdpress.con/2014'12Naiipgr ade-oracle8¢-16-dtabase-trom-11-2-0.3-040-1-2.0-4-0.0-es-runshe gale! ww5.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming Section 1: Grid upgrade. First we install the software by unpacking the downloaded files in a temp directory (we use /oracle/pateh in this post). This process is straightforward, so no need to document this. Since the database can't be upgraded without the grid first (yes, a warning will be issued when tried), we start with ‘upgrading grid/ASM. After the unpacking, we change to the unpacked grid software directory, and fire up the installer: § ed /oracie/p: 5 ./eoninstaller.sh Checking Temp space: must be greater than ae 08S MB assed Checking swap space: must be greater than 150 MB. Actual 5851 MB Passed paring to launch Oracle Universal Installer from /tsp/Oratnstall2014-11-07_03~33-19PM. Please wait + Prerequisites All these will be done on NODE 1 of the RAC unless stated otherwise. 1) Unset Oracle environment variables. Check ifthe variable ORA_CRS_HOMEis set. Ifset, unset it before starting an installation or upgrade. Check to ensure that installation owner login shell profiles (for example, .proftte or .esire) do not have ORA_CRS_HOME set. ‘We have an existing ASM installation running, and we use the same ‘user’ to install the upgrade, so at the least the following environment variables needs to be unset: ORA_CRS_HOME; ORACLE_HOME; ORA_NLS10; TNS_ADMIN; ORACLE_ROSTNAME © ORACLE_SID 1. ORACLE_UNONAME ‘The setting ‘ORACLE_TERM=xterm’ stays set. 2) Pre-check the upgrade process to determine if any patches are required. § /oracle/patch, AnvEy.sh stage upgrade “a bedc-edb01,h Ling ps lgemsorprogramming wordpress com/2014"1202)upgrade-oracle-r3c-qid-detabase-from-11-2-0-3-010-11-2-04-0-or-lets-runthe-gaurle 225.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming Performing pre-checks for cluster services setup Checking node reachability. Check: Node reachability from node "bede-odb02" Destination Node Reachable? nodeot yes rodeo? yes Result: Node reachability check passed from node "node02 checking user equivalence Check: User equivalence for user “oracle” bede-odb02 passed bede-odb0? passed Result: User equivalence check passed for user “oracle” Check: Time zone consistency Result: Time zone consistency check passed Checking VIP configuration Checking VIP Subnet configuration. Check for VIP Subnet configuration passed. Checking VIP reachability Check for VIP reachability passed, Checking Oracle Cluster Voting Disk configuration... ASM Running check passed. ASM ia running on all specified nodes oracle Cluster Voting Disk configuration check passed Clusterware version consistency passed Pre-check for cluster services setup was successful, ‘This is what we need to see. Ifthere is anything wrong, no matter how small it seems. FIXIT! After any issues are fixed, start the installer: Joracle/pateh/grid/runtastaller.sh Starting Oracle Universal Installer... Checking Temp space: must be gzeater than 120 MB. Actual 27009 MB Pagsed Checking swap space: must be greater than 150 MB. actual 49983 MB Pagsed Checking monitor: must be configured to display at least 296 colors. Actual 69926 Passed Preparing to launch Oracle Universal Installer from /tmp/Orainstali2014~11-11_11-08-36AM, Please wait ip lgemsorprogramming wordpress com/2014"1202iupgrade-oracle-r3c-qid-database-from-11-2-0-3-010-11-2-04-0-or-lets-runthe-gaurle 3m5.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming Skip Software updates. Press Next. Select Upgrade .. Press Next. Select English. Press Next. Make sure BOTH/ALL nodes are selected. Press Next. Follow os “GemsOfProgrammiiig™ Get every new post delivered ter your email adress Sclect the correct group for ASV Press Next. ip lgemsorprogramming wordpress com/2014"1202i-pgrade-oracle-r3c-qid-database-from-11-2-0-3-00-11-2-04-0-or-lets-runthe-gaurel5.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming (aCe) Press Yes, if this box pops up. re Enter the NEW location of the software. Press Next. ‘Wait until the checks have completed. [om No errors are expected, due to the runclufy command being successful. Press Install. Installer proceeds... ip lgemsorprogramming wordpress com/2014"1202i-pgrade-oracle-r3c-qid-database-from-11-2-0-3-00-11-2-04-0-or-lets-runthe-gaurel5.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming Installer keeps going... Run “rootupgrade.sh” ASM Node 1 goes down for update. ip lgemsorprogramming wordpress com/2014"1202i-pgrade-oracle-r3c-qid-database-from-11-2-0-3-00-11-2-04-0-or-lets-runthe-gaurel5.082018 Upgrade Oracle RAC GRID + Database rom 11.20.3010 11.2040...Or."Let's Run The Gault” | GemsOfProgramming Now root scripts on node 2,3, ip lgemsorprogramming wordpress com/2014"1202i-pgrade-oracle-r3c-qid-database-from-11-2-0-3-00-11-2-04-0-or-lets-runthe-gaurel m25.082018 Installation continues. Installation continues. Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming An error showed up during this phase, strange, since the previous check was ok. The log shows: wpe: 1wz0: INEO: IFo: INEO: Check for integrity of nane service switch configuration file "/etc/nsswitch.cont™ passed PRVG-1101 : SCAN name “zac~scan" failed to resolve ERROR PRVF-4657 ; Name resolution setup check for "raccscan" (IP address: 10,3.28.50) failed PRVE-4664 : Found Snconsistent name resolucion entries for SCAN nane “zactscan” 1 Verification of SCAN VIP and Listener setup failed This is not something critical for this procedure and can be considered as a Warning only, and can safely be ip lgemsorprogramming wordpress com/2014"1202i-pgrade-oracle-r3c-qid-database-from-11-2-0-3-00-11-2-04-0-or-lets-runthe-gaurel am5.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming ignored, so continue. Press Skip. Press Next. Press Yes Press Close. ASM Upgrade Finished! ‘Now this part is done, proceed with some sanity chee Check if ASM is upgraded to the correct version: Log in as “sqlplus / as sysasm” + from vversion: BARNER oracle Database 11g Enterprige dition Release 11.2.0.4.0 - 6ibit Production ps lgemsorprogramming wordpress com/2014"1202)upgrade-oracle-r3c-qid-detabase-from-11-2-0-3-010-11-2-04-0-or-lets-runthe-gaurle oe5.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming PL/SQL Release 11.2.0.4.0 ~ Production fe 11.2.0.4.9, oxs for Linux: Version 11,2.0.4.0 - Production Check if all the processes of the cluster are running: root@ =}# /oracle/11.2.0.4/geid/bin/ersct stat res —¢ ora.OanA.ay ONLINE ONLINE bede-odvoL ONLINE ONLINE bede~odb02 ONLINE ONLINE be ora. DATACCUNDG. ONETHE bede-ceb02 RA.dg {2 ONLINE hede-odhOL ONLINE ONLINE bede-odb02 ORLINE ONLINE bede-cdb01 ONLINE ONLINE he ora. PRACCUTDG.dg ONLINE ONLINE bede-od02 (ER. Lene ONLINE ONLINE bede-odvoL ONLINE ONLINE bede-odb02 (LINE ONLINE bede-odbOl started ONLINE ONLINE bede-ode02 Started oragsd bede-odh02 ora.netl neti ONLINE ONLINE bede-odbOL ONLINE ONLINE bede-odb02 (LINE ONLINE bede-cdo01 ONLINE ONLINE be ora.registry.acts ONLTIF bede-celb02 1ER_SCANL. lan 2 ONLINE ONLINE bede-oabO1 ora. bede-odb0l. vip ONDINE ONLINE bede-odb01 1 ONLINE OKLIKE bede-oab02 1 ONLINE ONLINE bede-odhO1 eb bede-odh01 open bede-oab02 open ps lgemsorprogramming wordpress com/2014"1202)upgrade-oracle-r3c-qid-detabase-from-11-2-0-3-010-11-2-04-0-or-lets-runthe-gaurle 10225.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming ora expeepr0 expeeprd.ave 1 ONLINE ONLINE 2 ONDINE ONLINE ora.erpecut0.db 12 ONLINE OKLIKE 2 ONLINE ONLINE ede-oab01 ede-oah02 bede-odb01 bede-oab02 ora. erpecut).expeouat.sve 1 ONDINE ONLINE 2 ONLINE ONLINE ora-expemea.db 1 ONLINE ONLIKE 2 ONLINE ONLINE 2 ONLINE ONLINE ova-ecanl.vip 1 ONDINE ONLINE ede-oah01 bede-oebo1 edo-oah02 bede-adh02 edo-och0l open open open open Section 2: Database upgrade Before starting the upgrade, a word of eaution: Your mileage may vary, but we found that while the database was running happily with Huge Pages(large ages in the init file) configured as ‘only’, and could bounce without any issues, it wreaked havoc with the Database Upgrade Assistant, probably somewhere with the ‘startup upgrade’ mode. Asa precaution we put our databases in shared memory (temporary 10G instead of the 48G Huge Pages) and make sure they bounce without any issues. The trial runs all failed with: Memlock limit too small: 32768 to accommodate segment size: 268435456 With the database in shared memory, we did not run into this issue. Unselect “I wish to be Press Next. ip lgemsorprogramming wordpress com/2014"1202i-pgrade-oracle-r3c-qid-database-from-11-2-0-3-00-11-2-04-0-or-lets-runthe-gaurel we5.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming Press Yes, los ee Select “skip...” Press Next. Select “Upgrade...” Press Next. Double check RAC installation is selected, all nodes are selected. Press Next, pee 1 Press Next, ip lgemsorprogramming wordpress com/2014"1202i-pgrade-oracle-r3c-qid-database-from-11-2-0-3-00-11-2-04-0-or-lets-runthe-gaurel vw5.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming Press Next. oma ao Change values to new oracle base/home location. Press Next. Select “oinstall”. Press Next. ‘Wait for next screen. ip lgemsorprogramming wordpress com/2014"1202i-pgrade-oracle-r3c-qid-database-from-11-2-0-3-00-11-2-04-0-or-lets-runthe-gaurel 1925.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming Expected error, select “Ignore All”, Press Next. NT tsa Cassa) Press Yes. Press Install. Process continues. Process continues. Execute the root scripts, on NODEo1 FIRST! ip lgemsorprogramming wordpress com/2014"1202i-pgrade-oracle-r3c-qid-database-from-11-2-0-3-00-11-2-04-0-or-lets-runthe-gaurel va5.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming PF eee Press Enter. Script is done. Run the root script on nodeo2-xx. When done: Press Ok. ip lgemsorprogramming wordpress com/2014"1202i-pgrade-oracle-r3c-qid-database-from-11-2-0-3-00-11-2-04-0-or-lets-runthe-gaurel 1325.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming Press Next. Make sure correct database is selected when more databases are available in this Oracle home. Press Next. Please wait. Validate the invalid objects, if applicable and make sure you at least check the warnings. Press Yes when done. ips:igemsofproaramming wordpress.com2014/202N-pgrade-oracle-rac-id-databasefrom-11-2.0-0-0410-11-2-04-0-or-lets-rutbe-gaurey5.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming ‘Make sure Recompile is ON. Press Next. Press Next. Optionally this screen can appear. check/uncheck this by choice. In this post it will not be checked, since the database is already registered in OEM 12¢. Press Next. Press Finish. ips:igemsofproaramming wordpress.com2014/202N-pgrade-oracle-rac-id-databasefrom-11-2.0-0-0410-11-2-04-0-or-lets-rutbe-gaurey we5.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming Pro-Upgrade starts. Upgrade of the Oracle Server. Upgrade of Java Virtual Machine, ete. ete. B aso = Performing Post Upgrade. Press Ok. ip lgemsorprogramming wordpress com/2014"1202i-pgrade-oracle-r3c-qid-database-from-11-2-0-3-00-11-2-04-0-or-lets-runthe-gaurel vw5.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming Press Close. Press Next. Press Close. And the database upgrade is done. ‘The database should be up and running again, but there are a couple of settings in need of change. * Put the database back into Huge Pages if it previously was configured like this. * Alter some parameter settings which still points to the old oracle home path. A small check before shutting down the database node: SQL select * from gvSveraion: INST_ID BANWER Oracle Database tig Enterprise Hdition Release 11.2.0.4.0 ~ 64bit Production PL/SQL Relea: CORE 11.2.0.4.0 Production IWS for Linux: Version 11.2.0.4.0 - Production NESREL Version 12.2.0.4.0 - Production Oracle Database i1g Enterprise Edition Release 11.2.0.4.0 - 64piz Production PL/SGh Release 11.2.0,6.0 ~ Production 2.0.4.0 = Production ps lgemsorprogramming wordpress com/2014"1202)upgrade-oracle-r3c-qid-detabase-from-11-2-0-3-010-11-2-04-0-or-lets-runthe-gaurle 1925.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming 2 CORE 1.2.0.4. 2 TNS for Linux: Version 11.2.0.4.0 - Production Production 10 cows selected. Nice! Beware of your environment variables: ORACLE_HOME and PATH should be updated to the NEW Oracle Home! Now, shut down the node, alter the pfile/spfile. Specific parameters to alter/check are: ground_dump_de: __dunp_dest vwsex cmp se Double check the / Do the regular database checks to see if all still looks correct, and keep an eye on the alert file. ‘That's it! Done! Really! Done! ‘You want more? Still more? Ok.-Let’s say the GUI is not your thing..or if the DBUA bails out after the new Oracle Home is installed, and the database is not upgraded. (yes, it happend to me also. Alot. ). No problem. Just do the upgrade manually.. Shutdown the database, up upgrade -> cluster database~false-—> FROM NEW ORACES_HOME! ACLE_HOwE/ zane /adnin/eatups duel Removing the old HOMES: Before removing/renaming the old GRID and DB homes: GRID: system set background dunp_dest=!/oracle/11_4/base/diag/asn/+asn/+Asti/trace' scope le: system set core_dump dest~' /oracle/11_4/base/diag/asn/sasn/+ASMi /edump' scopersptile system set user_dump_ dest=! /oracle/base/diag/asm/+asn/+ASMl/zrace! scope-spfile; Jp ab +ASW1 dat he_4ASML dat /oracte/11_4/base/db_1/ébs/ -p ab JASWI.dat he_/ASM1.dat /aracle/geid_21_4/dba/ Alter the pfile/spfile: ip lgemsorprogramming wordpress com/2014"1202i-pgrade-oracle-r3c-qid-database-from-11-2-0-3-00-11-2-04-0-or-lets-runthe-gaurel 20225.082018 Upsyade Oracle RAC GRID + Database fom 11.203010 112040. .Or."Let's Run The Gaurtlet” | GemsOFProggamming cle core dunp_dest Edit /ete/oratab: ele /L | Afease/ ab 1:8 # NeW Hous! Edit SORACLE_HOME/network/admin/sqinet.ora (NEW) ADR_BASE = /oracle/11_a/pase # new home: Bounce both the GRID and the DATABASE after renaming (not yet deleting!) the old homes to see if al is still working as expected. If there are no errors in any alert log, the old homes can be deleted. Or put on tape. Thi situation is different, so test,test and test to build up confidence before upgrading the production environment. And did someone mention backups? Might be a good idea also, just keep in mind to create a backup OUTSIDE of ASM before upgrading this software to prevent panic attacks. should cover most of the upgrade procedure. At the least it gives an idea of what to expect. Of course every ps: If there are any third party patches applied to the ORACLE_ HOME (as in ERPLN NLS patch) you need to re-apply this patch! This is a NEW vanilla oracle home. Success! Links to useful documentation: hittps://does.oracle,com/ed/E11882_ 01/install.112/e41961/proestop.htm#CWLIN430 https: //aprilesims.wordpress,com/migrating-to-igra, poste ae eke Related Building A Single Instance Physical Standby Duplicate a igR2 Database with Rman--the Installing Orade 12e on Ubuntu 12.04 (64 For ARAC Primary..Or.Actve Dataguard, one where we duplicate froma backup. bit. ahard journey, buti’s worth it! Save My (Daily) Data! in" Databases In "Database Tn"ASM ‘knack fortis, and since approx. 2000 Tm a fll time DRA. My experiences touches lt of Oracle procs ike Forms and Reports 9/30, IDAPI, Application Server, Weblogic Fusion and ofcourse: Orace Enterprise Databases, JavaFX, Swing and other Java ‘iewall pos About GemsOProgramming B Becng previously enthusatic ava programmer, rolled into the Ora Database Administration worl. tured out got a GomsofProgranming ip lgemsorprogramming wordpress com/2014"1202i-pgrade-oracle-r3c-qid-database-from-11-2-0-3-00-11-2-04-0-or-lets-runthe-gaurel au5.082018 Upgrade Oracle RAC GRID + Database rom 11.20.3010 11.2040...Or."Let's Run The Gault” | GemsOfProgramming GemsorProgramming ip lgemsorprogramming wordpress com/2014"1202i-pgrade-oracle-r3c-qid-database-from-11-2-0-3-00-11-2-04-0-or-lets-runthe-gaurel zm