Steps

1. Glance Server 1 output

2. Glance Server 2 output

3. Cronjob detail
30 03,19 * * * /home/oracle/arch/hot_archive_backup_CLRTYDB.sh >/dev/null 2>&1
00 20 * * 05 /home/oracle/script/alertlog.sh >/dev/null 2>&1
# -- Getting crs_stat output for monitoring
0,5,10,15,20,25,30,35,40,45,50,55 * * * *
/app/oracle/product/10.2.0/crs_1/bin/crs_stat>/home/oracle/crs_stat.out
00 02 2-8 * * /arc_prg/ap_script.sh >> /arc_prg/exp_dump/ap.log
#01 02 2 * * /arc_prg/arc.sh >> /arc_prg/exp_dump/clrt_arc.log
#00 02 4,5,6 * * /arc_prg/prg.sh >> /arc_prg/exp_dump/clrt_prg.log

4. Mount point detail

oracle@nwysum01:/home/oracle [CLRTYDB1] >bdf
Filesystem
kbytes
used
avail %used Mounted on
/dev/vg00/lvol3
10485760 928944 9482248
9% /
/dev/vg00/lvol1
2097152 417536 1666560
20% /stand
/dev/vg00/lvol8
10485760 4939360 5504960
47% /var
/dev/vg00/lvol7
10485760 7622776 2840736
73% /usr
/dev/vg00/lvol6
10485760 5444152 5002896
52% /tmp
/dev/vg00/lvol5
10485760 6648040 3807752
64% /opt
/dev/vg00/lvol4
10485760 300800 10105456
3% /home
/dev/vg01/lv_clalogs
33554432 1305627 30233261
4% /claritylogs
/dev/arch/archvpg 132645888 52761791 74891463
41% /arc_prg
/dev/vg01/lv_claapp
33554432 26850163 6285369
81% /app
oracle@nwysum01:/home/oracle [CLRTYDB1] >

5. Stop EM agent

o
o

. .oem_profile
emctl status agent
oracle@nwysum01:/home/oracle [CLRTYDB1] >emctl status agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
--------------------------------------------------------------Agent Version
: 10.2.0.5.0
OMS Version
: 10.2.0.5.0
Protocol Version : 10.2.0.5.0
Agent Home
:
/app/oracle/OracleHomes/agent10g_new/agent10g/nwysum01
Agent binaries
: /app/oracle/OracleHomes/agent10g_new/agent10g
Agent Process ID : 28144
Parent Process ID : 28129
Agent URL
: https://nwysum01:3872/emd/main
Repository URL
: https://bansum02:1159/em/upload
Started at
: 2013-03-13 22:23:33

Started by user
: oracle
Last Reload
: 2013-03-18 04:53:59
Last successful upload
: 2013-03-31 23:41:51
Total Megabytes of XML files uploaded so far : 1143.97
Number of XML files pending upload
:
0
Size of XML files pending upload(MB)
:
0.00
Available disk space on upload filesystem
:
18.73%
Last successful heartbeat to OMS
: 2013-03-31 23:41:51
--------------------------------------------------------------Agent is Running and Readyy

emctl stop agent

oracle@nwysum01:/home/oracle [CLRTYDB1] >emctl stop agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
Stopping agent ... stopped.

emctl status agent

oracle@nwysum01:/home/oracle [CLRTYDB1] >emctl status agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
--------------------------------------------------------------Agent is Not Running

6. Health Check

crs_stat t
oracle@nwysum01:/home/oracle [CLRTYDB1] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
ONLINE
ONLINE
nwysum01
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
ONLINE
ONLINE
nwysum01
ora....01.lsnr application
ONLINE
ONLINE
nwysum01
ora....m01.gsd application
ONLINE
ONLINE
nwysum01
ora....m01.ons application
ONLINE
ONLINE
nwysum01
ora....m01.vip application
ONLINE
ONLINE
nwysum01
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 00:04:38 MAL 2013

crsctl check crs

oracle@nwysum01:/home/oracle [CLRTYDB1] >crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 00:05:16 MAL 2013
ps -ef |grep pmon|grep -v grep

oracle@nwysum01:/home/oracle
grep
oracle 29169
1 0 Mar
oracle 19511
1 0 Mar
oracle@nwysum01:/home/oracle
Mon Apr 1 00:05:39 MAL 2013

[CLRTYDB1] >ps -ef |grep pmon|grep -v

18 ?
8:42 ora_pmon_CLRTYDB1
18 ?
3:35 asm_pmon_+ASM1
[CLRTYDB1] >date

ps -ef|grep tns|grep -v grep

oracle@nwysum01:/home/oracle [CLRTYDB1] >ps -ef|grep tns|grep -v grep

oracle 19604
1 0 Mar 18 ?
4:29
/app/oracle/product/10.2.0/asm_1/bin/tnslsnr LISTENER_NWYSUM01 -inherit
oracle 13196
1 0 Feb 24 ?
0:01
/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER_CLRTBKP -inherit
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 00:06:32 MAL 2013

srvctl status instance -d CLRTYDB -i CLRTYDB1

oracle@nwysum01:/home/oracle [CLRTYDB1] >srvctl status instance -d
CLRTYDB -i CLRTYDB1
Instance CLRTYDB1 is running on node nwysum01
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 00:07:03 MAL 2013

srvctl status asm n nwysum01

oracle@nwysum01:/home/oracle [CLRTYDB1] >srvctl status asm -n nwysum01
ASM instance +ASM1 is running on node nwysum01.
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 00:07:42 MAL 2013

srvctl status nodeapps -n nwysum01

oracle@nwysum01:/home/oracle [CLRTYDB1] >srvctl status nodeapps -n
nwysum01
VIP is running on node: nwysum01
GSD is running on node: nwysum01
Listener is running on node: nwysum01
ONS daemon is running on node: nwysum01
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 00:08:06 MAL 2013

SYS@CLRTYDB1>@ses
SYS@CLRTYDB1>@ses
CLOGNAME
---------------------------------------------------------------------Sessions_CLRTYDB1_130401000848
!!!!! No of Session in each node !!!!!
01/04/2013 00:08:48
==================================Session Count Group By Program======
Ins TYPE
Status
Cnt Program Name
---- ---------- --------- ----- -----------------------------1 BACKGROUND ACTIVE
26 oracle@nwysum01
********* ----sum
26
USER
ACTIVE
1 oracle@nwysum01
1 racgimon@nwysum01
1 sqlplusw.exe
********* ----sum
3
INACTIVE
34 JDBC
1 YesDBScheduler.exe
2 racgimon@nwysum01
1 toad.exe
********* ----sum
38
**** **********
----sum
67
2 BACKGROUND ACTIVE
26 oracle@nwysum02
********* ----sum
26
USER
ACTIVE
2 emagent@nwysum02
1 oracle@nwysum02

********* ----sum
3
INACTIVE
41 JDBC
1 cfg_file_gener@nwywin02
3 emagent@nwysum02
3 racgimon@nwysum02
********* ----sum
48
**** **********
----sum
77
----sum
144
=================================Session Count Group By Status========
Ins TYPE
---- ---------1 BACKGROUND
USER
USER
2 BACKGROUND
USER
USER
****
sum

Status
SESSIONS
--------- ---------ACTIVE
26
ACTIVE
3
INACTIVE
38
ACTIVE
26
ACTIVE
3
INACTIVE
48
---------144

================================Session Count (Background Excluded)===

Ins Status
SESSIONS
---- --------- ---------1 ACTIVE
3
INACTIVE
38
2 ACTIVE
3
INACTIVE
48
**** ********* ---------sum
92
!!!!! END !!!!!
SYS@CLRTYDB1>

SYS@CLRTYDB1>@sqls
SYS@CLRTYDB1>@sqls
CLOGNAME
------------------------------------------------------------------------------SQLs_CLRTYDB1_130401000926
!!!!! All About Running SQLs !!!!!
01/04/2013 00:09:26
USERNAME
Program Name
MACHINE
OSUSER
Status
PROCESS
--------------- ----------------------------------------------------------- SYS
oracle@nwysum02 (PZ99)
YTLCOMMS\CMMS0025
fairdaus
ACTIVE
CPRG
JDBC Thin Client
nwymon01
jboss
INACTIVE
SYS
oracle@nwysum01 (PZ99)
YTLCOMMS\CMMS0025
fairdaus
ACTIVE
CPRG
JDBC Thin Client
nwymon01
jboss
INACTIVE
CPRG
JDBC Thin Client
nwymon02
jboss
INACTIVE
CPRG
JDBC Thin Client
nwymon03
jboss
INACTIVE

LOGON_TIME
SID
SERIAL#
-------------------01/04/2013
3
31/03/2013
3198
01/04/2013
3
01/04/2013
3230
31/03/2013
3237
31/03/2013
3244

00:09:26
23:52:36
24874 1
00:09:26
00:00:06
27899 1
23:52:52
13282 1
23:53:11
24993 1

CPRG
JDBC Thin Client
nwymon01
jboss
SYS
sqlplusw.exe
YTLCOMMS\CMMS0025
fairdaus
!!!!! END !!!!!
SYS@CLRTYDB1>

INACTIVE
INACTIVE

31/03/2013
3268
01/04/2013
3281

23:52:37
27891 1
00:08:45
219

7. Monitor both instances

Monitor alert and cluster logs from both nodes

tail -f /app/oracle/admin/CLRTYDB/bdump/alert_CLRTYDB1.log

tail -f /app/oracle/admin/CLRTYDB/bdump/alert_CLRTYDB2.log
tail -f
/app/oracle/product/10.2.0/crs_1/log/nwysum01/crsd/crsd.log

tail -f
/app/oracle/product/10.2.0/crs_1/log/nwysum02/crsd/crsd.log

8. Stop Instance

oracle@nwysum01:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] >srvctl stop

instance -d CLRTYDB -i CLRTYDB1
oracle@nwysum01:/home/oracle [CLRTYDB1] >srvctl stop instance -d CLRTYDB
-i CLRTYDB1
PRKP-1002 : Error stopping instance CLRTYDB1 on node nwysum01
CRS-0216: Could not stop resource 'ora.CLRTYDB.CLRTYDB1.inst'.
oracle@nwysum01:/home/oracle [CLRTYDB1] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
OFFLINE
UNKNOWN
nwysum01
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
ONLINE
ONLINE
nwysum01
ora....01.lsnr application
ONLINE
ONLINE
nwysum01
ora....m01.gsd application
ONLINE
ONLINE
nwysum01
ora....m01.ons application
ONLINE
ONLINE
nwysum01
ora....m01.vip application
ONLINE
ONLINE
nwysum01
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02
oracle@nwysum01:/home/oracle [CLRTYDB1] >srvctl stop instance -d CLRTYDB
-i CLRTYDB1

oracle@nwysum01:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] >crs_stat -t

oracle@nwysum01:/home/oracle [CLRTYDB1] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
OFFLINE
OFFLINE
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
ONLINE
ONLINE
nwysum01
ora....01.lsnr application
ONLINE
ONLINE
nwysum01
ora....m01.gsd application
ONLINE
ONLINE
nwysum01
ora....m01.ons application
ONLINE
ONLINE
nwysum01
ora....m01.vip application
ONLINE
ONLINE
nwysum01
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 01:17:30 MAL 2013

oracle@nwysum01:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] >srvctl stop

asm -n nwysum01

oracle@nwysum01:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] >crs_stat -t

oracle@nwysum01:/home/oracle [CLRTYDB1] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
OFFLINE
OFFLINE
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
OFFLINE
OFFLINE
ora....01.lsnr application
ONLINE
ONLINE
nwysum01
ora....m01.gsd application
ONLINE
ONLINE
nwysum01
ora....m01.ons application
ONLINE
ONLINE
nwysum01
ora....m01.vip application
ONLINE
ONLINE
nwysum01
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 01:18:29 MAL 2013
oracle@nwysum01:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] >srvctl stop
nodeapps -n nwysum01

oracle@nwysum01:/home/oracle [CLRTYDB1] >srvctl stop asm -n

nwysum01

<<paste output here>>

oracle@nwysum01:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] >crs_stat -t

oracle@nwysum01:/home/oracle [CLRTYDB1] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
OFFLINE
OFFLINE
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
OFFLINE
OFFLINE
ora....01.lsnr application
OFFLINE
OFFLINE
ora....m01.gsd application
OFFLINE
OFFLINE
ora....m01.ons application
OFFLINE
OFFLINE
ora....m01.vip application
OFFLINE
OFFLINE
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 01:19:32 MAL 2013

check status crs_stat t on nwysum02

o oracle@nwysum02:/app/oracle/product/10.2.0/db_1 [CLRTYDB2] > crs_stat -t
o oracle@nwysum02:/home/oracle [CLRTYDB2] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
OFFLINE
OFFLINE
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
OFFLINE
OFFLINE
ora....01.lsnr application
OFFLINE
OFFLINE
ora....m01.gsd application
OFFLINE
OFFLINE
ora....m01.ons application
OFFLINE
OFFLINE
ora....m01.vip application
OFFLINE
OFFLINE
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02

ora....m02.ons application
ONLINE
ONLINE
ora....m02.vip application
ONLINE
ONLINE
oracle@nwysum02:/home/oracle [CLRTYDB2] >date
Mon Apr 1 01:20:17 MAL 2013

nwysum02
nwysum02

9. Connect as root and run the following command

o
o
o

mwysum01:/ # bash
mwysum01:/ # cd /app/oracle/product/10.2.0/crs_1/bin
mwysum01:/app/oracle/product/10.2.0/crs_1/bin # ./crsctl stop crs
Mon Apr 1 01:22:10 MAL 2013
nwysum01:/ # cd /app/oracle/product/10.2.0/crs_1/bin
nwysum01:/app/oracle/product/10.2.0/crs_1/bin # ./crsctl stop crs
Stopping resources.
This could take several minutes.
Successfully stopped Oracle Clusterware resources
Stopping Cluster Synchronization Services.
Shutting down the Cluster Synchronization Services daemon.
Shutdown request successfully issued.
nwysum01:/app/oracle/product/10.2.0/crs_1/bin #

10. Check Cluster Status in Node-2

oracle@nwysum02:/app/oracle/product/10.2.0/db_1 [CLRTYDB2] >crs_stat -t

oracle@nwysum02:/home/oracle [CLRTYDB2] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
OFFLINE
OFFLINE
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
OFFLINE
OFFLINE
ora....01.lsnr application
OFFLINE
OFFLINE
ora....m01.gsd application
OFFLINE
OFFLINE
ora....m01.ons application
OFFLINE
OFFLINE
ora....m01.vip application
OFFLINE
OFFLINE
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02

11. UNIX team will Reboot server before patch

12. Monitor Alert and Cluster logs for nwysum02

tail -f /app/oracle/admin/CLRTYDB/bdump/alert_CLRTYDB2.log
tail -f /app/oracle/product/10.2.0/crs_1/log/nwysum02/crsd/crsd.log

AFTER OS REBOOT

13. Stop Grid Console on nwysum01

o emctl stop agent

oracle@nwysum01:/home/oracle [CLRTYDB1] >emctl status agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
--------------------------------------------------------------Agent is Not Running
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 01:34:21 MAL 2013
14. (After OS reboot) Check Cluster Status in Node-1

oracle@nwysum01:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] >crs_stat -t

oracle@nwysum01:/home/oracle [CLRTYDB1] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
ONLINE
OFFLINE
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
ONLINE
ONLINE
nwysum01
ora....01.lsnr application
ONLINE
ONLINE
nwysum01
ora....m01.gsd application
ONLINE
ONLINE
nwysum01
ora....m01.ons application
ONLINE
ONLINE
nwysum01
ora....m01.vip application
ONLINE
ONLINE
nwysum01
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 01:35:15 MAL 2013

15. Check Cluster Status in Node-2

oracle@nwysum02:/app/oracle/product/10.2.0/db_1 [CLRTYDB2] >crs_stat -t

oracle@nwysum02:/home/oracle [CLRTYDB2] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
ONLINE
OFFLINE
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
ONLINE
ONLINE
nwysum01
ora....01.lsnr application
ONLINE
ONLINE
nwysum01
ora....m01.gsd application
ONLINE
ONLINE
nwysum01
ora....m01.ons application
ONLINE
ONLINE
nwysum01
ora....m01.vip application
ONLINE
ONLINE
nwysum01
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02
oracle@nwysum02:/home/oracle [CLRTYDB2] >date
Mon Apr 1 01:35:40 MAL 2013

16. Stop Instance back for OS patch

oracle@nwysum01:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] >srvctl stop

instance -d CLRTYDB -i CLRTYDB1

<<paste output here>>

oracle@nwysum01:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] >crs_stat -t

oracle@nwysum01:/home/oracle [CLRTYDB1] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
ONLINE
OFFLINE
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
ONLINE
ONLINE
nwysum01
ora....01.lsnr application
ONLINE
ONLINE
nwysum01
ora....m01.gsd application
ONLINE
ONLINE
nwysum01
ora....m01.ons application
ONLINE
ONLINE
nwysum01
ora....m01.vip application
ONLINE
ONLINE
nwysum01
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 01:36:57 MAL 2013

oracle@nwysum01:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] >srvctl stop

asm -n nwysum01

oracle@nwysum01:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] >crs_stat -t

oracle@nwysum01:/home/oracle [CLRTYDB1] >srvctl stop asm -n nwysum01
oracle@nwysum01:/home/oracle [CLRTYDB1] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
OFFLINE
OFFLINE
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
OFFLINE
OFFLINE
ora....01.lsnr application
ONLINE
ONLINE
nwysum01
ora....m01.gsd application
ONLINE
ONLINE
nwysum01
ora....m01.ons application
ONLINE
ONLINE
nwysum01
ora....m01.vip application
ONLINE
ONLINE
nwysum01
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02

oracle@nwysum02:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] >srvctl stop

nodeapps -n nwysum01

oracle@nwysum01:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] >crs_stat -t

oracle@nwysum01:/home/oracle [CLRTYDB1] >srvctl stop nodeapps -n
nwysum01
oracle@nwysum01:/home/oracle [CLRTYDB1] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
OFFLINE
OFFLINE
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
OFFLINE
OFFLINE
ora....01.lsnr application
OFFLINE
OFFLINE
ora....m01.gsd application
OFFLINE
OFFLINE
ora....m01.ons application
OFFLINE
OFFLINE
ora....m01.vip application
OFFLINE
OFFLINE
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02

17. Check CRS Status on nwysym02

oracle@nwysum02:/app/oracle/product/10.2.0/db_1 [CLRTYDB2] > crs_stat -t

oracle@nwysum02:/home/oracle [CLRTYDB2] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
OFFLINE
OFFLINE
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
OFFLINE
OFFLINE
ora....01.lsnr application
OFFLINE
OFFLINE
ora....m01.gsd application
OFFLINE
OFFLINE
ora....m01.ons application
OFFLINE
OFFLINE
ora....m01.vip application
OFFLINE
OFFLINE
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02
oracle@nwysum02:/home/oracle [CLRTYDB2] >date
Mon Apr 1 01:38:45 MAL 2013

18. Connect as root and run the following command

o
o
o

mwysum01:/ # bash
mwysum01:/ # cd /app/oracle/product/10.2.0/crs_1/bin
mwysum01:/app/oracle/product/10.2.0/crs_1/bin # ./crsctl stop crs
nwysum01:/app/oracle/product/10.2.0/crs_1/bin # ./crsctl stop crs
Stopping resources.
This could take several minutes.
Successfully stopped Oracle Clusterware resources
Stopping Cluster Synchronization Services.
Shutting down the Cluster Synchronization Services daemon.
Shutdown request successfully issued.
nwysum01:/app/oracle/product/10.2.0/crs_1/bin # date
Mon Apr 1 01:39:41 MAL 2013
19. Check Cluster Status in Node-2
o oracle@nwysum02:/app/oracle/product/10.2.0/db_1 [CLRTYDB2] >crs_stat -t
oracle@nwysum02:/home/oracle [CLRTYDB2] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
OFFLINE
OFFLINE
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
OFFLINE
OFFLINE
ora....01.lsnr application
OFFLINE
OFFLINE
ora....m01.gsd application
OFFLINE
OFFLINE
ora....m01.ons application
OFFLINE
OFFLINE
ora....m01.vip application
OFFLINE
OFFLINE
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02
oracle@nwysum02:/home/oracle [CLRTYDB2] >date
Mon Apr 1 01:40:00 MAL 2013

20. Monitor Alert and Cluster logs nywsum02

tail -f /app/oracle/admin/CLRTYDB/bdump/alert_CLRTYDB2.log
tail -f /app/oracle/product/10.2.0/crs_1/log/nwysum01/crsd/crsd.log

21. Server reboot after Patching. After Server up check

oracle@nwysum01:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] >crs_stat -t

oracle@nwysum01:/home/oracle [CLRTYDB1] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
ONLINE
OFFLINE
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
ONLINE
ONLINE
nwysum01
ora....01.lsnr application
ONLINE
ONLINE
nwysum01
ora....m01.gsd application
ONLINE
ONLINE
nwysum01
ora....m01.ons application
ONLINE
ONLINE
nwysum01
ora....m01.vip application
ONLINE
ONLINE
nwysum01
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02
oracle@nwysum01:/home/oracle [CLRTYDB1] >date

Mon Apr

1 02:02:27 MAL 2013

oracle@nwysum02:/app/oracle/product/10.2.0/db_1 [CLRTYDB2] >crs_stat -t

oracle@nwysum02:/home/oracle [CLRTYDB2] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
ONLINE
OFFLINE
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
ONLINE
ONLINE
nwysum01
ora....01.lsnr application
ONLINE
ONLINE
nwysum01
ora....m01.gsd application
ONLINE
ONLINE
nwysum01
ora....m01.ons application
ONLINE
ONLINE
nwysum01
ora....m01.vip application
ONLINE
ONLINE
nwysum01
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02
If CRS not Started automatic then run next step

Connect as root and run the following command

o mwysum01:/ # bash
o mwysum01:/ # cd /app/oracle/product/10.2.0/crs_1/bin
o mwysum01:/app/oracle/product/10.2.0/crs_1/bin # ./crsctl start crs

<<paste output here>>

oracle@nwysum01:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] >crs_stat -t

<<paste output here>>

oracle@nwysum01:/app/oracle/product/10.2.0/db_1 [CLRTYDB1] > srvctl start

instance -d CLRTYDB -i CLRTYDB1

<<paste output here>>

22. If any error while starting DB need to start DB manually from sqlplus (startup)
oracle@nwysum01:/home/oracle [CLRTYDB1] >sqlplus '/as sysdba'
SQL*Plus: Release 10.2.0.4.0 - Production on Mon Apr 1 02:03:19 2013
Copyright (c) 1982, 2007, Oracle.

All Rights Reserved.

Connected to an idle instance.

SQL> startup mount
ORACLE instance started.
Total System Global Area 1.0737E+10 bytes
Fixed Size
2074016 bytes
Variable Size
5419043424 bytes
Database Buffers
5301600256 bytes
Redo Buffers
14700544 bytes
Database mounted.
SQL> alter database open;
Database altered.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release
10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
oracle@nwysum01:/home/oracle [CLRTYDB1] >date

Mon Apr 1 02:04:05 MAL 2013

oracle@nwysum01:/home/oracle [CLRTYDB1] >

23. Health Check after successful startup

crs_stat t
oracle@nwysum02:/home/oracle [CLRTYDB1] >crs_stat -t
Name
Type
Target
State
Host
-----------------------------------------------------------ora....B1.inst application
ONLINE
ONLINE
nwysum01
ora....B2.inst application
ONLINE
ONLINE
nwysum02
ora.CLRTYDB.db application
ONLINE
ONLINE
nwysum02
ora....SM1.asm application
ONLINE
ONLINE
nwysum01
ora....01.lsnr application
ONLINE
ONLINE
nwysum01
ora....m01.gsd application
ONLINE
ONLINE
nwysum01
ora....m01.ons application
ONLINE
ONLINE
nwysum01
ora....m01.vip application
ONLINE
ONLINE
nwysum01
ora....SM2.asm application
ONLINE
ONLINE
nwysum02
ora....02.lsnr application
ONLINE
ONLINE
nwysum02
ora....m02.gsd application
ONLINE
ONLINE
nwysum02
ora....m02.ons application
ONLINE
ONLINE
nwysum02
ora....m02.vip application
ONLINE
ONLINE
nwysum02
oracle@nwysum02:/home/oracle [CLRTYDB1] >date
Mon Apr 1 02:04:47 MAL 2013
crsctl check crs
oracle@nwysum01:/home/oracle [CLRTYDB1] >crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 02:07:15 MAL 2013

ps -ef |grep pmon|grep -v grep

oracle@nwysum01:/home/oracle [CLRTYDB1] >ps -ef |grep pmon|grep -v grep
oracle 5430
1 0 02:00:34 ?
0:00 asm_pmon_+ASM1
oracle 8393
1 0 02:03:29 ?
0:04 ora_pmon_CLRTYDB1
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 02:07:34 MAL 2013

Up listener clrtbkp
oracle@nwysum01:/home/oracle [CLRTYDB1] >lsnrctl start LISTENER_CLRTBKP
LSNRCTL for HPUX: Version 10.2.0.4.0 - Production on 01-APR-2013 02:05:44
Copyright (c) 1991, 2007, Oracle. All rights reserved.
Starting /app/oracle/product/10.2.0/db_1/bin/tnslsnr: please wait...
TNSLSNR for HPUX: Version 10.2.0.4.0 - Production
System parameter file is /app/oracle/product/10.2.0/db_1/network/admin/listener.ora
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.25.38.12)(PORT=1523)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=nwysum01-bkp)(PORT=1523)))
STATUS of the LISTENER
-----------------------Alias
LISTENER_CLRTBKP
Version
TNSLSNR for HPUX: Version 10.2.0.4.0 - Production
Start Date
01-APR-2013 02:05:44
Uptime
0 days 0 hr. 0 min. 0 sec
Trace Level
off
Security
ON: Local OS Authentication
SNMP
OFF

Listener Parameter File /app/oracle/product/10.2.0/db_1/network/admin/listener.ora

Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.25.38.12)(PORT=1523)))
Services Summary...
Service "CLRTYDB" has 1 instance(s).
Instance "CLRTYDB1", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
oracle@nwysum01:/home/oracle [CLRTYDB1] >

o
o
o
o
o
o
o
o

ps -ef|grep tns|grep -v grep

oracle@nwysum01:/home/oracle [CLRTYDB1] >ps -ef|grep tns|grep -v grep
oracle 5508
1 0 02:00:35 ?
0:00
/app/oracle/product/10.2.0/asm_1/bin/tnslsnr LISTENER_NWYSUM01 -inherit
oracle 10465
1 0 02:05:44 ?
0:00
/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER_CLRTBKP -inherit
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 02:07:54 MAL 2013
srvctl status instance -d CLRTYDB -i CLRTYDB1
oracle@nwysum01:/home/oracle [CLRTYDB1] >srvctl status instance -d
CLRTYDB -i CLRTYDB1
Instance CLRTYDB1 is running on node nwysum01
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
sh: a^?date: not found.
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 02:08:30 MAL 2013
srvctl status asm -n nwysum01
oracle@nwysum01:/home/oracle [CLRTYDB1] >srvctl status asm -n nwysum01
ASM instance +ASM1 is running on node nwysum01.
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 02:08:55 MAL 2013
srvctl status nodeapps -n nwysum01
oracle@nwysum01:/home/oracle [CLRTYDB1] >srvctl status nodeapps -n
nwysum01
VIP is running on node: nwysum01
GSD is running on node: nwysum01
Listener is running on node: nwysum01
ONS daemon is running on node: nwysum01
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 02:09:55 MAL 2013

SYS@CLRTYDB1>@ses

o
o
o
o

SYS@CLRTYDB1>@ses

o
o
o
o
o
o
o
o
o

CLOGNAME
----------------------------------------------------------------------------Sessions_CLRTYDB1_130401021335
!!!!! No of Session in each node !!!!!
01/04/2013 02:13:35
==================================Session Count Group By
Program==============
Ins TYPE

Status

Cnt Program Name

o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

---- ---------- --------- ----- -----------------------------1 BACKGROUND ACTIVE

26 oracle@nwysum01
********* ----sum
26
USER
ACTIVE
1 oracle@nwysum01
1 racgimon@nwysum01
1 sqlplusw.exe
********* ----sum
3
INACTIVE
50 JDBC
2 racgimon@nwysum01
********* ----sum
52
**** **********
----sum
81
2 BACKGROUND ACTIVE
26 oracle@nwysum02
********* ----sum
26
USER
ACTIVE
2 emagent@nwysum02
1 oracle@nwysum02
********* ----sum
3
INACTIVE
25 JDBC
8 SOPAgent@nwywin02
1 cfg_file_gener@nwywin02
3 emagent@nwysum02
2 frmweb@nwymon01
3 racgimon@nwysum02
1 toad.exe
********* ----sum
43
**** **********
----sum
72
----sum
153
=================================Session Count Group By
Status================
Ins TYPE
---- ---------1 BACKGROUND
USER
USER
2 BACKGROUND
USER
USER
****
sum

Status
SESSIONS
--------- ---------ACTIVE
26
ACTIVE
3
INACTIVE
52
ACTIVE
26
ACTIVE
3
INACTIVE
43
---------153

================================Session Count (Background

Excluded)===========
Ins Status
SESSIONS
---- --------- ----------

o
o
o
o
o
o
o
o

1 ACTIVE
3
INACTIVE
52
2 ACTIVE
3
INACTIVE
43
**** ********* ---------sum
101
!!!!! END !!!!!
SYS@CLRTYDB1>
SYS@CLRTYDB1>@sqls
SYS@CLRTYDB1>@sqls
CLOGNAME
------------------------------------------------------------------------------SQLs_CLRTYDB1_130401021406
!!!!! All About Running SQLs !!!!!
01/04/2013 02:14:06
USERNAME
Program Name
MACHINE
OSUSER
Status
PROCESS
--------------- ----------------------------------------------------------- CPRG
JDBC Thin Client
nwymon01
jboss
INACTIVE
SYS
sqlplusw.exe
YTLCOMMS\CMMS0025
fairdaus
ACTIVE
SYS
oracle@nwysum02 (PZ99)
YTLCOMMS\CMMS0025
fairdaus
ACTIVE
SYS
oracle@nwysum01 (PZ99)
YTLCOMMS\CMMS0025
fairdaus
ACTIVE

LOGON_TIME
SID
SERIAL#
-------------------01/04/2013
3248
01/04/2013
3267
01/04/2013
3
01/04/2013
3

02:07:38
2 1
02:13:33
170
02:14:06
02:14:06

24. Monitor Alert and Cluster logs

tail -f /app/oracle/admin/CLRTYDB/bdump/alert_CLRTYDB1.log
tail -f /app/oracle/admin/CLRTYDB/bdump/alert_CLRTYDB2.log
tail -f /app/oracle/product/10.2.0/crs_1/log/nwysum01/crsd/crsd.log
tail -f /app/oracle/product/10.2.0/crs_1/log/nwysum02/crsd/crsd.log

25. Start EM agent

o
o

. .oem_profile
emctl start agent
oracle@nwysum01:/home/oracle [CLRTYDB1] >emctl start agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
Starting agent ............. started.
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 02:15:18 MAL 2013
.

emctl status agent

oracle@nwysum01:/home/oracle [CLRTYDB1] >emctl status agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
--------------------------------------------------------------Agent Version
: 10.2.0.5.0
OMS Version
: 10.2.0.5.0

Protocol Version : 10.2.0.5.0

Agent Home
:
/app/oracle/OracleHomes/agent10g_new/agent10g/nwysum01
Agent binaries
: /app/oracle/OracleHomes/agent10g_new/agent10g
Agent Process ID : 19563
Parent Process ID : 19495
Agent URL
: https://nwysum01:3872/emd/main
Repository URL
: https://bansum02:1159/em/upload
Started at
: 2013-04-01 02:15:00
Started by user
: oracle
Last Reload
: 2013-04-01 02:15:00
Last successful upload
: 2013-04-01 02:15:25
Total Megabytes of XML files uploaded so far :
0.20
Number of XML files pending upload
:
1
Size of XML files pending upload(MB)
:
0.00
Available disk space on upload filesystem
:
18.69%
Last successful heartbeat to OMS
: 2013-04-01 02:15:15
--------------------------------------------------------------Agent is Running and Ready
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 02:15:35 MAL 2013

26. Glance Server 1 output

27. Glance Server 2 output

28. Cronjob detail

oracle@nwysum01:/home/oracle [CLRTYDB1] >crontab -l
30 03,19 * * * /home/oracle/arch/hot_archive_backup_CLRTYDB.sh >/dev/null 2>&1
00 20 * * 05 /home/oracle/script/alertlog.sh >/dev/null 2>&1
# -- Getting crs_stat output for monitoring
0,5,10,15,20,25,30,35,40,45,50,55 * * * *
/app/oracle/product/10.2.0/crs_1/bin/crs_stat>/home/oracle/crs_stat.out
00 02 2-8 * * /arc_prg/ap_script.sh >> /arc_prg/exp_dump/ap.log
#01 02 2 * * /arc_prg/arc.sh >> /arc_prg/exp_dump/clrt_arc.log
#00 02 4,5,6 * * /arc_prg/prg.sh >> /arc_prg/exp_dump/clrt_prg.log
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 02:21:21 MAL 2013

29. Mount Point

oracle@nwysum01:/home/oracle [CLRTYDB1] >bdf
Filesystem
kbytes
used
avail %used Mounted on
/dev/vg00/lvol3
10485760 930008 9481192
9% /
/dev/vg00/lvol1
2097152 580424 1504912
28% /stand
/dev/vg00/lvol8
10485760 5495536 4952400
53% /var
/dev/vg00/lvol7
10485760 7631640 2831928
73% /usr
/dev/vg00/lvol6
10485760 5430832 5016120
52% /tmp
/dev/vg00/lvol5
10485760 6648440 3807352
64% /opt
/dev/vg00/lvol4
10485760 2185880 8235104
21% /home
/dev/vg01/lv_clalogs
33554432 1305627 30233261
4% /claritylogs
/dev/arch/archvpg 132645888 52761791 74891463
41% /arc_prg
/dev/vg01/lv_claapp
33554432 26861028 6275182
81% /app
oracle@nwysum01:/home/oracle [CLRTYDB1] >date
Mon Apr 1 02:21:36 MAL 2013