This article will provide information on moving OCR-Voting devices to ASM

diskgroups in newly upgraded 11.2.0.3.0 Grid infrastructure home from 10.2.0.4

CRS. Please check my earlier blog for upgrading 10.2.0.4 CRS to 11.2.0.3

As we can see from the below output that ASM resource is added to CRS
resources but not online because ASM instances and DG resources are not
created yet.
[root@rac1 bin]# ./crsctl stat res ora.asm -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.asm
OFFLINE OFFLINE rac1
OFFLINE OFFLINE rac2
OCR/VOTING disks are stored in Block devices as if now.
[root@rac1 bin]# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 31752502fce75fb6bfcbbe041949734c (/dev/raw/raw2) []
Located 1 voting disk(s).
[root@rac1 bin]# cat /etc/oracle/ocr.loc
ocrconfig_loc=/dev/raw/raw1
local_only=FALSE
We can see 6 DISKS for ASM however we'll create diskgroup DATADG1 using two
disks only and later on we'll multiplex voting disks into different diskgroup
DATADG2 using remaining four disks.
[root@rac1 bin]# oracleasm listdisks
DISK1
DISK2
DISK3
DISK4
DISK5
DISK6
Please make sure that same disks are available on both nodes before proceeding
for running of asmca.
[root@rac2 bin]# oracleasm listdisks
DISK1
DISK2
DISK3
DISK4
DISK5
DISK6

We can verify CRS version as 11.2.0.3 from below commands.

[oracle@rac1 ~]$ crsctl query crs softwareversion
Oracle Clusterware version on node [rac1] is [11.2.0.3.0]
[oracle@rac1 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.3.0]

Now we'll run asmca as oracle user from 11.2.0.3 home. We'll get first screen as
shown below. Please enter sys and asmsnmp password for ASM instances and
click on specify disk group button.

Once you click on specify disk group button, it'll show you below screen so
please choose External in redundancy category, Specify diskgroup name as
DATADG1 and select ORCL:DISK1 and ORCL:DISK2.
Once it's done then click ok and it'll come to first page where DISKGROUP will be
mentioned as DATADG1 as shown below and now click on Create ASM button.

Once it'll create ASM instances and Diskgroup, it'll display message like below.
Once you click on ok in above screen, it'll display DG information as shown below
so review it and exit from ASMCA.

Adding OCR into ASM diskgroup

================================
[root@rac1 bin]# ls -tlr /dev/raw
total 0
crw-rw-r-- 1 oracle oinstall 162, 2 Sep 6 10:29 raw2 -> used for Voting
crw-rw---- 1 root oinstall 162, 1 Sep 6 11:08 raw1 ->used for OCR
[root@rac2 bin]# ./ocrconfig -add +DATADG1
[root@rac2 bin]# cat /etc/oracle/ocr.loc
#Device/file getting replaced by device +DATADG1
ocrconfig_loc=/dev/raw/raw1
ocrmirrorconfig_loc=+DATADG1
local_only=false[root@rac2 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 1039908
Used space (kbytes) : 4328
Available space (kbytes) : 1035580
ID : 133883669
Device/File Name : /dev/raw/raw1
Device/File integrity check succeeded
Device/File Name : +DATADG1
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded

Deleting OCR raw device and adding back

============================================

[root@rac2 bin]# ./ocrconfig -delete /dev/raw/raw1

[root@rac2 bin]# cat /etc/oracle/ocr.loc
#Device/file /dev/raw/raw1 getting replaced by device +DATADG1
ocrconfig_loc=+DATADG1
local_only=false[root@rac2 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 1039908
Used space (kbytes) : 4328
Available space (kbytes) : 1035580
ID : 133883669
Device/File Name : +DATADG1
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
Now we'll add it back OCR block device and this time it'll be added as mirror
device.

[root@rac2 bin]# ./ocrconfig -add /dev/raw/raw1

[root@rac2 bin]# cat /etc/oracle/ocr.loc
#Device/file getting replaced by device /dev/raw/raw1
ocrconfig_loc=+DATADG1
ocrmirrorconfig_loc=/dev/raw/raw1
local_only=false[root@rac2 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 1039908
Used space (kbytes) : 4328
Available space (kbytes) : 1035580
ID : 133883669
Device/File Name : +DATADG1
Device/File integrity check succeeded
Device/File Name : /dev/raw/raw1
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded

Moving/Replacing Voting Devices into ASM DG

[root@rac2 bin]# ./crsctl query css votedisk

## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 31752502fce75fb6bfcbbe041949734c (/dev/raw/raw2) []
Located 1 voting disk(s).
[root@rac2 bin]# ./crsctl add css votedisk +DATADG1
CRS-4671: This command is not supported for ASM diskgroups.
CRS-4000: Command Add failed, or completed with errors.
[root@rac2 bin]# ./crsctl replace votedisk +DATADG1
CRS-4256: Updating the profile
Successful addition of voting disk 0fd588d39f784fb8bf7d9fd3b7558dd3.
Successful deletion of voting disk 31752502fce75fb6bfcbbe041949734c.
Successfully replaced voting disk group with +DATADG1.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
[root@rac2 bin]# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
 1. ONLINE 0fd588d39f784fb8bf7d9fd3b7558dd3 (ORCL:DISK1) [DATADG1]
Located 1 voting disk(s).

Please note that we couldn't add voting devices to ASM while having voting
device into raw devices as well. We could only replace existing voting devices to
ASM DG using crsctl replace command as shown in above output.

Moving Voting files back to raw device

===================================
[root@rac2 bin]# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 0fd588d39f784fb8bf7d9fd3b7558dd3 (ORCL:DISK1) [DATADG1]
Located 1 voting disk(s).
[root@rac2 bin]# ./crsctl replace votedisk /dev/raw/raw2
Now formatting voting disk: /dev/raw/raw2.
CRS-4256: Updating the profile
Successful addition of voting disk 56c5891e74094fbbbf9a62bc50ce872a.
Successful deletion of voting disk 0fd588d39f784fb8bf7d9fd3b7558dd3.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
[root@rac2 bin]# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 56c5891e74094fbbbf9a62bc50ce872a (/dev/raw/raw2) []
Located 1 voting disk(s).

So from above exercise, we can conclude that we can OCR devices into ASM DG
and block devices at the same time as mirrored devices however for voting
devices, we can either into Block devices or ASM DG not both at the same time.

Example of replacingVoting devices into from one ASM

diskgroup to another.

Now we'll create second ASM diskgroup DATADG2 using remaining disks with
normal redundancy.
[oracle@rac1 ~]$ asmcmd
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB
Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 1048576 10228 9071 0
9071 0 Y DATADG1/
MOUNTED NORMAL N 512 4096 1048576 20456 20169
5114 7527 0 N DATADG2/
Since Voting disk is already located in DATADG1 and we want to move it to
DATADG2 while CRS is up, we get below error.
[root@rac1 bin]# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 43887170ea864f1cbf3c4008f21fc78a (ORCL:DISK1) [DATADG1]
Located 1 voting disk(s).
[root@rac1 bin]# ./crsctl add css votedisk +DATADG2
CRS-4671: This command is not supported for ASM diskgroups.
CRS-4000: Command Add failed, or completed with errors.
[root@rac1 bin]# ./crsctl replace votedisk +DATADG2
CRS-4602: Failed 27 to add voting file 6ac9fe19e3444fcfbf827a035155bd18.
CRS-4602: Failed 27 to add voting file a4ffb5d52ad14f7ebf563726fbe15761.
CRS-4602: Failed 27 to add voting file 92b962c4090c4f7abfd3184f721796cc.
Failure 27 with Cluster Synchronization Services while deleting voting disk.
Failed to replace voting disk group with +DATADG2.
CRS-4000: Command Replace failed, or completed with errors.
[root@rac1 bin]# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 43887170ea864f1cbf3c4008f21fc78a (ORCL:DISK1) [DATADG1]
Located 1 voting disk(s).

Please stop CRS on both nodes using crsctl stop crs and then start CRS in
exclusive mode on any of one node.
[root@rac1 bin]# ./crsctl start crs -excl
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac1'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
[root@rac1 bin]# ps -ef |grep -i d.bin
root 8102 1 2 10:55 ? 00:00:04 /u01/app/11.2.0/grid/bin/ohasd.bin
exclusive
oracle 8223 1 1 10:55 ? 00:00:02 /u01/app/11.2.0/grid/bin/oraagent.bin
oracle 8235 1 0 10:55 ? 00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin
oracle 8258 1 1 10:55 ? 00:00:01 /u01/app/11.2.0/grid/bin/gpnpd.bin
root 8268 1 0 10:55 ? 00:00:00 /u01/app/11.2.0/grid/bin/cssdmonitor
oracle 8270 1 1 10:55 ? 00:00:02 /u01/app/11.2.0/grid/bin/gipcd.bin
root 8290 1 0 10:55 ? 00:00:01 /u01/app/11.2.0/grid/bin/cssdagent
oracle 8314 1 2 10:55 ? 00:00:03 /u01/app/11.2.0/grid/bin/ocssd.bin -X
root 8367 1 1 10:56 ? 00:00:02
/u01/app/11.2.0/grid/bin/orarootagent.bin
root 8381 1 0 10:56 ? 00:00:00 /u01/app/11.2.0/grid/bin/octssd.bin
root 8675 1 1 10:56 ? 00:00:01 /u01/app/11.2.0/grid/bin/crsd.bin
reboot
root 8840 6677 0 10:58 pts/2 00:00:00 grep -i d.bin

[root@rac1 bin]# ./crsctl replace votedisk +DATADG2

Successful addition of voting disk ae7a31fc941d4fa6bf813b58cbf4ae6d.
Successful addition of voting disk 9a7cd52bf7e24fe4bfac1939a2433a2a.
Successful addition of voting disk 75459ba3e9134f0abfa54468ac51610c.
Successful deletion of voting disk 43887170ea864f1cbf3c4008f21fc78a.
Successfully replaced voting disk group with +DATADG2.
CRS-4266: Voting file(s) successfully replaced
[root@rac1 bin]# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE ae7a31fc941d4fa6bf813b58cbf4ae6d (ORCL:DISK3) [DATADG2]
2. ONLINE 9a7cd52bf7e24fe4bfac1939a2433a2a (ORCL:DISK4) [DATADG2]
3. ONLINE 75459ba3e9134f0abfa54468ac51610c (ORCL:DISK5) [DATADG2]
Located 3 voting disk(s).

From above output, we can see that now VOTING disk is moved into three disks
out of 4 ASM disks since redundancy of ASM diskgroup is normal. now stop crs
and start on both nodes. We've to use force option to shutdown CRS.
[root@rac1 bin]# ./crsctl stop crs
CRS-2796: The command may not proceed when Cluster Ready Services is not
running
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
[root@rac1 bin]# ps -ef |grep -i d.bin
root 8102 1 1 10:55 ? 00:00:06 /u01/app/11.2.0/grid/bin/ohasd.bin
exclusive
oracle 8223 1 1 10:55 ? 00:00:06 /u01/app/11.2.0/grid/bin/oraagent.bin
oracle 8235 1 0 10:55 ? 00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin
oracle 8258 1 0 10:55 ? 00:00:02 /u01/app/11.2.0/grid/bin/gpnpd.bin
root 8268 1 0 10:55 ? 00:00:01 /u01/app/11.2.0/grid/bin/cssdmonitor
oracle 8270 1 0 10:55 ? 00:00:03 /u01/app/11.2.0/grid/bin/gipcd.bin
root 8290 1 0 10:55 ? 00:00:01 /u01/app/11.2.0/grid/bin/cssdagent
oracle 8314 1 1 10:55 ? 00:00:07 /u01/app/11.2.0/grid/bin/ocssd.bin -X
root 8367 1 1 10:56 ? 00:00:05
/u01/app/11.2.0/grid/bin/orarootagent.bin
root 8381 1 0 10:56 ? 00:00:01 /u01/app/11.2.0/grid/bin/octssd.bin
root 8675 1 0 10:56 ? 00:00:01 /u01/app/11.2.0/grid/bin/crsd.bin
reboot
root 9331 6677 0 11:03 pts/2 00:00:00 grep -i d.bin
[root@rac1 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed
resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on
'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac1 bin]# ps -ef |grep -i d.bin
root 9500 6677 0 11:04 pts/2 00:00:00 grep -i d.bin

[root@rac1 bin]# ./crsctl query css votedisk

## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 44b55c5576154fc2bf3e54f08099a0e9 (ORCL:DISK3) [DATADG2]
2. ONLINE 0fcd8b592b604faabf9120ffef2a8e18 (ORCL:DISK5) [DATADG2]
 3. ONLINE 00dc165a6d5c4ffebf4ba43496477d13 (ORCL:DISK6) [DATADG2]
Located 3 voting disk(s).
[root@rac1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

[root@rac2 bin]# ./crsctl query css votedisk

## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 44b55c5576154fc2bf3e54f08099a0e9 (ORCL:DISK3) [DATADG2]
2. ONLINE 0fcd8b592b604faabf9120ffef2a8e18 (ORCL:DISK5) [DATADG2]
3. ONLINE 00dc165a6d5c4ffebf4ba43496477d13 (ORCL:DISK6) [DATADG2]
Located 3 voting disk(s).
[root@rac2 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

We've to perform same steps for OCR replacement like stopping CRS and starting
in exclusive mode. Since we added/removed copy of OCR, we need not to
performs stop/start(exclusive) operations.