Restore of OCR file from backup.

OCR file records cluster information of RAC and if OCR is affected then RAC will comes to halt.
By default, oracle takes the backup of OCR file for every four hours, at the end of the day & at the
end of the week. Out of all the nodes in RAC one of the nodes will be designated as Master Node
and it takes the backup of OCR file.

Whenever we make some structural changes to the cluster like adding a node, creating a new
services, database etc it is always recommended to take the back up of OCR file manually as it takes
backup for every 4 hours in version 10g.

From 11gr2 it takes the OCR backup automatically whenever there is a structural change.

OCR contains the following resource information

 RAC databases and instances information

 Nodes and node applications
 ASM diskgroups, volumes, filesystems, and instances
 SCAN listeners and local listeners
 SCAN VIPs and Local VIPs
 Nodes and node applications
 User defined Services

To find master node in the Cluster

[oracle@rac2 crsd]$ grep -i 'ocr master' crsd.log |tail -1

2018-11-27 22:27:49.268: [ OCRMAS][1102379328]th_master: NEW OCR MASTER IS 1

When you bring down the node 1 then Node 2 will become master node.
[oracle@rac2 crsd]$
[oracle@rac2 crsd]$ grep -i 'ocr master' crsd.log |tail -1
2018-11-27 22:47:27.079: [ OCRMAS][1102379328]th_master:13: I AM THE NEW OCR MASTER at
incar 2. Node Number 2

Checking backup of OCR file

[oracle@rac2 crsd]$ ocrconfig -showbackup

rac1 2016/08/13 15:55:40 /u01/app/11.2.0/grid/cdata/rac-scan/backup00.ocr

rac1 2016/06/25 16:41:23 /u01/app/11.2.0/grid/cdata/rac-scan/backup01.ocr

rac1 2016/08/13 15:55:40 /u01/app/11.2.0/grid/cdata/rac-scan/day.ocr

rac1 2016/08/13 15:55:40 /u01/app/11.2.0/grid/cdata/rac-scan/week.ocr

PROT-25: Manual backups for the Oracle Cluster Registry are not available
Manually taking backup of OCR file can be done with root user only.

[oracle@rac2 bin]$ ocrconfig -manualbackup

PROT-20: Insufficient permission to proceed. Require privileged user
[oracle@rac2 bin]$

[root@rac2 bin]# ./ocrconfig -manualbackup

rac2 2018/11/27 22:51:41 /u01/app/11.2.0/grid/cdata/rac-scan/backup_20181127_225141.ocr

[root@rac2 bin]#

[oracle@rac2 bin]$ ocrconfig -showbackup

rac1 2016/08/13 15:55:40 /u01/app/11.2.0/grid/cdata/rac-scan/backup00.ocr

rac1 2016/06/25 16:41:23 /u01/app/11.2.0/grid/cdata/rac-scan/backup01.ocr

rac1 2016/08/13 15:55:40 /u01/app/11.2.0/grid/cdata/rac-scan/day.ocr

rac1 2016/08/13 15:55:40 /u01/app/11.2.0/grid/cdata/rac-scan/week.ocr

rac2 2018/11/27 22:51:41 /u01/app/11.2.0/grid/cdata/rac-scan/backup_20181127_225141.ocr

[oracle@rac2 bin]$

[oracle@rac2 bin]$ ocrcheck

Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3148
Available space (kbytes) : 258972
ID : 1237155669
Device/File Name : +OVDG
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

[oracle@rac2 bin]$
Imagine OCR file has been corrupted and we are going to restore it from backup.

Here we have 2 node RAC setup with node name rac1 and rac2. Here we are performing restore of
OCR file from rac2 node.

[root@rac2 bin]# ./crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac2'
CRS-2673: Attempting to stop 'ora.crf' on 'rac2'
CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac2' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac2 bin]#

[root@rac1 bin]# ./crsctl stop crs

[root@rac2 bin]# ./crsctl check crs

CRS-4639: Could not contact Oracle High Availability Services

Start the crs in the exclusive mode, so that ASM would be accessible by reading OLR file.
Make sure CRSD is not running while restoring OCR file.

OLR is the first file that is accessed to start-up clusterware when OCR is stored on ASM. OCR
should be accessible to find out the resources which need to be started on a node. If OCR is on
ASM, it can’t be read until ASM is up. To resolve this problem, information about the resources
which need to be started on a node is stored in an operating system file which is called Oracle
Local Registry or OLR. Since OLR is an operating system file, it can be accessed by various
processes on the node for read/write irrespective of the status of the clusterware (up/down).
Hence, when a node joins the cluster, OLR on that node is read, various resources ,including
ASM are started on the node . Once ASM is up, OCR is accessible and is used henceforth to
manage all the clusterware resources. If OLR is missing or corrupted, clusterware can’t be
started on that node.

OCR is managed by CRSD process.

OLR is managed by OHASD process.

[root@rac2 bin]# ./crsctl start crs -excl

CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac2'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rac2'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac2'
CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'rac2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac2'
CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac2'
CRS-2676: Start of 'ora.crsd' on 'rac2' succeeded
[root@rac2 bin]#

[oracle@rac2 bin]$ crsctl stat res -init

NAME=ora.asm
TYPE=ora.asm.type
TARGET=ONLINE
STATE=ONLINE on rac2

NAME=ora.cluster_interconnect.haip
TYPE=ora.haip.type
TARGET=ONLINE
STATE=ONLINE on rac2

NAME=ora.crf
TYPE=ora.crf.type
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.crsd
TYPE=ora.crs.type
TARGET=ONLINE
STATE=INTERMEDIATE on rac2

[oracle@rac2 bin]$ crsctl stat res -init|grep ora.crsd

NAME=ora.crsd
[oracle@rac2 bin]$ ps -ef|grep crsd
root 15503 1 0 23:14 ? 00:00:00 /u01/app/11.2.0/grid/bin/crsd.bin reboot
oracle 15876 9536 0 23:16 pts/2 00:00:00 grep crsd
[oracle@rac2 bin]$

Stopping CRS resource

[oracle@rac2 bin]$ ./crsctl stop resource ora.crsd -init

CRS-2673: Attempting to stop 'ora.crsd' on 'rac2'
CRS-2677: Stop of 'ora.crsd' on 'rac2' succeeded
[oracle@rac2 bin]$

[oracle@rac2 bin]$ ps -ef|grep pmon

oracle 15433 1 0 23:14 ? 00:00:00 asm_pmon_+ASM2
oracle 16216 9536 0 23:18 pts/2 00:00:00 grep pmon
[oracle@rac2 bin]$ ps -ef|grep crsd
oracle 16230 9536 0 23:18 pts/2 00:00:00 grep crsd
[oracle@rac2 bin]$

[root@rac2 bin]# ./ocrconfig -showbackup

rac1 2016/08/13 15:55:40 /u01/app/11.2.0/grid/cdata/rac-scan/backup00.ocr

rac1 2016/06/25 16:41:23 /u01/app/11.2.0/grid/cdata/rac-scan/backup01.ocr

rac1 2016/08/13 15:55:40 /u01/app/11.2.0/grid/cdata/rac-scan/day.ocr

rac1 2016/08/13 15:55:40 /u01/app/11.2.0/grid/cdata/rac-scan/week.ocr

rac2 2018/11/27 22:51:41 /u01/app/11.2.0/grid/cdata/rac-scan/backup_20181127_225141.ocr

Restoring OCR file from manually backup

[root@rac2 bin]# ./ocrconfig -restore /u01/app/11.2.0/grid/cdata/rac-

scan/backup_20181127_225141.ocr
[root@rac2 bin]#

[root@rac2 bin]# ./ocrcheck

Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3120
Available space (kbytes) : 259000
ID : 1237155669
Device/File Name : +OVDG
 Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

Once OCR file has been restored, then stop the crs services.

[root@rac2 bin]# ./crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac2' succeeded
CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac2 bin]#

Starting the cluster and it will start all the services.

[root@rac2 bin]# ./crsctl start crs

CRS-4123: Oracle High Availability Services has been started.
[root@rac2 bin]#

[root@rac1 bin]# ./crsctl start crs

CRS-4123: Oracle High Availability Services has been started.
[root@rac1 bin]#
[root@rac2 bin]# ./crsctl start cluster -all
CRS-4690: Oracle Clusterware is already running on 'rac1'
CRS-4690: Oracle Clusterware is already running on 'rac2'
CRS-4000: Command Start failed, or completed with errors.

[root@rac2 bin]# ps -ef|grep pmon

oracle 17828 1 0 23:25 ? 00:00:00 asm_pmon_+ASM2
oracle 18652 1 0 23:25 ? 00:00:00 ora_pmon_prod2
root 19536 7580 0 23:27 pts/1 00:00:00 grep pmon

Verification of OCR integrity by using cluvfy utility

[oracle@rac2 bin]$ cluvfy comp ocr -n all -verbose

Verifying OCR integrity

Checking OCR integrity...

Checking the absence of a non-clustered configuration...

All nodes free of non-clustered, local-only configurations

ASM Running check passed. ASM is running on all specified nodes

Checking OCR config file "/etc/oracle/ocr.loc"...

OCR config file "/etc/oracle/ocr.loc" check successful

Disk group for ocr location "+OVDG" available on all the nodes

NOTE:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user
to verify the contents of OCR.

OCR integrity check passed

Verification of OCR integrity was successful.

[oracle@rac2 bin]$