How to Restore OCR and Voting Disk

  • by

There're 2 procedures of restoring OCR and voting disk in this post. They are:

  1. Restore OCR and Voting Disk For 12c Clusterware
  2. Restore OCR and Voting Disk For 10g Clusterware

A. For 12c Clusterware

Suppose we have lost all disk groups as well as OCR data, voting disk and management database (MGMTDB) in a disaster, we have to rebuild 12c clusterware nearly from the ground up for later 12c RAC database restoration. The only thing that you must have is a latest OCR backup.

In this procedure, I will show you how to restore 12c clusterware with the following steps:

  1. Recreate Disk Groups
  2. Restore OCR Data
  3. Restore Voting Disk
  4. Recreate ASM SPFILE
  5. Restart CRS
  6. Recreate MGMTDB
  7. Create ASM Password File

1. Recreate Disk Groups

Login as root.

[root@primary01 ~]# . /home/grid/.bash_profile

Stop CRS.

[root@primary01 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'primary01'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'primary01'
CRS-2673: Attempting to stop 'ora.gipcd' on 'primary01'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'primary01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'primary01'
CRS-2673: Attempting to stop 'ora.evmd' on 'primary01'
CRS-2677: Stop of 'ora.cssdmonitor' on 'primary01' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'primary01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'primary01' has completed
CRS-4133: Oracle High Availability Services has been stopped.

Start CRS exclusively with nocrs.

[root@primary01 ~]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'primary01'
CRS-2672: Attempting to start 'ora.mdnsd' on 'primary01'
CRS-2676: Start of 'ora.evmd' on 'primary01' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'primary01'
CRS-2676: Start of 'ora.gpnpd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'primary01'
CRS-2672: Attempting to start 'ora.gipcd' on 'primary01'
CRS-2676: Start of 'ora.cssdmonitor' on 'primary01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'primary01'
CRS-2672: Attempting to start 'ora.diskmon' on 'primary01'
CRS-2676: Start of 'ora.diskmon' on 'primary01' succeeded
CRS-2676: Start of 'ora.cssd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'primary01'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'primary01'
CRS-2672: Attempting to start 'ora.ctssd' on 'primary01'
CRS-2676: Start of 'ora.ctssd' on 'primary01' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'primary01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'primary01'
CRS-2676: Start of 'ora.asm' on 'primary01' succeeded

Make sure that shared disks are existing by user grid.

[grid@primary01 ~]$ ll /dev/oracleasm/disks/
total 0
brw-rw---- 1 grid asmadmin 8, 33 Jun 26 11:27 ASM01
brw-rw---- 1 grid asmadmin 8, 17 Jun 26 11:27 ASM02

Make sure the disk group name. As usual, it's called DATA or CRS. If you have no clue about it, you can check the following file.

[grid@primary01 ~]$ cat /etc/oracle/ocr.loc
#Device/file +DATA getting replaced by device +DATA/primary-cluster/OCRFILE/registry.255.887454747
ocrconfig_loc=+DATA/primary-cluster/OCRFILE/registry.255.887454747
local_only=false

Create ASM diskgroups DATA as it was.

[grid@primary01 ~]$ sqlplus / as sysasm
...
SQL> alter system set asm_diskstring='/dev/oracleasm/disks/*' scope=memory;

System altered.

SQL> create diskgroup DATA external redundancy disk '/dev/oracleasm/disks/ASM01' attribute 'COMPATIBLE.ASM' = '12.1.0';

Diskgroup created.

Disk group DATA has been created. Please note that, ATTRIBUTE clause in the above statement is pretty important, don't miss it.

2. Restore OCR Data

Back to root. We start to restore OCR and then check the current setting of it.

[root@primary01 ~]# ocrconfig -restore /u01/app/12.1.0/grid/cdata/primary-cluster/backup_20190612_191205.ocr
[root@primary01 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       1680
         Available space (kbytes) :     407888
         ID                       : 1125371174
         Device/File Name         :      +DATA
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

Now we check OCR location file again.

[root@primary01 ~]# cat /etc/oracle/ocr.loc
#Device/file +DATA/primary-cluster/OCRFILE/registry.255.887454747 getting replaced by device +DATA/primary-cluster/OCRFILE/registry.255.1011959477
ocrconfig_loc=+DATA/primary-cluster/OCRFILE/registry.255.1011959477
local_only=false

The destination of OCR file has been replaced.

3. Restore Voting Disk

Since we have restored OCR, we restored voting file as well. You can just replace with new voting file on disk group DATA.

[root@primary01 ~]# crsctl replace votedisk +DATA
Successful addition of voting disk a6f4735a13a64f60bf31092700473fad.
Successfully replaced voting disk group with +DATA.
CRS-4266: Voting file(s) successfully replaced
[root@primary01 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   a6f4735a13a64f60bf31092700473fad (/dev/oracleasm/disks/ASM01) [DATA]
Located 1 voting disk(s).

4. Recreate ASM SPFILE

We use current ASM instance to recreate an ASM SPFILE by grid.

SQL> create spfile='+DATA' from memory;

File created.

5. Restart CRS

Now we can restart CRS to verify the result.

[root@primary01 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'primary01'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'primary01'
CRS-2673: Attempting to stop 'ora.ctssd' on 'primary01'
CRS-2673: Attempting to stop 'ora.asm' on 'primary01'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'primary01'
CRS-2673: Attempting to stop 'ora.evmd' on 'primary01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'primary01'
CRS-2677: Stop of 'ora.drivers.acfs' on 'primary01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.asm' on 'primary01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'primary01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'primary01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'primary01'
CRS-2677: Stop of 'ora.cssd' on 'primary01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'primary01'
CRS-2677: Stop of 'ora.gipcd' on 'primary01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'primary01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@primary01 ~]# crsctl start crs -wait
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'primary01'
CRS-2672: Attempting to start 'ora.evmd' on 'primary01'
CRS-2676: Start of 'ora.mdnsd' on 'primary01' succeeded
CRS-2676: Start of 'ora.evmd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'primary01'
CRS-2676: Start of 'ora.gpnpd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'primary01'
CRS-2676: Start of 'ora.gipcd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'primary01'
CRS-2676: Start of 'ora.cssdmonitor' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'primary01'
CRS-2672: Attempting to start 'ora.diskmon' on 'primary01'
CRS-2676: Start of 'ora.diskmon' on 'primary01' succeeded
CRS-2676: Start of 'ora.cssd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'primary01'
CRS-2672: Attempting to start 'ora.ctssd' on 'primary01'
CRS-2676: Start of 'ora.ctssd' on 'primary01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'primary01'
CRS-2676: Start of 'ora.asm' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'primary01'
CRS-2676: Start of 'ora.storage' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'primary01'
CRS-2676: Start of 'ora.crf' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'primary01'
CRS-2676: Start of 'ora.crsd' on 'primary01' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: primary01
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'primary01'
CRS-2672: Attempting to start 'ora.net2.network' on 'primary01'
CRS-2672: Attempting to start 'ora.net1.network' on 'primary01'
CRS-2672: Attempting to start 'ora.oc4j' on 'primary01'
CRS-2676: Start of 'ora.net2.network' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.primary01_2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.primary02_2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan1_net2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan2_net2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan3_net2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan4_net2.vip' on 'primary01'
CRS-2676: Start of 'ora.net1.network' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'primary01'
CRS-2672: Attempting to start 'ora.ons' on 'primary01'
CRS-2672: Attempting to start 'ora.primary01.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.primary02.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan4.vip' on 'primary01'
CRS-2676: Start of 'ora.cvu' on 'primary01' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'primary01' succeeded
CRS-2676: Start of 'ora.ons' on 'primary01' succeeded
CRS-2676: Start of 'ora.primary02_2.vip' on 'primary01' succeeded
CRS-2676: Start of 'ora.scan2_net2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER2_SCAN2_NET2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan3_net2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER2_SCAN3_NET2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.primary01.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan1_net2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER2_SCAN1_NET2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.primary02.vip' on 'primary01' succeeded
CRS-2676: Start of 'ora.scan4_net2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER2_SCAN4_NET2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.primary01_2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan1.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan3.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan4.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN4.lsnr' on 'primary01'
CRS-2676: Start of 'ora.LISTENER2_SCAN3_NET2.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.smalldb.db' on 'primary01'
CRS-2676: Start of 'ora.LISTENER2_SCAN4_NET2.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER2.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER2_SCAN1_NET2.lsnr' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.compdb.db' on 'primary01'
CRS-2672: Attempting to start 'ora.mgmtdb' on 'primary01'
CRS-2676: Start of 'ora.LISTENER2_SCAN2_NET2.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN4.lsnr' on 'primary01' succeeded
CRS-5017: The resource action "ora.compdb.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/compdb/spfileprimdb.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/compdb/spfileprimdb.ora
ORA-15056: additional error message
ORA-17503: ksfdopn:2 Failed to open file +DATA/compdb/spfileprimdb.ora
ORA-15173: entry 'compdb' does not exist in directory '/'
ORA-06512: at line 4
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/primary01/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.compdb.db' on 'primary01' failed
CRS-2679: Attempting to clean 'ora.compdb.db' on 'primary01'
CRS-2676: Start of 'ora.oc4j' on 'primary01' succeeded
CRS-5017: The resource action "ora.mgmtdb start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/_mgmtdb/spfile-MGMTDB.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/_mgmtdb/spfile-MGMTDB.ora
ORA-15056: additional error message
ORA-17503: ksfdopn:2 Failed to open file +DATA/_mgmtdb/spfile-mgmtdb.ora
ORA-15173: entry '_mgmtdb' does not exist in directory '/'
ORA-06512: at line 4
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/primary01/crs/trace/crsd_oraagent_grid.trc".
CRS-2674: Start of 'ora.mgmtdb' on 'primary01' failed
CRS-2679: Attempting to clean 'ora.mgmtdb' on 'primary01'
CRS-2681: Clean of 'ora.compdb.db' on 'primary01' succeeded
CRS-2681: Clean of 'ora.mgmtdb' on 'primary01' succeeded
CRS-2676: Start of 'ora.smalldb.db' on 'primary01' succeeded
===== Summary of resource auto-start failures follows =====
CRS-2807: Resource 'ora.compdb.db' failed to start automatically.
CRS-2807: Resource 'ora.mgmtdb' failed to start automatically.
CRS-6016: Resource auto-start has completed for server primary01
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.

Now we can check where ASM's parameter file is.

SQL> show parameter spfile;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +DATA/primary-cluster/ASMPARAM
                                                 ETERFILE/registry.253.10120620
                                                 47

As you can see in the above CRS starting log, most of resources are back online, except MGMTDB and database related ones.

6. Recreate MGMTDB

You can also call MGMTDB as Grid Infrastructure Management Repository (GIMR), they are the same thing.

Before recreating MGMTDB, we have to stop and disable Cluster Health Monitor (CRF) service before recreating MGMTDB.

[root@primary01 ~]# crsctl stop res ora.crf -init
CRS-2673: Attempting to stop 'ora.crf' on 'primary01'
CRS-2677: Stop of 'ora.crf' on 'primary01' succeeded
[root@primary01 ~]# crsctl modify res ora.crf -attr ENABLED=0 -init

Remove stale configuration of MGMTDB if any.

[grid@primary01 ~]$ srvctl remove mgmtdb
Remove the database _mgmtdb? (y/[n]) y

Create a MGMTDB for ASM administration by DBCA.

[grid@primary01 ~]$ dbca -silent -createDatabase -sid -MGMTDB -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -gdbName _mgmtdb -storageType ASM -diskGroupName +DATA -datafileJarLocation $ORACLE_HOME/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck
Registering database with Oracle Grid Infrastructure
5% complete
Copying database files
7% complete
9% complete
16% complete
23% complete
30% complete
37% complete
41% complete
Creating and starting Oracle instance
43% complete
48% complete
49% complete
50% complete
55% complete
60% complete
61% complete
64% complete
Completing Database Creation
68% complete
79% complete
89% complete
100% complete
Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb0.log" for further details.

Check any new folder in disk group DATA.

[grid@primary01 ~]$ asmcmd ls -l +DATA
Type  Redund  Striped  Time             Sys  Name
                                        Y    _MGMTDB/
                                        Y    primary-cluster/
[grid@primary01 ~]$ asmcmd ls -l +DATA/_MGMTDB/
Type  Redund  Striped  Time             Sys  Name
                                        Y    8C4B00F2B5AF559BE0530B2AA8C0FE14/
                                        Y    CONTROLFILE/
                                        Y    DATAFILE/
                                        Y    FD9AC0F7C36E4438E043B6A9E80A24D5/
                                        Y    ONLINELOG/
                                        Y    PARAMETERFILE/
                                        Y    TEMPFILE/

Now we have _MGMTDB/ back.

Since GIMR is a multitenant database with a pluggable database (PDB) for the GIMR of each cluster, we need to create it, too.

[grid@primary01 ~]$ dbca -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName primarycluster -createPDBFrom RMANBACKUP -PDBBackUpfile $ORACLE_HOME/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile $ORACLE_HOME/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true -internalSkipGIHomeCheck
Creating Pluggable Database
4% complete
12% complete
21% complete
38% complete
55% complete
85% complete
Completing Pluggable Database Creation
100% complete
Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/primarycluster/_mgmtdb.log" for further details.

MGMTDB is back, we can enable and start CRF now.

[root@primary01 ~]# crsctl modify res ora.crf -attr ENABLED=1 -init
[root@primary01 ~]# crsctl start res ora.crf -init
CRS-2672: Attempting to start 'ora.crf' on 'primary01'
CRS-2676: Start of 'ora.crf' on 'primary01' succeeded
[root@primary01 ~]# crsctl status res ora.mgmtdb
NAME=ora.mgmtdb
TYPE=ora.mgmtdb.type
TARGET=ONLINE
STATE=ONLINE on primary01

7. Create ASM Password File

Switch to grid and use orapwd to create password file for ASM.

[grid@primary01 ~]$ orapwd file='+DATA' asm=y password=welcome1 force=y
[grid@primary01 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +DATA/ASM/PASSWORD/pwdasm.275.1012063795
ASM listener: LISTENER
[grid@primary01 ~]$ asmcmd ls -l +DATA/ASM/PASSWORD/
Type      Redund  Striped  Time             Sys  Name
PASSWORD  UNPROT  COARSE   JUN 26 16:00:00  Y    pwdasm.275.1012063795

Next, we should restore the entire 12c RAC database back.

B. For 10g Clusterware

Suppose we have lost the shared storage as well as OCR data, voting disk and disk groups in a disaster. Later, a new shared storage for 10g RAC database has been established.

It looks like the worst case, but we still have backups of OCR and voting disk on server, plus a full database backup. That is to say, we need to rebuild RAC environment almost from the ground, except installed clusterware and database software.

In this procedure, I will start from OCR recovery, and then voting disk recovery. Since the disk groups we created previously are all gone, we need to recreate them back.

  1. Restore OCR Data
  2. Restore Voting Disk
  3. Recreate Disk Groups

1. Restore OCR Data

We can manually change the location of OCR from one to another on both nodes.

Node 1:

[root@primary01 ~]# . /home/oracle/.bash_profile
[root@primary01 ~]# crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM
[root@primary01 ~]# vi /etc/oracle/ocr.loc
#ocrconfig_loc=/dev/raw/raw1
ocrconfig_loc=/dev/raw/raw6
local_only=FALSE

Node 2:

[root@primary02 ~]# . /home/oracle/.bash_profile
[root@primary02 ~]# crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM
[root@primary02 ~]# vi /etc/oracle/ocr.loc
#ocrconfig_loc=/dev/raw/raw1
ocrconfig_loc=/dev/raw/raw6
local_only=FALSE

Then we restore OCR configuration by importing the backup dump on node 1.

[root@primary01 ~]# ocrconfig -import ocr_backup_20190612.dmp

Let's see current configuration of OCR.

[root@primary01 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :   18874264
         Used space (kbytes)      :       3788
         Available space (kbytes) :   18870476
         ID                       :  559990555
         Device/File Name         : /dev/raw/raw6
                                    Device/File integrity check succeeded

                                    Device/File not configured

         Cluster registry integrity check succeeded

2. Restore Voting Disk

You can add a new voting disk and delete the old one while CRS is not working.

[root@primary01 ~]# crsctl add css votedisk /dev/raw/raw7 -force
Now formatting voting disk: /dev/raw/raw7
successful addition of votedisk /dev/raw/raw7.
[root@primary01 ~]# crsctl delete css votedisk /dev/raw/raw2 -force
successful deletion of votedisk /dev/raw/raw2.

Then we restore the voting disk by dd command, just like we did in backing up voting disk.

[root@primary01 ~]# dd if=vod_backup_20190612.dmp of=/dev/raw/raw7 bs=1k count=500k
512000+0 records in
512000+0 records out
524288000 bytes (524 MB) copied, 103.76 seconds, 5.1 MB/s

Please note that, if you want all content of the dump file to be restored, you can omit count option.

[root@primary01 ~]# dd if=vod_backup_20190612.dmp of=/dev/raw/raw7 bs=1k

Let's see current configuration of voting disk on both nodes.

Node 1:

[root@primary01 ~]# crsctl query css votedisk
 0.     0    /dev/raw/raw7

located 1 votedisk(s).

Node 2:

[root@primary02 ~]# crsctl query css votedisk
 0.     0    /dev/raw/raw7

located 1 votedisk(s).

Restart both nodes to verify the results.

[root@primary01 ~]# init 6 [root@primary02 ~]# init 6

Check all CRS resources.

[oracle@primary01 ~]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....B1.inst application    ONLINE    OFFLINE
ora....B2.inst application    ONLINE    OFFLINE
ora.PRIMDB.db  application    ONLINE    OFFLINE
ora....SM1.asm application    ONLINE    ONLINE    primary01
ora....01.lsnr application    ONLINE    ONLINE    primary01
ora....y01.gsd application    ONLINE    ONLINE    primary01
ora....y01.ons application    ONLINE    ONLINE    primary01
ora....y01.vip application    ONLINE    ONLINE    primary01
ora....SM2.asm application    ONLINE    ONLINE    primary02
ora....02.lsnr application    ONLINE    ONLINE    primary02
ora....y02.gsd application    ONLINE    ONLINE    primary02
ora....y02.ons application    ONLINE    ONLINE    primary02
ora....y02.vip application    ONLINE    ONLINE    primary02

So far, we at least get CRS back.

3. Recreate Disk Groups

We switch to ASM mode.

[oracle@primary01 ~]$ export ORACLE_SID=+ASM1
[oracle@primary01 ~]$ sqlplus / as sysdba
...
SQL> column diskgroup format a10;
SQL> column type format a10;
SQL> column disk_path format a40;
SQL> select g.name diskgroup, g.type, d.path disk_path from v$asm_diskgroup g, v$asm_disk d where g.group_number = d.group_number;

no rows selected

We have nothing here. Let's recreate two disk groups back.

SQL> create diskgroup ora_data external redundancy disk '/dev/raw/raw8';

Diskgroup created.

SQL> create diskgroup fra_data external redundancy disk '/dev/raw/raw9';

Diskgroup created.

Check disk group again.

SQL> select g.name diskgroup, g.type, d.path disk_path from v$asm_diskgroup g, v$asm_disk d where g.group_number = d.group_number;

DISKGROUP  TYPE       DISK_PATH
---------- ---------- ----------------------------------------
ORA_DATA   EXTERN     /dev/raw/raw8
FRA_DATA   EXTERN     /dev/raw/raw9

Disk groups are back. In fact, using DBCA in 10 g can be an alternative to create disk groups, which is somewhat equivalent to ASMCA in 11g.

[oracle@primary01 ~]$ srvctl config database
PRIMDB
[oracle@primary01 ~]$ srvctl config database -d primdb
primary01 PRIMDB1 /u01/app/oracle/product/10.2.0/db_1
primary02 PRIMDB2 /u01/app/oracle/product/10.2.0/db_1
[oracle@primary01 ~]$ srvctl status database -d primdb
Instance PRIMDB1 is not running on node primary01
Instance PRIMDB2 is not running on node primary02

Next, we should restore the entire 10g RAC database back from a backup set.

Leave a Reply

Your email address will not be published. Required fields are marked *