Clusterware

How to Restore 12c Clusterware OCR Voting Disk

For 10g clusterware, you may check: How to Restore 10g Clusterware.

Suppose we have lost all disk groups as well as OCR data, voting disk and management database (MGMTDB) in a disaster, we have to rebuild 12c clusterware nearly from the ground up for later 12c RAC database restoration. The only thing that you must have is a latest OCR backup.

In this post, I will show you how to restore 12c clusterware with the following steps:

  1. Recreate Disk Groups
  2. Restore OCR Data
  3. Restore Voting Disk
  4. Recreate ASM SPFILE
  5. Restart CRS
  6. Recreate MGMTDB
  7. Create ASM Password File

1. Recreate Disk Groups

Login as root.

[root@primary01 ~]# . /home/grid/.bash_profile

Stop CRS.

[root@primary01 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'primary01'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'primary01'
CRS-2673: Attempting to stop 'ora.gipcd' on 'primary01'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'primary01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'primary01'
CRS-2673: Attempting to stop 'ora.evmd' on 'primary01'
CRS-2677: Stop of 'ora.cssdmonitor' on 'primary01' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'primary01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'primary01' has completed
CRS-4133: Oracle High Availability Services has been stopped.

Start CRS exclusively with nocrs.

[root@primary01 ~]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'primary01'
CRS-2672: Attempting to start 'ora.mdnsd' on 'primary01'
CRS-2676: Start of 'ora.evmd' on 'primary01' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'primary01'
CRS-2676: Start of 'ora.gpnpd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'primary01'
CRS-2672: Attempting to start 'ora.gipcd' on 'primary01'
CRS-2676: Start of 'ora.cssdmonitor' on 'primary01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'primary01'
CRS-2672: Attempting to start 'ora.diskmon' on 'primary01'
CRS-2676: Start of 'ora.diskmon' on 'primary01' succeeded
CRS-2676: Start of 'ora.cssd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'primary01'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'primary01'
CRS-2672: Attempting to start 'ora.ctssd' on 'primary01'
CRS-2676: Start of 'ora.ctssd' on 'primary01' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'primary01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'primary01'
CRS-2676: Start of 'ora.asm' on 'primary01' succeeded

Make sure that shared disks are existing by user grid.

[grid@primary01 ~]$ ll /dev/oracleasm/disks/
total 0
brw-rw---- 1 grid asmadmin 8, 33 Jun 26 11:27 ASM01
brw-rw---- 1 grid asmadmin 8, 17 Jun 26 11:27 ASM02

Make sure the disk group name of OCR. As usual, it’s called DATA.

[grid@primary01 ~]$ cat /etc/oracle/ocr.loc
#Device/file +DATA getting replaced by device +DATA/primary-cluster/OCRFILE/registry.255.887454747
ocrconfig_loc=+DATA/primary-cluster/OCRFILE/registry.255.887454747
local_only=false

Create ASM diskgroups DATA as it was.

[grid@primary01 ~]$ sqlplus / as sysasm
...
SQL> alter system set asm_diskstring='/dev/oracleasm/disks/*' scope=memory;

System altered.

SQL> create diskgroup DATA external redundancy disk '/dev/oracleasm/disks/ASM01' attribute 'COMPATIBLE.ASM' = '12.1.0';

Diskgroup created.

Disk group DATA has been created. Please note that, ATTRIBUTE clause in the above statement is pretty important, don’t miss it.

2. Restore OCR Data

Back to root. We start to restore OCR and then check the current setting of it.

[root@primary01 ~]# ocrconfig -restore /u01/app/12.1.0/grid/cdata/primary-cluster/backup_20190612_191205.ocr
[root@primary01 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       1680
         Available space (kbytes) :     407888
         ID                       : 1125371174
         Device/File Name         :      +DATA
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

Now we check OCR location file again.

[root@primary01 ~]# cat /etc/oracle/ocr.loc
#Device/file +DATA/primary-cluster/OCRFILE/registry.255.887454747 getting replaced by device +DATA/primary-cluster/OCRFILE/registry.255.1011959477
ocrconfig_loc=+DATA/primary-cluster/OCRFILE/registry.255.1011959477
local_only=false

The destination of OCR file has been replaced.

3. Restore Voting Disk

Since we have restored OCR, we restored voting file as well. You can just replace with new voting file on disk group DATA.

[root@primary01 ~]# crsctl replace votedisk +DATA
Successful addition of voting disk a6f4735a13a64f60bf31092700473fad.
Successfully replaced voting disk group with +DATA.
CRS-4266: Voting file(s) successfully replaced
[root@primary01 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   a6f4735a13a64f60bf31092700473fad (/dev/oracleasm/disks/ASM01) [DATA]
Located 1 voting disk(s).

4. Recreate ASM SPFILE

We use current ASM instance to recreate an ASM SPFILE by grid.

SQL> create spfile='+DATA' from memory;

File created.

5. Restart CRS

Now we can restart CRS to verify the result.

[root@primary01 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'primary01'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'primary01'
CRS-2673: Attempting to stop 'ora.ctssd' on 'primary01'
CRS-2673: Attempting to stop 'ora.asm' on 'primary01'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'primary01'
CRS-2673: Attempting to stop 'ora.evmd' on 'primary01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'primary01'
CRS-2677: Stop of 'ora.drivers.acfs' on 'primary01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.asm' on 'primary01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'primary01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'primary01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'primary01'
CRS-2677: Stop of 'ora.cssd' on 'primary01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'primary01'
CRS-2677: Stop of 'ora.gipcd' on 'primary01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'primary01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@primary01 ~]# crsctl start crs -wait
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'primary01'
CRS-2672: Attempting to start 'ora.evmd' on 'primary01'
CRS-2676: Start of 'ora.mdnsd' on 'primary01' succeeded
CRS-2676: Start of 'ora.evmd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'primary01'
CRS-2676: Start of 'ora.gpnpd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'primary01'
CRS-2676: Start of 'ora.gipcd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'primary01'
CRS-2676: Start of 'ora.cssdmonitor' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'primary01'
CRS-2672: Attempting to start 'ora.diskmon' on 'primary01'
CRS-2676: Start of 'ora.diskmon' on 'primary01' succeeded
CRS-2676: Start of 'ora.cssd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'primary01'
CRS-2672: Attempting to start 'ora.ctssd' on 'primary01'
CRS-2676: Start of 'ora.ctssd' on 'primary01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'primary01'
CRS-2676: Start of 'ora.asm' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'primary01'
CRS-2676: Start of 'ora.storage' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'primary01'
CRS-2676: Start of 'ora.crf' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'primary01'
CRS-2676: Start of 'ora.crsd' on 'primary01' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: primary01
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'primary01'
CRS-2672: Attempting to start 'ora.net2.network' on 'primary01'
CRS-2672: Attempting to start 'ora.net1.network' on 'primary01'
CRS-2672: Attempting to start 'ora.oc4j' on 'primary01'
CRS-2676: Start of 'ora.net2.network' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.primary01_2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.primary02_2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan1_net2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan2_net2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan3_net2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan4_net2.vip' on 'primary01'
CRS-2676: Start of 'ora.net1.network' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'primary01'
CRS-2672: Attempting to start 'ora.ons' on 'primary01'
CRS-2672: Attempting to start 'ora.primary01.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.primary02.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan4.vip' on 'primary01'
CRS-2676: Start of 'ora.cvu' on 'primary01' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'primary01' succeeded
CRS-2676: Start of 'ora.ons' on 'primary01' succeeded
CRS-2676: Start of 'ora.primary02_2.vip' on 'primary01' succeeded
CRS-2676: Start of 'ora.scan2_net2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER2_SCAN2_NET2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan3_net2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER2_SCAN3_NET2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.primary01.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan1_net2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER2_SCAN1_NET2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.primary02.vip' on 'primary01' succeeded
CRS-2676: Start of 'ora.scan4_net2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER2_SCAN4_NET2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.primary01_2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan1.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan3.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan4.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN4.lsnr' on 'primary01'
CRS-2676: Start of 'ora.LISTENER2_SCAN3_NET2.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.smalldb.db' on 'primary01'
CRS-2676: Start of 'ora.LISTENER2_SCAN4_NET2.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER2.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER2_SCAN1_NET2.lsnr' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.compdb.db' on 'primary01'
CRS-2672: Attempting to start 'ora.mgmtdb' on 'primary01'
CRS-2676: Start of 'ora.LISTENER2_SCAN2_NET2.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN4.lsnr' on 'primary01' succeeded
CRS-5017: The resource action "ora.compdb.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/compdb/spfileprimdb.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/compdb/spfileprimdb.ora
ORA-15056: additional error message
ORA-17503: ksfdopn:2 Failed to open file +DATA/compdb/spfileprimdb.ora
ORA-15173: entry 'compdb' does not exist in directory '/'
ORA-06512: at line 4
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/primary01/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.compdb.db' on 'primary01' failed
CRS-2679: Attempting to clean 'ora.compdb.db' on 'primary01'
CRS-2676: Start of 'ora.oc4j' on 'primary01' succeeded
CRS-5017: The resource action "ora.mgmtdb start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/_mgmtdb/spfile-MGMTDB.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/_mgmtdb/spfile-MGMTDB.ora
ORA-15056: additional error message
ORA-17503: ksfdopn:2 Failed to open file +DATA/_mgmtdb/spfile-mgmtdb.ora
ORA-15173: entry '_mgmtdb' does not exist in directory '/'
ORA-06512: at line 4
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/primary01/crs/trace/crsd_oraagent_grid.trc".
CRS-2674: Start of 'ora.mgmtdb' on 'primary01' failed
CRS-2679: Attempting to clean 'ora.mgmtdb' on 'primary01'
CRS-2681: Clean of 'ora.compdb.db' on 'primary01' succeeded
CRS-2681: Clean of 'ora.mgmtdb' on 'primary01' succeeded
CRS-2676: Start of 'ora.smalldb.db' on 'primary01' succeeded
===== Summary of resource auto-start failures follows =====
CRS-2807: Resource 'ora.compdb.db' failed to start automatically.
CRS-2807: Resource 'ora.mgmtdb' failed to start automatically.
CRS-6016: Resource auto-start has completed for server primary01
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.

Now we can check where ASM’s parameter file is.

SQL> show parameter spfile;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +DATA/primary-cluster/ASMPARAM
                                                 ETERFILE/registry.253.10120620
                                                 47

As you can see in the above CRS starting log, most of resources are back online, except MGMTDB and database related ones.

6. Recreate MGMTDB

You can also call MGMTDB as Grid Infrastructure Management Repository (GIMR), they are the same thing.

Before recreating MGMTDB, we have to stop and disable Cluster Health Monitor (CRF) service before recreating MGMTDB.

[root@primary01 ~]# crsctl stop res ora.crf -init
CRS-2673: Attempting to stop 'ora.crf' on 'primary01'
CRS-2677: Stop of 'ora.crf' on 'primary01' succeeded
[root@primary01 ~]# crsctl modify res ora.crf -attr ENABLED=0 -init

Remove stale configuration of MGMTDB if any.

[grid@primary01 ~]$ srvctl remove mgmtdb
Remove the database _mgmtdb? (y/[n]) y

Create a MGMTDB for ASM administration by DBCA.

[grid@primary01 ~]$ dbca -silent -createDatabase -sid -MGMTDB -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -gdbName _mgmtdb -storageType ASM -diskGroupName +DATA -datafileJarLocation $ORACLE_HOME/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck
Registering database with Oracle Grid Infrastructure
5% complete
Copying database files
7% complete
9% complete
16% complete
23% complete
30% complete
37% complete
41% complete
Creating and starting Oracle instance
43% complete
48% complete
49% complete
50% complete
55% complete
60% complete
61% complete
64% complete
Completing Database Creation
68% complete
79% complete
89% complete
100% complete
Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb0.log" for further details.

Check any new folder in disk group DATA.

[grid@primary01 ~]$ asmcmd ls -l +DATA
Type  Redund  Striped  Time             Sys  Name
                                        Y    _MGMTDB/
                                        Y    primary-cluster/
[grid@primary01 ~]$ asmcmd ls -l +DATA/_MGMTDB/
Type  Redund  Striped  Time             Sys  Name
                                        Y    8C4B00F2B5AF559BE0530B2AA8C0FE14/
                                        Y    CONTROLFILE/
                                        Y    DATAFILE/
                                        Y    FD9AC0F7C36E4438E043B6A9E80A24D5/
                                        Y    ONLINELOG/
                                        Y    PARAMETERFILE/
                                        Y    TEMPFILE/

Now we have _MGMTDB/ back.

Since GIMR is a multitenant database with a pluggable database (PDB) for the GIMR of each cluster, we need to create it, too.

[grid@primary01 ~]$ dbca -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName primarycluster -createPDBFrom RMANBACKUP -PDBBackUpfile $ORACLE_HOME/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile $ORACLE_HOME/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true -internalSkipGIHomeCheck
Creating Pluggable Database
4% complete
12% complete
21% complete
38% complete
55% complete
85% complete
Completing Pluggable Database Creation
100% complete
Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/primarycluster/_mgmtdb.log" for further details.

MGMTDB is back, we can enable and start CRF now.

[root@primary01 ~]# crsctl modify res ora.crf -attr ENABLED=1 -init
[root@primary01 ~]# crsctl start res ora.crf -init
CRS-2672: Attempting to start 'ora.crf' on 'primary01'
CRS-2676: Start of 'ora.crf' on 'primary01' succeeded
[root@primary01 ~]# crsctl status res ora.mgmtdb
NAME=ora.mgmtdb
TYPE=ora.mgmtdb.type
TARGET=ONLINE
STATE=ONLINE on primary01

7. Create ASM Password File

Switch to grid and use orapwd to create password file for ASM.

[grid@primary01 ~]$ orapwd file='+DATA' asm=y password=welcome1 force=y
[grid@primary01 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +DATA/ASM/PASSWORD/pwdasm.275.1012063795
ASM listener: LISTENER
[grid@primary01 ~]$ asmcmd ls -l +DATA/ASM/PASSWORD/
Type      Redund  Striped  Time             Sys  Name
PASSWORD  UNPROT  COARSE   JUN 26 16:00:00  Y    pwdasm.275.1012063795

Next, we should restore the entire 12c RAC database back.

Leave a Reply

Your email address will not be published. Required fields are marked *