There're 2 procedures of restoring OCR and voting disk in this post. They are:
A. For 12c Clusterware
Suppose we have lost all disk groups as well as OCR data, voting disk and management database (MGMTDB) in a disaster, we have to rebuild 12c clusterware nearly from the ground up for later 12c RAC database restoration. The only thing that you must have is a latest OCR backup.
In this procedure, I will show you how to restore 12c clusterware with the following steps:
- Recreate Disk Groups
- Restore OCR Data
- Restore Voting Disk
- Recreate ASM SPFILE
- Restart CRS
- Recreate MGMTDB
- Create ASM Password File
1. Recreate Disk Groups
Login as root.
[root@primary01 ~]# . /home/grid/.bash_profile
Stop CRS.
[root@primary01 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'primary01'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'primary01'
CRS-2673: Attempting to stop 'ora.gipcd' on 'primary01'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'primary01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'primary01'
CRS-2673: Attempting to stop 'ora.evmd' on 'primary01'
CRS-2677: Stop of 'ora.cssdmonitor' on 'primary01' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'primary01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'primary01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Start CRS exclusively with nocrs.
[root@primary01 ~]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'primary01'
CRS-2672: Attempting to start 'ora.mdnsd' on 'primary01'
CRS-2676: Start of 'ora.evmd' on 'primary01' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'primary01'
CRS-2676: Start of 'ora.gpnpd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'primary01'
CRS-2672: Attempting to start 'ora.gipcd' on 'primary01'
CRS-2676: Start of 'ora.cssdmonitor' on 'primary01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'primary01'
CRS-2672: Attempting to start 'ora.diskmon' on 'primary01'
CRS-2676: Start of 'ora.diskmon' on 'primary01' succeeded
CRS-2676: Start of 'ora.cssd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'primary01'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'primary01'
CRS-2672: Attempting to start 'ora.ctssd' on 'primary01'
CRS-2676: Start of 'ora.ctssd' on 'primary01' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'primary01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'primary01'
CRS-2676: Start of 'ora.asm' on 'primary01' succeeded
Make sure that shared disks are existing by user grid.
[grid@primary01 ~]$ ll /dev/oracleasm/disks/
total 0
brw-rw---- 1 grid asmadmin 8, 33 Jun 26 11:27 ASM01
brw-rw---- 1 grid asmadmin 8, 17 Jun 26 11:27 ASM02
Make sure the disk group name. As usual, it's called DATA or CRS. If you have no clue about it, you can check the following file.
[grid@primary01 ~]$ cat /etc/oracle/ocr.loc
#Device/file +DATA getting replaced by device +DATA/primary-cluster/OCRFILE/registry.255.887454747
ocrconfig_loc=+DATA/primary-cluster/OCRFILE/registry.255.887454747
local_only=false
Create ASM diskgroups DATA as it was.
[grid@primary01 ~]$ sqlplus / as sysasm
...
SQL> alter system set asm_diskstring='/dev/oracleasm/disks/*' scope=memory;
System altered.
SQL> create diskgroup DATA external redundancy disk '/dev/oracleasm/disks/ASM01' attribute 'COMPATIBLE.ASM' = '12.1.0';
Diskgroup created.
Disk group DATA has been created. Please note that, ATTRIBUTE clause in the above statement is pretty important, don't miss it.
2. Restore OCR Data
Back to root. We start to restore OCR and then check the current setting of it.
[root@primary01 ~]# ocrconfig -restore /u01/app/12.1.0/grid/cdata/primary-cluster/backup_20190612_191205.ocr
[root@primary01 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 1680
Available space (kbytes) : 407888
ID : 1125371174
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
Now we check OCR location file again.
[root@primary01 ~]# cat /etc/oracle/ocr.loc
#Device/file +DATA/primary-cluster/OCRFILE/registry.255.887454747 getting replaced by device +DATA/primary-cluster/OCRFILE/registry.255.1011959477
ocrconfig_loc=+DATA/primary-cluster/OCRFILE/registry.255.1011959477
local_only=false
The destination of OCR file has been replaced.
3. Restore Voting Disk
Since we have restored OCR, we restored voting file as well. You can just replace with new voting file on disk group DATA.
[root@primary01 ~]# crsctl replace votedisk +DATA
Successful addition of voting disk a6f4735a13a64f60bf31092700473fad.
Successfully replaced voting disk group with +DATA.
CRS-4266: Voting file(s) successfully replaced
[root@primary01 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE a6f4735a13a64f60bf31092700473fad (/dev/oracleasm/disks/ASM01) [DATA]
Located 1 voting disk(s).
4. Recreate ASM SPFILE
We use current ASM instance to recreate an ASM SPFILE by grid.
SQL> create spfile='+DATA' from memory;
File created.
5. Restart CRS
Now we can restart CRS to verify the result.
[root@primary01 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'primary01'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'primary01'
CRS-2673: Attempting to stop 'ora.ctssd' on 'primary01'
CRS-2673: Attempting to stop 'ora.asm' on 'primary01'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'primary01'
CRS-2673: Attempting to stop 'ora.evmd' on 'primary01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'primary01'
CRS-2677: Stop of 'ora.drivers.acfs' on 'primary01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'primary01' succeeded
CRS-2677: Stop of 'ora.asm' on 'primary01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'primary01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'primary01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'primary01'
CRS-2677: Stop of 'ora.cssd' on 'primary01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'primary01'
CRS-2677: Stop of 'ora.gipcd' on 'primary01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'primary01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@primary01 ~]# crsctl start crs -wait
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'primary01'
CRS-2672: Attempting to start 'ora.evmd' on 'primary01'
CRS-2676: Start of 'ora.mdnsd' on 'primary01' succeeded
CRS-2676: Start of 'ora.evmd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'primary01'
CRS-2676: Start of 'ora.gpnpd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'primary01'
CRS-2676: Start of 'ora.gipcd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'primary01'
CRS-2676: Start of 'ora.cssdmonitor' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'primary01'
CRS-2672: Attempting to start 'ora.diskmon' on 'primary01'
CRS-2676: Start of 'ora.diskmon' on 'primary01' succeeded
CRS-2676: Start of 'ora.cssd' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'primary01'
CRS-2672: Attempting to start 'ora.ctssd' on 'primary01'
CRS-2676: Start of 'ora.ctssd' on 'primary01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'primary01'
CRS-2676: Start of 'ora.asm' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'primary01'
CRS-2676: Start of 'ora.storage' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'primary01'
CRS-2676: Start of 'ora.crf' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'primary01'
CRS-2676: Start of 'ora.crsd' on 'primary01' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: primary01
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'primary01'
CRS-2672: Attempting to start 'ora.net2.network' on 'primary01'
CRS-2672: Attempting to start 'ora.net1.network' on 'primary01'
CRS-2672: Attempting to start 'ora.oc4j' on 'primary01'
CRS-2676: Start of 'ora.net2.network' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.primary01_2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.primary02_2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan1_net2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan2_net2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan3_net2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan4_net2.vip' on 'primary01'
CRS-2676: Start of 'ora.net1.network' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'primary01'
CRS-2672: Attempting to start 'ora.ons' on 'primary01'
CRS-2672: Attempting to start 'ora.primary01.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.primary02.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'primary01'
CRS-2672: Attempting to start 'ora.scan4.vip' on 'primary01'
CRS-2676: Start of 'ora.cvu' on 'primary01' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'primary01' succeeded
CRS-2676: Start of 'ora.ons' on 'primary01' succeeded
CRS-2676: Start of 'ora.primary02_2.vip' on 'primary01' succeeded
CRS-2676: Start of 'ora.scan2_net2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER2_SCAN2_NET2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan3_net2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER2_SCAN3_NET2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.primary01.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan1_net2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER2_SCAN1_NET2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.primary02.vip' on 'primary01' succeeded
CRS-2676: Start of 'ora.scan4_net2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER2_SCAN4_NET2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.primary01_2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan1.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan2.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan3.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'primary01'
CRS-2676: Start of 'ora.scan4.vip' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN4.lsnr' on 'primary01'
CRS-2676: Start of 'ora.LISTENER2_SCAN3_NET2.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.smalldb.db' on 'primary01'
CRS-2676: Start of 'ora.LISTENER2_SCAN4_NET2.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER2.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER2_SCAN1_NET2.lsnr' on 'primary01' succeeded
CRS-2672: Attempting to start 'ora.compdb.db' on 'primary01'
CRS-2672: Attempting to start 'ora.mgmtdb' on 'primary01'
CRS-2676: Start of 'ora.LISTENER2_SCAN2_NET2.lsnr' on 'primary01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN4.lsnr' on 'primary01' succeeded
CRS-5017: The resource action "ora.compdb.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/compdb/spfileprimdb.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/compdb/spfileprimdb.ora
ORA-15056: additional error message
ORA-17503: ksfdopn:2 Failed to open file +DATA/compdb/spfileprimdb.ora
ORA-15173: entry 'compdb' does not exist in directory '/'
ORA-06512: at line 4
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/primary01/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.compdb.db' on 'primary01' failed
CRS-2679: Attempting to clean 'ora.compdb.db' on 'primary01'
CRS-2676: Start of 'ora.oc4j' on 'primary01' succeeded
CRS-5017: The resource action "ora.mgmtdb start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/_mgmtdb/spfile-MGMTDB.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/_mgmtdb/spfile-MGMTDB.ora
ORA-15056: additional error message
ORA-17503: ksfdopn:2 Failed to open file +DATA/_mgmtdb/spfile-mgmtdb.ora
ORA-15173: entry '_mgmtdb' does not exist in directory '/'
ORA-06512: at line 4
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/primary01/crs/trace/crsd_oraagent_grid.trc".
CRS-2674: Start of 'ora.mgmtdb' on 'primary01' failed
CRS-2679: Attempting to clean 'ora.mgmtdb' on 'primary01'
CRS-2681: Clean of 'ora.compdb.db' on 'primary01' succeeded
CRS-2681: Clean of 'ora.mgmtdb' on 'primary01' succeeded
CRS-2676: Start of 'ora.smalldb.db' on 'primary01' succeeded
===== Summary of resource auto-start failures follows =====
CRS-2807: Resource 'ora.compdb.db' failed to start automatically.
CRS-2807: Resource 'ora.mgmtdb' failed to start automatically.
CRS-6016: Resource auto-start has completed for server primary01
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
Now we can check where ASM's parameter file is.
SQL> show parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +DATA/primary-cluster/ASMPARAM
ETERFILE/registry.253.10120620
47
As you can see in the above CRS starting log, most of resources are back online, except MGMTDB and database related ones.
6. Recreate MGMTDB
You can also call MGMTDB as Grid Infrastructure Management Repository (GIMR), they are the same thing.
Before recreating MGMTDB, we have to stop and disable Cluster Health Monitor (CRF) service before recreating MGMTDB.
[root@primary01 ~]# crsctl stop res ora.crf -init
CRS-2673: Attempting to stop 'ora.crf' on 'primary01'
CRS-2677: Stop of 'ora.crf' on 'primary01' succeeded
[root@primary01 ~]# crsctl modify res ora.crf -attr ENABLED=0 -init
Remove stale configuration of MGMTDB if any.
[grid@primary01 ~]$ srvctl remove mgmtdb
Remove the database _mgmtdb? (y/[n]) y
Create a MGMTDB for ASM administration by DBCA.
[grid@primary01 ~]$ dbca -silent -createDatabase -sid -MGMTDB -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -gdbName _mgmtdb -storageType ASM -diskGroupName +DATA -datafileJarLocation $ORACLE_HOME/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck
Registering database with Oracle Grid Infrastructure
5% complete
Copying database files
7% complete
9% complete
16% complete
23% complete
30% complete
37% complete
41% complete
Creating and starting Oracle instance
43% complete
48% complete
49% complete
50% complete
55% complete
60% complete
61% complete
64% complete
Completing Database Creation
68% complete
79% complete
89% complete
100% complete
Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb0.log" for further details.
Check any new folder in disk group DATA.
[grid@primary01 ~]$ asmcmd ls -l +DATA
Type Redund Striped Time Sys Name
Y _MGMTDB/
Y primary-cluster/
[grid@primary01 ~]$ asmcmd ls -l +DATA/_MGMTDB/
Type Redund Striped Time Sys Name
Y 8C4B00F2B5AF559BE0530B2AA8C0FE14/
Y CONTROLFILE/
Y DATAFILE/
Y FD9AC0F7C36E4438E043B6A9E80A24D5/
Y ONLINELOG/
Y PARAMETERFILE/
Y TEMPFILE/
Now we have _MGMTDB/ back.
Since GIMR is a multitenant database with a pluggable database (PDB) for the GIMR of each cluster, we need to create it, too.
[grid@primary01 ~]$ dbca -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName primarycluster -createPDBFrom RMANBACKUP -PDBBackUpfile $ORACLE_HOME/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile $ORACLE_HOME/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true -internalSkipGIHomeCheck
Creating Pluggable Database
4% complete
12% complete
21% complete
38% complete
55% complete
85% complete
Completing Pluggable Database Creation
100% complete
Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/primarycluster/_mgmtdb.log" for further details.
MGMTDB is back, we can enable and start CRF now.
[root@primary01 ~]# crsctl modify res ora.crf -attr ENABLED=1 -init
[root@primary01 ~]# crsctl start res ora.crf -init
CRS-2672: Attempting to start 'ora.crf' on 'primary01'
CRS-2676: Start of 'ora.crf' on 'primary01' succeeded
[root@primary01 ~]# crsctl status res ora.mgmtdb
NAME=ora.mgmtdb
TYPE=ora.mgmtdb.type
TARGET=ONLINE
STATE=ONLINE on primary01
7. Create ASM Password File
Switch to grid and use orapwd to create password file for ASM.
[grid@primary01 ~]$ orapwd file='+DATA' asm=y password=Welcome1 force=y
[grid@primary01 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +DATA/ASM/PASSWORD/pwdasm.275.1012063795
ASM listener: LISTENER
[grid@primary01 ~]$ asmcmd ls -l +DATA/ASM/PASSWORD/
Type Redund Striped Time Sys Name
PASSWORD UNPROT COARSE JUN 26 16:00:00 Y pwdasm.275.1012063795
Next, we should restore the entire 12c RAC database back.
B. For 10g Clusterware
Suppose we have lost the shared storage as well as OCR data, voting disk and disk groups in a disaster. Later, a new shared storage for 10g RAC database has been established.
It looks like the worst case, but we still have backups of OCR and voting disk on server, plus a full database backup. That is to say, we need to rebuild RAC environment almost from the ground, except installed clusterware and database software.
In this procedure, I will start from OCR recovery, and then voting disk recovery. Since the disk groups we created previously are all gone, we need to recreate them back.
1. Restore OCR Data
We can manually change the location of OCR from one to another on both nodes.
Node 1:
[root@primary01 ~]# . /home/oracle/.bash_profile
[root@primary01 ~]# crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM
[root@primary01 ~]# vi /etc/oracle/ocr.loc
#ocrconfig_loc=/dev/raw/raw1
ocrconfig_loc=/dev/raw/raw6
local_only=FALSE
Node 2:
[root@primary02 ~]# . /home/oracle/.bash_profile
[root@primary02 ~]# crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM
[root@primary02 ~]# vi /etc/oracle/ocr.loc
#ocrconfig_loc=/dev/raw/raw1
ocrconfig_loc=/dev/raw/raw6
local_only=FALSE
Then we restore OCR configuration by importing the backup dump on node 1.
[root@primary01 ~]# ocrconfig -import ocr_backup_20190612.dmp
Let's see current configuration of OCR.
[root@primary01 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 18874264
Used space (kbytes) : 3788
Available space (kbytes) : 18870476
ID : 559990555
Device/File Name : /dev/raw/raw6
Device/File integrity check succeeded
Device/File not configured
Cluster registry integrity check succeeded
2. Restore Voting Disk
You can add a new voting disk and delete the old one while CRS is not working.
[root@primary01 ~]# crsctl add css votedisk /dev/raw/raw7 -force
Now formatting voting disk: /dev/raw/raw7
successful addition of votedisk /dev/raw/raw7.
[root@primary01 ~]# crsctl delete css votedisk /dev/raw/raw2 -force
successful deletion of votedisk /dev/raw/raw2.
Then we restore the voting disk by dd command, just like we did in backing up voting disk.
[root@primary01 ~]# dd if=vod_backup_20190612.dmp of=/dev/raw/raw7 bs=1k count=500k
512000+0 records in
512000+0 records out
524288000 bytes (524 MB) copied, 103.76 seconds, 5.1 MB/s
Please note that, if you want all content of the dump file to be restored, you can omit count option.
[root@primary01 ~]# dd if=vod_backup_20190612.dmp of=/dev/raw/raw7 bs=1k
Let's see current configuration of voting disk on both nodes.
Node 1:
[root@primary01 ~]# crsctl query css votedisk
0. 0 /dev/raw/raw7
located 1 votedisk(s).
Node 2:
[root@primary02 ~]# crsctl query css votedisk
0. 0 /dev/raw/raw7
located 1 votedisk(s).
Restart both nodes to verify the results.
[root@primary01 ~]# init 6
[root@primary02 ~]# init 6
Check all CRS resources.
[oracle@primary01 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....B1.inst application ONLINE OFFLINE
ora....B2.inst application ONLINE OFFLINE
ora.PRIMDB.db application ONLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE primary01
ora....01.lsnr application ONLINE ONLINE primary01
ora....y01.gsd application ONLINE ONLINE primary01
ora....y01.ons application ONLINE ONLINE primary01
ora....y01.vip application ONLINE ONLINE primary01
ora....SM2.asm application ONLINE ONLINE primary02
ora....02.lsnr application ONLINE ONLINE primary02
ora....y02.gsd application ONLINE ONLINE primary02
ora....y02.ons application ONLINE ONLINE primary02
ora....y02.vip application ONLINE ONLINE primary02
So far, we at least get CRS back.
3. Recreate Disk Groups
We switch to ASM mode.
[oracle@primary01 ~]$ export ORACLE_SID=+ASM1
[oracle@primary01 ~]$ sqlplus / as sysdba
...
SQL> column diskgroup format a10;
SQL> column type format a10;
SQL> column disk_path format a40;
SQL> select g.name diskgroup, g.type, d.path disk_path from v$asm_diskgroup g, v$asm_disk d where g.group_number = d.group_number;
no rows selected
We have nothing here. Let's recreate two disk groups back.
SQL> create diskgroup ora_data external redundancy disk '/dev/raw/raw8';
Diskgroup created.
SQL> create diskgroup fra_data external redundancy disk '/dev/raw/raw9';
Diskgroup created.
Check disk group again.
SQL> select g.name diskgroup, g.type, d.path disk_path from v$asm_diskgroup g, v$asm_disk d where g.group_number = d.group_number;
DISKGROUP TYPE DISK_PATH
---------- ---------- ----------------------------------------
ORA_DATA EXTERN /dev/raw/raw8
FRA_DATA EXTERN /dev/raw/raw9
Disk groups are back. In fact, using DBCA in 10 g can be an alternative to create disk groups, which is somewhat equivalent to ASMCA in 11g.
[oracle@primary01 ~]$ srvctl config database
PRIMDB
[oracle@primary01 ~]$ srvctl config database -d primdb
primary01 PRIMDB1 /u01/app/oracle/product/10.2.0/db_1
primary02 PRIMDB2 /u01/app/oracle/product/10.2.0/db_1
[oracle@primary01 ~]$ srvctl status database -d primdb
Instance PRIMDB1 is not running on node primary01
Instance PRIMDB2 is not running on node primary02
Next, we should restore the entire 10g RAC database back from a backup set.