Srvctl Start Database ORA-00304
Saw some errors when I tried to startup a cluster database by srvctl.
[oracle@primary01 ~]$ srvctl start database -d compdb
PRCR-1079 : Failed to start resource ora.compdb.db
CRS-5017: The resource action "ora.compdb.db start" encountered the following error:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 671785331
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/primary02/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.compdb.db' on 'primary02' failed
CRS-2632: There are no more servers to try to place resource 'ora.compdb.db' on that would satisfy its placement policy
Eventually, only one RAC node was running, the other node failed to be started up. More specifically, if NODE 1's instance was online, then NODE 2's instance was offline, or vice versa.
Let's inspect the log for more details:
[oracle@primary02 ~]$ vi /u01/app/grid/diag/crs/primary02/crs/trace/crsd_oraagent_oracle.trc
...
2019-06-27 12:04:42.803845 :CLSDYNAM:123709184: [ora.compdb.db]{1:56587:1766} [start] ORA-00304: requested INSTANCE_NUMBER is busy
2019-06-27 12:04:42.803999 :CLSDYNAM:123709184: [ora.compdb.db]{1:56587:1766} [start] InstAgent::start: 1 errcode 304
2019-06-27 12:04:42.804059 :CLSDYNAM:123709184: [ora.compdb.db]{1:56587:1766} [start] InstAgent::start: ORA-304 instance busy m_instanceType 1 m_lastOCIError 304
2019-06-27 12:04:42.804113 :CLSDYNAM:123709184: [ora.compdb.db]{1:56587:1766} [start] InstAgent::start: ORA-304 instance busy dbType RAC
2019-06-27 12:04:42.804659 :CLSDYNAM:123709184: [ora.compdb.db]{1:56587:1766} [start] InstConnection::connectInt: server not attached
2019-06-27 12:04:42.820420 :CLSDYNAM:123709184: [ora.compdb.db]{1:56587:1766} [start] ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3640
Additional information: 671785331
Process ID: 0
Session ID: 0 Serial number: 0
Solution
Did you see ORA-00304? It told us that "requested INSTANCE_NUMBER is busy". As we know, INSTANCE_NUMBER is a database parameter used for RAC. Next, let's output a PFILE from SPFILE in order to inspect parameters of the cluster database.
SQL> create pfile='/tmp/pfile.ora' from spfile;
File created.
[oracle@primary01 ~]$ vi /tmp/pfile.ora
...
*.instance_number=1
...
*.thread=1
*.undo_tablespace='UNDOTBS1'
As we can see, those parameters with wild cards above should be used for single-instance databases, not for clustered databases. For RAC database, they should be prefixed with INSTANCE_NAME. For example:
PRIMDB1.instance_number=1
PRIMDB2.instance_number=2
...
PRIMDB1.thread=1
PRIMDB2.thread=2
PRIMDB1.undo_tablespace='UNDOTBS1'
PRIMDB2.undo_tablespace='UNDOTBS2'
So I created a new shared SPFILE correctly from the modified PFILE for the cluster database, then restart it again by srvctl. I solved the problem.
In summary, the key actions to this problem are:
- Inspect the log.
- Inspect the parameters.