Finally, we have installed 1 NAS for shared storage, 1 DNS for SCAN round robin, and 2 Servers for RAC nodes. Now, we have to check everything is working fine. In this major step, we should verify two things:
- Grid Infrastructure
- Clustered Database
A. Grid Infrastructure
Check CRS service.
[root@primary01 ~]# su - grid
[grid@primary01 ~]$ olsnodes -n
primary01 1
primary02 2
[grid@primary01 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
HAS is working online, two nodes are supported.
Check the CRS stat in table format.
[grid@primary01 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE primary01
ora....ER.lsnr ora....er.type ONLINE ONLINE primary01
ora....N1.lsnr ora....er.type ONLINE ONLINE primary02
ora....N2.lsnr ora....er.type ONLINE ONLINE primary01
ora....N3.lsnr ora....er.type ONLINE ONLINE primary01
ora.MGMTLSNR ora....nr.type ONLINE ONLINE primary01
ora.asm ora.asm.type ONLINE ONLINE primary01
ora.compdb.db ora....se.type ONLINE ONLINE primary01
ora.cvu ora.cvu.type ONLINE ONLINE primary01
ora.mgmtdb ora....db.type ONLINE ONLINE primary01
ora....network ora....rk.type ONLINE ONLINE primary01
ora.oc4j ora.oc4j.type ONLINE ONLINE primary01
ora.ons ora.ons.type ONLINE ONLINE primary01
ora....SM1.asm application ONLINE ONLINE primary01
ora....01.lsnr application ONLINE ONLINE primary01
ora....y01.ons application ONLINE ONLINE primary01
ora....y01.vip ora....t1.type ONLINE ONLINE primary01
ora....SM2.asm application ONLINE ONLINE primary02
ora....02.lsnr application ONLINE ONLINE primary02
ora....y02.ons application ONLINE ONLINE primary02
ora....y02.vip ora....t1.type ONLINE ONLINE primary02
ora.scan1.vip ora....ip.type ONLINE ONLINE primary02
ora.scan2.vip ora....ip.type ONLINE ONLINE primary01
ora.scan3.vip ora....ip.type ONLINE ONLINE primary01
This time, we have resource ora.compdb.db online.
Show CRS stat verbosely.
[grid@primary01 ~]$ crs_stat -v
NAME=ora.DATA.dg
TYPE=ora.diskgroup.type
RESTART_ATTEMPTS=5
RESTART_COUNT=0
FAILURE_THRESHOLD=
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.LISTENER.lsnr
TYPE=ora.listener.type
RESTART_ATTEMPTS=5
RESTART_COUNT=0
FAILURE_THRESHOLD=
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.LISTENER_SCAN1.lsnr
TYPE=ora.scan_listener.type
RESTART_ATTEMPTS=5
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary02
NAME=ora.LISTENER_SCAN2.lsnr
TYPE=ora.scan_listener.type
RESTART_ATTEMPTS=5
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.LISTENER_SCAN3.lsnr
TYPE=ora.scan_listener.type
RESTART_ATTEMPTS=5
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.MGMTLSNR
TYPE=ora.mgmtlsnr.type
RESTART_ATTEMPTS=0
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.asm
TYPE=ora.asm.type
GEN_USR_ORA_INST_NAME@SERVERNAME(primary01)=+ASM1
GEN_USR_ORA_INST_NAME@SERVERNAME(primary02)=+ASM2
RESTART_ATTEMPTS=5
RESTART_COUNT=0
FAILURE_THRESHOLD=
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.compdb.db
TYPE=ora.database.type
GEN_USR_ORA_INST_NAME@SERVERNAME(primary01)=primdb1
GEN_USR_ORA_INST_NAME@SERVERNAME(primary02)=primdb2
RESTART_ATTEMPTS=2
RESTART_COUNT=0
USR_ORA_INST_NAME@SERVERNAME(primary01)=primdb1
USR_ORA_INST_NAME@SERVERNAME(primary02)=primdb2
FAILURE_THRESHOLD=1
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.cvu
TYPE=ora.cvu.type
RESTART_ATTEMPTS=5
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.mgmtdb
TYPE=ora.mgmtdb.type
GEN_USR_ORA_INST_NAME@SERVERNAME(primary01)=-MGMTDB
GEN_USR_ORA_INST_NAME@SERVERNAME(primary02)=-MGMTDB
RESTART_ATTEMPTS=2
RESTART_COUNT=0
FAILURE_THRESHOLD=1
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.net1.network
TYPE=ora.network.type
RESTART_ATTEMPTS=5
RESTART_COUNT=0
FAILURE_THRESHOLD=
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.oc4j
TYPE=ora.oc4j.type
RESTART_ATTEMPTS=1
RESTART_COUNT=0
FAILURE_THRESHOLD=2
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.ons
TYPE=ora.ons.type
RESTART_ATTEMPTS=3
RESTART_COUNT=0
FAILURE_THRESHOLD=
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.primary01.ASM1.asm
TYPE=application
GEN_USR_ORA_INST_NAME@SERVERNAME(primary01)=+ASM1
GEN_USR_ORA_INST_NAME@SERVERNAME(primary02)=+ASM2
RESTART_ATTEMPTS=5
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.primary01.LISTENER_PRIMARY01.lsnr
TYPE=application
RESTART_ATTEMPTS=5
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.primary01.ons
TYPE=application
RESTART_ATTEMPTS=3
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.primary01.vip
TYPE=ora.cluster_vip_net1.type
RESTART_ATTEMPTS=0
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.primary02.ASM2.asm
TYPE=application
GEN_USR_ORA_INST_NAME@SERVERNAME(primary01)=+ASM1
GEN_USR_ORA_INST_NAME@SERVERNAME(primary02)=+ASM2
RESTART_ATTEMPTS=5
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary02
NAME=ora.primary02.LISTENER_PRIMARY02.lsnr
TYPE=application
RESTART_ATTEMPTS=5
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary02
NAME=ora.primary02.ons
TYPE=application
RESTART_ATTEMPTS=3
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary02
NAME=ora.primary02.vip
TYPE=ora.cluster_vip_net1.type
RESTART_ATTEMPTS=0
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary02
NAME=ora.scan1.vip
TYPE=ora.scan_vip.type
RESTART_ATTEMPTS=0
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary02
NAME=ora.scan2.vip
TYPE=ora.scan_vip.type
RESTART_ATTEMPTS=0
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
NAME=ora.scan3.vip
TYPE=ora.scan_vip.type
RESTART_ATTEMPTS=0
RESTART_COUNT=0
FAILURE_THRESHOLD=0
FAILURE_COUNT=0
TARGET=ONLINE
STATE=ONLINE on primary01
Check the SCAN processes on node 1.
[grid@primary01 ~]$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
MGMTLSNR
LISTENER
LISTENER_SCAN2
LISTENER_SCAN3
Yes, this is what we expected, SCAN2 and SCAN3 is on node 1, and SCAN1 is on node 2 at this moment.
Check ASM status.
[grid@primary01 ~]$ srvctl status asm -a
ASM is running on primary01,primary02
ASM is enabled.
[grid@primary01 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 1544
Available space (kbytes) : 408024
ID : 1125371174
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Check the voting disk.
[grid@primary01 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE cf23f25e7b504fd8bf7a7c235d715d9c (/dev/oracleasm/disks/ASM01) [DATA]
Located 1 voting disk(s).
Check all resources under the database.
[grid@primary01 ~]$ crsctl status resource ora.compdb.db -p
NAME=ora.compdb.db
TYPE=ora.database.type
ACL=owner:oracle:rwx,pgrp:oinstall:r--,other::r--,group:dba:r-x,group:oper:r-x,user:grid:r-x
ACTIONS=startoption,group:"oinstall",user:"oracle",group:"dba",group:"oper",user:"grid"
ACTION_SCRIPT=
ACTION_START_OPTION=
ACTION_TIMEOUT=600
ACTIVE_PLACEMENT=0
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
AUTO_START=restore
CARDINALITY=2
CHECK_INTERVAL=1
CHECK_TIMEOUT=30
CLEAN_TIMEOUT=60
CLUSTER_DATABASE=true
DATABASE_TYPE=RAC
DB_UNIQUE_NAME=compdb
DEGREE=1
DELETE_TIMEOUT=60
DESCRIPTION=Oracle Database resource
ENABLED=1
FAILOVER_DELAY=0
FAILURE_INTERVAL=60
FAILURE_THRESHOLD=1
GEN_AUDIT_FILE_DEST=/u01/app/oracle/admin/compdb/adump
GEN_START_OPTIONS=
GEN_USR_ORA_INST_NAME=
GEN_USR_ORA_INST_NAME@SERVERNAME(primary01)=primdb1
GEN_USR_ORA_INST_NAME@SERVERNAME(primary02)=primdb2
HOSTING_MEMBERS=
INSTANCE_FAILOVER=0
INTERMEDIATE_TIMEOUT=0
LOAD=1
LOGGING_LEVEL=1
MANAGEMENT_POLICY=AUTOMATIC
MODIFY_TIMEOUT=60
NLS_LANG=
OFFLINE_CHECK_INTERVAL=0
ONLINE_RELOCATION_TIMEOUT=0
ORACLE_HOME=/u01/app/oracle/product/12.1.0/db_1
ORACLE_HOME_OLD=
PLACEMENT=restricted
PWFILE=+DATA/COMPDB/PASSWORD/pwdcompdb.276.887729933
RANK=0
RELOCATE_ACTION=0
RELOCATE_BY_DEPENDENCY=1
RESTART_ATTEMPTS=2
ROLE=PRIMARY
SCRIPT_TIMEOUT=60
SERVER_CATEGORY=
SERVER_POOLS=ora.compdb
SERVER_POOLS_PQ=
SPFILE=+DATA/COMPDB/PARAMETERFILE/spfile.289.887733743
START_CONCURRENCY=0
START_DEPENDENCIES=hard(ora.DATA.dg) pullup(ora.DATA.dg) weak(type:ora.listener.type, global:type:ora.scan_listener.type, uniform:ora.ons, global:ora.gns)
START_TIMEOUT=600
STOP_CONCURRENCY=0
STOP_DEPENDENCIES=hard(shutdown:ora.DATA.dg, intermediate:ora.asm)
STOP_TIMEOUT=600
TYPE_VERSION=3.3
UPTIME_THRESHOLD=1h
USER_WORKLOAD=yes
USE_STICKINESS=0
USR_ORA_DB_NAME=compdb
USR_ORA_DOMAIN=example.com
USR_ORA_ENV=
USR_ORA_FLAGS=
USR_ORA_INST_NAME=
USR_ORA_INST_NAME@SERVERNAME(primary01)=primdb1
USR_ORA_INST_NAME@SERVERNAME(primary02)=primdb2
USR_ORA_OPEN_MODE=open
USR_ORA_OPI=false
USR_ORA_STOP_MODE=immediate
B. Clustered Database
et's check the clustered database by oracle.
[root@primary01 ~]# su - oracle
[oracle@primary01 ~]$ srvctl status database -d compdb
Instance primdb1 is running on node primary01
Instance primdb2 is running on node primary02
[oracle@primary01 ~]$ srvctl status nodeapps
VIP primary01-vip is enabled
VIP primary01-vip is running on node: primary01
VIP primary02-vip is enabled
VIP primary02-vip is running on node: primary02
Network is enabled
Network is running on node: primary01
Network is running on node: primary02
ONS is enabled
ONS daemon is running on node: primary01
ONS daemon is running on node: primary02
Check the configuration of the database.
[oracle@primary01 ~]$ srvctl config database
compdb
[oracle@primary01 ~]$ srvctl config database -d compdb
Database unique name: compdb
Database name: compdb
Oracle home: /u01/app/oracle/product/12.1.0/db_1
Oracle user: oracle
Spfile: +DATA/COMPDB/PARAMETERFILE/spfile.289.887733743
Password file: +DATA/COMPDB/PASSWORD/pwdcompdb.276.887729933
Domain: example.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: primdb1,primdb2
Configured nodes: primary01,primary02
Database is administrator managed
Our mission is completed.