The series of post will install a 19c (19.3) RAC database on two Enterprise Linux 8.1. There're 4 parts in this series.
- Preparing Two Database Servers
- Installing Grid Infrastructure
- Installing Database Software
- Creating a RAC Database
This post is the first part.
Assuming that you have already prepared a shared storage for RAC databases, so we start the series from preparing 2 database servers. For better preparing 2 nodes for database servers, we list the necessary items to check.
Network
Let's see the network design of our cluster database.
[root@primary01 ~]# vi /etc/hosts
# Public
192.168.10.11 primary01 primary01.example.com
192.168.10.12 primary02 primary02.example.com
# Private
192.168.24.11 primary01-priv primary01-priv.example.com
192.168.24.12 primary02-priv primary02-priv.example.com
# VIP
192.168.10.111 primary01-vip primary01-vip.example.com
192.168.10.112 primary02-vip primary02-vip.example.com
# SCAN
# 192.168.10.81 primary-cluster-scan primary-cluster-scan.example.com
# 192.168.10.82 primary-cluster-scan primary-cluster-scan.example.com
# 192.168.10.83 primary-cluster-scan primary-cluster-scan.example.com
# NAS
192.168.10.101 nas nas.example.com
# DNS
192.168.10.199 dns dns.example.comThe network design has been summarized as below:
- Two nodes for RAC servers: primary01 and primary02.
- One NAS for shared storage.
- One DNS for resolving hostname, especially for SCAN name.
- Two network cards, one is for public subnet, another is for private subnet.
- Public subnet (192.168.10.0/24): only for public connection. Later, all virtual IP like SCAN and VIP will dynamically bind with this NIC after running CRS.
- Private subnet (192.168.24.0/24): for both private and ASM connections. This NIC is mainly for node interconnection, not for outside access.
Please note that, you can only have local DNS server(s) listed in NIC configuration, otherwise, OUI of grid infrastructure will complain about it.
Now, I have two questions for you. You can make a guess.
Note that, public IP requires broadcast.
Note that, private IP requires multicast.
Same hosts configuration must also appear in primary02 according to our design.
Security
We stopped firewall and selinux on both nodes to make RAC installation more smoothly.
[root@primary01 ~]# systemctl stop firewalld
[root@primary01 ~]# systemctl disable firewalld
[root@primary01 ~]# vi /etc/selinux/config
...
SELINUX=disabledMaybe you'd like to open port 1521 on firewall for Oracle connections instead of disabling firewall.
Time
Time synchronization is pretty important for a RAC database, if time in one node deviates from the other node's, CRS may decide to abandon it. Please make sure NTP is working well on both nodes. Here we use the default NTP service chronyd to play the role.
[root@primary01 ~]# systemctl status chronyd
● chronyd.service - NTP client/server
Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-09-28 12:58:51 CST; 3min 22s ago
Docs: man:chronyd(8)
man:chrony.conf(5)
Process: 1038 ExecStartPost=/usr/libexec/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
Process: 1020 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 1028 (chronyd)
Tasks: 1 (limit: 23830)
Memory: 3.3M
CGroup: /system.slice/chronyd.service
└─1028 /usr/sbin/chronyd
Sep 28 12:58:51 primary01.example.com chronyd[1028]: chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +>
Sep 28 12:58:51 primary01.example.com chronyd[1028]: Frequency -112.344 +/- 1.716 ppm read from /var/lib/chrony/drift
Sep 28 12:58:51 primary01.example.com chronyd[1028]: Using right/UTC timezone to obtain leap second data
Sep 28 12:58:51 primary01.example.com systemd[1]: Started NTP client/server.
Sep 28 12:59:02 primary01.example.com chronyd[1028]: Selected source 111.235.248.121
Sep 28 12:59:02 primary01.example.com chronyd[1028]: System clock TAI offset set to 37 seconds
Sep 28 12:59:03 primary01.example.com chronyd[1028]: Source 218.161.118.185 replaced with 2406:2000:fc:437::2000
Sep 28 12:59:03 primary01.example.com chronyd[1028]: Received KoD RATE from 118.163.74.161
Sep 28 12:59:04 primary01.example.com chronyd[1028]: Selected source 49.213.184.242
Sep 28 13:00:07 primary01.example.com chronyd[1028]: Selected source 118.163.170.6Packages
Since the installed edition of Linux is minimal one. We need to install all required packages in order to install Oracle software. For Enterprise Linux 8.1, we can leverage oracle preinstall package which can ease our task to prepare those packages. For example, installing required packages and create required users and groups.
On both nodes.
[root@primary01 ~]# yum -y install oracle-database-preinstall-19c
Last metadata expiration check: 0:22:11 ago on Mon 28 Sep 2020 01:44:09 PM CST.
Dependencies resolved.
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
oracle-database-preinstall-19c
x86_64 1.0-1.el8 ol8_baseos_latest 24 k
Installing dependencies:
smartmontools x86_64 1:6.6-3.el8 ol8_baseos_latest 502 k
pkgconf-m4 noarch 1.4.2-1.el8 ol8_baseos_latest 17 k
net-tools x86_64 2.0-0.51.20160912git.el8 ol8_baseos_latest 323 k
glibc-devel x86_64 2.28-72.0.1.el8 ol8_baseos_latest 1.0 M
lm_sensors-libs x86_64 3.4.0-21.20180522git70f7e08.el8
ol8_baseos_latest 59 k
keyutils x86_64 1.5.10-6.el8 ol8_baseos_latest 63 k
bc x86_64 1.07.1-5.el8 ol8_baseos_latest 129 k
libnsl x86_64 2.28-72.0.1.el8 ol8_baseos_latest 94 k
unzip x86_64 6.0-43.el8 ol8_baseos_latest 196 k
rpcbind x86_64 1.2.5-7.el8 ol8_baseos_latest 70 k
binutils x86_64 2.30-73.0.1.el8 ol8_baseos_latest 5.8 M
psmisc x86_64 23.1-4.el8 ol8_baseos_latest 151 k
pkgconf x86_64 1.4.2-1.el8 ol8_baseos_latest 38 k
libverto-libevent x86_64 0.3.0-5.el8 ol8_baseos_latest 16 k
quota-nls noarch 1:4.04-10.el8 ol8_baseos_latest 94 k
glibc-headers x86_64 2.28-72.0.1.el8 ol8_baseos_latest 469 k
make x86_64 1:4.2.1-10.el8 ol8_baseos_latest 498 k
quota x86_64 1:4.04-10.el8 ol8_baseos_latest 214 k
gssproxy x86_64 0.8.0-15.el8 ol8_baseos_latest 119 k
libxcrypt-devel x86_64 4.1.1-4.el8 ol8_baseos_latest 25 k
kernel-headers x86_64 4.18.0-193.19.1.el8_2 ol8_baseos_latest 4.0 M
libpkgconf x86_64 1.4.2-1.el8 ol8_baseos_latest 35 k
libaio-devel x86_64 0.3.112-1.el8 ol8_baseos_latest 19 k
pkgconf-pkg-config x86_64 1.4.2-1.el8 ol8_baseos_latest 15 k
nfs-utils x86_64 1:2.3.3-31.el8 ol8_baseos_latest 468 k
ksh x86_64 20120801-254.0.1.el8 ol8_appstream 927 k
libX11-xcb x86_64 1.6.8-3.el8 ol8_appstream 14 k
libXxf86misc x86_64 1.0.4-1.el8 ol8_appstream 23 k
libXxf86dga x86_64 1.1.4-12.el8 ol8_appstream 26 k
libXi x86_64 1.7.9-7.el8 ol8_appstream 49 k
bind-libs-lite x86_64 32:9.11.13-6.el8_2.1 ol8_appstream 1.2 M
bind-utils x86_64 32:9.11.13-6.el8_2.1 ol8_appstream 443 k
libXrandr x86_64 1.5.1-7.el8 ol8_appstream 33 k
libXxf86vm x86_64 1.1.4-9.el8 ol8_appstream 19 k
libdmx x86_64 1.1.4-3.el8 ol8_appstream 22 k
xorg-x11-utils x86_64 7.5-28.el8 ol8_appstream 136 k
libXv x86_64 1.0.11-7.el8 ol8_appstream 20 k
libxcb x86_64 1.13.1-1.el8 ol8_appstream 231 k
bind-libs x86_64 32:9.11.13-6.el8_2.1 ol8_appstream 172 k
libstdc++-devel x86_64 8.3.1-4.5.0.1.el8 ol8_appstream 2.0 M
libXinerama x86_64 1.1.4-1.el8 ol8_appstream 15 k
libXmu x86_64 1.1.2-12.el8 ol8_appstream 74 k
xorg-x11-xauth x86_64 1:1.0.9-12.el8 ol8_appstream 39 k
bind-license noarch 32:9.11.13-6.el8_2.1 ol8_appstream 101 k
libSM x86_64 1.2.3-1.el8 ol8_appstream 47 k
libXcomposite x86_64 0.4.4-14.el8 ol8_appstream 28 k
libX11-common noarch 1.6.8-3.el8 ol8_appstream 158 k
libXtst x86_64 1.2.3-7.el8 ol8_appstream 22 k
libXau x86_64 1.0.8-13.el8 ol8_appstream 36 k
sysstat x86_64 11.7.3-2.el8 ol8_appstream 426 k
libXt x86_64 1.1.5-12.el8 ol8_appstream 185 k
libICE x86_64 1.0.9-15.el8 ol8_appstream 74 k
libXrender x86_64 0.9.10-7.el8 ol8_appstream 33 k
libX11 x86_64 1.6.8-3.el8 ol8_appstream 611 k
libXext x86_64 1.3.3-9.el8 ol8_appstream 45 k
python3-bind noarch 32:9.11.13-6.el8_2.1 ol8_appstream 148 k
Transaction Summary
================================================================================
Install 57 Packages
Total download size: 22 M
Installed size: 67 M
Downloading Packages:
(1/57): pkgconf-m4-1.4.2-1.el8.noarch.rpm 11 kB/s | 17 kB 00:01
(2/57): net-tools-2.0-0.51.20160912git.el8.x86_ 139 kB/s | 323 kB 00:02
(3/57): lm_sensors-libs-3.4.0-21.20180522git70f 286 kB/s | 59 kB 00:00
(4/57): smartmontools-6.6-3.el8.x86_64.rpm 198 kB/s | 502 kB 00:02
(5/57): keyutils-1.5.10-6.el8.x86_64.rpm 623 kB/s | 63 kB 00:00
(6/57): bc-1.07.1-5.el8.x86_64.rpm 102 kB/s | 129 kB 00:01
(7/57): libnsl-2.28-72.0.1.el8.x86_64.rpm 70 kB/s | 94 kB 00:01
(8/57): rpcbind-1.2.5-7.el8.x86_64.rpm 568 kB/s | 70 kB 00:00
(9/57): oracle-database-preinstall-19c-1.0-1.el 231 kB/s | 24 kB 00:00
(10/57): unzip-6.0-43.el8.x86_64.rpm 136 kB/s | 196 kB 00:01
(11/57): glibc-devel-2.28-72.0.1.el8.x86_64.rpm 219 kB/s | 1.0 MB 00:04
(12/57): psmisc-23.1-4.el8.x86_64.rpm 107 kB/s | 151 kB 00:01
(13/57): libverto-libevent-0.3.0-5.el8.x86_64.r 50 kB/s | 16 kB 00:00
(14/57): binutils-2.30-73.0.1.el8.x86_64.rpm 1.8 MB/s | 5.8 MB 00:03
(15/57): pkgconf-1.4.2-1.el8.x86_64.rpm 26 kB/s | 38 kB 00:01
(16/57): quota-nls-4.04-10.el8.noarch.rpm 75 kB/s | 94 kB 00:01
(17/57): glibc-headers-2.28-72.0.1.el8.x86_64.r 308 kB/s | 469 kB 00:01
(18/57): gssproxy-0.8.0-15.el8.x86_64.rpm 1.1 MB/s | 119 kB 00:00
(19/57): quota-4.04-10.el8.x86_64.rpm 169 kB/s | 214 kB 00:01
(20/57): make-4.2.1-10.el8.x86_64.rpm 187 kB/s | 498 kB 00:02
(21/57): libxcrypt-devel-4.1.1-4.el8.x86_64.rpm 18 kB/s | 25 kB 00:01
(22/57): libpkgconf-1.4.2-1.el8.x86_64.rpm 29 kB/s | 35 kB 00:01
(23/57): libaio-devel-0.3.112-1.el8.x86_64.rpm 15 kB/s | 19 kB 00:01
(24/57): nfs-utils-2.3.3-31.el8.x86_64.rpm 754 kB/s | 468 kB 00:00
(25/57): pkgconf-pkg-config-1.4.2-1.el8.x86_64. 11 kB/s | 15 kB 00:01
(26/57): libX11-xcb-1.6.8-3.el8.x86_64.rpm 54 kB/s | 14 kB 00:00
(27/57): ksh-20120801-254.0.1.el8.x86_64.rpm 564 kB/s | 927 kB 00:01
(28/57): kernel-headers-4.18.0-193.19.1.el8_2.x 909 kB/s | 4.0 MB 00:04
(29/57): libXi-1.7.9-7.el8.x86_64.rpm 349 kB/s | 49 kB 00:00
(30/57): libXxf86misc-1.0.4-1.el8.x86_64.rpm 20 kB/s | 23 kB 00:01
(31/57): bind-utils-9.11.13-6.el8_2.1.x86_64.rp 1.5 MB/s | 443 kB 00:00
(32/57): bind-libs-lite-9.11.13-6.el8_2.1.x86_6 1.8 MB/s | 1.2 MB 00:00
(33/57): libXxf86vm-1.1.4-9.el8.x86_64.rpm 220 kB/s | 19 kB 00:00
(34/57): libXrandr-1.5.1-7.el8.x86_64.rpm 186 kB/s | 33 kB 00:00
(35/57): libXxf86dga-1.1.4-12.el8.x86_64.rpm 22 kB/s | 26 kB 00:01
(36/57): libXv-1.0.11-7.el8.x86_64.rpm 238 kB/s | 20 kB 00:00
(37/57): libdmx-1.1.4-3.el8.x86_64.rpm 18 kB/s | 22 kB 00:01
(38/57): xorg-x11-utils-7.5-28.el8.x86_64.rpm 108 kB/s | 136 kB 00:01
(39/57): bind-libs-9.11.13-6.el8_2.1.x86_64.rpm 871 kB/s | 172 kB 00:00
(40/57): libxcb-1.13.1-1.el8.x86_64.rpm 188 kB/s | 231 kB 00:01
(41/57): libXinerama-1.1.4-1.el8.x86_64.rpm 114 kB/s | 15 kB 00:00
(42/57): libXmu-1.1.2-12.el8.x86_64.rpm 49 kB/s | 74 kB 00:01
(43/57): xorg-x11-xauth-1.0.9-12.el8.x86_64.rpm 25 kB/s | 39 kB 00:01
(44/57): bind-license-9.11.13-6.el8_2.1.noarch. 302 kB/s | 101 kB 00:00
(45/57): libXcomposite-0.4.4-14.el8.x86_64.rpm 189 kB/s | 28 kB 00:00
(46/57): libstdc++-devel-8.3.1-4.5.0.1.el8.x86_ 715 kB/s | 2.0 MB 00:02
(47/57): libXtst-1.2.3-7.el8.x86_64.rpm 332 kB/s | 22 kB 00:00
(48/57): libSM-1.2.3-1.el8.x86_64.rpm 39 kB/s | 47 kB 00:01
(49/57): sysstat-11.7.3-2.el8.x86_64.rpm 967 kB/s | 426 kB 00:00
(50/57): libX11-common-1.6.8-3.el8.noarch.rpm 119 kB/s | 158 kB 00:01
(51/57): libXau-1.0.8-13.el8.x86_64.rpm 30 kB/s | 36 kB 00:01
(52/57): libICE-1.0.9-15.el8.x86_64.rpm 59 kB/s | 74 kB 00:01
(53/57): libXt-1.1.5-12.el8.x86_64.rpm 123 kB/s | 185 kB 00:01
(54/57): libXrender-0.9.10-7.el8.x86_64.rpm 28 kB/s | 33 kB 00:01
(55/57): python3-bind-9.11.13-6.el8_2.1.noarch. 971 kB/s | 148 kB 00:00
(56/57): libXext-1.3.3-9.el8.x86_64.rpm 36 kB/s | 45 kB 00:01
(57/57): libX11-1.6.8-3.el8.x86_64.rpm 416 kB/s | 611 kB 00:01
--------------------------------------------------------------------------------
Total 978 kB/s | 22 MB 00:22
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : bind-license-32:9.11.13-6.el8_2.1.noarch 1/57
Installing : bind-libs-lite-32:9.11.13-6.el8_2.1.x86_64 2/57
Installing : libICE-1.0.9-15.el8.x86_64 3/57
Installing : libXau-1.0.8-13.el8.x86_64 4/57
Installing : libxcb-1.13.1-1.el8.x86_64 5/57
Installing : libSM-1.2.3-1.el8.x86_64 6/57
Installing : bind-libs-32:9.11.13-6.el8_2.1.x86_64 7/57
Installing : python3-bind-32:9.11.13-6.el8_2.1.noarch 8/57
Installing : bind-utils-32:9.11.13-6.el8_2.1.x86_64 9/57
Installing : libX11-common-1.6.8-3.el8.noarch 10/57
Installing : libX11-1.6.8-3.el8.x86_64 11/57
Installing : libXext-1.3.3-9.el8.x86_64 12/57
Installing : libXi-1.7.9-7.el8.x86_64 13/57
Installing : libXrender-0.9.10-7.el8.x86_64 14/57
Installing : libXrandr-1.5.1-7.el8.x86_64 15/57
Installing : libXtst-1.2.3-7.el8.x86_64 16/57
Installing : libXxf86misc-1.0.4-1.el8.x86_64 17/57
Installing : libXxf86dga-1.1.4-12.el8.x86_64 18/57
Installing : libXxf86vm-1.1.4-9.el8.x86_64 19/57
Installing : libdmx-1.1.4-3.el8.x86_64 20/57
Installing : libXv-1.0.11-7.el8.x86_64 21/57
Installing : libXinerama-1.1.4-1.el8.x86_64 22/57
Installing : libXcomposite-0.4.4-14.el8.x86_64 23/57
Installing : libXt-1.1.5-12.el8.x86_64 24/57
Installing : libXmu-1.1.2-12.el8.x86_64 25/57
Installing : xorg-x11-xauth-1:1.0.9-12.el8.x86_64 26/57
Installing : libstdc++-devel-8.3.1-4.5.0.1.el8.x86_64 27/57
Installing : libX11-xcb-1.6.8-3.el8.x86_64 28/57
Installing : xorg-x11-utils-7.5-28.el8.x86_64 29/57
Installing : ksh-20120801-254.0.1.el8.x86_64 30/57
Running scriptlet: ksh-20120801-254.0.1.el8.x86_64 30/57
Installing : libaio-devel-0.3.112-1.el8.x86_64 31/57
Installing : libpkgconf-1.4.2-1.el8.x86_64 32/57
Installing : pkgconf-1.4.2-1.el8.x86_64 33/57
Installing : kernel-headers-4.18.0-193.19.1.el8_2.x86_64 34/57
Running scriptlet: glibc-headers-2.28-72.0.1.el8.x86_64 35/57
Installing : glibc-headers-2.28-72.0.1.el8.x86_64 35/57
Installing : make-1:4.2.1-10.el8.x86_64 36/57
Running scriptlet: make-1:4.2.1-10.el8.x86_64 36/57
Installing : quota-nls-1:4.04-10.el8.noarch 37/57
Installing : quota-1:4.04-10.el8.x86_64 38/57
Installing : libverto-libevent-0.3.0-5.el8.x86_64 39/57
Installing : gssproxy-0.8.0-15.el8.x86_64 40/57
Running scriptlet: gssproxy-0.8.0-15.el8.x86_64 40/57
Installing : psmisc-23.1-4.el8.x86_64 41/57
Installing : binutils-2.30-73.0.1.el8.x86_64 42/57
Running scriptlet: binutils-2.30-73.0.1.el8.x86_64 42/57
Running scriptlet: rpcbind-1.2.5-7.el8.x86_64 43/57
Installing : rpcbind-1.2.5-7.el8.x86_64 43/57
Running scriptlet: rpcbind-1.2.5-7.el8.x86_64 43/57
Installing : unzip-6.0-43.el8.x86_64 44/57
Installing : libnsl-2.28-72.0.1.el8.x86_64 45/57
Installing : bc-1.07.1-5.el8.x86_64 46/57
Running scriptlet: bc-1.07.1-5.el8.x86_64 46/57
Installing : keyutils-1.5.10-6.el8.x86_64 47/57
Running scriptlet: nfs-utils-1:2.3.3-31.el8.x86_64 48/57
Installing : nfs-utils-1:2.3.3-31.el8.x86_64 48/57
Running scriptlet: nfs-utils-1:2.3.3-31.el8.x86_64 48/57
Installing : lm_sensors-libs-3.4.0-21.20180522git70f7e08.el8.x8 49/57
Running scriptlet: lm_sensors-libs-3.4.0-21.20180522git70f7e08.el8.x8 49/57
Installing : sysstat-11.7.3-2.el8.x86_64 50/57
Running scriptlet: sysstat-11.7.3-2.el8.x86_64 50/57
Installing : net-tools-2.0-0.51.20160912git.el8.x86_64 51/57
Running scriptlet: net-tools-2.0-0.51.20160912git.el8.x86_64 51/57
Installing : pkgconf-m4-1.4.2-1.el8.noarch 52/57
Installing : pkgconf-pkg-config-1.4.2-1.el8.x86_64 53/57
Installing : glibc-devel-2.28-72.0.1.el8.x86_64 54/57
Running scriptlet: glibc-devel-2.28-72.0.1.el8.x86_64 54/57
Installing : libxcrypt-devel-4.1.1-4.el8.x86_64 55/57
Running scriptlet: smartmontools-1:6.6-3.el8.x86_64 56/57
Installing : smartmontools-1:6.6-3.el8.x86_64 56/57
Running scriptlet: smartmontools-1:6.6-3.el8.x86_64 56/57
Running scriptlet: oracle-database-preinstall-19c-1.0-1.el8.x86_64 57/57
Installing : oracle-database-preinstall-19c-1.0-1.el8.x86_64 57/57
Running scriptlet: oracle-database-preinstall-19c-1.0-1.el8.x86_64 57/57
Verifying : smartmontools-1:6.6-3.el8.x86_64 1/57
Verifying : pkgconf-m4-1.4.2-1.el8.noarch 2/57
Verifying : net-tools-2.0-0.51.20160912git.el8.x86_64 3/57
Verifying : glibc-devel-2.28-72.0.1.el8.x86_64 4/57
Verifying : lm_sensors-libs-3.4.0-21.20180522git70f7e08.el8.x8 5/57
Verifying : keyutils-1.5.10-6.el8.x86_64 6/57
Verifying : bc-1.07.1-5.el8.x86_64 7/57
Verifying : libnsl-2.28-72.0.1.el8.x86_64 8/57
Verifying : unzip-6.0-43.el8.x86_64 9/57
Verifying : rpcbind-1.2.5-7.el8.x86_64 10/57
Verifying : oracle-database-preinstall-19c-1.0-1.el8.x86_64 11/57
Verifying : binutils-2.30-73.0.1.el8.x86_64 12/57
Verifying : psmisc-23.1-4.el8.x86_64 13/57
Verifying : pkgconf-1.4.2-1.el8.x86_64 14/57
Verifying : libverto-libevent-0.3.0-5.el8.x86_64 15/57
Verifying : quota-nls-1:4.04-10.el8.noarch 16/57
Verifying : glibc-headers-2.28-72.0.1.el8.x86_64 17/57
Verifying : make-1:4.2.1-10.el8.x86_64 18/57
Verifying : quota-1:4.04-10.el8.x86_64 19/57
Verifying : gssproxy-0.8.0-15.el8.x86_64 20/57
Verifying : libxcrypt-devel-4.1.1-4.el8.x86_64 21/57
Verifying : kernel-headers-4.18.0-193.19.1.el8_2.x86_64 22/57
Verifying : libpkgconf-1.4.2-1.el8.x86_64 23/57
Verifying : libaio-devel-0.3.112-1.el8.x86_64 24/57
Verifying : pkgconf-pkg-config-1.4.2-1.el8.x86_64 25/57
Verifying : nfs-utils-1:2.3.3-31.el8.x86_64 26/57
Verifying : ksh-20120801-254.0.1.el8.x86_64 27/57
Verifying : libX11-xcb-1.6.8-3.el8.x86_64 28/57
Verifying : libXxf86misc-1.0.4-1.el8.x86_64 29/57
Verifying : libXxf86dga-1.1.4-12.el8.x86_64 30/57
Verifying : libXi-1.7.9-7.el8.x86_64 31/57
Verifying : bind-libs-lite-32:9.11.13-6.el8_2.1.x86_64 32/57
Verifying : bind-utils-32:9.11.13-6.el8_2.1.x86_64 33/57
Verifying : libXrandr-1.5.1-7.el8.x86_64 34/57
Verifying : libXxf86vm-1.1.4-9.el8.x86_64 35/57
Verifying : libdmx-1.1.4-3.el8.x86_64 36/57
Verifying : xorg-x11-utils-7.5-28.el8.x86_64 37/57
Verifying : libXv-1.0.11-7.el8.x86_64 38/57
Verifying : libxcb-1.13.1-1.el8.x86_64 39/57
Verifying : bind-libs-32:9.11.13-6.el8_2.1.x86_64 40/57
Verifying : libstdc++-devel-8.3.1-4.5.0.1.el8.x86_64 41/57
Verifying : libXinerama-1.1.4-1.el8.x86_64 42/57
Verifying : libXmu-1.1.2-12.el8.x86_64 43/57
Verifying : xorg-x11-xauth-1:1.0.9-12.el8.x86_64 44/57
Verifying : bind-license-32:9.11.13-6.el8_2.1.noarch 45/57
Verifying : libSM-1.2.3-1.el8.x86_64 46/57
Verifying : libXcomposite-0.4.4-14.el8.x86_64 47/57
Verifying : libX11-common-1.6.8-3.el8.noarch 48/57
Verifying : libXtst-1.2.3-7.el8.x86_64 49/57
Verifying : libXau-1.0.8-13.el8.x86_64 50/57
Verifying : sysstat-11.7.3-2.el8.x86_64 51/57
Verifying : libXt-1.1.5-12.el8.x86_64 52/57
Verifying : libICE-1.0.9-15.el8.x86_64 53/57
Verifying : libXrender-0.9.10-7.el8.x86_64 54/57
Verifying : libX11-1.6.8-3.el8.x86_64 55/57
Verifying : libXext-1.3.3-9.el8.x86_64 56/57
Verifying : python3-bind-32:9.11.13-6.el8_2.1.noarch 57/57
Installed:
oracle-database-preinstall-19c-1.0-1.el8.x86_64
smartmontools-1:6.6-3.el8.x86_64
pkgconf-m4-1.4.2-1.el8.noarch
net-tools-2.0-0.51.20160912git.el8.x86_64
glibc-devel-2.28-72.0.1.el8.x86_64
lm_sensors-libs-3.4.0-21.20180522git70f7e08.el8.x86_64
keyutils-1.5.10-6.el8.x86_64
bc-1.07.1-5.el8.x86_64
libnsl-2.28-72.0.1.el8.x86_64
unzip-6.0-43.el8.x86_64
rpcbind-1.2.5-7.el8.x86_64
binutils-2.30-73.0.1.el8.x86_64
psmisc-23.1-4.el8.x86_64
pkgconf-1.4.2-1.el8.x86_64
libverto-libevent-0.3.0-5.el8.x86_64
quota-nls-1:4.04-10.el8.noarch
glibc-headers-2.28-72.0.1.el8.x86_64
make-1:4.2.1-10.el8.x86_64
quota-1:4.04-10.el8.x86_64
gssproxy-0.8.0-15.el8.x86_64
libxcrypt-devel-4.1.1-4.el8.x86_64
kernel-headers-4.18.0-193.19.1.el8_2.x86_64
libpkgconf-1.4.2-1.el8.x86_64
libaio-devel-0.3.112-1.el8.x86_64
pkgconf-pkg-config-1.4.2-1.el8.x86_64
nfs-utils-1:2.3.3-31.el8.x86_64
ksh-20120801-254.0.1.el8.x86_64
libX11-xcb-1.6.8-3.el8.x86_64
libXxf86misc-1.0.4-1.el8.x86_64
libXxf86dga-1.1.4-12.el8.x86_64
libXi-1.7.9-7.el8.x86_64
bind-libs-lite-32:9.11.13-6.el8_2.1.x86_64
bind-utils-32:9.11.13-6.el8_2.1.x86_64
libXrandr-1.5.1-7.el8.x86_64
libXxf86vm-1.1.4-9.el8.x86_64
libdmx-1.1.4-3.el8.x86_64
xorg-x11-utils-7.5-28.el8.x86_64
libXv-1.0.11-7.el8.x86_64
libxcb-1.13.1-1.el8.x86_64
bind-libs-32:9.11.13-6.el8_2.1.x86_64
libstdc++-devel-8.3.1-4.5.0.1.el8.x86_64
libXinerama-1.1.4-1.el8.x86_64
libXmu-1.1.2-12.el8.x86_64
xorg-x11-xauth-1:1.0.9-12.el8.x86_64
bind-license-32:9.11.13-6.el8_2.1.noarch
libSM-1.2.3-1.el8.x86_64
libXcomposite-0.4.4-14.el8.x86_64
libX11-common-1.6.8-3.el8.noarch
libXtst-1.2.3-7.el8.x86_64
libXau-1.0.8-13.el8.x86_64
sysstat-11.7.3-2.el8.x86_64
libXt-1.1.5-12.el8.x86_64
libICE-1.0.9-15.el8.x86_64
libXrender-0.9.10-7.el8.x86_64
libX11-1.6.8-3.el8.x86_64
libXext-1.3.3-9.el8.x86_64
python3-bind-32:9.11.13-6.el8_2.1.noarch
Complete!Users, Groups and Directories
Although Oracle pre-installtion package installed some necessary users or groups for us, we need to create them again to make sure everything is ready.
Create Necessary Users and Groups
On both nodes.
[root@primary01 ~]# groupadd -g 54321 oinstall
groupadd: group 'oinstall' already exists
[root@primary01 ~]# groupadd -g 54322 dba
groupadd: group 'dba' already exists
[root@primary01 ~]# groupadd -g 54323 oper
groupadd: group 'oper' already exists
[root@primary01 ~]# groupadd -g 54324 backupdba
groupadd: group 'backupdba' already exists
[root@primary01 ~]# groupadd -g 54325 dgdba
groupadd: group 'dgdba' already exists
[root@primary01 ~]# groupadd -g 54326 kmdba
groupadd: group 'kmdba' already exists
[root@primary01 ~]# groupadd -g 54327 asmdba
[root@primary01 ~]# groupadd -g 54328 asmoper
[root@primary01 ~]# groupadd -g 54329 asmadmin
[root@primary01 ~]# groupadd -g 54330 racdba
groupadd: group 'racdba' already exists
[root@primary01 ~]# useradd -u 54322 -g oinstall -G asmoper,asmadmin,asmdba,racdba grid
[root@primary01 ~]# id grid
uid=54322(grid) gid=54321(oinstall) groups=54321(oinstall),54330(racdba),54327(asmdba),54328(asmoper),54329(asmadmin)
[root@primary01 ~]# usermod -a -G asmdba oracle
[root@primary01 ~]# id oracle
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(oper),54324(backupdba),54325(dgdba),54326(kmdba),54330(racdba),54327(asmdba)As you can see, Oracle pre-installation package has already created some groups for us.
User Credentials
grid and oracle still have no password, we should provide passwords for them.
On both nodes.
[root@primary01 ~]# passwd grid
Changing password for user grid.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
[root@primary01 ~]# passwd oracle
Changing password for user oracle.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.Directories
On both nodes, we create necessary directories for grid and oracle.
[root@primary01 ~]# mkdir -p /u01/app/19.0.0/grid
[root@primary01 ~]# mkdir -p /u01/app/grid
[root@primary01 ~]# mkdir -p /u01/app/oracle/product/19.0.0/db_1
[root@primary01 ~]# chown -R grid:oinstall /u01
[root@primary01 ~]# chown -R oracle:oinstall /u01/app/oracle
[root@primary01 ~]# chmod -R 775 /u01/Set Profiles for grid and oracle
First, we set profiles for user grid on both nodes.
[root@primary01 ~]# su - grid
[grid@primary01 ~]$ vi .bash_profile
...
# User specific environment and startup programs
umask 022
ORACLE_SID=+ASM1
ORACLE_BASE=/u01/app/grid
ORACLE_HOME=/u01/app/19.0.0/grid
LD_LIBRARY_PATH=$ORACLE_HOME/lib
TMP=/tmp
TMPDIR=/tmp
PATH=$PATH:$HOME/.local/bin:$HOME/bin:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch
export PATH ORACLE_SID ORACLE_BASE ORACLE_HOME LD_LIBRARY_PATH TMP TMPDIR
export CV_ASSUME_DISTID=OEL8.1
[grid@primary01 ~]$ exit
logoutNow turn to user oracle on both nodes.
[root@primary01 ~]# su - oracle
[oracle@primary01 ~]$ vi .bash_profile
...
# User specific environment and startup programs
umask 022
ORACLE_SID=ORCLCDB1
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=$ORACLE_BASE/product/19.0.0/db_1
LD_LIBRARY_PATH=$ORACLE_HOME/lib
TMP=/tmp
TMPDIR=/tmp
PATH=$PATH:$HOME/.local/bin:$HOME/bin:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch
export PATH ORACLE_SID ORACLE_BASE ORACLE_HOME LD_LIBRARY_PATH TMP TMPDIR
export CV_ASSUME_DISTID=OEL8.1
[oracle@primary01 ~]$ exit
logoutPlease change the number in ORACLE_SID for the second node.
Set Resource Limitations
On both nodes.
[root@primary01 ~]# vi /etc/security/limits.conf
...
grid soft nofile 1024
grid hard nofile 65536
grid soft nproc 2047
grid hard nproc 16384
grid soft stack 10240
grid hard stack 32768
grid soft memlock unlimited
grid hard memlock unlimited
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft stack 10240
oracle hard stack 32768
oracle soft memlock unlimited
oracle hard memlock unlimited [root@primary01 ~]# sysctl -p
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500Storage
In our design, we use a NAS which provides iSCSI sharing service. Our database servers play as iSCSI clients, so we need to install iSCSI initiator package. For Solaris database servers, you may check: How to Configure iSCSI Disk on Solaris.
As I mentioned earlier, I assume that you have already prepared a shared storage for database servers to connect. You may find a tutorial here: How to Build 12c RAC (1/6) - Building a NAS for Shared Storage of RAC.
First of all, install iSCSI initiator on both nodes.
[root@primary01 ~]# yum info iscsi-initiator-utils lsscsi
Last metadata expiration check: 0:00:25 ago on Mon 28 Sep 2020 01:36:42 PM CST.
Installed Packages
Name : lsscsi
Version : 0.30
Release : 1.el8
Architecture : x86_64
Size : 124 k
Source : lsscsi-0.30-1.el8.src.rpm
Repository : @System
From repo : anaconda
Summary : List SCSI devices (or hosts) and associated information
URL : http://sg.danny.cz/scsi/lsscsi.html
License : GPLv2+
Description : Uses information provided by the sysfs pseudo file system in
: Linux kernel 2.6 series to list SCSI devices or all SCSI hosts.
: Includes a "classic" option to mimic the output of "cat
: /proc/scsi/scsi" that has been widely used prior to the lk 2.6
: series.
:
: Author:
: --------
: Doug Gilbert <dgilbert(at)interlog(dot)com>
Available Packages
Name : iscsi-initiator-utils
Version : 6.2.0.878
Release : 4.gitd791ce0.0.1.el8
Architecture : i686
Size : 393 k
Source : iscsi-initiator-utils-6.2.0.878-4.gitd791ce0.0.1.el8.src.rpm
Repository : ol8_baseos_latest
Summary : iSCSI daemon and utility programs
URL : http://www.open-iscsi.org
License : GPLv2+
Description : The iscsi package provides the server daemon for the iSCSI
: protocol, as well as the utility programs used to manage it.
: iSCSI is a protocol for distributed disk access using SCSI
: commands sent over Internet Protocol networks.
Name : iscsi-initiator-utils
Version : 6.2.0.878
Release : 4.gitd791ce0.0.1.el8
Architecture : src
Size : 683 k
Source : None
Repository : ol8_baseos_latest
Summary : iSCSI daemon and utility programs
URL : http://www.open-iscsi.org
License : GPLv2+
Description : The iscsi package provides the server daemon for the iSCSI
: protocol, as well as the utility programs used to manage it.
: iSCSI is a protocol for distributed disk access using SCSI
: commands sent over Internet Protocol networks.
Name : iscsi-initiator-utils
Version : 6.2.0.878
Release : 4.gitd791ce0.0.1.el8
Architecture : x86_64
Size : 378 k
Source : iscsi-initiator-utils-6.2.0.878-4.gitd791ce0.0.1.el8.src.rpm
Repository : ol8_baseos_latest
Summary : iSCSI daemon and utility programs
URL : http://www.open-iscsi.org
License : GPLv2+
Description : The iscsi package provides the server daemon for the iSCSI
: protocol, as well as the utility programs used to manage it.
: iSCSI is a protocol for distributed disk access using SCSI
: commands sent over Internet Protocol networks.
Name : lsscsi
Version : 0.30
Release : 1.el8
Architecture : src
Size : 200 k
Source : None
Repository : ol8_baseos_latest
Summary : List SCSI devices (or hosts) and associated information
URL : http://sg.danny.cz/scsi/lsscsi.html
License : GPLv2+
Description : Uses information provided by the sysfs pseudo file system in
: Linux kernel 2.6 series to list SCSI devices or all SCSI hosts.
: Includes a "classic" option to mimic the output of "cat
: /proc/scsi/scsi" that has been widely used prior to the lk 2.6
: series.
:
: Author:
: --------
: Doug Gilbert <dgilbert(at)interlog(dot)com>
[root@primary01 ~]# yum -y install iscsi-initiator-utils lsscsi
Last metadata expiration check: 0:00:36 ago on Mon 28 Sep 2020 01:36:42 PM CST.
Package lsscsi-0.30-1.el8.x86_64 is already installed.
Dependencies resolved.
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
iscsi-initiator-utils
x86_64 6.2.0.878-4.gitd791ce0.0.1.el8 ol8_baseos_latest 378 k
Installing dependencies:
isns-utils-libs x86_64 0.99-1.el8 ol8_baseos_latest 105 k
iscsi-initiator-utils-iscsiuio
x86_64 6.2.0.878-4.gitd791ce0.0.1.el8 ol8_baseos_latest 101 k
Transaction Summary
================================================================================
Install 3 Packages
Total download size: 584 k
Installed size: 2.5 M
Downloading Packages:
(1/3): isns-utils-libs-0.99-1.el8.x86_64.rpm 316 kB/s | 105 kB 00:00
(2/3): iscsi-initiator-utils-iscsiuio-6.2.0.878 282 kB/s | 101 kB 00:00
(3/3): iscsi-initiator-utils-6.2.0.878-4.gitd79 714 kB/s | 378 kB 00:00
--------------------------------------------------------------------------------
Total 1.1 MB/s | 584 kB 00:00
warning: /var/cache/dnf/ol8_baseos_latest-e4c6155830ad002c/packages/isns-utils-libs-0.99-1.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ad986da3: NOKEY
Oracle Linux 8 BaseOS Latest (x86_64) 84 kB/s | 3.1 kB 00:00
Importing GPG key 0xAD986DA3:
Userid : "Oracle OSS group (Open Source Software group) <build@oss.oracle.com>"
Fingerprint: 76FD 3DB1 3AB6 7410 B89D B10E 8256 2EA9 AD98 6DA3
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : isns-utils-libs-0.99-1.el8.x86_64 1/3
Running scriptlet: isns-utils-libs-0.99-1.el8.x86_64 1/3
Installing : iscsi-initiator-utils-iscsiuio-6.2.0.878-4.gitd791ce 2/3
Running scriptlet: iscsi-initiator-utils-iscsiuio-6.2.0.878-4.gitd791ce 2/3
Installing : iscsi-initiator-utils-6.2.0.878-4.gitd791ce0.0.1.el8 3/3
Running scriptlet: iscsi-initiator-utils-6.2.0.878-4.gitd791ce0.0.1.el8 3/3
Verifying : isns-utils-libs-0.99-1.el8.x86_64 1/3
Verifying : iscsi-initiator-utils-iscsiuio-6.2.0.878-4.gitd791ce 2/3
Verifying : iscsi-initiator-utils-6.2.0.878-4.gitd791ce0.0.1.el8 3/3
Installed:
iscsi-initiator-utils-6.2.0.878-4.gitd791ce0.0.1.el8.x86_64
isns-utils-libs-0.99-1.el8.x86_64
iscsi-initiator-utils-iscsiuio-6.2.0.878-4.gitd791ce0.0.1.el8.x86_64
Complete!Then enable and start iSCSI services on both nodes.
[root@primary01 ~]# systemctl enable iscsid
Created symlink /etc/systemd/system/multi-user.target.wants/iscsid.service → /usr/lib/systemd/system/iscsid.service.
[root@primary01 ~]# systemctl enable iscsi
[root@primary01 ~]# systemctl start iscsid
[root@primary01 ~]# systemctl start iscsiNow, we discover all targets from NAS on both nodes. In this case, we prepared 8 iSCSI entents, 16 GB each to associate targets in NAS.
[root@primary01 ~]# iscsiadm -m discovery -t sendtargets -p nas
192.168.10.101:3260,257 iqn.2005-10.org.freenas.ctl:primary-target
[root@primary01 ~]# iscsiadm -m node --op update -n node.startup -v automatic
[root@primary01 ~]# iscsiadm -m node -p nas --login
Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:primary-target, portal: 192.168.10.101,3260] (multiple)
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:asm-primary-target, portal: 192.168.10.101,3260] successful.
...List all iSCSI lun by path.
[root@primary01 ~]# ll /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root 9 Sep 28 13:44 ip-192.168.10.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:primary-target-lun-0 -> ../../sdb
lrwxrwxrwx 1 root root 9 Sep 28 13:44 ip-192.168.10.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:primary-target-lun-1 -> ../../sdc
lrwxrwxrwx 1 root root 9 Sep 28 13:44 ip-192.168.10.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:primary-target-lun-2 -> ../../sdd
lrwxrwxrwx 1 root root 9 Sep 28 13:44 ip-192.168.10.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:primary-target-lun-3 -> ../../sde
...Then we have to partition all the 4 shared disks only on one node. Usually, node 1 is our first choice. The procedures to partition a disk are all the same. So we take /dev/sdb for an example here.
[root@primary01 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xa2316ff2.
Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p):
Using default response p.
Partition number (1-4, default 1):
First sector (2048-209715199, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-209715199, default 209715199):
Created a new partition 1 of type 'Linux' and of size 100 GiB.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.Let's see disks after partitioning.
[root@primary01 ~]# ll /dev/disk/by-path/
total 0
lrwxrwxrwx. 1 root root 9 Sep 28 13:52 ip-192.168.10.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:primary-target-lun-0 -> ../../sdb
lrwxrwxrwx. 1 root root 10 Sep 28 13:52 ip-192.168.10.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:primary-target-lun-0-part1 -> ../../sdb1
lrwxrwxrwx. 1 root root 9 Sep 28 13:56 ip-192.168.10.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:primary-target-lun-1 -> ../../sdc
lrwxrwxrwx. 1 root root 10 Sep 28 13:56 ip-192.168.10.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:primary-target-lun-1-part1 -> ../../sdc1
lrwxrwxrwx. 1 root root 9 Sep 28 13:56 ip-192.168.10.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:primary-target-lun-2 -> ../../sdd
lrwxrwxrwx. 1 root root 10 Sep 28 13:56 ip-192.168.10.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:primary-target-lun-2-part1 -> ../../sdd1
lrwxrwxrwx. 1 root root 9 Sep 28 13:56 ip-192.168.10.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:primary-target-lun-3 -> ../../sde
lrwxrwxrwx. 1 root root 10 Sep 28 13:56 ip-192.168.10.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:primary-target-lun-3-part1 -> ../../sde1
...On the other node, i.e. node 2, we have to refresh the partition table to get the newest disk information.
[root@primary02 ~]# partprobe -s
...Persistent Naming
Since those iSCSI disks require to be recognized as block devices and used by grid in the server, we should apply some persistent rules on them. This time, we don't use ASMLib, we use udev to solve the problem.
First of all, check their uuid on node 1.
[root@primary01 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdb1
36589cfc000000adffe11b1eaed4524b0
[root@primary01 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdc1
36589cfc000000528749509e95d7a4752
[root@primary01 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdd1
36589cfc0000009a888c47ef263153fd7
[root@primary01 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sde1
36589cfc0000004539fdf0756d082d400Set udev rules on both nodes. The owner should be grid and the group should be asmadmin
[root@primary01 ~]# vi /etc/udev/rules.d/99-oracle-asm-devices.rules
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36589cfc000000adffe11b1eaed4524b0", SYMLINK+="asm/disk01", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36589cfc000000528749509e95d7a4752", SYMLINK+="asm/disk02", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36589cfc0000009a888c47ef263153fd7", SYMLINK+="asm/disk03", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36589cfc0000004539fdf0756d082d400", SYMLINK+="asm/disk04", OWNER="grid", GROUP="asmadmin", MODE="0660"
Then reload the new rules on both node.
[root@primary01 ~]# partprobe -s
...
[root@primary01 ~]# udevadm control --reload-rulesLet's check the result.
[root@primary01 ~]# ll /dev/asm
total 0
lrwxrwxrwx 1 root root 7 Sep 28 14:34 disk01 -> ../sdb1
lrwxrwxrwx 1 root root 7 Sep 28 14:34 disk02 -> ../sdc1
lrwxrwxrwx 1 root root 7 Sep 28 14:34 disk03 -> ../sdd1
lrwxrwxrwx 1 root root 7 Sep 28 14:34 disk04 -> ../sde1
[root@primary01 ~]# ll /dev/sd[b-e]1*
brw-rw---- 1 grid asmadmin 8, 17 Sep 28 14:34 /dev/sdb1
brw-rw---- 1 grid asmadmin 8, 33 Sep 28 14:34 /dev/sdc1
brw-rw---- 1 grid asmadmin 8, 49 Sep 28 14:34 /dev/sdd1
brw-rw---- 1 grid asmadmin 8, 65 Sep 28 14:34 /dev/sde1 The owner of disks have been changed into grid:asmadmin. That is, they can be discovered by grid infra.
Software
This section, we upload and unzip Oracle installation software of grid infrastructure and oracle home to the first node.
Starting with Oracle Grid Infrastructure 12c Release 2 (12.2), installation and configuration of Oracle Grid Infrastructure software is simplified with image-based installation. That is to say, we have to upload and extract the source file in grid's ORACLE_HOME.
Upload
Suppose that you have uploaded both installation software to node 1.
- Grid Infrastructure
- Oracle Home
[root@primary01 ~]# ll /home/grid/sources
total 2821472
-rwxr-xr-x 1 grid oinstall 2889184573 Sep 28 14:40 LINUX.X64_193000_grid_home.zip [root@primary01 ~]# ll /home/oracle/sources
total 2987996
-rwxr-xr-x 1 oracle oinstall 3059705302 Sep 28 14:46 LINUX.X64_193000_db_home.zipUnzip
[root@primary01 ~]# su - grid
[grid@primary01 ~]$ unzip -q /home/grid/sources/LINUX.X64_193000_grid_home.zip -d $ORACLE_HOME
[grid@primary01 ~]$ exit
[root@primary01 ~]# su - oracle
[oracle@primary01 ~]$ unzip -q /home/oracle/sources/LINUX.X64_193000_db_home.zip -d $ORACLE_HOME
[oracle@primary01 ~]$ exitInstall Cluster Verification Utility
Since we have already had unzipped software, we need to install a very special package, Cluster Verification Utility, which will be used during installation of grid infrastructure.
On node 1.
[root@primary01 ~]# cp -p /u01/app/19.0.0/grid/cv/rpm/cvuqdisk-1.0.10-1.rpm /tmp
[root@primary01 ~]# scp -p /u01/app/19.0.0/grid/cv/rpm/cvuqdisk-1.0.10-1.rpm primary02:/tmp
The authenticity of host 'primary02 (192.168.10.12)' can't be established.
ECDSA key fingerprint is SHA256:...
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'primary02,192.168.10.12' (ECDSA) to the list of known hosts.
root@primary02's password:
cvuqdisk-1.0.10-1.rpmOn both nodes.
[root@primary01 ~]# rpm -Uvh /tmp/cvuqdisk-1.0.10-1.rpm
Verifying... ################################# [100%]
Preparing... ################################# [100%]
Using default group oinstall to install package
Updating / installing...
1:cvuqdisk-1.0.10-1 ################################# [100%]Check the result.
[root@primary01 ~]# rpm -q cvuqdisk
cvuqdisk-1.0.10-1.x86_64Cluster Verification
Before we install Grid infra, we should verify everything is ready for grid installation. Here we use Oracle Cluster Verification Utility (CLUVFY) to check if there's any problem by grid.
[grid@primary01 ~]$ cd $ORACLE_HOME
[grid@primary01 grid]$ ./runcluvfy.sh stage -pre crsinst -n primary01,primary02 -verbose
ERROR:
PRVG-10467 : The default Oracle Inventory group could not be determined.
Verifying User Equivalence ...FAILED (PRVG-2019, PRKC-1191)
PRVF-4009 : User equivalence is not set for nodes: primary02
Verification will proceed with nodes: primary01
Verifying Physical Memory ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
primary01 3.6695GB (3847752.0KB) 8GB (8388608.0KB) failed
Verifying Physical Memory ...FAILED (PRVF-7530)
Verifying Available Physical Memory ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
primary01 3.219GB (3375316.0KB) 50MB (51200.0KB) passed
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
primary01 3.9531GB (4145148.0KB) 3.6695GB (3847752.0KB) passed
Verifying Swap Size ...PASSED
Verifying Free Space: primary01:/usr,primary01:/var,primary01:/etc,primary01:/sbin,primary01:/tmp ...
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/usr primary01 / 78.4101GB 25MB passed
/var primary01 / 78.4101GB 5MB passed
/etc primary01 / 78.4101GB 25MB passed
/sbin primary01 / 78.4101GB 10MB passed
/tmp primary01 / 78.4101GB 1GB passed
Verifying Free Space: primary01:/usr,primary01:/var,primary01:/etc,primary01:/sbin,primary01:/tmp ...PASSED
Verifying User Existence: grid ...
Node Name Status Comment
------------ ------------------------ ------------------------
primary01 passed exists(54322)
Verifying Users With Same UID: 54322 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying Group Existence: asmadmin ...
Node Name Status Comment
------------ ------------------------ ------------------------
primary01 passed exists
Verifying Group Existence: asmadmin ...PASSED
Verifying Group Existence: asmdba ...
Node Name Status Comment
------------ ------------------------ ------------------------
primary01 passed exists
Verifying Group Existence: asmdba ...PASSED
Verifying Group Membership: asmdba ...
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
primary01 yes yes yes passed
Verifying Group Membership: asmdba ...PASSED
Verifying Group Membership: asmadmin ...
Node Name User Exists Group Exists User in Group Status
---------------- ------------ ------------ ------------ ----------------
primary01 yes yes yes passed
Verifying Group Membership: asmadmin ...PASSED
Verifying Run Level ...
Node Name run level Required Status
------------ ------------------------ ------------------------ ----------
primary01 3 3,5 passed
Verifying Run Level ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...
Node Name Status
------------------------------------ ------------------------
primary01 passed
Verifying Root user consistency ...PASSED
Verifying Package: cvuqdisk-1.0.10-1 ...
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
primary01 cvuqdisk-1.0.10-1 cvuqdisk-1.0.10-1 passed
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
Verifying Hosts File ...
Node Name Status
------------------------------------ ------------------------
primary01 passed
Verifying Hosts File ...PASSED
Interface information for node "primary01"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
ens33 192.168.0.11 192.168.0.0 0.0.0.0 192.168.0.12 9 00:0C:29:DC:65:90 1500
ens34 192.168.24.11 192.168.24.0 0.0.0.0 192.168.0.12 9 00:0C:29:DC:65:9A 1500
ens33 2402:7500:52d:1a76:20c:29ff:fedc:6590 2402:7500:52d:1a76:0:0:0:0 UNKNOWN :: 00:0C:29:DC:65:90 1500
Check: MTU consistency of the subnet "192.168.0.0".
Node Name IP Address Subnet MTU
---------------- ------------ ------------ ------------ ----------------
primary01 ens33 192.168.0.11 192.168.0.0 1500
Check: MTU consistency of the subnet "2402:7500:52d:1a76:0:0:0:0".
Node Name IP Address Subnet MTU
---------------- ------------ ------------ ------------ ----------------
primary01 ens33 2402:7500:52d:1a76:20c:29ff:fedc:6590 2402:7500:52d:1a76:0:0:0:0 1500
Check: MTU consistency of the subnet "192.168.24.0".
Node Name IP Address Subnet MTU
---------------- ------------ ------------ ------------ ----------------
primary01 ens34 192.168.24.11 192.168.24.0 1500
Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...
Checking subnet "192.168.0.0" for multicast communication with multicast group "224.0.0.251"
Verifying Multicast or broadcast check ...PASSED
Verifying Network Time Protocol (NTP) ...PASSED
Verifying Same core file name pattern ...PASSED
Verifying User Mask ...
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
primary01 0022 0022 passed
Verifying User Mask ...PASSED
Verifying User Not In Group "root": grid ...
Node Name Status Comment
------------ ------------------------ ------------------------
primary01 passed does not exist
Verifying User Not In Group "root": grid ...PASSED
Verifying Time zone consistency ...PASSED
Verifying resolv.conf Integrity ...
Node Name Status
------------------------------------ ------------------------
primary01 passed
checking response for name "primary01" from each of the name servers specified
in "/etc/resolv.conf"
Node Name Source Comment Status
------------ ------------------------ ------------------------ ----------
primary01 192.168.0.199 IPv4 passed
Verifying resolv.conf Integrity ...PASSED
Verifying DNS/NIS name service ...PASSED
Verifying Domain Sockets ...PASSED
Verifying /boot mount ...PASSED
Verifying Daemon "avahi-daemon" not configured and running ...
Node Name Configured Status
------------ ------------------------ ------------------------
primary01 no passed
Node Name Running? Status
------------ ------------------------ ------------------------
primary01 no passed
Verifying Daemon "avahi-daemon" not configured and running ...PASSED
Verifying Daemon "proxyt" not configured and running ...
Node Name Configured Status
------------ ------------------------ ------------------------
primary01 no passed
Node Name Running? Status
------------ ------------------------ ------------------------
primary01 no passed
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying User Equivalence ...PASSED
Verifying RPM Package Manager database ...INFORMATION (PRVG-11250)
Verifying /dev/shm mounted as temporary file system ...PASSED
Verifying File system mount options for path /var ...PASSED
Verifying DefaultTasksMax parameter ...PASSED
Verifying zeroconf check ...PASSED
Verifying ASM Filter Driver configuration ...PASSED
Pre-check for cluster services setup was unsuccessful on all the nodes.
Failures were encountered during execution of CVU verification request "stage -pre crsinst".
Verifying User Equivalence ...FAILED
primary02: PRVG-2019 : Check for equivalence of user "grid" from node
"primary01" to node "primary02" failed
PRKC-1191 : Remote command execution setup check for node primary02
using shell /usr/bin/ssh failed.
No ECDSA host key is known for primary02 and you have requested
strict checking.Host key verification failed.
Verifying Physical Memory ...FAILED
primary01: PRVF-7530 : Sufficient physical memory is not available on node
"primary01" [Required physical memory = 8GB (8388608.0KB)]
Verifying RPM Package Manager database ...INFORMATION
PRVG-11250 : The check "RPM Package Manager database" was not performed because
it needs 'root' user privileges.
CVU operation performed: stage -pre crsinst
Date: Oct 12, 2020 1:25:41 PM
CVU home: /u01/app/19.0.0/grid/
User: gridAs we can see, there're some problems:
- Oracle Inventory group
- User equivalence
- Physical memory
PRVG-10467 : The default Oracle Inventory group could not be determined.Since we have never install any Oracle products on this server, so this error is normal. If you're not sure about this, you can try to figure out where Oracle central inventory is and how to move it.
PRVF-4009 : User equivalence is not set for nodes: primary02or this:
PRVG-2019 : Check for equivalence of user "grid" from nodeYes, we have not set user equivalency for grid yet, we'd like to establish SSH equivalency during grid installation, so this error is also normal. Of course, you can establish SSH equivalency for grid before installation.
PRVF-7530 : Sufficient physical memory is not available on nodeThat's true, we didn't reach the requirement of physical memory, but it can be ignored.
No major problem here, so we can just keep going.
By the way, if runcluvfy.sh seems stuck after issuing, you may revert scp fix for OpenSSH 8.x problem, then try again.
More Fixes
Various platforms may have various problems, if you met them, here are some fixed that may be helpful.
Version Fix
When you started OUI, you may see INS-08101 Unexpected error while executing the action at state: 'supportedOSCheck', if you were using Enterprise Linux 8 or above. You should set CV_ASSUME_DISTID environment variable to a lower value to solve INS-08101, say OEL7.8 to make OUI think it's a compatible OS release.
Shared Memory
For some servers, you may need to tune shared memory manually.
[root@primary01 ~]# vi /etc/fstab
...
tmpfs /dev/shm tmpfs size=6g 0 0Or use MB as the unit.
tmpfs /dev/shm tmpfs size=6144m 0 0To make it take effect immediately, you can do this:
[root@primary01 ~]# mount -o remount /dev/shmscp Fix
Since we choose Enterprise Linux 8.1 for our database servers, we should fix scp for INS-06006 problem during installation.
[root@primary01 ~]# cp -p /usr/bin/scp /usr/bin/scp-original
[root@primary01 ~]# echo "/usr/bin/scp-original -T \$*" > /usr/bin/scp
[root@primary01 ~]# cat /usr/bin/scp
/usr/bin/scp-original -T $*DNS Fix
DNS listed in /etc/resolv.conf should be all internal DNS for solving the hostnames of database servers, otherwise, a DNS warning may occur during system check of installation.
To fix the problem, you should modify the configuration of network interface. For example, I only keep the first DNS and comment out the rests in the network interface.
[root@primary01 ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33
...
DNS1=192.168.10.199
#DNS2=...
#DNS3=...Everything seems ready, we can go for installing Oracle 19c grid infrastructure.
Thanks, dude. I know what I’M doing tonight ;-P
My pleasure!