How to Build 19c RAC - 1 of 4 - Preparing Two Database Servers

The series of post will install a 19c (19.3) RAC database on two Enterprise Linux 8.1. There're 4 parts in this series.

• Preparing Two Database Servers (1 of 4)
• Installing Grid Infrastructure (2 of 4)
• Installing Database Software (3 of 4)
• Creating a RAC Database (4 of 4)

This post is the first part.

Assuming that you have already prepared a shared storage for RAC databases, so we start the series from preparing 2 database servers. For better preparing 2 nodes for database servers, we list the necessary items to check.

1. Network
2. Security
3. Time
4. Packages
5. Users, Groups and Directories
6. Storage
7. Software
8. Verification
9. More Fixes

Network

Let's see the network design of our cluster database by listing /etc/hosts.

Please note that, the format of every entry in /etc/hosts should be:

The network design has been summarized as below:

• Two nodes for RAC servers: primary01 and primary02.
• One NAS for shared storage.
• One DNS for resolving hostname, especially for SCAN name.

Please note that, you can only have local DNS server(s) listed in NIC configuration, otherwise, OUI of grid infrastructure will complain about it.

Now, I have two questions for you. You can make a guess before clicking the post.

• Can we setup SCAN VIP without DNS?
• Can we Setup Only One SCAN VIP?
• Two network cards, one is for public subnet, another is for private subnet.
• Public subnet (192.168.10.0/24): only for public connection. Later, all virtual IP like SCAN and VIP will dynamically bind with this NIC after running CRS.

Note that, public IP requires broadcast.

• Private subnet (192.168.24.0/24): for both private and ASM connections. This NIC is mainly for node interconnection, not for outside access.

Note that, private IP requires multicast.

Same hosts configuration must also appear in primary02 according to our design.

Security

We stopped firewall and selinux on both nodes to make RAC installation more smoothly.

Maybe you'd like to open port 1521 on firewall for Oracle connections instead of disabling firewall.

Time

Time synchronization is pretty important for a RAC database, if time in one node deviates from the other node's, CRS may decide to abandon it. Please make sure NTP is working well on both nodes. Here we use the default NTP service chronyd to play the role.

Packages

Since the installed edition of Linux is minimal one. We need to install all required packages (RPM) for installing Oracle 19c products later on both nodes.

Users, Groups and Directories

Although Oracle pre-installtion package installed some necessary users or groups for us, we need to create them again to make sure everything is ready.

Create Necessary Users and Groups

On both nodes.

As you can see, Oracle pre-installation package has already created some groups for us.

User Credentials

grid and oracle still have no password, we should provide passwords for them.

On both nodes.

Directories

On both nodes, we create necessary directories for grid and oracle.

Set Profiles for grid and oracle

First, we set profiles for user grid on both nodes.

Now turn to user oracle on both nodes.

Please change the number in ORACLE_SID for the second node.

Set Resource Limitations

On both nodes.

Storage

In our design, we use a NAS which provides iSCSI sharing service. Our database servers play as iSCSI clients, so we need to install iSCSI initiator package. For Solaris database servers, you may check: How to Configure iSCSI Disk on Solaris.

As I mentioned earlier, I assume that you have already prepared a shared storage for database servers to connect. You may find a tutorial here: How to Build 12c RAC (1/6) - Building a NAS for Shared Storage of RAC.

First of all, install iSCSI initiator on both nodes.

Then enable and start iSCSI services on both nodes.

Now, we discover all targets from NAS on both nodes. In this case, we prepared 8 iSCSI entents, 16 GB each to associate targets in NAS.

List all iSCSI lun by path.

Then we have to partition all the 4 shared disks only on one node. Usually, node 1 is our first choice. The procedures to partition a disk are all the same. So we take /dev/sdb for an example here.

Let's see disks after partitioning.

On the other node, i.e. node 2, we have to refresh the partition table to get the newest disk information.

Persistent Naming

Since those iSCSI disks require to be recognized as block devices and used by grid in the server, we should apply some persistent rules on them. This time, we don't use ASMLib, we use udev to solve the problem.

First of all, check their uuid on node 1.

Set udev rules on both nodes. The owner should be grid and the group should be asmadmin

[root@primary01 ~]# vi /etc/udev/rules.d/99-oracle-asm-devices.rules
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36589cfc000000adffe11b1eaed4524b0", SYMLINK+="asm/disk01", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36589cfc000000528749509e95d7a4752", SYMLINK+="asm/disk02", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36589cfc0000009a888c47ef263153fd7", SYMLINK+="asm/disk03", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="36589cfc0000004539fdf0756d082d400", SYMLINK+="asm/disk04", OWNER="grid", GROUP="asmadmin", MODE="0660"

Then reload the new rules on both node.

Let's check the result.

The owner of disks have been changed into grid:asmadmin. That is, they can be discovered by grid infra.

Software

This section, we upload and unzip Oracle installation software of grid infrastructure and oracle home to the first node.

Starting with Oracle Grid Infrastructure 12c Release 2 (12.2), installation and configuration of Oracle Grid Infrastructure software is simplified with image-based installation. That is to say, we have to upload and extract the source file in grid's ORACLE_HOME.

Upload

Suppose that you have uploaded both installation software to node 1.

• Grid Infrastructure
[root@primary01 ~]# ll /home/grid/sources
total 2821472
-rwxr-xr-x 1 grid oinstall 2889184573 Sep 28 14:40 LINUX.X64_193000_grid_home.zip
• Oracle Home
[root@primary01 ~]# ll /home/oracle/sources
total 2987996
-rwxr-xr-x 1 oracle oinstall 3059705302 Sep 28 14:46 LINUX.X64_193000_db_home.zip

Unzip

Install Cluster Verification Utility

Since we have already had unzipped software, we need to install a very special package, Cluster Verification Utility, which will be used during installation of grid infrastructure.

On node 1.

On both nodes.

Check the result.

Cluster Verification

Before we install Grid infra, we should verify everything is ready for grid installation. Here we use Oracle Cluster Verification Utility (CLUVFY) to check if there's any problem by grid.

As we can see, there're some problems:

1. Oracle Inventory group
PRVG-10467 : The default Oracle Inventory group could not be determined.

For more details about PRVG-10467, you may check How to Resolve PRVG-10467 : The default Oracle Inventory group could not be determined.

2. User equivalence
PRVF-4009 : User equivalence is not set for nodes: primary02

or this:

PRVG-2019 : Check for equivalence of user "grid" from node

Yes, we have not set user equivalency for grid yet, we'd like to establish SSH equivalency during grid installation, so this error is also normal. To solve it, you may check the post: How to Resolve PRVG-2019 : Check for equivalence of user from node.

3. Physical memory
PRVF-7530 : Sufficient physical memory is not available on node

That's true, we didn't reach the requirement of physical memory, but it can be ignored.

4. Root Privilege
PRVG-11250 : The check "RPM Package Manager database" was not performed because it needs 'root' user privileges.

For more details about PRVG-11250, you may check How to Resolve PRVG-11250 : The check "RPM Package Manager database" was not performed because it needs 'root' user privileges.

No major problem here, so we can just keep going.

By the way, if runcluvfy.sh seems stuck after issuing, you may revert scp fix for OpenSSH 8.x problem, then try again.

A perfect result may look like this:

More Fixes

Various platforms may have various problems, if you met them, here are some fixed that may be helpful.

Version Fix

When you started OUI, you may see INS-08101 Unexpected error while executing the action at state: 'supportedOSCheck', if you were using Enterprise Linux 8 or above. You should set CV_ASSUME_DISTID environment variable to a lower value to solve INS-08101, say OEL7.8 to make OUI think it's a compatible OS release.

Shared Memory

For some servers, you may need to tune shared memory manually.

Or use MB as the unit.

To make it take effect immediately, you can do this:

scp Fix

Since we choose Enterprise Linux 8.1 for our database servers, we should fix scp for INS-06006 problem during installation.

DNS Fix

DNS listed in /etc/resolv.conf should be all internal DNS for solving the hostnames of database servers, otherwise, a DNS warning may occur during system check of installation.

To fix the problem, you should modify the configuration of network interface. For example, I only keep the first DNS and comment out the rests in the network interface.

Everything seems ready, we can go for installing Oracle 19c grid infrastructure.