How to Build 12c RAC (2/6) - Building Two Servers for RAC

How to Build 12c RAC (1/6) - Building a NAS for Shared Storage of RAC
Two nodes cluster database for RAC is a very typical design for enterprises, I assumed that you have installed Linux for the two database servers already.

In this major step, we intend to build two servers and connect to the NAS which was built in the previous major step. The final result will be: Two ASM candidate disks are well prepared at each node.
  1. Check the basic information about the two servers
  2. Install iSCSI initiator on both servers
  3. Connect to NAS via iSCSI channel
  4. Format (or Clean) iSCSI disks
Let's start the steps.

A. Check the basic information about the two servers

The release of Linux is shown as following.
[root@primary01 ~]# uname -a
Linux primary01.example.com 3.8.13-44.1.1.el6uek.x86_64 #2 SMP Wed Sep 10 06:10:25 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@primary01 ~]# cat /etc/oracle-release
Oracle Linux Server release 6.6
[root@primary01 ~]# chkconfig ntpd on
[root@primary01 ~]# chkconfig | grep ntpd
ntpd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
...
[root@primary01 ~]# vi /etc/hostname
primary01.example.com

Let's see the network design in our case.

  1. Public, VIP and SCAN should be in the same subnet ( 192.168.15.0/24 in this case). Only public IP is bound to eth0  or bond0 . VIP and SCAN should not be bound to any NIC. You may refer to my post:  Why Don't We Explicitly Bind VIP and SCAN into NIC for more practices.
  2. Private IP are in a separate subnet ( 192.168.24.0/24  in our case) which is different from the subnet of the public IP, they are used as cluster interconnections. Private IP are bound to eth1  or bond1 in our case.

For the local hostnames setting:
[root@primary01 ~]# vi /etc/hosts
# Public
192.168.15.11 primary01 primary01.example.com
192.168.15.12 primary02 primary02.example.com

# Private
192.168.24.11 primary01-priv primary01-priv.example.com
192.168.24.12 primary02-priv primary02-priv.example.com

# VIP
192.168.15.111 primary01-vip primary01-vip.example.com
192.168.15.112 primary02-vip primary02-vip.example.com

# NAS
192.168.15.101 primary-nas primary-nas.example.com

# SCAN
# 192.168.15.81 primary-cluster-scan primary-cluster-scan.example.com
# 192.168.15.82 primary-cluster-scan primary-cluster-scan.example.com
# 192.168.15.83 primary-cluster-scan primary-cluster-scan.example.com

# Local
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Oracle 12c requires that SCAN name must be looked up dynamically, which means you have to register them in DNS, not in /etc/hosts . Otherwise, Oracle Universal Installer will throw warnings about this during installation.

If you'd like to know how to build a DNS service, you may refer to the post below:
How to Configure BIND Named Service on Enterprise Linux 7

B. Install iSCSI initiator on both servers

Install iscsi-initiator-utils and lsscsi two packages.
[root@primary01 ~]# yum -y install iscsi-initiator-utils lsscsi
Loaded plugins: security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package iscsi-initiator-utils.x86_64 0:6.2.0.873-14.0.1.el6 will be installed
---> Package lsscsi.x86_64 0:0.23-3.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                Arch    Version                Repository          Size
================================================================================
Installing:
 iscsi-initiator-utils  x86_64  6.2.0.873-14.0.1.el6   public_ol6_latest  724 k
 lsscsi                 x86_64  0.23-3.el6             public_ol6_latest   38 k

Transaction Summary
================================================================================
Install       2 Package(s)

Total download size: 762 k
Installed size: 2.5 M
Downloading Packages:
(1/2): iscsi-initiator-utils-6.2.0.873-14.0.1.el6.x86_64 | 724 kB     00:01
(2/2): lsscsi-0.23-3.el6.x86_64.rpm                      |  38 kB     00:00
--------------------------------------------------------------------------------
Total                                           250 kB/s | 762 kB     00:03
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
Importing GPG key 0xEC551F03:
 Userid : Oracle OSS group (Open Source Software group) <build@oss.oracle.com>
 Package: 6:oraclelinux-release-6Server-6.0.2.x86_64 (@anaconda-OracleLinuxServer-201410181705.x86_64/6.6)
 From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : lsscsi-0.23-3.el6.x86_64                                     1/2
  Installing : iscsi-initiator-utils-6.2.0.873-14.0.1.el6.x86_64            2/2
  Verifying  : iscsi-initiator-utils-6.2.0.873-14.0.1.el6.x86_64            1/2
  Verifying  : lsscsi-0.23-3.el6.x86_64                                     2/2

Installed:
  iscsi-initiator-utils.x86_64 0:6.2.0.873-14.0.1.el6
  lsscsi.x86_64 0:0.23-3.el6

Complete!

Enable the iscsi services
[root@primary01 ~]# chkconfig iscsi on
[root@primary01 ~]# chkconfig iscsid on
[root@primary01 ~]# service iscsi start
[root@primary01 ~]# service iscsid start
[root@primary01 ~]# init 6

C. Connect to NAS via iSCSI channel

Discover iSCSI disks.
[root@primary01 ~]# iscsiadm -m discovery -t sendtargets -p primary-nas
Starting iscsid:                                           [  OK  ]
192.168.15.101:3260,2 iqn.2005-10.org.freenas.ctl:asm-t1
192.168.15.101:3260,2 iqn.2005-10.org.freenas.ctl:asm-t2

Automatically connect to all iSCSI disks when node startup.
[root@primary01 ~]# iscsiadm -m node --op update -n node.startup -v automatic
Login NAS and map iSCSI disks.
[root@primary01 ~]# iscsiadm -m node -p primary-nas --login
Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:asm-t2, portal: 192.168.15.101,3260] (multiple)
Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:asm-t1, portal: 192.168.15.101,3260] (multiple)
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:asm-t2, portal: 192.168.15.101,3260] successful.
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:asm-t1, portal: 192.168.15.101,3260] successful.

Let's see the target disks.
[root@primary01 ~]# ll /dev/disk/by-path/
total 0
lrwxrwxrwx. 1 root root  9 Jul 29 16:37 ip-192.168.15.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:asm-t1-lun-1 -> ../../sdb
lrwxrwxrwx. 1 root root  9 Jul 29 16:37 ip-192.168.15.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:asm-t2-lun-2 -> ../../sdc
...
[root@primary01 ~]# lsscsi
...
[3:0:0:2]    disk    FreeBSD  iSCSI Disk       0123  /dev/sdc
[4:0:0:1]    disk    FreeBSD  iSCSI Disk       0123  /dev/sdb

By the way, if you want to clear discovered targets, you can do this:

[root@primary01 ~]# iscsiadm -m node -o delete -p 192.168.15.101:3260

D. Format (or Reuse) iSCSI disks

Before we can use the shared storage, we should format the two disks (i.e. /dev/sdb , /dev/sdc ).
[root@primary01 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x160980b9.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-16384, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-16384, default 16384):
Using default value 16384

Command (m for help): p

Disk /dev/sdb: 17.2 GB, 17179869184 bytes
64 heads, 32 sectors/track, 16384 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 1048576 bytes
Disk identifier: 0x160980b9

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       16384    16777200   83  Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@primary01 ~]# fdisk /dev/sdc
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xab60b776.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-16384, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-16384, default 16384):
Using default value 16384

Command (m for help): p

Disk /dev/sdc: 17.2 GB, 17179869184 bytes
64 heads, 32 sectors/track, 16384 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 1048576 bytes
Disk identifier: 0xab60b776

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1       16384    16777200   83  Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Let's see the result of disk format.
[root@primary01 ~]# ll /dev/disk/by-path/
total 0
lrwxrwxrwx. 1 root root  9 Jul 31 15:00 ip-192.168.15.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:asm-t1-lun-1 -> ../../sdb
lrwxrwxrwx. 1 root root 10 Jul 31 15:00 ip-192.168.15.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:asm-t1-lun-1 -part1 -> ../../sdb1
lrwxrwxrwx. 1 root root  9 Jul 31 15:00 ip-192.168.15.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:asm-t2-lun-2 -> ../../sdc
lrwxrwxrwx. 1 root root 10 Jul 31 15:00 ip-192.168.15.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:asm-t2-lun-2 -part1 -> ../../sdc1
...

On node 2, you don't have to format them again, just do a partprobe  to get the newest status.
[root@primary02 ~]# partprobe
If the block devices are already existent before the the installation job, you should be better to clean up the header of the ASM candidate disks on any one of the two nodes.
[root@primary01 ~]# dd if=/dev/zero of=/dev/sdb1 bs=1024 count=500
500+0 records in
500+0 records out
512000 bytes (512 kB) copied, 0.0839358 s, 6.1 MB/s
[root@primary01 ~]# dd if=/dev/zero of=/dev/sdc1 bs=1024 count=500
500+0 records in
500+0 records out
512000 bytes (512 kB) copied, 0.0506696 s, 10.1 MB/s

As you can see, we cleaned 500KB of the head of the devices. If you suspect that most of the devices are dirty, you can increase the block size (bs) and the loop counts.

So far, we have two RAC nodes with ASM candidate disks ready.

How to Build 12c RAC (3/6) - Preparing for Installations of Grid and RAC

2 thoughts on “How to Build 12c RAC (2/6) - Building Two Servers for RAC”

  1. Hello Ed,

    I am following your guide as a basis of setting up a test Oracle RAC 19c, in vCLoud, however I am a little confused on the network configuration.

    Should the RAC nodes have 4 nics in total? 3 for Public, VIP and SCAN and 1 for Private, or 2 nics, 1 public and 1 private, and VIP and SCAN being VLAN type based of the public?

Leave a Reply

Your email address will not be published. Required fields are marked *