Two nodes cluster database for RAC is a very typical design for enterprises, I assumed that you have installed Linux for the two database servers already.
In this major step, we intend to build two servers and connect to the NAS which was built in the previous major step. The final result will be: Two ASM candidate disks are well prepared at each node.
- Check the basic information about the two servers
- Install iSCSI initiator on both servers
- Connect to NAS via iSCSI channel
- Format (or Clean) iSCSI disks
Let's start the steps.
A. Check the basic information about the two servers
The release of Linux is shown as following.
[root@primary01 ~]# uname -a
Linux primary01.example.com 3.8.13-44.1.1.el6uek.x86_64 #2 SMP Wed Sep 10 06:10:25 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@primary01 ~]# cat /etc/oracle-release
Oracle Linux Server release 6.6
[root@primary01 ~]# chkconfig ntpd on
[root@primary01 ~]# chkconfig | grep ntpd
ntpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
...
[root@primary01 ~]# vi /etc/hostname
primary01.example.com
Let's see the network design in our case.
- Public, VIP and SCAN should be in the same subnet (192.168.15.0/24 in this case). Only public IP is bound to eth0 or bond0. VIP and SCAN should not be bound to any NIC. You may refer to my post: Why Don't We Explicitly Bind VIP and SCAN into NIC for more practices.
- Private IP are in a separate subnet (192.168.24.0/24 in our case) which is different from the subnet of the public IP, they are used as cluster interconnections. Private IP are bound to eth1 or bond1 in our case.
For the local hostnames setting:
[root@primary01 ~]# vi /etc/hosts
# Public
192.168.15.11 primary01 primary01.example.com
192.168.15.12 primary02 primary02.example.com
# Private
192.168.24.11 primary01-priv primary01-priv.example.com
192.168.24.12 primary02-priv primary02-priv.example.com
# VIP
192.168.15.111 primary01-vip primary01-vip.example.com
192.168.15.112 primary02-vip primary02-vip.example.com
# NAS
192.168.15.101 primary-nas primary-nas.example.com
# SCAN
# 192.168.15.81 primary-cluster-scan primary-cluster-scan.example.com
# 192.168.15.82 primary-cluster-scan primary-cluster-scan.example.com
# 192.168.15.83 primary-cluster-scan primary-cluster-scan.example.com
# Local
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
Oracle 12c requires that SCAN name must be looked up dynamically, which means you have to register them in DNS, not in /etc/hosts. Otherwise, Oracle Universal Installer will throw warnings about this during installation.
If you'd like to know how to build a DNS service, you may refer to the post: How to Configure BIND Named Service on Enterprise Linux 7
B. Install iSCSI initiator on both servers
Install iscsi-initiator-utils and lsscsi two packages.
[root@primary01 ~]# yum -y install iscsi-initiator-utils lsscsi
Loaded plugins: security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package iscsi-initiator-utils.x86_64 0:6.2.0.873-14.0.1.el6 will be installed
---> Package lsscsi.x86_64 0:0.23-3.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
iscsi-initiator-utils x86_64 6.2.0.873-14.0.1.el6 public_ol6_latest 724 k
lsscsi x86_64 0.23-3.el6 public_ol6_latest 38 k
Transaction Summary
================================================================================
Install 2 Package(s)
Total download size: 762 k
Installed size: 2.5 M
Downloading Packages:
(1/2): iscsi-initiator-utils-6.2.0.873-14.0.1.el6.x86_64 | 724 kB 00:01
(2/2): lsscsi-0.23-3.el6.x86_64.rpm | 38 kB 00:00
--------------------------------------------------------------------------------
Total 250 kB/s | 762 kB 00:03
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
Importing GPG key 0xEC551F03:
Userid : Oracle OSS group (Open Source Software group) <[email protected]>
Package: 6:oraclelinux-release-6Server-6.0.2.x86_64 (@anaconda-OracleLinuxServer-201410181705.x86_64/6.6)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : lsscsi-0.23-3.el6.x86_64 1/2
Installing : iscsi-initiator-utils-6.2.0.873-14.0.1.el6.x86_64 2/2
Verifying : iscsi-initiator-utils-6.2.0.873-14.0.1.el6.x86_64 1/2
Verifying : lsscsi-0.23-3.el6.x86_64 2/2
Installed:
iscsi-initiator-utils.x86_64 0:6.2.0.873-14.0.1.el6
lsscsi.x86_64 0:0.23-3.el6
Complete!
Enable the iscsi services
[root@primary01 ~]# chkconfig iscsi on
[root@primary01 ~]# chkconfig iscsid on
[root@primary01 ~]# service iscsi start
[root@primary01 ~]# service iscsid start
[root@primary01 ~]# init 6
C. Connect to NAS via iSCSI channel
Discover iSCSI disks.
[root@primary01 ~]# iscsiadm -m discovery -t sendtargets -p primary-nas
Starting iscsid: [ OK ]
192.168.15.101:3260,2 iqn.2005-10.org.freenas.ctl:asm-t1
192.168.15.101:3260,2 iqn.2005-10.org.freenas.ctl:asm-t2
Automatically connect to all iSCSI disks when node startup.
[root@primary01 ~]# iscsiadm -m node --op update -n node.startup -v automatic
Login NAS and map iSCSI disks.
[root@primary01 ~]# iscsiadm -m node -p primary-nas --login
Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:asm-t2, portal: 192.168.15.101,3260] (multiple)
Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:asm-t1, portal: 192.168.15.101,3260] (multiple)
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:asm-t2, portal: 192.168.15.101,3260] successful.
Login to [iface: default, target: iqn.2005-10.org.freenas.ctl:asm-t1, portal: 192.168.15.101,3260] successful.
Let's see the target disks.
[root@primary01 ~]# ll /dev/disk/by-path/
total 0
lrwxrwxrwx. 1 root root 9 Jul 29 16:37 ip-192.168.15.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:asm-t1-lun-1 -> ../../sdb
lrwxrwxrwx. 1 root root 9 Jul 29 16:37 ip-192.168.15.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:asm-t2-lun-2 -> ../../sdc
...
[root@primary01 ~]# lsscsi
...
[3:0:0:2] disk FreeBSD iSCSI Disk 0123 /dev/sdc
[4:0:0:1] disk FreeBSD iSCSI Disk 0123 /dev/sdb
By the way, if you want to clear discovered targets, you can do this:
[root@primary01 ~]# iscsiadm -m node -o delete -p 192.168.15.101:3260
D. Format (or Reuse) iSCSI disks
Before we can use the shared storage, we should format the two disks (i.e. /dev/sdb, /dev/sdc).
[root@primary01 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x160980b9.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-16384, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-16384, default 16384):
Using default value 16384
Command (m for help): p
Disk /dev/sdb: 17.2 GB, 17179869184 bytes
64 heads, 32 sectors/track, 16384 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 1048576 bytes
Disk identifier: 0x160980b9
Device Boot Start End Blocks Id System
/dev/sdb1 1 16384 16777200 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@primary01 ~]# fdisk /dev/sdc
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xab60b776.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-16384, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-16384, default 16384):
Using default value 16384
Command (m for help): p
Disk /dev/sdc: 17.2 GB, 17179869184 bytes
64 heads, 32 sectors/track, 16384 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 1048576 bytes
Disk identifier: 0xab60b776
Device Boot Start End Blocks Id System
/dev/sdc1 1 16384 16777200 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Let's see the result of disk format.
[root@primary01 ~]# ll /dev/disk/by-path/
total 0
lrwxrwxrwx. 1 root root 9 Jul 31 15:00 ip-192.168.15.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:asm-t1-lun-1 -> ../../sdb
lrwxrwxrwx. 1 root root 10 Jul 31 15:00 ip-192.168.15.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:asm-t1-lun-1
-part1
-> ../../sdb1
lrwxrwxrwx. 1 root root 9 Jul 31 15:00 ip-192.168.15.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:asm-t2-lun-2 -> ../../sdc
lrwxrwxrwx. 1 root root 10 Jul 31 15:00 ip-192.168.15.101:3260-iscsi-iqn.2005-10.org.freenas.ctl:asm-t2-lun-2
-part1
-> ../../sdc1
...
On node 2, you don't have to format them again, just do a partprobe to get the newest status.
[root@primary02 ~]# partprobe
If the block devices are already existent before the the installation job, you should be better to clean up the header of the ASM candidate disks on any one of the two nodes.
[root@primary01 ~]# dd if=/dev/zero of=/dev/sdb1 bs=1024 count=500
500+0 records in
500+0 records out
512000 bytes (512 kB) copied, 0.0839358 s, 6.1 MB/s
[root@primary01 ~]# dd if=/dev/zero of=/dev/sdc1 bs=1024 count=500
500+0 records in
500+0 records out
512000 bytes (512 kB) copied, 0.0506696 s, 10.1 MB/s
As you can see, we cleaned 500KB of the head of the devices. If you suspect that most of the devices are dirty, you can increase the block size (bs) and the loop counts.
So far, we have two RAC nodes with ASM candidate disks ready.
Hello Ed,
I am following your guide as a basis of setting up a test Oracle RAC 19c, in vCLoud, however I am a little confused on the network configuration.
Should the RAC nodes have 4 nics in total? 3 for Public, VIP and SCAN and 1 for Private, or 2 nics, 1 public and 1 private, and VIP and SCAN being VLAN type based of the public?
2 NICs are enough, one is for public, SCAN and VIP, the other is for private. The most important thing is that you should not bind SCAN and VIP into public NIC.