Rac Installation by Nadir Sb

Oracle 11gR2 2-Node RAC on VMware Workstation 8 This demo describes how to install Oracle 11gR2 (11.2.0.1) 2 Node (Linux) RAC on VMware Workstation-8. The hardware/software resources used in this demo are given b elow. Dell XPS-15 (Core i7) with Windows-7 (Home Premium)

64bit

RAM-16GB

8GB RAM is enough for 2 Node RAC in Virtual machine

50 GB free space for each RAC Node

Just to install 2 Node RAC, 20 GB free space on each node is enough

OEL-5.8 (64bit)

https://edelivery.oracle.com/linux https://edelivery.oracle. com/linux

VMware Workstation-8

VMware

Oracle 11.2.0.1 Grid

Grid

Oracle 11.2.0.1 Database (Part-1 & 2)

Database

Oracle ASMLib

ASMLib

Note: OEL-5.5 or advanced Linux version can be used for 11gR2 RAC installation. One advantage of using OEL-5.8 is , all required RPMs for ASMLib are already installed installed in OS.Prior to start RAC installation it is assumed that that VMware-8 and OEL-5.8 are installed in Laptop. Installation of LINUX and VMware are not covered in this document.

Create Linux Virtual Machine (Node-1) Start VMware workstation. Go to File --> New Virtual Machine

Oracle 11gR2 2-Node RAC on VMware Workstation 8

TusarRout'sOracleBlog http://gohappydba.wordpress.com/

Select "Custom" and proceed.

Click on Next.

Select option as appropriate and click Next.

2



select Linux with Version and click Next

3



Put Virtual Machine Name (RAC1) and Location to store VM files

Default No of Processor is 1, click on Next

4



Put Memory for Node-1 and click Next. I have 16GB RAM in my Laptop, so I put 3GB for Node-1. If you have 8GB RAM in Laptop, you can use 2GB (2.5 GB is preferable) for each nodes.

select "Use Bridge Networking" and click on Next

Select the default "LSI Logic" as I am using SCSI disk in my Laptop. 5



Create new virtual disk

Select SCSI and proceed. 6



Put Disk size and select the below option. I used 50GB for each RAC node (for future use , to stage Grid/ RDBMS software), but 20GB (10GB for Linux and 10GB for Grid+Database is enough for each RAC node just for installation purpose.

Put Disk name (file name which will be used a disk). By default it takes Node_Name.vmdk (RAC1.vmdk)

7



Verify Installation details and click on Finish to start Linux OS installation.

After successful installation of Linux OS on VM Node-1, it displays as given below. Start RAC1 by clicking on Power on button: 8



Check required RPMs and additional Setup for Oracle 11gR2 on OEL-5 Most of the RPMs are installed with OEL OS installation.

binutils-2.17.50.0.6 compat-libstdc++-33-3.2.3 compat-libstdc++-33-3.2.3 (32 bit) elfutils-libelf-0.125 elfutils-libelf-devel-0.125 gcc-4.1.2 gcc-c++-4.1.2 glibc-2.5-24 glibc-2.5-24 (32 bit) glibc-common-2.5 glibc-devel-2.5 glibc-devel-2.5 (32 bit) glibc-headers-2.5 ksh-20060214 libaio-0.3.106 libaio-0.3.106 (32 bit) libaio-devel-0.3.106 libaio-devel-0.3.106 (32 bit) libgcc-4.1.2 libgcc-4.1.2 (32 bit) libstdc++-4.1.2 libstdc++-4.1.2 (32 bit) libstdc++-devel 4.1.2 make-3.81 sysstat-7.0.2 compat-libstdc++-33-3.2.3 compat-libstdc++-33-3.2.3 (32 bit) elfutils-libelf-devel-0.125 glibc-devel-2.5 libaio-devel-0.3.106 libaio-devel-0.3.106 (32 bit) sysstat-7.0.2 unixODBC-2.2.11 unixODBC-2.2.11 (32 bit) unixODBC-devel-2.2.11 unixODBC-devel-2.2.11 (32 bit)

9



Check RPM is installed or not with below c ommand.

[root@rac1 ~]# rpm -qa | grep inutils-2.* binutils-2.17.50.0.6-20.el5 [root@rac1 ~]#

To install RPM ,use the below command with root user. All the RPMs are available in Server directory under Linux media-CD/DVD. Also many of the RPMs can be downloaded from http://rpm.pbone.net/

[root@rac1 ~]# rpm -ivh unixODBC-2.2.11-10.el5.x86_64.rpm

Check kernel parameters for Oracle as given below. Please note that any parameter value which is higher than below mentioned value shouldn't be modified. If the value is less than as mentioned below, then needs to be modified. Edit /etc/sysctl.conf as root user. Use “ /sbin/sysctl -p” command to apply the new settings.

fs.aio-max-nr = 1048576 fs.file-max = 6815744 kernel.shmall = 2097152 kernel.shmmax = 1054504960 kernel.shmmni = 4096 # semaphores: semmsl, semmns, semopm, semmni kernel.sem = 250 32000 100 128 net.ipv4.ip_local_port_range = 9000 65500 net.core.rmem_default=262144 net.core.rmem_max=4194304 net.core.wmem_default=262144 net.core.wmem_max=1048586

Edit the /etc/pam.d/login file and add following line:

[root@rac1 ~]# cat /etc/pam.d/login |grep pam_limits.so session required /lib/security/pam_limits.so

Edit the /etc/security/limits.conf file and add following lines:

oracle oracle oracle oracle

soft hard soft hard

nproc nproc nofile nofile

2047 16384 1024 65536

Change the setting of SELinux to permissive/disabled by editing the " /etc/selinux/config" file

[root@rac1 ~]# cat /etc/sysconfig/selinux |grep SELINUX= SELINUX=disabled [root@rac1 ~]# /usr/sbin/getenforce Disabled

Disable the Linux firewall. First stop the services. To p ermanently disable the firewall, use the second command. 10



[root@rac1 ~]# service iptables stop Flushing firewall rules: [ OK ] Setting chains to policy ACCEPT: filter [ Unloading iptables modules: [ OK ]

OK

]

[root@rac1 ~]# chkconfig iptables off

Create oracle user, OS groups and directories where Oracle software will be installed. [root@rac1 [root@rac1 [root@rac1 [root@rac1 [root@rac1

~]# ~]# ~]# ~]# ~]#

groupadd dba groupadd oinstall groupadd asmdba groupadd asmadmin useradd -g oinstall -G dba,asmdba,asmadmin oracle

Set password for user: oracle

[root@rac1 ~]# passwd oracle

Create directory where oracle software will be i nstalled and set the ownership.

[root@rac1 [root@rac1 [root@rac1 [root@rac1

~]# ~]# ~]# ~]#

mkdir mkdir chown chmod

-p /oracle/11.2.0/grid -p /oracle/11.2.0/product oracle:dba /oracle/11.2.0/ -R 775 /oracle/11.2.0

Login as the "oracle" user and add the following lines at the end of the "/home/oracle/.bash_profile" file.

# Oracle Settings TMP=/tmp; export TMP TMPDIR=$TMP; export TMPDIR if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi fi

ASMLib Setup The below RPMs are required for ASMLib. As we are using OEL-5.8 with kernel version: 2.6.32-300.10.1.el5uek, there is no separate oracleasmlib* RPM available. The ASMLib kernel module is build with EUK kernel, so no need to install that RPM to make ASM working without any issue. More information is available @ http://sethmiller.org/it/oracleasmlibnot-necessary/   

oracleasm-support-2.1.7-1.el5.x86_64.rpm oracleasm-2.6.18-194.el5-2.0.5-1.el5.x86_64.rpm oracleasmlib-2.0.4-1.el5.x86_64.rpm --> Not required in OEL-5.8 and higher OS version 11



[root@rac1 ~]# rpm -qa |grep oracleasm* oracleasm-support-2.1.8-1.el5 oracleasm-2.6.18-348.12.1.el5-2.0.5-1.el5 [root@rac1 ~]# rpm -qa |grep oracleasmlib* [root@rac1 ~]#

Configure the ASMLib driver (to be owned by oracle and the dba group). It needs to be loaded on every reboot.

[root@rac1 ~]# /etc/init.d/oracleasm configure Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface [ oracle ]: Default group to own the driver interface [ dba]: Start Oracle ASM library driver on boot (y/n) [ y]: Scan for Oracle ASM disks on boot (y/n) [ y]: Writing Oracle ASM library driver configuration: done Initializing the Oracle ASMLib driver: Scanning the system for Oracle ASMLib disks:

[ [

OK OK

] ]

[ [

OK OK

] ]

Start the ASMLib driver.

[root@rac1 ~]# /etc/init.d/oracleasm start Initializing the Oracle ASMLib driver: Scanning the system for Oracle ASMLib disks:

Create shared disk to be used by ASM ( Disk groups) Shutdown RAC1. Right click on RAC1 --> Settings --> Click on "Add"

12



Select Hard Disk and click on Next

Select "Create a new virtual disk" 13



Select SCSI, Independent and Persistent options.

Put the size of Disk. I put 10GB for d oing some testing on Disk group. Otherwise 5GB is enough for demo purpose.

14



Put a Disk file name and place the file in a separate location (C:\TUSAR\VMWARE\RACSHARE) because this disk will be shared with RAC2 node. Click Finish to c reate the disk. It takes few minutes to create the disk.

New Virtual Disk will be displayed as given below. Click on "Advanced"

15



Select device node as SCSI 1:0 from drop down menu (which means SCSI controller 1, device 0) and press OK.

Now we have to modify VMware hardware profile file (vmx) for RAC1 to make this disk as shared (in cluster).These changes force VM not try to buffer reads and writes to the disk directly.Take a backup of the below file before modifying.

16



We need to put the below lines in rac1.vmx file and save it. If any of the parameter is already present , don't put it twice (it will throw error during VM startup)

disk.locking = "FALSE" diskLib.dataCacheMaxSize = "0" diskLib.dataCacheMaxReadAheadSize = "0" diskLib.dataCacheMinReadAheadSize = "0" diskLib.dataCachePageSize = "4096" diskLib.maxUnsyncedWrites = "0" scsi1.sharedBus = "virtual"

The new disk (ASM_DISK1) has the following entries. No need to change anything with below entries.

scsi1:0.present = "TRUE" scsi1:0.fileName = "C:\TUSAR\VMWARE\RACSHARE\ASM_DISK1" scsi1:0.writeThrough = "TRUE" scsi1:0.mode = "independent-persistent" scsi1:0.deviceType = "disk" scsi1:0.redo = ""

Start RAC1 VM to format new disk. Use "fdisk" command to partition new disk (/dev/sdb).

[root@rac1 ~]# fdisk /dev/sdb Command (m for help): u Changing display/entry units to sectors Command (m for help): n Command action e extended p primary partition (1-4)

p Partition number (1-4): 1 First sector (63-10485759, default 63): 2048

17



Last sector or +size or +sizeM or +sizeK (2048-10485759, default 10485759): Using default value 10485759 Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.

After partitioning new disk (/dev/sdb1), you will find an entry like below with command "fdisk -l"

Disk /dev/sdb: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot /dev/sdb1

Start 1

End 1306

Blocks 10484736

Id 83

System Linux

Now we can use this d isk for ASM. Use the below command to create ASM disk.

[root@rac1 ~]# oracleasm createdisk DATA /dev/sdb1 Writing disk header: done Instantiating disk: done [root@rac1 ~]# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks...

Verify the ASM disks.

[root@rac1 ~]# oracleasm listdisks DATA

Shutdown RAC1 node to configure Network interfaces. Taking backup of RAC1 is a good idea here as we are done with all setups except Network Interface.

Configure Network Interfaces The Oracle RAC requires at least two network c onnections between cluster nodes. One network will be the public IP and the second will be a private IP reserved for inter-cluster traffic. In VMware Workstation, Go to Edit->Virtual Network Editor.

18



This allows us to create new virtual networks that allow our guest VMs to talk amongst themselves, to the host computer and/or to the outside world. We are going to create two virtual networks (VMnet2 and VMnet3) by clicking on Add Network. For both VMnet2 and VMnet3 DHCP is enabled. VMnet2 will be a host-only network allowing communication to other VMs and the host, but not to the outside world. We will assign a IP of 10.10.1.0 to VMnet2, and a Subnet mask of 255.255.255.0 VMnet3 will also be a host-only network allowing communication to other VMs and the host, but not to the outside world. We will assign a Subnet IP of 10.10.2.0 to VMnet3, and a Subnet mask of 255.255.255.0

Now we have our two networks created in VMware, we will add two new NICs to our VM.

Right click on the RAC1 in the VMware W orkstation menu, and select Settings. This will bring up the Virtual Machine Settings panel. Now click Add. The Add Hardware Wizard allows us to select Network Adapter

19



On the Network Adapter Type menu, we will select the Custom: Sp ecific virtual network radio button, and use the drop down to select VMNet2(Host-only). Click Finish to create the NIC.

Now repeat the above steps to add another Network Adapter, this time using VMNet3.

20



Now repeat the above steps to add another Network Adapter, this time using VMNet3.The final VM hardware configuration should look like this:

21



Now we can clone our VM (RAC1) which will be RAC2. From this point forward we will have two mac hines i.e. RAC1 and RAC2.As RAC1 is now shutdown, go to VM --> Manage --> Clone.

Select Create a full clone, put VM name: RAC2 and Location to store VM files. Click Finish to start clone.

22



After successful clone, we we have two VM i.e. i.e. RAC1 and RAC2. Now start both VM and login as root.Now we need to configure Network IP add, hostname. Log in to Node-1 (RAC1) as (RAC1) as root and select System->Administration->Network System->Administration->Network.. There will be three network interfaces (devices) as given below.   

eth0 : The bridged network adapter that connects us to the outside world (no configuration required) eth1 : eth1 : It is using VMnet2, the network we intend to be our public RAC network eth2 : eth2 : It is using VMnet3 which is what we plan to use for private cluster traffic

Select the eth1 eth1 adapter adapter and click the edit button to b ring up the Ethernet Device control panel. We will select “Statically set IP addresses” to assign a static IP address to this NIC. We will assign IP address 10.10.1.10. Make sure that “Activate device when computer starts” remains e nabled.

Now repeat the above steps for eth2 eth2,, this time statically assigning the IP address 10.10.2.10

23



Make sure you save your chances before closing the Network Configuration editor window.

24



[root@rac1 ~]# /etc/init.d/network restart Shutting down interface eth0: Shutting down interface eth1: Shutting down interface eth2: Shutting down loopback interface: Bringing up loopback interface: Bringing up interface eth0: Determining IP information for eth0... done. Bringing up interface eth1: Bringing up interface eth2:

[ [ [ [ [

OK OK OK OK OK

] ] ] ] ]

[ [ [

OK OK OK

] ] ]

Check with "ifconfig -a" command in RAC1

[root@ rac1 ~]# ifconfig -a eth0 Link encap:Ethernet HWaddr 00:0C:29:74:C4:4D inet addr:192.168.0.103 Bcast:192.168.0.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:120 errors:0 dropped:0 overruns:0 frame:0 TX packets:137 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:30398 (29.6 KiB) TX bytes:29031 (28.3 KiB) eth1

Link encap:Ethernet HWaddr 00:0C:29:74:C4:57 inet addr:10.10.1.10 Bcast:10.10.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:384 errors:0 dropped:0 overruns:0 frame:0 TX packets:542 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:88679 (86.6 KiB) TX bytes:104687 (102.2 KiB)

eth2

Link encap:Ethernet HWaddr 00:0C:29:74:C4:61 inet addr:10.10.2.10 Bcast:10.10.2.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8632 errors:0 dropped:0 overruns:0 frame:0 TX packets:7619 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5275335 (5.0 MiB) TX bytes:2998636 (2.8 MiB)

lo

Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:22178 errors:0 dropped:0 overruns:0 frame:0 TX packets:22178 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:10344538 (9.8 MiB) TX bytes:10344538 (9.8 MiB)

Now log in to RAC2 and follow the above steps for IP Add configuration. Once this is d one, we have the below n/w interfaces ready with following IP Add.

N/W Interfaces

RAC1

RAC2

eth1

10.10.1.10

10.10.1.20

eth2

10.10.2.10

10.10.2.20

25



Check hostname in the file /etc/sysconfig/network with root account, edit this file and change the host name on each machine. If we change hostname in this file, we need to reboot VM. My domain name is "miracle.com" (Oracle's Miracle) RAC1:

[root@rac1 ~]# cat /etc/sysconfig/network NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=rac1.miracle.com

RAC2:

[root@rac2 ~]# cat /etc/sysconfig/network NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=rac2.miracle.com

Check with "ifconfig -a" command in RAC2

[root@ rac2 ~]# ifconfig -a eth0 Link encap:Ethernet HWaddr 00:0C:29:04:BC:50 inet addr:192.168.0.104 Bcast:192.168.0.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:188 errors:0 dropped:0 overruns:0 frame:0 TX packets:142 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:47953 (46.8 KiB) TX bytes:33374 (32.5 KiB)

eth1

Link encap:Ethernet HWaddr 00:0C:29:04:BC:5A inet addr:10.10.1.20 Bcast:10.10.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:999 errors:0 dropped:0 overruns:0 frame:0 TX packets:718 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:196608 (192.0 KiB) TX bytes:175282 (171.1 KiB)

eth2

Link encap:Ethernet HWaddr 00:0C:29:04:BC:64 inet addr:10.10.2.20 Bcast:10.10.2.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:14402 errors:0 dropped:0 overruns:0 frame:0 TX packets:15877 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5795575 (5.5 MiB) TX bytes:10262372 (9.7 MiB)

lo

Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:36301 errors:0 dropped:0 overruns:0 frame:0 TX packets:36301 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:11833654 (11.2 MiB) TX bytes:11833654 (11.2 MiB)

It's a good idea to bounce both VM (RAC1 and RAC2) and check if everything back with above as expected. Please note, now onwards whatever changes we do , it mean for both the nodes (RAC1 and RAC2) ac cordingly.

26



Configure DNS for using SCAN Grid instiller demands a VIP to be used as SCAN VIP (for SCAN listener). It doesn't allow to proceed with installation without providing the VIP (SCAN). As a work around, we c an place VIP (which must be free and is not used in the network) in /etc/hosts files in both the nodes to make the installer continue with Grid installation. Later you can use local listener instead of SCAN (disable) and remove these entries from /etc/hosts. In our demo, we are going to use SCAN listener (which is an added benefit to handle user connections irrespective of number of nodes present in a cluster. For using SCAN, we need to configure DNS to resolve VIP addresses to be used by SCAN. Oracle recommends three VIP should be enough for SCAN to handle multiple number of database connection. Here we are using two VIP for SCAN (just for demo purpose, later we can add one more VIP to SCAN). We are using SCAN VIP, still I suggest to place SCAN VIP entries in /etc/hosts during installation, later we can remove these SCAN entries.

[root@ rac1 ~]# cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 10.10.1.10 rac1.miracle.com rac1 10.10.1.20 rac2.miracle.com rac2 10.10.1.11 rac1-vip.miracle.com rac1-vip 10.10.1.21 rac2-vip.miracle.com rac2-vip 10.10.2.10 rac1-priv.miracle.com rac1-priv 10.10.2.20 rac2-priv.miracle.com rac2-priv

10.10.1.12 rac-scan.miracle.com rac-scan 10.10.1.22 rac-scan.miracle.com rac-scan

We will be using these VIPs for SCAN and these need to be configured in DNS.

[root@ rac2 ~]# cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 10.10.1.10 rac1.miracle.com rac1 10.10.1.20 rac2.miracle.com rac2 10.10.1.11 rac1-vip.miracle.com rac1-vip 10.10.1.21 rac2-vip.miracle.com rac2-vip 10.10.2.10 rac1-priv.miracle.com rac1-priv 10.10.2.20 rac2-priv.miracle.com rac2-priv

10.10.1.12 rac-scan.miracle.com rac-scan 10.10.1.22 rac-scan.miracle.com rac-scan

N/W Interfaces

RAC1

RAC2 27



Public IP

10.10.1.10

10.10.1.20

Private IP

10.10.2.10

10.10.2.20

VIP IP

10.10.1.11

10.10.1.21

SCAN IP

10.10.1.12

10.10.1.22

The below RPM needs to be installed to start DNS in Linux Server. It is already a vailable in OEL-5.8.

[root@rac1 ~]# rpm -qa |grep bind-* bind-9.3.6-20.P1.el5

For DNS setup, we need to create file /etc/named.conf with below entries. If the file is already p resent, then before editing take a backup.

[root@ rac1 ~]# cat /etc/named.conf options { directory "/var/named"; dump-file "/var/named/data/cache_dump.db"; statistics-file "/var/named/data/named_stats.txt"; forwarders { 192.168.0.1; }; }; include "/etc/rndc.key"; zone "miracle.com" IN { type master; file "miracle.zone"; allow-update { none; }; };

In real scenario, DNS server holds this file. In our case you can place this file in both RAC1 and RAC2 (with same entries) as given below. It helps to operate both the nodes as DSN server.

[root@ rac2 ~]# cat /etc/named.conf options { directory "/var/named"; dump-file "/var/named/data/cache_dump.db"; statistics-file "/var/named/data/named_stats.txt"; forwarders { 192.168.0.1 ; }; }; include "/etc/rndc.key"; zone "miracle.com" IN { type master; file "miracle.zone"; allow-update { none; }; };

Change forwarders IP as per your Tertiary DNS (In my case it is 192.168.0.1). Go to the below location to check what is the Tertiary DNS. Generally it is 192.168.1.1 in most of the home networks.. 28



System->Administration->Network

Create file /var/named/miracle.zone with following entries in both nodes. RAC1:

[root@ rac1 ~]# cat /var/named/miracle.zone $TTL 86400 @ IN SOA miracle.com. miracle.com.( 42 ; serial (d. adams) 3H ; refresh 15M ; retry 1W ; expiry 1D ) ; minimum miracle.com. localhost rac1.miracle.com. rac2.miracle.com. rac1-vip.miracle.com. rac2-vip.miracle.com. rac-scan.miracle.com. rac-scan.miracle.com.

IN IN IN IN IN IN IN IN

NS A A A A A A A

10.10.1.20 127.0.0.1 10.10.1.10 10.10.1.20 10.10.1.11 10.10.1.21 10.10.1.12 10.10.1.22

RAC2:

29



[root@ rac2 ~]# cat /var/named/miracle.zone $TTL 86400 @ IN SOA miracle.com. miracle.com.( 42 ; serial (d. adams) 3H ; refresh 15M ; retry 1W ; expiry 1D ) ; minimum miracle.com. localhost rac1.miracle.com. rac2.miracle.com. rac1-vip.miracle.com. rac2-vip.miracle.com. rac-scan.miracle.com. rac-scan.miracle.com.

IN IN IN IN IN IN IN IN

NS A A A A A A A

10.10.1.20 127.0.0.1 10.10.1.10 10.10.1.20 10.10.1.11 10.10.1.21 10.10.1.12 10.10.1.22

RAC1: Put the below entries in /etc/resolve.conf file

[root@ rac1 ~]# cat /etc/resolv.conf ; generated by /sbin/dhclient-script nameserver 10.10.1.10 # rac1 DNS server nameserver 10.10.1.20 # rac2 DNS server nameserver 192.168.0.1 # Primary DNS in the domain search miracle.com # Local Domain

RAC2:

[root@ rac2 ~]# cat /etc/resolv.conf ; generated by /sbin/dhclient-script nameserver 10.10.1.10 # rac1 DNS server nameserver 10.10.1.20 # rac2 DNS server nameserver 192.168.0.1 # Primary DNS in the domain search miracle.com # Local Domain

Now we have to modify Ethernet adapter files to avoid any overwrite of /etc/resolve.conf file.

[root@rac1 network-scripts]# pwd

/etc/sysconfig/network-scripts [root@rac1 network-scripts]# ls -lrt ifcfg-eth* -rw-r--r-- 3 root root 216 Jul 27 10:45 ifcfg-eth2 -rw-r--r-- 3 root root 216 Jul 27 10:45 ifcfg-eth1 -rw-r--r-- 3 root root 153 Jul 27 10:45 ifcfg-eth0

Place "PEERDNS=no" in both ifcfg-eth1 and ifcfg-eth2 files (both nodes) to avoid any overwrite of /etc/resolve.conf

[root@ rac1 network-scripts]# pwd

/etc/sysconfig/network-scripts [root@rac1 network-scripts]# cat ifcfg-eth1 # Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)

30



DEVICE=eth1 BOOTPROTO=none ONBOOT=yes HWADDR=00:0c:29:74:c4:57 NETMASK=255.255.255.0 IPADDR=10.10.1.10 TYPE=Ethernet USERCTL=no IPV6INIT=no PEERDNS=no [root@ rac1 network-scripts]# cat ifcfg-eth2 # Intel Corporation 82545EM Gigabit Ethernet Controller (Copper) DEVICE=eth2 BOOTPROTO=none ONBOOT=yes HWADDR=00:0c:29:74:c4:61 NETMASK=255.255.255.0 IPADDR=10.10.2.10 TYPE=Ethernet USERCTL=no IPV6INIT=no PEERDNS=no

Now we can start DNS s ervice on both nodes.

[root@ rac1 ~]# service named start Starting named:

[

OK

]

[root@ rac2 ~]# service named start Starting named:

[

OK

]

We also need to set the DNS service to auto-start on reboot (both nodes)

[root@ rac1 ~]# chkconfig named on [root@ rac2 ~]# chkconfig named on

Now we can do look up the SCAN IP addresses. Now check whether rac-scan can resolve the IP add from DNS with round-robin format (on both nodes).

[root@rac1 network-scripts]# nslookup rac-scan Server: 10.10.1.10 Address: 10.10.1.10#53 Name: rac-scan.miracle.com Address: 10.10.1.22 Name: rac-scan.miracle.com Address: 10.10.1.12

--> 1st SCAN VIP --> 2nd SCAN VIP

31







Also check nslookup for other RAC IP add (on both nodes)

[root@rac1 network-scripts]# nslookup rac1 Server: 10.10.1.10 Address: 10.10.1.10#53 Name: rac1.miracle.com Address: 10.10.1.10 [root@rac1 network-scripts]# nslookup rac1-vip Server: 10.10.1.10 Address: 10.10.1.10#53 Name: rac1-vip.miracle.com Address: 10.10.1.11 [root@rac1 network-scripts]# nslookup rac2 Server: 10.10.1.10 Address: 10.10.1.10#53 Name: rac2.miracle.com Address: 10.10.1.20 [root@rac1 network-scripts]# nslookup rac2-vip Server: 10.10.1.10 Address: 10.10.1.10#53 Name: rac2-vip.miracle.com Address: 10.10.1.21

It confirms that DNS is working fine, so SCAN can used as expected.

Setup NTP service to synchronize time between our cluster nodes Check if NTP service is running. 32



[root@rac1 ~]# pgrep ntpd 4489

If it is running we need to stop for few adjustments (both nodes)

[root@rac1 ~]# service ntpd stop

Backup /etc/ntp.conf and modify as given below(both nodes) RAC1:

[root@ rac1 ~]# cat /etc/ntp.conf # --- GENERAL CONFIGURATION ---

server server fudge

rac1 rac2 rac1 stratum 10

# Drift file. driftfile /etc/ntp/drift

RAC2:

[root@ rac2 ~]# cat /etc/ntp.conf # --- GENERAL CONFIGURATION ---

server server fudge

rac1 rac2 rac2 stratum 10

# Drift file. driftfile /etc/ntp/drift

change the ownership of the /etc/ntp directory:

[root@rac1 etc]# chown ntp:ntp /etc/ntp [root@rac1 etc]# ls -lrt |grep drw |grep ntp drwxr-xr-x 2 ntp 4096 Jul 9 20:24 ntp ntp

Add the slewing option (it prevents the NTP daemon from resetting the clock in the event of a gap occurring).Modify /etc/sysconfig/ntpd file and add the -x option.

[root@rac1 etc]# cat /etc/sysconfig/ntpd # Drop root to id 'ntp:ntp' by default. OPTIONS=" -x -u ntp:ntp -p /var/run/ntpd.pid" # Set to 'yes' to sync hw clock after successful ntpdate SYNC_HWCLOCK=no # Additional options for ntpdate

33



NTPDATE_OPTIONS=""

Restart NTP service.

[root@rac1 etc]# service ntpd start Starting ntpd: [ OK ]

Check NTP services on both nodes.

[root@ rac1 ~]# ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================== rac1.miracle.co .INIT. 16 u - 1024 0 0.000 0.000 0.000 rac2.miracle.co .INIT. 16 u 62 1024 0 0.000 0.000 0.000 [root@ rac2 ~]# ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================== rac1.miracle.co .INIT. 16 u 78 1024 0 0.000 0.000 0.000 rac2.miracle.co .INIT. 16 u - 1024 0 0.000 0.000 0.000

Make NTP service auto-start on reboot (both nodes)

[root@rac1 ~]# chkconfig ntpd on [root@rac2 ~]# chkconfig ntpd on

Shared disk setup in Node-2 (RAC2) We have already created the shared disk and configured that disk to be used by ASM in RAC1. After cloning of RAC2 (from RAC1), this shared disk must be visible to RAC2. In case the partition table is not updated in RAC2, then we need to do partition the same disk in RAC2 to update the partition table. Check partition of new disk (/dev/sdb1) is visible in RAC2 with command "fdisk -l".

Disk /dev/sdb: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot /dev/sdb1

Start 1

End 1306

Blocks 10484736

Id 83

System Linux

If it is not visible then we ha ve to do partition of /dev/sdb once again in RAC2.

[root@rac2 ~]# fdisk /dev/sdb

34



Command (m for help): u Changing display/entry units to sectors Command (m for help): n Command action e extended p primary partition (1-4)

p Partition number (1-4): 1 First sector (63-10485759, default 63): 2048 Last sector or +size or +sizeM or +sizeK (2048-10485759, default 10485759): Using default value 10485759 Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.

Please note if and only if the partition is not reflected on RAC2, then do the partition as given above. In general it reflects on RAC2 as it is being c opied from RAC1 (clone).

Verify the ASM disks in RAC2.

[root@rac2 ~]# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... [root@rac2 ~]# oracleasm listdisks DATA

Now the shared disk is visible to both RAC1 and RAC2.

Install the Oracle Grid Infrastructure software Log in with oracle user, copy downloaded software (Grid and Database) in Node-1 (RAC1) and unzip with user "oracle"

[oracle@rac1 oracle]$ ls -lrt *.zip -rwxr-xr-x 1 oracle oinstall 1052897657 Jul 10 19:24 linux.x64_11gR2_grid.zip -rwxr-xr-x 1 oracle oinstall 1239269270 Jul 12 11:27 linux.x64_11gR2_database_1of2.zip -rwxr-xr-x 1 oracle oinstall 1111416131 Jul 12 11:29 linux.x64_11gR2_database_2of2.zip [oracle@rac1 oracle]$ ls -lrt grid total 40 -rwxrwxr-x 1 oracle oinstall 3795 Jan 29 2009 runcluvfy.sh -rwxr-xr-x 1 oracle oinstall 3227 Aug 15 2009 runInstaller drwxrwxr-x 2 oracle oinstall 4096 Aug 15 2009 rpm drwxr-xr-x 4 oracle oinstall 4096 Aug 15 2009 install drwxr-xr-x 14 oracle oinstall 4096 Aug 15 2009 stage drwxrwxr-x 2 oracle oinstall 4096 Aug 15 2009 response drwxrwxr-x 2 oracle oinstall 4096 Aug 15 2009 sshsetup drwxr-xr-x 9 oracle oinstall 4096 Aug 16 2009 doc

35



-rw-r--r--

1 oracle oinstall 4228 Aug 18

2009 welcome.html

Before starting with Grid installation, we need to setup SSH on both the nodes. In 10g CRS installation, we use to do SSH setup manually in all the nodes, but in 11gR2 Grid, the following script will setup SSH on both nodes with running from one node. Log in to RAC1 as oracle user and open a command terminal, run the below script. RAC1:

[oracle@rac1 ~]$ cd /oracle/grid/sshsetup [oracle@rac1 sshsetup]$ ./sshUserSetup.sh \

-user oracle -hosts "rac1 rac2" \ -noPromptPassphrase -confirm -advanced

It will prompt twice for oracle password for each node (RAC1, RAC2) and will setup SSH on both nodes. After successful run of this script, check ssh on both nodes. RAC1:

[oracle@rac1 ~]$ ssh -x Sun Jul 27 15:50:39 IST [oracle@rac1 ~]$ ssh -x Sun Jul 27 15:50:43 IST [oracle@rac1 ~]$ ssh -x Sun Jul 27 15:50:45 IST [oracle@rac1 ~]$ ssh -x Sun Jul 27 15:50:48 IST

-l oracle 2014 -l oracle 2014 -l oracle 2014 -l oracle 2014

rac1 date

-l oracle 2014 -l oracle 2014 -l oracle 2014 -l oracle 2014

rac1 date

rac2 date rac1-priv date rac2-priv date

RAC2:

[oracle@rac2 ~]$ ssh -x Sun Jul 27 15:53:16 IST [oracle@rac2 ~]$ ssh -x Sun Jul 27 15:53:21 IST [oracle@rac2 ~]$ ssh -x Sun Jul 27 15:53:26 IST [oracle@rac2 ~]$ ssh -x Sun Jul 27 15:53:30 IST

rac1-priv date rac2 date rac2-priv date

Now run runcluvfy.sh in RAC1 and co nfirm all the settings are fine before installation of Grid.

[oracle@rac1 oracle]$ cd /oracle/grid [oracle@rac1 grid]$ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2

The output of above script is attached here.

36



Before Grid installation check the Display setting in terminal. VNC, Hummingbird or any X-Windows system can be used to access the server terminal to start the runinstaller. Run "xclock" in terminal and check the Display setting is correct or not. If you find a clock as given below, that it confirms that Display setting is fine.

It looks all the setup is fine, so we c an proceed with Grid installation.

[oracle@rac1 grid]$ ./runInstaller &

It will start the graphical installer as given below.

37



Cluster Name:

rac-cluster

Scan Name:

rac-scan

SCAN Port:

1561

38



Click Add to include second node hostname, vip.

39



40



Change Discovery Path: /dev/oracleasm/disks 41



Put account passwords (SYS, SYSASM).

42



43



44



45



Run orainstRoot.sh in RAC1, after finish up then run orainstRoot.sh in RAC2. Then run root.sh in RAC1 and after finish up, run root.sh in RAC2.Please note, run all the scripts one b y one and after success ful run of all scripts press OK.

46



The output of these scripts are attached here. Verify the output and c onfirm that there is no error.

The Grid install is now complete.

Install Oracle Database 11gR2 software 47



Unzip Database software with oracle user in RAC1.

[oracle@rac1 /oracle [oracle@rac1 -rwxr-xr-x 1 -rwxr-xr-x 1

oracle]$ pwd oracle]$ ls -lrt *database*zip oracle oinstall 1239269270 Jul 12 11:27 linux.x64_11gR2_database_1of2.zip oracle oinstall 1111416131 Jul 12 11:29 linux.x64_11gR2_database_2of2.zip

[oracle@rac1 oracle]$ cd database/ [oracle@rac1 database]$ ls -lrt total 36 -rwxr-xr-x 1 oracle oinstall 3226 drwxr-xr-x 2 oracle oinstall 4096 drwxr-xr-x 4 oracle oinstall 4096 drwxr-xr-x 14 oracle oinstall 4096 drwxrwxr-x 2 oracle oinstall 4096 drwxrwxr-x 2 oracle oinstall 4096 drwxr-xr-x 12 oracle oinstall 4096 -rw-r--r-- 1 oracle oinstall 5402

Aug Aug Aug Aug Aug Aug Aug Aug

15 15 15 15 15 15 17 18

2009 2009 2009 2009 2009 2009 2009 2009

runInstaller rpm install stage sshsetup response doc welcome.html

Run runinstaller to start Database installation. [oracle@rac1 database]$ ./runInstaller &

48



49



50



51



52



53



54



55



56



57



58



59



run root.sh in RAC1 and RAC2 one by one.

60



Oracle RAC 11gR2 Database is now installed in two nodes.

Check status of Grid services in both nodes.

[root@rac1 ~]# /oracle/11.2.0/grid/bin/crs_stat -t -v Name Type R/RA F/FT Target State Host ---------------------------------------------------------------------ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1 ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1 ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac1 ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1 ora.eons ora.eons.type 0/3 0/ ONLINE ONLINE rac1 ora.gsd ora.gsd.type 0/5 0/ ONLINE ONLINE rac1 ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1 ora.oc4j ora.oc4j.type 0/5 0/0 ONLINE ONLINE rac1 ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1 ora.prod.db ora....se.type 0/2 0/1 ONLINE ONLINE rac1 ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1 ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1 ora.rac1.gsd application 0/5 0/0 ONLINE ONLINE rac1 ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1 ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1 ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2 ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2 ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2 ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2 ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2 ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac1 [root@rac2 ~]# /oracle/11.2.0/grid/bin/crs_stat -t -v

61


Rac Installation by Nadir Sb

Recommend Documents