Pre-requisites :

Before you install the Grid,Clusterware and the Database, you must complete several pre-installation tasks.

1) 2 Public IP’s – Needs a NIC card on the server
2) 2 Virtual IP’s – Binded with Public IP
3) 2 Private IP’s — Needs a NIC card on the server
4) SCAN IP – Does not need a NIC card however needs to resolved by DNS i.e it should have a fully qualified domain name in the hosts file as well as the DOMAIN.
5) Min 2 GB RAM on each server
6) RPM’s and kernel Parameters

7) Lun’s created on the storage and mounted on the server which will be seen as a raw device on the server
For e.g : /dev/sdb, /dev/sdc etc
8) Software for Oracle 11g Grid and Database (64 bit)
9) Access to root to create users and run post installation steps
10) 20 GB free on local hard disk for grid and database software installation

Starting the Configuration and Installation :

1) IP’s required for configuration

[root@erpuatdbs01 ~]# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
######PUBLIC IP FOR RAC#####################
10.10.160.21 erpuatdbs01.mbcgroup.local erpuatdbs01
10.10.160.22 erpuatdbs02.mbcgroup.local erpuatdbs02
##### VIP for RAC ########################
10.10.160.121 erpuatdbs01-vip.mbcgroup.local erpuatdbs01-vip
10.10.160.122 erpuatdbs02-vip.mbcgroup.local erpuatdbs02-vip
##########Private Interconnect for RAC#############
10.10.20.20 erpuatdbs01-priv.mbcgroup.local erpuatdbs01-priv
10.10.20.21 erpuatdbs02-priv.mbcgroup.local erpuatdbs02-priv
##########SCAN VIP#############################
10.10.160.23 erpuatdbs.mcbgroup.local erpuatdbs

2)Use ”free –m” to check the size of memory in Linux.
3) If using RAW DEVICES with ASM i.e without creating disks at OS level using oracleasm

##########IF USING RAWDEVICES DIRECTLY WITH ASM##############
[root@erpuatdbs01 ~]# fdisk –l
Disk /dev/sdb: 1073 MB, 1073741824 bytes --> For OCR and Voting Disk
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes
Device Boot Start End Blocks Id System --> For OCR and Voting Disk -- Redundant if needed
Disk /dev/sdc: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes
Device Boot Start End Blocks Id System --> For Database Files
Disk /dev/sdd: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System --> For Archive Logs
Disk /dev/sde: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
#########################################################
4) If using ASMLibraries for raw devices then follow the steps as stated below
<code>
#uname –r
2.6.18-194.el5

Note : Download the rpm as per your kernel version

[root@erpuatdbs02 rpm]# rpm -Uvh oracleasm*.rpm
warning: oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID
e5e0159
Preparing... ########################################### [100%]
1:oracleasm-support ########################################### [ 33%]
2:oracleasm-2.6.18-194.el########################################### [ 67%]
3:oracleasmlib ########################################### [100%]
[root@erpuatdbs02 rpm]#
[root@erpuatdbs01 ~]# rpm -qa oracleas*
oracleasmlib-2.0.4-1.el5 -- COMMON for all 64 bit OS
oracleasm-2.6.18-194.el5-2.0.5-1.el5 -- As per Kernel Ver
oracleasm-support-2.1.3-1.el5 -- COMMON for all 64 bit OS
Format the raw device to have atleast 1 partition
[root@erpuatdbs01 ~]# fdisk /dev/sdb
The number of cylinders for this disk is set to 26108.
There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with:

1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other Oss

(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-26108, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-26108, default 26108):
Using default value 26108
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

REPEAT THE SAME FOR ALL THE RAW DEVICES

[root@erpuatdbs01 ~]# fdisk –l
Device Boot Start End Blocks Id System
/dev/sdb1 1 1011 1048376+ 83 Linux
Disk /dev/sdc: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 1 1011 1048376+ 83 Linux
Disk /dev/sdd: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdd1 1 26108 209712478+ 83 Linux
Disk /dev/sde: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sde1 1 26108 209712478+ 83 Linux

Configure Oracle ASM

[root@erpuatdbs01 ~]# oracleasm configure –i
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]').
Hitting &lt;ENTER&gt; without typing an answer will keep that current value.
Ctrl-C will abort.

Default user to own the driver interface [oracrs]:
Default group to own the driver interface [oinstall]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
[root@erpuatdbs01 ~]#
[root@erpuatdbs01 oracleasm]# /usr/sbin/oracleasm init
Mounting ASMlib driver filesystem: /dev/oracleasm
[root@erpuatdbs01 etc]# /usr/sbin/oracleasm createdisk CRS /dev/sdb1
Writing disk header: done
Instantiating disk: done
[root@erpuatdbs01 etc]# /usr/sbin/oracleasm createdisk DATA /dev/sdd1
Writing disk header: done
Instantiating disk: done
[root@erpuatdbs01 etc]# /usr/sbin/oracleasm createdisk ARC /dev/sde1
Writing disk header: done
Instantiating disk: done
[root@erpuatdbs01 etc]# /usr/sbin/oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@erpuatdbs01 etc]# /usr/sbin/oracleasm listdisks
ARC
CRS
DATA

Change the permission manually on the devices

cd /dev
chown orauat:oinstall /dev/sd*
Edit /etc/rc.local
chown orauat:oinstall /dev/sd*

5) RPM’s and Kernel Parameters

#vi /etc/sysctl.conf
kernel.msgmax = 65536
# Controls the default maxmimum size of a mesage queue
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
Add below 4 lines in /etc/security/limits.conf
* soft nproc 2047
* hard nproc 16384
* soft nofile 1024
* hard nofile 65536

6) Create groups and users

#groupadd dba
#groupadd oinstall
#useradd orauat –g oinstall –G dba

7) Download the software from
http://otn.oracle.com
a) linux.x64_11gR2_grid.zip  —; For Grid and Cluster Installation
b) linux.x64_11gR2_database_1of2.zip —; For Database Home Installation
c) linux.x64_11gR2_database_2of2.zip

Starting the Grid Installation :

1)      Unzip the grid software as root user and give oracrs and orauat user to run the Installation
2)      Export DISPLAY and start the Installer
For e.g : From your local machine you should have
a)      Putty
b)      Xming software
c)       Double Click on Xming and it will set the local desktop display and will run in the background. Open putty and Enable X11 forwarding


3) Using “orauat” user start the runInstaller


We changed the SCAN-NAME to reflect as it is in our hosts file
Below diagram contains – erpuatd-cluster whereas it should be “erpuatdbs”


Click on the SSH Connectivity and Setup ssh for both nodes




IF using raw devices then it will look like






Execute orainstRoot.sh and root.sh from all the nodes
Node 1 : orainstRoot.sh

[root@erpuatdbs01 oraInventory]# ./orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@erpuatdbs01 oraInventory]#

Node 2 : orainstRoot.sh

[root@erpuatdbs02 oraInventory]# ./orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

Node 1 : root.sh


Node 2 : root.sh

Continue with the installation


CLUSTER AND ASM INSTALLATION COMPLETED. During this installation we used only one Disk i.e for OCR and Voting Disk.
Now using ASMCA we will add the the other 2 disks for DATA and ARCHIVE.
Login as orauat user

$export ORACLE_HOME=/u01/app/11.2.0/grid
$export ORACLE_SID=+ASM1
$export PATH=$PATH:$ORACLE_HOME/bin

To access ASM instance from command line

$./asmca





Exit the application and start the database home installation

Create Oracle Database Software :

Using orauat execute the runInstaller for database










Node 1 :

[root@erpuatdbs01 dbhome_1]# ./root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= orauat
ORACLE_HOME= /u02/app/oracle/product/11.2.0/dbhome_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.

Node 2 :

[root@erpuatdbs02 dbhome_1]# ./root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= orauat
ORACLE_HOME= /u02/app/oracle/product/11.2.0/dbhome_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
[root@erpuatdbs02 dbhome_1]#

Check the status of Cluster

#cd /u01/app/11.2.0/grid/bin
#./crs_stat -t
[root@erpuatdbs01 bin]# ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE erpuatdbs01
ora....ER.lsnr ora....er.type ONLINE ONLINE erpuatdbs01
ora....N1.lsnr ora....er.type ONLINE ONLINE erpuatdbs02
ora.OCR.dg ora....up.type ONLINE ONLINE erpuatdbs01
ora.asm ora.asm.type ONLINE ONLINE erpuatdbs01
ora.eons ora.eons.type ONLINE ONLINE erpuatdbs01
ora....SM1.asm application ONLINE ONLINE erpuatdbs01
ora....01.lsnr application ONLINE ONLINE erpuatdbs01
ora....s01.gsd application OFFLINE OFFLINE
ora....s01.ons application ONLINE ONLINE erpuatdbs01
ora....s01.vip ora....t1.type ONLINE ONLINE erpuatdbs01
ora....SM2.asm application ONLINE ONLINE erpuatdbs02
ora....02.lsnr application ONLINE ONLINE erpuatdbs02
ora....s02.gsd application OFFLINE OFFLINE
ora....s02.ons application ONLINE ONLINE erpuatdbs02
ora....s02.vip ora....t1.type ONLINE ONLINE erpuatdbs02
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE erpuatdbs01
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE erpuatdbs01
ora....ry.acfs ora....fs.type ONLINE ONLINE erpuatdbs01
ora.scan1.vip ora....ip.type ONLINE ONLINE erpuatdbs02

Node 1 :

[root@erpuatdbs01 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@erpuatdbs01 bin]#

Node 2 :

[root@erpuatdbs02 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@erpuatdbs02 bin]#
Tagged with →  
Share →
0 comments
Skip to toolbar