Oracle 11g Rac On Multipath

ORACLE 11G R2 RAC
install
On Multipath Environment
Author
Creation Date
2010-12-27
Last Updated
2011-08-09
Version
1.0
Copyright(C) 2004 Goodus Inc.

All Rights Reserved
Version
()
2010-12-27
1.2
2011-08-09
ORACLE 11g R2 RAC install ON Multipath Environment

Contents
1.
. .................................................................................................................................... 4
2.
System Overview ................................................................................................................. 4
3.
2.1.
Hardware ...................................................................................................4
2.2.
Network ....................................................................................................5
2.3.
........................................................................................................5
2.4.
H/W and OS Information ..........................................................................................8

2.4.1.
Hardware ..................................................................................................................... 8
2.4.2.
OS ................................................................................................................................ 8
2.4.3.
2.4.4.
Partition .............................................................................................................. 9
ASM ......................................................................................................... 12
2.5.
IP ............................................................................................................. 13
2.6.
OS USER ......................................................................................................... 13
2.7.
..................................................................................................................... 13
Pre-Installation Steps ......................................................................................................... 15

3.1.
3.2.
3.3.
Preparing the system ............................................................................................. 15

3.1.1.
Host file Configuration ............................................................................................. 15
3.1.2.
Kernel Sysctl Configuration ...................................................................................... 16
3.1.3.
User resource limit .......................................................................................... 16
3.1.4.
NTP ................................................................................................................... 17
3.1.5.
User Shell Configuration .......................................................................................... 18
3.1.6.
.................................................................................................................. 19
3.1.7.
SSH ................................................................................................................... 21
Preparing Oracle ASM ............................................................................................ 25

3.2.1.
Installation ASM LIBRARY ......................................................................................... 25
3.2.2.
ASM Configuration ................................................................................................... 26
3.2.3.
Multipath ................................................................................................. 27
3.2.4.
Create ASM VOLUME. .............................................................................................. 28
Install requirement ......................................................................................... 33

3.3.1.
4.
cluster verify utility ................................................................................................... 33
Installation step ............................................................................................................... 46

4.1.
Grid Infrastracture Install .................................................................................... 46

4.1.1.
runInstaller ................................................................................................................. 46
4.2. Install .................................................................................................... 52
4.2.1.
Resource Service ....................................................................................... 52
- 2

.

4.2.2.
network ............................................................................................................. 53
4.2.3.
Process ............................................................................................................ 57
4.3. Database Installation ........................................................................................... 57
4.3.1.
ASMCA ...................................................................................................................... 57
4.3.2.
Database Software Installation ................................................................................. 59
4.3.3.
Database .......................................................................................................... 62
4.4. Install .................................................................................................... 65
4.4.1.
Resource Service ....................................................................................... 65
5.
................................................................................................................................... 68
- 3

.

1.
.
Database oracle
.
Storage : storage RAID (0+1 )

Storage Network : Mpath , MPIO, PVLINKS, MPxIO, PowerPATH, HDLM ..
Network Device : IPMP, BONDING, EtherChannel, APA , Teaming
RAC Storage , ASMLib

path .
Linux OS EMC PowerPath RAC OS

hardware .
2.
System Overview
2.1.
Hardware .
- 4

.

2.2.
Network
2.3.
DB Interconnect Network
- 5

.

Oracle Interconnect Network , OS bonding
.
[grid@rdb1 admin]$ ifconfig

bond0
Link encap:Ethernet HWaddr 00:18:71:7A:F0:84

inet addr:10.10.20.45 Bcast:10.10.20.255 Mask:255.255.255.0
inet6 addr: fe80::218:71ff:fe7a:f084/64 Scope:Link
[root@rdb1 network-scripts]# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.4.0 (October 7, 2008)
Bonding Mode: fault-tolerance (active-backup)

Primary Slave: eth1 (primary_reselect always)
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth1

MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:18:71:7a:f0:84
Slave Interface: eth2

MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:1b:78:56:35:5c
[root@rdb1 network-scripts]#
Oracle cross-cable direct , Switch

active-standby .
Storage Network
Server Switch , Switch Storage Line

.
- 6

.

switch server-switch-storage .
OS EMC Powerpath path solution .
[root@rdb1 ~]# powermt display dev=all

Pseudo name=emcpowera
CLARiiON ID=CK200050501062 [rac1db]
Logical device ID=600601607A2114002EFCE001A517DF11 [LUN 2]
state=alive; policy=BasicFailover; priority=0; queued-IOs=0
Owner: default=SP A, current=SP A
Array failover mode: 1
==============================================================================
---------------- Host --------------### HW Path
- Stor -
I/O Paths
-- I/O Path - -- Stats --Interf.
Mode
State Q-IOs Errors
==============================================================================
0 qla2xxx
sda
SP A0
active alive
0 qla2xxx
sdah
SP B0
active
alive
Pseudo name=emcpowerab
CLARiiON ID=CK200050501062 [rac1db]
Logical device ID=60060160B0701000549F38CFAC17DF11 [LUN 93]
ASM sda , sdah Native name emcpowera pseudo name

.
Storage RAID Level
Storage RAID 1+0 .

Storage ASM Disk redundancy level external
.
- 7

.

2.4.
H/W and OS Information
2.4.1. Hardware
RDB1 ( rdb1)
RDB2 ( rdb2 )
4cpu Dual-Core AMD Opteron(tm) Processor
4cpu Dual-Core AMD Opteron(tm) Processor
2214 HE
2214 HE
MEMORY
8G Memory
8G Memory
Total Local Disk
72 GB HDD * 2 (Mirroring)
72 GB HDD * 2 (Mirroring)
Swap
8004 MB
8004 MB
/tmp
/ (/dev/cciss/c0d0p1 free: 6144948K /)
/ (/dev/cciss/c0d0p1 free: 6145700K /)
/dev/shm
8G
8G
CPU
memory 8G Swap 8G Warnning

.
Local Disk cciss RAID-1 .
[root@rdb1 sbin]# hpacucli ctrl slot=1 show config

Smart Array P400 in Slot 1
(sn: P61620D9SUO3XH)
array A (SAS, Unused Space: 0 MB)

logicaldrive 1 (68.3 GB, RAID 1, OK)
physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SAS, 72 GB, OK)
physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SAS, 72 GB, OK)
/dev/shm oracle 11g r2 AMM oracle

memory . /dev/shm
.
mount -t tmpfs tmpfs /dev/shm -o size=5120M
2.4.2. OS
OS
Hostname
RDB1 ( rdb1)
RDB2 ( rdb2 )
Red Hat Enterprise Linux Server release 5.5
Red Hat Enterprise Linux Server release 5.5
(Tikanga) (Red Hat 4.1.2-48)
(Tikanga) (Red Hat 4.1.2-48)
rdb1
rdb2
- 8

.

Linux
Kernel
gcc version
rdb1
2.6.18-194.3.1.el5
#1
SMP
Linux rdb1 2.6.18-194.3.1.el5 #1 SMP x86_64
x86_64 x86_64 x86_64 GNU/Linux
x86_64 x86_64 GNU/Linux
gcc version 4.1.2 20080704
gcc version 4.1.2 20080704
2.4.3. Partition
Partitions (RDB1, RDB2)
RDB1
[root@rdb1 disks]# df -k
Filesystem
1K-blocks
/dev/cciss/c0d0p1
10483700
/dev/emcpowera2
3806200
48070504
none
6144948
184324
8388608
/dev/cciss/c0d0p3
Local
Used Available Use% Mounted on
26171860
39% /
45444300
1% /DB1ARC
8388608
0% /dev/shm
176200 24666192
/dev/cciss/c0d0p4
26171860
/dev/emcpowera1
178826660
1% /data
1205820 23636572
5% /rac
192012 169550696
1% /DGBACKUP
[root@rdb1 disks]#
RDB2
[root@rdb2 ~]# df -k
Filesystem
1K-blocks
/dev/cciss/c0d0p1
none
Used Available Use% Mounted on
10483700
3805448
6145700
8388608
8388608
/dev/cciss/c0d0p3
/dev/cciss/c0d0p4
/dev/emcpowera3
26171860
39% /
0% /dev/shm
176200 24666192
26171860
48070504
1% /data
176200 24666192
1% /rac
184324
1% /DB2ARC
45444300
[root@rdb2 ~]#
[root@rdb1 disks]# fdisk l
.
.
Disk /dev/sda: 286.0 GB, 286068310016 bytes
255 heads, 63 sectors/track, 34779 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot
Storage
Start
End
Blocks
Id System
/dev/sda1
22618
/dev/sda2
22619
28698
181679053+ 83 Linux
48837600
83
Linux
/dev/sda3
28699
34778
48837600
83
Linux
Disk /dev/sdb: 57.9 GB, 57982058496 bytes

Device Boot
/dev/sdb1
Start
End
1
55296
Blocks
56623088
Id System
83
.
.
- 9

.
Linux

Disk /dev/sdbn: 57.9 GB, 57982058496 bytes
Device Boot
Start
/dev/sdbn1
End
1
Blocks
55296
Id System
56623088
83
Linux
Disk /dev/emcpowera: 286.0 GB, 286068310016 bytes

End
Blocks
/dev/emcpowera1
Device Boot
Start
1
22618
181679053+ 83 Linux
Id System
/dev/emcpowera2
22619
28698
48837600
83 Linux
/dev/emcpowera3
28699
34778
48837600
83 Linux
Disk /dev/emcpowerab: 57.9 GB, 57982058496 bytes

Device Boot
Start
/dev/emcpowerab1
End
1
55296
Blocks
Id
56623088
System
83 Linux
.
.
Disk /dev/emcpowern: 57.9 GB, 57982058496 bytes
Device Boot
Start
/dev/emcpowern1
End
Blocks
55296
56623088
Id System
83
Linux
Disk /dev/emcpowero: 57.9 GB, 57982058496 bytes

Device Boot
/dev/emcpowero1
Start
End
1
55296
Blocks
Id System
56623088
83
Linux
ASM disk emcpowerb~emcpowerag .
OCR,VOTE emcpowerb1,emcpowerc1,emcpowerd1 disk 1G .

Nomal redundancy 3 asm disk , storage RAID 1+0
OCR,VOTE Normal redundancy .
Oracle ASM Diskgroup stripe data , Diskgroup

.
- 10

.

Diskgroup disk .
OCR,VOTE emcpowerb emcpowerb1, emcpowerb2 partitioning
storage OCR,VOTE disk disk
disk .
disk

disk
/rac Grid Infrastructure , Oracle RDMS 24.5G

.
(patch ) 13G , 11.2.0.1.2 patch
(9655006) requirement size patch check
20G .
ORACLE binary /data .
[root@rdb1 source]# pwd

/data/source
[root@rdb1 source]# ls -al
total 24
drwxr-xr-x 6 root
dba 4096 Dec 27 17:15 .
drwxr-xr-x 4 root
root 4096 Dec 27 13:13 ..
drwxr-xr-x 2 grid
dba
4096 Dec 22 17:55 asm
drwxr-xr-x 4 grid
dba
4096 Dec 23 14:14 grid
drwxr-xr-x 4 oracle dba
4096 Dec 27 13:40 oracle
- 11

.

drwxrwxr-x 5 oracle dba 4096 Dec 27 11:48 patch
[root@rdb1 source]#
2.4.4. ASM
Partitions
RDB1, RDB2
ASM (OCR,VOTE)
/dev/emcpowerb1
OV01 (/dev/oracleasm/disks/OV01)
ASM (OCR,VOTE)
/dev/emcpowerc1
ASM (OCR,VOTE)
/dev/emcpowerd1
/dev/emcpowerb2
VOL01 (/dev/oracleasm/disks/VOL01)
/dev/emcpowerc2
/dev/emcpowerd2
ASM(DGDATA1)
/dev/emcpowere1
ASM(DGDATA1)
/dev/emcpowerf1
ASM(DGDATA1)
/dev/emcpowerg1
ASM(DGDATA1)
/dev/emcpowerh1
ASM(DGDATA1)
/dev/emcpoweri1
ASM(DGDATA1)
/dev/emcpowerj1
ASM(DGDATA1)
/dev/emcpowekg1
ASM(DGDATA1)
/dev/emcpowelg1
ASM(DGDATA1)
/dev/emcpowemg1
ASM(DGDATA1)
/dev/emcpowern1
ASM(DGDATA1)
/dev/emcpowero1
ASM(DGDATA1)
/dev/emcpowerp1
ASM(DGDATA1)
/dev/emcpowerq1
ASM(DGDATA1)
/dev/emcpowerr1
ASM(DGBACKUP1)
/dev/emcpowers1
ASM(DGBACKUP1)
/dev/emcpowert1
ASM(DGBACKUP1)
/dev/emcpoweru1
ASM(DGBACKUP1)
/dev/emcpowerv1
ASM(DGBACKUP1)
/dev/emcpowerw1
ASM(DGBACKUP1)
/dev/emcpowerx1
ASM(DGBACKUP1)
/dev/emcpowery1
ASM(DGBACKUP1)
/dev/emcpowerz1
ASM(DGBACKUP1)
/dev/emcpoweraa1
ASM(DGBACKUP1)
/dev/emcpowerab1
ASM(DGBACKUP1)
/dev/emcpowerac1
- 12

.

ASM(DGBACKUP1)
/dev/emcpowerad1
ASM(DGBACKUP1)
/dev/emcpowerae1
ASM(DGBACKUP1)
/dev/emcpoweraf1
ASM(DGBACKUP1)
/dev/emcpowerag1
emcpowerb2 , emcpowerc2 , emcpowerd2 ASM Disk Group .
2.5.
IP
eth1
bond0
eth2
RDB1 (rdb1)
RDB2 (rdb2)
10.10.20.45
10.10.20.46
bond mode : active backup

Interconnect network IP
eth0
eth0
192.168.20.135
192.168.20.136
public
eth0:1
eth0
192.168.20.145
192.168.20.146
virtual IP
eth0:2
eth0
192.168.20.147
interconnect network bonding , active_standby mode .
2.6.
OS USER
USERNAME
root(0)
GROUP
root(0)
oracle(600)
dba(1001)
asmadmin(1002)
dba(1001)
grid(601)
Scan IP
asmadmin(1002)
SHELL
Home Direcoty
PROFILE
/bin/bash
/root
.bash_profile
/bin/bash
/home/oracle
.bash_profile
/bin/bash
/home/grid
.bash_profile
oracle oracle user: oinstall, dba, asmdba , grid user: oinstall, dba, asmadmin, asmdba,
asmoper .
OS User Home Directory Oracle Directory .

Grid directory owner root user
home directory directory permission .
2.7.
Oracle package package
- 13

.

ORACLE packages
RDB1 (rdb1)
RDB2 (rdb2)
KERNEL
2.6.18
2.6.18-194.3.1.el5
Packages
binutils-2.17.50.0.6 (x86_64)
binutils-2.17.50.0.6-14.el5
binutils-2.17.50.0.6-14.el5
compat-libstdc++-33-3.2.3 (i386)
compat-libstdc++-33-3.2.3-61
compat-libstdc++-33-3.2.3 (x86_64)
elfutils-libelf-0.125 (x86_64)
elfutils-libelf-0.137-3.el5
elfutils-libelf-0.137-3.el5
elfutils-libelf-devel-0.125 (x86_64)
elfutils-libelf-devel-0.137-3.el5
elfutils-libelf-devel-0.137-3.el5
elfutils-libelf-devel-static-0.125 (x86_64)
elfutils-libelf-devel-static-0.137-3.el5
elfutils-libelf-devel-static-0.137-3.el5
gcc-4.1.2 (x86_64)
gcc-4.1.2-48.el5
gcc-4.1.2-48.el5
gcc-c++-4.1.2 (x86_64)
gcc-c++-4.1.2-48.el5
gcc-c++-4.1.2-48.el5
glibc-2.5-24 (i686)
glibc-2.5-49
glibc-2.5-49
glibc-2.5-24 (x86_64)
glibc-2.5-49
glibc-2.5-49
glibc-common-2.5 (x86_64)
glibc-common-2.5-49
glibc-common-2.5-49
glibc-devel-2.5 (i386)
glibc-devel-2.5-49
glibc-devel-2.5-49
glibc-devel-2.5 (x86_64)
glibc-devel-2.5-49
glibc-devel-2.5-49
glibc-headers-2.5 (x86_64)
glibc-headers-2.5-49
kernel-headers-2.6.18 (x86_64)
kernel-headers-2.6.18-194.el5
kernel-headers-2.6.18-194.el5
ksh-20060214 (x86_64)
ksh-20100202-1.el5
ksh-20100202-1.el5
libaio-0.3.106 (i386)
libaio-0.3.106-5
libaio-0.3.106-5
libaio-0.3.106 (x86_64)
libaio-0.3.106-5
libaio-0.3.106-5
libaio-devel-0.3.106 (i386)
libaio-devel-0.3.106-5
libaio-devel-0.3.106 (x86_64)
libgcc-4.1.2 (i386)
libgcc-4.1.2-48.el5
libgcc-4.1.2-48.el5
libgcc-4.1.2 (x86_64)
libgcc-4.1.2-48.el5
libgcc-4.1.2-48.el5
libgomp-4.1.2 (x86_64)
libgomp-4.4.0-6.el5
libgomp-4.4.0-6.el5
libstdc++-4.1.2 (i386)
libstdc++-4.1.2-48.el5
libstdc++-4.1.2 (x86_64)
libstdc++-devel-4.1.2 (x86_64)
libstdc++-devel-4.1.2-48.el5
libstdc++-devel-4.1.2-48.el5
make-3.81 (x86_64)
make-3.81-3.el5
make-3.81-3.el5
sysstat-7.0.2 (x86_64)
sysstat-7.0.2-3.el5
sysstat-7.0.2-3.el5
unixODBC-2.2.11 (i386)
unixODBC-2.2.11-7.1
unixODBC-2.2.11-7.1
unixODBC-2.2.11 (x86_64)
unixODBC-2.2.11-7.1
unixODBC-2.2.11-7.1
unixODBC-devel-2.2.11 (i386)
unixODBC-devel-2.2.11-7.1
unixODBC-devel-2.2.11 (x86_64)
ASM
- 14

.

oracleasm ( )
oracleasm-2.6.18-194.3.1.el5-2.0.5-1.el5
oracleasm-2.6.18-194.3.1.el5-2.0.5-1.el5
oracleasmlib
oracleasmlib-2.0.4-1.el5
oracleasm-support
oracleasm-support-2.1.3-1.el5
EMCpower.LINUX-5.3.1.00.00-111
EMCpower.LINUX-5.3.1.00.00-111
EMC Powerpath
EMCpower.LINUX-5.3.1
32bit 64bit package .
oracleasm hardware CPU architecture OS kernel version .

Red Hat Enterprise Linux Server 5 asm download :
http://www.oracle.com/technetwork/topics/linux/downloads/rhel5-084877.html
Powerpath powerpath version 5.3.1 .

ASMLib: oracleasm createdisk command fails: Device '/dev/emcpowera1 is not a partition [Failed] [ID
469163.1]
Symptoms
The issue is resolved with EMC PowerPath version 5.3 SP1 (5.3.1) - refer:
The "oracleasm createdisk" command fails with the following error despite the device having been correctlypartitioned:
Marking disk "/dev/emcpowera1" as an ASM disk: asmtool: Device "/dev/emcpowera1" is not a partition
Cause
EMC PowerPath versions <= 5.3.0 do not properly support the I/O calls performed by oracleasm
Solution
The issue is resolved with EMC PowerPath version 5.3 SP1 (5.3.1) - refer:
EMC PowerPath Family for Linux Version 5.3 Release Notes - January 25, 2010 (Fixed Problems)
EMC Knowledgebase Solution emc203137
3.
Pre-Installation Steps
3.1.
Preparing the system
3.1.1. Host file Configuration
/etc/hosts Pulic IP VIP, Interconnect .
(root .)
[root@rdb1etc]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1
localhost.localdomain localhost
- 15

.
[FAILED]

192.168.20.135
rdb1.goodus.com
rdb1
192.168.20.136
rdb2.goodus.com
rdb2
# Private LAN For Oracle RAC Interconnection (bond0)

10.10.20.45
rdb1-priv
10.10.20.46
rdb2-priv
# Public Oracle Virtual IP -(VIP)

192.168.20.145
rdb1-vip.goodus.com
rdb1-vip
192.168.20.146
rdb2-vip.goodus.com
rdb2-vip
# Public Oracle Scan IP

192.168.20.147
rdb-scan.goodus.com
rdb-scan
Oracle Grid infrastructure DNS 3 scan IP DNS hosts file

scan IP 1 .
scan EM(Enterprise Manager)
SCAN .
3.1.2. Kernel Sysctl Configuration
Oracle OS kernel .
/etc/sysctl.conf .
(root .)
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmni = 4096
kernel.sem = 2700 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
kernel.shmmax = 8589934592
(root .)
[root@rdb1 ~]# sysctl -p
3.1.3. User resource limit
- 16

.
OS user limit .
/etc/security/limits.conf .
(root .)
oracle
soft
oracle
hard
oracle
soft
oracle
hard
grid
soft
grid
hard
grid
soft
grid
hard
nproc
2047
nproc
nofile
1024
nofile
65536
nproc
2047
nproc
nofile
16384
16384
1024
nofile
65536
limit user login pam .

/etc/pam.d/login .
(root .)
session
required
/lib64/security/pam_limits.so
3.1.4. NTP
11g r2 RAC Time .
NTP
NTP cluster node Time x ntp
.
/etc/sysconfig/ntpd -x ntp service restart .
(root .)
# Drop root to id 'ntp:ntp' by default.
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"
# Set to 'yes' to sync hw clock after successful ntpdate

SYNC_HWCLOCK=no
# Additional options for ntpdate

NTPDATE_OPTIONS=""
- 17

.

[root@rdb1 ~]# service ntpd restart
Shutting down ntpd: [ OK ]
Starting ntpd: [ OK ]
[root@rdb1 sysconfig]# ps -ef | grep ntp
ntp
17721
1 0 Dec22 ?
root
22970 22391 0 21:50 pts/0
00:00:00 ntpd -x -u ntp:ntp -p /var/run/ntpd.pid

00:00:00 grep ntp
[root@rdb1 sysconfig]#
NTP
NTP Grid Infrastructure CTSSD daemon
. .
3.1.5. User Shell Configuration
grid , oracle shell .
( .)
grid user
rdb1
rdb2
# .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
. ~/.bashrc
fi
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/
PATH=$PATH:$HOME/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/
bin:/usr/local/sbin
bin:/usr/local/sbin
export PATH
export PATH
export GRID_HOME=/rac/grid/product/11.2.0
export GRID_HOME=/rac/grid/product/11.2.0
export ORACLE_SID=+ASM1
export ORACLE_SID=+ASM2
export PATH=$GRID_HOME/bin:$GRID_HOME/OPatch:$PATH
export PATH=$GRID_HOME/bin:$GRID_HOME/OPatch:$PATH
export ORACLE_HOME=/rac/grid/product/11.2.0
export ORACLE_HOME=/rac/grid/product/11.2.0
export DISPLAY=192.168.*.*:0.0
export DISPLAY=192.168.*.*:0.0
if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -p 16384
ulimit -n 65536
ulimit -n 65536
else
else
- 18

.

ulimit -u 16384 -n 65536
ulimit -u 16384 -n 65536
fi
fi
umask 022
umask 022
fi
fi
oracle user
rdb1
rdb2
# .bash_profile
# .bash_profile
. ~/.bashrc
. ~/.bashrc
fi
fi
PATH=$PATH:$HOME/bin
PATH=$PATH:$HOME/bin
export PATH
export PATH
stty erase ^H
stty erase ^H
export ORACLE_BASE=/rac/oracle
export ORACLE_BASE=/rac/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0
export ORACLE_SID=RDB1
export ORACLE_SID=RDB2
export ORACLE_UNQNAME=RAC1DB
export ORACLE_UNQNAME=RAC1DB
export ORACLE_TERM=xterm
export ORACLE_TERM=xterm
export PATH=/usr/sbin:$PATH
export PATH=/usr/sbin:$PATH
export
export
PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH
PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export
export
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACL
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACL
E_HOME/rdbms/jlib
E_HOME/rdbms/jlib
export DISPLAY=192.168.6.17:0.0
export DISPLAY=192.168.6.40:0.0
ulimit -p 16384
ulimit -p 16384
ulimit -n 65536
ulimit -n 65536
else
else
ulimit -u 16384 -n 65536
ulimit -u 16384 -n 65536
fi
fi
umask 022
umask 022
fi
fi
3.1.6.
.
SELinux
- 19

.

( root .)
[root@rdb1 ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#
enforcing - SELinux security policy is enforced.
permissive - SELinux prints warnings instead of enforcing.
disabled - SELinux is fully disabled.
SELINUX=disabled
# SELINUXTYPE= type of policy in use. Possible values are:
#
targeted - Only targeted network daemons are protected.
strict - Full SELinux protection.
SELINUXTYPE=targeted
# SETLOCALDEFS= Check local definition changes

SETLOCALDEFS=0
[root@rdb1 ~]#
SELinux rwx:rwx:rwx daemon(process) level

disable .(reboot ).
SELinux sqlplus library access fail
.
SELinux library file context .
IPTABLES
( root .)
[root@rdb1 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target
prot opt source
destination
Chain FORWARD (policy ACCEPT)

target
prot opt source
destination
Chain OUTPUT (policy ACCEPT)

target
prot opt source
destination
[root@rdb1 ~]#
- 20

.

Linux Network
.
TCP Wrapper
( root .)
[root@rdb2 ~]# cat /etc/hosts.allow
#
# hosts.allow
This file describes the names of the hosts which are
allowed to use the local INET services, as decided
by the '/usr/sbin/tcpd' server.
sshd : 192.168.20.136, 192.168.20.135, 10.10.20.45, 10.10.20.46
[root@rdb2 ~]# cat /etc/hosts.deny

#
# hosts.deny
This file describes the names of the hosts which are
*not* allowed to use the local INET services, as decided
by the '/usr/sbin/tcpd' server.
#
# The portmap line is redundant, but it is left to remind you that
# the new secure portmap uses hosts.deny and hosts.allow.
In particular
# you should know that NFS uses portmap!
ALL : ALL
[root@rdb2 ~]#
TCP Wrapper IP TCP tcpd

.
TCP Wrapper 11gr2 RAC SSH daemon
IP .
3.1.7. SSH
RAC password .
ssh deamon .
- 21

.

(grid , oracle user .)
[grid@rdb1 ~]$ ps -ef | grep ssh
root
grid
9434
1 0 Dec22 ?
11711 11674 0 20:59 pts/0
00:00:00 /usr/sbin/sshd
00:00:00 grep ssh
[grid@rdb1 ~]$
SSH
grid user
rdb1
rdb2
[grid@rdb1 ~]$ ssh-keygen -t rsa
[grid@rdb2 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/rac/grid/.ssh/id_rsa):
Enter file in which to save the key (/rac/grid/.ssh/id_rsa):
Created directory '/rac/grid/.ssh'.
Created directory '/rac/grid/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /rac/grid/.ssh/id_rsa.
Your identification has been saved in /rac/grid/.ssh/id_rsa.
Your public key has been saved in /rac/grid/.ssh/id_rsa.pub.
Your public key has been saved in /rac/grid/.ssh/id_rsa.pub.
The key fingerprint is:
90:82:e3:53:37:cf:8d:16:5b:c7:6a:9e:20:c2:a7:05 grid@rdb1
e6:cd:f3:73:43:2f:de:5b:53:96:d8:b3:8a:03:74:e6 grid@rdb2
[grid@rdb1~]$ cd .ssh
[grid@rdb2 ~]$ cd .ssh
[grid@rdb1.ssh]$ cat id_rsa.pub > authorized_keys
[grid@rdb2 .ssh]$ cat id_rsa.pub > authorized_keys
[grid@rdb1.ssh]$ cat authorized_keys
[grid@rdb2 .ssh]$ cat authorized_keys
ssh-rsa
ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAQEAzSmTrcOtC7/QIPhTnfSk/d+
AAAAB3NzaC1yc2EAAAABIwAAAQEA1+u2IIN+zNAIVt+RHtQpa5
R32q99QJrKxbq6N6FWabYMhZMV4ND8CN7A+5oYlgl8TR6uYuX
7SI2Cjjad1aDB2UGsKsFI2+DO7daKjlR+nhcXi4g26im9nPbv3TwcH
86nOUY+rZZhVi2yMn98j5bgcyDThrT0dB0xvvCQHH0lyiTh+sCiQi
14ho5hyFh+BT5Km3O3z4Jg8GNAO4F3Lj2IIH8uhY9ZMc3ofwdEV
Y3DgV8chTlnbRlxHhFcHxyqZdSWX7U44JI/2rh8QX3EgrnUu9/PfQ
caeuPO0d+qEKuZwhx2Pum6XXtD1SDPv0m4RmU1Ik/0gkqDj2Sa
SNOO9yvtXzA/KkvYpVGH5YjkDLgYvlzmYqYBEpeF0C5ccD/jme+Y
y1vqnUonBtgdv7wEJqMEkg97i66KXxI3jpVuE81X0Rl+gEleWs8m4
2mrVsMa4SLmu0JJoRzNEYSSzdVEakwniyYBf5+udTehF3PBtqSlAiil
d32TlTsty8w8yzrn18L1N5E5bk21HuInbAHk/5rHhevU559zw6SFEz
ncxUn//W54xkc3364t7IAdCXwoEKhRXIBUoGVRdx57V5ZM+bK0y
2Q1qVtsdUyMvncMEhZr2My/KyZLQlIxkk6ajGOaziliy2dk9YwVoCB
QOoFRw== grid@rdb1
UiWw== grid@rdb2
authorized_keys key .
[grid@rdb1.ssh]$ cat authorized_keys
[grid@rdb2 .ssh]$ cat authorized_keys
ssh-rsa
ssh-rsa
caeuPO0d+qEKuZwhx2Pum6XXtD1SDPv0m4RmU1Ik/0gkqDj2Sa
y1vqnUonBtgdv7wEJqMEkg97i66KXxI3jpVuE81X0Rl+gEleWs8m4
d32TlTsty8w8yzrn18L1N5E5bk21HuInbAHk/5rHhevU559zw6SFEz
2Q1qVtsdUyMvncMEhZr2My/KyZLQlIxkk6ajGOaziliy2dk9YwVoCB
QOoFRw== grid@rdb1
UiWw== grid@rdb2
- 22

.

ssh-rsa
ssh-rsa
caeuPO0d+qEKuZwhx2Pum6XXtD1SDPv0m4RmU1Ik/0gkqDj2Say
1vqnUonBtgdv7wEJqMEkg97i66KXxI3jpVuE81X0Rl+gEleWs8m4d
32TlTsty8w8yzrn18L1N5E5bk21HuInbAHk/5rHhevU559zw6SFEz2
Q1qVtsdUyMvncMEhZr2My/KyZLQlIxkk6ajGOaziliy2dk9YwVoCB
UiWw== grid@rdb2
QOoFRw== grid@rdb1
oracle user
rdb1
rdb2
[oracle@rdb1~]$ ssh-keygen -t rsa
[oracle@rdb2 ~]$ ssh-keygen -t rsa
Enter file in which to save the key (/rac/oracle/.ssh/id_rsa):
Enter file in which to save the key (/rac/oracle/.ssh/id_rsa):
Created directory '/rac/oracle/.ssh'.
Created directory '/rac/oracle/.ssh'.
Your identification has been saved in /rac/oracle/.ssh/id_rsa.
Your identification has been saved in /rac/oracle/.ssh/id_rsa.
Your public key has been saved in /rac/oracle/.ssh/id_rsa.pub.
Your public key has been saved in /rac/oracle/.ssh/id_rsa.pub.
bf:8c:e2:a0:69:bb:6b:62:7b:1a:c3:da:5b:45:c9:79 oracle@rdb1
a4:8d:58:03:3d:1d:fb:20:99:3c:d3:e7:4b:f1:c2:28 oracle@rdb2
[oracle@rdb1~]$ cd .ssh
[oracle@rdb2 ~]$ cd .ssh
[oracle@rdb1.ssh]$ cat id_rsa.pub > authorized_keys
[oracle@rdb2 .ssh]$ cat id_rsa.pub > authorized_keys
[oracle@rdb1.ssh]$ cat authorized_keys
[oracle@rdb2 .ssh]$ cat authorized_keys
ssh-rsa
ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAQEAyZbCFTVYE/r6QaRidXOIc8A
AAAAB3NzaC1yc2EAAAABIwAAAQEAvtSY969Kk/QvHfDWByVxD
hWnTUxow6iPAC6700EiZ5qY+XPv+/BjcmjCLYIGSWAHnFiyG+ae0
ZV2IX8eGPcSTU+1xC7L4fNnUuw8fuiwAPX+xR+Q21FcLpO6ENo6
/d1HwyWX1GbGnb0fVXsFWf6bzQJke0BxmscCa/GXkDGMnsNNs
KPVAE2d/G9+L/1A7CJ9PAUUgXywKwx+reyKrGB9/LHeMg3eBttd
36TQ/VpQ9Bkq/YnHeCiZo1EFS3klQISOiV0Dd3EkQShx4RT9taQ7o
51cjVnZO6tVFT9FwBHo8ihwurpFJr0yJgDAGoIXxVhbmgpjBF0Ln3
+S4HfPpl/15I3t+fIb4jwKp/tprJB6IQH3yka6V6gc+mJfIe3Z3aaWD9
O649hul3PyAb1ld0p94q1zDyegAZ+YaLrVaSIHS8VDIWr6zo8P+er
YLUJhSRdPPebhzqkfyw/D0vjnSO6RniIBaY3YscCzno8l9L2XhdPYa
95MuDL1B8bXZfAq/c7ojURWw1x72Dt/i4BFdlBG5AwaLm+7N1Uy
wzwYGDR1gia7cseaWzhsFJKpVW5QR5CQecUCz+2/aw0jLcZeGaP
yBn9kZq2tFI1EQcRkOHSG6AvP0SPX5Tn2ND0z4GTxLq8rcxhyMze
7sfw== oracle@rdb1
LlFtGndyew== oracle@rdb2
[oracle@rdb1.ssh]$ cat authorized_keys
[oracle@rdb2 .ssh]$ cat authorized_keys
ssh-rsa
ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAQEAvtSY969Kk/QvHfDWByVxD
ZV2IX8eGPcSTU+1xC7L4fNnUuw8fuiwAPX+xR+Q21FcLpO6ENo6
KPVAE2d/G9+L/1A7CJ9PAUUgXywKwx+reyKrGB9/LHeMg3eBttd
36TQ/VpQ9Bkq/YnHeCiZo1EFS3klQISOiV0Dd3EkQShx4RT9taQ7o
51cjVnZO6tVFT9FwBHo8ihwurpFJr0yJgDAGoIXxVhbmgpjBF0Ln3
+S4HfPpl/15I3t+fIb4jwKp/tprJB6IQH3yka6V6gc+mJfIe3Z3aaWD9
O649hul3PyAb1ld0p94q1zDyegAZ+YaLrVaSIHS8VDIWr6zo8P+er
YLUJhSRdPPebhzqkfyw/D0vjnSO6RniIBaY3YscCzno8l9L2XhdPYa
95MuDL1B8bXZfAq/c7ojURWw1x72Dt/i4BFdlBG5AwaLm+7N1Uy
- 23

.

wzwYGDR1gia7cseaWzhsFJKpVW5QR5CQecUCz+2/aw0jLcZeGaP
yBn9kZq2tFI1EQcRkOHSG6AvP0SPX5Tn2ND0z4GTxLq8rcxhyMze
7sfw== oracle@rdb1
LlFtGndyew== oracle@rdb2
ssh-rsa
ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAQEAvtSY969Kk/QvHfDWByVxDZ
V2IX8eGPcSTU+1xC7L4fNnUuw8fuiwAPX+xR+Q21FcLpO6ENo6K
PVAE2d/G9+L/1A7CJ9PAUUgXywKwx+reyKrGB9/LHeMg3eBttd5
1cjVnZO6tVFT9FwBHo8ihwurpFJr0yJgDAGoIXxVhbmgpjBF0Ln3O
36TQ/VpQ9Bkq/YnHeCiZo1EFS3klQISOiV0Dd3EkQShx4RT9taQ7
649hul3PyAb1ld0p94q1zDyegAZ+YaLrVaSIHS8VDIWr6zo8P+er9
o+S4HfPpl/15I3t+fIb4jwKp/tprJB6IQH3yka6V6gc+mJfIe3Z3aaW
5MuDL1B8bXZfAq/c7ojURWw1x72Dt/i4BFdlBG5AwaLm+7N1Uyy
D9YLUJhSRdPPebhzqkfyw/D0vjnSO6RniIBaY3YscCzno8l9L2XhdP
Bn9kZq2tFI1EQcRkOHSG6AvP0SPX5Tn2ND0z4GTxLq8rcxhyMzeLl
YawzwYGDR1gia7cseaWzhsFJKpVW5QR5CQecUCz+2/aw0jLcZeG
FtGndyew== oracle@rdb2
aP7sfw== oracle@rdb1
, grid , oracle user password

.
(grid , oracle user .)
rdb1 grid user
[grid@rdb1 ~]$ ssh rdb1 date

The authenticity of host 'rdb1 (192.168.20.135)' can't be established.
RSA key fingerprint is 09:48:2b:bf:0e:c3:d5:c8:62:2d:6b:92:20:3c:d5:40.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rdb1,192.168.20.135' (RSA) to the list of known hosts.
Wed Dec 22 17:33:13 KST 2010
Wed Dec 22 17:33:16 KST 2010

The authenticity of host 'rdb2 (192.168.20.136)' can't be established.
Warning: Permanently added 'rdb2,192.168.20.136' (RSA) to the list of known hosts.
Wed Dec 22 17:33:22 KST 2010
Wed Dec 22 17:33:25 KST 2010
[grid@rdb1 ~]$ ssh rdb1-priv date

The authenticity of host 'rdb1-priv (10.10.20.45)' can't be established.
- 24

.

Warning: Permanently added 'rdb1-priv,10.10.20.45' (RSA) to the list of known hosts.
Wed Dec 22 17:34:44 KST 2010
Wed Dec 22 17:34:47 KST 2010

The authenticity of host 'rdb2-priv (10.10.20.46)' can't be established.
Warning: Permanently added 'rdb2-priv,10.10.20.46' (RSA) to the list of known hosts.
Wed Dec 22 17:37:25 KST 2010
[oracle@rdb1 ~]$ ssh rdb2-priv date
Wed Dec 22 17:37:28 KST 2010
3.2.
Preparing Oracle ASM
3.2.1. Installation ASM LIBRARY
Linux ASM .
1)
ASM with ASMLIB I/O : ASM raw block devices ASMLib . Raw device
.
2)
ASM with Standard Linux I/O : ASM raw block devices Linux system I/O call . ASM
partition raw device .
ASMLIB .
OS Kernel version kernel 2.6.18-194.3.1.el5 .
( root .)
[root@rdb1~]# uname -r
2.6.18-194.3.1.el5
[root@rdb1 asm]# ls -rlt
total 252
-rw-r--r-- 1 root root 137887 Dec 22 17:53 oracleasm-2.6.18-194.3.1.el5-2.0.5-1.el5.x86_64.rpm
-rw-r--r-- 1 root root 90444 Dec 22 17:53 oracleasm-support-2.1.3-1.el5.x86_64.rpm
-rw-r--r-- 1 root root 14176 Dec 22 17:53 oracleasmlib-2.0.4-1.el5.x86_64.rpm
[root@rdb1 asm]#
[root@rdb1 asm]# rpm -Uvh oracleasm-support-2.1.3-1.el5.x86_64.rpm
warning: oracleasm-support-2.1.3-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...
########################################### [100%]
- 25

.

1:oracleasm-support
########################################### [100%]
[root@rdb1 asm]# rpm -Uvh oracleasm-2.6.18-194.3.1.el5-2.0.5-1.el5.x86_64.rpm

warning: oracleasm-2.6.18-194.3.1.el5-2.0.5-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...
########################################### [100%]
1:oracleasm-2.6.18-194.3.########################################### [100%]
[root@rdb1 asm]# rpm -Uvh oracleasmlib-2.0.4-1.el5.x86_64.rpm
warning: oracleasmlib-2.0.4-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...
1:oracleasmlib
########################################### [100%]
########################################### [100%]
[root@rdb1 asm]#
[root@rdb1 asm]# rpm -qa | grep oracleasm
oracleasm-2.6.18-194.3.1.el5-2.0.5-1.el5
[root@rdb1 asm]#
3.2.2. ASM Configuration
Oracle ASM Module configuration .

asm module .
(root .)
rdb1
[root@rdb1 init.d]# ./oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver.
The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: grid
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK
[root@rdb1 init.d]#
rdb2
[root@rdb2 asmlib]# cd /etc/init.d
[root@rdb2 init.d]# ./oracleasm configure
Configuring the Oracle ASM library driver.
- 26

.

This will configure the on-boot properties of the Oracle ASM library
driver.
The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: grid
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK
[root@rdb2 init.d]#
3.2.3. Multipath
ASM multipath path .

/dev/sda , /dev/sdb , path /dev/multipatha , ASM /dev/sda ,
/dev/sdb , /dev/multipatha path , storage driver
.
ASM multipath disk : http://www.oracle.com/technology/tech/linux/asmlib/multipath.html
multipath path .
/etc/sysconfig/oracleasm .
(root .)
[root@rdb1 sysconfig]# cat oracleasm
#
# This is a configuration file for automatic loading of the Oracle
# Automatic Storage Management library kernel driver.
# By running /etc/init.d/oracleasm configure.
It is generated
Please use that method
# to modify this file

#
# ORACLEASM_ENABELED: 'true' means to load the driver on boot.
ORACLEASM_ENABLED=true
# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.
ORACLEASM_UID=grid
# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.
ORACLEASM_GID=dba
- 27

.

# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.
ORACLEASM_SCANBOOT=true
# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="emcpower sd"
# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE=""
[root@rdb1 sysconfig]#
3.2.4. Create ASM VOLUME.
ASM VOLUME DISK .

/root/cr_asm_disk.sh script .
(root rdb1 .)
[root@rdb1 disks]# cat /root/cr_asm_disk.sh
oracleasm createdisk OV01 /dev/emcpowerb1
oracleasm createdisk OV02 /dev/emcpowerc1
oracleasm createdisk OV03 /dev/emcpowerd1
oracleasm createdisk VOL01 /dev/emcpowerb2
oracleasm createdisk VOL02 /dev/emcpowerc2
oracleasm createdisk VOL03 /dev/emcpowerd2
oracleasm createdisk VOL04 /dev/emcpowere1
oracleasm createdisk VOL05 /dev/emcpowerf1
oracleasm createdisk VOL06 /dev/emcpowerg1
oracleasm createdisk VOL07 /dev/emcpowerh1
oracleasm createdisk VOL08 /dev/emcpoweri1
oracleasm createdisk VOL09 /dev/emcpowerj1
oracleasm createdisk VOL10 /dev/emcpowerk1
oracleasm createdisk VOL11 /dev/emcpowerl1
oracleasm createdisk VOL12 /dev/emcpowerm1
oracleasm createdisk VOL13 /dev/emcpowern1
oracleasm createdisk VOL14 /dev/emcpowero1
oracleasm createdisk VOL15 /dev/emcpowerp1
oracleasm createdisk VOL16 /dev/emcpowerq1
oracleasm createdisk VOL17 /dev/emcpowerr1
oracleasm createdisk VOL18 /dev/emcpowers1
oracleasm createdisk VOL19 /dev/emcpowert1
oracleasm createdisk VOL20 /dev/emcpoweru1
oracleasm createdisk VOL21 /dev/emcpowerv1
oracleasm createdisk VOL22 /dev/emcpowerw1
oracleasm createdisk VOL23 /dev/emcpowerx1
oracleasm createdisk VOL24 /dev/emcpowery1
oracleasm createdisk VOL25 /dev/emcpowerz1
oracleasm createdisk VOL26 /dev/emcpoweraa1
oracleasm createdisk VOL27 /dev/emcpowerab1
- 28

.

oracleasm createdisk VOL28 /dev/emcpowerac1
oracleasm createdisk VOL29 /dev/emcpowerad1
oracleasm createdisk VOL30 /dev/emcpowerae1
oracleasm createdisk VOL31 /dev/emcpoweraf1
oracleasm createdisk VOL32 /dev/emcpowerag1
[root@rdb1 disks]#
[root@rdb1 ~]# ./cr_asm_disk.sh
Writing disk header: done
Instantiating disk: done
.
.
[root@rdb1 ~]#
[root@rdb1 ~]# oracleasm listdisks
OV01
OV02
OV03
VOL01
VOL02
VOL03
VOL04
VOL05
VOL06
VOL07
VOL08
VOL09
VOL10
VOL11
VOL12
VOL13
VOL14
VOL15
VOL16
VOL17
VOL18
VOL19
VOL20
VOL21
VOL22
VOL23
VOL24
VOL25
- 29

.

VOL26
VOL27
VOL28
VOL29
VOL30
VOL31
VOL32
[root@rdb1 ~]#
ASM VOL Disk .
(root rdb2 .)
[root@rdb2 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "VOL27"
Instantiating disk "OV01"
- 30

.

[root@rdb2 ~]#
[root@rdb2 ~]#
[root@rdb2 ~]# oracleasm listdisks
OV01
OV02
OV03
VOL01
VOL02
VOL03
VOL04
VOL05
VOL06
VOL07
VOL08
VOL09
VOL10
VOL11
VOL12
VOL13
VOL14
VOL15
VOL16
VOL17
VOL18
VOL19
VOL20
VOL21
VOL22
VOL23
VOL24
VOL25
VOL26
VOL27
VOL28
VOL29
VOL30
VOL31
VOL32
[root@rdb2 ~]#
major, minor number .

cluster node disk major,minor number .
reboot major , minor reboot
.
(Linux mpath LVM cluster node
reboot .)
- 31

.

rdb1
rdb2
[root@rdb1 sysconfig]# cd /dev/oracleasm/disks
[root@rdb2 sysconfig]# cd /dev/oracleasm/disks
[root@rdb1 disks]# ls -al
[root@rdb2 disks]# ls -al
total 0
total 0
drwxr-xr-x 1 root root

0 Dec 22 17:58 .
0 Dec 22 17:59 .
0 Dec 22 17:58 ..
brw-rw---- 1 grid dba
120,
17 Dec 22 18:05 OV01
120,
17 Dec 22 18:05 OV01
120,
33 Dec 22 18:06 OV02
120,
33 Dec 22 18:06 OV02
120,
49 Dec 22 18:05 OV03
120,
49 Dec 22 18:05 OV03
120,
18 Dec 22 18:05 VOL01
120,
18 Dec 22 18:05 VOL01
120,
34 Dec 22 18:05 VOL02
120,
34 Dec 22 18:05 VOL02
120,
50 Dec 22 18:05 VOL03
120,
50 Dec 22 18:05 VOL03
120,
65 Dec 22 18:05 VOL04
120,
65 Dec 22 18:05 VOL04
120,
81 Dec 22 18:05 VOL05
120,
81 Dec 22 18:05 VOL05
120,
97 Dec 22 18:05 VOL06
120,
97 Dec 22 18:05 VOL06
120, 113 Dec 22 18:05 VOL07
120, 113 Dec 22 18:05 VOL07
120, 129 Dec 22 18:05 VOL08
120, 129 Dec 22 18:05 VOL08
120, 145 Dec 22 18:05 VOL09
120, 145 Dec 22 18:05 VOL09
120, 161 Dec 22 18:05 VOL10
120, 161 Dec 22 18:05 VOL10
120, 177 Dec 22 18:05 VOL11
120, 177 Dec 22 18:05 VOL11
120, 193 Dec 22 18:05 VOL12
120, 193 Dec 22 18:05 VOL12
120, 209 Dec 22 18:05 VOL13
120, 209 Dec 22 18:05 VOL13
120, 225 Dec 22 18:05 VOL14
120, 225 Dec 22 18:05 VOL14
120, 241 Dec 22 18:05 VOL15
120, 241 Dec 22 18:05 VOL15
120, 257 Dec 22 18:05 VOL16
120, 257 Dec 22 18:05 VOL16
120, 273 Dec 22 18:05 VOL17
120, 273 Dec 22 18:05 VOL17
120, 289 Dec 22 18:05 VOL18
120, 289 Dec 22 18:05 VOL18
120, 305 Dec 22 18:05 VOL19
120, 305 Dec 22 18:05 VOL19
120, 321 Dec 22 18:05 VOL20
120, 321 Dec 22 18:05 VOL20
120, 337 Dec 22 18:05 VOL21
120, 337 Dec 22 18:05 VOL21
120, 353 Dec 22 18:05 VOL22
120, 353 Dec 22 18:05 VOL22
120, 369 Dec 22 18:05 VOL23
120, 369 Dec 22 18:05 VOL23
120, 385 Dec 22 18:05 VOL24
120, 385 Dec 22 18:05 VOL24
120, 401 Dec 22 18:05 VOL25
120, 401 Dec 22 18:05 VOL25
120, 417 Dec 22 18:05 VOL26
120, 417 Dec 22 18:05 VOL26
120, 433 Dec 22 18:05 VOL27
120, 433 Dec 22 18:05 VOL27
120, 449 Dec 22 18:05 VOL28
120, 449 Dec 22 18:05 VOL28
120, 465 Dec 22 18:05 VOL29
120, 465 Dec 22 18:05 VOL29
120, 481 Dec 22 18:05 VOL30
120, 481 Dec 22 18:05 VOL30
120, 497 Dec 22 18:05 VOL31
120, 497 Dec 22 18:05 VOL31
120, 513 Dec 22 18:05 VOL32
120, 513 Dec 22 18:05 VOL32
[root@rdb1 disks]#
0 Dec 22 17:59 ..
[root@rdb2 disks]#
major number emc powerpath 120 , 8 powerpath

sd# single path .
- 32

.

3.3.
Install requirement
3.3.1. cluster verify utility
.

.
(grid rdb1 .).
Grid Infrastructure CD .
[grid@rdb1 grid]$ ./runcluvfy.sh stage -pre crsinst -n rdb1,rdb2 -verbose
Performing pre-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "rdb1"
Destination Node
Reachable?
------------------------------------
------------------------
rdb1
yes
rdb2
yes
Result: Node reachability check passed from node "rdb1"
Checking user equivalence...

Check: User equivalence for user "grid"
Node Name
Comment
------------------------------------
------------------------
rdb2
passed
rdb1
passed
Result: User equivalence check passed for user "grid"

Checking node connectivity...
Checking hosts config file...
Node Name
------------
Status
Comment
------------------------
rdb2
passed
rdb1
passed
------------------------
Verification of the hosts config file successful
Interface information for node "rdb2"

Name
IP Address
Subnet
Gateway
Def. Gateway
HW Address
- 33

.
MTU

------ --------------- --------------- --------------- --------------- ----------------- -----bond0 10.10.20.46
10.10.20.0
0.0.0.0
192.168.20.1
00:1B:78:95:AC:3A 1500
eth0
192.168.20.0
0.0.0.0
192.168.20.1
00:1B:78:95:AC:38 1500
192.168.20.136
Interface information for node "rdb1"

Name
IP Address
Subnet
Gateway
Def. Gateway
HW Address
------ --------------- --------------- --------------- --------------- ----------------- -----bond0 10.10.20.45
10.10.20.0
0.0.0.0
192.168.20.1
00:18:71:7A:F0:84 1500
eth0
192.168.20.0
0.0.0.0
192.168.20.1
00:18:71:7A:F0:82 1500
192.168.20.135
Check: Node connectivity of subnet "10.10.20.0"

Source
-----------------------------rdb2:bond0
Destination
------------------------------
Connected?
----------------
rdb1:bond0
yes
Result: Node connectivity passed for subnet "10.10.20.0" with node(s) rdb2,rdb1
Check: TCP connectivity of subnet "10.10.20.0"

Source
-----------------------------rdb1:10.10.20.45
Destination
------------------------------
Connected?
----------------
rdb2:10.10.20.46
passed
Result: TCP connectivity check passed for subnet "10.10.20.0"
Check: Node connectivity of subnet "192.168.20.0"

Source
-----------------------------rdb2:eth0
Destination
------------------------------
Connected?
----------------
rdb1:eth0
yes
Result: Node connectivity passed for subnet "192.168.20.0" with node(s) rdb2,rdb1
Check: TCP connectivity of subnet "192.168.20.0"

Source
-----------------------------rdb1:192.168.20.135
Destination
------------------------------
Connected?
----------------
rdb2:192.168.20.136
passed
Result: TCP connectivity check passed for subnet "192.168.20.0"
Interfaces found on subnet "192.168.20.0" that are likely candidates for VIP are:
rdb2 eth0:192.168.20.136
rdb1 eth0:192.168.20.135
Interfaces found on subnet "10.10.20.0" that are likely candidates for a private interconnect are:
rdb2 bond0:10.10.20.46
rdb1 bond0:10.10.20.45
Result: Node connectivity check passed
- 34

.
MTU

Check: Total memory
Node Name
Available
------------
Required
------------------------
Comment
------------------------
----------
rdb2
7.8GB (8177484.0KB)
1.5GB (1572864.0KB)
passed
rdb1
7.8GB (8177484.0KB)
1.5GB (1572864.0KB)
passed
Result: Total memory check passed

Check: Available memory
Node Name
Available
------------
Required
Comment
------------------------
------------------------
rdb2
7.65GB (8023820.0KB)
50MB (51200.0KB)
---------passed
rdb1
7.49GB (7850852.0KB)
50MB (51200.0KB)
passed
Result: Available memory check passed

Check: Swap space
Node Name
Available
Required
------------ ------------------------
------------------------
Comment
----------
rdb2
7.46GB (7817272.0KB)
7.8GB (8177484.0KB)
failed
rdb1
7.46GB (7817272.0KB)
7.8GB (8177484.0KB)
failed
Result: Swap space check failed

Check: Free disk space for "rdb2:/tmp"
Path
Node Name
----------------
------------
/tmp
Mount point
------------
rdb2
Available
------------
Required
------------
6.14GB
Comment
------------
1GB
passed
Result: Free disk space check passed for "rdb2:/tmp"

Check: Free disk space for "rdb1:/tmp"
Path
Node Name
----------------
------------
/tmp
rdb1
Mount point
------------
Available
------------
Required
------------
6.12GB
1GB
Comment
-----------passed
Result: Free disk space check passed for "rdb1:/tmp"

Check: User existence for "grid"
Node Name
Status
------------
------------------------
Comment
------------------------
rdb2
exists
passed
rdb1
exists
passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
Node Name
Status
------------ ------------------------
Comment
------------------------
rdb2
does not exist
failed
rdb1
does not exist
failed
Result: Group existence check failed for "oinstall"
- 35

.

Check: Group existence for "dba"
Node Name
Status
------------
Comment
------------------------
------------------------
rdb2
exists
passed
rdb1
exists
passed
Result: Group existence check passed for "dba"

Check: Membership of user "grid" in group "oinstall" [as Primary]
Node Name
User Exists
---------------- ------------
Group Exists User in Group Primary
------------
Comment
------------ ------------ ------------
rdb2
yes
yes
no
N/A
failed
rdb1
yes
yes
no
N/A
failed
Result: Membership check for user "grid" in group "oinstall" [as Primary] failed
Check: Membership of user "grid" in group "dba"
Node Name
User Exists
Group Exists User in Group Comment
----------------
------------
------------
------------
----------------
rdb2
yes
yes
yes
passed
rdb1
yes
yes
yes
passed
Result: Membership check for user "grid" in group "dba" passed

Check: Run level
Node Name
------------
run level
Required
------------------------
Comment
------------------------
----------
rdb2
3,5
passed
rdb1
3,5
passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
Node Name
Type
Available
------------
Required
------------
Comment
----------------
------------
rdb2
hard
65536
65536
---------------passed
rdb1
hard
65536
65536
passed
Result: Hard limits check passed for "maximum open file descriptors"
Check: Soft limits for "maximum open file descriptors"
Node Name
Type
Available
------------
Required
------------
Comment
----------------
------------
rdb2
soft
1024
1024
---------------passed
rdb1
soft
1024
1024
passed
Result: Soft limits check passed for "maximum open file descriptors"
Check: Hard limits for "maximum user processes"
Node Name
Type
Available
------------
Required
------------
Comment
----------------
------------
----------------
rdb2
hard
16384
16384
passed
rdb1
hard
16384
16384
passed
Result: Hard limits check passed for "maximum user processes"
- 36

.

Check: Soft limits for "maximum user processes"
Node Name
Type
Available
------------
Required
------------
Comment
----------------
------------
rdb2
soft
2047
2047
---------------passed
rdb1
soft
2047
2047
passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
x86_64
x86_64
passed
rdb1
x86_64
x86_64
passed
Result: System architecture check passed

Check: Kernel version
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
2.6.18-194.3.1.el5
2.6.18
passed
rdb1
2.6.18-194.3.1.el5
2.6.18
passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
Node Name
------------
Configured
------------------------
Required
------------------------
Comment
----------
rdb2
2700
250
passed
rdb1
2700
250
passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
Node Name
------------
Configured
------------------------
Required
------------------------
Comment
----------
rdb2
32000
32000
passed
rdb1
32000
32000
passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
Node Name
------------
Configured
------------------------
Required
------------------------
Comment
----------
rdb2
100
100
passed
rdb1
100
100
passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
Node Name
------------
Configured
------------------------
Required
------------------------
Comment
----------
rdb2
128
128
passed
rdb1
128
128
passed
Result: Kernel parameter check passed for "semmni"
- 37

.

Check: Kernel parameter for "shmmax"
Node Name
------------
Configured
------------------------
Required
------------------------
Comment
----------
rdb2
8589934592
536870912
passed
rdb1
8589934592
536870912
passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
Node Name
------------
Configured
------------------------
Required
------------------------
Comment
----------
rdb2
4096
4096
passed
rdb1
4096
4096
passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
Node Name
------------
Configured
------------------------
Required
------------------------
Comment
----------
rdb2
2097152
2097152
passed
rdb1
2097152
2097152
passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
Node Name
------------
Configured
------------------------
Required
------------------------
Comment
----------
rdb2
6815744
6815744
passed
rdb1
6815744
6815744
passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
Node Name
------------
Configured
------------------------
Required
------------------------
Comment
----------
rdb2
between 9000 & 65500
between 9000 & 65500
passed
rdb1
between 9000 & 65500
between 9000 & 65500
passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
Node Name
------------
Configured
------------------------
Required
------------------------
Comment
----------
rdb2
262144
262144
passed
rdb1
262144
262144
passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
Node Name
------------
Configured
------------------------
Required
------------------------
Comment
----------
rdb2
4194304
4194304
passed
rdb1
4194304
4194304
passed
Result: Kernel parameter check passed for "rmem_max"
- 38

.

Check: Kernel parameter for "wmem_default"
Node Name
------------
Configured
------------------------
Required
------------------------
Comment
----------
rdb2
262144
262144
passed
rdb1
262144
262144
passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
Node Name
------------
Configured
------------------------
Required
------------------------
Comment
----------
rdb2
1048586
1048576
passed
rdb1
1048586
1048576
passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
Node Name
------------
Configured
------------------------
Required
------------------------
Comment
----------
rdb2
1048576
1048576
passed
rdb1
1048576
1048576
passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make-3.81"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
make-3.81-3.el5
make-3.81
passed
rdb1
make-3.81-3.el5
make-3.81
passed
Result: Package existence check passed for "make-3.81"

Check: Package existence for "binutils-2.17.50.0.6"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
binutils-2.17.50.0.6-14.el5 binutils-2.17.50.0.6
passed
rdb1
binutils-2.17.50.0.6-14.el5 binutils-2.17.50.0.6
passed
Result: Package existence check passed for "binutils-2.17.50.0.6"

Check: Package existence for "gcc-4.1"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
gcc-4.1.2-48.el5
gcc-4.1
passed
rdb1
gcc-4.1.2-48.el5
gcc-4.1
passed
Result: Package existence check passed for "gcc-4.1"

Check: Package existence for "libaio-0.3.106 (i386)"
Node Name
------------
Available
Required
Comment
------------------------
------------------------
----------
rdb2
libaio-0.3.106-5 (i386)
libaio-0.3.106 (i386)
passed
rdb1
libaio-0.3.106-5 (i386)
libaio-0.3.106 (i386)
passed
Result: Package existence check passed for "libaio-0.3.106 (i386)"
- 39

.

Check: Package existence for "libaio-0.3.106 (x86_64)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
libaio-0.3.106-5 (x86_64) libaio-0.3.106 (x86_64)
passed
rdb1
libaio-0.3.106-5 (x86_64) libaio-0.3.106 (x86_64)
passed
Result: Package existence check passed for "libaio-0.3.106 (x86_64)"

Check: Package existence for "glibc-2.5-24 (i686)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
glibc-2.5-49 (i686)
glibc-2.5-24 (i686)
passed
rdb1
glibc-2.5-49 (i686)
glibc-2.5-24 (i686)
passed
Result: Package existence check passed for "glibc-2.5-24 (i686)"

Check: Package existence for "glibc-2.5-24 (x86_64)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
glibc-2.5-49 (x86_64)
glibc-2.5-24 (x86_64)
passed
rdb1
glibc-2.5-49 (x86_64)
glibc-2.5-24 (x86_64)
passed
Result: Package existence check passed for "glibc-2.5-24 (x86_64)"

Check: Package existence for "compat-libstdc++-33-3.2.3 (i386)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
compat-libstdc++-33-3.2.3-61 (i386) compat-libstdc++-33-3.2.3 (i386) passed
rdb1
compat-libstdc++-33-3.2.3-61 (i386) compat-libstdc++-33-3.2.3 (i386) passed
Result: Package existence check passed for "compat-libstdc++-33-3.2.3 (i386)"

Check: Package existence for "compat-libstdc++-33-3.2.3 (x86_64)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
compat-libstdc++-33-3.2.3-61 (x86_64) compat-libstdc++-33-3.2.3 (x86_64) passed
rdb1
compat-libstdc++-33-3.2.3-61 (x86_64) compat-libstdc++-33-3.2.3 (x86_64) passed
Result: Package existence check passed for "compat-libstdc++-33-3.2.3 (x86_64)"

Check: Package existence for "elfutils-libelf-0.125 (x86_64)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
elfutils-libelf-0.137-3.el5 (x86_64) elfutils-libelf-0.125 (x86_64) passed
rdb1
elfutils-libelf-0.137-3.el5 (x86_64) elfutils-libelf-0.125 (x86_64) passed
Result: Package existence check passed for "elfutils-libelf-0.125 (x86_64)"

Check: Package existence for "elfutils-libelf-devel-0.125"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
elfutils-libelf-devel-0.137-3.el5 elfutils-libelf-devel-0.125 passed
rdb1
elfutils-libelf-devel-0.137-3.el5 elfutils-libelf-devel-0.125 passed
Result: Package existence check passed for "elfutils-libelf-devel-0.125"
- 40

.

Check: Package existence for "glibc-common-2.5"
Node Name
------------
Available
------------------------
Required
Comment
------------------------
----------
rdb2
glibc-common-2.5-49
glibc-common-2.5
passed
rdb1
glibc-common-2.5-49
glibc-common-2.5
passed
Result: Package existence check passed for "glibc-common-2.5"

Check: Package existence for "glibc-devel-2.5 (i386)"
Node Name
------------
Available
------------------------
Required
Comment
------------------------
----------
rdb2
glibc-devel-2.5-49 (i386)
passed
rdb1
glibc-devel-2.5-49 (i386)
passed
Result: Package existence check passed for "glibc-devel-2.5 (i386)"

Check: Package existence for "glibc-devel-2.5 (x86_64)"
Node Name
------------
Available
------------------------
Required
Comment
------------------------
----------
rdb2
glibc-devel-2.5-49 (x86_64) glibc-devel-2.5 (x86_64)
passed
rdb1
glibc-devel-2.5-49 (x86_64) glibc-devel-2.5 (x86_64)
passed
Result: Package existence check passed for "glibc-devel-2.5 (x86_64)"

Check: Package existence for "glibc-headers-2.5"
Node Name
------------
Available
------------------------
Required
Comment
------------------------
----------
rdb2
glibc-headers-2.5
passed
rdb1
glibc-headers-2.5
passed
Result: Package existence check passed for "glibc-headers-2.5"

Check: Package existence for "gcc-c++-4.1.2"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
gcc-c++-4.1.2-48.el5
gcc-c++-4.1.2
passed
rdb1
gcc-c++-4.1.2-48.el5
gcc-c++-4.1.2
passed
Result: Package existence check passed for "gcc-c++-4.1.2"

Check: Package existence for "libaio-devel-0.3.106 (i386)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
libaio-devel-0.3.106-5 (i386) libaio-devel-0.3.106 (i386) passed
rdb1
libaio-devel-0.3.106-5 (i386) libaio-devel-0.3.106 (i386) passed
Result: Package existence check passed for "libaio-devel-0.3.106 (i386)"

Check: Package existence for "libaio-devel-0.3.106 (x86_64)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
libaio-devel-0.3.106-5 (x86_64) libaio-devel-0.3.106 (x86_64) passed
rdb1
libaio-devel-0.3.106-5 (x86_64) libaio-devel-0.3.106 (x86_64) passed
Result: Package existence check passed for "libaio-devel-0.3.106 (x86_64)"
- 41

.

Check: Package existence for "libgcc-4.1.2 (i386)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
libgcc-4.1.2-48.el5 (i386)
libgcc-4.1.2 (i386)
passed
rdb1
libgcc-4.1.2-48.el5 (i386)
libgcc-4.1.2 (i386)
passed
Result: Package existence check passed for "libgcc-4.1.2 (i386)"

Check: Package existence for "libgcc-4.1.2 (x86_64)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
libgcc-4.1.2-48.el5 (x86_64) libgcc-4.1.2 (x86_64)
passed
rdb1
libgcc-4.1.2-48.el5 (x86_64) libgcc-4.1.2 (x86_64)
passed
Result: Package existence check passed for "libgcc-4.1.2 (x86_64)"

Check: Package existence for "libstdc++-4.1.2 (i386)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
libstdc++-4.1.2-48.el5 (i386) libstdc++-4.1.2 (i386)
passed
rdb1
libstdc++-4.1.2-48.el5 (i386) libstdc++-4.1.2 (i386)
passed
Result: Package existence check passed for "libstdc++-4.1.2 (i386)"

Check: Package existence for "libstdc++-4.1.2 (x86_64)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
libstdc++-4.1.2-48.el5 (x86_64)
rdb1
libstdc++-4.1.2-48.el5 (x86_64) libstdc++-4.1.2 (x86_64) passed
libstdc++-4.1.2 (x86_64) passed
Result: Package existence check passed for "libstdc++-4.1.2 (x86_64)"

Check: Package existence for "libstdc++-devel-4.1.2 (x86_64)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
libstdc++-devel-4.1.2-48.el5 (x86_64) libstdc++-devel-4.1.2 (x86_64) passed
rdb1
libstdc++-devel-4.1.2-48.el5 (x86_64) libstdc++-devel-4.1.2 (x86_64) passed
Result: Package existence check passed for "libstdc++-devel-4.1.2 (x86_64)"

Check: Package existence for "sysstat-7.0.2"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
sysstat-7.0.2-3.el5
sysstat-7.0.2
passed
rdb1
sysstat-7.0.2-3.el5
sysstat-7.0.2
passed
Result: Package existence check passed for "sysstat-7.0.2"

Check: Package existence for "unixODBC-2.2.11 (i386)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
unixODBC-2.2.11-7.1 (i386) unixODBC-2.2.11 (i386)
passed
rdb1
unixODBC-2.2.11-7.1 (i386) unixODBC-2.2.11 (i386)
passed
Result: Package existence check passed for "unixODBC-2.2.11 (i386)"
- 42

.

Check: Package existence for "unixODBC-2.2.11 (x86_64)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
unixODBC-2.2.11-7.1 (x86_64) unixODBC-2.2.11 (x86_64) passed
rdb1
unixODBC-2.2.11-7.1 (x86_64) unixODBC-2.2.11 (x86_64) passed
Result: Package existence check passed for "unixODBC-2.2.11 (x86_64)"

Check: Package existence for "unixODBC-devel-2.2.11 (i386)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
unixODBC-devel-2.2.11-7.1 (i386) unixODBC-devel-2.2.11 (i386) passed
rdb1
unixODBC-devel-2.2.11-7.1 (i386) unixODBC-devel-2.2.11 (i386) passed
Result: Package existence check passed for "unixODBC-devel-2.2.11 (i386)"

Check: Package existence for "unixODBC-devel-2.2.11 (x86_64)"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
unixODBC-devel-2.2.11-7.1 (x86_64) unixODBC-devel-2.2.11 (x86_64) passed
rdb1
unixODBC-devel-2.2.11-7.1 (x86_64) unixODBC-devel-2.2.11 (x86_64) passed
Result: Package existence check passed for "unixODBC-devel-2.2.11 (x86_64)"

Check: Package existence for "ksh-20060214"
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
ksh-20100202-1.el5
ksh-20060214
passed
rdb1
ksh-20100202-1.el5
ksh-20060214
passed
Result: Package existence check passed for "ksh-20060214"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed
Check: Current group ID
Result: Current group ID check passed
Checking Core file name pattern consistency...
Core file name pattern consistency check passed.
Checking to make sure user "grid" is not in "root" group
Node Name
------------
Status
------------------------
Comment
------------------------
rdb2
does not exist
passed
rdb1
does not exist
passed
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
Node Name
------------
Available
------------------------
Required
------------------------
Comment
----------
rdb2
0022
0022
passed
rdb1
0022
0022
passed
Result: Default user file creation mask check passed
- 43

.

Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed
Checking daemon liveness...
Check: Liveness for "ntpd"
Node Name
Running?
------------------------------------
------------------------
rdb2
yes
rdb1
yes
Result: Liveness check passed for "ntpd"

Checking NTP daemon command line for slewing option "-x"
Check: NTP daemon command line
Node Name
Slewing Option Set?
------------------------------------
------------------------
rdb2
yes
rdb1
yes
Result:
NTP daemon slewing option check passed
Checking NTP daemon's boot time configuration, in file "/etc/sysconfig/ntpd", for slewing option "-x"
Check: NTP daemon's boot time configuration
Node Name
Slewing Option Set?
------------------------------------
------------------------
rdb2
yes
rdb1
yes
Result:
NTP daemon's boot time configuration check for slewing option passed
NTP common Time Server Check started...
NTP Time Server "218.50.1.115" is common to all nodes on which the NTP daemon is running
NTP Time Server "211.39.136.4" is common to all nodes on which the NTP daemon is running
Check of common NTP Time Server passed
Clock time offset check from NTP Time Server started...
Checking on nodes "[rdb2, rdb1]"...
Check: Clock time offset from NTP Time Server
Time Server: 218.50.1.115
Time Offset Limit: 1000.0 msecs
Node Name
-----------rdb2
Time Offset
-----------------------5.637
Status
-----------------------passed
- 44

.

rdb1
7.231
passed
Time Server "218.50.1.115" has time offsets that are within permissible limits for nodes "[rdb2, rdb1]".
Time Server: 211.39.136.4
Time Offset Limit: 1000.0 msecs
Node Name
------------
Time Offset
------------------------
Status
------------------------
rdb2
2.85
passed
rdb1
3.998
passed
Time Server "211.39.136.4" has time offsets that are within permissible limits for nodes "[rdb2, rdb1]".
Clock time offset check passed
Result: Clock synchronization check using Network Time Protocol(NTP) passed
Pre-check for cluster services setup was unsuccessful on all the nodes.
[grid@rdb1 grid]$
SWAP size fail oinstall group fail pass

.
- 45

.

4.
Installation step
4.1.
Grid Infrastracture Install
4.1.1. runInstaller
Grid Infrastructure grid user GRID binary directory runInstaller

.
(grid rdb1 .)
[grid@rdb1 grid]$ ./runInstaller
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB.
Actual 3180 MB
Passed
Checking swap space: must be greater than 150 MB.
Actual 7633 MB
Passed
Checking monitor: must be configured to display at least 256 colors

>>> Could not execute auto check for display colors using command /usr/bin/xdpyinfo. Check if the DISPLAY variable is set.
Failed <<<<
Some requirement checks failed. You must fulfill these requirements before
continuing with the installation,
Continue? (y/n) [n] y
>>> Ignoring required pre-requisite failures. Continuing...

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2010-12-27_02-32-02PM. Please wait ...[grid@rdb1 grid]$ You
can find the log of this install session at:
/rac/grid/oraInventory/logs/installActions2010-12-27_02-32-02PM.log
GUI
1. Select Installation Option
2. Select Installation Type
Install and Configure Grid Infrastructure for a Cluster
Advanced Installation
- 46

.

3. Select Product Languages
4. Grid Plug and Play Information
Korean
Cluster Name : rac1db
SCAN Name : rdb-scan
SCAN Port : 1521
Configure GNS :
5. Cluster Node Information
6. Specify Network Interface Usage
bond0 : Interconnect Network Private
Edit .
rdb1.goodus.com rdb1
rdb1-vip.goodus.com rdb1-vip
eth0 : Public Network Public
Add .
rdb2 , rdb2-vip
7. Storage Option Information
8. Create ASM Disk Group - I
ASM
OCR , VOTE ASM Disk Group

Disk Group Name : RAC_OV_DG
- 47

.
Redundancy : Normal

ASM Disk Change Discovery Path
Disk Discovery Path : /dev/oracleasm/disks
9. Create ASM Disk Group - II
10. Specify ASM Password I
OCR, VOTE disk Redundancy Normal 3 ASM
SYS , ASMSNMP password
Disk .
OV01, OV02, OV03
11. Specify ASM Password II
12. Failure Isolation Support
Password Warning : Yes
IPMI :
IPMI module .
IPMI management network .
URL :
13. Privileged Operating System Groups - I
14. Privileged Operating System Groups - II
OSDBA : dba
group Warning : Yes
- 48

.

OSOPER : dba
OSASM : asmadmin
15. Specify Installation Location
16. Create Inventory
Directory .
Inventory Directory : /rac/grid/oraInventory
Oracle Base : /rac/grid/base

Software Location : /rac/grid/product/11.2.0
17. Perform Prerequisite Checks - I
18. Perform Prereuisite Checks - II
requirement check
Swap Warning : Ignore All
19. Summary
20. Setup
Summary
- 49

.

21. Execute Configuration Scripts
22. orainstRoot.sh
rdb1 :
[root@rdb1 ~]# /rac/grid/oraInventory/orainstRoot.sh
Changing permissions of /rac/grid/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /rac/grid/oraInventory to dba.
The execution of the script is complete.
[root@rdb1 ~]#
rdb2 :
[root@rdb2 oracle]# /rac/grid/oraInventory/orainstRoot.sh
Changing permissions of /rac/grid/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /rac/grid/oraInventory to dba.
The execution of the script is complete.
[root@rdb2 oracle]#
23. root.sh (rdb1)
23. root.sh (rdb1)
[root@rdb1 ~]# /rac/grid/product/11.2.0/root.sh
profile reader peer cert TP
Running Oracle 11g root.sh script...
peer user cert

pa user cert
The following environment variables are set as:
Adding daemon to inittab
ORACLE_OWNER= grid
CRS-4123: Oracle High Availability Services has been started.
ORACLE_HOME= /rac/grid/product/11.2.0
ohasd is starting
CRS-2672: Attempting to start 'ora.gipcd' on 'rdb1'
Enter the full pathname of the local bin directory: [/usr/local/bin]:
CRS-2672: Attempting to start 'ora.mdnsd' on 'rdb1'
Copying dbhome to /usr/local/bin ...
CRS-2676: Start of 'ora.gipcd' on 'rdb1' succeeded
Copying oraenv to /usr/local/bin ...
CRS-2676: Start of 'ora.mdnsd' on 'rdb1' succeeded
Copying coraenv to /usr/local/bin ...
CRS-2672: Attempting to start 'ora.gpnpd' on 'rdb1'

CRS-2676: Start of 'ora.gpnpd' on 'rdb1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rdb1'
Creating /etc/oratab file...
CRS-2676: Start of 'ora.cssdmonitor' on 'rdb1' succeeded
Entries will be added to the /etc/oratab file as needed by
CRS-2672: Attempting to start 'ora.cssd' on 'rdb1'
Database Configuration Assistant when a database is created
CRS-2672: Attempting to start 'ora.diskmon' on 'rdb1'
Finished running generic part of root.sh script.
CRS-2676: Start of 'ora.diskmon' on 'rdb1' succeeded
Now product-specific root actions will be performed.
CRS-2676: Start of 'ora.cssd' on 'rdb1' succeeded
2010-12-27 14:47:38: Parsing the host name
CRS-2672: Attempting to start 'ora.ctssd' on 'rdb1'
2010-12-27 14:47:38: Checking for super user privileges
CRS-2676: Start of 'ora.ctssd' on 'rdb1' succeeded
2010-12-27 14:47:38: User has super user privileges

Using configuration parameter file:
ASM created and started successfully.
/rac/grid/product/11.2.0/crs/install/crsconfig_params
Creating trace directory
DiskGroup RAC_OV_DG created successfully.
LOCAL ADD MODE

Creating OCR keys for user 'root', privgrp 'root'..
clscfg: -install mode specified
Operation successful.
Successfully accumulated necessary OCR keys.
root wallet
root wallet cert
root cert export
CRS-2672: Attempting to start 'ora.crsd' on 'rdb1'
peer wallet
CRS-2676: Start of 'ora.crsd' on 'rdb1' succeeded
profile reader wallet
CRS-4256: Updating the profile
pa wallet
Successful addition of voting disk 95d59037a3054fcbbfcf2a4360a61250.
peer wallet keys
Successful addition of voting disk 8ceebb0896af4fdbbf64cf602e08f87e.
pa wallet keys
Successful addition of voting disk 95f52f6a470e4f56bf8df3178cf46594.
peer cert request
Successfully replaced voting disk group with +RAC_OV_DG.
pa cert request
CRS-4256: Updating the profile
peer cert
CRS-4266: Voting file(s) successfully replaced
pa cert
## STATE
peer root cert TP
--
profile reader root cert TP
-----
File Universal Id
-----------------
1. ONLINE
pa root cert TP
[RAC_OV_DG]
peer pa cert TP
2. ONLINE
File Name Disk group

--------- ---------
95d59037a3054fcbbfcf2a4360a61250 (/dev/oracleasm/disks/OV01)
8ceebb0896af4fdbbf64cf602e08f87e (/dev/oracleasm/disks/OV02)
- 50

.

pa peer cert TP
[RAC_OV_DG]
profile reader pa cert TP
3. ONLINE
95f52f6a470e4f56bf8df3178cf46594 (/dev/oracleasm/disks/OV03)
23. root.sh (rdb1)
24. root.sh (rdb2)
[RAC_OV_DG]
[root@rdb2 oracle]# /rac/grid/product/11.2.0/root.sh
Located 3 voting disk(s).
CRS-2673: Attempting to stop 'ora.crsd' on 'rdb1'

CRS-2677: Stop of 'ora.crsd' on 'rdb1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rdb1'
ORACLE_OWNER= grid
CRS-2677: Stop of 'ora.asm' on 'rdb1' succeeded
ORACLE_HOME= /rac/grid/product/11.2.0
CRS-2673: Attempting to stop 'ora.ctssd' on 'rdb1'

CRS-2677: Stop of 'ora.ctssd' on 'rdb1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rdb1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rdb1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rdb1'
CRS-2677: Stop of 'ora.cssd' on 'rdb1' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on 'rdb1'
CRS-2677: Stop of 'ora.gpnpd' on 'rdb1' succeeded
Creating /etc/oratab file...
CRS-2673: Attempting to stop 'ora.gipcd' on 'rdb1'
CRS-2677: Stop of 'ora.gipcd' on 'rdb1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rdb1'
CRS-2677: Stop of 'ora.mdnsd' on 'rdb1' succeeded
2010-12-27 14:55:16: Parsing the host name
2010-12-27 14:55:16: Checking for super user privileges
2010-12-27 14:55:16: User has super user privileges
Using configuration parameter file:
/rac/grid/product/11.2.0/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS
daemon on node rdb1, number 1, and is terminating
CRS-2672: Attempting to start 'ora.asm' on 'rdb1'
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2676: Start of 'ora.asm' on 'rdb1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rdb1'
CRS-2676: Start of 'ora.evmd' on 'rdb1' succeeded
CRS-2672: Attempting to start 'ora.RAC_OV_DG.dg' on 'rdb1'
CRS-2676: Start of 'ora.RAC_OV_DG.dg' on 'rdb1' succeeded
CRS-2672: Attempting to start 'ora.registry.acfs' on 'rdb1'
CRS-2676: Start of 'ora.registry.acfs' on 'rdb1' succeeded

rdb1
2010/12/27 14:54:04
/rac/grid/product/11.2.0/cdata/rdb1/backup_20101227_145404.olr
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rdb2'
Updating inventory properties for clusterware
CRS-2676: Start of 'ora.drivers.acfs' on 'rdb2' succeeded

Actual 7633 MB
Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /rac/grid/oraInventory
CRS-2672: Attempting to start 'ora.evmd' on 'rdb2'
'UpdateNodeList' was successful.
CRS-2676: Start of 'ora.evmd' on 'rdb2' succeeded
[root@rdb1 ~]#
rdb2
2010/12/27 14:58:50
/rac/grid/product/11.2.0/cdata/rdb2/backup_20101227_145850.olr
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /rac/grid/oraInventory
- 51

.
Actual 7633 MB
Passed

'UpdateNodeList' was successful.
[root@rdb2 oracle]#
25. Oracle Grid Infrastructure
26. Oracle Grid Infrastructure
log
====================================================
INFO: Checking Single Client Access Name (SCAN)...
INFO: Checking name resolution setup for "rdb-scan"...
INFO: ERROR:
INFO: PRVF-4664 : Found inconsistent name resolution entries for SCAN name "rdbscan"
INFO: ERROR:
INFO: PRVF-4657 : Name resolution setup check for "rdb-scan" (IP address:
192.168.20.147) failed
INFO: ERROR:
INFO: PRVF-4664 : Found inconsistent name resolution entries for SCAN name "rdbscan"
INFO: Verification of SCAN VIP and Listener setup failed
====================================================
OUI rdb-scan nslookup DNS
hosts install error
error .
4.2.
Install
4.2.1. Resource Service
crs resource service .
( )
[root@rdb2 oracle]# crsctl stat res -t
-------------------------------------------------------------------------------NAME
TARGET
STATE
SERVER
STATE_DETAILS
-------------------------------------------------------------------------------Local Resources
-------------------------------------------------------------------------------ora.RAC_OV_DG.dg
- 52

.

ONLINE ONLINE
rdb1
ONLINE ONLINE
rdb2
ONLINE ONLINE
rdb1
Started
ONLINE ONLINE
rdb2
Started
ONLINE ONLINE
rdb1
ONLINE ONLINE
rdb2
ora.asm
ora.eons
ora.gsd
OFFLINE OFFLINE
rdb1
OFFLINE OFFLINE
rdb2
ora.net1.network
ONLINE ONLINE
rdb1
ONLINE ONLINE
rdb2
ONLINE ONLINE
rdb1
ONLINE ONLINE
rdb2
ONLINE ONLINE
rdb1
ONLINE ONLINE
rdb2
ora.ons
ora.registry.acfs
-------------------------------------------------------------------------------Cluster Resources
-------------------------------------------------------------------------------ora.LISTENER_SCAN1.lsnr
1
ONLINE
ONLINE
rdb1
ora.oc4j
1
OFFLINE OFFLINE
ora.rdb1.vip
1
ONLINE
ONLINE
rdb1
ONLINE
ONLINE
rdb2
ONLINE
ONLINE
rdb1
ora.rdb2.vip
1
ora.scan1.vip
1
[root@rdb2 oracle]#
4.2.2. network
- 53

.
network ( public , vip , interconnect , scan-ip )
( )
rdb1
[grid@rdb1 ~]$ ifconfig
bond0

inet addr:10.10.20.45 Bcast:10.10.20.255
Mask:255.255.255.0

UP BROADCAST RUNNING MASTER MULTICAST
MTU:1500 Metric:1
RX packets:7652980 errors:0 dropped:0 overruns:0 frame:0

TX packets:6095879 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4754176882 (4.4 GiB) TX bytes:2734383099 (2.5 GiB)
eth0

inet addr:192.168.20.135
Bcast:192.168.20.255 Mask:255.255.255.0

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:201 Memory:f6000000-f6012800
eth0:1

eth0:2

eth1

UP BROADCAST RUNNING SLAVE MULTICAST
MTU:1500
Metric:1

- 54

.

Interrupt:233 Memory:fa000000-fa012800
eth2

MTU:1500
Metric:1

RX bytes:6306522 (6.0 MiB) TX bytes:0 (0.0 b)
Memory:f9fe0000-fa000000
lo
Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
[grid@rdb1 ~]$
rdb2
[root@rdb2 ~]# ifconfig
bond0
Link encap:Ethernet HWaddr 00:1B:78:95:AC:3A

inet addr:10.10.20.46 Bcast:10.10.20.255
Mask:255.255.255.0
inet6 addr: fe80::21b:78ff:fe95:ac3a/64 Scope:Link

UP BROADCAST RUNNING MASTER MULTICAST
MTU:1500 Metric:1

eth0
Link encap:Ethernet HWaddr 00:1B:78:95:AC:38

inet addr:192.168.20.136
Bcast:192.168.20.255 Mask:255.255.255.0
inet6 addr: fe80::21b:78ff:fe95:ac38/64 Scope:Link

- 55

.

RX bytes:34060438051 (31.7 GiB) TX bytes:614893628 (586.4 MiB)
eth0:1
Link encap:Ethernet HWaddr 00:1B:78:95:AC:38

eth1

MTU:1500
Metric:1

Interrupt:233 Memory:fa000000-fa012800
eth2

MTU:1500
Metric:1

RX bytes:6306870 (6.0 MiB) TX bytes:90 (90.0 b)
Memory:f9fe0000-fa000000
lo
Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX bytes:1041294337 (993.0 MiB) TX bytes:1041294337 (993.0 MiB)
[root@rdb2 ~]#
- 56

.

4.2.3. Process
process .
( )
[root@rdb1 ~]# ps -ef | grep oh
root
10981
1 0 13:45 ?
root
12921 12536 0 13:55 pts/0
root
27289
1 0 Dec24 ?
00:00:01 /rac/grid/product/11.2.0/bin/ohasd.bin reboot

00:00:00 grep oh
00:00:00 /bin/sh /etc/init.d/init.ohasd run
[root@rdb1 ~]#
[root@rdb1 ~]# ps -ef | grep d.bin
root
3020
1 0 14:48 ?
00:00:01 /rac/grid/product/11.2.0/bin/ohasd.bin reboot
grid
5504
1 0 14:50 ?
00:00:00 /rac/grid/product/11.2.0/bin/mdnsd.bin
grid
5515
1 0 14:50 ?
00:00:00 /rac/grid/product/11.2.0/bin/gipcd.bin
grid
5526
1 0 14:50 ?
00:00:00 /rac/grid/product/11.2.0/bin/gpnpd.bin
grid
5578
1 0 14:51 ?
00:00:00 /rac/grid/product/11.2.0/bin/ocssd.bin
root
5719
1 0 14:52 ?
00:00:00 /rac/grid/product/11.2.0/bin/octssd.bin
grid
5830
1 0 14:52 ?
00:00:00 /rac/grid/product/11.2.0/bin/oclskd.bin
root
5846
1 1 14:52 ?
00:00:01 /rac/grid/product/11.2.0/bin/crsd.bin reboot
grid
5866
1 0 14:52 ?
00:00:00 /rac/grid/product/11.2.0/bin/evmd.bin
root
5890
1 0 14:52 ?
00:00:00 /rac/grid/product/11.2.0/bin/oclskd.bin
root
6700 2363 0 14:54 pts/1
00:00:00 grep d.bin
[root@rdb1 ~]#
[root@rdb1 ~]# ps -ef | grep pmon
grid
22760
1 0 14:12 ?
root
23484 23343 0 14:13 pts/0
00:00:00 asm_pmon_+ASM1
00:00:00 grep pmon
[root@rdb1 ~]#
[root@rdb1 ~]# ps -ef | grep LISTENER
grid
6571
1 0 14:54 ?
00:00:00 /rac/grid/product/11.2.0/bin/tnslsnr LISTENER_SCAN1 -inherit
grid
7292
1 0 15:01 ?
00:00:00 /rac/grid/product/11.2.0/bin/tnslsnr LISTENER -inherit
root
15556 15521 0 15:31 pts/0
00:00:00 grep LISTENER
[root@rdb1 ~]#
4.3.
Database Installation
4.3.1. ASMCA
Database ASM diskgroup .
- 57

.

grid user
asmca .
( )
1. Disk Groups
2. Create Disk Group - I
Disk Group Name : DGDATA1

Redundancy : External (None)
Disk
3. Create Disk Group - II
4. Create Disk Group
5. DGBACKUP1
5.
- 58

.

4.3.2. Database Software Installation
oracle user database media runInstaller
(oracle rdb1 .)
[oracle@rdb1 database]$ ./runInstaller
Checking Temp space: must be greater than 120 MB.
Actual 5928 MB
Passed
Actual 7633 MB
Passed
Checking monitor: must be configured to display at least 256 colors

>>> Could not execute auto check for display colors using command /usr/bin/xdpyinfo. Check if the DISPLAY variable is set.
Failed <<<<
Some requirement checks failed. You must fulfill these requirements before
continuing with the installation,
Continue? (y/n) [n] y
>>> Ignoring required pre-requisite failures. Continuing...

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2010-12-24_10-37-32AM. Please wait ...[oracle@rdb1
database]$ You can find the log of this install session at:
/rac/grid/oraInventory/logs/installActions2010-12-24_10-37-32AM.log
1. Configure Security Updates
2. Select Installation Option
Email :
Install database software only
- 59

.

I wish to receive . :
KO16KSC5601 DB DBCA .
software only .
Create and Configure a database KO16MSWIN949
.
3. Node Selection
4. Select Product Languages
Real Application Clusters database installation
Korean
select all
5. Select Database Edition
6. Specify Installation Location
Enterprise Edition (4.29GB)
Oracle Base : /rac/oracle
Select Options : Choose Components
Software Location : /rac/oracle/product/11.2.0
7. Privileged Operating System Groups
8. Perform Prerequisite Checks - I
dba group
requirement .
- 60

.

9. Perform Prerequisite Checks - II
10. Summary
Swap .
Finish
Ignore All
11. Install Product
12. Execute Configuration scripts
root.sh cluster node .

13. Execute Configuration scripts (rdb1)
14. Execute Configuration scripts (rdb2)
[root@rdb1 ~]# /rac/oracle/product/11.2.0/root.sh
[root@rdb2 ~]# /rac/oracle/product/11.2.0/root.sh
ORACLE_OWNER= oracle
ORACLE_OWNER= oracle
ORACLE_HOME= /rac/oracle/product/11.2.0
ORACLE_HOME= /rac/oracle/product/11.2.0
The file "dbhome" already exists in /usr/local/bin.
The file "dbhome" already exists in /usr/local/bin.
Overwrite it? (y/n)
[n]: y
The file "oraenv" already exists in /usr/local/bin.
The file "oraenv" already exists in /usr/local/bin.

Overwrite it? (y/n)
[n]: y
Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.

The file "coraenv" already exists in /usr/local/bin.
Overwrite it? (y/n)
[n]:
Overwrite it? (y/n)
[n]:
Overwrite it? (y/n)
[n]: y

Finished product-specific root actions.

Finished product-specific root actions.
[root@rdb2 ~]#
- 61

.

[root@rdb1 ~]#
15. Finish
4.3.3. Database
KO16KSC5601 DBCA DB .
(oracle rdb1 dbca .)

1. Welcome
2. Welcome
Oracle Real Application Clusters database
Create a Database
3. Welcome
4. Welcome
General Purpose or Transaction Processing
Admin-Managed
- 62

.

Global Database Name : RAC1DB
SID Prefiex : RAC1DB
Select All
5. Enterprise Manager
6. Automatic Maintenance Tasks
Configure Enterprise Manager
Enable automatic maintenance tasks
7. Password
8. Storage Type and Locations
Password
Use Oracle-Managed Files

Database Area : +DGDATA1
9. ASM SNMP password
10. Recovery Options
ASMSNMP password
Specify Flash Recovery Area

Flash Recovery Area : +DGBACKUP1
Flash Recovery Area Size : 774009
Enable Archiving
- 63

.

11. Sample Schemas
12. Custom Scripts
No scripts to run
13. Memory
14. Sizing
Typical
Processes : 1000
Memory Size (SGA and PGA) : 5120MB

Use Automatic Memory Management
15. Character Sets
16. Connection Mode
Show recommended character sets only
Dedicated Server Mode
KO16KSC5601
National Character Set : UTF8
Default Language : Korean
Default Territory : Korea
- 64

.

17. Datafiles
18. Database Creation Options
Redo size : 100M
Finish
19. Setup
20. Finish
Exit
4.4.
Install
4.4.1. Resource Service
crs resource service .
( )
[root@rdb1 source]# crsctl stat res -t
-------------------------------------------------------------------------------NAME
TARGET
STATE
SERVER
STATE_DETAILS
-------------------------------------------------------------------------------Local Resources
-------------------------------------------------------------------------------ora.DGBACKUP1.dg
- 65

.

OFFLINE OFFLINE
rdb1
OFFLINE OFFLINE
rdb2
ora.DGDATA1.dg
ONLINE ONLINE
rdb1
ONLINE ONLINE
rdb2
ora.LISTENER.lsnr
ONLINE ONLINE
rdb1
ONLINE ONLINE
rdb2
ora.RAC_OV_DG.dg
ONLINE ONLINE
rdb1
ONLINE ONLINE
rdb2
ONLINE ONLINE
rdb1
Started
ONLINE ONLINE
rdb2
Started
ONLINE ONLINE
rdb1
ONLINE ONLINE
rdb2
ora.asm
ora.eons
ora.gsd
OFFLINE OFFLINE
rdb1
OFFLINE OFFLINE
rdb2
ora.net1.network
ONLINE ONLINE
rdb1
ONLINE ONLINE
rdb2
ONLINE ONLINE
rdb1
ONLINE ONLINE
rdb2
ONLINE ONLINE
rdb1
ONLINE ONLINE
rdb2
ora.ons
ora.registry.acfs
-------------------------------------------------------------------------------Cluster Resources
-------------------------------------------------------------------------------ora.LISTENER_SCAN1.lsnr
1
ONLINE
ONLINE
rdb2
ora.oc4j
1
OFFLINE OFFLINE
ora.rac1db.db
1
ONLINE ONLINE
rdb1
Open
- 66

.

2
ONLINE ONLINE
rdb2
Open
ora.rdb1.vip
1
ONLINE
ONLINE
rdb1
ONLINE
ONLINE
rdb2
ONLINE
ONLINE
rdb2
ora.rdb2.vip
1
ora.scan1.vip
1
[root@rdb1 source]#
- 67

.
5.
RAC OS / Hardware RAC .

11.2.0.2 RAC Muticast , HAIP Oracle Grid
.
, , Cluster solution IT infra

ORACLE RAC .

. , check ,
.
Oracle Certification
OS , hardware , oracle Certification .
.
solution Bug , Error

error , .
Test
Test plan Test .
- 68

.

Oracle 11g Rac On Multipath

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Oracle 11g Rac On Multipath

Uploaded by

Copyright:

Available Formats

ORACLE 11G R2 RAC

Copyright(C) 2004 Goodus Inc.

ORACLE 11g R2 RAC install ON Multipath Environment

System Overview ................................................................................................................. 4

H/W and OS Information ..........................................................................................8

Pre-Installation Steps ......................................................................................................... 15

Preparing the system ............................................................................................. 15

Host file Configuration ............................................................................................. 15

Kernel Sysctl Configuration ...................................................................................... 16

User resource limit .......................................................................................... 16

User Shell Configuration .......................................................................................... 18

Preparing Oracle ASM ............................................................................................ 25

Installation ASM LIBRARY ......................................................................................... 25

ASM Configuration ................................................................................................... 26

Create ASM VOLUME. .............................................................................................. 28

Install requirement ......................................................................................... 33

cluster verify utility ................................................................................................... 33

Installation step ............................................................................................................... 46

Grid Infrastracture Install .................................................................................... 46

ORACLE 11g R2 RAC install ON Multipath Environment

ORACLE 11g R2 RAC install ON Multipath Environment

Storage : storage RAID (0+1 )

RAC Storage , ASMLib

Linux OS EMC PowerPath RAC OS

ORACLE 11g R2 RAC install ON Multipath Environment

ORACLE 11g R2 RAC install ON Multipath Environment

[grid@rdb1 admin]$ ifconfig

Link encap:Ethernet HWaddr 00:18:71:7A:F0:84

[root@rdb1 network-scripts]# cat /proc/net/bonding/bond0

Bonding Mode: fault-tolerance (active-backup)

Slave Interface: eth1

Slave Interface: eth2

Oracle cross-cable direct , Switch

Server Switch , Switch Storage Line

ORACLE 11g R2 RAC install ON Multipath Environment

[root@rdb1 ~]# powermt display dev=all

Array failover mode: 1

-- I/O Path - -- Stats --Interf.

State Q-IOs Errors

ASM sda , sdah Native name emcpowera pseudo name

Storage RAID Level

Storage RAID 1+0 .

ORACLE 11g R2 RAC install ON Multipath Environment

H/W and OS Information

4cpu Dual-Core AMD Opteron(tm) Processor

4cpu Dual-Core AMD Opteron(tm) Processor

Total Local Disk

/ (/dev/cciss/c0d0p1 free: 6144948K /)

/ (/dev/cciss/c0d0p1 free: 6145700K /)

memory 8G Swap 8G Warnning

Local Disk cciss RAID-1 .

[root@rdb1 sbin]# hpacucli ctrl slot=1 show config

array A (SAS, Unused Space: 0 MB)

/dev/shm oracle 11g r2 AMM oracle

mount -t tmpfs tmpfs /dev/shm -o size=5120M

Red Hat Enterprise Linux Server release 5.5

Red Hat Enterprise Linux Server release 5.5

(Tikanga) (Red Hat 4.1.2-48)

(Tikanga) (Red Hat 4.1.2-48)

ORACLE 11g R2 RAC install ON Multipath Environment

Linux rdb1 2.6.18-194.3.1.el5 #1 SMP x86_64

x86_64 x86_64 x86_64 GNU/Linux

x86_64 x86_64 GNU/Linux