Clusterware Networking

When designing your cluster architecture you will need to consider the physical networking that will be required.
You will need to consider the network adapters, high availability architectures, use of jumbo frames and
assignment of IP addresses. et!s look at each of these topics in more detail.

Network Adapters
"ach server to be used in an #racle $lusterware configuration will require a minimum of two separate network
components. %hese two adapters are
%he public network adapter
%he private cluster interconnect
Public Network Adapter
%he first is the public network adapter. %he public network adapter must support %$P&IP and is generally connected
to the local internet via a switch. 'dministrators and users will connect to the database via the public network
adapter. (any servers have more than one public network adapter and these are often )bonded*. +onding provides
the ability to combine each public network adapter into one logical network adapter. +onding provides more
throughput and thus better performance. Performance on the public network is quite important for the overall
performance of the database. ,igh performing networks make for faster file transfers -for e.ample backups or
e.port&import/ and faster transfer of large amounts of application data.
Private Network Adapter
%he second required network adapter is required to act as the cluster interconnect. %he cluster interconnect should
support %$P&IP and 01P -when using 02I3 or Windows platforms/. %he cluster interconnect is a local connection
between the individual servers, and should not be part of a public network. %he public interconnects are for local
traffic between the clustered servers. +ecause the private interconnect traffic should not mi. with public traffic then
the private interconnect connections should have their own switches. $ross4over cables are not supported, and you
must ensure that your configuration is supported by #racle. +onded interfaces are supported for improved
throughput.
#nce you have configured #racle $lusterware and installed 5'$ databases you may wish to confirm that the 5'$
databases are using the correct adapter for the Private 2etwork 'dapter. You can use the
67$08%"59I2%"5$#22"$% view from any node of the 5'$ database. You can also use the #racle oradebug
command to determine this information. You can access oradebug through the #racle 8:;Plus utility. You can
then use the oradebug setmypid and oradebug ipc commands to determine if the correct adapter is being used for
the private interconnect. ,ere is an e.ample<
/u01>sqlplus / as sysdba
SQL> ORADEBUG SETMYPID
SQL> oad!bu" #p$
No Single Points of Failure Please
It is very important to avoid a single point of failure when it comes to your networking. %his includes making sure
you have redundant interfaces, redundant switches, redundant everything to ensure that no one piece can break.
Administering RAC Network Settings
#racle $lusterware&5'$ administrators will want to be able to monitor and administer network settings related to
the cluster. %ools like oifcfg and svrctl are available for these tasks.
0sing oifcfg you can<
ist the interfaces available to the cluster as seen in this e.ample<
O#%$%" #&#s' (p )*
1etermine the public and private interfaces that have been configured for use as seen in this e.ample<
O#%$%" "!'#%
You can use the srvctl command to determine the 6IP hostname, address, subnet mask and interface name as
seen in this e.ample<
S+$'l $o*," *od!apps )a
Changing the Private Interconnect Adapter
5ecall that all 5'$ $lusters have at least two types of network adapters. %he first is the public adapter and the
second is the private interconnect. You may have cause to change the specification for the interconnect adapter
-such as changing the IP address/. ,ere are the instructions to do just that<
=. og4on to one node of the cluster. 'dd the new global interface specification using the oifcfg command as
seen here<
O#%$%" s!'#% ("lobal !'-./1/.000.001$lus'!2#*'!$o**!$'
>. $heck the changes to make sure they took effect with the oifcfg command as seen here<
O#%$%" "!'#%
?. 8top the cluster using the crsctl command. ".ecute this on each node of the cluster<
3s$'l s'op $s
@. #n each node, use the ifconfig command to assign the network address to the adapters<
I%$o*," !'-. 1/.000.014 *!'5as6 .440.440.4400 boad$as' 1/.000.0.44
A. 5emove the previous adapter specification using the oifcfg command as seen in this e.ample<
O#%$%" d!l#% ("lobal !'-1/1/.01780100
Do *o' !9!$u'! '-#s s'!p u*'#l you a! su! '-a' '-! !pla$!5!*' #*'!%a$! -as b!!* pop!ly add!d #*'o you
$lus'!0
B. 5estart the $lusterware using the crsctl command<
3s$'l s'a' $s

High Availabilit Networking Architectures
Your networking configurations should be architected for high availability. In a best case situation, both the private
interconnect and the public interconnect would have bonded, redundant network cards and redundant switches. 's
a reminder, the interface names on each node of the cluster -and slots/ need to be the same. %hus if the private
interconnect is on eth= on node one, it needs to be on eth= on the remaining nodes.

!umbo Frames
"thernet packages network messages in )frames* which can be of a variable siCe. %he frame siCe is called the (%0
or the ma.imum transmission unit. When a message is larger than the (%0 siCe -typically =ADD bytes/, then it is
split into multiple messages. %his splitting of messages incurs additional overhead and network traffic. %his can
lead to 5'$ performance problems. In #racle two main factors can influence the ma.imum siCe of a message<
1+9+#$E98IF" ; 1+9GI"9(0%I9+#$E95"'19$#02% -(+5$/ impacts the ma.imum siCe of a
message for the global cache. Gor e.ample, a 1+9+#$E98IF" of Hk and a (+5$ of ?> will result in a
ma.imum message siCe of >ABk. %his will result in appro.imately =ID separate packets. 0sing Jumbo
frames -Kk packet siCes/ would result in only appro.imately >H packets.

P'5""9"3"$0%I#29("88'L"98IF" -defaults to =B?H@ bytes/ determines the ma.imum siCe of a
message used in parallel e.ecution. %hese messages can range from >k to over B@k.
#ne solution to these issues is to configure the cluster interconnect to use Jumbo Grames. When you configure
Jumbo Grames the "thernet frame siCe can be as large as Kk in siCe. $onfiguring for Jumbo Grames requires some
careful configuration. %he following steps are required to configure jumbo frames -these steps might vary based on
your hardware and operating system/<
$heck #racle (etalink for more information on implementing Jumbo Grames. $heck for information
specific to your hardware, operating system and any bugs that might e.ist.
$onfigure the hostMs network adapter with a persistent (%0 siCe of KDDD. #n a 0ni. system you might use
the command ifconfig 4mtu KDDD.
$heck your vendor 2I$ configuration requirements. %he 2I$ may well require some configuration.
"nsure your switches will support the larger frame siCe and configure the '2 switches to increase the
(%0 for Jumbo Grame support.
You can use traceroute for some basic configuration testing as seen in this e.ample where we do a
traceroute with a Kk packet siCe and a KDD= byte packet siCe<
'a$!ou'! (: l#*u90105y-os'0$o5 /000
'a$!ou'! (: l#*u90105y-os'0$o5 /001
2ote that Jumbo Grames do not have any standard that they adhere to. 's a result Jumbo Grames the
interoperability between switches can be problematic and can require advanced networking skills to troubleshoot.
'lso keep in mind that the smallest (%0 used by any device in a given network path will determine the ma.imum
(%0 for all traffic travelling along that path. 's with all changes make sure that you completely test your
configuration before implementing it in production.

Assignment of IP Addresses
+efore installing #racle $lusterware you will want to define the public and private adapter IP addresses. %here are
two options you can choose with respect to IP address assignment. You can choose static IP addresses for each
adapter which will require the assignment of several static IP addresses. 'lternatively you can use #racle Lrid
2aming 8ervice -L28/ which will require the assignment of one static IP address. L28 will then assign dynamic IP
addresses to the nodes of the cluster.
When installing #racle $lusterware you will be prompted to identify the private and public adapters. 'lso you
should be aware if you perform any maintenance or change the network adapters that the same adapter name -for
e.ample "%,=/ should be used for the public and private adapters on each server.
When assigning the IP address for the private adapter you need to make sure that you use a non4routable IP
address, and make sure that all interfaces are configured for the correct speed, duple. and so on.

"he #IP$ %NS and Single Client Access Name &SCAN'
8everal networking components are associated with #racle $lusterware and #racle 5'$. %hese include the 6IP, the
L28 and the 8$'2.

"he #IP
%he purpose of the 6irtual IP -6IP/ is to provide for high availability. ' 6IP address is assigned to a given node.
When that node goes down, the 6IP is moved to one of the surviving nodes. %his new 6IP is re4arped. 's a result
new connections will get sent to a surviving node, and connections connected to the failed node will receive an
error and attempt to re4connect -assuming the application handles the errors gracefully which can be a problem/.
Gor clients connected to the failed node, this can speed up the timeout much quicker since they don!t have to wait
for a %$P&IP timeout, thus facilitating a quicker failover.
When assigning an IP address to the 6IP make sure that it is on the same subnet as the default gateway.

%rid Naming Service &%NS'
L28 is essentially a 128 for the #racle $luster. L28 is an optional networking component of #racle $lustereware
==g 5elease > that provides for dynamic name resolution in the cluster. L28 removes the need to have static IP
addresses assigned to the network nodes.L28 also removes the need to request 6IPs if the cluster changes.
Instead only one static IP is required which is the L28 virtual IP address. In conjunction with your 1,$P server,
L28 will then assign dynamic IP addresses&6IPs to the cluster nodes as required. %he end result is L28 makes it
easier to add and remove cluster nodes.

SCAN
#racle $lusterware ==g 5elease > introduced the 8ingle $lient 'ccess 2ame -8$'2/. %he 8$'2 address provides a
single address for clients to use when accessing the cluster. +efore 8$'2, if the cluster changed, then the client
%282'("8.#5' files -or other tns connect strings like "asy $onnect strings/ would need to change. %he 8$'2 is a
fully qualified domain name that typically resolves to three separate IP addresses. %he 8$'2 provides a stable
address that does not change regardless of the number of nodes on the cluster. %he 8$'2 also provides for a
highly available address, much like the 6IPs. %he 8$'2 simplifies network connectivity for applications using "asy
$onnect strings or J1+$ thin client connection strings.
If you are using L28, then you have 1,$P support, then the L28 will assign the addresses to the 8$'2
dynamically. If you are not using L28 then the 8$'2 will be defined in the 128 server and access requests will be
resolved to one of three different IP addresses by the 128. %hese addresses resolve to the 8$'2 listener, and the
8$'2 listener will resolve the connect request to one of the nodes of the cluster. %he 8$'2 listener will load
balance the connection requests to the least busy cluster as they come in.
%he following diagram displays the relationships between L28, 8$'2, isteners and the #racle $luster and its
associated databases.

Administering SCAN Resources
's an $lusterware&5'$ administrator you may need to administer 8$'2 resources on your cluster. %his can
include<
'dding a 8$'2 6IP resource.
Us! '-! s+$'l $o55a*d 'o add a S3A; <IP !sou$! as s!!* #* '-#s !9a5pl! =-!! =! add a *!= !sou$! $all!d
a$0>)s$a*0 ;o'! '-a' '-#s $o55a*d =#ll $!a'! '-! sa5! *u5b! o% S3A; <IP !sou$!s as '-!! a! '-! *u5b! o%
IP add!ss!s '-a' S3A; !sol+!s 'o0 I% you a! us#*" D?3P o GP;P '-!* Oa$l! =#ll $o*,"u! > S3A; <IP@s0 O'-!
op'#o*s #* '-! $o55a*d allo= you 'o d!,*! '-! *!'=o6 *u5b!A sub*!' a*d so o*1
S+$'l add s$a* (* a$0>)s$a*
5emoving a 8$'2 6IP resource.
Us! '-! s+$'l $o55a*d 'o !5o+! a* !9#s'#*" S3A; <IP %o5 '-! $lus'! as s!!* #* '-#s !9a5pl!1
S+$'l !5o+! s$a* )%
You may want to add a 8$'2 listener resource. %o do so use the srvctl command as seen in the following
e.ample. 2ote in the e.ample that we are assigning the listener to a non4default port<
S+$'l add s$a*2l#s'!*! (p 74001
You may need to remove $lusterware resources from the 8$'2 listeners. 0se the srvctl command to
perform this activity as seen here<
S+$'l !5o+! s$a*2l#s'!*! )%
You may need to modify the 8$'2 6IP to that it matches another 8$'2 6IP. %o do so use srvctl again as
seen here<
5od#%y s$a* (* *!=2s$a*2*a5!
If you have changed the 8$'2 6IP configuration you will need to update the 8$2 listeners too. 0se the
srvctl command to perform this action<
S+$'l 5od#%y s$a*2l#s'!*! )u
'fter making changes to the 8$'2 listener, you can verify those changes with the srvctl command as seen
in this e.ample<
S+$'l $o*," s$a*2l#s'!*!