12/20/2013

Document 787420.1

Cluster Interconnect in Oracle 10g and 11gR1 RAC (Doc ID 787420.1)

Modified: 17-Dec-2013

Type: BULLETIN

In this Document
Purpose
Scope
Details
Physical Layout of the Private Interconnect
Why Do We Need a Private Interconnect ?
Interconnect Failure
Interconnect High Availability
Private Interconnect for Oracle Clusterware
Private Interconnect for RAC
Identification of the Private Interconnect for RAC
Private Interconnect for ASM
Different Interconnects for Clusterware and RAC
CLUSTER_INTERCONNECTS Parameter
Changing the Private Interconnect
References

APPLIES TO:
Oracle Database - Enterprise Edition - Version 10.1.0.2 to 11.1.0.7 [Release 10.1 to 11.1]
Information in this document applies to any platform.
Oracle Server Enterprise Edition - Version: 10.1.0.2 to 11.1.0.7
Last Updated: 20-Feb-2012
***Checked for relevance on 20-Aug-2010***

PURPOSE
The purpose of this note is to make DBAs and System Administrators familiar with the concept of the cluster or
private interconnect and its usage in Oracle Clusterware and Oracle Real Application Clusters

SCOPE
The main audience is DBAs and System Administrators. This note applies to Oracle Clusterware and Oracle Real
Application Clusters releases 10.1, 10.2 and 11.1.

DETAILS
The term 'Private Interconnect' or 'Cluster Interconnect' is the private communication link between cluster nodes.
For Oracle Real Application Clusters and Oracle Clusterware we can differentiate between
1. Physical interface (including NIC, cables and switches)
2. Private Interconnect used by Oracle Clusterware
3. Private Interconnect used by Oracle Real Application Clusters (RAC)

Physical Layout of the Private Interconnect

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=q1iylk9uz_325&id=787420.1

1/4

12/20/2013

Document 787420.1

The basic requirements are described in the Installation Guide for each platform. Additional information about
certification can be found on Metalink Certify.
The interconnect as identified by both subnet number and interface name must be configured on all clustered
nodes.
A switch between the clustered nodes is an absolute requirement.

Why Do We Need a Private Interconnect ?

Clusterware uses the interconnect for cluster synchronization (network heartbeat) and daemon communication
between the the clustered nodes. This communication is based on the TCP protocol.
RAC uses the interconnect for cache fusion (UDP) and inter-process communication (TCP). Cache Fusion is the
remote memory mapping of Oracle buffers, shared between the caches of participating nodes in the cluster. The
volume and traffic patterns of this type of data, shared between nodes can vary greatly depending on the
applications.
There are some vendor specific protocol exceptions in Oracle 10g.

Interconnect Failure
The private interconnect is the critical communication link between nodes and instances. Network errors will
negatively impact Oracle Clusterware communication as well as RAC communication and performance.
Private interconnect failures are recognized by Oracle Clusterware and result in what is known as a 'split-brain' or
subdivided cluster.
A subdivided cluster can result in data corruption, consequently immediate action is taken to resolve this condition.
Interconnect failures, therefore, result in a node or subset of nodes in the cluster shutting down. In the case of two
equally sized sub clusters, it is basically random which sub cluster will survive, and a customer's architecture and
design should be based on this. At this time the Oracle Clusterware happens to use the node numbers to resolve
this, but this could change in the future.
RAC instances will wait for the end of cluster reconfiguration to start their own reconfiguration.

Interconnect High Availability

It is an Oracle Best Practice to make the private interconnect highly available. Depending on the operating system
and vendor you can use OS network drivers such as bonding, teaming, IPMP, Etherchannel, APA, MultiPrivNIC. For
that purpose the interconnect should be configured across two NICs as well as two switches for complete
redundancy so that the cluster can survive a single point of failure.
The setup of the highly available interconnect is transparent to Oracle Clusterware and RAC. This means that an
underlying failure is handled by the OS or networking drivers that manage the interfaces. Oracle software does not
recognize the change underneath, because the failover is handled by the operating system transparently to Oracle.

Private Interconnect for Oracle Clusterware

The private node name determines the interface being used for Oracle Clusterware and is defined during
installation of the Clusterware.
With 3rd party Vendor Clusterware in place Oracle Clusterware should be configured to use the same interconnect
(often referred to as the heartbeat) as the underlying vendor cluster software.
There exist three ways to identify the private node name after installation:
olsnodes -n -p can be used to identify the private node name.
[oracle@mynode1 ~]$ olsnodes -n -p
mynode1 1 mynode1-priv
mynode2 2 mynode2-priv
[oracle@mynode1 ~]$

You may also check the private node name in the ocrdump output.
[SYSTEM.css.node_numbers.node1.privatename]
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=q1iylk9uz_325&id=787420.1

2/4

12/20/2013

Document 787420.1

ORATEXT : node1-priv

ocssd.log has a line with clssnmClusterListener

[ CSSD]2009-02-23 03:09:06.945 [3086] >TRACE: clssnmClusterListener: Listening on
(ADDRESS=(PROTOCOL=tcp)(HOST=node1-priv)(PORT=49895))

Although the Oracle Universal Installer allows IP addresses when prompted for the private node name you should
always use the host names defined in the hosts file or DNS. This enables you to change the IP addresses when you
need to move the server to a different IP range. You would have to reinstall Oracle Clusterware when you used IPaddresses instead of host names and you are changing the server's IP adresses.

Private Interconnect for RAC

The database uses the private interconnect for communication and cache fusion. The private interconnect is also
referred to as "The Cluster Interconnect". Oracle highly recommends that the database and Oracle Clusterware
share the same interconnect.
The instances get the private interconnect definition from
spfile or init.ora when the parameter CLUSTER_INTERCONNECTS was set
Oracle Cluster Regsitry (OCR) defined during the installation (default)
automatically from platform defaults if not defined in the spfile/init.ora and OCR.
A value in the spfile or init.ora overrides a definition in the OCR.

Identification of the Private Interconnect for RAC

The value of the private interconnect for an instance can be identified using
1. The views V$CLUSTER_INTERCONNECTS and V$CONFIGURED_INTERCONNECTS
V$CLUSTER_INTERCONNECTS displays one or more interconnects that are being used for
cluster communication.
V$CONFIGURED_INTERCONNECTS displays all the interconnects that Oracle is aware of. This
view aims to answer the question on where Oracle found the information about a specific
interconnect.
These views are not available in Oracle 10g 10.1.
2. The alert.log
CLUSTER_INTERCONNECTS parameter is set
Value read from Cluster Registry
Interface type 1 Database Team 10.0.0.0 configured from OCR for use
as a cluster interconnect

Interconnect finally used

Cluster communication is configured to use the following
interface(s) for this instance
10.0.0.1

Note that there might be a difference to the interconnect defined in the OCR because
the value was overridden by setting the CLUSTER_INTERCONNECTS parameter or when
Oracle selected another interconnect because the one in the OCR was not available.
Protocol
Unix and Linux
cluster interconnect IPC version:Oracle UDP/IP (generic)

Windows:
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=q1iylk9uz_325&id=787420.1

3/4

12/20/2013

Document 787420.1

cluster interconnect IPC version:Oracle 9i Winsock2 TCP/IP IPC

Private Interconnect for ASM

Oracle ASM must use the same interconnect as Oracle Clusterware.

Different Interconnects for Clusterware and RAC

Oracle Clusterware does not monitor interfaces other than the one specified via the private host name during
installation. If RAC instances run with an interconnect that is different from the Oracle Clusterware interconnect any
failure will remain undetected by Oracle Clusterware. The instances will detect this failure themselves and clear the
situation by a mechanism called Instance Membership Recovery (IMR). Instance evictions due to IMR happen after
10 minutes.
This configuration should be avoided.

CLUSTER_INTERCONNECTS Parameter
This parameter overrides the value configured in the OCR. You may specify multiple IP addresses in
CLUSTER_INTERCONNECTS which will cause all defined interfaces to be used. Keep in mind that a failure of one of
those interfaces will cause the instance to fail. The failed instance can only be restarted when all the interfaces are
fixed or you start all the instances without the faulty interface.

Changing the Private Interconnect

Clusterware
Change of the IP Address: change the IP address in the hosts file and/or DNS and make sure that
ASM and the database also use the same interconnect
Change of the private node name used by Oracle Clusterware: requires a reinstall of Oracle
Clusterware
Real Application Clusters
OCR: Use Note 283684.1 How to Change Interconnect/Public Interface IP or Subnet in Oracle
Clusterware
CLUSTER_INTERCONNECTS: shutdown all instances, change the IP address and restart the instances

Database - RAC/Scalability Community

To discuss this topic further with Oracle experts and industry peers, we encourage you to review, join or start a
discussion in the My Oracle Support Database - RAC/Scalability Community

REFERENCES
NOTE:283684.1 - How to Modify Private Network Information in Oracle Clusterware
NOTE:341788.1 - Recommendation for the Real Application Cluster Interconnect and Jumbo Frames
NOTE:368464.1 - How to Setup IPMP as Cluster Interconnect

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=q1iylk9uz_325&id=787420.1

4/4