PURPOSE

The purpose of this note is to illustrate how to change a public hostname, public IP, a Virtual IP Add
ress (VIP), VIP hostname or other VIP attributes in an Oracle Clusterware/Grid Infrastructure enviro
nment.

SCOPE
Oracle Database 10g and 11g use VIPs (Virtual IP) in clustered environments for clients to connect to
the database. These VIPs are static IP addresses associated with (virtual) hostnames and resolved thro
ugh DNS (except when using 11gR2 GNS).

During the installation of the Oracle Clusterware users are prompted to enter a Virtual IP and Virtual
hostname for each of the node in the cluster. These are stored within the OCR (Oracle Cluster Registr
y) and different components within the HA framework depend on these VIPs. If for some reason the
need arises to change either the VIP, the VIP hostname, or the subnet, netmask etc, this procedure sho
uld be followed.

For changes associated with private network/cluster interconnect, please refer to Note 283684.1

DETAILS
Case I. Changing public hostname
Public hostname is recorded in OCR, it is entered during installation phase. It can not be modified aft
er the installation. The only way to modify public hostname is by deleting the node, then adding the n
ode back with a new hostname, or reinstalling the clusterware or following the clone procedure to reco
nfigure the clusterware.

Case II. Changing public IP or VIP only without changing interface, subnet or netmask or changing
MAC address only without changing anything else
If the change is only public IP or VIP address and the new ones are still in the same subnet, same inter
face, or if the change is only for public IP MAC address, IP/interface/subnet/netmask all remain the sa
me, nothing needs to be done at clusterware layer, all changes need to be done at OS layer to reflect th
e change.
1. Shutdown Oracle Clusterware stack
2. Modify the IP address at network layer, DNS and /etc/hosts file to reflect the change or modify the
MAC address at network layer
3. Restart Oracle Clusterware stack
Above change can be done in rolling fashion, eg: one node at a time.

Case III. Changing public network interface, subnet or netmask

If the change involves different subnet(netmask) or interface, delete the existing interface information
from OCR and add it back with the correct information is required. In the example here, the subnet is
changed from 10.2.156.0 to 10.2.166.0 via two separate commands - first a 'delif' followed by a
'setif':
% $CRS_HOME/bin/oifcfg/oifcfg delif -global <if_name>[/<subnet>]
% $CRS_HOME/bin/oifcfg/oifcfg setif -global <if_name>/<subnet>:public

For example:
% $CRS_HOME/bin/oifcfg delif -global eth0/10.X.156.0
% $CRS_HOME/bin/oifcfg setif -global eth0/10.X.166.0:public
Then make the change at OS layer. There is no requirement to restart Oracle clusterware unless OS ch
ange requires a node reboot. This can be done in rolling fashion.

Once public network is changed, its associated VIP and SCAN VIP are also required to change, refer
to CASE IV and CASE V.
Note: for 11gR2, above command requires clusterware RUNNING on ALL nodes, otherwise PRIF-33
and PRIF-32 will be reported, i.e.
[grid@racnode1 bin]$ ./oifcfg delif -global <if_name>/192.168.1.0
PRIF-33: Failed to set or delete interface because hosts could not be discovered
CRS-02307: No GPnP services on requested remote hosts.
PRIF-32: Error in checking for profile availability for host <nodename>2
CRS-02306: GPnP service on host "<nodename>2" not found.

Case IV. Changing VIPs associated with public network change

Planning for VIP changes
In general, a complete outage is only required for pre-10.2.0.3 release. From 10.2.0.3 onwards, the AS
M/database instance dependency on the VIP resource is removed, so the VIP could be modified witho
ut having to take down the ASM/database instance, only client connections to the database will be aff
ected when VIP is down. If the modification is a node specific, then only connection to that node will
be affected during the time of change.

Please follow Case III to ensure public network changes are made first. If there is a node reboot or Clu
sterware restart after the OS network change, vip will not start, please skip to the step "Modifying VIP
and its Associated Attributes".

Gathering Current VIP Configuration

1. Gather the existing setup
for 10g and 11gR1, as Oracle Clusterware owner:
$ srvctl config nodeapps -n <nodename> -a
eg:
$ srvctl config nodeapps -n <nodename>1 -a
VIP exists.: /<nodename>1-vip/101.XX.XX.184/255.255.254.0/<if_name>

for 11gR2, as Grid Infrastructure owner:

$ srvctl config nodeapps -a
eg:
$ srvctl config nodeapps -a
Network exists: 1/101.17.80.0/255.255.254.0/<if_name>, type static
VIP exists: /racnode1-vip/101.17.XX.184/101.17.80.0/255.255.254.0/<if_name>, hosting node
<nodename>1
VIP exists: /racnode2-vip/101.17.XX.186/101.17.80.0/255.255.254.0/<if_name>, hosting node
<nodename>2

2. Verify VIP status

10.2 and 11.1:
$ crs_stat -t

11.2:
$ crsctl stat res -t
- it should show VIPs are ONLINE
$ ifconfig -a
(netstat -in for HP and ipconfig /all for Windows)
- VIP logical interface is bound to the public network interface

Stopping Resources
3. Stop the nodeapps resources (and all dependent resources ASM/DB only if required):
10g and 11gR1, as Oracle Clusterware owner:
$ srvctl stop instance -d <db_name> -i <inst_name> (optional for 10.2.0.3+)
$ srvctl stop asm -n <node_name> (optional for 10.2.0.3+)
$ srvctl stop nodeapps -n <node_name>

eg,
$ srvctl stop instance -d <DBNAME> -i <INSTANCENAME>
$ srvctl stop asm -n <nodename>1
$ srvctl stop nodeapps -n <nodename>1

11gR2, as Grid Infrastructure owner:

$ srvctl stop instance -d <db_name> -n <node_name> (optional)
$ srvctl stop vip -n <node_name> -f

eg,
$ srvctl stop instance -d <DBNAME> -n <nodename>1
$ srvctl stop vip -n <nodename>1 -f

Note 1: The -f option is required for 11gR2 to stop listener resource, otherwise following error will
occur:
PRCR-1014 : Failed to stop resource ora.<nodename>1.vip
PRCR-1065 : Failed to stop resource ora.<nodename>1.vip
CRS-2529: Unable to act on 'ora.<nodename>1.vip' because that would require stopping or relocating
'ora.LISTENER.lsnr', but the force option was not specified
...

4. Verify VIP is now OFFLINE and the interface is no longer bound to the public network interface
$ crs_stat -t (or $ crsctl stat res -t for 11gR2)
$ ifconfig -a
(netstat -in for HP and ipconfig /all for windows)

Modifying VIP and Its Associated Attributes

5. Determine the new VIP IP/subnet/netmask or VIP hostname, make the network change on OS first,
ensure the new VIP is registered in DNS or modified in /etc/hosts (for Unix/Linux) and \WINDOWS\
System32\drivers\etc\hosts file (for Windows). If the network interface is changed, ensure the new int
erface is available on the server before proceeding with the modification.
For example:
New VIP is: 110.XX.XX.11 <nodename>1-nvip
new subnet is 110.11.70.0
new netmask is 255.255.255.0
new interface is <if_name>

6. Modify the VIP resource, as root user:

# srvctl modify nodeapps -n <node> -A <new_vip_address or new_vip_hostn ame>/<netmask>/<[if1
[if2...]]>
eg:
# srvctl modify nodeapps -n <nodename>1 -A <nodename>1-nvip/255.255.255.0/<if_name>

Note 1: Starting with 11.2, the VIPs depend on the network resource (ora.net1.network), the OCR onl
y records the VIP hostname or the IP address associated with the VIP resource. The network attribute
s(subnet/netmask/interface) are recorded with the network resource. When the nodeapps resource is m
odified, the network resoure(ora.net1.network) attributes are also modified implicitly.
From 11.2.0.2 onwards, if only subnet/netmask/interface change is required, network resource can be
modified directly via srvctl modify network command.
as root user:
# srvctl modify network -k <network_number>] [-S <subnet>/<netmask>[/if1[|if2...]]
eg:
# srvctl modify network -k 1 -S 110.XX.XX.0/255.255.255.0/<if_name>
There is no need to modify VIP or SCAN if other attributes are not changed.

Note 2: For 12.1.0.1 release, due to unpublished Bug 16608577 - CANNOT ADD SECOND PUBLIC
INTERFACE IN ORACLE 12.1, the srvctl modify network command fails with:
# srvctl modify network -k 1 -S 110.XX.XX.0/255.255.255.0/<if_name>
PRCT-1305 : The specified interface name "<if_name>2" does not match the existing network interfa
ce name "<if_name>1"

Workaround is to modify network resource with an empty interface name, then modify it again with
the desired interface name, eg:
# srvctl modify network -k 1 -S 110.XX.XX.0/255.255.255.0
# srvctl modify network -k 1 -S 110.XX.XX.0/255.255.255.0/<if_name>2

The bug has been fixed in 12.1.0.2 and above.

* A special case for 11gR2 modifying the VIP hostname only without changing the IP address.

For example: only VIP hostname changes from <nodename>1-vip to <nodename>1-nvip, IP and other
attributes remain the same.

If IP address is not changed, above modify command will not change the USR_ORA_VIP value in
'crsctl stat res ora.<nodename>.vip -p' output. Please use the following command:
# crsctl modify res ora.<nodename>.vip -attr USR_ORA_VIP=<nodename>1-nvip

Verify the changes for USR_ORA_VIP field:

# crsctl stat res ora.<nodename>1.vip -p |grep USR_ORA_VIP

Note: For Windows platform, the interface name needs to be in quote (") if there is a space in between,
eg:
As administrator user or software install user:
srvctl modify nodeapps -n <nodename> -A 110.XX.XX.11/255.255.255.0/"Local Area Connection 1"

7. Verify the change

$ srvctl config nodeapps -n <node> -a (10g and 11gR1)
$ srvctl config nodeapps -a (11gR2)
eg:
$ srvctl config nodeapps -n <nodename>1 -a
VIP exists.: /<nodename>1-nvip/110.11.70.11/255.255.255.0/<if_name>2

Restarting Resources
8. Start the nodeapps and the other resources
10g and 11gR1, as Oracle Clusterware owner:
$ srvctl start nodeapps -n <node_name>
$ srvctl start asm -n <node_name> (optional for 10.2.0.3+)
$ srvctl start instance -d <dbanme> -i <inst> (optional for 10.2.0.3+)

eg:
$ srvctl start nodeapps -n <nodename>1
$ srvctl start asm -n <nodename>1
$ srvctl start instance -d <DBNAME> -i <INSTANCENAME>1
11gR2, as Grid Infrastructure owner:

$ srvctl start vip -n <node_name>

$ srvctl start listener -n <node_name>
$ srvctl start instance -d <db_name> -n <node_name> (optional)

eg,
$ srvctl start vip -n <nodename>1
$ srvctl start listener -n <nodename>1
$ srvctl start instance -d <DBNAME> -n <nodename>1
Note: if the network attributes are changed, i.e. netmask changed, restart the nodeapps

9. Verify the new VIP is ONLINE and bind to the public network interface
$ crs_stat -t (or $ crsctl stat res -t for 11gR2)
$ ifconfig -a
(netstat -in for HP or ipconfig /all for windows)

10. Repeat the same steps for the rest nodes in the cluster only if the similar change is required.

Others
11. Modify listener.ora, tnsnames.ora and LOCAL_LISTENER/REMOTE_LISTENER parameter to
reflect the VIP change if necessary.

Note, LOCAL_LISTENER for ASM and DB are set automatically by Grid Infrastructure. The VIP ch
ange in LOCAL_LISTENER should take effect automatically. Due to Bug 22824602, under some rac
e condition, the LOCAL_LISTENER for ASM instance might not reflect the VIP change. The workar
ound is to restart the Clusterware on the affected node.

Case V. Change SCAN VIP associated with public network change

With Grid Infrastructure 11gR2, SCAN and SCAN VIP are introduced for client connections. To
modify the SCAN VIP, please refer to

Note 952903.1 How to update the IP address of the SCAN VIP resources (ora.scan<n>.vip)
Note 972500.1 How to Modify SCAN Setting or SCAN Listener Port after Installation

Note: if rolling back the change is required, repeat the commands which have been run, replace the
new value with original value to restore the original configuration.

GOAL
The purpose of this note is to describe how to change or update the private network (cluster_interconn
ect) information in Oracle Clusterware.

It may be necessary to change or update interface names, or subnet associated with an interface if ther
e is a network change affecting the servers, or if the original information that was input during the inst
allation was incorrect. It may also be the case that for some reason, the Oracle Interface Configuratio
n Assistant ('oifcfg') did not succeed during the installation.

Please refer to Note 276434.1 for modifying public network and VIP associated information and refer
to Note 1386709.1 for basics of IPv4 subnet and Oracle Clusterware.
Note: for Oracle Engineered system (Exadata) and Oracle Database Appliance (ODA), please do not
make such changes following this note.

SOLUTION
Network information(interface, subnet and role of each interface) for Oracle Clusterware is managed b
y 'oifcfg', but actual IP address for each interfaces are not, 'oifcfg' can not update IP address informatio
n. 'oifcfg getif' can be used to find out currently configured interfaces in OCR:

% $CRS_HOME/bin/oifcfg getif
<interfacename>0 10.X.XXX.0 global public
<interfacename>1 192.XXX.X.0 global cluster_interconnect
On Unix/Linux systems, the interface names are generally assigned by the OS, and standard names var
y by platform. For Windows systems, see additional notes below. Above example shows currently inte
rface <interfacename>0 is used for public with subnet 10.2.156.0, and <interfacename>1 for cluster_i
nterconnect/private with subnet 192.168.0.0.

The 'public' network is for database client communication (VIP also uses the same network though it's
stored in OCR as separate entry), whereas the 'cluster_interconnect' network is for RDBMS/ASM cach
e fusion. Starting with 11gR2, cluster_interconnect is also used for clusterware heartbeats - this is sign
ificant change compare to prior release as pre-11gR2 uses the private nodename that were specified at
installation time for clusterware heartbeats.

If the subnet or interface name for 'cluster_interconnect' interface is incorrect, it needs to be changed
as crs/grid user.

Case I. Changing private hostname

In pre-11.2 Oracle Clusterware, private hostname is recorded in OCR, it can not be updated. Generally
private hostname is not required to change. Its associated IP can be changed. The only way to change
private hostname is by deleting/adding nodes, or reinstall Oracle Clusterware.

In 11.2 Grid Infrastructure, private hostname is no longer recorded in OCR and there is no dependenc
y on the private hostname. It can be changed freely in /etc/hosts.

Case II. Changing private IP only without changing network interface, subnet and netmask or changin
g private IP MAC address only without changing anything else
For example, private IP is changed from 192.XXX.X.10 to 192.XXX.X.21, network interface name an
d subnet remain the same, or MAC address is changed, private IP address/interface name/subnet/ netw
ork all remain the same.

Simply shutdown Oracle Clusterware stack on the node where change required, make IP or MAC mod
ification at OS layer as required (eg: /etc/hosts, network config etc) for private network, restart Oracle
Clusterware stack will complete the task.

Case III. Changing private network MTU only

For example, private network MTU is changed from 1500 to 9000 (enable jumbo frame), network inte
rface name and subnet remain the same.
1. Shutdown Oracle Clusterware stack on all nodes
2. Make the required network change of MTU size at OS network layer, ensure private network is ava
ilable with the desired MTU size, ping with the desired MTU size works on all cluster nodes
3. Restart Oracle Clusterware stack on all nodes

Case IV. Changing private network interface name, subnet or netmask

Note: When the netmask is changed but the subnet ID doesn't change, for example:
The netmask is changed from 255.255.0.0 to 255.255.255.0 with private IP like 192.168.0.x, the subne
t ID remains the same as 192.168.0.0, the network interface name is not changed.
Shutdown Oracle Clusterware stack on all cluster nodes where change required, make IP modification
at OS layer (eg: OS network config etc) for private network, restart Oracle Clusterware stack on all no
des will complete the task. Please note, this change can not be done in rolling manner.

When the netmask is changed, the associated subnet ID is often changed. Oracle only store network in
terface name and subnet ID in OCR, not the netmask. Oifcfg command can be used for such change,
oifcfg commands only require to run on 1 of the cluster node, not all.

A. For pre-11gR2 Oracle Clusterware

1. Use oifcfg to add the new private network information, delete the old private network information:
% $ORA_CRS_HOME/bin/oifcfg/oifcfg setif -global <if_name>/<subnet>:cluster_interconnect
% $ORA_CRS_HOME/bin/oifcfg/oifcfg delif -global <if_name>[/<subnet>]]

For example:
% $ORA_CRS_HOME/bin/oifcfg setif -global <interfacename>3/192.168.2.0:cluster_interconnect
% $ORA_CRS_HOME/bin/oifcfg delif -global <interfacename>1/192.168.1.0

To verify the change

% $ORA_CRS_HOME/bin/oifcfg getif
eth0 10.X.XXX.0 global public
eth3 192.XXX.2.0 global cluster_interconnect

2. Shutdown Oracle Clusterware stack

As root user: # crsctl stop crs

3. Make required network change at OS level, /etc/hosts file should be modified on all nodes to reflect
the change.
Ensure the new network is available on all cluster nodes:
% ping <private hostname/IP>
% ifconfig -a on Unix/Linux
or
% ipconfig /all on windows

4. restart the Oracle Clusterware stack

As root user: # crsctl start crs

Note: If running OCFS2 on Linux, one may also need to change the private IP address that OCFS2 is
using to communicate with other nodes. For more information, please refer to Note 604958.1

B. For 11gR2 Oracle Clusterware and 12c Cluster without Flex ASM
As of 11.2 Grid Infrastructure, the private network configuration is not only stored in OCR but also in
the gpnp profile. If the private network is not available or its definition is incorrect, the CRSD process
will not start and any subsequent changes to the OCR will be impossible. Therefore care needs to be
taken when making modifications to the configuration of the private network. It is important to
perform the changes in the correct order. Please also note that manual modification of gpnp profile is
not supported.

Please take a backup of profile.xml on all cluster nodes before proceeding, as grid user:
$ cd $GRID_HOME/gpnp/<hostname>/profiles/peer/
$ cp -p profile.xml profile.xml.bk
1. Ensure Oracle Clusterware is running on ALL cluster nodes in the cluster

2. As grid user:
Get the existing information. For example:
$ oifcfg getif
<interfacename>1 100.17.10.0 global public
<interfacename>0 192.168.0.0 global cluster_interconnect

Add the new cluster_interconnect information:

$ oifcfg setif -global <interface>/<subnet>:cluster_interconnect

For example:
a. add a new interface bond0 with the same subnet
$ oifcfg setif -global bond0/192.168.0.0:cluster_interconnect

b. add a new subnet with the same interface name but different subnet or new interface name
$ oifcfg setif -global eth0/192.65.0.0:cluster_interconnect
or
$ oifcfg setif -global eth3/192.168.1.96:cluster_interconnect

1). This can be done with -global option even if the interface is not available yet, but this can not be do
ne with -node option if the interface is not available, it will lead to node eviction.

2). If the interface is available on the server, subnet address can be identified by command:
$ oifcfg iflist
It lists the network interface and its subnet address. This command can be run even if Oracle Clusterw
are is not running. Please note, subnet address might not be in the format of x.y.z.0, it can be x.y.z.24,
x.y.z.64 or x.y.z.128 etc. For example,
$ oifcfg iflist
lan1 18.1.2.0
lan2 10.2.3.64 << this is the private network subnet address associated with private network IP:
10.2.3.XX

3). If it is for adding a 2nd private network, not replacing the existing private network, please ensure
MTU size of both interfaces are the same, otherwise instance startup will report error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if MTU failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpcini2
ORA-27303: additional information: requested interface lan1:801 has a different MTU (1500) than
lan3:801 (9000), which is not supported. Check output from ifconfig command

4). For 11gR2 and higher, it is not recommended to set cluster_interconnects in ASM or Database
spfile or pfile. If this parameter is set for any reason, it needs to be changed to the new private IP in
spfile or pfile prior to clusterware shutdown, otherwise it will result a failure during restart due to the
interconnect mismatch.
Verify the change:
$ oifcfg getif

3. Shutdown Oracle Clusterware on all nodes and disable the Oracle Clusterware as root user:
# crsctl stop crs
# crsctl disable crs
4. Make the network configuration change at OS level as required, ensure the new interface is
available on all nodes after the change.
$ ifconfig -a
$ ping <private hostname>

5. Enable Oracle Clusterware and restart Oracle Clusterware on all nodes as root user:

# crsctl enable crs

# crsctl start crs

6. Remove the old interface if required:

$ oifcfg delif -global <interfacename>[/<subnet>]
eg:
$ oifcfg delif -global eth0/192.168.0.0

C. For 12c and 18c Oracle Clusterware with Flex ASM

Please review above section B and pay attention to the Note section, take a backup as follows:
Please take a backup of profile.xml on all cluster nodes before proceeding, as grid user:
$ cd $GRID_HOME/gpnp/<hostname>/profiles/peer/
$ cp -p profile.xml profile.xml.bk
1. Ensure Oracle Clusterware is running on ALL cluster nodes in the cluster

2. As grid user:
Get the existing information. For example:
$ oifcfg getif
<interfacename>1 100.17.10.0 global public
<interfacename>0 192.168.0.0 global cluster_interconnect,asm
Above example shows network eth0 is used for both cluster_interconnect and ASM network.

Add the new cluster_interconnect information:

$ oifcfg setif -global <interface>/<subnet>:cluster_interconnect[,asm]
For example:
a. add a new interface bond0 with the same subnet
$ oifcfg setif -global bond0/192.168.0.0:cluster_interconnect,asm

b. add a new subnet with the same interface name but different subnet or new interface name
$ oifcfg setif -global eth0/192.68.10.0:cluster_interconnect,asm
or
$ oifcfg setif -global eth3/192.168.1.96:cluster_interconnect,asm
If different network is used for private network and ASM network, then modify them accordingly.

3. As ASMLISTENER is using the private network, modifying the private network will affect ASMLI
STENER. It is required to add a new ASMLISTENER with the new network configuration. Skip this
step if the subnet for the ASM network is not changed.
3.1. Add a new ASMLISTENER (for example: ASMNEWLSNR_ASM) with the new subnet, as grid
user:
$ srvctl add listener -asmlistener -l <new ASM LISTENER NAME> -subnet <new subnet>
eg:
$ srvctl add listener -asmlistener -l ASMNEWLSNR_ASM -subnet 192.168.10.0
3.2. Drop the existing ASMLISTENER (ASMLSNR_ASM in this example) and remove the dependen
cy, as grid user:
$ srvctl update listener -listener ASMLSNR_ASM -asm -remove -force
$ lsnrctl stop ASMLSNR_ASM
Note. -force option is required, otherwise the following error will occur:
$ srvctl update listener -listener ASMLSNR_ASM -asm -remove
PRCR-1025 : Resource ora.ASMLSNR_ASM.lsnr is still running
$ srvctl stop listener -l ASMLSNR_ASM
PRCR-1065 : Failed to stop resource ora.ASMLSNR_ASM.lsnr
CRS-2529: Unable to act on 'ora.ASMLSNR_ASM.lsnr' because that would require stopping or reloc
ating 'ora.asm', but the force option was not specified

3.3 Verify the configuration

$ srvctl config listener -asmlistener
$ srvctl config asm

4. Shutdown Oracle Clusterware on ALL nodes and disable the Oracle Clusterware as root user:
# crsctl stop crs
# crsctl disable crs

5. Make the network configuration change at OS level as required, ensure the new interface is availa
ble on all nodes after the change.
$ ifconfig -a
$ ping <private hostname>

6. Enable Oracle Clusterware and restart Oracle Clusterware on all nodes as root user:
# crsctl enable crs
# crsctl start crs

7. Remove the old interface if required:

$ oifcfg delif -global <interfacename>[/<subnet>]
eg:
$ oifcfg delif -global <interfacename>0/192.168.0.0

Something to note for 11gR2+

1. If underlying network configuration has been changed, but oifcfg has not been run to make the sam
e change, then upon Oracle Clusterware restart, the CRSD will not be able to start.

The crsd.log will show:

2010-01-30 09:22:47.234: [ default][2926461424] CRS Daemon Starting
..
2010-01-30 09:22:47.273: [ GPnP][2926461424]clsgpnp_Init: [at clsgpnp0.c:837] GPnP client pid=
7153, tl=3, f=0
2010-01-30 09:22:47.282: [ OCRAPI][2926461424]clsu_get_private_ip_addresses: no ip addresses
found.
2010-01-30 09:22:47.282: [GIPCXCPT][2926461424] gipcShutdownF: skipping shutdown, count 2,
from [ clsinet.c : 1732], ret gipcretSuccess (0)
2010-01-30 09:22:47.283: [GIPCXCPT][2926461424] gipcShutdownF: skipping shutdown, count 1,
from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)
[ OCRAPI][2926461424]a_init_clsss: failed to call clsu_get_private_ip_addr (7)
2010-01-30 09:22:47.285: [ OCRAPI][2926461424]a_init:13!: Clusterware init unsuccessful : [44]
2010-01-30 09:22:47.285: [ CRSOCR][2926461424] OCR context init failure. Error: PROC-44: Error
in network address and interface operations Network address and interface operations error [7]
2010-01-30 09:22:47.285: [ CRSD][2926461424][PANIC] CRSD exiting: Could not init OCR, code:
44
2010-01-30 09:22:47.285: [ CRSD][2926461424] Done.
Above errors indicate mismatches between OS setting (oifcfg iflist) and gpnp setting profile.xml.

Workaround: restore the OS network configuration back to the original status, start Oracle Clusterwa
re. Then follow above steps to make the changes again.
If the underlying network has not been changed, but oifcfg setif has been run with a wrong subnet add
ress or interface name, same issue will happen.

2. If any one node is down in the cluster, oifcfg command will fail with error:
$ oifcfg setif -global bond0/192.168.0.0:cluster_interconnect
PRIF-26: Error in update the profiles in the cluster
Workaround: start Oracle Clusterware on the node where it is not running. Ensure Oracle Clusterware
is up on all cluster nodes. If the node is down for any OS reason, please remove the node from the clus
ter before performing private network change.

3. If a user other than Grid Infrastructure owner issues above command, it will fail with same error:
$ oifcfg setif -global bond0/192.168.0.0:cluster_interconnect
PRIF-26: Error in update the profiles in the cluster
Workaround: ensure to login as Grid Infrastructure owner to perform such command.

4. From 11.2.0.2 onwards, if attempt to delete the last private interface (cluster_interconnect) without
adding a new one first, following error will occur:
PRIF-31: Failed to delete the specified network interface because it is the last private interface
Workaround: Add new private interface first before deleting the old private interface.

5. If Oracle Clusterware is down on the node, the following error is expected:

$ oifcfg getif
PRIF-10: failed to initialize the cluster registry
Workaround: Start the Oracle Clusterware on the node

Notes for Windows Systems

The syntax for changing the interfaces on Windows/RAC cluster is the same as on Unix/Linux, but the
interface names will be slightly different. On Windows systems, the default names assigned to the inte
rfaces are generally named such as:
Local Area Connection
Local Area Connection 1
Local Area Connection 2

If using an interface name that has space in it, the name must be enclosed in quotes. Also, be aware th
at it is case sensitive. For example, on Windows, to set cluster_interconnect:
C:\oracle\product\10.2.0\crs\bin\oifcfg setif -global "Local Area Connection
1"/192.168.1.0:cluster_interconnect

However, it is best practice on Windows to rename the interfaces to be more meaningful, such as rena
ming them to 'ocwpublic' and 'ocwprivate'. If interface names are renamed after Oracle Clusterware
is installed, then you will need to run 'oifcfg' to add the new interface and delete the old one, as descr
ibed above.

You can view the available interface names on each node by running the command:
oifcfg iflist -p -n
This command must be run on each node to verify the interface names are defined the same.
Ramifications of Changing Interface Names Using oifcfg
For the Private interface, the database will use the interface stored in the OCR and defined as a
'cluster_interconnect' for cache fusion traffic. The cluster_interconnect information is available at
startup in the alert log, after the parameter listing - for example:

For pre 11.2.0.2:

Cluster communication is configured to use the following interface(s) for this instance
192.XXX.X.1

For 11.2.0.2+: (HAIP address will show in alert log instead of private IP)
Cluster communication is configured to use the following interface(s) for this instance
169.XXX.XX.97

If this is incorrect, then instance is required to restart once the OCR entry is corrected. This applies to
ASM instances and Database instances alike. On Windows systems, after shutting down the instance,
it is also required to stop/restart the OracleService<SID> (or OracleASMService<ASMSID> before
the OCR will be re-read.

oifcfg usage
To see the full options of oifcfg, simply type:
$ <CRS_HOME>/bin/oifcfg

Case V. Add or remove cluster_interconnect for 11gR2 and above with HAIP
1. To add another private network into existing cluster using HAIP, as grid user:
$ oifcfg setif -global <interface>/<subnet>:cluster_interconnect
For example:
$ oifcfg setif -global <interfacename>/192.XXX.XX.0:cluster_interconnect

Shutdown CRS on ALL nodes, then restart CRS on ALL nodes for HAIP to pick up the new interface.
It is insufficient to restart CRS in rolling manner.
2. To remove a private network from a cluster with HAIP, as grid user:
$ oifcfg delif -global <interfacename>
For example:
$ oifcfg delif -global <interfacename>

HAIP will failover to the remaining interface and clusterware/database continue to function after the
interface removal.

To remove the extra HAIP interface, it is required to shutdown CRS on ALL nodes, then restart CRS
on ALL nodes. It is insufficient to restart CRS in rolling manner.