GOAL

The purpose of this document is to demonstrate how to integrate a 10g (or 11.1) database with 11gR2
Grid Infrastructure.

With 11gR2 Grid, client connections are supposed to use the Single Client Access Name (SCAN) to
connect to instances in a RAC cluster.

While 10g/11gR1 sqlnet does not support certain aspects of this new feature, it is still possible to con
figure 10g/11gR1 clients and database instances to integrate with the 11gR2 Grid infrastructure using
the SCAN. This document provides an example on how to implement this.

Please note that there is no need to use the SCAN VIPs when running a 10g/11gR1 database under 11
gR2 Grid, it is perfectly fine to continue use the node listeners, both for instance registration as well as
client connect strings.

The solution outlined in this document requires that the SCAN VIPs are fixed, hence it cannot be used
if the SCAN VIPs are dynamically allocated via GNS (using DHCP).

SOLUTION
Pre-11gR2 clients/database instances do not support multiple IP addresses associated with the same h
ost name (SCAN), consequently the 11gR2 syntax should not be used for the remote_listener parame
ter or the client connect string as it probably would only register with/connect to the first VIP of the S
CAN only.

The VIPs of the SCAN need to be determined, the SCAN can have between 1 to 3 IP addresses
associated with it:

oragrid@<HOSTNAME1> $ nslookup <CLUSTER_NAME>_scan.rac.example.com

Server: dns1
Address: 10.xxx.10.183#53

Name: <CLUSTER_NAME>_scan.rac.example.com
Address: 10.xxx.10.200
Name: <CLUSTER_NAME>_scan.rac.example.com
Address: 10.xxx.10.201
Name: <CLUSTER_NAME>_scan.rac.example.com
Address: 10.xxx.10.202

Using above information, TNS aliases for instance parameters local_listener / remote_listener can be
created. As these aliases are used by the 10g/11gR1 database instances they need to go into the tnsna
mes.ora accessed by these instances - default location is $ORACLE_HOME/network/admin. Please
note that the alias for 'local_listener' uses the node VIPs, while the alias for 'remote_listener' uses the
SCAN VIPs.
This matches the setup for 11gR2 databases - see the documentation.

LISTENER_<INST_NAME_1> =
(ADDRESS = (PROTOCOL = TCP)(HOST = <HOSTNAME1>-vip)(PORT = 1521))

LISTENER_<INST_NAME_2> =
(ADDRESS = (PROTOCOL = TCP)(HOST = <HOSTNAME2>-vip)(PORT = 1521))
LISTENERS_GRID =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.xxx.10.200)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.xxx.10.201)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.xxx.10.202)(PORT = 1521))
)

Please note:
Typically, clients of pre-11.2 databases were configured with an ADDRESS_LIST listing node VIPs.
When any clients are still configured with connect strings that list node VIPs, the REMOTE_LISTEN
ERS parameter of the pre-11.2 databases need to list every node VIP plus every SCAN VIP. Only whe
n all the client connect strings have been updated to no longer use node VIPs, the node VIPs can be re
moved from REMOTE_LISTENERS.

Set the local_listener/remote_listener parameters for the 10g/11gR1 database instances, please note th
at the value for 'sid' is case sensitive and needs to match the actual values in the SPFILE:

alter system set local_listener='LISTENER_<INSTNAME_1>' scope=spfile sid='<INST_NAME_1>';

alter system set local_listener='LISTENER_<INSTNAME_2>' scope=spfile sid='<INST_NAME_2>';
alter system set remote_listener='LISTENERS_GRID' scope=spfile sid='*';

There should be a CRS service configured in the OCR for clients to connect this database - e.g.:

$ srvctl status service -d <DB_NAME>

Service <DB_NAME>_SCAN is running on instance(s) <INST_NAME_1>, <INST_NAME_2>

A corresponding client connect string could look like this:

<DB_NAME>_VIA_SCAN =
(DESCRIPTION =
(ADDRESS_LIST =
(LOAD_BALANCE = yes)
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.xxx.10.200)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.xxx.10.201)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.xxx.10.202)(PORT = 1521))
)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = <DB_NAME>_SCAN)
)
)

To check if the 10g/11gR1 database instances have registered their services names with the SCAN
listeners the command 'lsnrctl services scan_listener' can be used, however since the SCAN can run
on any node of the cluster, it first needs to be established on which node the SCAN listeners are
currently running:

$ srvctl status scan_listener

SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node <HOSTNAME2>
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node <HOSTNAME1>
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node <HOSTNAME1>

Please note that CRS dynamically adds entries for the SCAN listeners to the listener.ora under $GR
ID_HOME/network/admin, so for the following command to succeed, it needs to be executed from
the GRID_HOME on the node that currently runs the SCAN listener and environment variables like
ORACLE_HOME, TNS_ADMIN (if set) need to point to the GRID_HOME.

All SCAN listeners should show the service names of the 10g/11gR1 database instances - e.g.:

grid@<HOSTNAME2> $ lsnrctl service LISTENER_SCAN1

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 05-FEB-2010 14:26:33
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
Services Summary...
Service "<DB_NAME>" has 2 instance(s).
Instance "<INST_NAME_1>", status READY, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:0 refused:0 state:ready
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=<HOSTNAME1>-vip)(PORT=1521))
Instance "<INST_NAME_2>", status READY, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:0 refused:0 state:ready
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=<HOSTNAME2>-vip)(PORT=1521))
...
Service "<DB_NAME>_SCAN" has 2 instance(s).
Instance "<INST_NAME_1>", status READY, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:0 refused:0 state:ready
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=<HOSTNAME1>-vip)(PORT=1521))
Instance "<INST_NAME_2>", status READY, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:0 refused:0 state:ready
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=<HOSTNAME2>-vip)(PORT=1521))

SCAN Listeners Error

This issue happened on a RAC database, version 12.1. This RAC database has two instances, one on
server1 and one on server2. While we were working on this, server1 had two of the SCAN listeners
(SCAN1 and SCAN3) while server2 had the last one (SCAN2).

The issue started when we saw that “crsctl status res -t” on server1 showed both scan listeners in “Not
All Endpoints Registered,STABLE” state. Checking the SCAN listener logs showed “Address already
in use“.

This is a quite self explanatory message, some other process is listening on the same IP and port. Rem
ember, only the combination of IP and port must be unique. If the server has multiple IP addresses, pr
ocesses can listen on the same IP and different ports or different IPs and the same port. But processes
cannot listen on the same IP and same port. With this we had to find the process. To find that we can
use the netstat command and look for processes in “LISTENING” mode on the IP and port we are loo
king for. In this case we looked for the SCAN1 and SCAN3 IP addresses, and surprisingly the process
that listened on this was the local listener process. Why was it listening on the SCAN IPs? Good ques
tion.

At this point we went to check if MOS has something interesting to say. MOS is really great when you
have a well define message or error to look for, so I logged in and looked for the “Not All Endpoints
Registered,STABLE” message. I managed to find a note talking about listeners configured manually
in the listener.ora, but this was not the case here. Then I found note 1667873.1. This note explains that
this might happen because of incorrect hosts file configuration. And indeed this was the case.

If you work with Linux/Unix (don’t know how it behaves in Windows), duplicate entries in /etc/hosts
are illegal. Although usually it doesn’t cause any problems, sometimes it does (and RAC is very sensi
tive regarding host naming and IPs).

What does a duplicate entry mean? The hosts file is responsible for translating hostnames to IP. It doe
sn’t care if many hosts are translated into the same IP (in the same line or different ones), but when the
same hostname is translated into different IPs it’s a problem.

I’ve seen quite a lot of servers where the real hostname is configured in the 127.0.0.1 line as well as th
e real IP in /etc/hosts. For example:
127.0.0.1 localhost localhost.localdomain server1 server1.mydomain
10.10.10.10 server1 server1.mydomain
This duplicate translation (server1 is translated to both 127.0.0.1 and 10.10.10.10) is illegal.

So this was the case here. The solution was to remove the server name from 127.0.0.1 and restart the
listeners (local and SCAN). Problem solved, right? Hmmm, not quite.

Connection Problem
After we removed the duplicate hostname from /etc/hosts, we restarted the listeners and the message
indeed disappeared. But then we realized that clients cannot connect to the database.

Clients who tried to connect to the database got “ORA-12520: TNS:listener could not find available
handler for requested type of server”. So fixing the SCAN listeners issue introduced a new problem.

We checked and saw that everything is configured correctly, on both client side and server side. We
tried to ping the SCAN name and tnsping the tns entry, both worked, so now we need to analyze whe
re the problem is.

When running tnsping, the client goes to the host and port (as appears in the tnsnames.ora file) and tri
es to connect to a listener. If a listener responds, we see an “OK” message and the time it took. If not,
we will get an error. However, tnsping doesn’t try to connect to the database, only to the listener. The
fact that tnsping succeeded means that we could get to the SCAN listener, but clearly there is a proble
m further down the road.

At that point we enabled sqlnet trace for the client side (according to MOS note 395525.1). Looking at
the trace file immediately solved the mystery. When connecting to RAC, we first get to the SCAN list
ener, but the SCAN listener doesn’t connect us to the database. It is simply a mechanism created for si
mplicity, management and load balancing. To continue with the connection, the SCAN listener redire
cts us to one of the local listeners and the local listener is the one that actually connects us to the datab
ase. This redirection is based on the hostname, so our client got a redirection message to the VIP name
(server1-vip) in order to continue the connection. When we checked, the client couldn’t resolve the na
me server1-vip, so it returned an error. The solution was to fix the DNS (or local /etc/hosts), so the cli
ent will be able to resolve the VIP hostnames of the RAC servers.

Final Note
As a final note, I’d like to explain why fixing the SCAN listener issue caused the connection problem.
As I said, this is the flow when connecting to RAC database:
a). The client has the SCAN name, port and service name configured in the tnsnames.ora
b). The client first resolves the SCAN name to an IP address
c). With the IP and port it connects to the SCAN listener and asks to connect to the specific database
(service)
d). The SCAN listener knows which services run on which nodes and redirects the client to one of the
local listeners (using the VIP name/port)
e). The client then resolves the VIP name to an IP address
f). With the IP and port of the local listener, The client connects to the listener and asks to connect to
the specific database (service)
g). The local listener creates a server process for this connection and gets out of the picture
h). Now the client talks directly with the server process
In our case, the problem we had was at step e). That's why we couldn't connect to the database, but
tnsping worked.

Before fixing the initial problem, because of the hosts configuration, the local listener was listening on
all IP addresses of the server (including the SCAN addresses). In this case, when the client tried to con
nect to the SCAN listener (step 3), it actually reached the local listener which simply connected it to th
e database. No redirection was needed (as this was not the SCAN listener), so the client didn’t need to
resolve server1-vip address.

The CRSd process starts orarootagent & oraagent processes, orarootagent is responsible for starting
Node VIP, SCAN VIP and oraagent process is responsible for starting SCAN Listener and Node List
ener.

For each of the 3 IP addresses that the SCAN resolves to, a SCAN VIP resource is created and a SCA
N Listener is created. The SCAN Listener is dependent on the SCAN VIP and the 3 SCAN VIPs (alo
ng with their associated listeners) will be dispersed across the cluster.REMOTE_LISTENER paramet
er is set to the SCAN by default this allows the instances to register with the SCAN Listeners as remo
te listeners to provide information on what services are being provided by the instance, the current loa
d and a recommendation on how many incoming connections should be directed to the instance.

The LOCAL_LISTENER parameter is set to the node-VIP, the node listener run out of the Oracle
Grid Infrastructure home and listens on the node-VIP using the specified port (default port is 1521).

So up till now we understood, SCAN configuration, SCAN IPs, SCAN Listener & Node Listener and
how the Instances provide relevant information to SCAN Listeners, now we’ll understand the Client
Side Load balancing in RAC using SCAN which would further explain another major benefit of using
SCAN.

Connection Load Balancing：

Client Side Load Balancing – For clients connecting using Oracle SQL*Net 11g Release 2 and above,
3 IP addresses will be received by the client by resolving the SCAN name through DNS. The client wi
ll then go through the list it receives from the DNS and try connecting through one of the IPs received.
If the client receives an error, it will try the other addresses before returning an error to the user or app
lication. This is similar to how client connection failover works in previous releases when an address
list is provided in the client connection string.

Server Side Load Balancing –When a SCAN Listener receives a connection request, the SCAN Listen
er will check for the least loaded instance providing the requested service. It will then re-direct the con
nection request to the local listener on the node where the least loaded instance is running.
Subsequently, the client will be given the address of the local listener. The local listener will finally
create the connection to the database instance.

I hope the above information helps you in understanding the SCAN Concept, how it is configured and
how it is used for client/server side connect time load balancing. Please comment below if you need m
ore information on SCAN & Load balancing as in how to configure client/server side connect time loa
d balancing, Transparent Application Failover concept, FAN Concept, etc.

SCAN VIP is one of the resources you find in the output of “crsctl status resource –t” command.
Number of SCAN VIP’s you notice will be the same as the number of SCAN LISTENERS in the
setup.

SCAN VIP’s are physical IP addresses that you allocate to SCAN listeners. In the example that I use
later in this blog, 192.168.122.5, 192.168.122.6, 192.168.122.7 are SCAN VIP’s. If you identify that
SCAN VIP’s are online in the output of “crsctl status resource –t” command then IP addresses are
online on the physical network ports. Only when SCAN VIP’s are online we can start the SCAN
listeners.

SCAN Listener is the oracle component which starts running a service on the port (by default its 1521)
using the SCAN VIP (IP address). So SCAN listener doesn’t start if SCAN VIP is not online. This is
the major difference between a SCAN listener and SCAN VIP. The number of SCAN listeners you
notice in the output will be the same as number of SCAN VIP’s ONLINE. Name that is given to
SCAN LISTENER is referred as SCAN NAME and it is registered in DNS server. In our example
which you will find next, the SCAN name is “SCAN_LISTENER”.
So in short, always SCAN LISTENER and SCAN VIP are related to each other.

Node VIP is the IP address which has capability of relocating to local listener of other node when the
current node goes down. Drawback of using only node VIP is TNS entry of application server should
have all node VIP addresses updated. You can identify the IP address assigned to node VIP will be a
part of LOCAL listener. Test it yourself, run “lsnrctl status” on every node and you will find that local
listener has two IP addresses – 1 node IP and 2 node VIP.

The challenge of having multiple VIP addresses in TNS entry is addressed in 11gR2 introducing
SCAN.

Let us now understand the architecture of SCAN with the help of below image that serves the user
requests from application server.
Untitled_567
Assumptions:
4 Node RAC setup/3 SCAN listeners/DNS server in use
TNS of application uses SCAN_LISTENER as hostname
Database is new and no connections

Explanation for each stage numbered in the image:

a) 0 sec: User1 when tries to establish a session on database with connection request C1, it hits DNS
server first.
b) DNS server will then resolve the name “SCAN_LISTENER” to the first IP 192.168.122.5
c) C1 request reaches the first scan listener SCAN1 mostly the default name will be
“LISTENER_SCAN1” which is running on 192.168.122.5 SCAN VIP.
d) SCAN1 using details from LBA, identifies the load on each node in the setup and routes the
request C1 to node which has least load.
e) In this case it happened to be NODE 2 with least load or no load and the C1 request is addressed
by local listener on this node which helps C1 to establish a session on instance on NODE 2.
f) 5th sec: User2 when tries to establish a session on database with connection request C2, it hits
DNS serer first.
g) DNS server will now use Round-Robin algorithm and resolves the name “SCAN_LISTENER” to
second IP 192.168.122.6
h) C2 request reaches the second scan listener SCAN2 mostly the default name will be
“LISTENER_SCAN2” which is running on 192.168.122.6 SCAN VIP.
i) SCAN2 using details from LBA, identifies the load on each node in the setup and routes the
request C2 to node which has least load.
j) In this case it happened to be NODE 1 with least load or no load and the C2 request is addressed
by local listener on this node which helps C2 to establish a session on instance on NODE 1.
— The same cycle and principles are applicable for other 3 connection requests C3, C4 and C5 —