Les 08

8
High Availability
of Connections
Copyright 2007, Oracle. All rights reserved.
Objectives
After completing this lesson, you should be able to:

Configure client-side connect-time load balancing
Configure client-side connect-time failover
Configure server-side connect-time load balancing
Use the Load Balancing Advisory (LBA)
Describe the benefits of Fast Application Notification
(FAN)
Configure server-side callouts
Configure the server- and client-side ONS
Configure Transparent Application Failover (TAF)
8-2
Types of Workload Distribution
Connection balancing is rendered possible by

configuring multiple listeners on multiple nodes:
Client-side connect-time load balancing
Client-side connect-time failover
Server-side connect-time load balancing
Run-time connection load balancing is rendered

possible by using connection pools:
Work requests automatically balanced across the pool of
connections
Native feature of the JDBC implicit connection cache and
ODP.NET connection pool
8-3
Client-Side Connect-Time Load Balancing
ERP =
(DESCRIPTION =
(ADDRESS_LIST =
(LOAD_BALANCE=ON)
(ADDRESS=(PROTOCOL=TCP)(HOST=node1vip)(PORT=1521))
)
(CONNECT_DATA=(SERVICE_NAME=ERP)))
Random
access
node1
8-4
node2
Client-Side Connect-Time Failover
ERP =
(DESCRIPTION =
(ADDRESS_LIST =
(LOAD_BALANCE=ON)
3
(FAILOVER=ON)
4
)
node2vip
2 node1vip
8-5
Server-Side Connect-Time Load Balancing

ERP = (DESCRIPTION=
(ADDRESS_LIST=(LOAD_BALANCE=ON)(FAILOVER=ON)
(ADDRESS=(PROTOCOL=TCP)(HOST=node2vip)(PORT=1521)))
6
Listener
5
ERP started on both instances

1
Listener
1
PMON
Node1
PMON
*.REMOTE_LISTENER=RACDB_LISTENERS
Node2
RACDB_LISTENERS=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=node1vip)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=node2vip)(PORT=1521)))
8-6
Fast Application Notification: Overview

.NET app
C app
C app
Java app
Java app
ODP.NET
OCI API
ONS OCI
API
ONS Java
API
JDBC
ONS
ONS
AQ
Proxy
app
Callout
script
Callout
exec
8-7
HA
Events
HA
Events
CRS
HA
Events
ONS
HA
Events
EMD
Node1
DB
Control
Fast Application Notification: Benefits
No need for connections to rely on connection timeouts

Used by Load Balancing Advisory to propagate load
information
Designed for enterprise application and management
console integration
Reliable distributed system that:
Detects high-availability event occurrences in a timely
manner
Pushes notification directly to your applications
Tightly integrated with:

Oracle JDBC applications using connection pools
Enterprise Manager
Data Guard Broker
8-8
FAN-Supported Event Types
8-9
Event type
Description
SERVICE
Primary application service
SRV_PRECONNECT
Shadow application service event

(mid-tiers and TAF using primary and secondary
instances)
SERVICEMEMBER
Application service on a specific instance
DATABASE
Oracle database
INSTANCE
Oracle instance
ASM
Oracle ASM instance
NODE
Oracle cluster node
SERVICE_METRICS
Load Balancing Advisory
FAN Event Status
8 - 10
Event status
Description
up
Managed resource comes up.
down
Managed resource goes down.
preconn_up
Shadow application service comes up.
preconn_down
Shadow application service goes down.
nodedown
Managed node goes down.
not_restarting
Managed resource cannot fail over to a remote

node.
restart_failed
Managed resource fails to start locally after a

discrete number of retries.
Unknown
Status is unrecognized.
FAN Event Reasons
8 - 11
Event Reason
Description
user
User-initiated commands, such as srvctl and sqlplus
failure
Managed resource polling checks detecting a failure
dependency
Dependency of another managed resource that triggered

a failure condition
unknown
Unknown or internal application state when event is

triggered
autostart
Initial cluster boot: Managed resource has profile attribute

AUTO_START=1, and was offline before the last Oracle
Clusterware shutdown.
Boot
Initial cluster boot: Managed resource was running before

the last Oracle Clusterware shutdown.
FAN Event Format

<Event_Type>
VERSION=<n.n>
[service=<serviceName.dbDomainName>]
[database=<dbName>] [instance=<sid>]
[host=<hostname>]
status=<Event_Status>
reason=<Event_Reason>
[card=<n>]
timestamp=<eventDate> <eventTime>
SERVICE VERSION=1.0 service=ERP.oracle.com
database=RACDB status=up reason=user card=4
timestamp=16-Mar-2004 19:08:15
NODE VERSION=1.0 host=strac-1
status=nodedown timestamp=16-Mar-2004 17:35:53
8 - 12
Load Balancing Advisory: FAN Event
Parameter
Description
Version
Version of the event payload
Event type
SERVICE_METRICS
Service
Matches DBA_SERVICES
Database unique name
Unique DB name supporting the service
Time stamp
Date and time stamp (local time zone)

Repeated
Instance
Instance name supporting the service
Percent
Percentage of work to send to this database and instance
Flag
GOOD, VIOLATING, NO DATA, UNKNOWN
8 - 13
Server-Side Callouts Implementation
The callout directory:

<CRS Home>/racg/usrco
Can store more than one callout
Grants execution on callout directory and callouts only to
the Oracle Clusterware user
Callouts execution order is nondeterministic.

Writing callouts involves:
1. Parsing callout arguments: The event payload
2. Filtering incoming FAN events
3. Executing event-handling programs
8 - 14
Server-Side Callout Parse: Example

#!/bin/sh
NOTIFY_EVENTTYPE=$1
for ARGS in $*; do
PROPERTY=`echo $ARGS | $AWK -F"=" '{print $1}'`
VALUE=`echo $ARGS | $AWK -F"=" '{print $2}'`
case $PROPERTY in
VERSION|version)
NOTIFY_VERSION=$VALUE ;;
SERVICE|service)
NOTIFY_SERVICE=$VALUE ;;
DATABASE|database)
NOTIFY_DATABASE=$VALUE ;;
INSTANCE|instance)
NOTIFY_INSTANCE=$VALUE ;;
HOST|host)
NOTIFY_HOST=$VALUE ;;
STATUS|status)
NOTIFY_STATUS=$VALUE ;;
REASON|reason)
NOTIFY_REASON=$VALUE ;;
CARD|card)
NOTIFY_CARDINALITY=$VALUE ;;
TIMESTAMP|timestamp) NOTIFY_LOGDATE=$VALUE ;;
??:??:??)
NOTIFY_LOGTIME=$PROPERTY ;;
esac
done
8 - 15
Server-Side Callout Filter: Example

if ((( [ $NOTIFY_EVENTTYPE = "SERVICE"
]
[ $NOTIFY_EVENTTYPE = "DATABASE"
]
[ $NOTIFY_EVENTTYPE = "NODE"
]
) &&
( [ $NOTIFY_STATUS = "not_restarting" ]
[ $NOTIFY_STATUS = "restart_failed" ]
)) &&
( [ $NOTIFY_DATABASE = "HQPROD"
]
[ $NOTIFY_SERVICE = "ERP"
]
))
then
/usr/local/bin/logTicket $NOTIFY_LOGDATE
$NOTIFY_LOGTIME
$NOTIFY_SERVICE
$NOTIFY_DBNAME
$NOTIFY_HOST
fi
8 - 16
||
||
||
||
\
\
\
\
\
\
\
\
\
\
\
\
Configuring the Server-Side ONS

Mid-tier1
localport=6100
remoteport=6200
useocr=on
2
ONS
$ racgons add_config node1:6200 node2:6200

$ racgons add_config midtier1:6200
$ onsctl reconfig
$ onsctl reconfig
ONS
ONS
OCR
ons.config
Node1
8 - 17
Node2
ons.config
Optionally Configure the Client-Side ONS

Mid-tier1
ons.config
$ onsctl start
ONS
localport=6100
remoteport=6200
nodes=node1:6200,node2:6200
ONS
ONS
OCR
ons.config
ons.config
Node1
8 - 18
Node2
JDBC Fast Connection Failover: Overview
Service
UP event
Mid-tier1
JDBC ICC
Connections
reconnected
Event
handler
Connections
marked down &
cleaned up
Connection Cache
Connections
using
service names
ONS
Node1
8 - 19
Service or node
DOWN event
ONS
Listeners
Connections
using
service names
Connections
load balancing
ONS
Noden
Using Oracle Streams Advanced Queuing for FAN

Use AQ to publish FAN to ODP.NET and OCI.
Turn on FAN notification to alert queue.
exec
DBMS_SERVICE.MODIFY_SERVICE (
service_name => 'SELF-SERVICE', aq_ha_notification => TRUE);
View published FAN events:

SQL> select object_name,reason
2 from dba_outstanding_alerts;
OBJECT_NAME REASON
----------- ---------------------------------------xwkE
Database xwkE (domain ) up as of time
2005-12-30 11:57:29.000000000 -05:00;
reason code: user
JFSERV
Composite service xwkE up as of time
2006-01-02 05:27:46.000000000 -05:00;
reason code: user
8 - 20
JDBC/ODP.NET FCF Benefits
Database connections are balanced across preferred

instances according to LBA.
Database work requests are balanced across preferred
instances according to LBA.
Database connections are anticipated.
Database connection failures are immediately detected
and stopped.
8 - 21
Load Balancing Advisory
The Load Balancing Advisory (LBA) is an advisory for

sending work across RAC instances.
The LBA advice is available to all applications that send
work:
JDBC and ODP connection pools
Connection load balancing
The LBA advice sends work to where services are

executing well and resources are available:
Relies on service goodness
Adjusts distribution for different power nodes, different
priority and shape workloads, changing demand
Stops sending work to slow, hung, or failed nodes
8 - 22
JDBC/ODP.NET Runtime Connection

Load Balancing: Overview
CRM work requests
Connection Cache
10%
30%
60%
RAC
Inst1
8 - 23
RAC
CRM is
very busy.
CRM is
not busy.
Inst2
RAC
CRM is
busy.
Inst3
Connection Load Balancing in RAC
Connection load balancing allows listeners to distribute

connection requests to the best instances.
Three metrics are available for listeners to decide:
Session count by instance
Run queue length of the node
Service goodness
The metric used depends on the connection loadbalancing goal defined for the service:
LONG
NONE
SHORT
8 - 24
Load Balancing Advisory: Summary
Uses DBMS_SERVICE.GOAL
Service time: weighted moving average of elapsed time
Throughput: weighted moving average of throughput
AWR
Calculates goodness locally (MMNL), forwards to master MMON
Master MMON builds advisory for distribution of work across
RAC, and posts load balancing advice to AQ
IMON retrieves advice and send it to ONS
EMON retrieves advice and send it to OCI
Local MMNL post goodness to PMON
Listeners use DBMS_SERVICE.CLB_GOAL=SHORT

Use goodness from PMON to distribute connections.
Load Balancing Advisory users (inside the pools)

Use percentages and flags to send work.
8 - 25
Monitor LBA FAN Events
SQL> SELECT TO_CHAR(enq_time, 'HH:MI:SS') Enq_time, user_data

2 FROM sys.sys$service_metrics_tab
3 ORDER BY 1 ;
ENQ_TIME USER_DATA
-------- ----------------------------------------------------
04:19:46 SYS$RLBTYP('JFSERV', 'VERSION=1.0 database=xwkE
service=JFSERV { {instance=xwkE2 percent=50
flag=UNKNOWN}{instance=xwkE1 percent=50 flag=UNKNOWN}
} timestamp=2006-01-02 06:19:46')
04:20:16 SYS$RLBTYP('JFSERV', 'VERSION=1.0 database=xwkE
service=JFSERV { {instance=xwkE2 percent=80
flag=UNKNOWN}{instance=xwkE1 percent=20 flag=UNKNOWN}
} timestamp=2006-01-02 06:20:16')
SQL>
8 - 26
FAN Release Map

Oracle
Release
FAN Product
Integration
Event
System
FAN Event Received and Used
10.1.0.2
JDBC ICC&FCF
Server-side callouts
ONS
RAC
Up/down/Load Balancing Advisory

Up/down
10.1.0.3
CMAN
Listeners
ONS
PMON
Down
Up/down/LBA
10.2
OCI connection pool

OCI session pool
TAF
ODP.NET
Custom OCI
DG Broker (10.2.0.3)
AQ
AQ
AQ
AQ
AQ
AQ
Down
Down
Down
Up/down/LBA
All
Down
11.1
OCI session pool
AQ
LBA
8 - 27
Transparent Application Failover: Overview

TAF Basic
Application
OCI Library
AP
OCI Library
5
8
ERP
1
3
3
ERP ERP_PRECONNECT
1
8 - 28
Net Services
Application
Net Services
AP
TAF Preconnect
TAF Basic Configuration Without FAN: Example
$ srvctl add service -d RACDB -s AP -r I1,I2 \

> -P BASIC
$ srvctl start service -d RACDB -s AP
AP =
(DESCRIPTION =(FAILOVER=ON)(LOAD_BALANCE=ON)
(ADDRESS=(PROTOCOL=TCP)(HOST=N1VIP)(PORT=1521))
(CONNECT_DATA =
(SERVICE_NAME = AP)
(FAILOVER_MODE =
(TYPE=SESSION)
(METHOD=BASIC)
(RETRIES=180)
(DELAY=5))))
8 - 29
TAF Basic Configuration with FAN: Example

$ srvctl add service -d RACDB -s AP -r I1,I2
$ srvctl start service -d RACDB -s AP
execute dbms_service.modify_service (
service_name => 'AP'
aq_ha_notifications => true
failover_method => dbms_service.failover_method_basic
failover_type
=> dbms_service.failover_type_session
failover_retries => 180, failover_delay => 5
clb_goal => dbms_service.clb_goal_long);
AP =
(CONNECT_DATA = (SERVICE_NAME = AP)))
8 - 30
,,,,,,-
TAF Preconnect Configuration: Example

$ srvctl add service -d RACDB -s ERP -r I1 a I2 \
> -P PRECONNECT
$ srvctl start service -d RACDB -s ERP
ERP =
(CONNECT_DATA = (SERVICE_NAME = ERP)
(FAILOVER_MODE = (BACKUP=ERP_PRECONNECT)
(TYPE=SESSION)(METHOD=PRECONNECT))))
ERP_PRECONNECT =
(CONNECT_DATA = (SERVICE_NAME = ERP_PRECONNECT)))
8 - 31
TAF Verification
SELECT
1st
node
2nd
node
2nd
node
after
8 - 32
machine, failover_method, failover_type,
failed_over, service_name, COUNT(*)

FROM
v$session
GROUP BY machine, failover_method, failover_type,
failed_over, service_name;
MACHINE FAILOVER_M FAILOVER_T FAI SERVICE_N
COUNT(*)
------- ---------- ---------- --- --------------node1
BASIC
SESSION
NO AP
1
MACHINE FAILOVER_M FAILOVER_T FAI SERVICE_N
COUNT(*)
node1
PRECONNECT SESSION
NO ERP
1
------- ---------- ---------- --- ---------------MACHINE
FAILOVER_M
FAILOVER_T
FAI
node2
NONE
NONE
NO SERVICE_N
ERP_PRECO 1
COUNT(*)
------- ---------- ---------- --- --------------node2
BASIC
SESSION
YES AP
1
FAN Connection Pools and TAF Considerations
Both techniques are integrated with services and

provide service connection load balancing.
Do not use FCF when working with TAF, and vice versa.
Connection pools that use FAN are always
preconnected.
TAF may rely on operating system (OS) timeouts to
detect failures.
FAN never relies on OS timeouts to detect failures.
8 - 33
Summary
In this lesson, you should have learned how to:

Configure client-side connect-time load balancing
Configure client-side connect-time failover
Configure server-side connect-time load balancing
Use the Load Balancing Advisory
Describe the benefits of Fast Application Notification
Configure server-side callouts
Configure the server- and client-side ONS
Configure Transparent Application Failover
8 - 34
Practice 8: Overview
This practice covers the following topics:

Monitoring high availability of connections
Creating and using callout scripts
Using the Load Balancing Advisory
Using the Transparent Application Failover feature
8 - 35

Les 08

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Les 08

Uploaded by

Copyright:

Available Formats

8

Copyright 2007, Oracle. All rights reserved.

After completing this lesson, you should be able to:

Copyright 2007, Oracle. All rights reserved.

Types of Workload Distribution

Connection balancing is rendered possible by

Run-time connection load balancing is rendered

Copyright 2007, Oracle. All rights reserved.

Client-Side Connect-Time Load Balancing

Copyright 2007, Oracle. All rights reserved.

Client-Side Connect-Time Failover

Copyright 2007, Oracle. All rights reserved.

Server-Side Connect-Time Load Balancing

ERP started on both instances

Copyright 2007, Oracle. All rights reserved.

Fast Application Notification: Overview

Copyright 2007, Oracle. All rights reserved.

Fast Application Notification: Benefits

No need for connections to rely on connection timeouts

Tightly integrated with:

Copyright 2007, Oracle. All rights reserved.

FAN-Supported Event Types

Primary application service

Shadow application service event

Application service on a specific instance

Oracle ASM instance

Oracle cluster node

Load Balancing Advisory

Copyright 2007, Oracle. All rights reserved.

FAN Event Status

Managed resource comes up.

Managed resource goes down.

Shadow application service comes up.

Shadow application service goes down.

Managed node goes down.

Managed resource cannot fail over to a remote

Managed resource fails to start locally after a

Copyright 2007, Oracle. All rights reserved.

FAN Event Reasons

User-initiated commands, such as srvctl and sqlplus

Managed resource polling checks detecting a failure

Dependency of another managed resource that triggered

Unknown or internal application state when event is

Initial cluster boot: Managed resource has profile attribute

Initial cluster boot: Managed resource was running before

Copyright 2007, Oracle. All rights reserved.

FAN Event Format

Copyright 2007, Oracle. All rights reserved.

Load Balancing Advisory: FAN Event

Version of the event payload

Database unique name

Unique DB name supporting the service

Date and time stamp (local time zone)

Instance name supporting the service

Percentage of work to send to this database and instance

GOOD, VIOLATING, NO DATA, UNKNOWN

Copyright 2007, Oracle. All rights reserved.

Server-Side Callouts Implementation

The callout directory:

Callouts execution order is nondeterministic.

Copyright 2007, Oracle. All rights reserved.

Server-Side Callout Parse: Example