RAC, Data Guard, &

the Data Guard Broker

Bonnie J. Bizzaro
Nationwide Insurance

Bonnie Bizzaro
Database Engineering
Nationwide Insurance

Database Administrator since 1991

Past presenter, IOUW, Oracle Open World,
Collaborate
Served on board of Ohio Oracle Users group,
& Collaborate Conference Committee

Session prerequisite:
Knowledge and understanding of
Oracle RAC architecture
This session will provide all of the information
you need to successfully create the standby
database, create services, configure the
broker, perform switchovers, and verify that
CRS is aware of the database's correct state.

 Data Guard architecture

Configure Oracle Networking
Duplicate primary database
Cluster enable the standby
Data Guard Broker configuration
Troubleshooting
Perform a switchover test
Active Data Guard

I wont be talking about 10g

I wont be talking about Transparent Application
Failover

I wont be talking about scan listeners.

There is only so much that even I can say in 60
minutes..

Why is DG Broker and RAC problematic ?

Many parts
Too many log files
CRS integration not 100% (11.1)

Data Guard non-RAC without Broker

LNS

Primary

RFS
Standby

Data Guard non-RAC with Broker

Redo
LNS

RFS

Primary

NSVn

DRCn

Standby

Data, SQL
NSVn

DRCn

RSM

RSM
DMON

Config files

DMON

Config files

Data Guard on RAC with Broker

Node2

Node2
INVS

LNS
LNS

Redo
RFS
RFS
NSVn
NSVn

DRCn
DRCn
Node1

Data, SQL
NSVn
NSVn

Primary

INVS

LNS
DRCn
DRCn

Standby

RSM

RSM
DMON
DMON

DMON
DMON

Config files

Config files

Lets get started!

Data Guard patch bundle needs
applied if 11.1 (7628357)

 Data Guard architecture

Configure Oracle Networking
Duplicate primary database
Cluster enable the standby
Data Guard Broker configuration
Troubleshooting
Perform a switchover test
Active Data Guard

Configure Oracle Networking

Statically register *_DGMGRL entry for the instances in
listener.ora on each node of the primary and standby clusters
SID_LIST_LISTENER_$HOST =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = $DB_UNIQUE_NAME_dgmgrl)
(SID_NAME = $INSTANCE_NAME)
(ORACLE_HOME = /u01/home/oracle/11.1.0.7)
)
)

Create temporary tns entry for duplicate command

$DB_UNIQUE_NAME_TEMP =
(DESCRIPTION=
(ADDRESS= (PROTOCOL=TCP)(HOST=NODE1-vip)(PORT=1521))
(CONNECT_DATA=
(SERVER=DEDICATED)
(SERVICE_NAME=$DB_UNIQUE_NAME_dgmgrl)))

Only have 1 host listed. It will be used by RMAN

duplicate process and can be removed after the
duplicate is complete.

tnsnames.ora entries on all standby nodes

These will be referenced by REMOTE_LISTENER and LOCAL_LISTENER
parameters in the standby database.

LISTENERS_$DB_UNIQUE_NAME =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = NODE1-vip)(PORT=1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = NODE2-vip)(PORT=1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = NODE3-vip)(PORT=1521))
)
LISTENER_$INSTANCE =
(ADDRESS = (PROTOCOL = TCP)(HOST = NODE-vip)(PORT = 1521))

 Data Guard architecture

Configure Oracle Networking
Duplicate primary database
Cluster enable the standby
Data Guard Broker configuration
Troubleshooting
Perform a switchover test
Active Data Guard

Prepare Primary for duplication

Verify primary is in archivelog mode.
Turn on force logging
SQL> alter database force logging

Copy password file to standby servers.The

password file needs to be an exact binary
match on all servers. Very important!

Create standby redo logs

(# of redo logs per instance +1) * # of instances = total # of standby redo logs
In this example, assume the following:
$INSTANCE1.thread=1
$INSTANCE2.thread=2
$INSTANCE3.thread=3

$INSTANCE1 has 3 redo logs, 100m each

$INSTANCE2 has 3 redo logs, 100m each
$INSTANCE3 has 3 redo logs, 100m each

Therefore:
$INSTANCE1 (thread 1) needs 3 + 1 = 4 redo logs 100m each for thread 1
$INSTANCE2 (thread 2) needs 3 + 1 = 4 redo logs 100m each for thread 2
$INSTANCE3 (thread 3) needs 3 + 1 = 4 redo logs 100m each for thread 3

This database will need a total of 12 standby redo logs.

Here is the syntax to use to create them. Dont forget the thread parameter.

Run each statement 4 times.

ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 (+DISKGRP_DATA') size 100m;
.
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 (+DISKGRP_DATA') size 100m;
.
ALTER DATABASE ADD STANDBY LOGFILE THREAD 3 (+DISKGRP_DATA') size 100m;
.
.
At this time Oracle DOES NOT recommend multiplexing of standby redo logs. They can
cause problems at failover time.

Prepare Standby for duplication

Select one node to create initial instance
Create diskgroups for data and flash
Create audit directories
Create init$INSTANCE_NAME.ora file with only
db_name=$DB_NAME
db_unique_name=$DB_UNIQUE_NAMEs
instance_number=1

Start standby instance

SQL>startup nomount pfile=$ORACLE_HOME/dbs/
init$INSTANCE_NAME.ora

Test connection from primary side using the

service you registered with the standby listener.
$sqlplus sys/password@$DB_UNIQUE_NAME_dgmgrl as sysdba

RMAN duplicate command

run
{
allocate channel disk1 device type disk ;
allocate channel disk2 device type disk ;
allocate channel disk3 device type disk ;
allocate auxiliary channel aux1 device type disk ;
allocate auxiliary channel aux2 device type disk ;
allocate auxiliary channel aux3 device type disk ;
duplicate target database for standby from active database
spfile
parameter_value_convert '$DB_UNIQUE_NAMEp','$DB_UNIQUE_NAMEs'
set instance_number='1'
set db_unique_name='$DB_UNIQUE_NAMEs'
set audit_file_dest='/u01/home/oracle/admin/$DB_UNIQUE_NAMEs/adump
set control_files='+DISKGROUP_DATA/$DB_UNIQUE_NAMEs/controlfile/control01.dbf'
,'+DISKGROUP_FLASH/$DB_UNIQUE_NAMEs/controlfile/control02.ctl'
set remote_listener='LISTENERS_$DB_UNIQUE_NAMEs'
nofilenamecheck;
}
exit

After duplicate completes

Verify standby database
If using OMF ignore Clearing online log errors
Remove multiplexed standby redo logs
Created automatically, can cause problems at
switchover

 Data Guard architecture

Configure Oracle Networking
Duplicate primary database
Cluster enable the standby
Data Guard Broker configuration
Troubleshooting
Perform a switchover test
Active Data Guard

Create & edit text pfile from new standby spfile

Delete

all the entries which start with the primary instance names.
Change the instance names listed in instance_name.instance_number
entries.
Verify the standbys db_unique_name is used.
Make sure there are no duplicate entries or global (*.) entries that dont
belong.
Verify remote_listener is set to LISTENERS_$DB_UNIQUE_NAME
where $DB_UNIQUE_NAME is the standby.
Add entries for local_listener using the listener_$INSTANCE in the
tnsnames.ora file

Verify one entry for

*.log_archive_dest_1=LOCATION=USE_DB_RECOVERY_FILE_
DEST
Add one entry for *.log_archive_dest_state_1=ENABLE
Add one entry for *. log_archive_format='%t_%s_%r.dbf
Remove entry for *.instance_number=1
Edit the instance numbers for thread=, change them from the
primary instance names to the standby instance names.
Edit the instance numbers for undo_tablespace=, change them
from the primary instance names to the standby instance names.

Create new spfile in ASM

SQL> create
spfile='+$DISKGROUP_DATA/$DB_UNIQUE_NAMEs/spfile$DB_U
NIQUE_NAMEs.ora' from pfile='tmp_init.ora';

Create init$INSTANCE.ora pointing to new spfile.

Example:

spfile=+MYDB01P_DATA/MYDBPRR/spfileMYDBPRR.ora

Check new spfile for errors

Startup, shutdown database.

Register Standby with CRS

srvctl add database -d $DB_UNIQUE_NAME -o $ORACLE_HOME -m local
-p $name_of_spfile_including_path_inASM -n $DB_NAME -r
physical_standby -s mount
srvctl add instance -d $DB_UNIQUE_NAME -i $INSTANCE_NAME -n
$NODE

Add global floating service

srvctl add service -d $DB_UNIQUE_NAME -s $FLOATING_SERVICE -r
"$INSTANCE_NAME1,$INSTANCE_NAME2,$INSTANCE_NAME3" -P
NONE
(start on primary only)
srvctl start service -d $DB_UNIQUE_NAME -s "$FLOATING_SERVICE

Example of connect string using a global service name mydbprdg

(DESCRIPTION=(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=prod0003-vip)(PORT=1521))
(ADDRESS=(PROTOCOL=TCP)(HOST=prod0004-vip)(PORT=1521))
(ADDRESS=(PROTOCOL=TCP)(HOST=prod0005-vip)(PORT=1521))
(ADDRESS=(PROTOCOL=TCP)(HOST=drserv0063-vip)(PORT=1521))
(ADDRESS=(PROTOCOL=TCP)(HOST=drserv0064-vip)(PORT=1521))
(ADDRESS=(PROTOCOL=TCP)(HOST=drserv0065-vip)(PORT=1521)) )
(CONNECT_DATA=(SERVICE_NAME=mydbprdg))
(SERVER = DEDICATED) )

Verify CRS configuration

srvctl config database -d <db_name> -a
Here is sample output from a successful query of a primary database:
testsrv0335 bjbdbp1 /u01/home/oracle/11.1.0.7
testsrv0336 bjbdbp2 /u01/home/oracle/11.1.0.7
testsrv0337 bjbdbp3 /u01/home/oracle/11.1.0.7
DB_UNIQUE_NAME: bjbdbp
DB_NAME: bjbdbp
ORACLE_HOME: /u01/home/oracle/11.1.0.7
SPFILE: +ASSM01T_DATA/bjbdbp/spfilebjbdbp.ora
DOMAIN:
DB_ROLE: PRIMARY
Note DB_ROLE
START_OPTIONS: null
Note START_OPTIONS
POLICY: AUTOMATIC
ENABLE FLAG: DB ENABLED

 Data Guard architecture

Configure Oracle Networking
Duplicate primary database
Cluster enable the standby
Data Guard Broker configuration
Troubleshooting
Perform a switchover test
Active Data Guard

Prepare databases for broker

Turn off database broker processes
SQL>alter system set dg_broker_start=false;

Add DG broker config files

Using asmcmd create broker subdirectory
Use alter system command to set
dg_broker_config_file* parameters.

Create broker subdirectory

$. oraenv
$+ASMn (where n is the ASM instance number)
$asmcmd
ASMCMD> pwd
+$DISKGROUP _DATA/$DB_UNIQUE_NAME
ASMCMD> mkdir BROKER
ASMCMD> cd +$DISKGROUP_FLASH/$DB_UNIQUE_NAME
ASMCMD> mkdir BROKER

Create empty broker config files

on primary and standby
SQL> alter system set dg_broker_config_file1='+$DISKGROUP
_DATA/$DB_UNIQUE_NAMEp/BROKER/dr1$DB_UNIQUE_NAMEp
.dat scope=both;
SQL> alter system set dg_broker_config_file2='+$DISKGROUP
_DATA/$DB_UNIQUE_NAMEp/BROKER/dr2$DB_UNIQUE_NAMEp
.dat scope=both;

On Primary and Standby

Turn on broker background processes
SQL> alter system set dg_broker_start=true scope=both;

Verify broker and configuration files

SQL> show parameter broker
NAME
TYPE
VALUE
------------------------------------ ----------- -----------------------------dg_broker_config_file1
string
+MYDB01P_DATA/MYDBPRR/dr1mydbprr.dat
dg_broker_config_file2
string
+MYDB01P_DATA/MYDBPRR/dr2mydbprr.dat
dg_broker_start
boolean TRUE

Create Data Guard Configuration

On Primary node, startup dgmgrl command line interface to broker
$dgmgrl
DGMGRL>connect sys/xxx you must use sys and sys password
create configuration $DB_UNIQUE_NAMEp_$DB_UNIQUE_NAMEs
as primary database is DB_UNIQUE_NAMEp connect identifier is
$DB_UNIQUE_NAMEp;
add database $DB_UNIQUE_NAMEs as connect identifier is
$DB_UNIQUE_NAMEs maintained as physical;

Create Data Guard Configuration cont..

edit configuration set protection mode as maxperformance; set desired
mode
enable configuration;

this will turn on broker

show configuration;

this will show if there are errors

show database $DB_UNIQUE_NAMEp StatusReport shows db errors

show database $DB_UNIQUE_NAMEs StatusReport shows db errors

This is an example of a successful configuration:

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys/password
Connected.
DGMGRL> show configuration
Configuration
Name:
mydbprd_mydbprr
Enabled:
YES
Protection Mode: MaxPerformance
Databases:
mydbprd - Primary database
mydbprr - Physical standby database
Fast-Start Failover: DISABLED
Current status for mydbprd_mydbprr":
SUCCESS

Check for errors. Do not ignore warnings!

This message:
Warning: ORA-16610: command "EDIT DATABASE
$DB_UNIQUE_NAME SET PROPERTY" in progress
It means that the configuration is still in progress. Wait a few minutes and
try again

 Data Guard architecture

Configure Oracle Networking
Duplicate primary database
Cluster enable the standby
Data Guard Broker configuration
Troubleshooting
Perform a switchover test
Active Data Guard

Data Guard Broker commands

dgmgrl
<= enter at server prompt to access broker interface
connect sys/password
show database $DB_UNIQUE_NAME status
show database $DB_UNIQUE_NAME 'StatusReport'
show database $DB_UNIQUE_NAME 'LogXptStatus'
show database $DB_UNIQUE_NAME 'InconsistentLogXptProps'
show database $DB_UNIQUE_NAME 'InconsistentProperties'
show database verbose $DB_UNIQUE_NAME
show instance verbose $INSTANCE_NAME

Data Guard Broker instance properties

HostName(*)
SidName(*)
StaticConnectIdentifier(*)
StandbyArchiveLocation(*)
AlternateLocation(*)
LogArchiveTrace(*)
LogArchiveFormat(*)
LatestLog(*)
TopWaitEvents(*)

DGMGRL> show instance verbose mydbr1

Instance 'mydbr1' of database 'mydbr'
Host Name:
drserv0078
PFILE:
Properties:
HostName
= 'drserv0078'
SidName
= 'mydbr1'
StaticConnectIdentifier
=
'(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=172.24.93.239)(PO
RT=1521))(CONNECT_DATA=(SERVICE_NAME=mydbr_DGMGRL.nsc.net)(I
NSTANCE_NAME=mydbr1)(SERVER=DEDICATED)))'
StandbyArchiveLocation
= '+TWS01P_FLASH'
AlternateLocation
= ''
LogArchiveTrace
= '0'
LogArchiveFormat
= 'mydbr%t_%s_%r.dbf'
LatestLog
= '(monitor)'
TopWaitEvents
= '(monitor)

Apply instance & Apply instance failover

Apply process only runs on one instance at a
time in 11.1.
From dgmgrl show database verbose command

Intended State: APPLY-ON

Instance(s):
mydbr2 (apply instance)
mydbr1

Broker automatically moves log apply to available instance if apply instance

fails.

Apply instance & Apply instance failover

Set PreferredApplyInstance property

To specify the specific instance for the apply process.

This change wont be immediate, but will be in effect when the
current apply instance is restarted or there is a state change.

DGMGRL> show database verbose mydb01pr

Database
Name:
mydb01pr
Role:
PHYSICAL STANDBY
Enabled:
YES
Intended State: APPLY-ON
Instance(s):
mydb01pr2
mydb01pr1(apply instance)

Verify log apply status

V$archive_gap often doesnt show logs are
missing, will only show gap in logs recieved
Query last log received & applied (each thread)
Thread Last Seq Received Last Seq Applied
---------- ----------------- ---------------1
1463
1446
SQL> Select switchover_status from v$database;
SWITCHOVER_STATUS
----------------TO STANDBY

ORA-16532: Data Guard broker configuration does

not exist
DGMGRL> show database mydb 'StatusReport'
STATUS REPORT
INSTANCE_NAME SEVERITY ERROR_TEXT
mydb2
ERROR ORA-16532: Data Guard broker configuration does not
exist
mydb3
ERROR ORA-16532: Data Guard broker configuration does not
exist

The problem is communicating to other instances.

 Data Guard architecture

Configure Oracle Networking
Duplicate primary database
Cluster enable the standby
Data Guard Broker configuration
Troubleshooting
Perform a switchover test
Active Data Guard

Process for RAC Switchover

Stop all but 1 instance on primary and standby
clusters
Run dgmgrl from one of the current primary
database servers. (bug # 7457766)
$ dgmgrl
DGMGRL> connect sys/password
DGMGRL> switchover to
$STANDBY_DB_UNIQUE_NAME
.

Performing switchover NOW, please wait...

New primary database mydbprr" is opening...
Operation requires shutdown of instance mydbprd1" on
database mydbprd"
Shutting down instance mydbprd1"...
Switchover succeeded, new primary is mydbprr"
DGMGRL>

Switchover verify
Verify that OCR/CRS shows correct state for the database.
$srvctl config database d mydbprd -a
Prodsrv0003 mydbprd1 /u01/home/oracle/11.1.0.7
Prodsrv0004 mydbprd2 /u01/home/oracle/11.1.0.7
Prodsrv0005 mydbprd3 /u01/home/oracle/11.1.0.7
DB_UNIQUE_NAME: mydbprd
DB_NAME: mydbprd
ORACLE_HOME: /u01/home/oracle/11.1.0.7
SPFILE: +MYDB01P_DATA/mydbprd/spfilemydbprd.ora
DOMAIN: nsc.net
DB_ROLE: PRIMARY
Verify this is correct
START_OPTIONS: null
Verify this
POLICY: AUTOMATIC
ENABLE FLAG: DB ENABLED

Switchover verify
If necessary, correct database role or start option in CRS
srvctl modify database d DB_UNIQUE_NAME s mount
srvctl modify database d DB_UNIQUE_NAME r PHYSICAL_STANDBY

Verify the floating service is running on the correct cluster

ora.mydbprr.mydbprdg.mydbprr1.srv
ora.mydbprr.mydbprdg.mydbprr2.srv
ora.mydbprr.mydbprdg.mydbprr3.srv

ONLINE
ONLINE
ONLINE

ONLINE on drserv0065
ONLINE on drserv0066
ONLINE on drserv0067

 Data Guard architecture

Configure Oracle Networking
Duplicate primary database
Cluster enable the standby
Data Guard Broker configuration
Troubleshooting
Perform a switchover test
Active Data Guard

Active Data Guard

Applying redo while standby is open read-only
Compatible parameter must be 11.0 or higher

Can use block change tracking for fast

incremental backups against the standby
Easy to enable.

Active Data Guard

Extra cost option (Included

with Golden Gate

license)
In 11.1 RAC, turned on automatically after switchover (Be

aware in case you dont have a license)

No performance impact to primary (Except

when doing

large imports)
Open read-only (Applications

often write to
database for instrumentation purposes)

Active Data Guard

To enable, pause redo apply and open the database read only.
SQL> alter database recover managed standby database cancel;
Database altered.
SQL> alter database open read only;

Database altered.
SQL> alter database recover managed standby database disconnect;

Database altered.

Active Data Guard

Verify:
SQL> select database_role,open_mode from v$database;

DATABASE_ROLE
---------------PHYSICAL STANDBY

OPEN_MODE
---------READ ONLY

Fix CRS:
srvctl modify database -d mydbr -s open

Active Data Guard

Note If the apply instance goes down in an Active
Data Guard configuration, ALL of the instances will
go down.

 Data Guard architecture

Configure Oracle Networking
Duplicate primary database
Cluster enable the standby
Data Guard Broker configuration
Troubleshooting
Perform a switchover test
Active Data Guard

Conclusion
Important things to remember
-Statically register instances in listener.ora
-Often error messages are misleading
-Copy password file from 1 primary server to ALL
other servers. May have to do it more than once.
-Verify CRS config after switchovers
- v$archive_gap doesnt always show the gap

Please complete the session

evaluation!
RAC, Data Guard, & the Data Guard Broker
Bonnie J. Bizzaro
Nationwide Insurance

SAVE THE DATE!

COLLABORATE 13
April 7-11, 2013
Colorado Convention Center
Denver, Colorado
http://collaborate13.ioug.org