RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

ARCHIVE GAP
An archive gap means a set of archived redo logs that could not be transmitted to the
Standby site from the Primary database. Mostly this problem occurs the network
connectivity becomes unavailable between the Primary and Standby site. When the
network is available again Data Guard resumes redo data transmission from the Primary
to Standby site. Oracle DG provides 2 methods for GAP resolution. They are AUTOMATIC
and FAL.

 Automatic Gap Resolution & FAL Gap Resolution

Let’s consider an extended n/w failure occurred between the Primary and Standby
machines which causes the Standby is very far behind the Primary database, then an
RMAN INCREMENTALBACKUP can be used to roll the Standby database forward faster than
redo log apply.

When the archived logs are missing on the Standby database, simply we can ship
missing logs from the Primary to Standby database; (If missing logs are very less
count e.g. (below 15).We need to register all Shipped logs in the Standby database so
that Gap can be resolved.

In this article I will demonstrate how to resolve archive log gaps using following
methods.

 Manually resolving a Gap (by shipping logs from Primary to Standby).

 To Roll Forward a Standby database using RMAN Incremental backup.

DISASTER RECOVERY CONFIGURATION

Primary Database -> 192.168.222.131 -> DEV Database Server
Standby Database -> 192.168.222.132 -> UAT Database Server
Primary database db_unique_name -> crms
Standby database db_unique_name -> stbycrms

MANUALLY RESOLVING GAPS

# On Primary database

SYS> select name, db_unique_name, database_role from v$database;

NAME DB_UNIQUE_NAME DATABASE_ROLE
--------- ------------------------------ ----------------
CRMS crms PRIMARY

# On Standby database

SYS> select name, db_unique_name, database_role from v$database;

NAME DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE
--------- --------------------- ---------------- ------------
CRMS stbycrms PHYSICAL STANDBY MOUNTED

# On Primary database

SYS> select thread#, max (sequence#) from v$archived_log group by thread#;

THREAD# MAX (SEQUENCE#)
--------- --------------
1 551

# On Standby database

SYS> select thread#, max (sequence#) from v$archived_log where applied='YES' group
by thread#;
THREAD# MAX (SEQUENCE#)
--------- --------------
1 551

Oracle DBA Technology Explored by 8 Bits Virtual Training

 RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

# Standby database is waiting for SEQ# 552

SYS> select process, sequence#, status from v$managed_standby;

PROCESS SEQUENCE# STATUS

--------- ---------- ------------
ARCH 551 CLOSING
ARCH 558 CLOSING
ARCH 0 CONNECTED
ARCH 0 CONNECTED
RFS 0 IDLE
RFS 0 IDLE
RFS 559 IDLE
RFS 0 IDLE
MRP0 552 WAIT_FOR_GAP
9 rows selected.

# Snippet of Standby alert log

$ cd /u01/app/oracle/diag/rdbms/stbycrms/stbycrms/trace
$ tail -f alert_stbycrms.log
Fetching gap sequence in thread 1, gap sequence 552-556
..
... FAL [client]: Failed to request gap sequence

GAP - thread 1 sequence 552-556

DBID 1613387466 branch 913081878
FAL [client]: All defined FAL servers have been attempted.

DETECTING GAPS
Oracle Data Guard provide us with a simple view (v$archive_gap) to detect a gap.

# On Standby database

SYS> select * from v$archive_gap;

THREAD# LOW_SEQUENCE# HIGH_SEQUENCE#
---------- ------------- --------------
1 552 556

Output of Standby database is currently missing log files from sequence# 552 to 556; the
Standby database is 5 logs behind the Primary database. ORACLE NOTE: Refer BUG #10072528
V$ARCHIVE_GAP may not detect archive gap when Physical Standby is open read only.

FIND ARCHIVE LOGS AT PRIMARY SERVER

After identifying a gap (as shown above), as a DBA you need to query the primary database
to locate the archived redo logs on the primary database. I have configured the local
archive destination on primary is LOG_ARCHIVE_DEST_1.

# On Primary database (SEQUENCE 552 to 556)

SYS> select name

FROM v$archived_log
WHERE thread# = 1
AND dest_id = 1
AND sequence# BETWEEN 552 and 556;

NAME
--------------------------------------------------------------------------------------
/u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/o1_mf_1_552_cnvodqq5_.arc
/u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/o1_mf_1_553_cnvohcy1_.arc
Oracle DBA Technology Explored by 8 Bits Virtual Training
 RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

/u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/o1_mf_1_554_cnvokk6z_.arc
/u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/o1_mf_1_555_cnvom1l4_.arc
/u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/o1_mf_1_556_cnvomxkz_.arc

Copy the above redo log files to the Physical Standby database and register them using the
ALTER DATABASE REGISTER LOGFILE ... SQL statement on the Physical Standby database.

COPY ARCHIVELOG FILES TO STANDBY

$ ls /u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/

o1_mf_1_553_cnvohcy1_.arc o1_mf_1_555_cnvom1l4_.arc o1_mf_1_552_cnvodqq5_.arc

o1_mf_1_554_cnvokk6z_.arc o1_mf_1_556_cnvomxkz_.arc

COPY ARCHIVE LOG FILES TO STANDBY

# Example to transfer archive logs to Standby Server

$ scp log_file_n.arc oracle@standby:/archive_log_location/log_file_n.arc

$ scp o1_mf_1_552_cnvodqq5_.arc o1_mf_1_553_cnvohcy1_.arc o1_mf_1_554_cnvokk6z_.arc

o1_mf_1_555_cnvom1l4_.arc o1_mf_1_556_cnvomxkz_.arc oracle@192.168.222.134:/u01/app/
oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/

oracle@192.168.222.134's password:

o1_mf_1_552_cnvodqq5_.arc 100% 29KB 29.0KB/s 00:00

o1_mf_1_553_cnvohcy1_.arc 100% 73KB 72.5KB/s 00:00
o1_mf_1_554_cnvokk6z_.arc 100% 179KB 179.0KB/s 00:00
o1_mf_1_555_cnvom1l4_.arc 100% 61KB 60.5KB/s 00:00
o1_mf_1_556_cnvomxkz_.arc 100% 19KB 19.0KB/s 00:00

As per above example you need to transfer all archive logs to the Standby Server.

REGISTER LOGFILES AT STANDBY SERVER

SQL> alter database register logfile '/directory path/filename ';

SYS> alter database register logfile '/u01/app/oracle/flash_recovery_area/STBYCRMS/

archivelog/2016_06_01/o1_mf_1_552_cnvodqq5_.arc';

Database altered.

SYS> alter database register logfile '/u01/app/oracle/flash_recovery_area/STBYCRMS/

archivelog/2016_06_01/o1_mf_1_553_cnvohcy1_.arc';

Database altered.

SYS> alter database register logfile '/u01/app/oracle/flash_recovery_area/STBYCRMS/

archivelog/2016_06_01/o1_mf_1_554_cnvokk6z_.arc';

Database altered.

SYS> alter database register logfile '/u01/app/oracle/flash_recovery_area/STBYCRMS/

archivelog/2016_06_01/o1_mf_1_555_cnvom1l4_.arc';

Database altered.

SYS> alter database register logfile '/u01/app/oracle/flash_recovery_area/STBYCRMS/

archivelog/2016_06_01/o1_mf_1_556_cnvomxkz_.arc';

Database altered.

Recovery process would start automatically or else stop the managed recovery process
(MRP) and re-start it once again; that’s it!
Oracle DBA Technology Explored by 8 Bits Virtual Training
 RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

WHY WE NEED TO REGISTER ARCHIVE LOGS AT STANDBY?

On Primary to Standby Server, archive logs were not transferred by the log transfer
service, (but through SCP) so the managed recovery process will not have any info
about these logs. Then manually transferred logs need to be registered with the
managed recovery process before they applied by the log apply service.

CHECK WITH FOLLOWING QUERIES

SQL> archive log list;

SQL> select max (sequence#) from v$archived_log;
SQL> select sequence#, archived, applied from v$archived_log
Where applied='YES' order by 1;

This is simple way to handle manually Resolving Gaps.

ROLL FORWARD PHYSICAL STANDBY USING RMAN INCREMENTAL BACKUP

The Standby database lags far behind from the Primary database then i noticed that
there is a huge sequence mismatch between Primary & Standby database. There could be
some reasons.

 Network unavailable between Primary and Standby database.

 Archive logs deleted/corrupted on Primary database.

In cases where a Physical Standby database is out of Sync with Primary database. In
case of archive logs are missing/corrupt we have to rebuild the Standby from scratch.
If the database size in terabytes again rebuilding the Standby database could be a
tedious job; but we a solution to resolve this kind of issues.

As a DBA you can go for an RMAN Incremental backup to sync a Physical Standby with
the Primary database; using the command RMAN BACKUP INCREMENTAL FROM SCN … create a
backup on the Primary database that starts at the standby database’s current SCN,
which can then be used to roll the Standby database forward in time.

Please assume bunch of archive logs are deleted/corrupted on the Primary database
Server before they are transferring to the Standby database Server. In this case, I
demonstrate an efficient way to Sync Standby with Primary (an alternative to rebuild
the Standby DB)!

DISASTER RECOVERY
11g DATABASE SERVER FOR PRIMARY -> 192.168.222.133 -> DB_UNIQUE_NAME -> CRMS
11g ATABASE SERVER FOR STANDBY -> 192.168.222.134 -> DB_UNIQUE_NAME -> STBYCRMS

Oracle Database Software Version is 11.2.0.1.0 on OEL (5.5)

# On Primary database

SYS> select name, db_unique_name, database_role from v$database;

NAME DB_UNIQUE_NAME DATABASE_ROLE

--------- ------------------------------ ----------------
CRMS crms PRIMARY

# On Standby database

SYS> select name, db_unique_name, database_role, open_mode from v$database;

NAME DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE

--------- --------------------- --------------- --------------------
CRMS stbycrms PHYSICAL STANDBY MOUNTED

# On Primary database (Capture

Oracle DBA Technology Explored by 8 Bits Virtual Training
 RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

SYS> select thread#, max (sequence#) from v$archived_log group by thread#;

THREAD# MAX(SEQUENCE#)
---------- --------------
1 653

# On Standby database

SYS> select thread#, max (sequence#) from v$archived_log where applied='YES' group
by thread#;

THREAD# MAX (SEQUENCE#)

---------- --------------
1 558

Note the difference, On Standby last applied SEQUENCE# 558 but on Primary SEQUENCE# 653

# Finding Archive Gap SEQ# at Standby database

SYS> select * from v$archive_gap;

THREAD# LOW_SEQUENCE# HIGH_SEQUENCE#

-------- ------------- --------------
1 559 644

SYS> @archive_gap.sql

DB_NAME HOSTNAME LOG_ARCHIVED LOG_APPLIED APPLIED_TIME

LOG_GAP
------- ----------- ------------- ------------ --------------- ------
CRMS OEL5 653 558 01-JUN/04:33 95

CANCEL MANAGED RECOVERY PROCEES

# On Standby database

SYS> alter database recover managed standby database cancel;

Database altered.

GET LAST SCN OF THE STANDBY DATABASE

Last SCN of the STANDBY, will be used for the RMAN Incremental backup at Primary DB Server.

# On the Standby database Server (Checking last SCN of the Standby)

SYS> select current_scn from v$database;

CURRENT_SCN
-----------
3473651

# On the Primary database Server (Capture current_scn at primary)

SYS> select current_scn from v$database;

CURRENT_SCN
-----------
4538393

Note the SCN DIFFERENCE -> 4538393 – 3473651 = 1064742 (Now the Standby is lag behind).
But I want to know how long the Standby database is far behind in terms of (hh/dd).
To know that I used the scn_to_timestamp function to translate the SCN to timestamp.

# On Primary database

SYS> select scn_to_timestamp(4538393) from dual;

Oracle DBA Technology Explored by 8 Bits Virtual Training
 RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

SCN_TO_TIMESTAMP (4538393)
---------------------------------------
01-JUN-16 11.12.52.000000000 PM

# I execute following SQL Statement at Primary database (NOT workable at Standby)

SYS> select scn_to_timestamp(3473651) from dual;

SCN_TO_TIMESTAMP (3473651)
--------------------------------------------------
01-JUN-16 04.32.44.000000000 AM

The Standby database far behind more than 18 hours from the Primary database.
NOTE: Query NOT WORKABLE in OPEN READ ONLY MODE or MOUNTED MODE.

CHECKING ARCHIVE LOGS AT PRIMARY

$ ls /u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/
o1_mf_1_645_cnxo02c4_.arc o1_mf_1_647_cnxo9ndy_.arc o1_mf_1_649_cnxoc4d6_.arc
o1_mf_1_651_cnxpl4fd_.arc o1_mf_1_653_cnxpyh0c_.arc o1_mf_1_646_cnxo2329_.arc
o1_mf_1_648_cnxob5cl_.arc o1_mf_1_650_cnxp29x7_.arc o1_mf_1_652_cnxpyd51_.arc

To Sync Standby with Primary we need archive logs SEQUENCE# 559 to SEQUENCE# 644.
But these archives are NOT found at Primary Site; then only choice is RMAN
Incremental backup.

# Shutdown the Standby database

SYS> shutdown immediate;

On the Primary, take an Incremental backup from the SCN# (3473651) of the Standby
database. The last recorded SCN# (3473651) of the Standby database.

# connect target as Primary database

$ rman target sys/passwd@CRMSDB

.. Connected to target database: CRMS (DBID=1613387466)

RMAN> backup incremental from SCN 3473651 database format '/u03/rman-bkp/stby_%U'

TAG='FOR_STBY';
Starting backup at 02-JUN-16
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=52 device type=DISK backup will be obsolete on date 09-JUN-16 archived logs will
not be kept or backed up
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00003 name=/u01/app/oracle/oradata/crms/undotbs01.dbf
input datafile file number=00004 name=/u01/app/oracle/oradata/crms/users01.dbf
input datafile file number=00001 name=/u01/app/oracle/oradata/crms/system01.dbf
input datafile file number=00002 name=/u01/app/oracle/oradata/crms/sysaux01.dbf
input datafile file number=00007 name=/u01/app/oracle/oradata/crms/users02.dbf
input datafile file number=00008 name=/u01/app/oracle/oradata/crms/mytbs01.dbf
channel ORA_DISK_1: starting piece 1 at 02-JUN-16
channel ORA_DISK_1: finished piece 1 at 02-JUN-16
piece handle=/u03/rman-bkp/stby_03r73ftc_1_1 tag=FOR_STBY comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:13:13
using channel ORA_DISK_1
backup will be obsolete on date 09-JUN-16
archived logs will not be kept or backed up
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 02-JUN-16
channel ORA_DISK_1: finished piece 1 at 02-JUN-16
piece handle=/u03/rman-bkp/stby_04r73gm6_1_1 tag=FOR_STBY comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:04
Finished backup at 02-JUN-16
CREATE A NEW STANDBY CONTROLFILE

Oracle DBA Technology Explored by 8 Bits Virtual Training

 RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

On the Primary database, create a new Standby control file. We can use (SQL*Plus or
RMAN) to create a Standby control file; anyone method is enough.

SYS> alter database create standby controlfile as '/home/oracle/stdb_ctrl.ctl';

Database altered.

Or

# connect target as Primary & Create control file backup for Standby database

RMAN> backup current controlfile for standby format '/u03/rman-bkp/stdb_ctrl.ctl';

Starting backup at 02-JUN-16
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including standby control file in backup set
channel ORA_DISK_1: starting piece 1 at 02-JUN-16
channel ORA_DISK_1: finished piece 1 at 02-JUN-16
piece handle=/u03/rman-bkp/stdb_ctrl.ctl tag=TAG20160602T024937 comment=NONE channel ORA_DISK_1: backup
set complete, elapsed time: 00:00:01
Finished backup at 02-JUN-16

LISTING CREATED BACKUP SETS

$ ls -l /u03/rman-bkp/
total 4707832
-rw-r----- 1 oracle oinstall 4792238080 Jun 2 02:42 stby_03r73ftc_1_1
-rw-r----- 1 oracle oinstall 11927552 Jun 2 02:42 stby_04r73gm6_1_1
-rw-r----- 1 oracle oinstall 11927552 Jun 2 02:49 stdb_ctrl.ctl

TRANSFER BACKUP SETS TO STANDBY SERVER

Using OS utility to transfer these backups to the Standby Server from Primary Server.

# SCP all backup sets to the Standby Server at Primary Server

$ scp * oracle@192.168.222.134:/u03/bkp/
oracle@192.168.222.134's password: ******

stby_03r73ftc_1_1 100% 4570MB 2.4MB/s 31:43

stby_04r73gm6_1_1 100% 11MB 5.7MB/s 00:02
stdb_ctrl.ctl 100% 11MB 2.8MB/s 00:04

VERIFY RECEIVED BACKUP SETS ON THE STANDBY SERVER

$ ls -l /u03/bkp/
total 4707832

-rw-r----- 1 oracle oinstall 4792238080 Jun 2 05:40 stby_03r73ftc_1_1

-rw-r----- 1 oracle oinstall 11927552 Jun 2 05:40 stby_04r73gm6_1_1
-rw-r----- 1 oracle oinstall 11927552 Jun 2 05:40 stdb_ctrl.ctl

Note the location of all data files & Control file(s) at the Standby Database Server.

SYS> startup mount;

SYS> spool datafile_names.txt
SYS> set lines 100;
SYS> column name format a100;
SYS> show parameter control_files;
SYS> select count (*) from v$datafile;
SYS> select file#, name from v$datafile order by file#;
SYS> spool off SYS> shut immediate;
SYS>! cat datafile_names.txt

RELOCATE EXISTING CONTROL FILES

Oracle DBA Technology Explored by 8 Bits Virtual Training

 RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

# Renamed old control files at Standby Server

$ cd /u01/app/oracle/flash_recovery_area/stbycrms/
$ mv control02.ctl /tmp/control02.ctl.bkp
$ cd /u01/app/oracle/oradata/stbycrms/
$ mv control01.ctl /tmp/control01.ctl.bkp

NOTE: If you do not want rename control files, you can remove these files at OS
level.

BRINGUP THE STANDBY INSTANCE – NOMOUNT PHASE

$ export ORACLE_SID=stbycrms
$ rman target /

connected to target database (not started)

RMAN> startup nomount;

Oracle instance started

Total System Global Area 912306176 bytes

Fixed Size 1340244 bytes

Variable Size 536874156 bytes
Database Buffers 369098752 bytes
Redo Buffers 4993024 bytes

# Restore Standby control files from the backup

# Replacing newly created controlfile with old files

RMAN> restore standby controlfile from '/u03/bkp/stdb_ctrl.ctl';

Starting restore at 02-JUN-16

allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=19 device type=DISK
channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
output file name=/u01/app/oracle/oradata/stbycrms/control01.ctl
output file name=/u01/app/oracle/flash_recovery_area/stbycrms/control02.ctl
Finished restore at 02-JUN-16

SHUTDOWN THE STANDBY DATABASE & STARTUP MOUNT

RMAN> shutdown immediate;
Oracle instance shut down

# Startup the database with new controlfile

RMAN> startup mount;

connected to target database (not started) Oracle instance started database mounted
Total System Global Area 912306176 bytes
..
....

CATALOG THE INCREMENTAL BACKUP FILES AT STANDBY

RMAN does not know about these backup sets. We must catalog the new backup sets on
the Standby database. I.e. when we register these backup files to the Standby
database control file with RMAN CATALOG command, the Standby control file gets
updated about these backup files. It helps to recover the Standby using these RMAN
backup files.

# REGISTERING BACKUP SETS

Oracle DBA Technology Explored by 8 Bits Virtual Training

 RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

RMAN> catalog start with '/u03/bkp';

Starting implicit crosscheck backup at 02-JUN-16

allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=18 device type=DISK
Crosschecked 3 objects
Finished implicit crosscheck backup at 02-JUN-16
Starting implicit crosscheck copy at 02-JUN-16
using channel ORA_DISK_1
Crosschecked 2 objects
Finished implicit crosscheck copy at 02-JUN-16

searching for all files in the recovery area

cataloging files... cataloging done
List of Cataloged Files
=======================
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_646_cnxof4nl_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_647_cnxocnkr_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_652_cnxpz1lz_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_645_cnxodgj5_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_651_cnxodqq5_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_650_cnxp48lh_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_649_cnxoc5oq_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_653_cnxpzpvs_.arc
File Name:
/u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_648_cnxobwyc_.arc
searching for all files that match the pattern /u03/bkp

List of Files Unknown to the Database

=====================================
File Name: /u03/bkp/stby_04r73gm6_1_1
File Name: /u03/bkp/stby_03r73ftc_1_1 File
Name: /u03/bkp/stdb_ctrl.ctl
Do you really want to catalog the above files (enter YES or NO)? Y
cataloging files...
cataloging done
List of Cataloged Files
=======================
File Name: /u03/bkp/stby_04r73gm6_1_1
File Name: /u03/bkp/stby_03r73ftc_1_1
File Name: /u03/bkp/stdb_ctrl.ctl

APPLY INCREMENTAL BACKUP TO THE STANDBY DATABASE

The recovery process will use cataloged incremental backup sets because we have
registered. This backup taken for Physical Standby DB Sync; noredo key word is
required.

# Recover the database with Catalog incremental backup sets

RMAN> recover database noredo;

Starting recover at 02-JUN-16

using channel ORA_DISK_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 06/02/2016 05:55:09
RMAN-06094: datafile 6 must be restored

Oracle DBA Technology Explored by 8 Bits Virtual Training

 RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

If data files have been added to the Primary database during the archive log gap
time, they were not included in the incremental backup sets. We need to register
newly created files to the Standby database. We can find newly created files using
CURRENT_SCN of the Standby.

# Execute the following query at Primary DB - (Last SCN of the Standby database)

SYS> select file#, name from v$datafile where creation_change# > 3473651;

FILE# NAME
---------- --------------------------------------------
6 /u01/app/oracle/oradata/crms/users03.dbf

Newly created files should be restored at Standby before recovery operation.

RMAN> run {restore datafile 6;}

Starting restore at 02-JUN-16

using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00006 to
/u01/app/oracle/oradata/stbycrms/users03.dbf
channel ORA_DISK_1: reading from backup piece /u03/bkp/stby_03r73ftc_1_1
channel ORA_DISK_1: piece handle=/u03/bkp/stby_03r73ftc_1_1 tag=FOR_STBY
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:08
Finished restore at 02-JUN-16

# Recover the Standby database using Incremental backup Sets

RMAN> recover database noredo;

Starting recover at 02-JUN-16

using channel ORA_DISK_1 channel
ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00001:
/u01/app/oracle/oradata/stbycrms/system01.dbf
destination for restore of datafile 00002:
/u01/app/oracle/oradata/stbycrms/sysaux01.dbf
destination for restore of datafile 00003:
/u01/app/oracle/oradata/stbycrms/undotbs01.dbf
destination for restore of datafile 00004:
/u01/app/oracle/oradata/stbycrms/users01.dbf d
estination for restore of datafile 00005:
/u01/app/oracle/oradata/stbycrms/example01.dbf
destination for restore of datafile 00007:
/u01/app/oracle/oradata/stbycrms/users02.dbf
destination for restore of datafile 00008:
/u01/app/oracle/oradata/stbycrms/mytbs01.dbf
channel ORA_DISK_1: reading from backup piece /u03/bkp/stby_03r73ftc_1_1
channel ORA_DISK_1: piece handle =/u03/bkp/stby_03r73ftc_1_1 tag=FOR_STBY
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:42:18
Finished recover at 02-JUN-16

All changed blocks has been captured in the incremental backup and also updated at
thestandby database, thus brings the Standby database up to date with the primary
database.

# Start MRP at Standby database

SYS> alter database recover managed standby database disconnect;

Database altered.

Oracle DBA Technology Explored by 8 Bits Virtual Training

 RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

# Check CURRENT_SCN of the Standby Database & Primary database

SYS> select current_scn from v$database;

...

Check the SCN’s in Primary and Standby it should be close to each other.

ARCHIVE LOG SEQUENCE DETAILS AT PRIMARY & STANDBY DATABASE

# Maximum generated LOG SEQUENCE# at Primary

SYS> select thread#, max(sequence#) from v$archived_log group bythread#;

THREAD# MAX (SEQUENCE#)

---------- --------------
1 773

# Maximum archive log SEQUENCE# applied at Standby

SYS> select thread#, max (sequence#) from v$archived_log

Where applied='YES' group by thread#;

THREAD# MAX (SEQUENCE#)

---------- --------------
1 773

SYS> select process, status, thread#, sequence#, block#, blocks from

v$managed_standby;

...Wow! Successfully Standby database Synced with Primary database.

Check alert.log and status of processes if any problems;

REFERENCE DOCS:

https://docs.oracle.com/cd/E11882_01/server.112/e41134/rman.htm#SBYDB00759
http://docs.oracle.com/cd/B19306_01/backup.102/b14191/rcmdupdb.htm#sthref955
https://web.stanford.edu/dept/itss/docs/oracle/10gR2/backup.102/b14191/rcmdupdb008.htm

BELOW ARE THE STEPS I FOLLOWED FOR A SUCCESSFUL RECOVERY

# Check the archive log gap SEQUENCE#
SQL> select * from v$archive_gap;

# Find last SCN of the Standby database

SQL> select current_scn from v$database;

# Cancel the recovery

SQL> alter database recover managed standby database cancel;

# Shutdown the Standby

SQL> shutdown immediate;

# Backup the Source (Primary) using from SCN

RMAN> backup incremental from SCN XXXXX database format '/bkp_location/stby_U%'

# Backup current control file of the Source (Primary database)

RMAN> backup current controlfile for standby format '/bkp_location/stby_ctrl.ctl';

# SCP backup Sets to the target(Standby) Server from Primary

$scp /bkp_location/* oracle@standby:/standby_location/

# Startup Standby database at NOMOUNT mode

RMAN> startup nomount;

# Restore control files from the backup

RMAN> restore standby controlfile from '/standby _location';

# Shutdown the Standby database

Oracle DBA Technology Explored by 8 Bits Virtual Training
 RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

RMAN> shutdown immediate;

# Startup the Standby database at MOUNT

RMAN> startup mount;

# Register backup sets by a process called cataloging

RMAN> catalog startwith '/standby _location';

# Perform Recover
RMAN> recover database noredo;

# Restart MRP at standby

SQL> alter database recover managed standby database disconnect from session;

# Check SCN of the Primary and Standby database

SQL> select current_scn from v$database;

These are Steps I have done to roll the Standby database forward in time. That’s it!

METHODS OF GAP RESOLUTION

 Automatic Gap Resolution
 FAL (Fetch Archive Log) Gap Resolution
 Manual Gap Resolution
 Roll forward using RMAN Incremental Backup (Physical Standby)

AUTOMATIC GAP RESOLUTION

If connectivity is lost between the Primary and Standby databases (due to network
problems), redo data being generated on the primary database cannot be sent to the
Standby database. Once a connection is reestablished, the missing archived redo log
files are automatically detected by Data Guard, which then automatically transmits
the missing archived redo log files to the Standby databases. The Standby databases
are synchronized with the Primary database, without manual intervention by the DBA.
How this happens? Let’s dig into deep.

Automatic Gap Resolution is performed automatically by the Log Transport Services.

Basically the currently transferred Redo is compared with the last received.

On Primary ARCH/LGWR to transfer archived redo logs to the Standby.

On the Standby RFS process receives archive redo log file.

RFS process Compares the Sequence number of currently being archived file with the
sequence number of previously received archived redo file; (if currently being
archived redo file sequence# is greater than the last sequence# received plus one,
there is a gap). An archived redo log file is uniquely identified by its sequence
number and thread number.

 o1_mf_ 1 _ 648 _cnxob5cl_.arc -> Last archived log file.

 o1_mf_ 1 _ 652 _cnxpyd51_.arc -> currently being archived file.

Now 3 files are missing. There is a gap between both files, and then RFS
automatically requests the missing redo log sequence# from the Primary DB again via
the ARCH-RFS Heartbeat Ping. The archiver of the Primary then retransmits these
missing archived redo files.

o1_mf_1_649_cnxoc4d6_.arc, o1_mf_1_ 650 _cnxp29x7_.arc, o1_mf_1_651_cnxpl4fd_.arc

This type of Gap Resolution is using the Service defined in log_archive_dest_n on the
Primary databaseserving this Standby database.

SYS> show parameter log_archive_dest_2;

NAME TYPE VALUE

----------------------- ----------- --------------------------------------------
log_archive_dest_2 string Service=stby_crmsdb LGWR ASYNC
VALID_FOR= (ONLINE_LOGFILES, PRIMARY_ROLE)
DB_UNIQUE_NAME=stbycrms
Oracle DBA Technology Explored by 8 Bits Virtual Training
 RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

The archiver process of the Primary database polls the Standby databases every minute
- (Referred to as heartbeat) to see if there is a gap in the sequence of archived
redo logs. If a gap is detected, the ARCH process sends the missing archived redo log
files to the Standby databases that reported the gap. Once a gap is resolved then the
Transport Process (ARCH/LGWR) is notified about the Resolution of the Gap.

FAL GAP RESOLUTION

As I said above, RFS receives an archive log on the Standby database, and then the
archive logs are registered in the Standby Control file with name and location.
Missing log files are typically detected by the Log Apply Services (MRP) on the
Standby database. If the archived redo log file is missing/corrupted for any reason
(eg. it got deleted). FAL is required to resolve a Gap, to obtain a new copy of the
corrupted or deleted file.

WHY ESPECIALLY FAL IS REQUIRED?

Since MRP has NO direct communications (LOG TRANSPORT SERVICES) of the primary
database; it must use the FAL_SERVER and FAL_CLIENT initialization parameters to
resolve the gap. Both of these parameters must be set in the Standby Initialization
parameter file.

The FAL_CLIENT parameter is no longer required since Oracle 11g R2.

In earlier releases, you set the FAL_CLIENT parameter on the Standby database.

FAL_SERVER AND FAL_CLIENT

FAL_SERVER and FAL_CLIENT need to be defined in the initialization parameter file of
the Standby database(s). They mainly used for GAP RESOLUTION through FAL background
process. Using FAL parameters the MRP in the Physical Standby database checks
automatically and also Resolving gaps at the time redo is applied.

FAL_SERVER = NET_SERVICE_NAME of Primary database.

 From where the missing archive log(s) should be requested.

FAL_CLIENT = NET_SERVICE_NAME of Standby database.

 The destination where the FAL_SERVER database should send the archived log(s).

SYS> show parameter fal; # On Standby database

NAME TYPE VALUE

-------------------- ------------- ----------------------------------
fal_client string STBY_CRMSDB
fal_server string CRMSDB

In earlier release when you set FAL_CLIENT parameter on the Standby database, the
Primary database (FAL_SERVER)uses Standby database (NET SERVICE NAME) to connect the
Standby DB.

The Parameter log_archive_dest_n on FAL_SERVER database which is pointing to the

Standby database to resolve the Gap Resolution.

# On Primary database

SYS> show parameter log_Archive_dest_2;

NAME TYPE VALUE
--------------------- ----------- ------------------------------
log_archive_dest_2 string Service=stby_crmsdb LGWR ASYNC
VALID_FOR= (ONLINE_LOGFILES, PRIMARY_ROLE)
DB_UNIQUE_NAME=stbycrms

FAL_SERVER: An Oracle NET SERVICE NAME (TNS-Alias or Connect Descriptor). This

parameter must be configured on the Standby database System that points from where
the missing archive log(s) should be requested. It’s recommended to set FAL_SERVER on

Oracle DBA Technology Explored by 8 Bits Virtual Training

 RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE

the Primary DB with the value of the Standby database NET SERVICE NAME thus helps
when you do switch over event.

Once the Log Apply Services (MRP) detect an ARCHIVE GAP, then it sends a FAL Request
to the FAL_SERVER; Once communication has been established with the Primary database,
it passes the Sequence number of archived files (which are causing for the archive
gap) to retransmit from the archiver process of the Primary database; and
additionally it passes the service name defined by the FAL_CLIENT parameter to the
Primary ARCH process.

Fetching gap sequence in thread 1, gap sequence 1076-1077

FAL [client]: Trying FAL server: CRMSDB

An ARCH Process on the FAL_SERVER tries to pick up the requested Sequences from that
database and sends them to the FAL_CLIENT. I.e. The Primary database ARCH process
Ships the requested archived logs to the remote archive destination of corresponding
service name.
# SNIPPET OF STANDBY ALERT.LOG

RFS [54]: Opened log for thread 1 sequence 1076 dbid 1613387466 branch 913081878
Media Recovery Log u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_04/
o1_mf_1_1076_co5qn3sd_.arc

RFS [55]: Opened log for thread 1 sequence 1077 dbid 1613387466 branch 913081878
Media Recovery Log u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_04/
o1_mf_1_1077_co5qn4db_.arc

In order to successfully complete a Gap Request the requested archive log Sequence(s)
must be available on the FAL_SERVER database.

When you have multiple Physical Standby databases, the FAL mechanism can
automatically retrieve missing archived redo log files from another Physical Standby
database.

IF THE FAL_SERVER IS NOT ABLE TO RESOLVE THE GAP?

The FAL-Request fails and a corresponding Message will be put in the ALERT.LOG of the
Standby database. Example taken for some other SEQUENCE (1078 – 1081).

FAL [client]: Failed to request gap sequence

GAP - thread 1 sequence 1078-1081
DBID 1613387466 branch 913081878
FAL [client]: All defined FAL servers have been attempted.
-------------------------------------------------------------

Every minute the Primary database polls its Standby databases to see if there
are gaps in the sequence of archived redo log files.
 The FAL (Client) requests to transfer archived redo log files automatically.
 The FAL (Server) services the FAL requests coming from the FAL Client.
 A separate FAL server is created for each incoming FAL client.

FAL is available since Oracle 9.2.0 for Physical Standby database and Oracle 10.1.0
for Logical Standby databases. In 11g , if you not set FAL_CLIENT parameter, the
Primary databasewill obtain service name from related
LOG_ARCHIVE_DEST_n parameter.

REFERENCE LINKS
http://flylib.com/books/en/1.145.1.66/1/ https://docs.oracle.com/cd/B19306_01/server.
102/b14239/log_transport.htm#i1268294 Oracle

Note: Data Guard Gap Detection and Resolution Possibilities [ID1537316.1]

Oracle DBA Technology Explored by 8 Bits Virtual Training