<Insert Picture Here>

Developing a Successful Backup and Recovery

Strategy: Customer !periences
Timothy Chien
Senior Product Manager
Oracle
Jonathan Intner
Global Head of Database Security
Noartis
Gene Psoter
!ead " CIS Technology Team
Pacific Gas # $lectric
The follo%ing is intended to outline our general
&roduct direction' It is intended for information
&ur&oses only( and may not be incor&orated into any
contract' It is not a commitment to delier any
material( code( or functionality( and should not be
relied u&on in ma)ing &urchasing decisions'
The deelo&ment( release( and timing of any
features or functionality described for Oracle*s
&roducts remain at the sole discretion of Oracle'
<Insert Picture Here>
"genda
+
Oracle*s Data ,e&air Solution
+
Noartis Case Study
+
Pacific Gas # $lectric Case Study
+
Summary
+
-#.
Data Repair
#racle$s Integrated H" Solutions
System
C%anges
Data
C%anges
&nplanned
Do'ntime
Planned
Do'ntime
Real Application Clusters
ASM
H.A.R.D
Data Guard
Streams
Online Reconfiguration
Rolling Upgrades
Online Redefinition
O
r
a
c
l
e

M
A
A

B
e
s
t

P
r
a
c
t
i
c
e
s
Flas!ac" #ecnologies
RMA$ % Oracle Secure Bac"up
System
(ailures
Data
(ailures
(las%)ack *ime +avigation
+ (las%)ack ,uery
+ ,uery all data at point in time
+ (las%)ack -ersions ,uery
+ See all versions of a ro' )et'een t'o
times
+ See transactions t%at c%anged t%e ro'
+ (las%)ack *ransaction ,uery
+ See all c%anges made )y a transaction
*! .
*! /
*! 0
Select * from Emp AS OF 2:00 P.M. where
Select * from Emp VERSIONS E!"EEN
2:00 PM #$% &:00 PM where
Select * from F'AS(A)*+!RANSA)!ION+,-ER.
where /0% 1 0002000&000000223
(las%)ack rror Correction
+ Recovery at all levels
+ Data)ase 1evel
+ (las%)ack Data)ase restores t%e
'%ole data)ase to a point2in2time
+ &ses (las%)ack 1ogs
+ *a)le 1evel
+ (las%)ack *a)le restores ro's in a
set of ta)les to a point2in2time
+ &ses &+D# in data)ase
+ (las%)ack Drop restores a
dropped ta)le or an inde!
+ Recycle )in for DR#Ps
+ Ro' 1evel
+ (las%)ack ,uery restores ro's to
time
#rder
Data)ase
Customer
Recovery 3anager 4R3"+5
#racle$s Backup 6 Recovery &tility
3edia 3anagement 1ayer
nterprise nterprise
3anager 3anager
6 0 6 0
rd rd
Party *ools Party *ools
Network
+ Integral feature of the database that sends data
to dis) and ta&e
+ Smart
+ So&histicated bac)u& and recoery
strategies
+ /ast
+ O&timi0ed bac)u& to dis) for fastest
recoery
+ 1loc) leel incremental bac)u&
+ ,eliable
+ 1loc) contents are alidated during
bac)u&
+ $asy
+ Sim&le management %ith $nter&rise
Manager
+ Su&&orts oer 23 Media Managers
+ 4eritas( !egato( Tioli( HP( Oracle
Secure 1ac)u&( etc'
Installed and ready to use5
Recovery 3anager
#racle Secure Backup
*ape Backup 3anagement

Protects your entire

enironment

Oracle integrated
adantage

/astest database
ta&e bac)u&

Ma6imum bac)u&
security

1o' cost7
#racle Secure Backup
Centrali8ed *ape Backup 3anagement
(ile System Data (ile System Data
&+I9 1inu!
:indo's +"S
*ape
#racle Data)ases #racle Data)ases
Integration 'it%
R3"+
<Insert Picture Here>
+ovartis
Backup 6 Recovery Case Study
Jonathan Intner
Noartis
2 Use of RMAN at Novartis / J onathan Intner / 24 October, 2006
A4e$%#
"ho #m I5
"h6 #m I here5
RMAN Pro7ect:
#80c Pro7ect 9o#l8
#c:;p #rch0tect;re pr0or to RMAN
Re<;0re% e/pert08e for # 8;cce88f;l RMAN Pro7ect
RMAN #rch0tect;re
I88;e8 e$co;$tere% 0mpleme$t0$4 the S6m#$tec Net#c:;p=N->
API
3 Use of RMAN at Novartis / J onathan Intner / 24 October, 2006
#80c Pro7ect 9o#l8
Impro?e #?#0l#@0l0t6.
Impro?e reco?er#@0l0t6.
E$8;re the 0$te4r0t6 of Or#cle @#c:;p8.
!04hter 0$te4r#t0o$ of the Or#cle @#c:;p8 w0th the e/08t0$4 @#c:;p
0$fr#8tr;ct;re.
)o8t 8#?0$48
Allow the 2A8 to h#?e
!he #@0l0t6 to @#c:;p # %#t#@#8e #8 #$ e$t0t6.
More co$trol of the ?er0Ac#t0o$ of the 8;cce88 of the @#c:;p8
4 Use of RMAN at Novartis / J onathan Intner / 24 October, 2006
E$?0ro$me$t
Pr0m#r6 2A for h;$%re%8 of %#t#@#8e8 #cro88 te$8 of 8er?er8B
8er?0$4 #lmo8t e?er6 #ppl0c#t0o$ 0m#40$#@le.
!he %08: f#rm 8;pport0$4 the8e #ppl0c#t0o$8 #re 0$ the tee$8 of
ter#@6te8.
Ser?er OSe8 0$cl;%e:
Sol#r08
(PC-D
AID
"0$tel e/cl;%e% from co$80%er#t0o$ =for $ow>
Ver80o$8 of Or#cle:
EFGH I0R2 =I.2.0.J.0B I.2.0.G.0B I.2.0.K.0B I.2.0.F.0B I.2.0.L.0>
EM2H M04RM or R2
EM2.GH L.M.F.J
5Use of RMAN at Novartis / J onathan Intner / 24 October, 2006
Backup arc%itecture prior to R3"+
Serer"based 7ee)ly Cold 1ac)u&s
.rchie logs 8ty&ically9 )e&t for : month
. critical ;uestion for our customers became<
.re &lanned outages still outages=
6 Use of RMAN at Novartis / J onathan Intner / 24 October, 2006
Re;uired !pertise for a Successful
R3"+ Pro<ect
Pro>ect Manager
Document( document( document
/inance
+ Ca&ital
+ $6&ense
D1.
Media Manager
System .dministrator
Storage .dministrator
7 Use of RMAN at Novartis / J onathan Intner / 24 October, 2006
Arch0tect;re Alter$#t0?e8
Alter$#t0?e8:
(#?e RMAN cop6 the Ale8 from %#t#@#8eCtoC%08: #$% the$ h#?e
Net#c:;p=N-> cop6 the Ale8 from %08:CtoCt#pe.
Re7ecte% c#;8e m#$#40$4 the @#c:;p8 0$ N- wo;l% @e too
compl0c#te%.
-8e the N- API to h#?e RMAN cop6 Ale8 from %#t#@#8eCtoCt#pe.
Now co$80%er0$4 ;80$4 Or#cle Sec;re #c:;p #$% # V0rt;#l !#pe
'0@r#r6 =V!'> wh0ch wo;l% ;8e Or#cle Sec;re #c:;p #8 the Me%0#
M#$#4er.
Use of RMAN at Novartis / J onathan Intner / 24 October, 2006
(#?e RMAN cop6 the Ale8 from %#t#@#8eCtoC%08: #$% the$
h#?e Net#c:;pcop6 the Ale8 from %08:CtoCt#pe
RMAN
Catalog
RMAN Catalog
Server
Tape
Client
DB
DB Server
RMAN
S-ATA
Disk
NBU
! Use of RMAN at Novartis / J onathan Intner / 24 October, 2006
-8e the Net#c:;pAPI to h#?e RMAN cop6 Ale8
from %#t#@#8eCtoCt#pe.
RMAN Catalog
Server
NBU Vaulting
process copies data
fro ATA disk to
tape!
Tape
DB Server
RMAN
Catalog
S-ATA
Disk
"#isting
Media
Server
RMAN started on client
DB!
NBU A$% copies data&ase
onto ATA disk pool
connected to an e#isting
edia server!
RMAN
Client
DB
NBU
A$%
"0 Use of RMAN at Novartis / J onathan Intner / 24 October, 2006
(#r%w#re c;rre$tl6 0$ ;8e
'#r4e %08: pool of SANC#tt#che% SA!A %08:.
2e%0c#te% N- Me%0# Ser?er for RMAN.
)o$80%er0$4 # @#c:;p $etwor: to m0$0m0Ne the 0mp#ct o$ the m#0$
$etwor:.
Re#l Appl0c#t0o$ )l;8ter 8;pport0$4 the RMAN )#t#lo4
"" Use of RMAN at Novartis / J onathan Intner / 24 October, 2006
)o$cl;80o$
)h#lle$4e8
)r0t0c#l S;cce88 F#ctor8:
Profe880o$#l Pro7ect M#$#4eme$t
)omm0tte% #$% %e%0c#te% pro7ect p#rt0c0p#$t8
S;Oc0e$t re8o;rce8 to 8;cce88f;ll6 complete the pro7ect
"2 Use of RMAN at Novartis / J onathan Intner / 24 October, 2006
Pro7ect M#$#4eme$t for 2A8*
2oc;me$tB %oc;me$tB %oc;me$t
F0$#$ce
)#p0t#l
E/pe$8e
<Insert Picture Here>
Pacific =as and lectric
Backup 6 Recovery Case Study
$ugene Psoter
Pacific Gas and $lectric Com&any
Base Content Slide
+
/irst !eel 1ullet
+ Second leel bullet
+ Third leel bullet
+ /ourth leel bullet
+ /ifth leel bullet
PG#$ .genda
+ Introduction
+ 1ac)ground on the Com&any and .&&lication
+ Driers for Change and the Solution
+ Designing for /ailures
+ High !eel Princi&les and Trade"offs
+ /ailure .nalysis and ,ecoery by Design
PG#$ Com&any Info
+ Serice .rea" ?3(333 s;' mi' in northern and central
California
+ Serice .rea Po&ulation @ :A million &eo&le 8about AB
of CS &o&9
+ $lectric Distribution customer accounts @ A'3 million
+ Natural Gas Distribution Customer .ccounts @ D'2
million
+ ,eenue E::'? 1illion
+ .ssets< EFD 1illion
+ System< :DG(?DD circuit miles of electric transmission
and distribution lines( DG(HF2 miles of natural gas
transmission and distribution &i&elines( GD23 M7 of
generation
+ $lectricity deliered in 233A< H:(G2G G7h
+ Natural Gas Through&ut in 233A< HDD(3GH Million Cubic
/eet MMcf
+ $m&loyees :I(H33
+ Shareholders< a&&ro6 233(333
To become the
leading utility in
the United States
Our Vision
The .&&lication
Customer Care # 1illing
"pplication
+ CC#1 :'A from SP! 7orldgrou&
+ 2F33 concurrent online users(
F33(333J bills &er day( Internet
&resence
+ Su&&orts o&erational systems for field
force dis&atch( outage management(
etc'
+ Nearly ?62D6? u&time re;uirement 823
minute read"only nightly %indo% in
&lace( scheduled %ee)end outages
usually acce&table( %eather
de&endent9
*ec%nology
+ Database< :D T1 of data using
D12 on a 0Series mainframe
+ .&& serers< F Tu6edo 8online9(
D batch
+ 7eb Serers< I serers %ith 2D
instances of 7eb!ogic
+ Daily @ : 1illion selectKfetch(
:33 Million insertKu&dateKdelete
Backup and Recovery
+ S.N SN.PShot( bac)ed u& to
ta&e daily( PP,C to D, center
+ D12 online image co&ies daily
to ta&e "" all data and catalog'
1usiness Driers for Change
+ Increasing loads due to data gro%th J internet gro%th
+ !ong 8?yr9 data retention re;uirements from arious regulatory and
legislatie rules
+ SmartMeter initiatie %ill significantly increase database si0e 8DAJ T19
+ Costs associated %ith su&&orting gro%th on the mainframe &latform
8Cnit of Measure is MillionsE for mainframe( :33LsE Mid"Tier9
Ley Pro>ect Driers
+ Su&&ort the business re;uirements
+ Sim&lify o&erations
+ ,educe costs
@ Note &otential conflict %ith aboe 2 items
Solution
+ Moe to Oracle database running on Mid"Tier Cni6 serers
+ Ctili0e Oracle ,.C to &roide incrementally scalable gro%th and
high aailability to the system
+ Ctili0e Oracle technologies to the ma6 8.SM( ,M.N( /lashbac)(
etc9 to sim&lify the soft%are stac)
+ Moe a%ay from a reliance on ta&es as a first tier recoery
mechanism
Database Serer Design
Pro>ect Timelines
>an ?@ 3ar ?@ >un ?@ Sep ?@ Dec ?@ 3ar ?A
Pro<ect
Inception
Business Case
Re;uirements
Initial Design
Cost stimate
Small Scale
R"C Cluster
P#C
Specify Re;uirements
"c;uire 6 Install DRB*est System
Configure DRB*est System
"c;uire 6 Install PR#D System
(ull Scale Hard'are
*esting
Backup 6 RecoveryBData =uard
Sept C +ov D?E
"pplication 6 Interface
#ct C Dec D?E
#R* 2 >an $?A 4Cycle .5F (e) $?A
4Cycle /5F 3ar $?A 4Cycle 05
>an ?E 3ar ?E >un ?E Sept ?E Dec ?E >an ?A
"c;uire
Install
Decision to move for'ard
1arge Scale
P#C
Data
Conv
(ull Scale Data Conversion
.G*B
Production
"pr ?A
7hy 7orry=
Anything That Can Possibly Go
Wrong, Does
epigraph of
ohn Sac!"s
The Butcher: The Ascent of Yerupaja #$%&'(
Gene )ran*
+light Director ,ASA
During the Apollo -... Crisis
+ailure is not
an option
BUT ->
Designing for /ailures
+ 7hat can fail 8and ho% often9
+ 1usiness tolerance for outages
@ 8PG#$9 Planned %ee)end outages can be accommodated(
%eather &ermitting
+ 1usiness continuity re;uirements for disasters
@ 8PG#$9 DH Hours to recoer %ith M 2D hours data loss
+ Strategies to accommodate failures %hile meeting re;uirements
Design Tradeoffs
+ Meet the business re;uirements
@ Does the current re;uirement seem reasonable for a ne% system=
@ Can you do better %ithout increasing costs K ris)s=
@ Do the ne% re;uirements change the design that you are currently
using=
*%e fundamental ;uestion is:
Ho' muc% do you over)uildH
Design Tradeoffs
+ Sim&lify O&erations
@ O&erating a com&le6 system introduces ris)s that something %ill go
%rong
@ The o&erational costs for the system %ill( oer time( e6ceed the
construction costs
@ Nour design can hel& com&ensate for this by<
+ ,educe the number of soft%are # hard%are technologies in the design
+ .utomate &rocesses as much as you can
+ Cse redundancy for trans&arent failure recoeries
+ ,educe costs
@ ,educe the number of soft%are and hard%are technologies
@ Cse lo%er end hard%are %here &ossible
@ Sim&lify o&erations
Things to ConsiderO
+ ,.C s single instance s High .ailability OS Cluster
+ Storage arrays @ $nter&rise s Mid"Tier
+ ,M.N s other soft%are
+ Dis) bac)u&s s ta&e
+ /lashbac) o&tions s restores
+ S.N PSna&Shot* ca&abilities s storage costs
+ S&inning D, s ,estore from bac)u&
+ Net%or) band%idth bet%een centers
1ac)u& .nd ,ecoery Princi&les
+ /ailures %ill ha&&en( so &lan accordingly
+ Prefer hard%are redundancy %here it is cost effectie
+ If a deice is e6&ected to fail occasionally 8as o&&osed to
rarely9( assume you %ill hae a concurrent double failure on that
deice 8e'g' lose 2 dis)s on a raid array9
+ Cse the Qdefense in de&thR a&&roach %here multi&le layers of
&rotection are built in to the design'
+ Prefer to use database recoery tools 8sim&lify soft%are stac)9
oer other o&tions
/ailure .nalysis "
%hat can fail and ho% to recoer
+ Single and Multi&le hard%are failures
+ Physical database corru&tion
+ !ogical database corru&tion
+ Disaster
,ecoery by Design
Trans&arent recoery for routine failures
@ Cse ,.ID to &rotect<
+ Single Dis) Drie( Multi&le Dis) Dries( Dis) .rray
Dra%er
@ Cse redundant hard%are to &rotect<
+ Dis) .rray Controller( S.N S%itch( S.N Net%or)( H1.(
,.C Node
Interention recoery for rare failures<
@ Cse ,ecoery and D, to &rotect<
+ Multi&le ,.C node failures( Storage Subsystem failure(
Physical and !ogical database corru&tion( Data Center
failure
,ecoery Settings
+ Daily bac)u& to dis) 8/lash ,ecoery .rea9 on
&rimary and standby database
@ !eel 3 incremental first day( leel : incremental on days
thereafter
@ Incrementally C&dated 1ac)u&s 8roll for%ard eery F days9
@ 1loc) Change Trac)ing enabled on &rimary database
+ One month bac)u& cycle 8full leel 3 each month9
+ Lee& &reious month co&y 82 full on"dis) co&ies9
,ecoery Settings
+ .rchie logs
@ On &rimary database "" multi&le6ed F %ays 8/,.( : local( :
&hysical standby9'
@ On &hysical standby database "" multi&le6ed 2 %ay 8/,.( :
local9'
+ .rchie log s%itches tuned to 2":A minute interals
+ Data Guard !G7, .SNNC mode
+ 7ee)ly full bac)u& from &hysical standby database
/,. to ta&e
,ecoery O&tions
+ Physical Database Corru&tion
@ ,M.N datafile or tables&ace restore and recoery
@ ,M.N bloc) media recoery
+ !ogical Database Corru&tion
@ /lashbac) Table
@ /lashbac) -uery
@ /lashbac) Database
@ ,M.N TSPIT, or D1PIT,
,ecoery O&tions
+ Disaster ,ecoery
@ If &roduction storage affected( s%itch to /lash ,ecoery
.rea image co&ies
@ Data Guard /ailoer
@ S%itch 1ac)
PG#$ Summary
+ /ailure %ill ha&&en( so s&end the effort to design for it
+ Ta)e a holistic ie% of the database system and
reali0e your tradeoffs and leerage &oints
+ Ma)e the system as sim&le as &ossible( but no
sim&ler
+ ,eference Documents
@ Backup and Recovery Basics 10g Release 2
@ Backup and Recovery Advanced Users Guide 10g Release 2
@ Using Recovery Manager with racle !ata Guard in racle !ata"ase 10g
Summary
+
Oracle*s Data ,e&air Solution
+ /lashbac) Technologies
+ ,ecoery Manager
+ Oracle Secure 1ac)u&
+
$ffectie 1ac)u& and ,ecoery Im&lementation
+ Identify &ain &oints
+ Design for the long"term
+ Identify resources early
+ Phased a&&roach
+ Test( test( test
<Insert Picture Here>
,uestions and "ns'ers