Oracle 11g: Learning to Love the ADR

Presented by: Don Seiler

About Don Seiler

From Manitowoc, Wisconsin Oracle DBA since !!1 Starte" with Oracle #$%$& on '()*+ With (,thian since Ma, !!Blogger ) www$seiler$us

2011 Pythian

Why Companies Trust Pythian

.
.

Recognized Leader:

Global industry-leader in remote database administration ser ices and consulting !or "racle# "racle $%%lications# &yS'L and S'L Ser er (or) *ith o er 1+0 multinational com%anies such as ,orbes-com# ,o. /nteracti e &edia# 0ordion and (estern 1nion to hel% manage their com%le. /2 de%loyments

.
.

3.%ertise:

"ne o! the *orld4s largest concentrations o! dedicated# !ull-time D5$ e.%ertise- 3m%loy 6 "racle $73s8$73 DirectorsS%ecialized in "racle Database# R$7# Data (arehousing# Per!ormance 2uning# "racle 3.adata# GoldenGate and 3nter%rise Linu.-

.
.

Global Reach 9 Scalability:

2:8;8<6+ global remote su%%ort !or D5$ and consulting# systems administration# s%ecial %ro=ects or emergency res%onse

<

2011 Pythian

Diagnostic Files

What Are We /al0ing About1

Alert Log 2RDBMS, ASM, Listener3 /race Files

*ser)4enerate" 21!!&5, 1!!6%3 S,stem)4enerate" 2ORA error3

7ore Dum8s

2011 Pythian

Diagnostic Files in 1!g

9ORA7L:;BAS:<a"min<9ORA7L:;S=D<

b"um8 > alert log ? bac0groun" trace @iles c"um8 > core "um8 @iles u"um8 > user trace @iles 21!!&5, 1!!6%3 a"um8 > au"it @iles

These are defaults, overridden with %_dump_dest parameters.

2011 Pythian

Diagnostic Files in 11g

Automatic Diagnostic Re8ostor, 2ADR3A S8eci@ie" b, D=A4BOS/=7;D:S/

Bew initialiCation 8arameter, re8laces D;"um8;"est

=@ not set, "e@aults to ORA7L:;BAS:

=@ ORA7L:;BAS: not set, "e@aults to ORA7L:;'OM:<log

First creates a E"iagF "irector, insi"e its home$ 7hec0 G9D=A4;=BFO @or location "etails
2011 Pythian

G9D=A4;=BFO
SQL> select name, value from v$diag_info; NAME VALUE !iag Ena"led A!$ %ase A!$ (ome !iag #race !iag Alert !iag )ncident !iag *dum+ (ealt, Monitor !efault #race -ile Active 2ro"lem *ount Active )ncident *ount 33 ro4s selected1 #$UE c&'oracle c&'oracle'diag'rd"ms'orcl'orcl c&'oracle'diag'rd"ms'orcl'orcl'trace c&'oracle'diag'rd"ms'orcl'orcl'alert c&'oracle'diag'rd"ms'orcl'orcl'incident c&'oracle'diag'rd"ms'orcl'orcl'cdum+ c&'oracle'diag'rd"ms'orcl'orcl',m c&'oracle'diag'rd"ms'orcl'orcl'trace'orcl_ora_./0.1trc / /

2011 Pythian

WhatHs in the ADR1

(ro"uct /,8e: asm < r"bms < tnslsnr < clients Both human)rea"able 2trace3 an" +ML alert logs /race @iles 21!!&5 etc$ in trace "irector,3

=nclu"ing SILBet trace @iles

=nci"ent @iles 7ore "um8 @iles

2011 Pythian

>

Disabling the ADR

Set hi""en 8arameter:

alter s5stem set 6_diag_adr_ena"led67false sco+e7s+file;

Be sure to set the ol" J;"um8;"est 8arameters Bot a"vise" 2obviousl,3A

10

2011 Pythian

/oo Man, /race FilesA

First thing = notice" a@ter u8gra"ing to 11gA 7an be greatl, re"uce" with hi""en 8arameter:

Set ;"isable;health;chec0 K /R*: DonLt set in 8ro"uction without consulting Oracle Su88ortA

2011 Pythian

Iuic0 /erminolog, Lesson

(roblem > A critical error in the "atabase 2e$g$ ORA)!!5!!, !#&&6, !&!%13 =nci"ent > A single occurence o@ a 8roblem$ Follows =/=LA

12

2011 Pythian

ADR Benefits & ADRCI

1<

ADR7=
$ adrci adrci> s,o4 "ase; A!$ "ase is 68u/38a++8oracle6

adrci> s,o4 ,omes; A!$ (omes& diag8tnslsnr8m5,ost8listener diag8asm89asm89ASM diag8rd"ms8m5d"8m5d"

adrci> set ,ome diag8rd"ms8m5d"8m5d";

1: 2011 Pythian

*sing ADR7= to Giew Alert Log

Giew < /ail alert log 2even on Win"owsA3

adrci> s,o4 alert Rea" @rom +ML @iles$

tail

f;

Results o8ene" in teMt e"itor unless )term s8eci@ie"$

Giew alert log contents matching 8atterns$

adrci> s,o4 alert

+ 6message_te:t li;e <=>$A =<6

term;

=nclu"es message timestam8s, better than gre8$ 7ase)sensitiveA 7an 8attern)match on an, 8re"icate @iel" such as host;i", user;i", originating;timestam8, etc$
2011 Pythian

Giewing Alert Log b, /imestam8

adrci> s,o4 alert + 6originating_timestam+ >7 s5stimestam+ 38?.6 term; adrci> s,o4 alert + 6message_te:t li;e <=>$A @//=< and originating_timestam+ >7 s5stimestam+ A/6 term;

Obvious alert log monitoring bene@itsA ReNuires 8resence o@ +ML log @iles$ 7an S(OOL out8ut to new log @iles$
2011 Pythian

Giewing DDL in Alert Log

:nable DDL Logging:

alter s5stem set ena"le_ddl_logging7true sco+e7"ot,; alter session set ena"le_ddl_logging7true;

Giew out8ut:
+ 6message_te:t li;e <=!$>2=<6 term;

adrci> s,o4 alert

A!$ (ome 7 c&'oracle'diag'rd"ms'orcl'orcl& BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB ?/33 /0 3@ 3C&AA&.D10/C/// !$>2 #A%LE !#S_>%EE*#S?

/C&//

Remin"er: Logs an" search are case)sensitiveA

2011 Pythian

1;

File Maintenance

Alert log +ML @iles rotate" a@ter 1!M$ Ol" Alert log +ML @iles an" trace<inci"ent @iles 8urge" "e8en"ing on short or long 8urge 8olicies$ /race 2human)rea"able3 alert an" listener log @iles not rotate" or archive" b, Oracle$

1?

2011 Pythian

Short (urge (olic,

adrci> s,o4 control; S'OR/(;(OL=7O Galue in 'O*RS, "e@ault is # ! 2%! "a,s3 MaM is %6#P1%P& 2over &!!! ,earsA3 Setting to ! 2Cero3 means all Eshort li@eF @iles can be 8urge"$ /race @iles, core "um8s, 8ac0aging in@o set control FS(>$#2_2>L)*G 7 AA@H;
2011 Pythian

1>

Long (urge (olic,

LOB4(;(OL=7O De@ault is -#5! 2%56 "a,s3 Same maM ? Cero rules as S'OR/(;(OL=7O =nci"ent in@o, inci"ent "um8s, alert logs set control FL>NI2_2>L)*G 7 3../H;

20

2011 Pythian

Manuall, (urging Files

(urge alert logs ol"er than 1 "a,

)age 8arameters ta0es minutes

age 3../ t5+e alert;

adrci> +urge

(urge all @iles associate" @or an inci"ent

i 3?A.C;

adrci> +urge

For more

adrci> ,el+ +urge;

2011 Pythian

ADR (urging BugA

Bug 95 5!5: 114 L=S/:B:R LO4S LO4;+OQ$+ML AR: BO/ A*/OMA/=7ALLO (*R4:D

(otential to @ill u8 "iagnostic;"est "is0$ Oracle 11$1$!$# an" 11$ $!$1 /o manuall, 8urge @iles ol"er than 1 wee0:

adrci> +urge

age 3//0/

t5+e alert;

2011 Pythian

Inci"ent Pac#aging $er%ice

2<

=nci"ent (ac0aging Service 2=(S3

7reates logical E8ac0agesF @or Oracle Su88ort base" on criteria such as inci"ent number or time@rame$ :ither via :M or a"rci

adrci> s,o4 incident; adrci> s,o4 +ro"lem; adrci> s,o4 tracefile J i 3?AK J JrKtK;

2:

2011 Pythian

=nci"ent Floo" 7ontrol

Floo")controlle" inci"ents will generate an alert log entr, but not inci"ent "um8s$ /hreshol"s:

6 inci"ents @or the same 8roblem 0e, in one hour$ Will resume in the neMt hour$ 6 inci"ents @or the same 8roblem 0e, in one "a,$ Will resume in the neMt "a,$ 6!<hour or 6!<"a, K Bo more inci"ent recor"ing at all until hour<"a, eM8iresA
2011 Pythian

=(S > 7reating (ac0ages

adrci>)2S *$EA#E 2A*LAIE )N*)!EN# inc_num"er; adrci>)2S *$EA#E 2A*LAIE 2$>%LEM +ro"lem_)!; adrci>)2S *$EA#E 2A*LAIE 2$>%LEMLEG 6+ro"lem_;e56; adrci>)2S *$EA#E 2A*LAIE SE*>N!S sec; adrci>)2S *$EA#E 2A*LAIE #)ME <start_time< #> <end_time<; adrci>)2S *$EA#E 2A*LAIE;

*se )2S A!! )N*)!EN# or )2S A!! -)LE to a"" "ata to the 8ac0age be@ore generating it$

26

2011 Pythian

=(S > 4enerating (ac0age Files

7reates a 8h,sical @ile @rom the logical 8ac0age$

adrci>)2S IENE$A#E 2A*LAIE +ac;age_num"er )N +at, adrci>)2S IENE$A#E 2A*LAIE 3 )N 8,ome8seiler8diagnostics adrci>)2S S(>M -)LES 2A*LAIE 3

7reate an" generate 8ac0age with one comman": =(S (A7R

adrci>)2S 2A*L )N*)!EN# incident_id )N +at,; adrci>)2S 2A*L )N*)!EN# 3?A.C )N 8,ome8seiler8diagnostics;
2011 Pythian

ADRCI $cripts

2?

ADR7= Scri8ts

7an be calle" interactivel, a la sNl8lus:

adrci> N8,ome8seiler8scri+ts8test1adrci

7an be calle" on comman")line:

$ adrci scri+t78,ome8seiler8scri+ts8test1adrci

:Mam8le scri8t:

O A!$*) scri+t to find alert log errors S2>>L 8,ome8seiler8logs8alert_log_errors1log E*(> 6ALE$# L>I E$$>$S&6; SE# (>ME2A#( diag8rd"ms8orcl8orcl; S(>M ALE$# #E$M 2 6MESSAIE_#EP# L)LE <=>$A =<6 S2>>L >--

2011 Pythian

ADRCI Reports

<0

ADR7= 'ealth Re8orts

Onl, a88lies to 'ealth Monitor @or now

adrci> s,o4 ,m_run; O Loo; for $UN_NAME, e1g1 (M_$UN_3 adrci> create re+ort ,m_run ,m_run_3; adrci> s,o4 re+ort ,m_run ,m_run_3; O >ut+uts PML

'uman)rea"able re8ort:

SQL> set long DDDDD longc,un;siQe 3/// +agesiQe DDD linesiQe C3? SQL> select d"ms_,m1get_run_re+ortF<(M_$UN_3<H from dual;

2011 Pythian

&'ample Inci"ent $cenario

<2

Ste8 1: 7ause an =nci"entA

SQL> alter session set events <D.? incidentFta"le_missingH<; Session altered1

SQL> dro+ ta"le doesnote:ist; dro+ ta"le doesnote:ist B E$$>$ at line 3& >$A //D.?& ta"le or vie4 does not e:ist

SQL> alter session set events <D.? trace name conte:t off<; Session altered1
2011 Pythian

Alert Log o@ =nci"ent

-ri Aug 3? 3C&..&C. ?/33 Errors in file 8,ome8oracle8a++8oracle8diag8rd"ms8orcl8orcl8trace8orcl_ora_30.CR 1trc Fincident73??/3H& >$A //R//& soft internal error, arguments& JEVEN#_*$EA#E!_)N*)!EN#K, JD.?K, J#A%LE_M)SS)NIK, JK, JK, JK, JK, JK, JK, JK, JK, JK >$A //D.?& ta"le or vie4 does not e:ist )ncident details in& 8,ome8oracle8a++8oracle8diag8rd"ms8orcl8orcl8incident8incdir_3??/ 38orcl_ora_30.CR_i3??/31trc

2011 Pythian

Alert Log via ADR7=

adrci> s,o4 alert + 6message_te:t li;e <=>$A //D.?=<6 term adrci> s,o4 alert + 6message_te:t li;e <=>$A =< and originating_timestam+ >7 s5stimestam+ 38?.6 term A!$ (ome 7 c&'oracle'diag'rd"ms'orcl'orcl& BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB ?/33 /0 3. 3.&AC&?R10.A/// /C&//

Errors in file c&'oracle'diag'rd"ms'orcl'orcl'trace'orcl_ora_.@001trc Fincident73.@AAH& >$A //R//& soft internal error, arguments& JEVEN#_*$EA#E!_)N*)!EN#K, JD.?K, J#A%LE_M)SS)NIK, JK, JK, JK, JK, JK, JK, JK, JK, JK >$A //D.?& ta"le or vie4 does not e:ist
2011 Pythian

Ste8 : Giew the =nci"ent

adrci> set ,ome diag8rd"ms8orcl8orcl adrci> s,o4 incident A!$ (ome 7 8,ome8oracle8a++8oracle8diag8rd"ms8orcl8orcl& BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB BBBBBBBB )N*)!EN#_)! 2$>%LEM_LEG *$EA#E_#)ME

3??/3 J#A%LE_M)SS)NIK 3 ro4s fetc,

>$A R// JEVEN#_*$EA#E!_)N*)!EN#K JD.?K ?/33 /0 3? 3C&..&C.1.R?/// /C&//

2011 Pythian

Ste8 %: 7reate a (ac0age

adrci> i+s create +ac;age incident 3??/3; *reated +ac;age 3 "ased on incident id 3??/3, correlation level t5+ical adrci> i+s generate +ac;age 3 in 8,ome8oracle Ienerated +ac;age 3 in file 8,ome8oracle8>$AR//EVE_?/33/03?3C.DC3_*>M_31Qi+, mode com+lete

OR:
adrci> i+s +ac; incident 3??/3 in 8,ome8oracle Ienerated +ac;age ? in file 8,ome8oracle8>$AR//EVE_?/33/03?3@/D/3_*>M_31Qi+, mode com+lete

2011 Pythian

Giew (ac0age 7ontents

Obvious:
l 8,ome8oracle8>$AR//EVE_?/33/03?3C.DC3_*>M_31Qi+

unQi+

/he ADR7= Wa,:

adrci> i+s s,o4 incidents +ac;age 3 adrci> i+s s,o4 files +ac;age 3
-)LE_)! -)LE_L>*A#)>N -)LE_NAME LAS#_SEQUEN*E EP*LU!E 3 SA!$_(>ME>8incident8incdir_3??/3 orcl_ora_30.CR_i3??/31trc 3 )ncluded

2011 Pythian

Ste8 &: Sen" (ac0age to OSS

(ac0age will contain:

Alert log =nci"ent trace @iles :M8ort<"m8 @iles

'o8e@ull, will re"uce the @ile reNuests @rom Oracle Su88ortA

2011 Pythian

(i%e Demo)

:0

*uestions+

:1

/han0 OouA

:mail: seilerS8,thian$com

(lease write with an, Nuestions or criticismsA htt8:<<www$seiler$us htt8:<<www$8,thian$com

Blogs:

/witter: S"tseiler < S8,thian Follow (,thian on Faceboo0 ? Lin0e"=nA

2011 Pythian