Malhar Anaokar

Senior IT Architect
Sun Microsystems Pte Ltd
Practical Applications of DTrace
Agenda
What is DTrace?
Uses of DTrace
Additional information
Why Dynamic Tracing?
Well-defined techniques for deu!!in! f at al,
non- reproducible failure:
> Obtain core file or crash dump
>
Debug problem post m ort em using mdb(1),
dbx(1)
Techniques for debugging t ransient
failures are much more ad hoc
> Tpical techniques push traditional tools (e!g!
truss(1), mdb(1)) beond their design centers
> "an transient problems cannot be debugged
at all using extant techniques
Transient failure
Any unacce"tale eha#ior that does not result in fatal
failure of the system
May e a clear failure$
> %&ur a""lication occasionally doesn't recei#e its timer si!nal()
May e a failure ased on customer's definition of
%unacce"tale)$
> %We *ere e+"ectin! to accommodate ,-- users "er .PU / and
*e're ale to !et no more than 0- users "er .PU()
> %1#ery mornin! from aout 2am to aout ,-$3-am4 the system is a
do!()
#erformance inhibitors or incorrect expectations$
Debugging transient failure
5istorically4 *e ha#e deu!!ed transient failure
usin! "rocess-centric tools$ truss6,74 "stack6,74
"rstat6,74 etc(
These tools *ere not desi!ned to deu!
syst em ic "rolems
8ut the tools desi!ned for systemic "rolems 6i(e(4
md6,77 are desi!ned for "ostmortem analysis(((
Postmortem techniques
&ne technique is to use "ostmortem analysis to
deu! transient "rolems y inducing fatal
failure during period of transient failure
%etter than nothing, but not b much:
> &equires inducing fatal failure, 'hich nearl
al'as results in more do'ntime than the
transient failure
> &equires a (een intuition to be able to suss out
a dnamic problem from a static snapshot of
state
Invasive techniques
If e+istin! tools cannot root-cause transient failure4
more in#asi#e techniques must e used
Ty"ically4 custom instrumentation is de#elo"ed for
the failin! "ro!ram and9or the kernel
The customer re"roduces the "rolem usin! the
instrumented inaries
Dynamic instrumentation
Want to e ale to dynam ically modify the
system to record arbit rary data
Must e ale to do this on a product ion system
Must e com"letely saf e / there should e no *ay
to induce fatal failure
Introducing DTrace
Dynamic tracin! frame*ork introduced *ithin S,-
in Se"temer :--3
A#ailale on stock systems / ty"ical system has
more than 3;4--- "roes
Dynamically inter"reted lan!ua!e allo*s for
aritrary actions and "redicates
.an instrument at oth user-le#el and kernel-le#el
Introducing DTrace, cont.
Po*erful data mana!ement "rimiti#es eliminate
need for most "ost"rocessin!
Un*anted data is "runed as close to the source as
"ossile
Mechanism to trace durin! oot
Mechanism to retrie#e all data from a kernel crash
dum"
Much more(((
ey !oncepts
Proes
Pro#iders
.onsumers
The %D) lan!ua!e
>
<ariales
>
Actions
>
A!!re!ation
>
Predicates
>
=ormattin!
Probes
A probe is a point of instrumentation
) probe is made a*ailable b a provider
+ach probe identifies the m odule and
f unct ion that it instruments
+ach probe has a nam e
These four attributes define a tuple that
uniquel identifies each probe
+ach probe is assigned an integer
identifier
Providers
A "ro#ider re"resents a methodolo!y for
instrumentin! the system
Pro#iders make "roes a#ailale to the DTrace
frame*ork
DTrace informs "ro#iders *hen a "roe is to e
enaled
Pro#iders transfer control to DTrace *hen an
enaled "roe is hit
Providers, cont.
DTrace has quite a fe* "ro#iders4 e(!($
>
The f unct ion boundary t racing (FBT)
pro*ider can dynamically instrument e#ery function
entry and return in the kernel
>
The syscall "ro#ider can dynamically instrument the
system call tale
>
The lockst at "ro#ider can dynamically instrument
the kernel synchroni>ation "rimiti#es
>
The prof ile pro*ider can add a configureable,
rate profile interrupt of to the sstem
>
...
!onsumers
A DTrace consumer is a "rocess that interacts
*ith DTrace
?o limit on concurrent consumers@ DTrace
handles the multi"le+in!
Some "ro!rams are DTrace consumers only as an
im"lementation detail
dtrace6,M7 is a DTrace consumer that acts as a
!eneric front-end to the DTrace facility
"isting probes
Proes can e listed *ith the %-l) o"tion to dtrace
6,M7
.an list "roes
>
in a s"ecific function *ith %-f f unct ion)
>
in a s"ecific module *ith %-m m odule)
>
*ith a s"ecific name *ith %-n nam e-
>
from a specific pro*ider 'ith .,# provider-
/or each probe, pro*ider, module, function
and name are displaed
#ere are the Probes,
Providers and !onsumers
Actions
Act ions are taken *hen a "roe fires
Actions are com"letely "ro!rammale
Most actions record some s"ecified state in the
system
Some actions change the state of the system
system in a *ell-defined *ay
>
These are called dest ruct ive act ions
> Disabled b default
Many actions take as "arameters e+"ressions in
the D language
Actions$ %trace&

trace() records the result of a D e+"ression to

the trace uffer
=or e+am"le$
> trace(pid) traces the current "rocess ID
> trace(execname) traces the name of the current
e+ecutale
> trace(curthread->t_pri) traces the
t_pri field of the current thread
> trace(probefunc) traces the function name of
the "roe
Actions, cont.
Actions are indicated y follo*in! a "roe
s"ecification *ith %{ act ion }-
/or example:
dtrace -n 'readch{trace(pid)}'
dtrace -m 'ufs{trace(execname)}'
dtrace -n 'syscall:::entry {trace
(probefunc)}'
"ultiple actions can be specified0 the
must be separated b semicolons:
dtrace -n 'xcalls{trace(pid); trace
(execname)}'
Actions$ Destructive actions
DTrace allo*s for destructi#e actions
MUST 81 .&?SID1A1D .AA1=ULLB
1+tremely dan!erous on li#e 9 "roduction system
Must s"ecify %-*) o"tion to DTrace
'(ample of DTrace Action
Aggregations
When tryin! to understand suo"timal
"erformance4 one often looks for pat t erns that
"oint to ottlenecks
When lookin! for "atterns4 one often doesn't *ant
to study each datum / one *ishes to
aggregat e the data and loo( for larger
trends
Traditionally4 one has had to use con#entional
tools 6e(!( a*k6,74 "erl6,77
Aggregations, cont.
An aggregat ion is the result of an
aggregating function (eed b an arbitrar
tuple
/or example, to count all sstem calls on
a sstem b sstem call name:
dtrace -n 'syscall:::entry \
{ syscalls!probefunc" # count(); }'
8y default4 a!!re!ation results are "rinted *hen
dtrace6,M7 e+its
Aggregations, cont.
A!!re!ations need not e named
A!!re!ations can e keyed y more than one
e+"ression
=or e+am"le4 to count all ioctl system calls y oth
e+ecutale name and file descri"tor$
dtrace -n 'syscall::ioctl:entry \
{ !execname$ ar%&" # count(); }'
Aggregation at )or*
Predicates
redicat es allo* actions to only e taken *hen
certain conditions are met
A "redicate is a D e+"ression
Actions *ill only e taken if the "redicate
e+"ression e#aluates to true
A "redicate takes the form %'e!pression'-
and is placed bet'een the probe
description and the action
Predicates, cont.
=or e+am"le4 tracin! the "id of e#ery "rocess
named %date) that "erforms an o"en6:7$
()'usr'sbin'dtrace -s
syscall::open:entry
'execname ## *date+'
{
trace(pid);
}
"oo*ing through Predicates
The D language
D is a .-like lan!ua!e s"ecific to DTrace4 *ith
some constructs similar to a*k6,7
.om"lete access to kernel . ty"es
.om"lete access to statics and !loals
.om"lete su""ort for A?SI-. o"erators
Su""ort for strin!s as first-class citi>en
We'll introduce D features as *e need them(((
+uilt,in D variables
=or no*4 our D e+"ressions *ill consist only of
uilt-in #ariales
1+am"le of uilt-in #ariales$
> pid is the current "rocess ID
> execname is the current e+ecutale name
> timestamp is the time since boot, in
nanoseconds
> curthread is a pointer to the ,thread_t
structure that represents the current thread
> probemod, probefunc and probename are
the current probe1s module, function and
name
D -cripts
.om"licated DTrace enalin!s ecome difficult to
mana!e on the command line
dtrace6,M7 su""orts script s, specified 'ith the
.,s- option
Alternati#ely4 e+ecutale DTrace inter"reter files
may e created
Inter"reter files al*ays e!in *ith$
()'usr'sbin'dtrace -s
D -cripts, cont.
=or e+am"le4 a scri"t to trace the e+ecutale name
u"on entry of each system call$
()'usr'sbin'dtrace -s
syscall:::entry
{
trace(execname);
}
.utput formatting

The printf() function comines the trace

action *ith the aility to "recisely control out"ut

printf takes a "rintf63.7-like format strin! as

an ar!ument4 follo*ed y corres"ondin!
ar!uments to "rint
e(!($
printf(*-d .as here+$ pid);
printf(*/ am -s+$ execname);
.utput formatting, cont.
?ormally4 dtrace6,M7 "ro#ides details on the firin!
"roe4 "lus any e+"licitly traced data
Use the quiet o"tion 6%-q)7 to dtrace6,M7 to su"ress
the "roe details
The quiet o"tion may also e set in a D scri"t y
emeddin!$
(pra%ma 0 option 1uiet
'(ample of formatting
'(ploring DTrace, cont.
DTrace Cuick Aeference Duide
htt"$99de#elo"ers(sun(com9solaris9articles9dtraceEq
uickref9dtraceEquickref(html
DTrace User Duide
htt"$99docs(sun(com9a""9docs9doc9F,2-;GFF
8i!Admin has a "a!e and discussion forum
dedicated to DTrace$
htt"$99***(sun(com9i!admin9content9dtrace
The DTrace /evolution
DTrace ti!htens the dia!nosis loo"$ hypot hesis
inst rum ent at ion dat a gat hering
analysis hypot hesis
Ti!htened loo" effects a re#olution in the *ay *e
dia!nose transient failure
=ocus can shift from inst rum ent at ion stage
to hypot hesis stage:
>
"uch less labor intensi*e, less error prone
>
"uch m ore brain intensi*e
>
"uch more effecti*e2 ()nd a lot more fun)
0uestions?