go oracle: user manual

AlanDonovan adonovan@google.com August25,2013

ThegooracleisaprototypesourceanalysistoolthatanswersquestionsaboutGoprograms. Thisdocumentexplainshowtouseit. Thereisalsoadesigndocument. Howithelps Caveats Buildingit Commandsyntax Analysisscope Warnings Editorbindings Emacs Queries Notationforexamples callees callers callgraph callstack describe freevars implements peers

Theoraclemaybeinvokeddirectlyfromthecommandline,orindirectlyviaaneditorthat providesthetoolwiththecurrentcursorposition/selectionplusthekindofqueryyouwishto perform.

How it helps
Theoracleisdesignedtofullyautomatetheansweringmanyofthequestionsaboutelementsof yourprogramthatcomeupallthetimeduringatypicaldayofprogramming.Questionssuchas: Whatisthetypeofthisexpression?Whatareitsmethods? Whatsthevalueofthisconstantexpression? Whereisthedefinitionofthisidentifier? Whataretheexportedmembersofthisimportedpackage?

Whatarethefreevariablesoftheselectedblockofcode? Whatinterfacesdoesthistypesatisfy? Whichconcretetypesimplementthisinterface? And: Whatarethepossibleconcretetypesofthisinterfacevalue? Whatarethepossiblecalleesofthisdynamiccall? Whatarethepossiblecallersofthisfunction? Whatobjectsmightthispointerpointto? Wherewillavaluesentonthischannelbereceived? Whichstatementscouldupdatethisfield/local/global/map/array/etc? Whichfunctionsmightbecalledindirectlyfromthisone? Inmanycases,usingtheoracleisassimpleasselectingaregionofsourcecode,pressinga button,andreceivingapreciseanswertothequeryalmostimmediately. Thefirstsetofquestionsabovecanbeansweredusingonlylocal,modularreasoningbylooking atthesyntaxtree,thetypes,orthemembersofasinglepackage,butqueriesinthesecondset depend,ingeneral,uponglobalpropertiesofyourapplicationrequiringmoreanalytical efforthumanorrobottodeduce.

Caveats
Theoracleisaprototype.Itsuserinterfacehasmanyroughedgesandwillalmostcertainly needmajorchanges.Theanalysislibrariesmaycontainbugs.Configurationistrickierandless flexiblethanitoughttobe.Analysisisanorderofmagnitudeslowerthanourgoal.Nonetheless, theoracleofferssomeanalyticalservicesthatadvancethestateoftheartincode comprehensiontoolsforanylanguageandadventuroususersmayfinditusefultoday. Pleasereportbugsdirectlytotheauthorfornow.

Building it
Runthefollowingcommandtobuildtheoracle: % g o g e t c o d e . g o o g l e . c o m / p / g o . t o o l s / c m d / o r a c l e Thiswillcauseanexecutablenamedoracletoappearinyour$GOPATH/bindirectory.

Command syntax
Runningtheoraclewithnoargumentsprintsasummaryofthecommandsyntax. Heresanexamplecommandinvokingtheoracle: %oraclemode=describepos=src/pkg/net/http/triv.go:#1042,#1050format=json\ src/pkg/net/http/triv.go

Therearefourinputsofinterest: Themodeofthequery, m o d e = d e s c r i b e inthisexample. Eachsupportedqueryisdescribedinitsownsectionbelow. Thepositionofthecursororselectedsyntax,asaflagoftheform p o s = f i l e : # s t a r t , # e n d where[start,end)formsahalfopenintervalofbyte indiceswithinfile,startingatzero. Thedesiredoutputformat.Supportedformatsinclude: p l a i n ,ahumanreadableformatresemblingtypicalcompilerdiagnosticoutput j s o n ,astructureddataformatspecifiedatgo.tools/oracle/json/json.go Thescopeoftheanalysis.Inthesimplestcase,thisisjustthemainpackageofyour program.Inthisexample,itsapackagecontainingonlythesinglefile s r c / p k g / n e t / h t t p / t r i v . g o .Thisconceptisexpandedinthenextsection.

Analysis scope
Forglobalqueries(explainedabove),theoracleneedstoknowwhichpackagedefinesyour applicationsmainfunction,anditneedssourcecodeforallpackagesthataretransitively importedfromititcannotanalyseisolatedlibraries.Wecallthisthescopeoftheanalysis. Thescopemayconsistofseveralprograms,suchasaclientandaserver,oralltheprograms youroutinelyworkon,orasetoflibrariesandtheirtests.Biggerscopesarebetterbecausethey causetheanalysistovisitmorecode.Ifalibraryfunctionisnotreachableinagivenscope,the analysiscantansweranyquestionsaboutitasifitsnotthere.Thisisanalogoustothewaya linkerworks:libraryfunctionsthatcannotbecalledinagivenexecutablearediscarded. Examplescopes: 1. f m t theimportpathofalibrarydefiningoneormoreTest*functions 2. c o d e . g o o g l e . c o m / p / g o . t o o l s / c m d / o r a c l e theimportpathofapackagewithamainentrypoint. 3. s r c / p k g / n e t / h t t p / t r i v . g o anadhocmainpackageconsistingofasinglefile. (Multiplefilesinthesamepackageshouldbeseparatedwithcommas.) Allthreeofthesemaybespecifiedinthesamecommand.However,duetoacurrentlimitation ofthetypechecker,onlythefirstoftheimportpathstyleargumentswillcontributeanyteststo thepointeranalysisscope.

Warnings
Ifthestaticanalysismustmakeanassumptionthatitcannotprove(e.g.aboutthebehaviourof nativecodethatitcannotseeorunsafe.Pointerconversionsitcannotunderstand)itwillprinta

warningafteritsresults.Currentlythestandardlibrariescausetheoracletoprintmany warnings,sotheoutputcanberathernoisy.Thesewilldiminishovertimeasgapsinthe analysisarefilled.

Editor integration
Theoraclemaybeinvokedfromanyeditorcapableofrunninganexternaltool(suchasa compiler)anddisplayingitsoutput.Sincemanyeditorstreatfilenamesappearingincompiler diagnosticsashyperlinkstothelocationoftheerror,theoracleprintsitsanswerusingasimilar syntaxwheninvokedwith f o r m a t = p l a i n . CurrentlytheonlyeditorforwhichbindingsexistisEmacs,thoughwehopetoaddsupportfor othersbasedondemand.Pleasecontacttheauthorifyoudliketohelpconnecttheoracleto anothereditorsuchasVim,AcmeorEclipse.

Emacs
Emacsexpectstofindtheoracleexecutablein$GOROOT/bin,notwhereg o g e t placesit(i.e. P/binwherePisthefirstdirectorynamedby$GOPATH),sobuilditusingtheg o g e t command aboveandthenmoveit: % m v $ G O P A T H / b i n / o r a c l e $ G O R O O T / b i n / (ThiscommandassumesyourGOPATHconsistsofexactlyonedirectoryadjustaccordinglyif yourshasseveral.) WithinEmacs,loadtheoracle.elfileusingacommandsuchasthis: M x l o a d f i l e $ G O P A T H / s r c / c o d e . g o o g l e . c o m / p / g o . t o o l s / c m d / o r a c l e / o r a c l e . e l Typically,userswilladdthiscommandtotheir~/.emacsstartupconfiguration. Beforeyoucanruntheoracle,youmusttellEmacstheanalysisscope,whichisdoneusingthe command: M x g o o r a c l e s e t s c o p e Thiscommandpromptsyoufortheanalysisscope,describedabove,withwordsseparatedby spaces.Theeffectofg o o r a c l e s e t s c o p e persistsacrossalloracleinvocationsuntilitis calledagainwithadifferentvalue. Toinvoketheoracle,positionthecursoron(orselect)thesyntaxofinterestandcalltheEmacs commandgooraclexxxwherexxxisthemodeofthequery.Forexample: M x g o o r a c l e c a l l e e s Themostcommonlyusedquery,describe,isboundtotheshortcutkey< F 4 > .Totestyour configuration,loadafilewithinyouranalysisscope,selectanexpression,andhit< F 4 > .Aftera momentawindowshouldappearwiththeresults,lookingsomethinglikethis:

GoOracle referencetovarresultstring definedhere

Queries
Thissectiondescribesthesetoforaclequeries.SeetheTableofContentsforthecompletelist.

Notation for examples

Intheexamples,sourcecodeisshowningrey,userselectedsourcecodeishighlightedin yellow,and f o r m a t = p l a i n tooloutputiscoloredblue.Intheactualtooloutput,eachlineis precededbythesourcelocationmostrelevanttoit,buttoavoiddistractingdetailinthe examples,thefilenameshavebeenrenderedasasymbol.Insomecases,locationmarkers suchasL1havebeenaddedtomakethesource/resultscorrespondenceclear. Forbrevity,the f o r m a t = j s o n outputisnotshown,butitcontainsessentiallyallthesame informationasthep l a i n output,brokendownintoatreeofstructureddataforeaseofparsing. Readtheg o . t o o l s / o r a c l e / j s o n documentationformoredetails.

callees
Thecalleesqueryshowsthepossiblecalltargetsoftheselectedfunctioncallsite.Thecursor orselectionmustbewithinafunctioncallexpression. Example:acalleesqueryonthemaindispatcherofn e t / h t t p strivialwebserverrevealsallthe handlersthatareregisteredbytheapplication. func(fHandlerFunc)ServeHTTP(wResponseWriter,r*Request){ f(w,r) } thisdynamicfunctioncalldispatchesto: net/http.NotFound main.FlagServer main.ArgServer expvar.expvarHandler main.DateServer main.HelloServer main.Logger

func@1247.21

(Thelastoneisananonymousfunction.) Example:acalleesqueryonaninterfacemethodcall(itselfinananonymouscallback)reveals thesoletargetofthecall. funcStripPrefix(prefixstring,hHandler)Handler{ ifprefix==""{ returnh } returnHandlerFunc(func(wResponseWriter,r*Request){ ifp:=strings.TrimPrefix(r.URL.Path,prefix)len(p)<len(r.URL.Path){ r.URL.Path=p h.ServeHTTP(w,r) }else{ NotFound(w,r) } }) } thisdynamicmethodcalldispatchesto: (*http.fileHandler).ServeHTTP

callers
Thecallersqueryshowsthepossiblecallersofthefunctioncontainingtheselection. Example:FlagServerisanHTTPhandlerfunctioninthetrivialwebserverinn e t / h t t p .A callersqueryonthatfunctionrevealswherethewebserverdispatchesrequeststoit. funcFlagServer(whttp.ResponseWriter,req*http.Request){ w.Header().Set("ContentType","text/plaincharset=utf8") func(fHandlerFunc)ServeHTTP(wResponseWriter,r*Request){ f(w,r)//L1 ... L1 main.FlagServeriscalledfromthese1sites: dynamicfunctioncallfrom(http.HandlerFunc).ServeHTTP

Example:asecondcallersquery,thistimeonServeHTTP: func(fHandlerFunc)ServeHTTP(wResponseWriter,r*Request){ f(w,r) ... func(mux*ServeMux)ServeHTTP(wResponseWriter,r*Request){ ... h,_:=mux.Handler(r) h.ServeHTTP(w,r)//L2 } L2 (http.HandlerFunc).ServeHTTPiscalledfromthese1sites: dynamicmethodcallfrom(*http.ServeMux).ServeHTTP

callgraph
Thecallgraphqueryshowsthecallgraphoftheentireprogram,whichmayhavemany thousandsofnodesandedges.Theselectionisignored. Example:hereisanexcerptofthecallgraphofaprogramthatusesthef m t package. 278 log.Fatalf 279 fmt.Sprintf 280 fmt.newPrinter 281 (*fmt.cache).get (*sync.Mutex).Lock(52) (*sync.Mutex).Unlock(56) 282 func@161.23 283 (*fmt.fmt).init 284 (*fmt.fmt).clearflags 285 (*fmt.pp).doPrintf Thegraphisrenderedasaspanningtree,usingindentationtoshowparent/childrelationships. Whenanodeappearsforthefirsttime,itisprefacedbyafreshnumber(e.g.278forlog.Fatalf) andwhenitappearsagain,thenumberisshownafter(e.g.52for(*sync.Mutex).Lock). Afunctionmayhavemultiplenodesinthecallgraphiftheanalysisdecidestotreatitcontext sensitively:ineffectitmakestwodistinctcopiesofthefunctionbasedonwhereitiscalledfrom. Thiscanoftenimprovetheprecisionoftheanalysis.Lookatthenumbersifindoubt.

callstack
Thecallstackqueryshowsanarbitrarypathfromtherootofthecallgraphtothefunction containingtheselection.Thismaybeusefultounderstandhowthefunctionisreachedina givenprogram. Example:theresultofacallstackqueryfromtheServeHTTPfunctioninn e t / h t t p . func(ctr*Counter)ServeHTTP(whttp.ResponseWriter,req*http.Request){ ctr.mu.Lock() ... } Foundacallpathfromrootto(*main.Counter).ServeHTTP (*main.Counter).ServeHTTP dynamicmethodcallfrom(*http.ServeMux).ServeHTTP dynamicmethodcallfrom(http.serverHandler).ServeHTTP staticmethodcallfrom(*http.conn).serve staticmethodcallfrom(*http.Server).Serve staticmethodcallfrom(*http.Server).ListenAndServe staticmethodcallfromnet/http.ListenAndServe staticfunctioncallfrommain.main

Theprecisionandusefulnessofcallstackinformationvariesconsiderably,especiallyifthecall pathcontainsagreaterdegreeofdynamiccalls.Thechosencallstackmightbeinfeasible,i.e. neveroccurringduringanyexecution.

describe
Thedescribequeryshowsvariouspropertiesoftheselectedsyntax:itssyntactickind,type, methodset,constantvalue,pointofdefinition,pointstoset,etc,asappropriate.Almostany pieceofsyntaxmaybedescribed,andtheoraclewilltrytoprintalltheusefulinformationitcan. Example:adescribequeryonafieldselectionexpressioninthen e t / h t t p package. func(ctr*Counter)ServeHTTP(whttp.ResponseWriter,req*http.Request){ ... io.Copy(buf,req.Body) } referencetovarBodyio.ReadCloser definedhere interfacemaycontaintheseconcretetypes: *struct{*strings.Readerio.Closer},maypointto: complit *http.body,maypointto: complit complit complit *http.expectContinueReader,maypointto: complit

Theresponsetothisquerycontainsthetypeoftheexpression,thelocationofthedefinitionof thestructfield,thelistofconcretetypesthatit(aninterface)maycontain,andforeachofthose concretetypes,allofwhicharepointers,thesetofobjectstowhichitmaypoint,withsource locationswhereavailable.(c o m p l i t indicatestheobjectallocatedbyacompositeliteral.) Example:frompackagen e t . func(h*dnsHeader)Walk(ffunc(vinterface{},name,tagstring)bool)bool{ returnf(&h.Id,"Id","")&& func(dns*dnsMsg)Pack()(msg[]byte,okbool){ vardhdnsHeader//L1 func(dns*dnsMsg)Unpack(msg[]byte)bool{ vardhdnsHeader//L2

 L1 L2

unary&operationoftype*uint16 valuemaypointtotheselabels: dh.Id dh.Id

Thesyntaxdh.Iddenotesthe.Idfieldoftheobjectcreatedbytheidentifierdh.Infactthereare twodistinctlocalvariables,bothcalleddh,intowhichthisexpressionmaypoint. Example:anexcerptofadescribequeryonapackagename.Essentiallythesameresultsare obtainedwhentheselectionistheimportpathorwhenitisthepackageidentifier. import"net/url" varuurl.URL importofpackage"net/url" typeError struct{...} method(*url.Error)Error()string typeEscapeError string method(url.EscapeError)Error()string funcParse func(rawurlstring)(url*url.URL,errerror) funcParseQuery func(querystring)(murl.Values,errerror) funcParseRequestURI func(rawurlstring)(url*url.URL,errerror) funcQueryEscape func(sstring)string funcQueryUnescape func(sstring)(string,error) typeURL struct{...} method(*url.URL)IsAbs()bool method(*url.URL)Parse(refstring)(*url.URL,error) method(*url.URL)Query()url.Values ... Thedescriptionofapackageincludesallitsexportedmembers,theirtypes,methods,and values(forconstants).Ifthecurrentpackageisdescribed(byselectingthepackagedeclaration packagep),thedescriptionincludesthenonexportedmemberstoo.

freevars
Thefreevarsqueryenumeratesthefreevariablesoftheselection.Freevariablesisa technicaltermmeaningthesetofvariablesthatarereferencedbutnotdefinedwithinthe selection,orlooselyspeaking,itsinputs. Thisinformationisusefulifyoureconsideringwhethertorefactortheselectionintoafunctionof itsown,asthefreevariableswouldbethenecessaryparametersofthatfunction.Itsalso usefulwhenyouwanttounderstandwhattheinputsaretoacomplexblockofcodeevenifyou dontplantochangeit. Tomaketheresultsmoreuseful,theoutputofthequerydiffersslightlyfromthetextbook definitionoffreevariables: theoutputdoesnotreportanynamesdefinedatpackagelevel,sincetheywouldnot needtobepassedasparameterstoafunction foreachfreestructvariable,theoutputreportseachdistinctaccesspath(e.g.s.x.y)asa freevariable. theoutputalsoreportsreferencestofreeconstantsandtypes. Example:thefreevariablesofthebodyofaloopinthes t r i n g s package. //Secondpass:findrepeatsofpattern'ssuffixstartingfromthefront. fori:=0i<lasti++{ lenSuffix:=longestCommonSuffix(pattern,pattern[1:i+1]) ifpattern[ilenSuffix]!=pattern[lastlenSuffix]{ //(lasti)istheshift,andlenSuffixislen(suffix). f.goodSuffixSkip[lastlenSuffix]=lenSuffix+lasti } } Freeidentifiers: variint varlastint varpatternstring varf.goodSuffixSkip[]int

implements
Theimplementsqueryshowstheimplementsrelationforallinterfacesandconcretetypes definedinthispackage.Theselectionisignored. Example:anexcerptoftheimplementsrelationofthei o package. Interfaceio.Writer: *io.PipeWriter *io.multiWriter Interfaceio.Seeker: *io.SectionReader Interfaceio.Closer: *io.PipeReader *io.PipeWriter

peers
Thepeersqueryshowsthesetofpossiblesends/receivesonthechanneloperandofthe selectedsendorreceiveoperationtheselectionmustbea< token. Example:apeersqueryonareceiveoperationinthen e t / h t t p package. vartextprotoReaderCache=make(chan*textproto.Reader,4) funcnewTextprotoReader(br*bufio.Reader)*textproto.Reader{ select{ caser:=<textprotoReaderCache: //L3 r.R=br returnr default: returntextproto.NewReader(br) } } funcputTextprotoReader(r*textproto.Reader){ r.R=nil select{ casetextprotoReaderCache<r: //L2 default: } } L1 L2 L3 Thischanneloftypechan*textproto.Readermaybe: allocatedhere sentto,here receivedfrom,here //L1

referrers
Thereferrersqueryshowsthesetofidentifiersthatrefertothesameobjectasdoesthe selectedidentifier,withinanypackageintheanalysisscope. Example:findallreferencestoafunctionparameterinthef m t package. func(p*pp)fmtUint64(vuint64,verbrune,goSyntaxbool){ switchverb{ case'b': p.fmt.integer(int64(v),2,unsigned,ldigits)//L1 case'c': p.fmtC(int64(v))//L2 case'd': p.fmt.integer(int64(v),10,unsigned,ldigits)//L3 L1 L2 L3 definedhereasvarvuint64 referencedhere referencedhere referencedhere ...

Troubleshooting
TheoraclesaysthatfunctionFisreachable,butIknowthatitsnot. TheoraclesaysthatpointerPcanpointtolabelL,butIknowthatitcannot.

Thisclassoferrorsarisefromfalsepositivesorimprecisioninthepointeranalysis.Asound pointeranalysismaymakeconservativeapproximationswhenitisntcapableoffullycapturing thebehaviourofyourprogram.Thesekindsoffalsereportsaremostlynotconsideredbugs, althoughofcourseiftheyaretoonumerous,theusefulnessofthetoolmaybediminished. TheoraclesaysthatfunctionFisdeadcode,butIknowthatitsnot. TheoraclesaysthatpointerPmaynotpointtolabelL,butIknowthatitcan.

Thisclassoferrorsarisefromfalsenegativesorunsoundnessofthepointeranalysis,and theygenerallyindicateabug.Reflectionisnotcurrentlysupported,leadingtounsoundresults (missingedgesinthecallgraphandunderestimatesofpointstosets)thiswillbefixedindue course.unsafe.Pointerconversionsarealsonotsupported,andmayneverbe. Dontforgetthatthepointeranalysisonlylooksatcodereachableintheanalysisscopethatyou specified,e.g.theentireprogramwhosemainpackagewasnamedonthecommandline. EvenalargeGoprogrammightuseonlysmallpartsofsomeofthelibrariesitdependsupon,so pointeranalysisqueriesabouttheunusedpartswillreturnnullresults.Thisiscorrect,andthe expectedbehaviour.Specifyingalargerscope(moremainpackagesandtests)canimprove theanalyticalcoverageofyourlibraries.