Professional Documents
Culture Documents
Go Generate: A Proposal
Go Generate: A Proposal
Introduction
ThegobuildcommandautomatestheconstructionofGoprogramsbutsometimes
preliminaryprocessingisrequired,processingthatgobuilddoesnotsupport.Motivating
examplesinclude:
yacc:generating.gofilesfromyaccgrammar(.y)files
protobufs:generating.pb.gofilesfromprotocolbufferdefinition(.proto)files
Unicode:generatingtablesfromUnicodeData.txt
HTML:embedding.htmlfilesintoGosourcecode
bindata:translatingbinaryfilessuchasJPEGsintobytearraysinGosource
Thereareotherprocessingstepsonecanimagine:
stringmethods:generatingString() stringmethodsfortypesusedasenumerated
constants
macros:generatingcustomizedimplementationsgivengeneralizedpackages,suchas
sort.Intsfromints
Thisproposaloffersadesignforsmoothautomationofsuchprocessing.
Non-goal
ItisnotagoalofthisproposaltobuildageneralizedbuildsystemliketheUnixmake(1)utility.
Wedeliberatelyavoiddoinganydependencyanalysis.Thetooldoeswhatisaskedofit,
nothingmore.
Itishoped,however,thatitmayreplacemanyexistingusesofmake(1)intheGorepoat
least.
Design
Therearetwobasicelements,anewsubcommandforthegocommand,calledgogenerate,
anddirectivesinsideGosourcefilesthatcontrolgeneration.
Whengogenerateruns,itscansGosourcefileslookingforthosedirectives,andforeach
oneexecutesageneratorthattypicallycreatesanewGosourcefile.Thegogeneratetool
alsosetsthebuildtag"generate"sothatfilesmaybeexaminedbygogeneratebutignored
duringbuild.
Theusageis:
(Plustheusual-x,-n,-vand-tagsoptions.)Ifpackagesarenamed,eachGosourcefilein
eachpackageisscannedforgeneratordirectives,andforeachdirective,thespecified
generatorisruniffilesarenamed,theymustbeGosourcefilesandgenerationhappensonly
fordirectivesinthosefiles.Givennoarguments,generatorprocessingisappliedtotheGo
sourcefilesinthecurrentdirectory.
The-runflagtakesaregularexpression,analogoustothatofthegotestsubcommand,that
restrictsgenerationtothosedirectiveswhosecommand(seebelow)matchestheregular
expression.
GeneratordirectivesmayappearanywhereintheGosourcefileandareprocessed
sequentially(noparallelism)insourceorderaspresentedtothetool.Eachdirectiveisa//
commentbeginningaline,withsyntax
//go:generatecommandarg...
wherecommandisthegenerator(suchasyacc)toberun,correspondingtoanexecutablefile
thatcanberunlocallyitmusteitherbeintheshellpath(gofmt)orfullyqualified
(/usr/you/bin/mytool)andisruninthepackagedirectory.
Theargumentsarespaceseparatedtokens(ordoublequotedstrings)passedtothe
generatorasindividualargumentswhenitisrun.Shelllikevariableexpansionisavailablefor
anyenvironmentvariablessuchas$HOME.Also,thespecialvariable$GOFILEreferstothe
nameofthefilecontainingthedirective.(Wemayneedotherspecialvariablessuchas
$GOPACKAGE.Whenthegeneratorisrun,thesearealsoprovidedintheshellenvironment.)No
otherspecialprocessing,suchasglobbing,isprovided.
Nofurthergeneratorsarerunifanygeneratorreturnsanerrorexitstatus.
Asanexample,saywehaveapackage"my/own/gopher"thatincludesayaccgrammarin
filegopher.y.Insidemain.go(notgopher.y)weplacethedirective
(Moreaboutwhat"yacc"meansinthenextsection.)Wheneverweneedtoupdatethe
generatedfile,wegivetheshellcommand,
% go generate my/own/gopher
or,ifwearealreadyinthesourcedirectory,
% go generate
Ifwewanttomakesurethatonlytheyaccgeneratorisrun,weexecute
Ifwehavefixedabuginyaccandwanttoupdateallyaccgeneratedfilesinourtree,wecan
run
Thetypicalcycleforapackageauthordevelopingsoftwarethatusesgogenerateis
% edit
% go generate
% go test
andoncethingsaresettled,theauthorcommitsthegeneratedfilestothesourcerepository,
sothattheyareavailabletoclientsthatusegoget:
Commands
Theyaccprogramisofcoursenotthestandardversion,butisaccessedfromthecommand
lineby
Tomakeiteasytousetoolslikeyaccthatarenotinstalledin$PATH,havecomplexaccess
methods,orbenefitfromextraflagsorotherwrapping,thereisaspecialdirectivethatdefines
ashorthandforacommand.Itisago:generatedirectivefollowedbythekeyword/flag
"command"andwhichgeneratoritdefinestherestofthelineissubstitutedforthecommand
namewhenthegeneratorisrun.Thustodefine"yacc"asageneratorcommandweaccess
normallybyrunning"gotoolyacc",wefirstwritethedirective
andthenallothergeneratordirectivesusingyaccthatfollowinthatfile(only)canbewritten
asabove:
whichwillbetranslatedto
whenrun.
Discussion
Thisdesignisunusualbutisdrivenbyseveralmotivatingprinciples.
First,go generateisintendedtoberunbytheauthorofapackage,nottheclientofit.The
authorofthepackagegeneratestherequiredGofilesandincludestheminthepackagethe
clientdoesaregulargogetorgobuild.Generationthroughgogenerateisnotpartofthe
build,justatoolforpackageauthors.Thisavoidscomplicatingthedependencyanalysisdone
byGobuild.
Second,gobuildshouldnevercausegenerationtohappenautomaticallybytheclientofthe
package.Generatorsshouldrunonlywhenexplicitlyrequested.
Third,theauthorofthepackageshouldhavegreatfreedominwhatgeneratortouse(thatisa
keygoaloftheproposal),buttheclientmightnothavethatprocessoravailable.Asasimple
example,ifitisashellscript,itwillnotrunonWindows.Itisimportantthatautomated
generationnotbreakclientsbutbeinvisibletothem,whichisanotherreasonitshouldberun
onlybytheauthorofthepackage.
Finally,itmustfitwellwiththeexistinggocommand,whichmeansitappliesonlytoGo
sourcefilesandpackages.ThisiswhythedirectivesareinGofilesbutnot,forexample,in
the.yfileholdingayaccgrammar.
Onecanimaginescenarioswheretheauthorwishestheclienttorunthegenerator,butinsuchcasesthe
authormustguaranteethattheclienthasthegeneratoravailable.Regardless,gogetwillnotautomatethe
runningoftheprocessor,sofurtherinstallationinstructionswillneedtobeprovidedbytheauthor.
Examples
Herearesomehypotheticalworkedexamples.Therearecountlessmorepossibilities.
String methods
WewishtogenerateaStringmethodforanamedconstanttype.Wewriteatool,say
strmeth,thatreadsadefinitionforasingleconstanttypeandvaluesandprintsacomplete
Gosourcefilecontainingamethoddefinitionforthattype.
InourGosourcefile,main.go,wedecorateeachconstantdeclarationlikethis(withsome
blanklinesinterposedsothegeneratordirectivedoesnotappearinthedoccomment):
)
ThestrmethgeneratorparsestheGosourcetofindthedefinitionoftheDaytypeandits
constants,andwritesoutaString()stringmethodforthattype.Fortheuser,generation
ofthestringmethodistrivial:justrungo generate.
Yacc
Asoutlinedabove,wedefineacustomcommand
andthenanywhereinmain.go(say)wewrite
Protocol buffers
Theprocessisthesameaswithyacc.Insidemain.go,wewrite,foreachprotocolbufferfile
wehave,alinelike
Becauseofthewayprotocworks,wecouldgeneratemultipleprotodefinitionsintoasingle
.pb.gofilelikethis:
Sincenoglobbingisprovided,onecannotsay*.proto,butthisisintentional,forsimplicity
andclarityofdependency.
Caveat:Theprotocprogrammustberunattherootofthesourcetreewewouldneedto
provideacdoptiontoitorwrapitsomehow.
Binary data
AtoolthatconvertsbinaryfilesintobytearraysthatcanbecompiledintoGobinarieswould
worksimilarly.Again,intheGosourcewewritesomethinglike
Sort
Onecouldimagineavariantsortimplementationthatallowsonetospecifyconcretetypes
thathavecustomsorters,justbyautomaticrewritingofmacrolikesortdefinition.Todothis,
wewriteasort.gofilethatcontainsacompleteimplementationofsortonanexplicitbut
undefinedtypespelled,say,TYPE.Inthatfileweprovideabuildtagsoitisnevercompiled
(TYPEisnotdefined,soitwon'tcompile)butisprocessedbygogenerate:
// +build generate
Thenwewriteangeneratordirectiveforeachtypeforwhichwewantacustomsort:
orperhaps
Therenameprocessorwouldbeasimplewrappingofgofmt-r,perhapswrittenasashell
script.
Therearemanymorepossibilities,anditisagoalofthisproposaltoencourage
experimentationwithprebuildtimecodegeneration.