7/20/2015

OpenSourceJavascriptDevelopment

OpenSourceJavascriptDevelopment

25thJanuary2014

IvebeenprogrammingatGoogleforawhilenow.Therearelotsofgoodpoints,butonethingthatsbeen
worryingmeforafewyearsisthatIvebasicallyfallenoutofdevelopingonopensourcetoolchains.So
when I got a chance to write a small standalone open source Javascript utility
[https://github.com/google/instanthangouts]thismonth,Ijumpedatit.
You

can

read

all

about

the

utility,

Instant

Hangouts, in my previous post

[http://johnmcox.blogspot.com/2014/01/introducinginstanthangouts.html] .Briefly,ifyouwantyourusersto
beabletotalktoeachotherfacetofaceaboutawebpage,youcaninserttwolinesofHTMLandgeta
Google+ Hangout specific to that page. The project is very simple its just a shim around the existing
Google+HangoutButtonAPIbutitgavemeachancetogetbackuptodateonJavascripttooling.
ThispostisntacompareandcontrastwiththeGoogletoolchains.Instead,itsanaccountofwhatIfound
thatIwroteontheoffchancethatitmightsaveyousometimeifyourenewtoJavascripttoolchains.

Requirements
Ineededmytoolchaintogivemesixthings:
1. Revision control. This frees you to mess up. Messing up is the hearts blood of software

development.
2. Dependencymanagement.Atbothruntimeandduringdevelopment,Iwantedpinnedversionsofall
dependencies.Otherwise,youneverknowwhatcodeyourereallyrunning.Youcantpredicthowit
will behave, so you spend all your time dealing with integration problems rather than shipping your
product.Miserable.
3. Developmentserver.TheunderlyingGoogleAPIsIwasworkingwithcrashunlesstheycanPOSTto
the requesting server, so viewing my code locally in my browser via file:// was out. Also, my
userswillbeusingrealservers,andIwantmydevenvironmenttobeasclosetotheirsaspossibleso
Icanminimizethechancemycodeworksformebutnotthem.
4. Automaticversioning.Thisgivesouruserstheabilitytomanagetheirdependencyonourscript.
5. Codeisolationatruntime,andvisibilityattesttime.Whenyourerunningyourtestsyouwanttobe
able to inspect your code for debugging. But at runtime you cant expose all your symbols because
Javascript doesnt have namespaces and collisions will break your code (and everyone elses) in
entertainingandnovelways.
6. Automatictesting.Smallprojectshavethebadhabitofgrowingintorealproducts,soIliketohave
tests from the start. Even if theyre not perfectly comprehensive, tests give you the confidence you
needtodealwithcodechurnovertime,andtheyforceyoutouseyourownAPIsinwaysthatimprove
designearlyinyourproductslifecyclewhenchangeischeap.
My naive expectation was that these would all be solved problems. And they were, more or less. The
morepartisthatthereareofftheshelfsolutionsforeachoftheseproblems.Thelesspartisthatyou
needtocobblethesesolutionstogetherintoatoolchain,andeachtimeyouaddanewutilitythespaceof
whatcangowrongexpandsgeometrically.
Icallthistheprojectorproblem,aftersomethingIobservedrightafterjoiningGoogle.Wehavemeetings,
as everybody does. And at those meetings, someone invariably wanted to present something on their
laptop.Everymeetingroomhadaprojector.Everyoneknewhowtousetheirlaptop,andeveryoneknew
how to use the projector, but the minute they plugged the laptop into the projector things fell apart in
http://johnmcox.blogspot.in/2014/01/opensourcejavascriptdevelopment.html

1/7

7/20/2015

OpenSourceJavascriptDevelopment

apocalypticfashion.Themeetinghaltedwhileeverybodydebuggedtheconnectionbetweenthelaptopand
theprojector,usuallyburninganhourormoretotaltimeacrosseveryoneintheroom.Maybeyouvegotthe
wrongresolution?Orthecableisloose?Oryoudidnthittherightbuttoninyourpresentationsoftware?
Maybe.
Theproblemisthatthetotalcomplexityofanyintegrationistheproductofthecomplexityoftheindividual
parts. This scales up faster than the expertise of whoever is dealing with the integration, so it was no
surprise that I ended up burning more time on my toolchain than on writing my small utility. Heres how
thingsendedup.

Revisioncontrol
Solvedwithgit[http://gitscm.com/] andGitHub[https://github.com/] .gitisadistributedversioncontrolsystem
thatsbasicallybecomethedefactostandardfordevelopmentintheopensourceworldandinsidemany
startups.GitHubprovidesgitasaservice,andisenoughofastandardthataprogrammersGitHubprofile
iseffectivelytheirworkportfolio.Iwantedittobeeasyforuserstogetourcodeandfordeveloperstoadd
toit.Usingthestandardversioncontrolsystemandserviceproviderminimizesfrictiononbothcounts.
There are lots of ways to install git on your system [http://gitscm.com/book/en/GettingStartedInstallingGit] .
Picktheonethatsrightforyou.

Dependencymanagement
Solved with Node.js [http://nodejs.org/] and npm [https://npmjs.org/] , which come together. Node.js is a
platformformakingfastnetworkapplicationsinJavascript.npmisitspackagemanager.Withnpm,you
write a package.json [https://github.com/google/instanthangouts/blob/master/package.json] file that
describes your project. Its just a JSON object literal. Its dependencies field lets you specify your
runtimedependencies,anditsdevDependenciesfieldletsyouspecifyyourdevelopmentdependencies.
TherearelotsofwaystoinstallNode.jsandnpmonyoursystem[http://nodejs.org/download/] .Picktheone
thatsrightforyou.
Oncethatsdone,youcangrabInstantHangoutswith
$gitclonehttps://github.com/google/instanthangouts.git
andinstalldependencieswith
$npminstall
Asidebaraboutruntimedependencies:
Youvegottwochoices:packageyourdependenciesinyourproject,orloadthematruntime.Theformer
givesyousimplicityandpredictability.Youdistributeparticularversionsofeverythingsoyouknowexactly
whatyourerunning.Andonceyourusersloadallyourcode,theyreset.
http://johnmcox.blogspot.in/2014/01/opensourcejavascriptdevelopment.html

2/7

7/20/2015

OpenSourceJavascriptDevelopment

The latter gives better performance. Browsers cache the scripts they load. If two projects both load a
commonscript(say,jQuery [http://jquery.com/] )itsreallywastefulforboththoseprojectstoforceusersto
sitaroundwhilethecommonscriptisfetchedtwice.Instead,itsbetterforeachofthoseprojectstoloadthe
commonscriptfromastandardlocationsobrowsercachingwillkickinonallrequestsforitpastthefirst.
Nowthatcontentdeliverynetworks [http://en.wikipedia.org/wiki/Content_delivery_network] (CDNs)arecommon,
its good to load your runtime dependencies from them whenever you can. Thats what we do with our
dependencyontheGoogleHangoutButtonAPI.
Apartfromthatlibrary,InstantHangoutshasnoruntimedependencies.Forsimplicityweimplementedthe
few DOM manipulation and functional programming helpers so we can avoid loading in jQuery or
Underscore.js [http://underscorejs.org/] . While the world really doesnt need another Javascript
implementation of filter(), a few hours researching Javascript module loaders like Require.js
[http://requirejs.org/] convincedmethatthesesmallrepetitionswerecheapertowriteandfastertoexecute
thanbringinginhelperlibrariesforatinyhandfulofcallsites.
Atdevtimewedohaveahandfulofdependencies.Theseareenumeratedinpackage.jsonusingnpms
semantic versioning [https://npmjs.org/doc/misc/semver.html] . Developers can install these dependencies
locallyvianpminstallorgloballyvianpminstallgastheyprefer.Eitherway,onceadeveloper
hasinstalledNode.jsandnpmtherestofthedependenciesaremanagedandtheirversionspinned.

Developmentserver
Forsimplescripts,loadingofflocaldiskviaabrowserthatsupportsfile://duringdevelopmentisoften
sufficient.TheGoogleHangoutButtonAPIdoesntworkinthatenvironmentbecauseitneedstoPOST,so
weneedalocaldevelopmentserver.
WerealreadyusingNode.jsfordependencymanagement,sowealsouseitforourdevelopmentserverto
avoidintroducingadditionalcomplexitytothetoolchain.Thishastheaddedbenefitofgivingusbothclient
and server implementations in the same language, which likewise decreases complexity of the project
overall.
Ourserverlivesinscripts/server.js [https://github.com/google/instanthangouts/blob/master/scripts/server.js]
,andyoucanlaunchitwith
$nodescripts/server.js
It serves on port 8080. For Instant Hangouts youll want to use your hostname rather than localhost
sincetheURLispartofthekeyusedtogeneraterooms.Ifyouuseyourhostname,youcanloadthedev
serverfrommultiplemachinesanddoamanualendtoendtestbyjoiningtheresultingHangoutsfromtwo
differentGoogleaccounts.
Theserverimplementationissimple:
http://johnmcox.blogspot.in/2014/01/opensourcejavascriptdevelopment.html

3/7

7/20/2015

OpenSourceJavascriptDevelopment

//Loaddependencies:
//Express.jsisawebframeworkfornode.
//pathisforlocalfilesystemoperations.
varexpress=require('express')
varpath=require('path')
varapp=express()
//LogrequeststoSTDOUT.
app.use(express.logger())
//Bindrequestsfor*.jstofilesintheprojectroot.
//ThisservesourcompiledJS,notthefilesinsrc/.
app.get(/.js$/,function(request,response){
response.sendfile(path.resolve(request.url.substring(1)))
})
//Bindallotherrequeststoindex.html.
//index.htmlisalsousedasafixtureinourtests.Seebelow.
//ThisletsussetanypathcomponentwelikeintheURL,
//allowingustotestrooms(whichusetheURLintheirkey).
app.get('*',function(request,response){
response.sendfile(path.resolve('index.html'))
})
//Bindtheservertoport8080.
app.listen(8080)
DuringdevelopmentIlaunchtheserverinaterminaltabsoIcanwatchthelogsgoby.

Automaticversioning
Idontexpectthisprojecttohavelotsofchurn,butitsalwaysimportanttomakesurethatyouhavean
upgradepathifyouhavetoreleasechanges.Inthiscase,thatmeansmakingsureusersloadanexplicit
versionoftheInstantHangoutsscript.
We use package.json [https://github.com/google/instanthangouts/blob/master/package.json] to set the
version, then read this value when compiling the scripts our users load. These compiled files live in the
projectrootandarestrictlyadditive:everytimetheresanewversion,welladdnewfiles.Oldversionsare
neverremoved.Thismeansthatwewontbreakdeploymentsinthewildevenifwelaterservethefiles
fromaCDNandpeoplecopyandpastespecificURLsintotheirHTML.

Codeisolationatruntime,andvisibilityattesttime
Whileexecutingtestsitsusefultohaveaccesstotheinternalstateandbehaviorofyourcode.Otherwise
youre restricted to blackbox integration tests. Since Javascript has no namespaces, though, you dont
wanttostickallofyoursymbolsonwindowbecausetheycouldcollidewithcodethatexecutesbeforeor
afteryours.Thereareseveralstrategiesfordealingwithproperencapsulationofyourcode.
The

usual

technique

for

solving

this

the Javascript module pattern

[http://www.adequatelygood.com/JavaScriptModulePatternInDepth.html] . This is actually a family of patterns
dependingonyourprojectsrequirements.BecauseourprojectexposesnoJavascriptAPI,wecanuse
thesimplestform,whichiswrappingallourcodeinananonymousclosure:
http://johnmcox.blogspot.in/2014/01/opensourcejavascriptdevelopment.html

problem

is

4/7

7/20/2015

OpenSourceJavascriptDevelopment

(function(){
//Ourcodehere.
}())
Sinceallourcodelivesinsidethefunctionsscope,noneofoursymbolsarevisibleoutsidetheclosure.
Notethelastline:wereinvokingthefunction,soallourcoderunswhenthefileisloaded.
Thisisgreatforproductionbecauseitencapsulatesourcode.Butitslousyintestsbecause(waitforit!)it
encapsulates our code. Without exporting symbols we cant directly inspect the state or invoke the
behaviorofthecodeundertest.Whattodo?
Our solution is to have slightly different configurations at test time and in production. This is somewhat
heretical since any skew between test and prod keeps you from accurately testing how your code will
behave in the real world. So its important to minimize the skew and have a solid understanding of the
behaviorsitcandistort.
Inoursource files [https://github.com/google/instanthangouts/blob/master/src/instanthangouts.js] , we dont
writeourcodeinsideananonymousclosure.Whenourtestframeworkloadsthesourcefiles,allthetop
levelsymbolsarevisibleintheglobalJavascriptnamespace.Thismeanswecansimplyrefertothemin
our tests as if they were defined in the test source files [https://github.com/google/instant
hangouts/blob/master/test/instanthangouts.js] themselves. If we had multiple source files and needed
encapsulation within our project to manage its internal complexity, this approach would not be sufficient.
Foraprojectthissimple,itworkswellenough.
At runtime we dont load the source files we load processed [https://github.com/google/instant
hangouts/blob/master/instanthangouts0.1.0.js]

versions
[https://github.com/google/instant
hangouts/blob/master/instanthangouts0.1.0.uncompiled.js] fromtheprojectroot.Whenwegeneratethese
processedfiles,wewrapourcodeinaclosuretoencapsulateit.
WeuseGrunt[http://gruntjs.com/] ,aJavascriptautomatictaskrunner,togeneratetheseprocessedfilesin
theprojectroot.Therearetwoofthem:
instanthangouts<version>.js
instanthangouts<version>.uncompiled.js
Grunt is configured by Gruntfile.js [https://github.com/google/instanthangouts/blob/master/Gruntfile.js] .
Duringdevelopment,youinvokeitvia
$grunt
from the project root, assuming you have installed it globally. Otherwise, you can invoke it from
./node_modules/gruntcli/bin/grunt.
Gruntfile.jscontainstaskdefinitions.ThesetasksareunitsofworkthatGruntexecutesforyou.They
areinvokedautomaticallywhenevercertainconditionsaremet.Inourcase,thatconditionisachangeto
thecontentsofsrc/,test/,orindex.html.
In

the

filenames

above,

<version>

is

the

version

string

from

package.json

[https://github.com/google/instanthangouts/blob/master/package.json].Thelatterfileistheconcatenationof

src/*.js,wrappedinananonymousclosure.Theformerfileisaminifiedversionofthelatterfile,andis
whatourusersruninproduction.Thisminificationmakesourcodesmalleronthewire,decreasinglatency
inouruserspages.
http://johnmcox.blogspot.in/2014/01/opensourcejavascriptdevelopment.html

5/7

7/20/2015

OpenSourceJavascriptDevelopment

We want our users to load the minified version by default, which is why the unminified version has the
longer, more obscure name. During development, this unminified version is very helpful because it is
human readable and has line numbers you can refer to in your debugger of choice. By default, the
unminifiedversionisloadedbythedevelopmentserverviathe<script>taginindex.html. Because
Gruntregeneratesthecompiledfilesonsave,thismeansyoucanmakeachange,savethefile,andsee
theresultsimmediatelyinyournextresponsefromthedevserver.Onlychangestothedevservercode
itself in scripts/server.js [https://github.com/google/instanthangouts/blob/master/scripts/server.js] require a
serverrestart.

Automatictesting
We also want our tests to run automatically whenever the contents of src/, test/, or index.html
change. We could use Grunt for this, but we can get some really useful features from a dedicated test
runner.
WeuseKarma [http://karmarunner.github.io/0.10/index.html] .ThemainbenefitKarmaprovidesoverGruntis
theabilitytoexecuteourtestsinasmanyrealbrowsersaswewant.Karmadoesthisbyallowingyouto
attach browsers to its test server [http://karmarunner.github.io/0.10/intro/howitworks.html] . Whenever your
testsareexecuted,theyareruninsideeachattachedbrowser,andtheresultsaresenttoyourterminal.
Karmaisconfiguredbykarma.conf.js [https://github.com/google/instanthangouts/blob/master/karma.conf.js]
. In our configuration, we launch and attach Chrome locally. While developing, I run Karma in its own
terminalwith
$karmastart
EverytimeIchangeoneofthewatchedprojectfiles,Igetresultsinmyterminalprettymuchinstantly.
WewriteourtestsusingtheJasmine [http://pivotal.github.io/jasmine/] test framework. Karma executes the
testsJasminegivesyouadomainspecificlanguageforwritingthem.Ourtestsforthisprojecthavesome
importantlimitations:werenotsimulatingusersclickingontheHangoutcontrolsandstartingupaHangout.
Wereonlytestingthatweregeneratingthecontrolscorrectly.Wewouldhaveneededtologintoareal
Googleaccounttotestthingsendtoend,andthatmeanswedbetestingGooglescodeinadditiontoour
own.Generallythatsovertesting:itsalotofwork,anditsofquestionablevaluebecausethatcodeisnt
ourresponsibilityorunderourcontrol.
Instead,webasicallytreatthecontrolslikeablackbox:wetesttomakesuretheyreboxshapedandthe
rightshadeofblack.Weneedtoknow,forexample,thatwerecreatingthe<script>tagthatloadsthe
GoogleHangoutButtonAPIcorrectly,andthatwereinsertingourrendertargetscorrectly,andthatwere
settingwindowglobalsandparsingoptionsfromtheuserintherightway.OurJasminetestsdoallofthis,
andtheyusetheindex.html[https://github.com/google/instanthangouts/blob/master/index.html] fileourdev
serverservesasatestfixture(meaningitgivesustheDOMformanyofourtests).

Conclusion
TheJavascriptecosystemisabsolutelyteemingrightnow.Thisisgreatfordevelopersbecausethereare
abunchofreallygoodtoolsthatcanmakeourlivesaloteasier,butitalsomeansthatfindingtherighttools
andgettingthemtoworktogethercanbeabitofachallengebecausetheressuchaprofusionofchoice.I
spentabout50%moretimeassemblingmytoolchaincomparedtoactuallywritingprojectcode,whichisnt
agreatratio.Onfutureprojects,though,Illbeabletoreusealotofthetoolchainwork.Hopefullyitcan
saveyousometime,too.
Posted25thJanuary2014byjohnmcox
http://johnmcox.blogspot.in/2014/01/opensourcejavascriptdevelopment.html

6/7

7/20/2015

OpenSourceJavascriptDevelopment

0 Addacomment

Enteryourcomment...

Commentas:

Publish

GoogleAccount

Preview

http://johnmcox.blogspot.in/2014/01/opensourcejavascriptdevelopment.html

7/7