SUBSCRIBE&SAVE74%

MANAGEMYACCOUNT
STUDENT
GIVEAGIFT

Tech

searchNewScientist

Go

Logout

Home News InDepthArticles Opinion CultureLab Galleries TopicGuides LastWord Subscribe Dating
SPACE

TECH

ENVIRONMENT

HEALTH

LIFE

PHYSICS&MATH

SCIENCEINSOCIETY

MyNewScientist

LookforScienceJobs
Cookies&Privacy

MyNewScientist
Home | Tech | InDepthArticles

Nooneincontrol:Thealgorithmsthatrunourlives
04February2015byHalHodson
Magazineissue3007.Subscribeandsave

Like

72

Tweet

22

20

ShareShareShare

11

Thisweek'sissue
Subscribe

ADVERTISEMENT

07February2015
ADVERTISEMENT

More

Latestnews

Robotfirefighterputsoutitsfirstblaze
11:3805February2015

Designedforputtingout
flamesinthecramped
quartersofaship,the
SAFFiRrobothashadits
firstsuccessfultest

Docwatcherspotswhenphysicians
stoplistening
11:3205February2015

(Image:DavidLemm)
Softwareisdecidingwhogetsaloan,whocountsasacitizenandwhatprices
youpayonline.Whowillstepinwhenthemachinesgetoutofhand?
"AMAZONisallkindsofbroken."Ifyoucaughtthattweeton12December
lastyear,andwerequick,youmighthavegrabbedsomeexceptionalbargains.
Foranhouronly,Amazonwassellinganoddmixofitemscellphones,video
games,fancydresscostumes,mattressesforonepenny.
Thesurprisepricedropcostsellersdearly.Goodsusuallymarked100went
fora99.99percentdiscount.Hundredsofcustomersleaptatthechance,
oftenbuyinginbulk.EventhoughAmazonreactedquicklyandcancelledmany
orders,theywereunabletorecallthosethattheirautomatedsystemhad
alreadydispatchedfromwarehouses.Oncesetinmotion,theprocesswas
hardtostop.Thankstoasoftwareglitch,ahandfulofindependenttraders
usingAmazon'sMarketplaceloststockworthtensofthousandsofdollars.
Somefacedbankruptcy.

Asscreenusebecomes
evermoreubiquitousin
healthcare,doctorsmay
haveahardtimefocusing
onpatientsduring
consultations.LabinaBoxaimstohelp

Livingwiththealgorithmsthatrunour
lives
08:0005February2015

Mostofusdon'tunderstand
thesoftwarethathones
internetsearchesandour
newsfeeds,butwedon't
needtoinordertocometo
termswiththem

Addadashofquantumforsecure
cloudcomputing
18:3304February2015

Weonlynoticewhenalgorithmsgowrong.Mostofthetimetheygetonwith
businessoutofsightandoutofmind.Andbusinessisbooming.Automated
processesarenolongersimplytoolsatourdisposal:theyoftenmakethe
decisionsthemselves.Muchofthenewsweread,themusicwelistentoand
theproductswebuyareservedupautomatically,basedonstatistical
guessworkaboutwhatwewant.Invisiblechaperonesshapeouronline
experiences.Systemswecan'texamineanddon'tunderstanddeterminethe
routewetaketowork,therateswegetformortgages,andthepriceweseefor
airfares.
Manyareproprietaryandallarecomplex,pushingthembeyondpublicscrutiny.
Howcanwebesurethey'replayingfair?Anewwaveofalgorithmauditorsare
onthecase,intentonpullingbackthecurtainonthehiddenworkingsand
huntingforunduebiasordiscrimination.Butisthisthefix?Doalgorithmsneed
betterpolicing,ormustweaccepttheirnatureasapricewepayforour
automatedworld?
There'snothinginherentlymysteriousaboutthem:analgorithmissimplyaset
ofinstructionsforgettingsomethingdone.Thetroubleisthatalgorithmsget
nestedinsideorboltedontoothers,interactinginevermorecomplexways.It
canalsobehardtopredicthowalgorithmswillbehavewithrealworlddata
oncereleasedintothewild.
Thescopeoftheirinfluenceisoftenunclear.Somepeopleswearblindthat
they'veseenthepriceofflightsononewebsitejumpaftercheckingoutarival
site,forexample.Othersthinkthat'sbunk,anurbanmythforourtimes.Such
debateshighlighttheshadowynatureoftoday'ssystems.
Notonlyaremostalgorithmssecretrecipes,sometimeseventhedevelopers
whowrotethemareinthedark.WhenAnikoHannakatNortheastern
UniversityinBoston,Massachusetts,lookedcloselyathowmanyofushave
oursearchresultsskewedbyfactorslikelocationandbrowsinghistory,she
notedthingsevenGoogledidn'tknow:forexample,thataround12percentof
searchesgetpersonalised.Googleengineersthankedher.They'dnevermade
suchmeasurementsandhadn'tknowntheexactimpactoftheirpersonalisation
algorithms.
Exposinghiddenalgorithmscancauseoutrage.That'swhatChristianSandvig
andhiscolleaguesattheUniversityofMichigan,AnnArbor,foundwhenthey
liftedthelidonFacebook'snewsfeedalgorithms,whichdecidewhichposts
fromfriendsandfamilyweactuallysee.Theteamcomparedfilteredand
unfilteredfeedsandfoundthatFacebook'salgorithmshidpostsdeemed
uninteresting,accordingtounspecifiedcriteria.
AroundtwothirdsoftheparticipantsinSandvig'sstudyhadnoideathat
algorithmsweredecidingwhattheysaw.Manywereshockedandupsetwhen
postsfromclosefriendsorfamilywereexcluded.Somehadbeenblaming
themselvesortheirfriendsforthealgorithms'work."Ifyoupostsomethingand
itdoesn'tgetanycommentsorlikes,peopleassumethateithertheirfriends
don'tlikethetopic,ortheirfriendsdon'tlikethem,"saysSandvig.
Evenfornews,it'sapopularitycontest.Duringlastyear'sFergusonriotsin
Missouri,forexample,Facebook'snewsfeedswerefilledwithpostsaboutthe
IceBucketChallengebecausethesehadhundredsofthousandsoflikes.
WhatSandvig'steamdidforFacebook,Hannakandhercolleaguesaredoing
forotheronlineactivity.Hannakisinterestedinhowalgorithmscantailorprices
todifferentshoppers.Inonerecentstudy,theresearcherslookedathowonline
retailerssuchasWalmart,OfficeDepotandExpediavariedpricesaccordingto
factorsincludingauser'schoiceofbrowser,operatingsystemandpurchase
history.
Theyfoundmanyinstancesofwhattheyconsiderpricediscriminationthough
theyarenotsureoftherationale.Oftenthedifferencewassmall.Android
users,forexample,sawhigherpricesonabout6percentofitems,thoughonly
byafewcents.Inothercases,pricequotesvariedbyupto$100.Thegreatest
differencesweretypicallyseenbetweenuserswhowereloggedintoasiteand
thosewhowerenot.

Crashdamage
Hannak'sgroupnowwantstounderstandexactlyhowlocationinfluences
searchresults.TheyaresimulatinghundredsofAndroidphonesandspreading
themacrossOhiousingfakedGPScoordinates.They'llalsobelookingtosee
whetherpeoplefromrichandpoorneighbourhoodsgetdifferentsearchresults
whenhuntingforfinancialservices.
Evidenceofthatmayalreadyhavecometolight.Somethinkhidden
algorithmsplayedapartinthe2008subprimemortgagecrash.Between2000
and2007,USlenderslikeCountrywideHomeLoansandDeepGreendoledout

It'stoughtocomputeinthecloudwhilekeeping
yourdatasecret,butstrikingabalancebetween
usingquantumbitsandencryptioncouldmakeit
easier
seeallrelatedstories

Mostread
GreenlightformissiontoJupiter
moonEuropa
SubglacialLakeVostokcrackedfora
secondtime
Cosmicinflation's'smokinggun'goes
upinsmoke
Spermwhale'semergency
evacuation...ofitsbowels
Lilacandturquoiseareasbasicasred
andorange

FOLLOWUS
Geteditors'picksinyoursocial
streams

homeloansatanunprecedentedrateviaautomatedonlineapplications.
"Everyonewassayingwhatagreatinnovationitwas,"saysDanPoweratthe
UniversityofNorthernIowainCedarFalls."Everyonewasveryhighonthese
fastwebbasedloans.Nooneanticipatedtheproblem."
Theproblemwasgrantingsomanyhighriskloanswithouthumanoversight.
Americansfromminoritygroupssufferedmostintheresultingcrash.
Automatedprocessescrunchedthroughvastamountsofdatatoidentifyhigh
riskborrowerswhoarechargedhigherinterestratesandtargetedthemto
sellmortgages."ThoseborrowersturnedouttobedisproportionatelyAfrican
AmericanandLatino,"saysSeetaGangadharanoftheOpenTechnology
Institute,apublicpolicythinktankbasedinWashingtonDC."Algorithms
playedaroleinthatprocess."
Theexactdegreetowhichalgorithmsweretoblameremainsunclear.But
bankslikeWellsFargoandBankofAmericasettledwithseveralcities,
includingBaltimore,Chicago,LosAngelesandPhiladelphia,forhundredsof
millionsofdollarsoverclaimsthattheirsubprimelendinghad
disproportionatelyaffectedminorities.Althoughthedecisionmakingprocess
bigbanksusedtotargetandsellsubprimeloansmaynothavebeennewin
itself,thereachandspeedofthosedecisionswhenalgorithmswerethedriving
forcewasnew."It'sthescalefactor,"saysGangadharan."Thiswasaproblem
thataffectedmanypeopleintheUSandwehaveseentheeffectsfallalong
raceandclasslinesindevastatingways."
Automatedsystemsarereplacinghumandiscretioninevermoreimportant
decisions.In2012,theUSStateDepartmentstartedusinganalgorithmto
randomlyselectthewinnersofthegreencardlottery.Thesystemwasbuggy,
however:itawardedvisasonlytopeoplewhoappliedonthefirstday,says
JoshKroll,aPrincetonUniversitycomputerscientistwhoisinvestigatingthe
event.Thosevisaswererescinded,butit'sagoodexampleofhowhidden
algorithmscanhavealifechangingeffect.
Inasimilarexample,thedocumentsleakedbyEdwardSnowdenrevealedthat
theNationalSecurityAgencyusesalgorithmstodecidewhetherapersonisa
UScitizen.AccordingtoUSlaw,onlynoncitizenscanhavetheir
communicationsmonitoredwithoutawarrant.Intheabsenceofinformation
aboutanindividual'sbirthplaceorparents'citizenship,theNSAalgorithmsuse
othercriteria.Isthispersonincontactwithforeigners?Dotheyappeartohave
accessedtheinternetfromaforeigncountry?Dependingonwhatyoudo
online,yourcitizenshipmightchangeovernight."Onedayyoumightbea
citizen,anotheryoumightbeaforeigner,"saysJohnCheneyLippold,atthe
UniversityofMichiganinAnnArbor."It'sacategoricalassessmentbasedon
aninterpretationofyourdata,notyourpassportoryourbirthcertificate."
Algorithmsarealsousedtopolicevoterfraud.SeveralUSstatesusesoftware
calledCrosschecktoremoveduplicateentriesfromelectoralregisters.But
peoplehavebeendeletedsimplyforhavingthesamename.Aswiththesub
primealgorithms,minoritiesareagainhithardest.Thenamesitscrubsare
disproportionatelythoseofblack,AsianandHispanicvoters,whoaremore
likelytosharenamessuchasJackson,KimorGarcia.
Thenextscandalmaybeprisonsentencing.JudgesandlawyersinMissouri
canuseawebsitetomakean"AutomatedSentencingApplication".The
systemcalculatesincarcerationcostsfordefendants,andweighsthatagainst
thelikelihoodthedefendantwillreoffend,basedonpriorcriminalhistoryand
behaviouralanddemographicfactors.Somethinkthiswillleadtominorities
beinggivenharshersentences.Proxieslikeaddress,incomeandeducation
levelmakeitalmostimpossibletoavoidracialbias.Similarsystemsare
appearingacrosstheUS."Ithinkit'sterrifying,"saysSorelleFriedler,a
computerscientistatHaverfordCollegeinPennsylvania.
Thescalesarefallingfromoureyesastheimpactofalgorithmsisfeltin
almosteveryareaofourlives.Whatshouldwedoaboutit?Inmanyofthese
examples,theproblemisnotthealgorithmsthemselves,butthefactthatthey
overamplifyanexistingbiasinthedata.

Higherstandards
"Peoplewhoworkwithalgorithmsarecomfortablewiththeideathatthey
mightproducetheseunintendedresults,"saysSandvig.Butforagrowing
numberofpeople,that'snotgoodenough.ChristoWilson,whoworkswith
HannakatNortheasternUniversity,thinksthatlargetechnologycompanieslike
GoogleandFacebookoughttobeconsideredaspublicservicesthathuge
numbersofpeoplerelyon."Giventhattheyhaveabillioneyeballs,Ithinkthey
havearesponsibilitytoholdthemselvestoahigherstandard,"hesays.
Wilsonthinksthatautomatedsystemsmightbemademoretrustworthyifusers
cancontrolexactlyhowtheirresultsarepersonalisedsuchasleavinggender
outoftheequationorignoringincomebracketandaddress.Itwouldalsohelp

uslearnhowthesesystemswork,hesays.
Othersarecallingforanewregulatoryframeworkgoverningalgorithms,much
likewehaveforthefinancialindustry,forexample.Arecentreport
commissionedbytheWhiteHouserecommendsthatpolicymakerspaymore
attentiontowhatthealgorithmsdowiththedatatheycollectandanalyse.To
ensureaccountability,however,therewouldneedtobeindependentauditors
whoinspectalgorithmsandmonitortheirimpact.Wecannotleaveitto
governmentsorindustryalonetorespondtotheproblems,saysGangadharan.
"Thebigquestionnowformeiswhoarethewatchdogs,"saysSandvig.For
now,hesuggestsitshouldbetheresearcherswhoarebeginningtoreveal
algorithms'broadereffects.Wilson,forexample,islookingintosettingup
dummycreditprofilestobetterunderstandpricefixingsystems.But
independentauditorsfacetoughobstacles.Forastart,diggingaroundinside
manyautomatedservicesviolatestheirtermsofuseagreement,which
prohibitsattemptstoanalysehowtheywork.UndertheUSComputerFraud
andAbuseAct,suchsnoopingmayevenbeillegal.Andwhilepublicscrutinyis
important,thedetailsofproprietaryalgorithmsneedtobekeptsafefrom
competitorsorhackers,forexample.
What'smore,mostautomatedsystemsaretoocomplexforhumanstoinspect
byhand.Sosomeresearchershavedevelopedalgorithmsthatcheckother
algorithms.Krollisworkingonasystemthatwouldletanauditorverifythatan
algorithmdidwhatitwassupposedtowithwhatitwasgiven.Inotherwords,it
wouldprovideafoolproofwayofcheckingthattheoutcomeofthegreencard
lottery,forexample,wasinfactrandom.Orthatadriverlesscar'salgorithmfor
avoidingpedestrianstreatsbothpeoplewalkingandpeopleinwheelchairswith
thesamecaution.
Friedlerhasadifferentapproach.Byunderstandingthebiasesinherentinthe
underlyingdata,shehopestoeliminatebiasinthealgorithm.Hersystemlooks
forcorrelationsbetweenarbitrarypropertieslikeheightoraddressand
demographicgroupingslikeraceorgender.Ifthecorrelationisexpectedto
leadtounwantedbias,thenitwouldmakesensetonormalisethedata.Itis
essentiallyaffirmativeactionforalgorithms,shesays.
That'sfineforcaseswherediscriminationisclear,whereasystemisfoundto
beunfairorillegal.Butwhatifthereisdisagreementabouthowanalgorithm
oughttobehave?ManywouldsayFacebook'sfilteringofitsnewsfeedkeepsit
readable.Somewouldarguethathighlypersonalisedpriceadjustmentcan
benefitbothcustomersandretailers.What'sacceptabletosomewon'tbefor
others.
AsSandvignotes,unlikeforfinancialsystems,therearenostandardsof
practicegoverningalgorithms.Buthowwewantthemtobehavemayturnout
tobeaharderquestionforsocietytoanswerthanwethink.Maybewe'llneed
analgorithmforthat.
Leader:"Livingwiththealgorithmsthatrunourlives"
Thisarticleappearedinprintundertheheadline"Nooneincontrol"
HalHodsonisareporteratNewScientist
Fromissue3007ofNewScientistmagazine,
page3033.
Asasubscriber,youhaveunlimitedaccessto
ouronlinearchive.
WhynotbrowsepastissuesofNewScientist
magazine?

Like

72

Tweet

22

ShareShareShare

MOREFROMNEWSCIENTIST

20

11

Shoesvs
barefoot:The
mythofthe
normalfoot

Secretsofthe
home:What
yourhouse
saysaboutyou

Feedback:Nice
sliceoffried
man,Sir?

Talking
gibbonish:
Decipheringthe
banterofthe
apes

3Tipsfor
Calculatingthe
RightSoftware
Choice
(RepslyBlog)

Whatinthe
worldisSix
Sigma?
(Villanova
University)

PROMOTEDSTORIES

How3DPrinters
AreEmpowering
Customers
(Autodesk)

GivetheGiftof
Good
Communication
(CrisisPrevention
Institute)

Recommendedby

IfyouwouldliketoreuseanycontentfromNewScientist,eitherinprintoronline,please
contactthesyndicationdepartmentfirstforpermission.NewScientistdoesnotownrights
tophotos,butthereareavarietyoflicensingoptionsavailableforuseofarticlesandgraphics
weownthecopyrightto.

Backtotop

searchNewScientist

Logout

Go

Aboutus

UserHelp

Subscriptions

Links

ScienceJobs

NewScientist
Syndication
RecruitmentAdvertising
StaffatNewScientist
Advertise
RBIJobs

ContactUs
FAQ/Help
Disclaimer
Ts&Cs
Cookies
PrivacyPolicy

Subscribe
Renew
Giftsubscription
StudentSubscription
Backissues
TheCollection
TheAntiZoo

SiteMap
Browseallarticles
Magazinearchive
NewScientistJobs
TheLastWord
RSSFeeds
OnlineStore
iOS&Androidapps
Lowbandwidthsite

SearchallJobs
BiologyJobs
ChemistryJobs
ClinicalJobs
Earth&Environment
Jobs
EngineeringJobs
Maths&ITJobs
CareersAdvice

CopyrightReedBusinessInformationLtd.