11/20/12

When the Nerds Go Marching In - Technology - The Atlantic

SUBSCRIBE
RENEW
GIVEAGIFT
DIGITALEDITION
Print | Close

WhentheNerdsGoMarchingIn
ByAlexisMadrigal
HowadreamteamofengineersfromFacebook,Twitter,andGooglebuiltthesoftwarethatdrove
BarackObama'sreelection

ThreemembersofObama'stechteam,fromlefttoright:HarperReed,DylanRichard,andMark
Trammell(PhotobyDanielX.O'Neil).
TheObamacampaign'stechnologistsweretenseandtired.Itwasgamedayandeverythingwasgoing
wrong.
JoshThayer,theleadengineerofNarwhal,hadjustbeeninformedthatthey'dlostanotheroneofthe
servicespoweringtheirsoftware.Thatwasbad:Narwhalwasthecodenameforthedataplatform
thatunderpinnedthecampaignandletittrackvotersandvolunteers.Ifitbroke,sowouldeverything
else.
TheyweretalkingwithpeopleatAmazonWebServices,butalltheyknewwasthattheyhadpacket

www.theatlantic.com/technology/print/2012/11/when-the-nerds-go-marching-in/265325/

1/13

11/20/12

When the Nerds Go Marching In - Technology - The Atlantic

loss.Earlierthatday,theylosttheirdatabases,theirEastCoastservers,andtheirmemcacheclusters.
ThayerwasreadytokillNickHatch,aDevOpsengineerwhowastheofficialbearerofbadnews.
Anotheroftheirvendors,PalominoDB,wasfixingdatabases,butneededtorebuildthereplicas.It
wasgoingtotaketime,Hatchsaid.Theydidn'thavetime.
They'dbeenworking14hourdays,sixorsevendaysaweek,tryingtoreelectthepresident,andnow
everythinghadbeenbrokenatjustthewrongtime.ItwaslikesomeonehadwrittenaMurphy'sLaw
algorithmanddeployeditatscale.
Andthatwasthepoint."Gameday"wasOctober21.Theelectionwasstill17daysaway,andthiswas
aliveactionroleplaying(LARPing!)exercisethatthecampaign'schieftechnologyofficer,Harper
Reed,wasinflictingonhisteam."Weworkedthrougheverypossibledisastersituation,"Reedsaid.
"Wedidthreeactualalldaysessionsofdestroyingeverythingwehadbuilt."
Hatchwasplayingtheroleofdungeonmaster,callingoutdevilishlycomplexscenariosthatwere
designedtotesteachandeverypieceoftheirsystemastheyenteredtheexponentialtrafficgrowth
phaseoftheelection.MarkTrammell,anengineerwhoReedhiredafterheleftTwitter,sawacouple
gamedays.HesaidtheyremindedhimofhistimeintheNavy."Youranfirefightingdrillsoverand
overandover,tomakesurethatyounotjustknowwhatyou'redoing,"hesaid,"butyou'recalm
becauseyouknowyoucanhandleyourshit."
Theteamhadeliteand,fortech,seniortalentbywhichImeanthatmostofthemwereintheir30s
fromTwitter,Google,Facebook,Craigslist,Quora,andsomeofChicago'sownsoftwarecompanies
suchasOrbitzandThreadless,whereReedhadbeenCTO.Buteventhesepeople,maybe*especially*
thesepeople,knewenoughabouttechnologynottotrustit."IthinktheRepublicansfuckedupinthe
hubrisdepartment,"Reedtoldme."IknowwehadthebesttechnologyteamI'veeverworkedwith,
butwedidn'tknowifitwouldwork.Iwasincrediblyconfidentitwouldwork.Iwasbettingalotonit.
Wehadtime.Wehadresources.Wehaddonewhatwethoughtwouldwork,anditstillcouldhave
broken.Somethingcouldhavehappened."
Infact,thedayaftertheOctober21gameday,Amazonservicesonwhichthewholecampaign'stech
presencewasbuiltwentdown."Wedidn'thaveanydowntimebecausewehaddonethatscenario
already,"Reedsaid.HurricaneSandyhitonanothergameday,October29,threateningthe
campaign'swholeEastCoastinfrastructure."WecreatedahotbackupofallourapplicationstoUS
westinpreparationforUSeasttogodownhard,"Reedsaid.
"Weknewwhattodo,"Reedmaintained,nomatterwhatthescenariowas."Wehadarunbookthat
saidifthishappens,youdothis,this,andthis.TheydidnotdothatwithOrca."
THENEWCHICAGOMACHINEvs.THEGRANDOLDPARTY
OrcawassupposedtobetheRepublicananswertoObama'sperceivedtechadvantage.Inthedays
leadinguptotheelection,theRomneycampaignpushedits(notso)secretweaponastheanswerto
theDemocrats'vauntedgroundgame.Orcawasgoingtoallowvolunteersatpollingplacestoupdate
theRomneycamp'sdatabaseofvotersinrealtimeaspeoplecasttheirballots.Thatwouldsupposedly
allowthemtodeployresourcesmoreefficientlyandwringeverylastvoteoutofFlorida,Ohio,and
theotherbattlegroundstates.Theproductgotitsname,aRomneyspokespersontoldNPRPBS,
becauseorcasaretheonlyknownpredatoroftheonetuskednarwhal.
ThebillingtheRepublicansgavethetoolconfusedalmosteveryoneinsidetheObamacampaign.

www.theatlantic.com/technology/print/2012/11/when-the-nerds-go-marching-in/265325/

2/13

11/20/12

When the Nerds Go Marching In - Technology - The Atlantic

Narwhalwasn'tanappforasmartphone.Itwasthearchitectureofthecompany'ssophisticateddata
operation.NarwhalunifiedwhatObamaforAmericaknewaboutvoters,canvassers,eventgoers,and
phonebankers,anditdiditinrealtime.FromthedescriptionsoftheRomneycamp'ssoftwarethat
wereavailablethenandnow,OrcawasnoteveninthesamecategoryasNarwhal.Itwasliketouting
theiPadasaFacebookkiller,orcomparingaGPSdevicetoanengine.Andbesides,intheschemeofa
campaign,adigitizedstrikelistiscool,butit'snot,like,agamechanger.It'sjustanicethingtohave.
So,itwaswithmorethanahintofschadenfreudethatReed'steamhearthatOrcacrashedearlyon
electionday.Laterreportspostedbyrankandfilevolunteersdescribechaosdescendingonthe
pollinglocationsasonlyafractionofthetensofthousandsofvolunteersorganizedfortheeffortwere
abletouseitproperlytoturnoutthevote.
Ofcourse,theycouldn'tsnickertooloudly.Obama'scampaignhadcreatedasimilarappin2008
calledHoudini.AsdetailedinSashaIssenberg'sgroundbreakingbook,TheVictoryLab,Houdini's
rolloutwentgreatuntilabout9:30amEasternonthedayoftheelection.Thenitcrashedinmuchthe
samewayOrcadid.
In2012,Democratshadanewversion,builtbythevendor,NGPVAN.ItwascalledGordon,afterthe
manwhokilledHoudini.Butthe2008failure,amongotherneeds,drovethe2012Obamateamto
bringtechnologistsinhouse.
Withelectiondaybearingdownonthem,theyknewtheycouldnotgodown.Andyettheyhadto
accommodatemuchmorestrainonthesystemsasinterestintheelectionpickeduptowardtheend,
asitalwaysdoes.MarkTrammell,whoworkedforTwitterduringitsperiodofexponentialgrowth,
thoughtitwouldhavebeeneasyfortheObamateamtofallintomanyofthepitfallsthatthesocial
networkdidbackthen.Butwhiletheproblemsofscalingbothtechnologyandculturequicklymight
havebeensimilar,thestakesweremuchhigher.Afailwhale(cough)inthedaysleadinguptooron
November6wouldhavebeenneithercharmingnorfunny.Inaracethatatleastsomepeoplethought
mightbeveryclose,itcouldhavecostthePresidenttheelection.
Andofcourse,theteam'sonlyrealgoalwastoelectthePresident."WehavetoelectthePresident.
Wedon'tneedtoselloursoftwaretoOracle,"Reedtoldhisteam.Butthesecondaryimpactoftheir
successorfailurewouldbetoprovethatcampaignscouldeffectivelyhireanddeploytoplevel
programmingtalent.Iftheyfailed,itwouldbeevidencethatthisstuffmightbebestlefttooutside
politicaltechnologyconsultants,bywhomthearenahadlongbeenhandled.IfReed'steamsucceeded,
engineersmightbecomeasenshrinedinthemechanicsofcampaignsassocialmediateamsalready
are.
Wenowknowwhathappened.Thegrandtechnologyexperimentworked.Solittlewentwrongthat
TrammellandReedevenhadtimetocookupalittlepintocelebrate.Itsaid,"YOLO,"shortfor"You
OnlyLiveOnce,"withtheObamaOs.
WhenObamacampaignchiefJimMessinasignedoffonhiringReed,hetoldhim,"Welcometothe
team.Don'tfuckitup."AsElectionDayendedandthedustsettled,itwasclear:Reedhadnotfucked
itup.
Thecampaignhadturnedoutmorevolunteersandgottenmoredonorsthanin2008.Sure,thefield
organizationwasmoreentrenchedandexperienced,butthedifferencestemmedinlargepartfrom
bettertechnology.Thetechteam'skeyproductsDashboard,theCallTool,theFacebookBlaster,
thePeopleMatcher,andNarwhalmadeitsimplerandeasierforanyonetoengagewiththe

www.theatlantic.com/technology/print/2012/11/when-the-nerds-go-marching-in/265325/

3/13

11/20/12

When the Nerds Go Marching In - Technology - The Atlantic

President'sreelectioneffort.
Butitwasn'teasy.Reed'steamcameinasoutsiderstothecampaignandbymostaccounts,remained
thatway.Thedivisionsamongthetech,digital,andanalyticsteamneverquitegotresolved,evenif
theendproducthassalvedthesorespotsthatdevelopedoverthestressfulmonths.Attheirworst,in
early2012,theculturaldifferencesbetweentechandeverybodyelsethreatenedtoderailthewhole
grandexperiment.
Bytheend,thecampaignproducedexactlywhatitshouldhave:ahybridofthedesiresofeveryoneon
Obama'steam.Theyraisedhundredsofmillionsofdollarsonline,madeunprecedentedprogressin
votertargeting,andbuilteverythingatopthemoststabletechnicalinfrastructureofanypresidential
campaign.Togoastepfurther,I'devensaythatthisclashofcultureswasagoodthing:Thenerds
shookupanossifyingDemocratictechstructureandthepoliticostaughtthenerdsathingortwo
aboutstress,smallppolitics,andthesignificanceofelections.
YOLO:MEETTHEOBAMACAMPAIGN'SCHIEFTECHNOLOGYOFFICER

ThePresidenthuggingHarperReedasshownonhisInstagramfeed.
Ifyou'reanerd,HarperReedisaneasyguytolike.He'sbrashandfunnyandsmart.Hegetsyouand
whereyoucamefrom.He,too,playedwithcomputerswhentheyweren'tcool,andlearnedtocode
becausehejustcouldnothelphimself.Youcouldcalloutnouns,phenomena,andhe'dberightthere
withyou:BBS,warez,selforganizingsystems,Rails,thequantifiedself,Singularity.Hewrotehisfirst
programsatageseven,gamesthathismomtypedintotheirAppleIIC.He,too,hasamemorythatall
nerdsshare:Lateatnight,lightfromachunkymonitorilluminatinghisface,fingersflyingacrossa
keyboard,hefiguredsomethingout.
TVnewssegmentsaboutcybersecuritymightlookliftedstraightfromhismemories,butthebroll
theyshotofdarkenedroomsandtypinghandscouldnotconveythesenseofexhilarationhefeltwhen
hebuiltsomethingthatworks.HarperReedgotthecityofChicagotocreateanopenandrealtime

www.theatlantic.com/technology/print/2012/11/when-the-nerds-go-marching-in/265325/

4/13

11/20/12

When the Nerds Go Marching In - Technology - The Atlantic

feedofitstransitdatabyreverseengineeringhowtheyservedbuslocationinformation.Why?
BecauseitmadehiswifeHiromi'scommutealittleeasier.Becauseitwasfuntoextractthedatafrom
thebureaucracyandmakeitavailabletoanyonewhowantedit.Becauseheisanerd.
YetReedhasfriendslikethemanagerofthehiphopclubEmpirewho,whenwewalkintotheplace
earlyontheFridayaftertheelection,says,"Letmegrabyouashot."Surprisingly,HarperReedisa
chilledvodkakindofguy.Unsurprisingly,HarperReedreadStevenLevy'sHackersasakid.
Surprisingly,themanager,whoistallandhandsomewithrockandrollhairflowingfrombeneatha
redbeanie,returnstoshowHarperphotographsofhiskids.They'veknowneachotherforalong
while.Theyarereallygrowingup.
Asthenightrollson,andtheclubstartstofillup,anotherfriendapproachedus:DJHiroki,whowas
spinningthatnight.HarperReedknowstheDJ.Ofcourse.AndHirokigrabsusanothershot.(Atthis
pointI'mthinking,"Bytheendofthenight,eitherIpassoutorReedtellsmesomethinggood.")
Hiroki'sbeenDJingatEmpireforyears,sinceHarperReedwasthecrazyguyyoucanseeonhis
publicFacebookphotos.Inoneshotfrom2006,askinnyReedsitsinabathtubwithabeerinhis
hand,twothickbandtattoosrunningacrosshischestandshoulders.Heisnotwearinganyclothes.
Thecaptionreads,"Stopstaring,it'snotthereiswear!"WhatmakesHarperReeddifferentisn'tjust
thatthephotoexists,butthathekeptitpublicduringtheelection.
Yetifyou'vespentalotoftimearoundtechpeople,aroundBurningMandevotees,aroundstartups,
aroundSanFrancisco,aroundBBSs,aroundReddit,HarperReedprobablymakessensetoyou.He'sa
coolhacker.HegetsprofiledbyMotherJoneseventhoughhecouldn'ttalkwithTimMurphy,their
reporter.Hesupportsopensource.HelikesJapan.Hesaysfuckalot.Hegoestohipsterbarsthat
serveveganMexicanfood,andwhereaquarterofthestaffandclientelehavemustaches.
Hemaybelikeyou,buthealsojugglesbetterthanyou,andiswilderthanyou,morefunthanyou,
coolerthanyou.He'swhatakingofthenerdsreallylookslike.Sure,hemightgrowabeardandput
onalittlepotbelly,buthewouldn'ttuckinhistshirt.Heisnotthatkindofnerd.Instead,he'sgot
plugsinhisearsandashockofgloriouslyproductmussedhairandhipsterglassesandhedoesn'town
alongsleevedressshirt,incaseyouwerewondering.
"Harperisaneasyguytounderestimatebecausehelooksfunny.Thatmightbepartofhisbrand,"
saidChrisSacca,awellknownSiliconValleyventurecapitalistandmajorObamabundlerwho
broughtateamofmorethanadozentechnologistsoutforanObamacampaignhackday.
Reed,forhispart,hasthekindofselfawarenessthatfacesoutward.Hisselfannouncedflawsbristle
likequills."Ialwayslooklikeafuckingidiot,"Reedtoldme."Andifyoulooklikeanasshole,you
havetobereallygood."
ItwasalessonhelearnedearlyoutinGreeley,Colorado,wherehegrewup."Ihadthisexperience
wheremydadhiredsomeonetohelphimoutbecausehisnetworkwasmessedupandhewantedme
towatch.AndthiswasataveryunfortunatetimeinmylifewhereIwaswearingverybaggypantsand
IhadaMarilynMansonshirtonandIlookedlikeanasshole.Andmyfathertookmeasideandwas
like,'Whydoyoulooklikeanasshole?'AndIwaslike,'Idon'tknow.Idon'thaveananswer.'ButI
realizedIwasjustasgoodastheguysfixingit,"Reedrecalled."Andtheydidn'tlooklikemeandI
didn'tlooklikethem.AndifI'mgoingtodothis,andlooklikeanidiot,Ihavetostepup.Likeifwe're
allatzero,Ihavetobeat10becauseIhavethisstupidmustache."
Andinfact,hemayactuallybeat10.Saccasaidthatwithtechnicalpeople,it'sonethingtolookat

www.theatlantic.com/technology/print/2012/11/when-the-nerds-go-marching-in/265325/

5/13

11/20/12

When the Nerds Go Marching In - Technology - The Atlantic

theirresumesandanothertoseehowtheyareviewedamongtheirpeers."Anditwasamazinghow
manyincrediblywellregardedhackersthatIfollowonTwitterrejoicedandcelebrated[whenReed
washired],"Saccasaid."Lotsofguyswhoknowhowtospitoutcode,theyreallyboughtthat."
BythetimeSaccabroughthisSiliconValleycontingentouttoChicago,hecalledthetechnicalteam
"topnotch."Afterall,we'retalkingaboutagroupofpeoplewhohadEricSchmidtsittinginwiththem
onElectionDay.Youhadtobegood.Thetechworldwaswatching.
TerryHowerton,theheadoftheIllinoisTechnologyAssociationandafrankobserverofChicago's
techscene,hadonlyglowingthingstosayaboutReed."HarperReed?Ithinkhe'swickedsmart,"
Howertonsaid."Heknowshowtopullpeopletogether.Ithinkthatwasprobablywhatattractedthe
restofthepeoplethere.Harperisresponsibleforahugepercentageofthepeoplewhowentover
there."
Reed'sownteamfoundtheircoworkersparticularlyimpressive.Onetestamenttothatisseveral
startupsmightspinoutoftheconnectionspeoplemadeattheOFAheadquarters,suchasOptimizely,
atoolforwebsiteA/Btesting,whichspunoutofObama's2008bid.(Sacca'sactuallyaninvestorin
thatone,too.)
"ACTOroleisaweirdthing,"saidCarolDavidsen,wholeftMicrosofttobecometheproduct
managerforNarwhal."Theprimaryresponsibilityisgettinggoodengineers.Andtherereallywasno
oneelselikehimthatcouldhaveassembledthesepeoplethatquicklyandgetthemtotakeapaycut
andmovetoChicago."
Andyet,theverythingsthatmakeReedaninterestingandbelovedpersonarethesamethingsthat
makehimanunlikelypicktobecomethechieftechnologyofficerofthereelectioncampaignofthe
PresidentoftheUnitedStates.Politicalpeoplewearkhakis.Theyonlyownlongsleevedressshirts.
TheiroldphotosonFacebookshowthemcanvassingforlocalpoliticiansandwinningcrosscountry
meets.
IaskedMichaelSlaby,Obama's2008chieftechnologyofficer,andtheguywhohiredHarperReed
thistimearound,ifitwasn'triskytohirethiswildguyintoapresidentialcampaign."It'sfunnyto
hearyoucallitrisky,itseemsobvioustome,"Slabysaid."Itseemscrazytohiresomeonelikemeas
CTOwhenyoucouldhavesomeonelikeHarperasCTO."
THENERDSAREINSIDETHEBUILDING
Thestrangetruthisthatcampaignshavelongbeenlowtechnologist,ifnotlowtechnology,affairs.
Thinkofthemasaweirdkindofnichestartupandyoucanseewhy.Youhaveverylittletime,maybea
year,really.Youcan'taffordtopayverymuch.Thejobsecurity,bydesign,isnonexistent.Andeven
thoughyouneedtobuildamassive"customer"baseanddeveloptheinfrastructuretogetmoneyand
votesfromthem,noonegetstoexitandmakeabunchofmoney.So,campaigntechhasbeen
dominatedbypeoplewhocareaboutthepoliticsofthething,notthetechnologyofthething.The
websitesmighthavelookedlikesolidconsumerwebapplications,buttheywerenotunderthehood.
ForallthehooplasurroundingthedigitalsavvyofPresidentObama's2008campaign,andasmuchas
everyoneIspokewithlovedit,itwasnotasheavilydigitalortechnologicalasitisnowremembered.
"Facebookwasaboutonetenthofthesizethatitisnow.Twitterwasanothingburgerforthe
campaign.Itwasn'tacoreorevenperipheralpartofourstrategy,"saidTeddyGoff,DigitalDirector
ofObamaforAmericaandaveteranofbothcampaigns.Thinkaboutthekillertoolofthatcampaign,

www.theatlantic.com/technology/print/2012/11/when-the-nerds-go-marching-in/265325/

6/13

11/20/12

When the Nerds Go Marching In - Technology - The Atlantic

my.barackobama.comItborrowedthemyfromMySpace.
Sure,the'08campaignhadFacebookcofounderChrisHughes,butHugheswasthespokespersonfor
thecompany,notitstechnicalguy.The'08campaigners,Slabytoldme,hadbeen"opportunisticusers
oftechnology"who"bruteforcedandbalingwired"differentpiecesofsoftwaretogether.Things
worked(mostofthetime),buteveryone,Slabyespecially,knewthattheyneededamorestable
platformfor2012.
Campaigns,however,evenHowardDean'sfamous2004InternetenabledrunattheDemocratic
nomination,didnothireabunchoftechnologists.Thoughtheyhiredacouple,likeClayJohnson,they
boughttechnologyfromoutsideconsultants.For2012,Slabywantedtochangeallthat.Hewanted
dozensofengineersinhouse,andhegotthem.
"Therealinnovationin2012isthatwehadworldclasstechnologistsinsideacampaign,"Slabytold
me."Thetraditionaltechnologystuffinsidecampaignshadnotbeenatthesamelevel."Andyetthe
technologists,nomatterhowgoodtheywere,broughtadifferentworldview,setofpersonalities,and
expectations.
CampaignsarenotjustanotherFortune500companyortop50website.Theyhavetheirownculture
anddemands,strangerigorsandschedules.Thedeadlinesarehardandthepressurewouldbeenough
topressthetshirtofeventhemostbattletestedstartupveteran.
ToreallyunderstandwhathappenedbehindthescenesattheObamacampaign,youneedtoknowa
littlebitaboutitsorganizationalstructure.TechwasHarperReed'sdomain."Digital"wasJoe
Rospars'kingdomhisteamwascomposedofthepeoplewhosentyouallthoseemails,designed
someoftheconsumerfacingpiecesofBarackObama.com,andranthecampaigns'mostexcellent
accountsonFacebook,Twitter,Tumblr,video,andthelike.AnalyticswasrunbyDanWagner,and
thoseguyswereresponsibleforcomingupwithwaysoffindingandtargetingvoterstheycould
persuadeorturnout.JeremyBirdranField,theonthegroundoperationsoforganizingvotersatthe
communitylevelthatmanyconsiderObama'ssecretsauce.Thetechforthecampaignwassupposed
tohelptheField,Analytics,andDigitalteamsdotheirjobsbetter.Tech,inacampaignoratleastthis
campaignorperhapsanysuccessfulcampaign,hastoplayasupportingrole.Thegoalwasnottobuild
aproduct.ThegoalwastoreelectthePresident.AsReedputit,ifthecampaignwereMoneyball,he
wouldn'tbeBillyBeane,he'dbe"GoogleBoy."
There'soneotherinterestingcomponenttothecampaign'sstructure.Andthat'sthepresenceoftwo
bigtechvendorsinterfacingwiththevariousteamsBlueStateDigitalandNGPVan.Themost
obviousisthefirmthatRospars,JaschaFranklinHodge,andClayJohnsoncofounded,BlueState
Digital.They'rethepreeminentprogressivedigitalagency,andadecentchunkmaybe30percent
oftheirbusinesscomesfromprovidingtechnologytocampaigns.Ofcourse,BSD'sbiggestclientwas
theObamacampaignandhasbeenforsometime.BSDandObamaforAmericawereandareso
deeplyenmeshed,itwouldbedifficulttosaywhereoneendedandtheotherbegan.Afterall,both
GoffandRospars,thecompany'sprincipals,werepaidstaffersoftheObamacampaign.Andyet
between2008and2012,BSDwaspurchasedbyWPP,oneofthelargestadagenciesintheworld.
Whathadbeenanobviouslyprogressiveorganizationwasnowownedbyahugeconglomerateand
hadclientsthatweren'totherDemocraticpoliticians.
OneotherthingtoknowaboutRospars,specifically:"He'stheKarlRoveoftheInternet,"someone
whoknowshimverywelltoldme.WhatRovewastodirectmailtheundisputedkingofthe

www.theatlantic.com/technology/print/2012/11/when-the-nerds-go-marching-in/265325/

7/13

11/20/12

When the Nerds Go Marching In - Technology - The Atlantic

mediumRosparsistoemail.HeandGoffarethebrainsbehindObama'sunprecedentedonline
fundraisingefforts.Theyknowwhattheyweredoingandhadproventhattimeandagain.
ThecomplexrelationshipbetweenBSDandtheObamacampaignaddsanotherdimensiontothe
introductionofaninsideteamoftechnologists.Ifallcampaignsstartedbringingtheirtechnologyin
house,perhapsBSD'stechbusinesswouldbegintoseemlessattractive,particularlyifmanyofthe
toolsthatsuchaninsideteamcreatedweredevelopedasopensourceproducts.
So,perhapsthetechteamwasboundtobuttheadswithRospars'digitalsquad.Slabywouldnote,too,
thattheorganizationalstylesofthetwooperationswereverydifferent."Campaignsaren't
traditionallythatcollaborative,"hesaid."Departmentstendtobefreestanding.Theyareorganized
kindoflikedisasterresponsesiloedandsuperhierarchicalsothatthingscanmoveveryquickly."
Lookingatitallfromtheoutside,boththedigitalandtechteamshadreallygood,missionoriented
reasonsforwantingtheirwaytocarrytheday.Thetechteamcouldsay,"Hey,we'vedonethiskindof
techbeforeatlargerscaleandwithmorestabilitythanyou'veeverhad.Letusdothis."Andthe
digitalteamcouldsay,"Yeah,well,weelectedthepresidentandweknowhowtowin,regardlessof
thetechnologystack.Justmakewhatweaskfor."
Thewaythattheconflictplayedoutwasoverthingsliketheuserexperienceonthewebsite.Jason
KuneshwasthedirectorofUXforthetechteam.Hehadmanyyearsofconsultingunderhisbeltfor
bigandsmallcompanieslikeMicrosoftandLeapFrog.He,too,fromanindustryperspectiveknew
whathewasdoing.So,heransomeuserinterrupttestsonthewebsitetodeterminehowpeoplewere
experiencingwww.barackobama.com.Whathefoundwasthatthewebsitewasn'teventryingtomake
agoatpersuadingvoters.Rather,everyonegotfunneledintothefundraising"trap."Whenheraised
thatissuewithGoffandRospars,hegotaresponsethatIimaginewassomethinglike,"Duh.Now
STFU,"butperhapsinmorewords.AndfromtheGoff/Rosparsperspective,thinkaboutit:the
systemthey'ddevelopedcouldraise$3million*fromasingleemail.*Thesortsofmovestheyhad
learnedhowtomakehadmadeadifferenceoftens,ifnothundredsofmillionsofdollars.Whywas
thisKuneshguycomingaroundtryingtotellthemhowtorunacampaign?
FromKunesh'sperspective,though,therewasnoreasontothinkthatyouhadtorunthiscampaign
thesameasyoudidthelastone.Theoutsiderstatusthattheteambothadoptedandhadappliedto
themgavethemtherighttoquestionpreviouspractices.
TechsometimeshaddifficultybuildingwhattheFieldteam,ahallowedgroupwithinthecampaign's
world,wanted.MostofthatrelatedtothewaythattheylaunchedDashboard,theonlineoutreach
tool.IfyoulookatDashboardattheendofthecampaign,youseeabeautifullypolishedproductthat
letyouvolunteeranywayyouwanted.It'ssecureandintuitiveandhadtremendouslygooduptimeas
thecampaigndrewtoaclose.
Butthatwasn'thowthefirstversionofDashboardlooked.
Thetechteam'splanwastorolloutversion1withalimitedfeatureset,iterate,rolloutversion2,
iterate,andsoonandsoforthuntilthesoftwarewascompleteandbulletproof.PerKunesh'stelling,
theFieldpeoplewereusedtosoftwarethatlookedcompletebutthatwasunreliableunderthehood.
Itlookedasifyoucoulddothethingsyouneededtodo,butthesoftwarewouldkeepfallingdownand
gettingpatched,fallingdownandgettingpatched,allthewaythroughacampaign.Thetechteamdid
notwantthat.Theymightbeslower,buttheyweregoingtobuildsolidproducts.

www.theatlantic.com/technology/print/2012/11/when-the-nerds-go-marching-in/265325/

8/13

11/20/12

When the Nerds Go Marching In - Technology - The Atlantic

Reed'steambegantotrickleintoChicagobeginninginMayof2011.Theypromised,over
optimistically,thatthey'dreleaseaversionofDashboardjustafewmonthsaftertheteamarrived.
Thefirstversionwasnotimpressive."August29,2011,mybirthday,weweresupposedtohavea
prototypeoutofDashboard,thatwasgoingtobethepubliclaunch,"Kuneshtoldme."Itwasfreaking
horrible,youcouldn'tshowittoanyone.ButI'donlybeenthere13weeksandmostoftheteamhad
beentherehalfthattime."
Asthetechteamstruggledtotranslatewhatpeoplewantedintousablesoftware,trustinthetech
teamalreadyshakykepteroding.ByFebruraryof2012,Kuneshstartedtogetwordthatpeople
onboththedigitalandfieldteamshadagitatedtopulltheplugonDashboardandreplacethetech
teamwithsomebody,anybody,else.
"Alotofthesoftwareiskindoflate.It'slookinguglyandIgooutonthisFieldcall,"Kunesh
remembered."Andpeoplearelike,'Man,weshouldfireyourbossesman...Wegottagettheguys
fromtheDNC.Theydon'tknowwhatthehellyou'redoing.'I'msittingtheregoing,'I'mgonnaget
anothermargarita.'"
Whiletheresponsibilityfortheirearlystrugglescertainlyfallstothetechteam,therewere
mitigatingfactors.Forone,noonehadeverdonewhattheywereattemptingtodo.Narwhalhadto
connecttoabunchofdifferentvendors'software,someofwhichturnedouttobesurprisinglyarcane
anddifficult.Notonlythat,butthereweredifferencesinthewayfieldofficesinsomestatesdidthings
andhowcampaignHQthoughttheydidthings.Techwastedtimebuildingthingsthatitturnedout
peopledidn'tneedorwant.
"Inthemovieversionofthecampaign,there'sprobablyameetingwhereI'mabouttogetfiredandI
throwmyselfonthetable,"Slabytoldme.Butinreality,whatactuallyhappenedwasObama's
campaignchiefJimMessinawouldcomebySlaby'sdeskandtellhim,"Dude,thishastowork."And
Slabywouldrespond,"Iknow.Itwill,"andthengobacktowork.
Infact,someshakeupswerenecessary.ReedandSlabysentsomeproductmanagerspackingand
broughtinmoretraditionaloneslikeformerMicrosoftPMCarolDavidsen."Youverymuchhaveto
understandthecampaign'shiringstrategy:'We'llhiretheseproductmanagerswhohavecampaign
experience,thenhireengineerswhohavetechnicalexperienceandthesetwoworldswillmagically
cometogether.'Thatfailed,"Davidsensaid."Thosetwogroupsofpeoplecouldn'ttalktoeachother."
Then,inthelatespring,alltheproductsthatthetechteamhadbeenpromisingstartedtoshowup.
Dashboardgotsolid.Youdidn'thavetologinabunchoftimesifyouwantedtododifferentthingson
thewebsite.Othersmallerproductsrolledout."Thestuffwetoldyouaboutforayearisactually
happening,"KuneshrecalledtellingtheFieldteam."You'regoingtohaveoneloginandhaveallthese
toolsandit'salljustgonnawork."
Perhapsmostimportantly,Narwhalreallygotontrack,thanksnodoubttoDavidsen'seffortsaswell
asJoshThayer's,theleadengineerwhoarrivedinApril.WhatNarwhalfixedwasaproblemthat's
longplaguedcampaigns.Youhaveallthisdatacominginfromalltheseplacesthevoterfile,
variousfieldoffices,theanalyticspeople,thewebsite,mobilestuff.In2008,andallpreviousraces,
thenumberschangedonceaday.Itwasn'trealtime.Andthepeoplelookingtohittheirnumbersin
variouswaysoutinthefieldofficesnumberofvolunteersanddollarsraisedandvoterspersuaded
wereusedtoseeingthatupdatehappenlikethat.
Butfromaninfrastructurelevel,howmuchbetterwoulditbeifyoucouldsyncthatdatainrealtime

www.theatlantic.com/technology/print/2012/11/when-the-nerds-go-marching-in/265325/

9/13

11/20/12

When the Nerds Go Marching In - Technology - The Atlantic

acrosstheentirecampaign?That'swhatNarwhalwassupposedtodo.Davidsen,intrueproduct
managerform,brokedownpreciselyhowitallworked.First,shesaid,Narwhalwasn'treallyone
thing,butseveral.Narwhalwasjustaninitiallyhelpfulbrandforthebundleofsoftware.
Inreality,ithadthreecomponents."Oneisvendorintegration:BSD,NGP,VAN[thelattertwo
companiesmergedin2010].Justgettingallofthatdataintothesystemandgettingitinrealtimeas
soonasitgoesinonesystemtoanother,"shesaid."ThesecondpartisanAPIportion.Youdon'twant
amillionconsumersgettingdataviaSQL."TheAPIallowedpeopletoaccesspartsofthedatawithout
lettingthemgetattheSQLdatabaseonthebackend.ItprovidedasafewayforDashboard,theCall
Tool(whichhelpedpeoplemakecalls),andtheTwitterBlastertopulldata.Andthelastpartwasthe
presentationofthedatathatwasinthesystem.Whilethedreamhadbeenforallapplicationstorun
throughNarwhalinrealtime,itturnedoutthatcouldn'twork.So,theysplitthingsintorealtime
applicationsliketheCallToolorthingsontheweb.Andthentheyprovidedaseparatewayforthe
Analyticspeople,whohadveryspecificneeds,togetthedatainadifferentform.Then,whateverthey
cameupwithwasfedbackintoNarwhal.
Bytheend,Davidsenthoughtalltheteams'relationshipshadimproved,evenbeforeObama'sbigwin.
ShecreditedaweeklyWednesdaydrinkingandhangingoutsessionthatbroughttogetherallthe
variouspeopleworkingonthecampaign'stechnology.Bytheveryend,somefrontenddesignerswho
weretechnicallyonthedigitalteamhadembeddedwiththetechsquadtogetworkdonefaster.Tech
mightnothavebeenfullyintegrated,butitwasfullyoperational.Highfiveswereintheair.
Slaby,withtypicalpragmatism,putitlikethis."Oursupportersdon'tgiveashitaboutourorgchart.
Theyjustwanttohaveameaningfulexperience.Wepromisethemtheycanplayameaningfulrolein
politicsandtheydon'tcareaboutthedepartmentsinthecampaign.Sowehavetodotheworkonour
sidetolookintegratedandhaveourshittogether,"hesaid."Thattooksometime.Youhaveto
developnewtrustwithpeople.It'sjustchangemanagement.It'snotcomplicatedit'sjusthard."
WHATTHEYACTUALLYBUILT

www.theatlantic.com/technology/print/2012/11/when-the-nerds-go-marching-in/265325/

10/13

11/20/12

When the Nerds Go Marching In - Technology - The Atlantic

Ofcourse,thetechdidn'texistforitsownsake.Itwasmeanttobeusedbytheorganizersinthefield
andtheanalystsinthelab.Let'sjustrunthroughsomeofthethingsthatactuallygotaccomplishedby
thetech,digital,andanalyticsteamsbeyondofNarwhalandDashboard.
Theycreatedthemostsophisticatedemailfundraisingprogramever.Thedigitalteam,underRospars
leadership,tooktheirdatadrivenstrategytoanewlevel.Anytimeyoureceivedanemailfromthe
Obamacampaign,ithadbeentestedon18smallergroupsandtheresponserateshadbeengauged.
Thecampaignthoughtallthelettershadagoodchanceofsucceeding,buttheworstperforming
lettersdidonly15to20percentofwhatthebestperformingemailscoulddeliver.So,ifagood
performercoulddo$2.5million,apoorperformermightonlynet$500,000.Thegeniusofthe
campaignwasthatitlearnedtostopsendingpoorperformers.
ObamabecamethefirstpresidentialcandidatetoappearonReddit,themassivepopularsocial
networkingsite.Andyes,hereallydidtypeinhisownanswerswithGoffathisside.Onefascinating
outcomeoftheAMAisthat30,000RedditorsregisteredtovoteafterPresidentdroppedinalinkto
theObamavoterregistrationpage.Oh,andthecampaignalsoofficiallyhasthemosttweetedtweet
andthemostpopularFacebookpost.Notbad.IwouldalsonotethatLauraOlin,aformerstrategist
atBlueStateDigitalwhomovedtotheObamacampaign,ranthebestcampaignTumblrtheworld
willprobablyeversee.
WithDavidsen'shelp,theAnalyticsteambuiltatooltheycalledTheOptimizer,whichallowedthe
campaigntobuyeyeballsontelevisionmorecheaply.Theytooksettopbox(thatistosay,yourcable
orsatelliteboxorDVR)datafromDavidsen'soldstartup,NavikNetworks,andcorrelateditwiththe
campaign'sowndata.ThisoccurredthroughathirdpartycalledEpsilon:thecampaignsentitsvoter
fileandthetelevisionprovidersenttheirbillingfileandboom,alistcamebackofpeoplewhohad
donecertainthingslike,forexample,watchedthefirstpresidentialdebate.Havingthatdataallowed
thecampaigntobuyadsthattheyknewwouldgetinfrontofthemostoftheirpeopleattheleastcost.
Theydidn'thavetobuythetraditionalstufflikethelocalnews,either.Instead,theycouldrunads
targetedtospecifictypesofvotersduringrerunsoroffpeakhours.
AccordingtoCMAG/Kantar,theObama'scampaign'scostperadwaslower($594)thantheRomney
campaign($666)oranyothermajorbuyerinthecampaigncycle.Thatdifferencemaynotsound
impressive,buttheObamacampaignitselfairedmorethan550thousandads.Anditwasn'tjustabout
cost,either.TheycouldseethatsomehouseholdswereonlywatchingacouplehoursofTVadayand
mightbewillingtospendmoretogetinfrontofthosehardertoreachpeople.
GoffdescribedtheFacebooktoolas"themostsignificantnewadditiontothevotercontact
arsenalthat'scomearoundinyears,sincethephonecall."
Thedigital,tech,andanalyticsteamsworkedtobuildTwitterandFacebookBlasters.Theyranona
servicethatgeneratedmicrotargetingdatathatwasbuiltbyWillSt.Clair.ItwascodenamedTrgus
Taargsforsomereason.WithTwitter,oneofthecompany'sformeremployees,MarkTrammell,
helpedbuildatoolthatcouldspecificallysendindividualusersdirectmessages."Webuiltan
influencescoreforthepeoplefollowingthe[ObamaforAmerica]accountsandthencrossreferenced
thoseforspecificthingsweweretryingtotarget,battlegroundstates,thatsortofstuff."Meanwhile,
theteamsalsobuiltanoptinFacebookoutreachprogramthatsentpeoplemessagessaying,
essentially,"Yourfriend,DaveinOhio,hasn'tvotedyet.Gotellhimtovote."Goffdescribedthe
Facebooktoolas"themostsignificantnewadditiontothevotercontactarsenalthat'scomearoundin

www.theatlantic.com/technology/print/2012/11/when-the-nerds-go-marching-in/265325/

11/13

11/20/12

When the Nerds Go Marching In - Technology - The Atlantic

years,sincethephonecall."
Lastbutcertainlynotleast,youhavethedigitalteam'sQuickDonate.Itessentiallybroughttheease
ofAmazon'soneclickpurchasestopoliticaldonations."It'stheabsoluteepitomeofhowyoucan
makeiteasyforpeopletogivemoneyonline,"Goffsaid."Intermsoffundraising,that'sasinnovative
asweneededtobe."Storingpeople'spaymentinformationalsoletthecampaignreceivedonations
viatextmessageslongbeforetheFederalElectionsCommissionapprovedanofficialwayofdoingso.
Theycouldsimplytextpeoplewho'doptedinasimplemessagelike,"Textbackwithhowmuch
moneyyou'dliketodonate."Sometimespeopletextedmuchlargerdollaramountsbackthanthe$10
incrementsthatmobilecarriersallow.
It'sanimpressivearrayofaccomplishments.Whatyoucandowithdataandcodejustkeeps
advancing."Afterthelastcampaign,IgotintroducedastheCTOofthemosttechnicallyadvanced
campaignever,"Slabysaid."Butthat'strueofeveryCTOofeverycampaigneverytime."Or,rather,
it'strueofonecampaignCTOeverytime.
EXITMUSIC

HarperReedaftertheelection(PhotobyDanielX.O'Neil).
ThatnextmosttechnicallyadvancedCTO,in2016,willnotbeHarperReed.Afewdaysafterthe
election,sittingwithhiswifeatWickerPark'sHandlebar,eatingfishtacos,anddrinkingaDaisy
Cutterpaleale,Reedlookshappy.He'dtoldmeearlierinthedaythathe'dneverexperiencedstress
untiltheObamacampaign,andIbelievehim.
Heregaleduswithstoriesabouthisoldperformancetroupe,JugglersAgainstHomophobia,wild
clubbingandDJs."Itwasthiswholeworldofhavingfunandlivinginthemoment,"Reedsaid."And
therewasalotofdoingthatontheInternet."
"Ispentalotoftimehackingdoingallthisstuff,buildingwebsites,buildingcommunities,workingall
thetime,"Reedsaid,"andthenalotoftimedrinking,partying,andhangingout.AndIhadtochoose

www.theatlantic.com/technology/print/2012/11/when-the-nerds-go-marching-in/265325/

12/13

11/20/12

When the Nerds Go Marching In - Technology - The Atlantic

whentodowhich."
WeleftHandlebarandmadeaquickpitstopatthecoffeeshop,Wormhole,wherehefirstmetSlaby.
Reedcracksthatit'slikeRedditcometolife.Bothofthemrememberthemeetingthesameway:
Slabyplayingtheroleofsquare,Reedplayingtheroleofhipster.Andtwominuteslater,theywere
readytoworktogether.Whatbegan18monthsagointhatveryspotwasfinallycomingtoanend.
ReedcouldstopbeingObamaforAmerica'sCTOandreturntobeing"HarperReed,probablyoneof
thecoolestguysever,"ashispersonalwebpageistitled.
Butofcourse,heandhiswholeteamofnerdswerechangedbytheexperience.Theylearnedwhatit
wasliketohaveandworkwithpeoplewhohadahigherpurposethanbuildingcoolstuff."Teddy
[Goff]wouldtearuptalkingaboutthePresident.Iwouldbelike,'Yeah,thatguy'scool,'"Reedsaid.
"Itwasonlytowardstheend,themiddleof2012,whenwerealizedthegravityofwhatwewere
doing."
PartofthatprocesswasReed,atechnologist'stechnologist,learningthelimitsofhisownpower."I
rememberatonepointbasicallybreakingdownduringthecampaignbecauseIwaslosingcontrol.
Thesuccessofitwasoutofmyhands,"hetoldme."IfeltlikethepeopleIhiredwereright,the
resourceswearguedforwereright.Andbecauseofastupidmistake,orpeoplewerescaredandthey
didn'tadoptthetechnologyorwhatever,somethingcouldgoawry.Wecouldlose."
Andlosing,theyfeltmoreandmoredeeplyasthecampaignwenton,wouldmeanhorriblethingsfor
thecountry.TheystartedtoworryaboutthenextSupremeCourtJusticeswhiletheycoded.
"Thereistheegoismoftechnologists.Wedoitbecausewecancreate.Icanhandleallofthe
parametersgoingintothemachineandIknowwhatisgoingtocomeoutofit,"Reedsaid."Inthis,the
controlweallenjoyedabouttechnologywasgone."
Wefinishedourdrinks,readyforwhatwasalmostcertainlygoingtobealongnight,andheadedto
ourfirstclub.ThelastthingmyrecorderpickedupoverthebasswasmesayingtoHarper,"Ijustsaw
someonebuyHennessy.I'veneverseensomeonebuyHennessy."Then,allIcanhearisthatmusic.
Thisarticleavailableonlineat:
http://www.theatlantic.com/technology/archive/2012/11/whenthenerdsgomarchingin/265325/
Copyright2012byTheAtlanticMonthlyGroup.AllRightsReserved.

www.theatlantic.com/technology/print/2012/11/when-the-nerds-go-marching-in/265325/

13/13