20TerabytesaNight

DesigningtheLargeSynopticSurveyTelescopesImageProcessingPipelines

Contents

Foreword: 1
GeoffSparks,CEOSparxSystemsPtyLtd
Prologue: 2
LongAgo(andSomeGalaxiesFarAway)

Chapter1: TheLargeBinocularTelescope 8
JumpStartingtheLBTSoftware 8
StretchingICONIXProcess:LBTsObservatoryControlSystem 11
Chapter2: WhatsaLargeSynopticSurveyTelescope(andwhydoweneedone)? 17

Chapter3: DataChallenges:From0to20Terabytesanight 23

Chapter4: TailoringICONIXProcessforAlgorithmDevelopment 27

Chapter5: HowDoYouDetectanAsteroidThatMightHittheEarth? 38

"Anyopinions,findings,andconclusionsorrecommendationsexpressedinthismaterialarethoseof
theauthor(s)anddonotnecessarilyreflecttheviewsoftheNationalScienceFoundation,LSST
Corporation,oranyoneelse."

Foreword
GeoffSparks,SparxSystemsCEO

Since 2002, Sparx Systems has benefitted by having ICONIX as a member of its Global
Partner Program. ICONIX CEO Doug Rosenberg is a luminary in the field of Software
Engineering and noted author. Over the years he has contributed valuable expertise,
helping to ensure Enterprise Architect provides world-leading support for the ICONIX
process.

At Sparx, we have enjoyed the hands-on experience Doug has related to us from years of
successfully applying the ICONIX process to projects in industry. Wed like to share some of
these insights with the broader Enterprise Architect community.

In this e-Book, we asked Doug to distill his experiences and lessons-learned from the Large
Synoptic Survey Telescope (LSST) project. The sheer size and complex nature of LSST,
bring a unique set of challenges and a massive software modeling endeavor. However, the
unchanging principles behind use of model abstractions, UML and the ICONIX process
remain beneficial in such an undertaking, as highlighted throughout this account.

We hope you also enjoy and benefit from Dougs shared experience on the amazing LSST
project!

2

Prologue
LongAgo(andSomeGalaxiesFarAway)

Beforewegetstarted,heresashortsummaryofhowIcametobeinvolvedwiththeLargeSynoptic
SurveyTelescope(LSST),andanintroductiontoacoupleofthekeyplayersontheLSSTteam(and
goodfriendsofmine),TimAxelrodandJeffKantor.
NASAJPL:TheBirthplaceofImageProcessing
IgraduatedfromtheUniversityofSouthernCaliforniain1980withadegreeinElectrical
Engineeringandtheabilitytoprogramcomputersin12or13differentlanguagesandhaving
takenonlyoneAstronomycourse(whichIenjoyedquitealot).Ibouncedaroundbetweenacouple
ofaerospacecompaniesinSouthernCaliforniaandaVLSICADcompanyinSiliconValleyforafew
years,anddiscoveredthat
5%ofthepeopleinhightechkneweverything,anddid95%ofthework
Ihadabsolutelynostomachforcompanypolitics
Badtechnicaldecisionsmadeforpoliticalreasonsonmyprojectskeptmeupatnight
Consultants/contractprogrammersmadedoublethesalaryofregularemployees
Ihadtheabilitytocreatesoftwareofsignificantvalue
Giventhesediscoveries,itseemedtomakesensetobecomeacontractprogrammer,doublemy
salary,andinvesttheextramoneyinstartingmyownbusinesstocreatesoftware(andonlyhire
top5%overachievers).IttookmeacoupleofyearstofigureoutwhatkindofsoftwareIwanted
tocreate,andIfinallysettledondevelopingbettertoolsforprogrammers.
So25yearsago(1984),IfoundmyselfasacontractprogrammerattheNASAJetPropulsion
Laboratory(JPL)workinginalabcalledtheMultiMissionImageProcessingLaboratory.
1
This
happenedtobethelabthatwasprocessingthephotosreceivedbyVoyagerasitpassedJupiter,
Saturn,Uranus,andNeptune.Iwasntpersonallydoingimageprocessing,Iwasworkingona
commandandcontrolsystemthatdidsomethingcalledTacticalDataFusion,wherewewouldtake
allsortsofdifferentinformation,fuseittogether,anddisplayitonamap.ButIwassurroundedby
folkswhoweredoingrealimageprocessingandIalwaysfoundittobeinterestingstuff.Plusthe
giantphotoofJupitersRedSpot
2
onthewalloftheofficewhereIworkedwasprettycool.Its
possiblethatsomebody,somewherewasdoingimageprocessingbeforeJPL,buttheystarteddoing
itin1966,soMIPLwascertainlyoneoftheplaceswheretheimageprocessingtechniquesnow
beingusedonLSSTweredeveloped.
Iworkedfour10hourdaysaweekatJPL,andspenttherestofmytimestartingICONIX.Ihad
boughtaLisa2/10computer(thepredecessortotheMacintosh,whichcameoutin1984)thathad
a32bitprocessor,2MegabytesofRAM,anda10Megabyteharddisk,whichwasalotofcomputer

1
http://www-mipl.jpl.nasa.gov/
2
http://photojournal.jpl.nasa.gov/catalog/PIA02259
3
for$10,000backthen.OurdepartmentVAX11/780minicomputersupported16concurrentusers
onsomethinglikeasinglemegabyteofRAM.Bycontrast,thetopicofthisbookisanimage
processingsystemthatwillprocess20Terabytesofdataeverynightforadecade.
NASAJohnsonSpaceStationSSE
ICONIXchangedfrombeingapipedreamtoarealbusinessin198687afterImetJeffKantorata
conferenceinSeattlecalledtheStructuredDevelopmentForum(OOmethodologieshadntbeen
inventedyet).JeffwasworkingnearNASAJohnsoninHouston,definingthecommonSoftware
SupportEnvironment(SSE)fortheSpaceStation.
3

JeffwantedanoptionfordeveloperstouseMacintoshcomputers,andICONIXwasjustaboutthe
onlygameintown.WeopenedanofficeafterJeffbought88licensesofourMacCASEtools(called
ICONIXPowerTools),andICONIXbecamearealcompany.JeffisnowtheLSSTDataManagement
ProjectManager,andakeyplayerinthisstory.
NASAGoddardHubbleRepairProject
AquickcheckoftheNASAwebsiteshowsthatthefirstservicingmissiontotheHubbleSpace
TelescopewasflowninDecember1993(anotherservicingmissionisabouttobeflownasIwrite
this
4
),whichmeansthatitwassometimein1992whenIfoundmyselfinGreenbelt,Marylandat
theNASAGoddardSpaceFlightCenter,teachingaclassonStructuredAnalysisandDesigntothe
teamthatwasrehostingthecoprocessorsoftware.
ManypeopleareawarethatwhentheHubblewasfirstbuilt,therewasaproblemwiththe
curvatureofthemainmirror(itwasoffbysomethinglikethe1/50
th
thewidthofahumanhair)that
requiredcorrectivelensestobeinstalled.Alesserknownfactisthattheonboardcoprocessorsof
theHubble,originallysomesortofproprietarychip,werefailingatanalarmingratedueto
radiationdamage,andpartoftherepairmissionwastoreplacethemwithradiationhardchips(I
believetheywereIntel386processors).Thecoprocessorsoftware
5
didthingslikepointthesolar
panelsatthesun.Soallofthesoftwareneededtoberehosted.TheHubbleRepairprojectwasmy
firstexperiencewithlargetelescopes,andIgotacoolpostertoputupinmyoffice,nexttothe
SpaceStationposter.
ICONIX:PuttingtheUinUML
ICONIXspentabout10yearsintheCASEtoolbusiness,andalongthewaydevelopedoneofthe
firstObjectOrientedAnalysisandDesign(OOAD)tools,whichwecalledObjectModeler.JeffKantor
hadlefttheSpaceStationprogramandworkedwithmeatICONIXforawhile.Oneofthethingshe
didwasanalyzetheemergingplethoraofOOmethodologybooks,lookingforcommonalityand
figuringoutwhichofthesemethodologieswewantedtosupportinObjectModeler.Wecameup
withBooch,Rumbaugh,JacobsonandCoad/Yourdon,whichofcourseincludesthe3methodologies
thatwentintoUML.WedidthisseveralyearsbeforeBooch,Rumbaugh,andJacobsongottogether
tocreateUML,whichhappenedacoupleofyearsafterIpublishedaCDROMcalledAUnified
ObjectModelingApproach.SoIliketothinkthatJeffandIputtheUinUML.
AfterUMLcameout,itbecamecleartomethatICONIXasatoolvendorwasntlikelytoremain
competitiveforverylong.ButIhaddevelopedaninterestingtrainingcoursethattaughtpeople
howtouseBooch,Rumbaugh,andJacobsonmethodstogether,andwiththeadventofUML,that
classbecamemarketable.SoICONIXbecameatrainingcompany,focusingonourJumpStart

3
http://www.nasa.gov/mission_pages/station/main/index.html
4
http://www.nasa.gov/mission_pages/shuttle/shuttlemissions/hst_sm4/index.html
5
http://hubble.nasa.gov/a_pdf/news/facts/CoProcessor.pdf
4
approachtostartingclientprojectsusingourlightweightunifiedUMLprocess.Ialsostarted
writingbooks,initiallyUseCaseDrivenObjectModelingAPracticalApproach,withKendallScott,
whichbecameprettypopular.
StewardObservatoryTheLargeBinocularTelescope
Fastforwarding8or10yearsandanotherbookwritten,IreceivedaphonecallonedayfromTim
AxelrodattheUniversityofArizonaStewardObservatory.Tim,inhisquiet,softspokenway,said
thathehadreadmybookandwashopingthatImightbeabletopopouttoTucsonandrunoneof
myJumpStartworkshopsbecausesomebodyhasbuiltaverybigtelescope(theLargeBinocular
Telescope
6
)andspent10yearsworkingonthehardwareandcompletelyforgotaboutthe
software,andIvejustbeenputinchargeofgettingthesoftwarebuilt.TimisnowtheProject
ScientistforLSSTDataManagement.

Asithappened,theLBTclasswasthefirstoccasionIhadtouseEnterpriseArchitect.Ifiguredout
howtouseitonthe(veryshort)flightfromLosAngelestoTucson.Welltellthewholestoryin
Chapter1,buttomakealongstoryshort,theSparxSystemssoftwareworkedlikeachampand
solvedthesharedmodelissuesthatthe(then)industrystandardtoolsignored.
AsaresultofthepositiveexperiencewehadwithLBT,ICONIXjoinedtheSparxSystemspartner
programimmediatelyafterIreturnedfromTucson.Ourinitialprojectwastoproduceamultimedia
tutorialtitledMasteringUMLwithEnterpriseArchitectandICONIXProcess,followedbyEnterprise
ArchitectforPowerUsersandwerapidlyswitchedourfocustowardstrainingEnterpriseArchitect
users.DuringthistimeIwasalsowritingExtremeProgrammingRefactored
7
,thefirstofseveral
booksthatIvewrittenwithMattStephens
8
.
Itwasduringthe5dayclassforLBT,wherewemodeledtheObservatoryControlSystem(OCS),
thatmywholeperspectiveabouttelescopeschanged.Atthetime,mysons8thgradescienceclass
wasgrindingan8inchmirrorbyhand,andbuildingatelescopesothatwasmyframeof
referencewhenIheadedforTucsonthefirsttime.LBThastwinprimarymirrors(henceBinocular)
andtheyareeach8.4metersindiameter.BycomparisontheHubblehasa2.4meterprimary
mirror,andthebigHaletelescopeatthePalomarObservatory
9
,whichwastheworldslargestfor
45years,hasa5.1meter(200inch)mirror.
Dependingonwhoyoutalkto,LBTiseitherthelargestopticaltelescopeontheplanet,oroneof
thetop2or3inanyevent,itsBIG
10 11
.TheKeckObservatory onMaunaKeahasa10meter
mirror,butitsmadeinsections.OntheotherhandLBTbeingabinoculartelescopemeansitstwin
primarymirrorsareworkingtogether,soIllleavethatdebatetotheastrophysicists.
Duringtheweek,TimarrangedformetohavealunchtimetouroftheMirrorLabatSteward.
Seeing8.4metersonapagedoesntreallyconveythescaleofthesemirrors.Eachmirrorweighs
20tons.TheMirrorLab
12
(whichisunderthefootballstadiumattheUniversityofArizona)hasan
oventhatmelts20tonsofglassinan8.4metermold,andspinsituntilthemoltenglassformsa
parabolicshape,thentheycoolitdown.Thissavesalotofgrindingtimeanditsaprettyunique
facility.OneoftheLBTprimarymirrorswasbeingpolishedwhenIwasthereandIgottocrawl

6
http://lbto.org
7
http://www.softwarereality.com/ExtremeProgrammingRefactored.jsp
8
http://www.softwarereality.com/MattStephens.jsp
9
http://www.astro.caltech.edu/palomar/hale.html
10
See http://medusa.as.arizona.edu/lbto/observatory_images.htm to get a sense of LBT.
11
http://keckobservatory.org/index.php/about/telescopes/
12
http://mirrorlab.as.arizona.edu/index.php
5
aroundunderneathitandlookatitupclose.WhenIreturnedtotheclass,thoseusecasesthatsaid
RaisetheMirror,LowertheMirror,TrackanObjectetcsuddenlyseemedalotmoresignificant.It
dawnedonmethatgettingachancetocontribute(eveninasmallway)totheLBTsoftwarewasan
opportunitythatnotmanypeopleget,andtohavehadafingertipinboththeHubbleandLBT
softwarewasreallyquiteamazing.

Figure1TimAxelrodpointsoutafeatureofoneofLBTstwinprimarymirrorstoJeffKantorononeofmytripstoMount
Graham.Thescaleisabitdeceptive,wewereseveralfloorsup.Youcanbetterappreciatethesizeofthesemirrorsby
noticingthattherearetwopeopledownbythewhiterailingnexttothemirror.
ThankstoTim,IwasfortunateenoughtomaketwotripstoMountGraham,thefirstwhenthefirst
mirrorhadbeeninstalledandthesecondtimeafterbothmirrorswereuponthemountainand
theywerepreparingtocommissionthetelescope.Thesecondtime,theyhadtheObservatory
ControlSystemupandrunning,andLBTisnowobservinggalaxiesover100lightyearsaway
13
.
TheFirstThingINeedtodoisHireaReallyGoodProjectManager
AfewyearsaftertheclasswedidfortheLBTOCS,TimandIspokeagainandhetoldmehehadleft
LBTandwasnowProjectScientistforanothertelescope,calledLSST(LargeSynopticSurvey
Telescope).LSSThasasingleprimarymirror,also8.4metersindiameter
14
,anda3.2gigapixelCCD
camera
15
.However,unlikeLBT,LSSTisasurveytelescopeandwillcontinuouslysweeptheentire
skyratherthanfocusingononespotatatime.Continuouslysweepingtheskyforadecadewitha
camerathatcaptures3billionpixelsineachimageiswhatwilldrivetheunprecedentedvolumesof
imagedatathatLSSTwillproduce.SoyoumightsaythatimageprocessingwasbornatJPLand

13
http://medusa.as.arizona.edu/lbto/astronomical_images.htm
14
http://www.lsst.org/lsst/gallery/mirror-casting/Group_photo
15
http://www.lsst.org/lsst/gallery/camera
6
theyreperfectingitonLSST.
WhenIspokewithTim,hesaid:ThefirstthingIneedtodoishireareallygoodprojectmanager.I
knewthatJeffKantorwasworkingatacompanyinGeorgiawheretheyweregrindinghimhalfto
deathwithovertime,andthatheneededtogetoutofthatsituation.SoItoldTimthatheshould
meetmyfriendJeff.Theymet,andthatbringsustoourstartingpointforthisstory.
Lucas,MeetSpielberg
Beforewestart,Idliketoshareonemorestory.Someyearsago,JeffandIwereatabaseballgame
inChicago,atWrigleyField,andsomedrunkCubsfanspointedouttousthatJeffbearsastrong
resemblancetoStevenSpielberg(theydidthisbyshoutingHey,Spielberg!athimformostofthe
game).Afewyearslater,mysonRobobservedtomethatTimhasaresemblancetoGeorgeLucas.
SoitsalmostasifIintroducedLucastoSpielberg,andweallknowtheresultsofthatcollaboration.
Inallseriousness,Icantimaginetwomorequalifiedpeopletospearheadanefforttofigureout
howtoanalyze20Terabytesofimagedatapernight,anditcontinuestobemyprivilegetowork
withbothofthem.
7
Chapter1
TheLargeBinocularTelescope

Ivebeenreadingyourbook,saidthevoiceonthephone,andIwashopingyoumightbeableto
comeouttoTucsonandgiveusoneofyourtrainingworkshops.Ivejustbeenputinchargeofthe
softwareforalargetelescopeproject,theyvebeenworkingonthehardwareforabout10years
andcompletelyforgotaboutthesoftware,andnowIhavetogetitdoneinahurry.
JumpStartingtheLBTSoftware
That,ascloseasIcanrememberit,wasmyintroductiontoTimAxelrod.TimisasoftspokenPhD
astrophysicistfromCaltech,andhesresponsibleformyinvolvementinbothLBTandLSST.Hes
oneofthesmartestguysthatIknow,andIthinkweshareacommondistastefordogmatic
approachestosoftwaredevelopment(andfordogmaingeneral).
Thiswassometimeduring2002,andIwasinthemiddleofwritingmythirdbook(whichwasmy
firstonewithMattStephens),ExtremeProgrammingRefactored:TheCaseAgainstXP.Mattalso
sharesmydistastefordogma;XPRisverymuchamykarmaranoveryourdogmasortofbook.
Atthetime,ExtremeProgramming(asdogmaticasitgets)hadbecomequitethetrendythingtodo
intheindustry,andtheCASEtoolmarketwasdominatedbyexpensivetoolsthathadsome
significantissues.Thethingthatdisturbedmethemostaboutmodelingtoolsbackthenwasthe
lackofaconcurrent,multiuser,serverbasedrepository.Ialwaysfeltthatthis,combinedwitha
highpricepoint,wasasignificantimpedimenttotheadoptionofUMLmodelinginthefield,andin
away,addedabigsupportingargumenttoXPproponents,manyofwhomusedXPtojustifynot
designingtheirsoftwareupfrontordocumentinganything(andthenskippedthehardpartsofXP).
IhadheardofEnterpriseArchitect(EA)previously,becauseoneoftheirearlyadopterswasafanof
myfirstbook,andsuggestedtoGeoffSparksthathesupportrobustnessdiagramsinhissoftware,
andGeoff,whoisoneofthemostprolificbuildersofhighqualitysoftwarethatIveevermet,went
aheadanddidso.Ineffect,SparxSystemschangedthewholeprice/performanceequationinthe
industrywithEnterpriseArchitect,flippingthesituationfromhighprice/lowperformancetohigh
performance/lowprice.
HighPerformanceandLowPriceMakesaGoodCombination
Butbackin2002,IhadneverusedEnterpriseArchitectwhenIgotTimscall,andaspartofthe
preparationfortheJumpStartworkshop,hearrangedformetogetasoftwarelicenseandIrecall
figuringouthowtouseitontheshortflightfromLosAngelestoTucson.Itseemedprettyintuitive,
andmyplanstospendtheeveninggettingacquaintedwiththesoftwareprovedlargely
unnecessary.
ModelingTip:Goodtoolsmakeabigdifference
Dontsettleforanythinglessthanamodelingtoolthatsaffordable,easytouse,andsupports
concurrent,multiusermodeling.GoodtoolslikeEnterpriseArchitectmakeabigimpactonyour
project.

IwasinterestedintryingEnterpriseArchitectbecauseitseemedtoaddressmytwobiggestissues
withmodelingtoolsatthetime;pricepoint(atthattime,anEnterpriseArchitectlicensewas$99
anditsstillamazinglyaffordable)andanoutoftheboxmultiuserserverbasedrepository.But
8
whenTimtoldmeofhisplanstorunitonLinuxmachinesusingaWindowsemulator,andtokeep
therepositoryforthelabonanetworkedmachineinanotherbuilding(wewereintheSteward
ObservatoryontheUniversityofArizonacampus),Iwaslessthanenthused,becauseIthoughtwe
wererunningasignificantriskofJumpStartingtheprojectintoabrickwall.
APushintheRightDirection
JumpStartisthenameofourweeklongtrainingclassinICONIXProcesswhereweworkaclients
realprojectasthelabexample.ClientslikeTimhireuswhentheyneedtogetasoftwareproject
movinginahurry.Thisisatrickierprocessthanitmightfirstappear,becauseifwegettheproject
movinginthewrongdirection,itcreatesbigproblemsfortheclient,andforus.AtICONIX,were
investedinourclientssuccess.
OurJumpStartclassesareabout20%lecture(wetrytokeepitsimple)and80%lab,andthelabis
notatextbookexample,buttherealprojectandmostofthetimeacriticallyimportantprojectto
theclient.SoanythingthatputsthelabsessionatriskofnotgoingwellissomethingthatItryto
avoidliketheplague.IexplainedmyconcernsaboutthenetworkconfigurationtoTim,he
understood,andproposedthatwetryit,andifitprovedproblematicwedquicklyswitchtoplanB.
ModelingTip:NoteverythingisintheUMLspec
ThereareseveralreallyusefulextensionstoUMLthatmakeabigdifferencetoasuccessfulproject.
Forexample,Requirements,andScreensarenotpartoftheUML,butareessentialtoasuccessful
project.Easytousedocumentgenerationisalsoimportant.Reliabilityofamodelingtoolisalso
veryimportant.

Astheweekprogressed,Ibecameincreasinglyimpressedwiththecapabilityandreliabilityofthe
SparxSystemsEnterpriseArchitectsoftware.Itwaseasytouse,nevercrashedonceduringthe
entireweek,andhadlotsofusefulfeatureslikebuiltindocumentgenerationandextendedUML
withimportantthingslikeRequirementandScreenelements.
HavingspentadecadebuildingCASEtools,IknewaqualitymodelingtoolwhenIusedone,and
ICONIXjoinedtheSparxpartnerprogramimmediatelyaftermyreturnfromTucson.Thusbegana
longandfruitfulassociationwiththefolksatSparx,whocontinuetoimplementmyideasabout
improvingprocesswithinEnterpriseArchitect.
ICONIXandSparxSystemscontinuetocollaborateonextensionsto
UMLandEnterpriseArchitect
ICONIXhasdevelopedaprocessroadmapthatdetailsallthestepsofICONIXProcessonasetof
activitydiagrams,andamethodfordrivingtestcases(andJUnit/NUnittestcode)fromUML
models,calledDesignDrivenTesting(DDT).SparxSystemsprovidesexcellentsupportforthese
ICONIXextensionsinEnterpriseArchitect.DDTissupportedintheAgile/ICONIXaddinfromSparx
Systems.Synergybetweenprocessandtoolsisawonderfulthing.

9
TheresBig,andThenTheresBIG
ThehighlightofmyweekinTucsonwasaThursdayafternoontourthatTimarrangedatthe
StewardObservatoryMirrorLab.Whenwearrived,theywereintheprocessofpolishingthefirstof
LBTstwo20tonmirrors.

Figure1.LBTstwinprimarymirrorseachweigh20tons.
TheMirrorLabatStewardisactuallyunderthebleachersofthefootballstadiumontheUniversity
ofArizonacampus.ThishasalwaysremindedmeofEnricoFermisnuclearreactorunderthe
footballstadiumattheUniversityofChicago.IfyoueverfindyourselfinTucson,Ihighly
recommendtakingatouroftheMirrorLab.
16
Theyhavearotatingoventhatspins20tonsof
moltenglassintoaparabolicshapeinan8.4metermold.Youveprobablyneverseenonebefore.
SizeMatters
Whiletakingthistour,myjawdroppedopenwhenIrealizedthetruescaleofLBT.Ittakesabig
telescopetoimagegalaxiesmorethan100lightyearsaway(especiallythroughtheatmosphere,
whichrequirestheuseofadaptiveopticstocorrectfordistortion),andwiththatsizecomes
complexity.
Forexample,becauseglassisnotperfectlyrigid,whenyoutilta20tonmirrorvertically,gravity
affectsthecurvatureofthemirror.Forthisreasonthemirrorsskinhasdevicescalledactuators
attachedtothebackofit,whichcaneitherpushoutorpullinundersoftwarecontroltomaintain
thepropercurvatureofthetelescopemirrors.BackintheJumpStartlabthatIwasrunning,someof
mystudentswerewritingtheusecasestocontroltheseactuators.Othersweredoingmirror
pointing,aiming,andtracking.Anotherteam(whichasIrecallincludedTimswifeRobyn,whois
nowtheQAManageronLSST)wasdoingTelemetryusecases.
Irememberthinking,asIwalkedbacktotheclassroomfromthefootballstadium,thatIwaspretty
luckytobeinvolvedwiththisproject,andthatbetweenLBTandtheweekIhadoncespentworking
withtheHubblesoftwarefolks,Iwasdoublylucky.IresolvedtostayintouchwithTimandhoped
thatImightgettovisitLBToneday.SofarIvemadeitupthemountaintwice,andIhopesomeday
around2015ImightgettovisitChiletohavealookatLSST.

16
http://mirrorlab.as.arizona.edu/MISC.php?navi=tours

10
StretchingICONIXProcess:LBTsObservatoryControlSystem
TheObservatoryControlSystem(OCS)forLBTwasbynomeansaclassicalusecasesystem,but
ratherasystemwithmanyrealtimeembeddedaspects,thatwascontrolledbyanoperatorthrough
arelativelysimpleGUI.
TheresaGUI,butThatsNottheHardPart
DuringtheLBTJumpStartworkshopinTucson,thestrategythatTimandIadoptedwastousethe
operatorusecasesasanorganizingframeworkforthemodel,eventhoughweknewthattheuser
interfacewasnotwheretherealcomplexityoftheLBTsoftwarelived.

Figure2a.LBTsObservatoryControlSystemGUIisnotwherethecomplexityoftheOCSsoftwareresides.

Figure2b.LBTsObservatoryControlSystemdoeshavesomelegitimateusecases.
11
StartingfromtheGUI,wewereabletousethisapproachtoidentifyembeddedoperational
scenariosthatwereinvokedfromtheoperatorcentricusecases,andrecognizedthatwithinthese
embeddedscenarios(whichinvolvesoftwarecontrollinghardware),therewouldbesomedeviation
fromclassicalusecasedrivenICONIXProcess.
ModelingEmbeddedRealtimeSystemsWithScenarios
LBT,althoughabehemothamongtelescopes,behavesinasimilarmannertomosttraditional
telescopesinthatthetelescopeisaimedatacelestialobject,andhastotrackthatobjectwhilethe
earthrotatesandphotosaretaken.Sotherearevariouscommandsissuedbythetelescope
softwarethatdrivetheservomotorsandotherhardwarethatmovethemirrors,thetelescope
(andactuallytheentirebuildingthatenclosesit,whichswivelsongiantrollers)around.
Whilemovingthetelescopeisntausecaseintensiveprocess,itsfairlystraightforwardtoidentify
scenariosofoperationsuchas:Pointthetelescopeatapresetlocationontheskyandwecan
modelthesescenarioswithusecases,withanunderstandingthattherulesformodelingusecases
canbebentalittletoaccommodatetherealtimenatureofthesoftware.

ModelingTip:Scenariotechniquesworkwellforrealtimeembedded
systems
Usecasedrivendevelopmentisascenariobasedapproach.Itstretchesreasonablywelltorealtime
embeddedsystemswhenyoucaneasilyidentifycontrolscenarios.Someoftheusecaserules
canbebentwhenthereisnouserinterface.

Figure3aTheOCShasplentyofscenarios,eventhoughthosescenariosarentclassicalusecases.
12
SotheOCSisntreallytoobigofastretchforICONIXProcessbecauseitsstraightforwardtoidentify
scenarios.Ingeneral,ourexperienceovertheyearshasbeenthatICONIXProcesshandlesrealtime
embeddedsystemsprettywell.Bycontrast,asyoullseelater,theLSSTDataManagementsoftware
isalmostpurelyalgorithmicinnature,makingforamuchbiggerstretch.
Eventhoughthescenariosarefairlysimple,likemovingthetelescopetoapresetpositiononthe
sky,thesoftwarewithinthemneedstobedesignedprettycarefully,becausesoftwarefailurescan
getprettycostlywithhardwareonthescaleofLBT.

Figure3b.Aimingthetelescopeinvolvessomesignificanteffort(JeffKantorexplainsittomysonRob)
ModelingRealtimeSystems:ItsOKtoBendtheRulesaLittle
IhadverylittleinteractionwithTimsgroupaftertheinitialJumpStartworkshopuntilwestarted
workingtogetheronLSST.ThisisntparticularlyunusualwithourengagementsatICONIX,asour
fivedayworkshopsaregenerallyprettysuccessfulatgettingtheUMLmodelorganized,andgiving
themodelingeffortabigpushintherightdirection.ButIwasntaroundtoguidethemodeling
effort,andcreditfortheprojectssuccessgoesentirelytoTim.
Muchofthebenefitrealizedfromamodelingeffortissimplyhavingasharedmediumwithwhich
peopleontheprojectcancommunicateideasaboutthedesign.Soitsmuchbettertodoamodel,
andbendtherulesalittle,thannottodoamodelatall.
AgoodUMLmodelingtoollikeEnterpriseArchitectprovidesasharedworkspacethatallows
differentteammemberssimultaneousaccesstothemodel,and,whileprovidingassistancein
checkingsemantics,isnotoverlyintrusiveaboutadherencetothosesemantics.
ModelingTip:Asharedworkspacefacilitatescommunication
UMLmodelingisacollaborativeactivity,whosepurposeistocreateasharedunderstandingofthe
problemandthesolution.Agoodmodelingtoolthatprovidesasharedworkspacemakesthis
communicationmoreeffective.

13
Tim,likemanyotherexperiencedalgorithmdevelopersisquitefamiliarwithusingdataflow
diagramstodecomposecomplexprocesses,and,withoutmebeingaroundtoharasshimabout
stayingpureobjectorientedinhisthinking,heoccasionallyexpressedsomemodelingideasin
DFDlikeform.
Ofcourse,ImnostrangertoDFDsandfunctionaldecompositionmyself.Backintheearlydaysof
ICONIXwebuiltandsoldaDFDeditorCASEtool,andItaughtmanyclassesinStructuredAnalysis
andDesign.
17
SoIrecognizethemwhenIseethem,andIunderstandthedifferencebetweena
functionaldecompositionwithDFDsandascenariodecompositionwithusecases.
Imafirmbelieverthatascenariodecompositionisbetter,inmostcases,especiallyifthe
implementationisgoingtobeobjectoriented,becauseweknowhowtoreliablydrivegoodOO
designsfromscenarios.

Figure4.AfewoftheLBTusecasediagramshaveadistinctfunctionaldecompositionflavortothem

SemanticConsistencyisaGoodThing,butItsNottheOnlyThing
SuccessfuldevelopersandprojectmanagerslikeTim(whodoesboth)recognizethatmodelsexist
togetthejobdone,andgettingthejobdonecomesbeforesemanticpurityinmodeling.
SinceitwasobviousduringtheJumpStartworkshopthatthemostcomplexpartoftheLBT
softwaredidntinvolveverymuchuserinteraction,IencouragedTimtobendtheICONIXusecase

17
This included the class I taught at NASA Goddard to the folks who were re-hosting the
Hubble co-processor software.
14
modelingrulesasheneededto,andusetheUMLmodelasacommunicationvehicleforhimto
communicatethedesignhewantedtohisdevelopmentteam.
Whiletherobustnessdiagrambelow(whichconsistsalmostentirelyofcontrollerobjectsthose
arethecircleswiththearrow)probablywouldntwinaprizeforsemanticconsistencywiththe
textbookspecifiednotation,
18
itdoessucceedinshowingtheconceptualdesignofagroupof
controllerobjectsinteractingtogethertopointthetelescopeatadesiredspotonthesky
(CelestialPosition).

Figure5aLBTsObservatoryControlSystemisjustthatacontrolsystemthatusessoftwaretocontrolhardware.
Normalusecasemodelingrulesdontalwaysapply.
Itsasimplescenario,butyoureallywouldntwanttoscrewituprefactoringtheLBThardware
couldgetexpensive.SoTestDrivenDevelopingthestopMirrorsmethodbyfirstmakingaunittest
failandrunningthemirrorsintothewallmightnotbetheidealapproachhere.

18
The controllers represent verbs, or actions, in the emerging software design.
15

Figure 5b. Tims controller objects are hauling 40 tons of mirrors (and a lot of steel) around.Unambiguous
Communication is the Only Option
ICONIXProcessmakesheavyuseofsequencediagramstodescribeasoftwaredesignatan
object/messagelevel.Whiletheprocesssystematicallydrivessequencediagramsfromusecases,
UMLsequencediagramsarestillhighlyusefulevenwhentherearenousecases,astheStatus
RequestdiagraminFigure6shows.
Byspecifyingthedesignatthislevel,Timwasabletounambiguouslycommunicatetohis
developersexactlywhattheLBTsoftwareneededtodo.Tothebestofmyknowledge,thisdesignis
stillthebasisofthesoftwarethatsrunningtheLBTObservatorytoday.

Figure6.Unambiguouscommunicationbetweenmanagementanddevelopersisessentialforprojectsuccess.

16
Chapter2
WhatsaLargeSynopticSurveyTelescope
(andwhydoweneedone)?

19
ManythankstothefolksatLSSTCorporation forallowingustousesomecontent(virtuallyallof
thischapter)fromtheLSST.orgwebsite!
Initsfirstmonthofoperation,theLSSTwillseemoreoftheUniversethanallprevioustelescopes
combined.Itsrapidfire,3billionpixeldigitalcamerawillopenamovielikewindowonobjectsthat
changeormove;itssuperbimageswillenablemappingthecosmosin3Dasneverbefore.
Surveyingtheentireskyeveryfewdays,LSSTwillprovidedatainrealtimetobothastronomersand
thepublic.Forthefirsttime,everyonecandirectlyparticipateinourjourneyofcosmicdiscovery.

Figure1.SuzanneJacobywiththeLSSTfocalplanearrayscalemodel.Thearraysdiameteris64cm.Thismosaicwill
provideover3Gigapixelsperimage.Theimageofthemoon(30arcminutes)isplacedthereforscaleoftheFieldofView.
(Imagecredit:LSSTCorporation)
LSSTsmainscienceareasincludeDarkMatter/DarkEnergy,NearEarthObjects,TheOuterSolar
System,MappingtheMilkyWay,andtheTransientHighEnergyUniverse.LSSTwillproduceand
distributeasetofdataproducts,includingImages,Catalogs,andAlerts,thatastronomerswilluse
toexplorethesescienceareas.

19
LSST is a public-private partnership. Design and development activity is supported in part by the National
Science Foundation. Additional funding comes from private foundation gifts, grants to universities, and in-kind
support of Department of Energy laboratories and other LSST Member Institutions. The project is overseen by the
LSST Corporation, a non-profit 501(c)3 corporation formed in 2003, with headquarters in Tucson, AZ.
17
Synoptic=AllSeeing
The8.4meterLSSTwillsurveytheentirevisibleskydeeplyinmultiplecolorseveryweekwithits
threebillionpixeldigitalcamera,probingthemysteriesofDarkMatterandDarkEnergy,and
openingamovielikewindowonobjectsthatchangeormoverapidly:explodingsupernovae,
potentiallyhazardousnearEarthasteroids,anddistantKuiperBeltObjects.

Figure2.LSSTGroupPicture.MembersoftheteambuildingtheLSST,alargesurveytelescopebeingbuiltinNorthern
Chile,gathertocelebratethesuccessfulcastingofthetelescopes27.5ftdiametermirrorblank,August2008.(Image
credit:HowardLester/LSSTCorporation)
PlansforsharingthedatafromLSSTwiththepublicareasambitiousasthetelescopeitself.Anyone
withacomputerwillbeabletoflythroughtheUniverse,zoomingpastobjectsahundredmillion
timesfainterthancanbeobservedwiththeunaidedeye.TheLSSTprojectwillprovideanalysis
toolstoenablebothstudentsandthepublictoparticipateintheprocessofscientificdiscovery.
20
WevesummarizedsomescienceinformationfromtheLSSTwebsitebelow.
DarkMatter
About90%oftheUniverseisdarkwecantseeitexceptthroughitsgravitationalpull.Although
thiswassuspectedmorethan60yearsago,wearejustnowinapositiontoexplorethedarkmatter
inlargeareasoftheUniversethroughatechniquecalledweakgravitationallensing.
DarkEnergy
Darkenergyisamysteriousforcethatisacceleratingtheexpansionoftheuniverse.Theexpansion
hasslowedtheclusteringofdarkmatter,oneoftheuniversesmainbuildingblocks.

20
You can find out much more about the science enabled by LSST at
http://www.lsst.org/lsst/science
18

Figure3.Spacetimewarp:thedetailedmassdistributionintheclusterCL0024isshown,withgravitationallydistorted
graphpaperoverlaid.Thisdetaileddarkmatterdistributioncanbeusedtoconstraintheoriesofdarkmatter.
NearEarthObjects
ThequestionsofhowthesolarsystemcameintobeingandhowlifeonEarthmightendaretwoof
thenearestastronomicalissuestomankind.Answeringtheformerrequiresasaninitialstepa
censusofthesolarsystem.Butidentifyingobjectsintheoutersolarsystemhasproventobea
difficultchallengedespitesomerecentsuccess.Answeringthelatter,atleastinthecaseofa
possibleasteroidimpact,isnotstrictlyascientificquestionbutmightbethemostimportant
contributionastronomymakestolifeonEarth.TheLSSTwouldmakeuniquelypowerful
contributionstothestudyofnearEarthobjects(NEOs).
TheOuterSolarSystem
TheLSSThasbeenidentifiedasanationalscientificpriorityinreportsbydiversenationalpanels,
includingseveralNationalAcademyofSciencesandfederalagencyadvisorycommittees.
InvestigatingtheextentandcontentofthesolarsystembeyondNeptunerequiresadetailed
understandingoftheKuiperBelt,whichinturnwillleadtoanimprovedunderstandingofthelink
betweenourSolarSystemandthosebeingdiscoveredaroundotherstars.Thestudyoftheouter
solarsystemwillnotonlyclarifytheformationhistoryofoursolarsystem,butwillpointthewayto
howothersolarsystemsmayformandhowstarformationingeneralproceeds.
NewLightOnTheTransientHighEnergyUniverse
TheLSSTwillexploreanewuniverseoftransientsourcesofluminosity.Wealreadyhavehintsof
unusualandviolenteventsatgreatdistances.Recently,acombinationofobservationswiththe
BeppoSAXsatelliteandtwogroundbasedopticaltelescopeshasproducedproofthatgammaray
burstsourcesareathighredshiftandsomustbethemostluminousobjectsintheuniverse.This
mustbenewphysicsoramanifestationofphysicsinnewandextremeconditions:attemptsata
theoreticalexplanationusingknownphysicshaveencounteredmajordifficulties.Moregenerally,
ourmonitoringofandknowledgeoftransientsourcesinthecosmosisinitsinfancy.
19
MappingtheMilkyWay
TheLSSTisideallysuitedtoansweringtwobasicquestionsabouttheMilkyWayGalaxy:
*WhatisthestructureandevolutionhistoryoftheMilkyWay?
*WhatarethefundamentalpropertiesofallthestarsintheneighborhoodofourSun?
LSSTwillproduceamassiveandexquisitelyaccuratedata.Comparedtothethebestcurrently
availableopticalsurvey,SloanDigitalSkySurey(SDSS)LSSTwillcoveranareamorethantwiceas
large,makinghundredsofobservationsofthesameregion(insteadofjustoneortwo)andeach
observationwillbeabout2magnitudesdeeper.ThecoverageoftheplaneofourGalaxywillyield
datafornumerousstarformingregions,andevenpenetratingthroughtheinterstellardustlayer.
LSSTwilldetectoftheorderof10billionstars.

Figure4.The8.4meterLSSTwilluseaspecialthreemirrordesign,creatinganexceptionallywidefieldofviewandwill
havetheabilitytosurveytheentireskyinonlythreenights.(Imagecredit:LSSTCorporation)
LSSTDataProducts
TheLSSTdataproductsareorganizedintotwogroups,distinguishedbythecadencewithwhich
theyaregenerated.LevelOneproductsaregeneratedbypipelineprocessingthestreamofdata
fromthecamerasystemduringnormalobserving.LevelOneproductsarethereforebeing
continuouslygeneratedandupdatedeveryobservingnight.Thisprocessisofnecessityhighly
automated,andmustproceedwithabsolutelyminimalhumaninteraction.LevelOnedataproducts
aredividedintoImage,Catalog,andAlertcategories,asshowninFigure5.
20

Figure5.LSSTDataProducts
HighLevelRequirements
TheFunctionalRequirementSpecificationsofthedatamanagementsystemincludethefollowing:
1)Theincomingstreamofimagesgeneratedbythecamerasystemduringobservingisprocessedto
generateandarchivethenightlydataproducts
*Rawscienceimages
*Catalogofvariablesources
*Transientalerts
2)Periodicallytheaccumulatednightlydataproductsareprocessedto
*Generatecoaddedimagesofseveraltypes
*Optimallymeasurethepropertiesoffainterobjects
*Performastrometricandphotometriccalibrationofthefullsurveyobjectcatalog
*Classifyobjectsbasedonboththeirstaticpropertiesandtimedependentbehavior
LevelTwoproducts,includingcalibrationimages,coaddedimages,andtheresultingcatalogs,are
generatedonamuchslowercadence,andtheirreleasewillbedrivenbydataqualityassessments.
AlthoughmanyofthestepsthatgenerateLevelTwoproductswillbeautomated,theyneednotall
beso,andsignificanthumaninteractionmaybetolerated.
LSSTDataManagementPipelines
Thepipelinesprocesstheimagestoproducethecatalogs,whicharethenmadeaccessibletothe
communityviaopeninterfacesinaVirtualObservatorymodel.Sincenewdataisbeingcollected
nightlythroughouttheLSSTs10yearsurveyperiod,andscientificalgorithmswillevolveduringthis
timeframe,significantreprocessingwilloccur.ThismustbetakenintoaccountinsizingtheLSST
technologyresourcesandmakingtheLSSTmiddlewareeasilyextendable.
21

Figure6.LSSTsImageProcessingPipelinesoftwareisthetopicofthisbook
Thepipelinescanbecategorizedintermsofeitherbeingnearrealtimeorstatic,dependingon
howstringenttheassociatedlatencyandthroughputdeadlinesare.Examplesofnearrealtime
pipelinesincludedataqualityassessmentprovidingfeedbacktotelescopeoperations,instrument
calibrationprocessing,andtimedomainortransientscienceanalysis.Thesepipelinesexecuteat
themountainbasefacilityinordertoavoidthelatencyassociatedwithlonghaultransmissionof
therawdata.Thestaticpipelinesincludedeepimagecoaddition,weaklensingshearprocessing
neededfordarkenergydarkmatterscience,andobjectcataloging.Thesepipelinesexecuteatthe
archivecenter.Thearchivecenteralsoperformsreprocessingofthenearrealtimepipelines.

22
Chapter3
DataChallenges:From0to20Terabytesanight

21
summarizesthechallengefacingJeffandTimverynicely: TheLSSTwebsite
Thesciencearchivewillconsistof400,000sixteenmegapixelimagespernight(for10
years),comprising60PBofpixeldata.ThisenormousLSSTdataarchiveandobject
databaseenablesadiversemultidisciplinaryresearchprogram:astronomy&
astrophysics;machinelearning(datamining);exploratorydataanalysis;extremelylarge
databases;scientificvisualization;computationalscience&distributedcomputing;and
inquirybasedscienceeducation(usingdataintheclassroom).Manypossiblescientific
dataminingusecasesareanticipatedwiththisdatabase.
TheLSSTscientificdatabasewillinclude:
*Over100databasetables
*Imagemetadataconsistingof700millionrows
*Asourcecatalogwith3trillionrows
*Anobjectcatalogwith20billionrowseachwith200+attributes
*Amovingobjectcatalogwith10millionrows
*Avariableobjectcatalogwith100millionrows
*Analertscatalog.Alertsissuedworldwidewithin60seconds.
*Calibration,configuration,processing,andprovenancemetadata
SkyMoviesChallengesofLSSTDataManagement
TheDataManagement(DM)partoftheLSSTsoftwareisabeastofaproject.LSSTwilldealwith
unprecedenteddatavolumes.Thetelescopescamerawillproduceastreamofindividualimages
thatareeach3.2billionpixels,withanewimagecomingalongeverycoupleofminutes.
Inessence,theLSSTskysurveywillproducea10yearskymovie.Ifyouthinkoftelescopeslike
LBTproducingaseriesofsnapshotsofselectedgalaxiesandothercelestialobjects,andsurvey
telescopessuchasSloanproducingaskymap
22
,thenLSSTsdatastreamismoreanalogousto
producinga10year,framebyframevideoofthesky.
LSSTsUseCasesWillInvolveAccessingtheCatalogs
LSSTsmandateincludesawidedistributionofsciencedata.Virtuallyanyonewhowantstowillbe
abletoaccesstheLSSTdatabase.SopartsoftheLSSTDMsoftwarewillinvolveusecasesanduser
interfacesforaccessingthedataproducedbythetelescope.Thosedataminingpartsofthe
softwarewillbedesignedusingregularusecasedrivenICONIXProcess,buttheyrenotthepartof
thesoftwarethatwereconcernedwithinthisbook.

21
www.lsst.org/lsst/science/technology
22
www.sdss.org/dr7/products/index.html
23

Figure1LSSTwillproducemanycatalogs,whichwillbewidelyaccessiblebythepublic
LotsofBrainsandaFairAmountofTime
ThereareacoupleofthingsthatJeffandTimdohaveworkingintheirfavor:plentyofbrains(not
onlytheirown,butawidespreadandlargelybrilliantteamofastrophysiciststhatareexpertson
variouspiecesoftheproblem),andafairamountoftime(LSSTisscheduledtogooperationalin
2015,andiscurrentlyinanR&Dphase).However,itssafetosaythatmostoftheastrophysicists
ontheteamwouldntconsiderthemselvessoftwareengineers,althoughmostofthemare
programmers.Inthissituation,agoodstrategyistomakeextensiveuseofrapidprototyping(inthis
casealgorithmdevelopmentviaprototyping)inadditiontotheUMLmodeling.Soatwopronged
strategyofprototypingandmodelinghasbeenunderwayonLSSTforafewyearsnow.
TheLSSTprototypingstrategyinvolvesannualDataChallenges(seeFigure2).TheseData
Challengesaredevelopmenteffortswithalimitedfunctionalandperformancescope,and
somewhatrelaxedmodelingrequirements.DuringLSSTsConstructionPhase,prototypingwill
switchtoincrementaldevelopment,wheretheactualsystemwillbedevelopedinasequenceof
incrementalreleases,andsomewhatmoremodelingwillbeexpected.

Figure2.LSSTsR&DPhaseisbeingconductedasaseriesofDataChallenges
InthenextChapter,welltakeyouintoamodelingworkshopthatIhelpedtoconduct,forData
Challenge3(DC3),wheretheneedforsomeprocesstailoringbecameobvious.Butfirstlet'slookat
someofthechallengesfacedbytheLSSTmodelingteam.
24
LetCodersWriteCode
AveryimportantpartofthedevelopmentstrategyforLSSTistoimplementkeyalgorithmsand
validatethem,inparallelwithdevelopinganoverallUMLmodelfortheLSSTsoftware.Thisis
importantformanyreasons,amongthem(asyoullseeinthenextchapter)thatlotsof
programmerspreferwritingcodetomodelinginUML
23
.Butinordertomakeallofthese
independentlydevelopedprototypepiecesworktogetherwhenLSSTgetstoConstruction,weneed
tobeabletoimporttheprototypecodeintothemodel.EnterpriseArchitectsreverseengineering
capabilitiesmakethisworkquitesmoothly(seeFigures2and3).

Figure2.LSSTsscheduleallowsanR&DPhasethatslongenoughtoallowlotsofprototyping.
EnterpriseArchitectcaninhalecodeasfastasitcanbewritten,soJeffandTimareabletoallow
programmerstoprototypevariouscriticalpiecesofthedatamanagementsoftwareduringtheData
Challenges,withfullconfidencethatwhenthetimecomestobeginworkontherealproduction
software,theprototypecodecanbeminedasneededandmadetoworkwithintheoverall
systemarchitecture.
ModelingTip:Combinemodelingwithprototyping
Dontbeafraidtobuildprototypecode,but
Domakesuretoreverseengineeritintoyourproductionmodel.
Makesureyourmodelingtooliseffectiveatreverseengineering.
Combiningmodelingwithprototypingisaneffectiveriskmitigationstrategy.

23
We know this will come as a shock to many readers.
25

Figure3.EnterpriseArchitectsreverseengineeringcapabilitiesallowcomplexalgorithmstobeprototyped,then
incorporatedintothemodel.
DistributedDevelopmentTeam
AfurthercomplicationinmodelingLSSTsDMsoftwareisthatthedevelopmentisbeingdonein
multiple,independent,geographicallydistributedresearchorganizationsplaceslikePrinceton,
Stanford,UniversityofWashingtonandCaltech,amongothers.Thisplacessomesignificant
requirementsonthemodelingtoolsthatareusedbytheproject.Fortunately,EnterpriseArchitect
supportsdistributeddevelopmentusingversioncontrolledpackages,allowingsomeoneatCaltech
orStanfordtocheckoutapackage,addinformationtothemodel,andcheckitbackintothe
centralrepositoryinTucson.
ModelingTip:Managedistributedteamswithversioncontrol
Makesurethatyourmodelingtoolallowsyoutomanagelargeprojectsbyprovidinginterfacesto
versioncontrolsystemandallowingcheckin/checkoutonapackagebypackagebasis.
HereToday,GoneTomorrow
MyinvolvementwithLSSThasbeenongoingbutsporadic.Duetobudgetconstraintsandthe
incrementalnatureofthevariousDataChallenges(andtorelaxedmodelingrequirementsduring
LSSTsR&Dphase),weneverdidafull5dayLSSTJumpStartworkshopaswedidwithLBT.Somy
involvementhasbeenmoreofanoccasionalvisitortothevariousworkshopsontheproject,onthe
orderofafewdaysayearforthelastfewyears.
Myapproachhasalwaysbeentoshowupwhenrequested,anddothebestIcantohelpout.In
somecasesthishasbeenassimpleasbeingTimspersonalmodelingassistant.Inothercasesit
hasinvolvedstrategysessionswithJeffandTimabouttheoverallapproach,and,asyoullseeinthe
nextchapter,tailoringICONIXProcesstomakeitworkbetterforLSST.Wellspendtherestofthe
bookexploringthewhyandhowoftheresult:ICONIXProcessforAlgorithms.
26
Chapter4
TailoringICONIXProcessforAlgorithmDevelopment

Thoseofuswhodefinesoftwareprocessesdoourbesttomakeaprocessasgenerallyusefulas
possible.However,noprocesscanbeaperfectfitforeveryproject,andwhileICONIXProcesshas
beenusefultothousandsofsoftwareprojectsoverthelast15years,theresnodisgraceinvolvedin
tailoringittofitaspecificproblem,orclassofproblems.Inthiscase,tailoringittofitLSSTData
Managementresultedinaprocessthatsapplicabletoallkindsofalgorithmintensiveprojects.
TheneedtoconsidertailoringICONIXProcesstobetterfitLSSTsdatamanagementsoftwarecanbe
tracedbacktoanemaildiscussionbetweenJeff,Tim,andIbackinAugustof2008,inpreparation
fortheDC3DesignworkshopwhichwasheldatCaltech.
ThePasadenaDC3WorkshopHowtheNeedforProcessTailoring
Surfaced
TheDC3workshophadaveryspecificpurpose:tobringkeyplayersfromtheLSSTteamtogetherto
brainstormandcaptureideasforDataChallenge3inthemodel.Jeffwasmuchlessconcernedwith
thepreciseformthattheinformationcapturetook,thanwithgettingasmanyalgorithmscaptured
aspossibleinashorttimeperiod,becausehewasonthehooktoproduceestimatesforthe
upcomingPreliminaryDesignReview(PDR).
Diggingthroughmyoldemailsfindsthisdiscussion,beginningwithanemailfrommetoJeffand
Timregardinghowwewantedtoapproachtheworkshop:
JeffandTim
HereareafewthingsthatIbelievehelptodefinetherightwaytodotheseusecase/UML
models.IflefttomyowndevicesIwouldteachtheworkshopattendeestheseprinciples,butyou
needtogivememarchingordersastowhetheryouwantmetodothisornot.Ithinkoneofthekey
valuesofrobustnessanalysisisthatithelpstoensurewellformedorproperusecases.Tobe
honestIdontknowwhattherobustnessdiagramswilllooklikeifwedontfollowtheseguidelines
usecasediagramsarenotDataFlowDiagrams(ausecasedoesnthaveINPUTS:OUTPUTS:and
drilldowntoseelowerlevelfunctions)
torestatethepreviouspoint,usecasesareaSCENARIOdecomposition,notfunctional
decomposition
scenariosaregenerallydistinguishedbyhavinganactor,andrepresentevent/responsedialog
betweenasystemandsomethingoutsideofthatsystem,whereasfunctionsarealgorithmic(allone
system)
torestatethatone,algorithmsarenotscenarios(algorithmsaregenerallyrepresentedas
controllerswithinscenarios)
functionalrequirementsarenotscenarios(buttheyshouldbelinkedtothescenarios).
Requirementscanalsobelinkedtocontrollers.
Doug

Timsresponsetothisemailwasasfollows:
HiDoug,
Ithinktheresalongtermandashorttermresponsetoyourconcerns.First,thelongterm.Ive
graduallybeencomingtotheviewthatavariantoftheIconixprocessneedstobedevelopedthatis
27
bettertunedtopeoplelikeus(astronomers,physicists,scientificapplicationprogrammers)andthe
systemsweneedtoconstruct.IthinkthatthedifficultiesthatweverunintowithLBTandLSSTare
notonlyduetoourownlackofdisciplineandcatlikenatures.Eventhoseofuswhosupporttheuse
ofaformaldesignprocess(likeme)havenotreallyuseditasintended,asyouareseeing.Atleast
sometimes,theprocessjustdoesntseemanaturalfittothewaywethinkaboutoursystems.The
partofLSSTwereworkingonnowisdominatedbycomplexpipelineprocessing,andwhenwe
scribbleonourwhiteboardswealwaysendupwithinputsandoutputsandboxesthattransform
data.Andso,whenwegointoUMLwewanttocontinuethinkingthatway,andenduptorquing
thedesignprocessintosomethingthatdoesntworkright.So,perhapsafterPDRiscomplete(and
wevesurvivedit),itwouldbeworthwhiletodevotesomeseriousthoughttocreatingamodified
Iconixprocess(TheIconixProcessforScientificCatsbyRosenberg,Kantor,andAxelrod??)
Intheshortterm,Imnotsurehowtoproceed.Jeffhassomeprettydefiniteideasofwhatweneed
togetthroughPDR,andsomehowweneedtomobilizetheteamtoproducewhatsneeded.And
thereistheminormatterthatwehavealotofcodetoproduceaswell!Forthemodeling,itmight
actuallymakemostsensetotakeasubteam(sayJeff,Nicole,andme)andnotionallylockusina
roomtoproducethemodel,drawingasneededontheotherapplicationfolksandtranslatingwhat
theytellusintoUML.Ifthatstherightapproach,weshouldtryitonforsizenextweek.Maybe
theressomesmallmodification/additiontotheIconixprocessthatcouldhelpintheshortterm
youmentionedDFDs.IllingoodpartleavethissetofdecisionsuptoyouandJeff,thoughIll
certainlywanttokibitz(andmaybeprotest!).
Keepthinking,
Tim
Sohereswhereitstarted.AsuggestionfromTimthatmaybeitwastimetothinkabouttailoring
theprocess.SinceTimsoneofthesmartestguysthatIknow,andIhavenevereverknownhimto
speakwithoutagooddealofforethought,Itookthisemailveryseriously,andIstartedmullingover
whatanalgorithmicvariantofICONIXProcessmightlooklike.
Wedecidedtotakeadayforthethreeofus(Tim,Jeff,andmyself)toworkthroughanexample
whichweduseasamodelforworkshopattendees.WeproceededtoproduceastretchedICONIX
Processexample,whichcapturedhighlevelalgorithmsasusecases,andlowerlevelalgorithmsas
controllersonrobustnessdiagrams.Idontthinkanyofusfeltcompletelycomfortablewiththe
example,butJefffeltprettystronglythattryingtointroducesomethingotherthanICONIXProcess
aspublishedinmybooksatthisparticularworkshopwouldcausemoreproblemsthanitmight
solve,whenwhattheyreallyneededwassomeforwardprogresscapturingalgorithmdetailwithin
themodel.Sothiswasntthetimetobethinkingaboutprocesstailoring.
ButallthreeofusknewthatwewerestretchingtextbookICONIXProcessfurtherthanwefelt
comfortablewith.
Asweworkedthroughourexample,thethoughtthatkeptgrowinginmyheadwastryingto
modelalgorithmsasusecaseswasthefundamentalproblem,becausescenariomodeling(which
hadworkedonLBT)andalgorithmmodelingweredifferent,andneededtohavedifferent
guidelines.IntheLSSTimageprocessingsoftware,scenarioswerescarce,andalgorithmswere
plentiful.Tofurthercomplicatematters,thealgorithmswereorganizedintoworkflowsanddata
productions
24
.
Asitturnsout,thePasadenaworkshopreinforcedtheseideasquitestronglyinmymind.Butitwas
duringourdayofpreworkshoppreparationthatIfirstsuggestedwestopusingusecasestomodel
algorithms,distinguishbetweenworkflowandsciencealgorithms(anduseactivitydiagramsto
describeworkflow,andcontrollersonrobustnessdiagramstocapturedetailsaboutscience
algorithms).

24
In LSST terminology, workflows are the image processing pipelines, and productions involve running a set of
pipelines to produce a set of data.
28
WereNotBuildingInternetBookstores
Ihadbeenwarnedonseveraloccasionsaboutthelevelofantimodelingsentimentinsomepartsof
theLSSTteam,andinparticularthatonehighlybrilliantsuperheroprogrammerwasthemost
vocal,andthatIdbemeetinghimforthefirsttimeatthisworkshop.Iwontusehisnameinthis
discussionbutsinceheisalegitimatesuperheroprogrammer(beinglargelyresponsiblefor
anothermajorsurveytelescopessoftware),IllrefertohimassuperCoderwhichisafairlyaccurate
descriptionofhisskilllevel.Therewasalsoanotherindividualwho,havingbeenintheindustryfor
quitealongtime,datedbacktothedaysofFORTRAN,andwhohaddubbedtherobustness
diagramnotationMartianandproudlypronouncedthatIdontspeakMartian.Forthis
discussionwellcallhimFORTRAN_JOCK.
SoJeffandTimhadwarnedmetoexpectactiveresistancefromsuperCoderandFORTRAN_JOCK
duringtheworkshop,andIwasnottobedisappointed.Jeff(andtosomeextentTim)hasspent
manyhoursbangingheadswiththesefolksregardingthevalueandnecessityofmodeling.
superCoder,havingmoreorlesssinglehandedlyprogrammedtheothersurveytelescopedidntsee
whyhecouldntdothesamewithLSST.LSST,ofcourseismassivelybiggerandmorecomplexthan
anyprevioussurveytelescopeandthiswouldbeacompleteimpossibility,evenifitweredesirable.
Thefirstmorningoftheworkshopturnedouttobequitecontentious.Ihadacoupleofhourson
themeetingagendatopresenttheexamplewehaddevelopedandgiveaquickoverviewofthe
planofattackforthenext2days.JeffhadpredictedthatthistimewouldbeusedbysuperCoderand
FORTRAN_JOCKtobasicallyattackthewholeapproachwewereusing,andopenadebateabout
modelingthat,ifleftunchecked,wouldsuckuptheentire2daysthatwereallocatedtocapture
algorithmsintothemodel.Jeffspredictionswereentirelycorrect,ledoffbysuperCoders
proclamationtome(andtheassembledgroup)thathehadreadmybook(whoseexample
happenedtobeanInternetbookstore),andthatwhileitcontainedlovelyguidanceforthosewho
mightbebuildingInternetbookstores,thattheLSSTDMgroupwasmostemphaticallynotbuilding
internetbookstores,andthustheguidancepresentedinmybookwasgenerallynotusefultohim.
ThiswasfollowedbyFORTRAN_JOCKspronouncementthathedidntspeakMartian,whichgot
chucklesfromeverybody,includingme.Inowroutinelyintroducetherobustnessnotationto
studentsasMartian.
Jeff,havingfullyanticipatedthissortofspeechfromsuperCoder,wasabsolutelyinnomoodto
tolerateit.JeffisresponsiblefortheLSSTprojectpassingPDR,andcontinuedfundingforthe
projectiscontingentuponthecostandscheduleestimatesthatheneededtoproduce,andhe
neededinformationcapturedintothemodelduringthisworkshopforthatpurpose.Ina
performanceworthyofGeneralPatton
25
,hestoodupandbasicallyoutshoutedthenaysayersand
declaredthatwewouldNOTspendthismeetingdebatingtheapproach,andthatweWOULDspend
themeetingcapturingalgorithmdetail,endofdebate,LETSGETTOWORK.
IwasprivatelysomewhatsympathetictotheInternetbookstoreargumentbecauseIknewwe
werestretchingtheprocessfartherthanIwascomfortablewith,andbythenIhadsomeideason
howtotailorittobeabetterfittotheLSSTDataManagementproblem,butIcertainlyunderstood
whereJeffwascomingfrom,andIwasquiteimpressedwithhisperformance,whichhehad
forewarnedmemightbecoming.Inanyevent,itworkedexactlyasheintended,andtheworkshop
proceededtocapturealgorithmdetailforthenext2days.

25
J effs performance actually reminded me of one of my favorite scenes from the movie Patton, which is one of
my favorite movies. In the scene, Pattons army is marching 100 miles through the snow trying to rescue a
battalion at Bastogne during the Battle of the Bulge. His officers are lamenting the bad weather and the fact that
they cant get any air cover. Patton launches into a tirade that ends with: If we are not victorious, LET NO MAN
COME BACK ALIVE. His aide pulls him aside and says: General, sometimes the men cant tell when youre
acting. To which General Patton responds: It isnt important for them to know its only important for me to
know.
29
ModelingTipDontconfuseusecasesandalgorithms
Manypeopleconfuseusecaseswithalgorithmsbecausebotharegenerallynamedwithaverb
phraseandconsistofasequenceofsteps.Inmostsystems,algorithmsarerepresentedassteps
withinusecases(controllersonrobustnessdiagrams).LSSTsimageprocessingpipelinesdont
reallyhaveanyusecases.
Iwouldratetheresultofthisworkshopasaqualifiedsuccess.Wecapturedalotofalgorithms,but
wecapturedtheminalessthanoptimalform.Essentially,theconclusionofourpreworkshop
meetingwasthatweknewitwasgoingtobemessy,butthatwedcleanthemessuplater.Ifithad
beenanyonebutJeffpromisingthatwedcleanituplater,Iwouldhavebeenprettyskepticalabout
iteverhappening,butIknewhemeantit.Andasyoullseeshortly,TimandIweredestinedto
becomethecleanupcrew,duringwhichtimetheactualprocesstailoringtookplace,ona
whiteboardinTimsoffice.
Weorganizedtheworkshopintoteams,eachspecializedinthefunctionalareaoftheirparticular
laboratory.TherewerepeoplefromPrinceton,UniversityofWashington,Caltech,Stanford,and
severalotherlocations.Itookchargeofonelabteam,Jefftookanother,andTimanother,andwe
allfloatedaroundasbestwecouldduringthelabsessions.
Theapproachwastomodelhighlevelalgorithmswithusecases,andlowerlevelalgorithmsas
controllersontherobustnessdiagramsforthoseusecases.Sincethesoftwaredidnthaveanyuser
interface,wedidntexpecttoseeanyboundaryobjectsontherobustnessdiagrams,butwedid
wanttousetherobustnessdiagramstodiscovermissingdomainobjectsandtoidentifylowerlevel
algorithmswithinthehigherlevelalgorithms.TheproblemwiththisparticularprojectisthatLSST
imageprocessinghasmanylevelsofalgorithmswithinalgorithmswithinalgorithmswithin
algorithmsandtheguidancewasntrealclearwhentomodelwithausecaseandwhentousea
robustnessdiagram.
Tosavetime,wedecidedtohaveeachlabteamproduceitsownminidomainmodelbefore
tryingtodescribeanyusecases.IwasmuchmorerigorousaboutthiswiththeteamsthatIworked
withthanJeffandTimwere,andIthinkitmadeasignificantdifferenceintheamountofprogress
madebythevariousteams.JeffhadinitiallypenciledhimselfintoworkwithsuperCoderslabteam,
butduringthelunchbreakIapproachedhimandsuggestedthatIworkwiththatteaminstead
becausethefrictionlevelbetweenthemwasobviouslywaytoohigh.Ithinkthatwasthequickest
agreementIhaveeverreceivedfromJeffonanything.
Asitturnsout,havingmeworkwithsuperCodersteamwasaprettygoodidea.Ivebeendeveloping
softwarewithVerySmartPeoplesincemycollegedays(oftenPhDphysicistswhosescienceand
mathabilitiesgowayovermyhead),andithasexposedmetosomeveryinterestingprojects.I
haveanapproachthatIoftenusewhenImworkingwithTim,anditprettymuchinvolvesme
startingadiscussionwith:So,tellmeaboutX(ifIremembercorrectly,XinthiscasewastheDeep
DetectionPipeline),andlettingtheVerySmartPersonstartexplaininghowitallworks,whileI
takenotesinUML.
TheresslightlymorestructurebehindmyquestionsthanjusttellmeabouttheDeepDetection
PipelineinthatImfishingforspecificinformationwithmyquestions.ThefirstthingIfishforis
namesofthings(nouns).SowhiletheVSP(inthiscaseourgoodfriendsuperCoder)istellingme
whatarethethingsthatareinvolvedindeepdetection,adomainmodeldiagramisassembling
itselfinEnterpriseArchitect(seeFigure1).
30

Figure1.Someofthedomainclassesthatareinvolvedindeepdetection.Notethelinktoaseparatediagramshowing
reverseengineeredcodefromtheprototypeDeepDetectionPipeline.
OccasionallyIllstopandaskwhetherthedomainmodelImassemblinglooksrightornot.Thekey
tothisapproachisthatIneedtobeabletocaptureinformationwithoutslowingdownthenarrative
explanationoftheVSP.Inthiscase,superCoderhadalreadybuiltadeepdetectionpipelinebefore,
knewexactlyhowhewantedittowork,andwasperfectlycapableofexplainingitatamind
bendingpace.
ModelingTipAlwaysStartwithaDomainModel
Failuretodefineyourtermswillalwayscauseproblemswhenmodeling.Soyoushouldalwaysstart
withadomainmodel.Butdontgetstucktryingtomakeitperfectupfront.
Idontliketospendtoomuchtimeondomainmodeling,butrathertrytonaildownthe80%most
importantdomainobjectsasquicklyaspossiblewithanunderstandingthatwellcontinuallyrefine
thisaswediscovernewdomainclasses.Soafter15or20minutesofdomainmodeling,Iswitched
myquestionsovertoalgorithms.Asitturnsout,themagicquestionsformodelingusecases
26

workreasonablywellforalgorithmstooandImakesureanydetailthatIcaptureusesthenouns
fromthedomainmodel.InthiscaseIwastryingtoseparatepipelineworkflowfromscience
algorithms,soIstartedwithtellmeaboutthehighlevelalgorithmfortheDeepDetection
Pipelineandproceededfromthere.Onceagain,thekeyisthatIhavetobeabletocapture
informationasfastasaVerySmartPersoncanexplainittome,sotheCASEtoolthatImusing
(EnterpriseArchitect)hadbetternotgetinmywaywhileImworking(whichitdidnt).
Inasense,IuseUMLandvisualmodelingthewayagoodsecretaryusesshorthandtocapturethe
textofamemoorbusinessletter.SothenotationandtoolthatImusingareanalogousto
shorthandandastenopad.Thegoodnewsoutofallofthiswasthatwithin5minutesofstarting

26
As described in Use Case Driven Object Modeling, the magic questions are: 1. What happens? 2. And then what
happens? (repeat as necessary) 3. What else can happen? (repeat as necessary). These questions work well for
capturing algorithms, too.

31
thelabportionoftheworkshop,ourfriendsuperCoderwasstandingupatthewhiteboard
explainingalgorithms(whichhewasverygoodatandlikedtodo),andthealgorithmshewas
explainingweregettingcapturedintoJeffsmodel,byme.Forthemoment,wehadovercomethe
resistancetomodeling,butIknewthattheneedforprocesstailoringwasanissuethatwasnt
goingawayanytimesoon.
TimandIBecometheCleanupCrew,andICONIXProcessGets
Tailored
AftertheAugustPasadenaDC3workshop,IwasntinvolvedwithLSSTforseveralmonths.Butas
thePDRdeadlinestartedcreepingcloser,gettingthemodelcleanedupstartedmovinghigherup
theprioritylistforJeffandTim.SoitwasinmidDecemberthatIfoundmyselfinTucsonagainfora
2dayworkingsessionwithTimtotrytogetthemodelinbettershape.Ithinkallofuswouldhave
beenquitehappyformetobeinvolvedwiththemodelingonamorefrequentbasis,butthiswas
whatthebudgetpermitted.MyapproachwastotrytobeashelpfulasIcouldwithinthelimited
timeavailable.
IhadntseenanythingoftheLSSTmodelsinceAugust,anditdidntlooklikeTimhadreallygotten
anyrealtractionondoingmuchwithit.JeffhadgottentheresultsofthePasadenaworkshop
inhaledintotheexistingLSSTmodel,butitwassortofjustsittingthereofftotheside,waitingfor
thecleanupthatweallknewwasnecessary.Myarrivalwasthecatalystforthiscleanuptostart,
andTimandIwerethecleanupcrew.
Thefirstissueweranintowasthedomainmodel,whichwasactuallynotasingledomainmodel
anymorebutinsteadacollectionofmultiple,fragmented,andinconsistentdomainmodels.
Becauseofthesplitintomultipleteams(eachwithaminidomainmodel)inPasadena,therewas
nosingleplacetolookforthedomainmodelwhichdefinitivelyestablishedaconsistentsetof
namesfordomainclasses,andtherelationshipsbetweenthem.Thedomainmodelincluded
reverseengineeredcodefromprototypingefforts,and(asonmanyprojectsthathavent
discoveredthepowerofdomainmodeling)nameswereusedinconsistentlythroughoutthemodel.
TimandIranheadfirstintotheneedtocleanthisupwithin10minutesofgettingstarted.Ihad
askedhimsomequestionwhichcausedhimtotrytofindsomethinginthedomainmodel,andit
wasmuchtoodifficult.ItbecamecleartomethatdespiteTimhavingeveryintentionoftryingto
makethemodelwork,hewascontinuallyhavingtofightwithit.
ModelingTip:Ifyourefightingthemodel,figureoutwhy
UMLmodelsexisttohelpdefineandcommunicateadevelopmentproblemanditssolution.Ifyou
findyourselfhavingtofightthemodel,somethingswrong.Takethetimetofigureoutwhatthe
problemis,andfixit.Modelsshouldbeahelp,notahindrance.
Sowedecidedtoattackthisproblemfirst,andspentafewhoursprintingouteachandevery
domainmodeldiagram,makingnotesonthemwithmarkersofvariouscolors,andonebyone
addingdozensofobjectsthathadbeendiscoveredinPasadenaintotheprojectwidedomainmodel
(renamingmanyofthemaswewentasnameusagehadbeeninconsistentpreviously).
Wealsoremovedreverseengineeredcodefromthedomainmodelintoaseparatepackage,but
grabbedthenamesofsomeoftheseclassesandaddedthosetothedomainmodel.Withinafew
hourswesawanoticeableimprovement.

32
ModelingTip:Keepreverseengineeredcodeseparatefromthe
domainmodel
ItsOKtoprototypecodeandreverseengineeritintoaUMLmodel.OnsystemslikeLSST,its
essential.Butdontmixthereverseengineeredclasseswiththedomainmodel.
Astheconsolidateddomainmodelgrew,IsuggestedtoTimthatitwasreallyprettyimportantfor
himtoadddefinitionstoallthedomainclasses,andheagreed(withsomereluctance,becauseit
wasalotofwork).Hewasheadingtoaconferenceinsomeremoteplacethenextweekand
promisedtospendsometimeaddingthedefinitions,becausehesawthevalueinhavingaproject
glossary.Ireceivedanemailfromhimwhilehewasinthemiddleofthisexercisethatsaid:
Wouldyoubelieveit?Theexerciseofwritingthedefinitionsisturningupsomemoredomain
objects...Tim
Helatertoldmethatsomemorewasreallyabouta20%increaseinthenumberofdomain
objectsdiscovered,justbyaddingdefinitionstotheclasses.So,ifyoulookforhighleverage
modelingactivities,likeIdo,thisisdefinitelyonetomakeanoteabout.Defineyourproject
vocabulary,itworks.ThankstoTimscontinuedefforts(withsomehelpfromRobyn),theLSST
DomainModelisnowanextremelyusefulsectionofthemodel.
ModelingTip:Useyourdomainmodelasaprojectglossary
Trustme,youllbegladyoudefinedyourdomainclassesunambiguously.Itgenerallyonlytakesa
fewhours,andpaysdividendsoverthelifetimeofyourproject.
Oncethedomainmodelgotconsolidated,thenextthingweneededtodowastobringallofthe
usecasesandrobustnessdiagramsfromthePasadenaworkshopinlinewiththe(now
standardized)domainterminology.Thiswasnosmalltask,certainlynotonethatcouldbe
completedbyTimandIinlessthantwodays,anditwasduringourinitialattemptsatthisthatI
becameconvincedthattheprocesstailoringcouldntwaitmuchlonger.
IhadntseenthemodelsinceAugust,andduringtheworkshopinPasadenaIwasmostlyfocused
onthemodelsproducedbytheteamsthatIworkedwith.AsTimandIstarteddigging,hereswhat
wefound:
Pipelineworkflowsweremodeledasusecases.Usecasediagramswereusedtodescribethe
variouspipelinestagesandhowtheyrelatedtogether.
Algorithms(pipelinestages)werealsomodeledasusecases.Sowhenyoulookedatadiagramit
wasnteasytotellifyouwerelookingatapipeline,apipelinestage,orasciencealgorithm.
Everythinglookedthesame.
Schizophrenicusecasedescriptionswhichhadinputs/outputslikeanalgorithmanda
basic/alternatestructurelikeausecase.Notsurprisingly,withusecasesbeingusedtorepresent
algorithms,themodelwasfullofalgorithmusecasesthathadasplitpersonality(halfalgorithm,
halfusecase).Theproblemhereisthatthebestpracticeguidelinesforwritinggoodusecasesare
notthesameastheguidelinesfordescribingalgorithms.Thusrepresentingalgorithmswithuse
casestendstobeconfusingtobothcreatorsandreadersofthemodel.
Whilenoneofthiswasparticularlysurprisingtous,itstruckmerightbetweentheeyesthatitwas
timetodosomethingaboutit,andthatthecompellingreasontodoitnowwasthatresistanceto
modeling(whichdidntneedanyfurtherammunition)wasbeingfueledbythefactthatthemodel
simplywasntworking(i.e.wasnoteasilyunderstandable)aswellasitshould.

33
ModelingTip:Watchforsignsthatyoursoftwareprocessmayneed
tailoring
Herearesometelltalewarningsignstolookoutfor:
Theargumentsagainstmodelingseemcompelling
ThereisinevitablysomeresistancetoanysortofUMLmodeling.Generallythereisahardcore
groupofdisbelieverswhothinkthatanysortofprecodingmodelingactivityisawasteoftime,and
someotherswhounderstandtheneedforamoredisciplineddevelopmentapproachandwillmake
someattemptatmakingmodelingwork.Whenthedisbelieverspresentargumentsthatseem
compellingtoalargepercentageoftheteam,andwhenthosewhoaretryingtomakemodeling
workarefindingthatitstoughsledding,itsacluethattheprocessneedstailoring.
Themodelfeelslikeahindranceratherthanahelp
ThepurposeofUMLmodelsisnottoslowaprojectdownfornogoodpurpose(werealizethismay
comeasasurprisetosome).Thepurposeofthemodelistocapturealltheinformationthatneeds
tobecapturedinorderfortheentireteamtodevelopacommonunderstandingoftheproblemas
completelyaspossiblebeforecodingbegins.Thisenablesnotjustthedeveloperstoproceedwith
theirworkinamoreconsistentway,butallowsforparallelworkbytestdesignersandproject
management,essentialfortimelyprojectdeliveryandmanaging.Forthegreatmajorityofsystems
whichinvolveusersanduserinterfaces,alotofthisinformationtakestheformofusecases.For
systemsthatrunwithlittleornouserinteraction,usecasetechniquesarenotnecessarilythemost
appropriate.
Themodelisconfusingtothosewhotrytoreadandunderstandit
Agoodmodelshouldexplaintherequirementsandintendedbehaviorofthesystemclearlyand
concisely.Ifthemodelisnoteasytofollowandunderstand,itmaybeasignthattailoringis
required.
AsTimandIsatinhisofficediscussingthereasonsthatweshouldtailortheprocessforLSSTright
now,itwassomewhatproblematicbecauseJeffhadmadeitveryclearthathedidntthinkthere
wasenoughtimetotakeonanysortofprocessmodificationsbeforePDR.Soitwasabitofa
dilemma.FinallyIsuggestedtoTimthatheandIconsiderworkingasmallexampleandshowingit
toJeffinthehopeshewouldagreethatitwasobviouslyanimprovement.Tim,whoyoumayrecall
hadsuggestedprocesstailoringinthefirstplace,agreed.Weinitiallyundertooktoconvertone
pipeline(InstrumentSignatureRemoval)intothenewformattotryitonforsize.
Bythistime,havinghadseveralmonthstoreflectontheproblem,Ihadprettywellformedanidea
ofhowIthoughttheprocessshouldbetailored.IhadaprecedentfortailoringICONIXProcessfor
BusinessProcessModelingwherewehadusedactivitydiagramstodescribeworkflowsinthe
ICONIXBusinessModelingRoadmap.
27

Thishadproventoworkprettywell,andthemoreIlookedattheusecasediagramsfortheLSST
pipelinesthemoreconvincedIbecamethattherewasafundamentaldifferencebetweenthe
pipelines,whichdefinedtheworkflowforastreamofimages,andthesciencealgorithms,which
performmathematicaloperationsontheimages.Ithoughtthemodelwouldbemuchless
confusingifwemodeledtheworkflowsonactivitydiagrams.

27
www.iconixsw.com/BPRoadmap.html
34
ModelingTip:Guidelinesfortailoringadevelopmentprocess
Theoverridingruleinprocesstailoringisfittheprocesstotheproblem(andnotviceversa).
ICONIXProcessusesacoresubsetofUMLdiagrams,notbecausetheotherpartsofUMLarenever
useful,butbecausethissubsethasproventoworkconsistentlyandeffectivelywhileavoiding
analysisparalysisonawiderangeofprojects.
WhenaprojecthasneedforadditionalUMLdiagramtypes,ICONIXProcesshasalways
recommendedtailoringuptoincludethem.Ifweoverzealouslysticktothe4diagramtypes
(class,sequence,usecase,androbustness)inthecoreICONIXsubset,weruntheriskofeverything
lookslikeanailbecauseallwevegotisahammer.Inthiscasethenailisausecase,andthe
hammerisausecasediagram.
Notethatstartingfromaminimalistmodelingsubsetandtailoringupisaverydifferentapproach
fromthattakenbyprocessessuchasRationalUnifiedProcess(RUP),whichstartwitheverything
andrequirechippingouteverythingthatsnotneeded.Generally,tailoringdowninvolvesalot
moreworkthantailoringup.
TheoverridingissuewithapplyingICONIXProcesstotheLSSTDMsoftwareisthatICONIXProcessis
fundamentallyausecasedrivenapproachtoobjectorienteddesign,andtheLSSTDMsoftwarehas
veryfewrealusecasesbutisinsteadheavilyalgorithmintensive.
ThetailoringIproposedtoTimwasactuallyaprettysimpleextensionwhichretainedallthe
benefits
28
ofusecasedrivenICONIXProcesswhilebeingalotlessconfusingtoeveryoneinvolved.
Thesechangeswere:
1) Dontdescribeanythingotherthanarealusecasewithusecases.Specifically,dont
describeworkflowasusecases,anddontdescribealgorithmsasusecases.LSSTwillhave
realusecases;manyofthesewillinvolveusingthedataproducedbytheimage
processingpipelines.Butthepartofthesoftwarewemodeledisalmostcompletelydevoid
ofthem.
2) Describeworkflowusingactivitydiagrams.Essentially,forLSST,thismeantthatPipelines
andProductionswouldbedescribedusingactivitydiagrams.
3) Describehighlevel,policydirectedsciencealgorithmsonrobustnessdiagrams.
RecognizethatpolicyisessentiallyaproxyforahumansittinginfrontofaGUIdriving
theimageprocessing,andtreatitasanactor.Thisallowstherobustnessdiagramrulesto
beusedcorrectly.
4) Describemoreatomicnumbercrunchingsciencealgorithmsascontrollerson
robustnessdiagrams.SoeachcontrollerwillspecifyInputs,Outputs,Algorithm,and
Exceptions.

ThistailoringofICONIXProcessiswidelyapplicablebeyondLSSTtoanentireclassofalgorithm
intensivesystems.WepresentithereasICONIXProcessforAlgorithmIntensiveSystems(or,ICONIX
ProcessforAlgorithms).

28
Another useful principle to follow is: Fix whats broken, but dont try to fix what isnt broken. What wasnt
broken about ICONIX Process was using robustness diagrams to identify missing domain objects and identify
controllers (science algorithms), and using sequence diagrams to do a responsibility-driven allocation of behavior
to objects. J eff, Tim, and I were all unanimously agreed on not sacrificing these benefits. We just needed to find a
way to make them work when there were no real use cases.
35
AFewMoreThoughtsAboutICONIXProcessforAlgorithmsasUsed
onLSST
Modelingpipelinesasactivitydiagramsinvolvednotonlytransmogrifyingthediagramfroma
usecasediagramtoanactivitydiagram,butalsoincorporatingPolicyasanactorwhichdefined
pathsthroughthevariouspipelinestages.AlthoughtheLSSTDMsoftwarewillrunwithouthuman
intervention,variouspredefinedPoliciesactasproxiesforhowahumanuserwouldguidethe
software.AsitturnedoutonLSST,thereweretwoparallelsetsofimageprocessingpipelinesthat
differedonlyinthepoliciestoguidethem,somakingthepipelineactivitydiagrampolicydriven
immediatelyallowedustocutthenumberofpipelineusecasediagramsinhalf.Thiswasan
encouragingsignasanimmediatesimplificationofthemodelresultedfromtheprocesstailoring
wedid.
Modelingpipelinestagesashighlevelalgorithmsmeantreplacingtheschizophrenicalgorithm
usecasetemplateof
Inputs:
Outputs:
BasicCourse:
AlternateCourses:
Withanactivityspecificationtemplatemoresuitedtoalgorithms,namely:
Inputs:
Outputs:
Algorithm:
Exceptions:
Notsurprisingly,writingalgorithmdescriptionsasalgorithmsandnotasscenariosmadethemodel
mucheasiertounderstand.Thissimpleprocessmodificationwentalongwaytowardsaddressing
thelackofsemanticconsistencyinthemodel.
Weusedrobustnessdiagramstoelaborateactivities(isthatlegal?)
29
Thealgorithmusecases
thathadbeenwritteninPasadenahadbeenelaboratedonrobustnessdiagrams,andwemadethe
nonstandardprocessenhancementtoelaboratethepipelinestageactivitieswiththeserobustness
diagramsaswell.EnterpriseArchitectwasflexibleenoughtosupportthis.
ModelingTip:Goodmodelingtoolsareflexible
Ivebeenbendingtherules(andwritingnewones)ofsoftwaredevelopmentprocessesformore
than20years.OneofthekeyattributesthatIlookforinatoolisflexibility.Overtheyears,Ive
foundthatIcanmakeEnterpriseArchitectdoalmostanything.Ithelpsme,butdoesntgetinmy
way.
Keepingthiselaborationofpipelinestagealgorithmsonrobustnessdiagramswasimportantfora
numberofreasons,oneoftheprimaryreasonsbeingthatwewantedtomaintainthe
decompositionintocontrollers(lowerlevelalgorithms)andentities(domainclasses).Another
importantreasonwasthatprojectestimationtoolsandtechniquesreliedonthenumberof
controllerswithinagivenpipelinestage(andanestimateoflevelofeffortforeachcontroller)for
costandscheduleestimation.

29
Luckily I have a license from the UML police to make these sorts of non-standard process enhancements.
36
Notethatsincethedomainmodelhadnotbeenconsolidatedwhentheoriginalrobustness
diagramswerecreated,thesediagramshadtoberevisitedanyway(sincethedomainclasses
appearasentitiesontherobustnessdiagrams).Notsurprisingly,numerousothererrorswere
uncoveredduringthiscleanuppassthroughthepipelinestagealgorithms.Allofthisworkwas
absolutelynecessaryinordertogettheprojectreadyforPDR.
SomeFinalThoughtsonProcessTailoring
Whileitsgenerallyagoodthingtogiveaprocessafairchancetowork,andwevefoundthat
ICONIXProcessoutoftheboxworksverywellforaverywiderangeofprojects,itsalsoadvisable
tobealertforanyfundamentalissuesthatmightrequiretailoring.InthecaseofLSSTDM,itproved
toocumbersome(althoughnotimpossible)tobendtherulesandmodelalgorithmsasusecases.
Whileweknewfromthebeginningthatwewerestretchingandtwistingtherulesduringthe
Pasadenaworkshop,theteamstillmanagedtoproducealargeamountofusefulwork.Inhindsight
itmighthavebeenadvisabletotailortheprocessabitearlier,but,aswithmanythings,theissues
becameclearerasmoreprogresswasmade.
Themostimportantconclusionistonotbeafraidtotweaktheprocessifitsnotworkingaswell
asitshould.NeitherICONIXProcessnoranyothersoftwareprocessischiseledonstonetablets,
andtheyshouldneverbetreatedasiftheywere.
Ifitaintbrokedontfixit,butifitisdonthesitatetodowhatsnecessarytomakeyourprocess
workforyou.
AllsWellThatEndsWell
Asthingsturnedout,theLSSTPDRdatewasslippedbythefundingagencies,allowingtimeforTim,
withassistancefromRobynandacoupleofworkingsessionswithme,toupdatetheentiremodel
inthisfashionwithexcellentresultssomeofthisupdatingisstillinprogressasImwritingthis.
Wellusethealgorithmicallytailoredversionofthemodeltoshowhowtodetectasteroidsthat
mighthittheEarth(andothermovingobjects)inthenextchapter.

37
Chapter5
HowDoYouDetectanAsteroidThatMightHitthe
Earth?

ThischapterpullstogetherafewpiecesoftheLSSTmodeltotellthestoryofhowtheLSSTsoftware
willidentifymovingobjectsfromitstorrentofimages.
DomainModel
LetsstartwithasmallfragmentfromtheLSSTDomainModel,focusingonAstronomicalObjects
(AstroObjects).Thedomainmodel,inadditiontoitslargemainalmosteverythingdiagram,alsois
organizedintosubsetdiagramsshowingclustersofrelatedobjects.Itsoftenaloteasiertolookat
thesubsetdiagrams,andthatsthecasehere.
YoucanseefromFigure1thatweveidentifiedaSolarSystemObjectasadistinctkindofAstroObject,
andthatSolarSystemObjectshaveOrbits.Wevealsoidentifiedsomethingmysteriouscalled
Ephemerides.Asitturnsout,Ephemeridesareimportanttoourstory.But...(asIaskedTimona
recentvisittoTucson)whatsanEphemeride?

Figure1.DomainmodelshowingdifferentkindsofAstronomicalObjects.
HereswhereTimhavingdefinedhisdomainobjectscomesinhandy.WecanjustlookitupunderE
forEphemeride.Usingthedomainmodelasaglossaryisusefulonanyproject,butwhenyouhave
adistributedteamlikeLSST,itsabsolutelyessential(seeFigure2).
38

Figure2.UsingtheDomainModelasaProjectGlossaryisaveryusefultechnique.
Asyoucansee,anEphemerideisaTimeSeriesofpredictedSkyCoordinatesforaSolarSystem
Object.Inotherwords,thepredictedtrackofanobjectinthesolarsystem(likeaMartian
Spacecraft,oranasteroidthatmightbeheadingforEarth,ortheplanetNeptune,oroneof
Saturnsmoons,oreventheHubbleSpaceTelescope).Inthewordsofanoldelectromagnetic
theoryprofessorofmine,thisisintuitivelyobvioustothecasualobserver.
ModelingTip:ItallstartswithDomainModeling
IncomplexsystemslikeLSST,theDomainModelcanbecomequitelarge.Whenthishappens,
createsubsetviewdiagramsthatshowarelatesetofobjects.Dontforgettodefineallyour
domainclassesunambiguously.
Sofar,sogood.TheLSSTDomainModelhasanobjectthatexistsspecificallyforthepurposewe
haveinmind.NowletstakealookathowweregoingtocomputetheseEphemerides.
39
ImageProcessingPipelines
LSSTsimageprocessingsoftwareusesapipelinearchitecture.Imagesgoinoneendofthe
pipelinethroughanInputQueue,andareanalyzedastheypassthroughvariousprocessingstages,
thenexitthroughanOutputQueue.LSSTsmiddlewaredefinesageneralpurposearchitecturefor
pipelineswhichallowsforparallelprocessingoftheimagestream.Parallelprocessingisanabsolute
necessitywhenyouredealingwithastreamof3gigapixelimageswithanewimagecoming
througheveryfewminutes.WeregoingtobelookingatoneofmanyLSSTpipelineslaterinthis
chapter,apipelinecalledDayMOPS.

Figure3.LSSTsmiddlewaremanagestheimageprocessingpipelines.
Policies
LSSTssoftwarewilloperateatmuchtoohigharatefortheretobehumanguidanceanddirection
duringtheexecutionofapipeline.However,therearemanyoccasionswherehumanguidanceis
necessary.LSSTpipelinescanbecontrolledbyPolicies,whicharesetsofparametersthathuman
experts(astrophysicists)candefine.SoaPolicyisreallylikeaproxyobjectthatreplacesaperson
whowouldbeguidingtheimageprocessingsoftwareifyousloweddowntheprocessingbya
coupleofmilliontimes.(SeeFigure4).
40

Figure4.LSSTspipelinesarepolicydriven.Policyobjectsaresetsofparametersthateffectivelyactasproxiesforexpert
astrophysicists.Theseparametersguidetheimageanalysis,sincetheimageprocessingrunsfartooquicklytoallowfor
actualhumanintervention.
TheDayMOPSPipeline
SinceLSSTPipelinesareusedtodefineworkflow(andarenotreallyusecases)wereusingan
activitydiagramtodescribetheworkflow,insteadoftryingtodescribethepipelinedetailonause
casediagram.Soitsclearinthemodelthatwhenweseeanactivitydiagram,wereworkingatthe
upperlevelsofthemodel.
Figure5showstheactivitydiagramthatdescribesourmovingobjectdetectionpipeline.

Figure5.TheDayMOPSpipelinedetectsmovingobjectsandupdatesseveralofLSSTsCatalogs,includingtheSolar
SystemObjectCatalog.
41
NearEarthObjectDetection
LSSTsDayMOPSpipelineisbeingdevelopedinclosecollaborationwithaprojectcalledPAN
30
STARRS (PanoramicSurveyTelescope&RapidResponseSystem).ThePANSTARRSwebsitehasan
interestingdiscussionaboutNearEarthObject(NEO)detectionandthepotentialthreatfrom
31
asteroids.
Also,NASAJPLmaintainsaNearEarthObjectwebsiteifyoudliketotakeapeekatwhichasteroids
32
willbemakingclosepassestousinthenearfuture .ThePANSTARRSwebsiteliststheirdigital
33
camerasastheworldslargest,at1.4gigapixels ;thiswontbetrueanymoreafterLSSTs3.2
34
gigapixel(thats3200megapixels) cameragetsbuilt.
ALookInsidethePipelineStages
Withineachactivity,wereelaboratingtheprocessingonarobustnessdiagramasiftheactivity
wereausecase,andwererepresentingPolicyasanactor,and(tomakeourrobustnessdiagram
rulesworkcorrectly)definingPolicyReadersasboundaryobjects.Sincetherobustnessdiagramisa
conceptualdesigndiagram,itdoesntreallymatterwhetherapolicyreaderobjectwilleverbe
implemented.Wereusingitinthemodelasadevicetohelpusidentifywhatthedifferentpolicy
parametersshouldbe.
ModelingTip:RobustnessdiagramshelpObjectDiscovery
Robustnessdiagrams(akaMartian)areconceptualdesigndiagramsthatareusefultohelpyou
discoverdetailsaboutyourobjectmodel.Sinceitsnotanimplementationmodel,itsOKtoadd
conceptualobjectslikePolicyReaderstohelpusdiscoverwhatdatawerereadingfromthePolicy.
SeeFigure6foranexampleofarobustnessdiagram.
Byelaboratingpipelinestageswithinactivities,andallowingforcontrollers(lowerlevel
algorithms)withintherobustnessdiagram,wereputtingalimitonhowmanylevelsofalgorithms
withinalgorithmswithinalgorithmswithinalgorithmsweregoingtoshowinthemodel.The
controllersthatareconnectedtothePolicyReadershelpustoidentifyexactlywhatneedstogointo
thePolicy.
ModelingTip:Avoiddeeptreesofnestedfunctions
Oneoftheproblemswithfunctionaldecompositionapproachesthataretypicallyusedin
algorithmintensivesystemsisthattheretendtobemanylevelsofnestedalgorithms.ICONIX
ProcessforAlgorithmsrepresentshighlevel(workflow)algorithmsonactivitydiagrams,elaborates
policy/parameterdrivenactivitiesonrobustnessandsequencediagrams,andlowerlevelnumber
cruncheralgorithmsascontrollerswithintherobustnessdiagrams.
Soitsmucheasiertofigureoutwhatlevelofalgorithmyourelookingatwhenyoureadthemodel.

30
http://pan-starrs.ifa.hawaii.edu/public/
31
http://pan-starrs.ifa.hawaii.edu/public/asteroid-threat/asteroid_threat.html
32
http://neo.jpl.nasa.gov/
33
http://pan-starrs.ifa.hawaii.edu/public/design-features/cameras.html
34
http://www.lsst.org/lsst/gallery/camera/suzanne

42

Figure6.ArobustnessdiagramforTrackletCreation.Martiannotationisactuallyprettysimple.
Noticethateventhoughwereusingittodescribeapolicydirectedalgorithminsteadofausecase,
therobustnessdiagramstillworks.ThisallowsustoleverageothercapabilitiesofICONIXProcess
andtheSparxAgile/ICONIXaddintogenerateskeletonsequencediagramsandtestcases
automatically.
ModelingTip:ICONIX/AlgorithmsretainsallthebenefitsofICONIX
Process
Thestandard,usecasedriven,ICONIXProcessdiscoversdetailsaboutanobjectmodelusing
robustnessanalysis,anddoesaresponsibilitydrivenallocationofbehaviorusingsequence
diagrams.Additionally,theAgile/ICONIXaddinfromSparxSystemssupportsautomaticgeneration
oftestcasesandJUnit/NUnittestcodefromrobustnessandsequencediagrams.ICONIXProcessfor
Algorithmsretainsallofthesebenefits.
ICONIXProcessforAlgorithmsisusefulforawiderangeofalgorithmintensivesystems.
43
Figure7showsanalgorithmforapipelinestagethathasntbeenelaborated(yet)onarobustness
diagram.ExactlyaswithusecasedrivenICONIXProcess,drawingtherobustnessdiagramhelpsus
discoveradditionaldomainobjectsthatmaystillbemissingfromthemodel.Soourprocess
tailoringhasretainedthebenefitsofusecasedrivenICONIXProcess,withouttryingtoforcefit
algorithmsintousecases.

Figure7.ThisalgorithmforcomputingEphemerideswillsoonbedetailedonarobustnessdiagram.
44
ConvergingTowardsaSolution
Youcanseethebenefitsofaforwardmodelingandreverseengineeringapproachhere,witha
reverseengineereddatabaseschemaforMOPS(seeFigure8).

Figure8.ReverseengineeredMOPSdatabaseschema.
ByusingLSSTsR&DPhaseforbothmodelingandprototypingdifferentareasoffunctionality,the
projectminimizesrisk.WhenLSSTreachestheConstructionphase,therewillbeverylittledoubtof
asuccessfuloutcome.
ModelingTip:Modeling+ReverseEngineeringminimizesrisk
Onceagain,youcanseehowthecapabilitiesofthemodelingtool(inthiscase,EAsabilityto
reverseengineerdatabaseschemasinadditiontoawiderangeofprogramminglanguages)allows
forariskmitigationstrategythatscrucialforaprojectofthecomplexityofLSST.

Conclusion
DetectingasteroidswhichmightimpacttheEarthisasmall,butimportant,portionofLSSTsoverall
sciencemission.Hopefully,youvebeenabletofollowthediscussionpresentedinthischapterthat
explainshowthiscapabilitywillbeprovidedbytheDayMOPSimageprocessingpipeline.Ifyou
have,thatsagoodsignthatourtailoringofICONIXProcesshasresultedinaUMLmodelthat
successfullyandunambiguouslyexplainsthedesignoftheLSSTsoftware.

45