You are on page 1of 251

AcceptanceTest

EngineeringGuide
VolumeI:ThinkingaboutAcceptance
HowtoDecideifSoftwareisReadyforYou
andYourCustomers

GrigoriMelnik,GerardMeszaros,JonBach

ForewordbyKentBeck

ReleaseCandidate1October26,2009

COMMUNITYPREVIEWLICENSE
Thisdocumentisapreliminaryreleasethatmaybechangedsubstantiallypriortofinalcommercial
release.ThisdocumentisprovidedforinformationalpurposesonlyandMicrosoftmakesnowarranties,
eitherexpressorimplied,inthisdocument.Informationinthisdocument,includingURLandother
InternetWebsitereferences,issubjecttochangewithoutnotice.Theentireriskoftheuseorthe
resultsfromtheuseofthisdocumentremainswiththeuser.Unlessotherwisenoted,thecompanies,
organizations,products,domainnames,emailaddresses,logos,people,places,andeventsdepictedin
exampleshereinarefictitious.Noassociationwithanyrealcompany,organization,product,domain
name,emailaddress,logo,person,place,oreventisintendedorshouldbeinferred.Complyingwithall
applicablecopyrightlawsistheresponsibilityoftheuser.Withoutlimitingtherightsundercopyright,
nopartofthisdocumentmaybereproduced,storedinorintroducedintoaretrievalsystem,or
transmittedinanyformorbyanymeans(electronic,mechanical,photocopying,recording,or
otherwise),orforanypurpose,withouttheexpresswrittenpermissionofMicrosoftCorporation.

Microsoftmayhavepatents,patentapplications,trademarks,copyrights,orotherintellectualproperty
rightscoveringsubjectmatterinthisdocument.Exceptasexpresslyprovidedinanywrittenlicense
agreementfromMicrosoft,thefurnishingofthisdocumentdoesnotgiveyouanylicensetothese
patents,trademarks,copyrights,orotherintellectualproperty.

2009MicrosoftCorporation.Allrightsreserved.

MicrosoftaretrademarksoftheMicrosoftgroupofcompanies.
Allothertrademarksarepropertyoftheirrespectiveowners.

TableofContents

ForewordbyKentBeck 1
Preface:AcceptanceTestEngineeringGuide 3
Introduction 10
ACautionaryTale 20
PARTI.ThinkingAboutAcceptance
Chapter1 TheAcceptanceProcess 27
Chapter2 ElaboratingontheAcceptanceProcess 53
Chapter3 DecisionMakingModel 71
Chapter4 ProjectContextModel 82
Chapter5 SystemsRequirementsModel 91
Chapter6 RiskModel 99
Chapter7 DonenessModel 104
PARTII.PerspectivesonAcceptance
Chapter8 BusinessLeadsPerspective 119
Chapter9 ProductManagersPerspective 131
Chapter10 TestManagersPerspective 138
Chapter11 DevelopmentManagersPerspective 143
Chapter12 UserExperienceSpecialistsPerspective 148
Chapter13 OperationsManagersPerspective 153
Chapter14 SolutionArchitectsPerspective 156
Chapter15 EnterpriseArchitectsPerspective 159
Chapter16 LegalPerspective 161
PARTIII.AcceptingSoftware
Chapter17 PlanningforAcceptance 166
Chapter18 AssessingSoftware 190
Chapter19 ManagingtheAcceptanceProcess 207
Chapter20 FinetuningtheAcceptanceProcess 219
Appendices
AppendixA OrganizationStereotypesandReaderPersonas 231
AppendixB KeyPoints 237

Foreword

Ifigureit'smyjobinaforewordtotellyouwhyIthinkabookisworthreading.Oftenthisinvolves
settingcontext,placingthebookwithrespecttohistoricalorsocialortechnicaltrends.SometimesI
pointouthowtheauthorshavebeentoohumbleintheirpresentation.SometimesItalkaboutwhatI've
learnedfromabook.InthiscaseIamabletouseallthreetechniques.
Technologistshavespentthelasthalfcenturyswingingonapendulumbetweenbeinglowlyserfs
obeyingthecommandsoftheirbusinessmastersandactingashighpriestsatthealtarofComputing.
Thelastdecadehasseenthefirsttentativestepstowardsamorerealisticandsustainablepositionof
partnershipwithbusinesssponsorsandusers.Thenewemphasisonacceptancetestingisevidencethat
thistrendcontinues.
Isay"emphasis"becauseacceptancetestingisnotnew.Eversincethefirstusertoldthefirst
programmer"That'snotquiteright,"acceptancetestinghasbeenpartofthedevelopmentworkflow.
Sincethosefocusedontechnologyandthosefocusedonbusinessbringdifferentvalues,expectations,
andvocabulary,somemisunderstandingisinevitable.What'sthatProfessorHeisenberg?Oh,yes,thanks
forthereminder.Theappearanceofasystemautomatically(itseems)changesitsrequirements.
Betweenthesefactors,acceptancetestingispartofeffectivedevelopment.
Legitimatelyshiftingrequirementshaveaprofoundeffectonthetiming,role,andtechniquesof
acceptancetesting.Oneofthetrendsindevelopmentoverthepastdecadeisthattestingtakesplace
soonerandmorefrequentlyindevelopment.Evenwiththebulkofacceptancetestingshiftedtojustin
advanceofdevelopment(AcceptanceTestDrivenDevelopment),someonestillneedstobelisteningfor
"Thanksforthat,butwhatIreallywantis..."
Oneimplicationofthegrowingfrequencyofacceptancetestingistheneedforautomationofrepetitive
testingtasks.Thisisnotajobfortestersalone.Toincreasetherateoffeedback,programmersmust
maketheirsystemsmoretestable.Automationfreesresourcesforhigherleveragedmanualtesting.
OnethingIappreciatedinthisbookwastheconsistentconnectionbetweenacceptancetestingandrisk
management.Acceptancetestingisworthwhileforthevalueitaddstodevelopment:increased
probabilityofcustomersatisfaction,reducedriskoferrors,andbettertrackingofprogress.Theimageof
riskmanagementasawaytobuyreactiontimewillstickwithme.Earlyincrementalacceptancetests
canbethescoutssentouttoinformthearmyoftroubletocome.
Oneofthestrengthsofthisseriesofbooksisthevarietyofformatsfortheinformation.Thefirst
volumepresentstheintellectualfoundationsofacceptancetesting.Remainingvolumespresent
concretepracticesthatworkwithinthatframework,amenuof"Here'stechniquesthathaveworked"as
wellasaworkedexampleforthoselookingtounderstandacceptancetestingincontext.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 1

I'llclosebypointingoutwhereIthinkGrigori,Gerard,andJonshumilitygotthebetterofthem.Whatif
softwaredevelopmentwasturnedonitshead?Whatifthewholeprocessworkedbackwardsfrom"The
customerisdelightedwiththesystem"?Beforeyoucouldexpecttohearastatementlikethat,you'd
needtospendalotoftimetestingthesysteminrealisticenvironments.You'dneedtomakesureallthe
easilyunderstoodfeaturesworkedasexpected.Acceptancetestingandacceptancetestdriven
developmentcanencouragethis.Novelfeaturesandinteractionsoffeatureswouldalsoneedtoworkin
asensibleway,anothergoalacceptancetestingcancontributeto.Whenthose"Great,butwhatI'd
reallyliketoseeis..."momentshappen,incrementalacceptancetestingcanhelpthemoccurearly
enoughthatthewholeteamcanrespond.
Astheconcretemanifestationofthedialogbetweenbusinessandtechnology,acceptancetestingcan
playacentralroleinsoftwaredevelopment.Hereinyourhandsisabookthatdistillsyearsof
experiencewithacceptancetesting,abagofideasyoucanreachintoagainandagainasyoutravel
towards:"Thecustomerisdelightedwiththesystem..."
KentBeck,June2009

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 2

Preface:AcceptanceTestEngineering
Guide

WhyWeWroteThisGuide
Acceptingsoftwareisoneofthemostimportantdecisionsforsoftwaretotakeflightandtostart
deliveringvalue.Yet,thereexistslittleguidanceaboutwhatthisdecisionshouldbebasedonandhowto
makeiteffectively.
Overmanyyearsinindustryandappliedresearch,itwasourobservationthatpeopleoften
misunderstoodthechallengesofacceptingsoftware,orworse,didnotevenhaveaclearnotionofwho
isinchargeofacceptance.Manyappearnottobeawareofthearsenalofpracticesandtoolsavailable
tothemtoaddressthosechallenges.Thisledtodisappointments,failedopportunities,brokenbusiness
relationships,andlitigations.Thereexistsarichbodyofknowledgeonsoftwaretesting(withnotable
worksbyBorisBezier,BertrandMeyer,GeraldWeinberg,CemKaner,JamesBach,BrettPettichord,Lee
Copeland,MarkFewster,DorothyGraham,RexBlack,BrianMarick,JamesWhittakertonameafew),
butsoftwareacceptanceingeneral,andacceptancetesting,inparticular,asaprocess,receivedno
serious,wellreasonedorcomprehensivetreatment.Wesetouttofillthisgapwithtwokeyobjectives:
devisementalmodelsforthinkingaboutsoftwareacceptanceandofferactionableguidanceon
achievingsoftwareacceptance.

WhoShouldReadThisGuide
Thisguideisintendedforanyonewhoisinvolvedintheprocessofdecidingthedegreetowhicha
softwareintensiveproductmeetstheacceptancecriteriaofthosewhocommissioneditsconstruction.
Specifically,thisguidewillhelpyouifanyofthefollowingapplytoyou:
Youareinvolvedindecidingwhethertoacceptthesoftwareasithasbeenbuilt.Thisisthe
acceptancedecision.
Youareinvolvedindecidingonwhatacceptancecriteriashallbeusedtomaketheacceptance
decision.
Youareinvolvedincollectingdatathatthepersonmakingtheacceptancedecisionrequiresto
makethatdecision.Thisisacceptancetesting.
Youareinvolvedindecidingwhethertheproductisreadytobeseenbythepeopleinvolvedin
theacceptancedecisionoracceptancetesting.Thisisthereadinessdecision.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 3

Youareinvolvedincollectingdatathatthepersonmakingthereadinessdecisionrequiresto
makethatdecision.Thisisreadinessassessment.
Youareinvolvedindefiningtheexpectationsagainstwhichthereadinessassessmentor
acceptancetestingactivitieswillbeconducted.Thisisacombinationofrequirements
gathering,testplanningandtestdesign.
Youareinvolvedinmanaginganyoftheprecedingactivities.

Thisguidedescribesthepracticesusedbypeopleintheprecedingroles,regardlesswhatyourjobtitle
is.Ifanyofthemdescribesyourrole,youshouldfindsomethingofinterestinthisguide.Appendix
ReaderPersonasdescribesourtargetaudienceinmoredetail.Thesepersonasareusedto:
remindtheauthorsandreviewersofourprimarytargetaudience.
giveyou,thereaderafiguretoempathizeorassociateyourselfwith.Thisshouldhelpyoupick
thepersonathatmostcloselyresemblesyourownjobresponsibilities.
IfanyofthegoalsdepictedinFigure1areapplicabletoyou,theguideandtheonlineknowledgebase
companion(TestingGuidance.codeplex.com)areforyou.


Figure1.
Readergoalsandpathsthroughtheguide.

GuideStructure
Thisguideisstructuredasafourvolumeseries(Figure2):
VolumeIprovidesanoverviewoftheacceptanceprocessandhowacceptancetestingandother
keypracticesfitintotheprocess.Thisvolumeisintendedtobereadfrombeginningtoend.
Itissubdividedintothreemainparts:
VolumeIIcoversAcceptancePlanningandManagementpracticesandpatterns.
VolumeIIIcoversFunctionalAcceptanceTestpracticesandpatterns.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 4

VolumeIVcoversOperationalAcceptanceTestpracticesandpatterns.Operationalacceptance
includesnotonlyperformance,recoverability,availability,securitybutalsotestingofother
nonfunctionalrequirementsandqualities,suchaslearnability,usability,supportability,
upgradeabilityetc.


Figure2.
GuideStructure.
VolumeIisintendedtobereadasanintroductiontotheconcepts,termsandpracticesdescribedin
VolumesIIIV.VolumesIIIVareintendedtobeusedasareference;mostreaderswillnotreadthem
frombeginningtoend.
VolumeIThinkingAboutAcceptanceconsistsof3parts:
PartIThinkingaboutAcceptanceexplainssixmentalmodelsthatareusefulwhen
thinkingabouttheacceptanceprocess.
PartIIPerspectivesonAcceptancedescribestheacceptanceprocessfromthe
perspectivesofkeystakeholdersintwodifferentkindsoforganizations:theInformation
TechnologyDepartmentinabusinessandtheProductDevelopmentCompany.Most

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 5

readersinvolvedintheacceptanceprocessshouldfindsomecommonalitywithatleast
oneoftherolesdescribes.
PartIIIAcceptingSoftwareintroducesthepracticesthatarenecessaryforplanningthe
acceptanceprocess,forperformingacceptancetestingandforimprovingthe
acceptanceprocess.

VolumesIIthroughIVcontainacollectionofthumbnailsandsamples:
AThumbnailisashortoverviewofapracticethatexplainswhatitis,whenyoumaywanttouse
it,therisksthatitmitigates,andanoverviewofhowtoperformthepractice.Thumbnails
alsoincludealistofreferencestopapers,books,andotherresourcesthatprovidemore
completedescriptionsofthepracticesinquestion.Themainpurposeofathumbnailisto
describeatopicwellenoughtoprovideanoverview,serveasamentalreminderfor
someonewhohasusedthepracticeonhowtodoit,andgivesomeoneunfamiliarwiththe
practiceenoughinformationaboutthepracticeanditsapplicabilitytodetermineifthey
wanttolearnmoreaboutit.Someofthesetopicsandpracticeshaveentirebookswritten
aboutthemthatdescribetheconceptsingreaterdetailanddepththanthisguidecould
possiblydo.
ASampleisaanartifactgeneratedbyapplyingdifferentpracticesinafictionalbutrealistic
situationforthefictionalcompanyGlobalBank.Theseartifactsareembeddedinaseriesof
casestudiesofwhattheGlobalBankteammayhaveproducedwhilebuildingthe
application.Thecasestudiesprovidesomecontexttotheindividualartifacts.Theyalso
providecrossreferencestothethumbnails.Theartifactsareintendedtobeusedaswayto
learnmoreabouthowtoperformapractice;theycanalsobeusedastemplatesforyour
ownartifacts.

HowtoReadThisGuide
Thewayyouapproachthisguidewilldependonwhatroleyouhaveandwhatyouwanttolearnabout
theprocessofacceptingsoftware.Dependingonwhatyouwanttodo,youwillwanttoapplydifferent
strategies.

GetanOverviewofAcceptancePracticesandProcesses
StartbyreadingVolumeIifyouwanttodoanyorallofthefollowing:
Learngeneralinformationaboutacceptancetesting.
Findacceptancetestingpractices.
Createaprojectplanthatincorporatessoftwareacceptance.
Justifyaprojectplanthatincorporatessoftwareacceptance.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 6

Justifyanapproachusedforacceptancetesting.
Validatethatyouareontrackwithyouracceptancetestingstrategyorapproach.
Getyourprojectunstuckwhensoftwareisunacceptable.
Determinewheretheremaybegapsinyouracceptancetestingapproachorstrategy.

AfterreadingVolumeI,youmaywanttoskimparticularpracticesofinterestandthecorresponding
samplesinVolumesIIthroughIV.

DecideWhichAcceptancePracticestoUseonYourProject
StartbyreadingVolumeItogetanoverviewofpossiblepractices,andthenrefertothethumbnailsin
VolumesIIIVforspecificpracticesyouareconsidering.Eachthumbnailincludesasectiontitled"When
toUseIt,"whichincludesadviceaboutwhenthepracticeshouldbeused,andasectiontitled
"Limitations,"whichprovidesguidelinesaboutwhenthepracticeshouldnotbeapplied.

LearnHowtoPerformaSpecificAcceptancePractice
StartbyfindingathumbnailinVolumesIIIVifyouwanttodoanyofthefollowing:
Learnaspecificacceptancetestingpracticeorstrategy.
Teachaspecificacceptancetestingpracticeorstrategytosomeoneelse.
Reviewaspecificacceptancetestingpractice.
Findmoreinformationandrelatedresourcestoconsultaboutaparticularpractice.

Afteryoulocatethethumbnailforthespecificpracticeyouwanttolearnabout,readitandanyrelated
samples.Ifyouneedmoredetailedinformationaboutthepractice,seethe"References"sectioninthe
thumbnail.

GetaTemplateforaSpecificArtifact
StartbyfindingansampleinVolumesIIIVifyouwanttodoanyofthefollowing:
Findatemplateforaspecificartifact.
Learnhowtofillinaspecificartifact.

FindtheexampleyouwantinVolumesIIIV,removethesampleinformation,andpopulateit
appropriately.Ifyouneedtoreviewthepracticethatgeneratedtheexample,theexamplelistsallthe
appropriatethumbnailsinVolumesIIIV.

PlantheExecutionofthePracticesonYourProject
StartbyreadingVolumeItogetanoverviewofhowthepracticesfittogetherandsupporteachother.In
particular,thesectionsontheAcceptanceProcess,theDecisionMakingModel,andtheDoneness
Modelmaybeofparticularinterest.Afterthat,reviewthespecificthumbnailsinVolumesIIIV,paying

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 7

particularattentiontothesubsection,"TestLifeCycleApplicability"inthesection,"WhentoUseIt."
Eachsampleartifactisaccompaniedbyanotationthatindicatesatwhatpointinthehypothetical
projecttheartifactwasproduced.Notethatsomeartifactsappearatseveralpointsintheproject
timelinebecausetheyevolveovertime.Ifyoufindyouracceptanceprocesstakestoolongyoumayfind
theStreamliningAcceptanceProcesschapterofVolumeItobehelpful.

FindToolsforAcceptanceTesting
Ifyouarelookingfortoolstoperformacceptancetesting,youmayusetheguidetoexplorethe
availablespace.Althoughsomeofthecasestudiesillustrateusingspecifictools,rememberthatthe
primaryfocusofthisguideisondescribingpractices.
Thechoiceoftoolsusedwhileproducingthisguideshouldnotbeinterpretedasanendorsementofany
tool,norshoulditbeinterpretedasanindicationthatanytoolusedisthebestoneforthejob.Bythe
timeyoureadthisguide,thetoolsillustratedinthesamplesintheguidemayhavebeensupplantedby
newerandbetterones.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 8

TheTeamWhoBroughtYouThisGuide

Mainauthors:
GrigoriMelnik,GerardMeszaros,JonBach

Contributors:
HakanErdogmus,MichaelPuleio,RohitSharma

AdvisoryBoardMembers:
MakAgashe,MohammadAlSabt,LoisPhilippeCarignan,SreekumarChandra,BartHsu,ChrisShaffer,
EngChongLim,DonnaMcLeod,KarlMetivier,TracyMonteith,NoelNyman,KenPerilman,BobSnead,
KeithStobie,JamesWhittaker

Reviewers:
JennittaAndrea,MarkAustin,DanBarnes,KentBeck,LarryBrader,HansBuwalda,JanetGregory,Sam
Guckenheimer,MihaKralj,JamesLyndsay,DanielPiessens,MaxPoliashenko,TomPoppendieck,BJ
Rollison,DonSmith,BasVodde,ThomasWoisetchlger

SpecialThanksto:
JamesBachforconstantremindertotestourtestingguidance
JeffPattonfortheProductContextchart(VolumeI,Ch.4)
BobBrumfieldfortheMultiplexingsidebarsuggestion(VolumeI,Ch.17)andmanyinsightful
conversations
patterns&practicesleadsforsupportingthisendeavor

Production:
RichardBurte,RoAnnCorbisier,DennisDeWitt,TinaBurdenMcGrayne,VeronicaRuiz

Community:
AttendeesatSTARWEST,patterns&practicessummitsandTechEdconferenceswhoprovidedinformal
feedbackonthis.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 9

Introduction
Thisguideisaboutacceptingsoftware.Acceptingsoftwareinvolvesacceptancetesting,butitismuch
morethanthat.Theconceptofacceptancetestingmeansdifferentthingstodifferentpeople.Insimple
terms,acceptancetestingisthesetofactivitiesweperformtogathertheinformationweneedtomake
thedecisionIsthissoftwarereadyforme(ormycustomer)anddoesitfullfillmy(andmycustomers)
requirements?Thisdecisionisusuallycomposedofseveraldecisions,eachwithsupportingactivities.
Therefore,todefineacceptancetesting,itmaybeusefultounderstandtheprocessbywhichthe
decision(s)aremade.Thisprocessmayinvolveseveralorganizationalentities,eachwithoneormore
decisionmakers.Thesoftwareistypicallypassedbetweentheorganizationalentitiesforthemtodecide
whetherthesoftwareisreadytogothroughthenextstep.Thisprocessisintroducedinmoredetailin
thesection"AcceptanceProcessModel"andfollowedupwithamoredetaileddescriptionofthe
decisionmakingprocessinthesectioncalled"DecisionMakingModel."

SoftwareAcceptanceandAcceptanceTesting
Acceptancereferstotheactofdeterminingwhetherapieceofsoftwareorasystemmeetstheproduct
ownersexpectations.Itincludesboththefinaldecisiontoacceptthesoftwareandanyactivities,
includingacceptancetesting,requiredtocollectthedataonwhichtheacceptancedecisionisbased.
Boththeacceptancetestingandtheacceptancedecisioncanberelegatedtoaseparateacceptance
phaseoftheprojectortheycanbedonethroughouttheproject,whichisknownasIncremental
AcceptanceTesting.

MentalModelsforAcceptanceTesting
Whilewritingthisguide,westruggledtodetermineasuitabledefinitionofacceptancetestingthat
wouldmakesensetoabroadrangeofreaders.Itseemedlikethereweremanydifferentvocabulariesin
usebydifferentcommunitiessuchasconsumerproductcompanies,informationtechnology
departmentsoflargebusinesses,anddataprocessingdivisionsoftelecommunicationserviceproviders,
tonamejustafew.Toassistusindescribingacceptance,wecameupwithseveralmentalmodelsof
variousaspectsofacceptancetesting.Wetestedthemodelsagainstnumerousexamplesfromour
collectiveprojectexperiencesatMicrosoftpatterns&practices,telecommunicationproductcompanies,
ITdepartmentsandbeyond.ThenwetestedthemodelswiththepeopleontheAdvisoryBoardforthe
project.Thiswasaniterativeprocess.Wealsotestedthesethroughthepublicreviewprocessby
releasingearlydraftsofthisguidetothecommunityandsolicitingfeedback.
Itisimportanttonotethatourearlymodelsfailedtheiracceptancetests!Thatwasagreatlessonabout
theneedtogetfeedbackincrementally,apracticeweadvocateforacceptancetesting.Basedon
feedbackfromouradvisorswerefactoredthemodelsandcameupwithadditionalmodelstofillthe
gaps.ThekeybreakthroughwaswhenwecameupwiththeDecisionMakingModel,whichtiestogether

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 10

mostoftheconceptsaroundacceptingasystem.ItbuildsontheAcceptanceProcessModelwhich
describesthekeystepsandactivitiesasthesystemundertestmovesfromrequirements,through
developmentandintotestingandfinallyproduction;italsodescribeshowthedecisiontoacceptthe
systemismade.TheDecisionMakingmodeldescribeswhomakesthedecisionsandwhoprovidesthe
dataforthosedecisions.
Thedecisionsarenotmadeinavacuum;thereareanumberofinputs.Theseincludetheproject
context,thenatureofthesystembeingbuiltandtheprocessbeingusedtobuildit.Thelatteris
importantbecauseitaffectshowwedefinedone.
Figure1illustratestherelationshipsbetweenthekeymodels.


Figure1

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 11

TheKeyMentalModelsofAcceptance

ThesemodelsarethefocusofPartIThinkingaboutAcceptancebutheresashortintroductiontoeach
modeltogetusstarted:
TheAcceptanceProcessModel.Thismodeldefinestheoverallstagesofsoftwaredevelopment
andthe"gates"thatmustbepassedthroughonthejourneyfromconstructiontosoftware
inuse.
DecisionMakingModel.Thismodeldescribeshowtodecidewhethersoftwarecangothrougha
gatetothenextstageandwhomakesthedecision.Italsodefinesthesupportingrolesthat
mayhelpthedecisionmakergathertheinformationneededtomakethedecision.
ProjectContextModel.Thismodeldescribesthebusinessandprojectfactorsthatinfluencethe
decision,includingtimeframesanddeadlines,resourceavailability,budget,andanything
contributingtoprojectrisks.
SystemModel.Thismodeldescribestheattributesofthesoftwareintensivesystemthatmay
factorintothedecision.Thisincludesbothfunctionalandnonfunctionalattributes.The
systemmodelmayariseoutofrequirementsgatheringactivitiesoramoreformalproduct
designprocess.Techniquesforcapturingfunctionalrequirementsincludesimpleprose,use
casemodels,protocolspecificationsandfeaturelists.Thenonfunctionalrequirementsare
typicallycapturedindocumentsorchecklists.
RiskModel.Thismodelintroducestheconceptsofevents,likelihood/probability,and
consequence/impact.Ithelpseveryoneinvolvedunderstandwhatcouldgowrongand,
thereby,prioritizetheacceptancecriteriaandthekindsofinformationtogathertohelp
maketheacceptancedecision.Italsodescribesseveraldifferentriskmitigationstrategies,
includingthefollowing:
Dosomethingearliertobuyreactiontime.
Doadditionalactivitiestoreducelikelihoodofsomethingoccurring.
DonenessModel.Thismodeldescribesdifferentdimensionsofdonenessthatneedtobe
consideredinareleasedecisionandhowtheyareaffectedbytheprocessmodel.

ThechaptersinPartIIIAcceptingSoftwareintroduceothermodelsthatbuildonthiscoremodel:
TestLifecycleModel.Thisdescribesthestagesthatanindividualtestcasegoesthroughhowto
gatherinformationformakingreadinessandacceptancedecisions.
ConcernResolutionModel.Thisdescribeshowtohandleanyconcernsthatareraisedduringthe
acceptancetestingprocess.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 12

DevelopmentProcesses
Thesoftwaredevelopmentprocesshasasignificantimpactonhowacceptancetestingisperformed.
ThroughouttherestofthisvolumeandtheotherstofollowwefoundourselvessayingOnsequential
projectsandOnagileprojectsbutmanypeoplehavetheirowndefinitionsofwhattheseterms
mean.Wewantedtomakesureallreadersunderstoodwhatwemeantbytheseterms.Wefeelthat
thesenamesrefertopointsonaprocesscontinuumwithotherlabelledpointspossible.Thissection
describestheprocesscontinuumwithtwodistinctprocessstereotypesontheoppositeendsofthescale.

SequentialProcesses
Sequentialprocessesorganizeworkbasedonthekindsofactivitiesinvolvedandtheinterdepencies
betweentheactivities.Theclassicwaterfallapproachinvolvesasinglereleasewhileincremental
waterfallprojectshavemultiplereleases.

ClassicWaterfall
Thewaterfallapproach(sonamedafterthediagramsusedinapaperbyWinstonRoyce[Royce])
involvesorganizingtheprojectintoaseriesofdistinctphases.Eachphasecontainsaspecifictypeof
work(suchasrequirementsanalysis)andhasspecificentryandexitcriteria.Intheclassicorpure
waterfallapproachthephasesdonotoverlap.Theentryandexitcriteriarequiretheoutcomeofa
previousphasetobecompleteandcorrect(validated)beforethenextphasecanstart.Thispushesthe
deliveryoftheproductsfunctionalitytotheend:theproductasawholeisdeployedinabigbang
approach.Figure2illustratesthemajorphasesofawaterfallproject.


Figure2
AClassicalWaterfallProject
Thisprocessisusuallyimplementedbybreakingdowneachphaseintohierarchicallyorganizedunitsof
workappropriatetothetypeofworkinvolved.Forexample,withintherequirementsphase,thework
maybedividedbetweenanalystsbyrequirementtopic,butduringtheconstructionphase,workmaybe
dividedamongthedevelopersbymodule.Thehandoffsbetweenphasesareusuallyintheformof
documents,exceptthatthehandofffromconstructiontotestingalsoinvolvesthecodebase.Readiness
assessmentisdonebythesupplierorganization,whichwerefertoastheProductDevelopmentTeam,
afteralltheconstructioniscompleted;acceptancetestingisperformedbytheproductownerafterthe
softwareisdeemedtobeready.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 13

IncrementalWaterfall
Itiscommonlyacceptedthatthelongeraprojectgoesbeforedeliveringsoftware,thehigherthe
probabilityoffailure.Ifthecontextorrequirementschangebeforetheprojectiscompleted,thepure
waterfallcannotsucceed.Onewaytocombatthisistobreaktheprojectintoincrementsof
functionality.Theseincrementscanbetestedandinsomecasesevendeployed.Figure3aillustratesa
waterfallprojectwithtwoincrementsofindependentfunctionalityeachofwhichistestedand
deployed.Thistypeofprojectisalsocalledcheckpointedwaterfall).

time
Requirements

Dply
Test
Archiecture
High-Level

Construction
Functionality

& Design
Planning

Analysis

Dplyt
Test
Construction



Figure3a
Multireleasewaterfallwithindependentfunctionality
Inthisapproach,theplanningphase,requirementsanalysisphase,anddesignphaseareperformed
onceearlyintheprojectwhiletheconstructionphase,testphase,anddeploymentphasearerepeated
severaltimes.Theworkwithineachphaseisdecomposedthesamewayasforsinglereleaseprojects.If
thefunctionalitybuiltinthesecondreleaseoverlapsthefunctionalityinthefirstrelease,thetestingand
deploymentmustencompasstheentirefunctionality.Figure3billustratesmultiplereleaseswith
overlappingfunctionality.Notehowthetestactivitymustspanthefunctionalityofbothreleases.


Figure3b
Multireleasewaterfallwithoverlappingfunctionality
Inthemultireleasewaterfallprocess,sometimesthetestphaseoftheareleasemayoverlapwiththe
constructionphaseofthesubsequentreleaseiftheconstructionandtestingteamsareseparateand
featuresacrossthethetworeleasesaresufficientlyindependent.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 14

AgileProcesses
Mostagilemethodsuseaniterativeandincrementalapproachtodevelopment.Afteraninitialplanning
period,theprojectdurationisbrokenintodevelopmentiterationsthatdeliverincrementsofworking
software.Figure4illustratesaprojectwithtwoiterations;mostprojectswouldhavemanymore
iterationsthanthis.


Figure4
Iterative&IncrementalDevelopment
Figure4illustratestwoiterations,eachofwhichstartswithaniterationplanningsessionandendswith
someacceptancetesting.Ineachiterationtheworkisbrokendownintofeaturesoruserstories,each
ofwhichindependentlygoesthroughtheentiresoftwaredevelopmentlifecycle.Theproductowner,
onsitecustomerorcustomerproxy,whoisreadilyaccessibletotheProductDevelopmentTeam,is
responsiblefordescribingthedetailsoftherequirementstothedevelopers.Itisalsotheproduct
ownersresponsibilitytodefinetheacceptancetestsforeachfeatureoruserstory.Theymayprepare
theteststhemselves,delegatethejobtorequirementsortestspecialistswithinthProductOwnerTeam
orpreparethemincollaborationwiththeProductDevelopmentTeam.Thetestsactasamoredetailed
versionoftherequirementsdescriptioninaprocessknownasAcceptanceTestDrivenDevelopment
orStorytestDrivenDevelopment.Thisallowsthedeveloperstoexecutetheacceptancetestsduring
thedevelopmentcycle.
Whenallthetestspassforaparticularfeatureoruserstory,thedevelopersturnoverthefunctionality
totheproductowner(orproxy)forimmediate"incrementalacceptancetesting."Iftheproductowner
findsanybugs,thedeveloperfixesitassoonaspossible.ItistheProductOwnersdiscretiontoinsist
thatitbefixedrightawayortoconsideritpartofanotherfeaturetobeimplementedinalater
iteration.Iftheywantitfixedandthedeveloperhasalreadystartedworkingonanotherfeature,the
developertypicallyputstheotherfeatureonholdwhiletheyaddresstheProductOwnersconcerns;the

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 15

featureisntconsidereddoneuntilalltheconcernsareaddressed.Theproductownerconcernsare
notstockpiledinabugdatabaseforfixingduringabugfixingphaseoftheproject.

MultiReleaseAgileProjects
Mostagilemethodsadvocate"deliverearly,deliveroften."Intheory,theresultofanydevelopment
iterationcouldbedetermined,afterthefact,tobesufficienttobeputintoproduction.Thiswouldlead
directlytothedeploymentactivities.Inpractice,mostagileprojectsplanonmorethanonereleaseto
productionandtheiterationsarethenplannedtodeliverthenecessaryfunctionality.Figure6Multi
ReleaseAgileProjectillustratesanagileprojectwithtworeleases.


Figure5
MultiReleaseAgileProject.
Notehowthereisatestingcycleforthesecondreleasewhichincludesregressiontestingofthe
functionalitydeliveredinthefirstrelease.Mostagilemethodsemphasizetestautomationsothe
regressiontestingcostisminimized.

KanbanbasedAgileProcess
Someagilemethodologiesdispensewithiterationsinfavourofallowingafixednumberoffeaturesin
progressatanytime.Thisisdesignedtoemphasizetheconceptofacontinuousflowofworkingcode
fortheonsiteproductowner(orproxy)toaccept.Fromanacceptancetestingperspective,these
Kanbanbasedmethods[Kanban]stilldoincrementalacceptancetestingatthefeatureleveland
formal/finalacceptancetestingbeforeeachrelease,butthereisnologicalpointatwhichtotriggerthe
interimacceptancetestingthatwouldhavebeendoneatiteration'sendiniterationbasedagile
methods.Figure6KanbanbasedAgileProjectillustratesthisstyleofdevelopment.Notethelackof
iterations.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 16


Figure6
KanbanbasedAgileProcess
Intheexample,itisimportanttonoteherethattherearenevermorethanthreefeatures(inthis
example,oneudergoingdesign,asecondundergoingconstruction,andathirdundergoingtesting)in
progressatanyonetime.Inotherwords,thereareonlythreedevelopment"slots,"andaslotbecomes
availableforanotherfeatureonlyafterithasfinisheditsincrementalacceptancetesting.Thisissimilar
tohowKanbanareusedtocontroltheinventoryinleanfactoryproductionlines.Intheory,theproduct
ownercandecideatanytimethatthereisenoughfunctionalitytowarrantdeployingtheproduct
followingashortperiodofregressiontesting.
Kanbanbasedsoftwareproceses[Scrumban]areimplementationsofamoregeneralphilosophyknown
asLeansoftwaredevelopment[LSD].

ProcessasaContinuum
"Agile"and"waterfall"areexamplesoftwohighlevelprojectstreotypesconsistingofcertain
combinationsofcharacteristics.Itiseasytoimaginethedecisiononeachofthesecharacteristicsas
beingthesettingofaprocessslidercontrol.Forexample,theNumberofreleasesslidermighthave
stopsat1,2,3,andsoon.TheNumberofiterationsslidercouldhavevaluesof1andsoon,which
indicatewhetherthereareintermediatecheckpointswithinarelease.TheMaximumnumberof
featuresinprogressslidersimilarlymaytakeonvaluesdependingonthenumberofdevelopmentslots
availableinaKanbanbasedsystem.AnotherdimensionmightbeIntegrationfrequency,withsettings
ofBigBang,MajorMilestone,Quarterly,Monthly,Biweekly,Weekly,Daily,andContinuous.
Thefollowingtablesummarizesthepositionsoftheseslidersforwhatisconsideredtobeastereotypical
projectofeachkind.Thesepositionsarenotdefinitiveorcomplete,buttheychallengeyoutocreate
yourownslidersandsettingsforyourcontext.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 17

Type of Process

Classic Incremental Agile (Iteration) Agile (Kanban)


Project Attributes Waterfall Waterfall

Number of releases 1 1 2 or more 2 or more

Number of iterations 1 26 4 or more 1

Iteration length Not applicable Many months 1-4 weeks Not applicable

Maximum number of No maximum No maximum 1 iterations worth Less than the


features in progress number of
team members

Integration frequency Big Bang Quarterly Daily or hourly Daily or hourly

Requirement-to-test Months or years Months Days Days


duration

Test timing Separate phase Separate phase Mostly incremental Mostly


incremental

Release criteria Scope-based Scope-based Time-boxed Time-boxed

Average Requirement task Person months Person months Person days Person days
effort

Average development task Person days or Person days or Person hours Person hours
effort weeks weeks

Culture Hierarchical Hierarchical Collaborative Collaborative

Skills Highly Highly Generalists Generalists or


specialized specialized Specialists

Determining progress Work Work completed Delivery of working Delivery of


completed relative to plan code working code
relative to plan

Work remaining Estimate Estimate Estimated time for Estimated time


duration of duration of remaining features for remaining
remaining tasks remaining tasks features

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 18

References
[Royce]WinstonW.Royce,ManagingtheDevelopmentofLargeSoftwareSystems,Proceedingsof
IEEEWESCON,Aughust1970,pp19.
[Kanban]KenjiHiranabe,KanbanAppliedtoSoftwareDevelopment:fromAgiletoLean,published
onlineat<http://www.infoq.com/articles/hiranabeleanagilekanban>,January14,2008
[LSD]Poppendieck,Mary&TomLeanSoftwareDevelopmentAddisonWesley(2003)ISBN:032
1150783
[Scrumban]CoreyLadas,Scrumban:EssaysonKanbanSystemsforLeanDevelopment,Modus
CooperandiPress,2008

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 19

ACautionaryTale
Thefollowingisthestoryofaprojectwherethepersoninchargeofdefiningandacceptingaproduct
failstorisetothechallenge.Itdescribeswhatgoeswrongandwhy.Italsoprovidesanalternate
outcomeofhowtheprojectcouldhaveworkedoutifproperacceptancepracticeshadbeenused.
BobKellyisamidlevelmanagerinthemarketingdepartmentatAcmeProducts.Heisinchargeofthe
MidSizedMarketsproductgroupandistheproductmanagerofitscoreproduct,theXCelRator.Hes
justfinishedabunchofmarketresearchandhascomeupwithagreatideaforanewmoduleforthe
productthathebelieveswilldoubletherevenueoftheproduct.TheProductDevelopmentdivisionhas
ateamthatisjustwindingdownitscurrentprojectandBobiskeentogettheteamstartedonbuilding
thenewmodule.HecallsameetingandrunsthroughthePowerPointslidedeckhesbeenusingwhen
talkingwithfocusgroupsandpotentialclients.Heconcludesthepresentationbylayingoutthekey
deliverydateswhichmustbemettoallowthecompanytoshowcasetheproductattheannualtrade
show.Heaskstheprojectmanagertodrawuptheprojectplanandgettheteamworkingonthenew
module.
TheDevManagermeetswithhisteamtocollectsomeinformationandthenhedefinestheprojectplan.
BasedonthedeliverydatehedefinesintermediatemilestonesforRequirementsComplete,Design
Complete,CodeCompleteandTestComplete.
Teamstartswritingrequirementsdocuments.AtRequirementsComplete,devmanagerdeclares
requirementscompleteontime.Theproductdevelopmentteam(devteam)knowsfullwellthatsome
areasoftherequirementsarestilltoovaguetoimplement.Theywillhavetogetclarificationonthe
detailsastheydothedesign.Hopefullytheproductmanager,Bob,willbemoreavailabletoanswer
questionsabouttherequirementsashislackofavailabilityisatleastpartiallytoblameforthe
requirementsstillbeingvague.
AtDesignComplete,ditto.Devteamknowsithasntdesignedsomepartsofthefunctionalityexceptata
verycursorylevel.Theywillhavetofillinthedetailsastheycode.Meanwhile,Bob,theproduct
managerthinkseverythingisproceedingaccordingtoplanandthathellhaveagreatproducttodemo
atthetradeshow.Hestartsthinkingabouthowhellspendthebonushessuretogetforcompleting
thisprojectontime.Hecallshisarchitectandaskshertostartdrawingupplansfortheextensiontohis
house,completewithindoor/outdoorswimmingpool.
Atcodecompletetheteamrequestsanextramonthtofinishcodingpartsofthefunctionalitythey
haventhadtimetodoyet.TheTestManageraskswhatpartsofthesoftwarearecompletesothat
testerscanstarttestingandtheteamrespondsNone,weareeachworkingonadifferentpartofthe
systemanditwillallbefinishedataboutthesametime.
Toensuretheproductisreadyforthetradeshow,Bobasksthetestmanagertomakeuptheschedule
bycompressingthedurationofthetestphase.Hereducesthenumberofplannedtestcyclestoreduce
theelapsedtimebasedonassurancesthatthecodewillbeingreatshapeduetotheextratimethedev
teamisbeinggiven.Bobstillbelievestheproductcanbereadyforthetradeshow(butnonethelesshe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 20

asksthearchitecttoscalebacktheextensiontohishousebyremovingtheenclosureforthepool
thinkingtohimselfIllstillbeabletouseitmostoftheyearandIcanalwaysaddtheenclosurewithmy
release2bonus.)
AsthenewCodeCompletedeadlineapproaches,theteamasksforanothermonth.Bobreluctantly
givesthem2weeks.Twoweekslaterwillmakeittightforthetradeshowbutwecandemothebeta,
hehopes.
Thedevteamfinallydeliversthecodetwomonthslate.Thetestteamstartstestingbutfindssignificant
problemsthatpreventcompletionofthefirstpassofthetestcases.Testinghaltsafterlessthanaweek.
Thedevteamtakesamonthtofixallthemajorshowstopperbugsbeforehandingthecodebackto
test.Testteammakesitthroughthefulltestcyclethistimebutfindsseveralhundreddefects.
Developmentstartsfixingthebugsastheyarefound.
ThetestteamfinallyreceivesanewbuildthatthedevteamsayshasalltheSev1and2bugfixes.
AlmostimmediatelytheyfindseveralnewSev1regressionbugs.Developmentdisagreesthatsomeof
thebugsareevenvalidrequirements.TherequirementsneversaidanythingaboutandWhywould
anyoneeverdothat?theyexclaim.Bob,theproductmanager,hastobebroughtintosettlethe
dispute.Heagreesthatsomeofthebugsarentrealrequirementsbutmostarevalidscenariosthathe
neverconsideredbutwhichneedtobesupported.TesttellsBobthatthereisnowayinthatthe
originalschedulecanbemet.
After4test&fixcyclesthattook50%longerthantheoriginalschedule(letalonemakingupthe
schedule),mostoftheSev1and2bugsappeartohavebeenfixed.ThereisstillseveralhundredSev3
and4bugsandtestteamhasstoppedbotheringtologtheSev5s(poorlywordedmessages,fieldlabels
orbuttonnames,etc.).Testteamsaystheproductisnotreadytoshipandwillrequireatleast2more
test&fixcyclesbeforetheywillagreetoreleaseit.
Theproductisnowseveralmonthslateandwontbereadyforthetradeshoweveninalpharelease.
Wecanstillshowtheboxanddosomeverycontrolleddemos.Bobassureshisboss.Bobisseeinghis
bonusdiminishwitheachweekofreducedsalesinthisfiscalyear.Hedecidestoshiptheproduct,
overrulingtheTestdepartment.Herevisesthesalesforecastsbasedonthelatelaunchcausedbythe
underperformanceofthedevelopmentandtestteams.
ThemanageroftheoperationsdepartmenthearsaboutthistoolateandcomesstormingintoBobs
office.Ihearyouareplanningtoreleasethenewmodulewith300knownSeverity3defects.Doyou
haveanyideawhatthiswilldotooursupportcosts?Acounterproductiveargumentensuesbecause
thereisnoturningbackatthispoint;theannouncementshavebeenmadeandtochangeplansnow
wouldbehugelyembarrassingtothecompany.IsurehopethisworksoutOK.thinksBobtohimself;
atthispointhedoesntfeellikehesinchargeofhisowndestiny.
Bobsmarketingmachinehasbeeninhighgearforquiteawhileandhasgeneratedquiteabitofpent
updemand,especiallysincesomeuserswerehopingtobeusingtheproductmonthsago.Userstry
usingthenewproductindroves.Theyrunintoallmannerofproblemsandcallthehelplinewhichis
overwhelmedbythecallvolumes.Manyusersgetbusysignals;theluckyoneswaitlisteningtorecorded
announcementsforlongperiodsoftime.Thehelpdeskhastobringonextrastaffwhoneedtobe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 21

trainedveryhastilyandthereforecannotprovideverygoodservicetothecustomers.Somecustomers
giveuptryingtousetheproductbecauseoflongwaitsorpoorsupport.
Manyoftheuserproblemsarerelatedtousabilityissuesthatwerenotdetectedduringtherushed
testingbecauseitwassofocusedontheSev1&2bugs(andtheusabilitystuffwasusuallyrated3or
below,manyofwhichwerentevenloggedduetothefocusonthe1sand2s.)Atpeakusagetimesthe
systemslowstoacrawl;thedevteamiscalledintofigureoutwhyitissoslowdistractingthemfrom
workingonsomeoftheimprovementsidentifiedduringtesting.Usershavetroubleimportingdatafrom
priorversionsofthesoftwareorfromcompetitorsproducts.
Alargepercentageoftheusersabandontheproductafterthefreetrialisover;theconversionrateis
lessthanhalfoftheprojectedrate.Revenuesarerunningat40%oftherevisedprojectionsandless
than20%oftheoriginalprojections.Supportcostsare50%overoriginalprojectionsduetothehiring
bingeintheusersupportcentre.
Thecapitalcostis35%overtheoriginalbudgetandhaseatenintotheplannedbudgetforasecond
modulewithadditionalmusthavefunctionality.Instead,the2ndreleasehastofocusonimprovingthe
quality.Thenewmodulewillhavetowaituntil2ndhalfofnextyear.Theproductmanagerrevisesthe
revenueprojectionsyetagain(andcallsthecontractortocanceltheadditiontohishouse).Shortlyafter
sendinginhismonthlystatusreporthisphonerings.ItistheVP,hisboss,requestinghecometohis
officeimmediately...

WhatWentWrong
1. ProductmanagerBobdecidedscope,timeframesandresourcestherebyleavingonlyqualityas
thederivablevariable.Thisresultedinthedevteambeingovercommittedattheproduct
managersinsistence.(Sometimesthedevteamwillovercommitoutofoptimismbutinthis
casetheproductmanagerdidittothem.)
2. Sequentialprocessisinherentlyopaquefromaprogressperspective.Thefirstmilestonethat
isnteasytofudgeisCodeComplete.Thefirstrealisticassessmentofprogressiswellintothe
testcycle.
3. Productmanagerwasntavailabletoclarifyvagueandmissingrequirements.Testerswerenot
involvedsoonenoughtoidentifythetestscenariosthatwouldhavehighlightedthemissing
requirements.ButnoonecouldprovetherequirementswereincompletesoRCwasdeclared
ontime.
4. Productdevelopmentteamcouldntprovedesignwasntdone(becauseitisamatterof
opinionastohowdetailedthedesignneedstobe)soDesignCompletewasdeclaredontime.
Thesequentialprocessdoesntgivethedevelopmentteamagoodwaytoestimateitstrue
velocityuntillateintheproject,whenthecodeismostlywrittenanddebugged.
5. Devteamcutcornerstomakethenew(alreadylate)CodeCompletedeadline.Thecodewas
writtenbutmuchofitwasntproperlyunittested.Theteamwasdeepinunittestdebt,they
knewit,andwouldhavetoldanyonewhoaskedbutnoonewantedtoheartheanswer.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 22

6. ThequalitywasawfulwhendeliveredtoTest.Soithadtoberedone(weneverhavetimetodo
itrightbutwealwaysmaketimetodoitover!)
7. Testwasaskedtorecovertheschedule(typical!)buttestingtooklongerbecausethepoor
qualitycoderequiredmoretest&fixcyclestogetitintoshape.
8. Nocleardefinitionofdonesodecisionismadeontheflywhenemotionsgetinthewayof
clearthinking.Theproductmanagerlethisattachementtohiscommitmentsoverrideany
sensibilityaboutquality.
9. Theoperationsacceptancecriteriawereneversolicitedandbythetimetheywereknownitwas
toolatetoaddressthem.
10. Sequentialprocesshidtrueprogress(velocity)untilitwastoolatetorecover.Therewasnoway
toscalebackfunctionalitybythetimeitbecameundeniablethatitwontallbedoneontime.
Therewasnowaytoreducescopetofitthetimelinebecausethesequentialstyleapproachto
theprojectplan(RequirementsComplete,DesignComplete,CodeComplete,TestingComplete
milestones)causedallfeaturestobeatroughlythesamestageofdevelopment.Therefore
cuttinganyscopewouldresultinalargewasteofeffortandverylittlesavingsofelapsedtime.
(Onlytheeffortofbugfixingwouldhavebeensaved.)
11. DevelopmentwasrushedinfixingtheSev1problemssothattestingcouldgetthroughatleast
onefulltestcycle.Thiscausedthemtomakemistakesandintroduceregressionbugs.Ittook
severaltest&fixcyclesjusttofixtheoriginalSev1&2sandtheresultingregressionbugs.Inthe
meantimetheSev3spiledupandupandup.Thisresultedinseveralmoretest&fixcyclestofix
andeventhenmorethanhalfwerestilloutstanding.
12. Lackofplanningfortheusagephaseoftheprojectresultedinapoorcustomersupport
experiencewhichexacerbatedthepoorproductquality.

HowitCouldHaveGone
Bob,theproductmanagercomestoproductdevelopmentteamwithrequirementsthatare
representativeofthecustomersexpectations.
Devteamestimatesrequirementsasbeing50%overteamscapabilitybasedondemonstrated
developmentvelocity.Theproductmanagerisnothappyaboutthis.Devteamproposesincremental
development&acceptancetesting.Insteadof4sequentialmilestonesbasedonphasesofdevelopment
theysuggest4incrementalinternalreleasesoffunctionalitywhereeachincrementcanbetested
properly.Thiswillallowthemtodeterminetheirdevelopmentcapacity(orvelocity)whichwillhelp
theproductmanagerplansubsequentmilestonesmoreaccurately.
Bobselectsfirstincrementoffunctionalitytodevelop.Withthehelpofthedevteamhedefinesthe
usagemodelconsistingofuserpersonasandtasks.TheDevteamwhipsupsomesketchesorpaper
prototypesandhelpsBobrunsomeWizardofOztestsonthepaperprototypesthatrevealsome
usabilityissues.BobadjuststherequirementsanddevteamadjuststheUIdesign.Bob,theproduct

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 23

managerworkswithdevteamandthetesterstodefinetheacceptancetests.Thedevteamautomates
thetestssotheycanberunondemand.Theyalsobreakdownthefeaturesintouserstoriesthateach
takejustafewdaystodesignandtest.
Teamdesignssoftwareandwritescodeusingtestdrivendevelopment.Allcodeisproperlyunittested
asitiswritten.Allpreviouslydefinedautomatedtestsarererunseveraltimesadaytomakesureno
regressionbugsareintroducedasthenewfeatureisimplemented.Aseachfeatureisfinished,as
definedbythefeatureleveldonedonechecklist,thedeveloperdemostoproductmanagerand
testerwhocanpointoutanyobviouslymissingfunctionalitythatneedstobefixedbeforethesoftware
isconsideredreadyforacceptancetesting.
Aspartofincrementalacceptancetestingtheydoidentifyafewnewusagescenariosthatwerenotpart
oftherequirementsandprovidethesetoBobassuggestionsforpotentialinclusioninthesubsequent
increments.Bobadjuststhecontentofthenextincrementbyincludingafewofthemorecriticalitems
andremovinganequivalentamountoffunctionality.Healsocontactstheoperationsmanagerto
validatesomeoftheoperationalusagescenariosidentifiedbythedevteamandtesters.Theoperations
managersuggestsafewadditionaloperationaluserstorieswhichBob,theproductmanager,addsto
thefeaturebacklogforthenextincrement.
Attheendoffirstincrementoffunctionality(whichtook5iterationstodevelop,onemorethan
expected)devteamrunsthenonfunctionalqualitiesteststoverifythesoftwareperformsupto
expectationsevenwith110%oftheratednumbersofusers.Thefirsttestresultsindicateitcanonly
handleroughly50%oftheexpectedusersandgetsevenslowerasthedatabasestartstoaccumulate
records.Theyaddsomeworkitemstothescheduleforthesecondincrementtoaddresstheseissues
andwarntestingaboutthelimitationstoavoidtheirwastingtimestumblingontothem.Testingfinds
onlyafewminorbugsduringexecutionofthefunctionaltestscripts(themajorstuffwasallcaught
duringincrementalacceptancetesting.)Theymoveontodoingsomeexploratorytestingusingsoap
operasandscenariosastheirtestsessioncharters.ThesetestsidentifyseveralpotentialscenariosBob
neverthoughtof;headdsthemtothefeaturebacklog.
Bob,astheproductmanagerarrangestodosomeusabilitytestingofthefirstincrementwithsome
friendlyusersbasedonsomeoftheusagescenariosidentifiedintheusermodel.Theresultsofthe
testingidentifyseveralenhancementsthatwouldimprovetheusersatisfaction.Bobaddsthesetothe
thingstodointhenextincrementoffunctionality.
Bobcalculatesthatdemonstrateddevelopmentvelocityis25%lessthanoriginalestimates.Basedon
thisheadjustshisexpectationsforthefunctionalitytobedeliveredinthereleasebyremovingsomeof
thelesscriticalfeaturesandthinningsomeofthecriticalfeaturesbyremovingsomenicetohave
glitz.Bettertodelivergoodquality,ontimethantotrytocraminextrafunctionalityandrisk
everythinghethinkstohimself.
Thedevelopmentteamdeliversthe2ndincrementoffunctionalitywithsimilarresultsasthefirst.The
worktheydidtoimproveperformanceresultsintheperformancetestspassingwithflyingcolorswith
acceptableresponsetimesat120%ofratedcapacityandnodegradationasthedatabasefillsupwith
transactionaldata.Theyaddsometestsforpenetrationtestingandscheduleasecurityreviewwiththe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 24

securitydepartment.Bobmakessomeadjustmentstothefunctionalityplannedforthe3rdincrementof
functionality.Heincludessomefunctionalitytoaddressoperationalrequirementssuchasmigrating
datafromearlierversionsofthesoftwareandimportingdatafromcompetitorsproducts;hewantsto
makeitrealeasyforuserstoadopthisproduct.Heshappywithhowtheprojectisprogressingand
confidentthattheywillbeabletodeliverexcellentqualityandonschedule.
Inthethirdincrement,thedevelopmentteamdelivers20%morefunctionalitythanoriginallyplanned.
Theproductmanagerhadtoscrambletoprovidetheacceptancetestsfortheextrafeaturesbrought
forwardfromthefourthincrement.Basedonthis,theproductmanagerisabletoplanforsomeextra
functionalityforthefourthincrement.Heconsidersrevivingsomeofthefunctionalitythathedcutafter
thefirstincrementbutdecidesitreallywasntthatuseful;amuchbetteruseofthedevteamsefforts
wouldbesomeoftheusabilityenhancementssuggestedbythelastroundofusabilitytesting.Healso
addsfunctionalitytomakeiteasytoupgradetothenext(yetunplanned)releasewithouthavingtotake
downtheserver.Thatwillhelpreducetheoperationalcostsofthesoftware.Yes,thisisgoingtoa
greatproduct!hesaystohimself.
AsthedevelopmentteamisworkingonIncrement4,theproductmanagerdiscussestheAcceptance
TestPhaseoftheproject.Wehadoriginallyplanned3fulltest&fixcycleseachof2weeksdurationwith
aweekforfixesinbetweenforatotalof8weeksoftesting.recountstheTestManager.Butbasedon
theresultsoftestingIncrements1,2and3Impredictingthatwellonlyneedonefulltestcycleof2
weeksplusa1weekminicycleofregressiontestingforanyfixesthatneedtobedone(andImnot
expectingmany.)Theautomatedregressiontestingthedevteamisdoingaspartofreadiness
assessmenthasbeenpreventingtheintroductionofmanyregressionbugsandtheautomatedstory
testswevebeencoauthoringwithyouandthedevteamhaspreventedmisunderstandingsofthe
requirements.ThisisthemostconfidentIveeverfeltaboutaproductatthispointintheproject!

Summary
Therearegoodwaystoeffectivelydefine,buildandacceptproductsandtherearewaysthatmayresult
indisappointment.Therestofthisguideisdedicatedtohelpingyouunderstandwhichiswhich.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 25

PartIThinkingAboutAcceptance
Akeypartofunderstandinganysystem,whetherwearedesigningit,buildingit,testingit,acceptingor
usingitistobuildmentalmodelsabouthowweexpectittowork.Systemsthatareeasytousemakeit
easyforuserstodevelopsimplementalmodelsoratleastfamiliarmentalmodelsthatallowthemto
reuseknowledgeabouthowothersystemswork.Thispartoftheguidedescribessixsimplemodelsthat
helpusreasonabouttheprocessofacceptingsoftware.
Theultimategoalofanydevelopmentprocessistobuildtherightproductofhighqualityandtobuildit
withminimumwaste.Theprimarydecisionswemustmakearetheworkflowandtheamountofwork
thatwillbemovedthroughtheworkflowatatime.Theworkflowconsistsofthesetofactivitiesthe
teammustpursueandtheorderingofthoseactivities,thatis,whethertheactivitieswillbecarried
sequentiallyorasneeded.Ourorganizationschoicesonthesetwocriticaldimensionsaffecthowwewill
goaboutmakingtheacceptancedecisions.Inaddition,howaprojectchoosestomeasuretherightness
andthequalityoftheproductwillmakeadifferenceindecidinghowtheproductdevelopmentteam
andtheproductownerwillknowifitsdoneandreadytodeploy.Themodelshelpaddresstheimpact
ofthechoicesthatthestakeholdersmakeonthesedimensions.

Chapter1TheAcceptanceProcessintroducestheoverallacceptanceprocess.Itdescribesthedecisions
thatmustbemadeandhowacceptancerelatestotheoverallsoftwaredevelopmentlifecycle,the
processeswithstagesandgates,andtheclassicVmodelofsoftwaredevelopment.
Chapter2TheDecisionMakingModeldescribestheprocessformakingthedecisionsintroducedin
thepreviouschapter.Italsoprovidesguidanceonwho(whichroles)shouldmakethedecisionsandwho
shouldcollecttheinformationonwhichthedecisionsarebased.
Chapter3TheProjectContextModeldescribesthevariousprojectfactorsthatmightinfluencethe
acceptanceprocess.
Chapter4TheSystemRequirementsModeldescribesvariouswaysrequirementscanbepartitioned
anddescribedandhowthatrelatestotheacceptanceprocess.
Chapter5TheRiskModeldescribesasimplewayofthinkingaboutpotentiallyundesirableeventsand
howtheymightinfluencetheacceptanceprocess.
Chapter6TheDonenessModeldescribeswaystodescribetowhatdegreetheproductisfinished.It
givesuswaystoaskthequestionwhatdoesdonemean?asawaytodefineouracceptancecriteriafor
individualfeaturesandforentirereleasesofproducts.Italsointroducestheconceptofincremental
acceptance.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 26

Chapter1 TheAcceptanceProcess
ThischapterdefinestheprocessbywhichsoftwareisdeemedacceptablebytheProductOwner.It
introducesthethreemajorconstituencieswhomustmakedecisionsatkeypointsintheprocessandhow
thesedecisionsrelateto,andinfluence,eachother.WestartbydrillingintotheAcceptphaseofthe
softwaredevelopmentlifecycletoexaminethekeydecisionsandhowreleasecandidatesflowthrough
theprocess.Thenweexaminehowmorecomplexscenariossuchasmultiplereleases,complex
organizationsandcomplexproductsinfluencetheprocess.Sidebarsexaminehowthedecisionprocess
relatestoStageGateprocessesandQualityGates.Wefinishthischapterwithtechniquesfor
streamliningtheacceptanceprocess.Subsequentchaptersdescribetherolesandresponsibilitiesof
variouspartiesinvolvedinmakingthedecisionsandhowthisprocessportsfromsequentialtohighly
iterativeandincrementalagileprojects.

1.1 AcceptanceasPartoftheProductDevelopmentLifecycle
Softwareintensivesystemsgothroughanumberofstagesontheirjourneyfromconcept,ahalfformed
ideainsomeonesmind,toprovidingvaluetoitsusers.ThisprocessisillustratedinFigure1ASequential
ProductDevelopmentLifecyclewiththestagesplacedintoswimlanes[OMGOnSwimlanes]basedon
whichpartyhasprimaryresponsibilityforthestage.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 27


Fiigure1A
ASequentialP
ProductDevelo
opmentLifecyycle
Acommonscenarioisforabusinessspersontoh haveanideafforhowtosolveabusinessproblemwith
software;thisistheco
oncept.Thebusinessperso on(hereincallledtheProdu uctOwner)ellaborateonthhe
concepttoproduceasspecificationoftheproducctthatonceb builtwilldelivvervalue.TheeProduct
Developm mentTeambu uildssoftwareetosatisfytheespecificationeventhougghfocussingo onthe
specificationinsteadofcustomersaatisfactionmaayoftenleadtothewrong gproductbuiiltphenomen non.
TheProduuctOwnerthe enassesseshhowwellitsatisfiestheneedsoftheinttendedusersandifsatisfieed,
acceptsth
hesoftware.W Whenthesofftwareisdeemedacceptable,thesoftw wareismadeavailabletotthe
userswhoothenusethe esoftwareto
orealizetheinntendedvaluethatsolvestheoriginallyyidentified
businesspproblem.The eProductDevvelopmentTeeamisusuallyyobligatedtoprovidesupp portfora
warrantyperiod.
ductdevelopm
Agileprod mentprocesssblurrthelinesbetweenthetimingand dresponsibiliitiesofthe
Elaboratee,BuildandAccceptactivitieesoftheprod
ductdevelopmentprocesssasillustrated
dinFigure1B
B
AnAgileP
ProductDevelopmentLifeccycle.

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 28


Fiigure1B
AnAgileProdu
uctDevelopmentLifecycle
Agileprodductdevelopm mentischaraacterizedbyfaacetofacecollaborationbetweentheProductOwn ner
andtheProductDevelopmentTeam m.Whileeach hhastheirowwnresponsibiilities,theydo onthandofff
artifactsb
butrathercollaborateonp producingtheem.Thedevelopmentteam mismorelikeelytobedirecctly
involvedintheelaboraationofthereequirementsandproductdesignandth heproductow wnerisavailaable
toassesstheproductaasitisbeingb
builtratherth
hanwaitingfo orafinalacceeptancephaseoftheprojeect.
Elaboratioon,BuildingaandAcceptanceisdoneinccrementallyin nsmallchunkksratherthan ninasinglep
pass.
Asaresult,testingtheproductforbbothacceptanceandqualityassuranceestartsearlyiintheprocesss,
therebyspreadingthetestingactiviitythroughou utthedevelop pmentlifecyccleratherpusshingittotheevery
end.Increementaldeve elopmentand dincremental,earlytestinggallowalotm moreopportu unityforlearnning
aboutwhatistrulynee ededandtrullypossibleanndtypicallyreesultsinprodu uctswithbetterfitnessfor
purpose.Seethesideb barUsingWhholeTeamsto oBuildProductsformoreo onthemotivaationbehindthis.
Thereareeseveraldiffe
erentwaystodescribetheroleoftestin
ngandaccepttanceinthed
development
lifecycleo
ofasoftwareintensivepro
oduct.Someo
ofthewellknnownmodelsare:
TheSttageGateTMProcess.
TheV
VModel.

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 29

Thefirstisaproductdevelopmentprocessthatisnotspecifictosoftwareintensivesystems.Itdescribes
howtransitionsbetweenthedifferentstagesoftheproductdevelopmentlifecyclecanbemanaged.The
latterisaclassicalmodelspecifictosoftwaredevelopment.Itemphasizesequallyboththefrontend
andbackendoftheproductdevelopmentlifecyclebyexplainingtherelationshipsbetweendifferent
kindsofstagesandartefactscommonlyassociatedwithfrontendofthelifecycleanddifferenttypesof
testingandvalidationactivitiesthattraditionallytakeplaceatitsbackend.

Sidebar:UsingWholeTeamstoBuildProducts
Oneofthekeyformsofwasteinsoftwaredevelopmentishandoffsbetweenhighlyspecializedroles.A
keypracticeinagileandleanproductdevelopmentistheeliminationofhandoffswheneverpossible
throughtheuseoftheWholeTeamapproach.TheWholeTeamapproachbringseveryskillneededto
developtheproductontoasingleProductDevelopmentTeamwhichworkswiththeProductOwnerto
designthemostsuitableproductbasedontheneedsandconstraints.Theteamcollectivelycommits
todeliveringincrementsofpotentiallyshippableproductfunctionality.Thiseliminatesdependencies
betweenteamsandallowstheProductOwnertoworkdirectlywiththeteamtodesignthebest
possiblesolutiontothebusinessneeds.
Asanexample,theScrummethodadvocatestheuseoftheWholeTeamapproach.Scrumproponents
claimtypicalimprovementinproductivityof5xto10x(500%to1000%)[PrimaveraOnScrum,
KnibergOnScrumFromTrenches].Includingeveryoneneededtodelivertheproductonasingleteam
allowstheteamtocontinuallyimproveitsprocessbyeliminatingwaste.Organizationalboundariesdo
notgetinthewayofthisprocessstreamliningbecauseeveryoneisonthesameteamstrivingtoreach
thesamegoal:deliveringvaluableproducttotheProductOwner.
ThereisanongoingdebateintheagilecommunitywhethertheProductOwnerispartoftheteamor
separatefromit.Thereisnosingleanswertothisquestionasitdependsonthenatureofthe
organization(s)andpeopleinvolved.Thepragmaticansweristhatthereareoftentwolevelsofteam
atplay:
1. TheWholeTeamincludingtheproductownerandanyonehelpingthemdefine,buildandverify
theproduct.
2. TheProductDevelopment(sub)TeamwhichbuildstheproductandtheProductOwner(sub)Team
whichacceptstheproduct.Thesetwosubteamsmustworkcloselytogetherandoftensitinthe
sameteamroom.
<maybeagraphicshowingthetwosubteamsandtheskillswhichmightliveineachandwherethey
overlap>
Theexactbreakdownofwhichskillsareoneachsubteamvariesfromcasetocase.Developersusually
belongtotheProductDevelopmentTeamandanalystsusuallybelongtotheProductOwnerTeam.
Testers,documentationwriters,interactiondesignsmayallbelongtoeithersubteam.Theimportant
thingisthatalltheskillsarepresentandaccountableforworkingtogethertodeliverthebestpossible
productwithfewifanyexternaldependencies.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 30

1.1.1 Processesw
P withStage
esandGattes
Manyprooductdevelop pmentprojecttsgothrough htheirlifecycllebyadoptinganessentiaallysequential
processto
oevolveaniddeaintoadep ployedproduct.Ifthepro ocesssdistincctstagesaregguardedby
decisionsthatcontrolprogressionffromonestaggetoanotherr,theprocessscanberepreesentedasa
streamoffalternatingsstagesandgates,aspropo osedintheStaageGateProcess
[CooperOOnDoingItRigh ht].Thegatesareassociattedwithgo/n nogoorscoppechangedeccisionswith
associateddresourcecoommitmentim mplications.Inastrictlyseequentialpro
ocess,thestaggesasmutually
exclusive::theprojectccanonlybein
nonestageattatimealtho oughwithinaastagemanykindsofactivvities
mightbehappeningattthesametim me.:Figure1cASequenttialProcesswwithStagesan ndGatesdepicts
suchapro
ocess.


Figure1c
withStagesan
ASequentialProcessw ndGates

Stagegateprocessesn neednotbesstrictlysequenntial[CooperOnStageGatee]inthesenseeofeverystaage
andeveryygatebeingddistinct.Anexxampleofaniterativeproccesswithstaggesandgatessisthe
IncrementalFundingM Method(IFM)[DenneHuan ngOnIFM]whosestagesan ndgatesinvollvethesame
activitiesanddecisionssthatarerep
peated.Inconntrasttothestrictlysequeentialdepictio
oninFigure11c
andsimilaartotheIFM,,theacceptannceprocesstthatwedescrribeinthisguidehasthesaameactivitiessthat
areexecuutedacrossstagesanddeccisionsthatarremademanyytimesovertthelifetimeo ofaproject.FFor
example,theproductd developmenttteammaybuildmanyreleasecandidaatesfortestin ngbutonlyaffew
maybeacccepted.Theprojectcould dpartiallybeintheDeployystage(apreeviousreleaseehavingpasseed

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 31

throughthegateaftertheAcceptstageGate5inFigure1c)eventhoughtheproductdevelopment
teamisworkingonprovidinganotherreleasecandidateforacceptancetesting.Theresultofsuch
dynamicsistheintroductionofmanystaggeredparallelprocessstreamswiththeirownstagesand
gates,interactionsbetweentheseparallelstreams,andloopswithinandacrossthestreams(seethe
SidebartitledRecastingaSequentialProcessesthroughWorkflowScheduleIndependenceand
Parallelization).Thedivergencefromthetraditionalsinglesequentialflowisinlinewiththemore
modernandflexibleinterpretationsandadaptationsadvocatedby[CooperOnStageGate].Howeverthe
acceptanceprocesswillfeatureintermediatestagesthatdirectlyfeedintosubsequentstageswithout
explicitgatesinbetween.Inaddition,thegates,ratherthanexplicitlycontrollingfundingdecisionsfor
subsequentstages,primarilydictatecontrolflow.Thesevariationsmaybeconsideredasdepartures
fromtheoriginalStageGateProcess[CooperOnDoingItRight].Thestagegatelikemodelunderlying
theacceptanceprocessisexpoundedinanothersidebartitledRepresentingtheAcceptanceProcess
withStagesandGatesinChapter2.

Sidebar:RecastingaSequentialProcessesthroughWorkflowScheduleIndependenceand
Parallelization
CoreyLadas[LadasOnScrumban]providestwopowerfulinsightsonsoftwareprocessesbasedon
Kanbansystems[HiranabeOnKanban]andleandevelopment[PoppendiecksOnLean]thatallowsusto
seetheminanewlight.Thefirstinsightexplainshowworkflow,theorderandinterdependenceof
stepsinherentinasoftwareprocess,canbeseparatedfromhowtheworkthatflowsthroughthe
processisscheduled.Thisiscalledworkflowscheduleindependence.Thesecondinsightallows
reorganizingasequentialworkflowasparallelstreamswithmergingpointswhereworkfromthe
streamscanbeintegrated.Whenbothideasarecombined,softwareprocesses,regardlessofwhether
theiressentialworkflowissequentialcanbemadeiterative,incremental,andparallel.Forany
process,theessentialworkflowisthesequencingofsteps,frombeginningtoend,appliedtoaworking
system,toimplementanewimprovement,whetheranewpieceoffunctionalityoranewnon
functionalrequirement.
Thiskeyideabehindworkflowscheduleindependenceisindependenceofrequirements.
Independenceofrequirementsinturnresultsfromhowworkisdividedupinsmallerchunksinthe
beginningofaworkflow,oftenbyanelaborationorrequirementsanalysisactivity.Ifthechunks,the
resultinglowlevelrequirementsthattheproductdevelopmentteamtransformsintoworking
functionality,aresmallandcanindividuallybecompletedallthewaytointegration,acceptance
testing,anddeployment,theyareindependent.Theextenttowhichthisconditionissatisfieddictates
whetherthechunkscanbescheduledtoflowthroughtheprocessindividually,ingroupsofsmaller
batches,orasasinglebigbatch.Themoreindependentthechunksare,thesmallerthebatchescan
get,downtotheleveloftheindividualrequirements.Iftheyformindependentgroupsof
interdependentchunks,thenthegroupscanbescheduledindependently,butnotthechunks
themselves.Regardlessofthegranularityofthebatchesthatflowthoughtheprocess,theunderlying
essentialworkflowstaysthesame,buttheworkflowexecutedagainforeachbatch.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 32

Thesecondinsight,whichLadasreferstoaspipelining,leveragesanyindependenceofthestepsofthe
essentialworkflowitselfinsteadofthechunksofworkthatflowthroughit.Thestepsthatdontshare
resourcescanbeparallelized.Theresultofsuchparallelizationisworkflowwithstaggeredbranches
andminimalunusedcapacity.Parellelizationincreasestheefficiencyoftheprocessprovidedthatthe
outputsofparallelstreamscanbeintegratedinsuchawaythecapacitygainintroducedbythe
parallelizationexceedstheextraoverheadofintegration.
Thusworkflowscheduleindependenceandparallelizationtogethermayallowaseeminglystrictly
sequentialprocesstobeexecutedinahighlyiterative,incremental,andparallelmanner,makingit
moreefficient,flexibleandresponsivetochange.Theyapplytovariousdegreestomanysoftware
processesandlifecyclemodelswithsequentialdepictions.Inparticularworkflowschedule
independenceandparallelizationexplainhowthevariousdevelopmentprocessesontheprocess
continuumdiscussedinIntroductioncanbederivedfromtheessentiallysequentialworkflow
underlyingtheclassicalwaterfallmodel.
Ladassinsightsalsoapplytotwoadditionalrelatedprocessmodelsthatwediscussinthischapter.
BothofthesemodelsprocessesexpressedintermsofstagesandgatesandtheVModelhave
sequentialdepictionsnottoodissimilartothatoftheclassicalwaterfallprocessdiscussedin
Introduction.Andbothofthesemodels,liketheclassicalwaterfallprocess,areamenabletorecasting
inaniterative,incremental,andparallelmannerbyapplyingworkflowscheduleindependenceand
parallelization.Suchrecastingmayoccuratdifferentgranularities:atthelevelofwholereleases,
iterationswithinreleases,featuresets,orindividualfeatures.

1.1.2 TheVModelofSoftwareDevelopment
InthedevelopmentofsoftwareintensivesystemstheVModeliscommonlyusedtodescribethe
relationshipsbetweenactivitiesfocusingonbuildingthesystemandactivitiesforsystemverification].A
variantoftheVModeladaptedfrom[FewsterGrahamOnTestAutomation]isshowninFigure1dTheV
ModelofSoftwareDevelopment.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 33

Analysis Business Tests Acceptance


Requirements Testing
Architecting Product Tests System
Requirements Testing
Design Components
Tests Component
Testing

Coding Units Tests Unit


Testing
Build Verify
Figure1dTheVModelofSoftwareDevelopment

InFigure1d,activitiesandartefactsthatareassociatedwithbuildingthesystemareshownontheleft
sideoftheVshape.Testingactivitiesareassociatedwithsystemverification,andtheseareshownon
therightsideofthefigure.Testsspecifictoalayertietogethertheactivitiesandartefactsofthatlayer
shownontherightsideofthelayertotheverificationrelatedactivitiesofthesamelayershownonthe
left.Testsforeachlayercanbedefinedearlyduringtheactivitiesthatareassociatedwithbuildingthe
system,buttheycanbeexecutedonlyafterdevelopmenthasprogressedsufficientlytotherightsideof
theVshapeintothecorrespondingverificationactivity.NotethatthisinterpretationoftheVModelcan
accommodatebothsequentialortestlaststyleverificationandincrementalortestasyougostyle
verification.TheprinciplesdescribedintheSidebartitledRecastingaSequentialProcessesthrough
WorkflowScheduleIndependenceandParallelizationapplytotheVModel.Thereforetheunderlying
workflowcanbeexecutediterativelyatdifferentlevelsofgranularity,onafeaturebyfeature,iteration
byiteration,orreleasebyreleasebasis.
Aftercodinghasbegunandtheunittestsfortheimplementedpiecesareinplace,unittestingmay
start.Asacomponentsunitsareimplemented,certainAPIsofthecomponentmaybecomefunctional.
IfthecorrespondingcomponenttestsforthoseAPIsareinplacethencomponenttestingforthatmay
begin.Continuingthisprogression,weclimbuptherightsideoftheVshape:theunitsarerolledinto
componentsandultimatelycomponentsarecomposedintoproductfeaturessuchthatthebusiness
requirementsarereadytobeexercised,thusreachingtheacceptancetestingactivityatthepinnacleof
therightside.Theacceptanceprocessdescribeswhathappensatthisfinalactivityofthetoplayer.We
makethedecisiontoacceptorrejectthesoftwarebasedonhowwellitmeetsthebusiness
requirementsshownonthetopleftsideoftheVModel.
Whenappliedinastrictlysequentialmanner,weassociatetheAcceptingTestingactivitywithadistinct
AcceptanceTestPhase.ContrastedwithpracticessuchasAcceptanceTestDrivenDevelopmentand
IncrementalAcceptanceTestingadvocatedinagilesoftwaredevelopment,theAcceptanceTestPhase
undertakentowardtheendofthedevelopmentlifecyclerepresentsthetestlastapproachto
acceptancetesting..Thespecificpracticescommonlyusedintheagilesoftwaredevelopmentcontext

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 34

aredescribedlaterinthischapterinsectionTheAcceptanceProcessforHighlyIncremental
Development.

1.2 PartiesoftheAcceptanceProcess
Theacceptanceprocessdescribestheinteractionsbetweentwokeyparties:theProductOwnerandthe
ProductDevelopmentTeam.Eachhasveryspecificresponsibilitiesintheproductdevelopmentprocess.
Whilethenamesofthesepartiesmaycoincidewithnamesusedinspecificmethods(suchasScrum),we
provideourowndefinitionsforthepurposesofthisbook.

1.2.1 ResponsibilitiesoftheProductOwner
TheProductOwneristhepartywhohascommissionedtheconstructionoftheproductbutdoesnthave
thetimeorskillstobuilditthemselves.TheProductOwnerisresponsibleforclearlycommunicating
theirexpectationsoftheproducttotheProductDevelopmentTeamandfordecidingwhetherthose
expectationshavebeensatisfiedbytheproductdeliveredtothem.Thedetailedresponsibilitiesofthe
ProductOwnerinclude:
Understandthepotentialusersoftheproductoridea.
Determinewhatcapabilitiestheproductneedstohavetoaddresstheusersneeds.
Determinetheamountofresourcestheyarepreparedtoinvestintobuildingtheproduct.
Specifythedesignoftheproduct(notthedesignofthesoftwareinsidetheproduct).
HelptheProductDevelopmentTeamunderstandthepotentialusersandtheirneedsandhowthe
productwilladdressthem.
Definetheacceptancecriteriathatmustbemetbeforetheproductcanbeaccepted.Theseinclude:
Examplesofhowtheproductwillbeusedbyusersandhowtheproductshouldreactin
eachexample.
Criteriaregardingnonfunctionalpropertiesoftheproductincludingresponsetime,
availability,accessability,configurability,etc.
Defineanyconstraintsontheproductincludingtechnicalconstrainsts(whattechnologycanormust
beused).
AssessthecostanddeliverydateestimatesprovidedbytheProductDevelopmentTeam.
Makethedecisionastowhethertheproductisworthbuilding.
TheProductOwnermaycarryouttheseresponsibilitiesthemselvesortheymaydelegatetomembersof
theirteam(knownastheProductOwnerTeam.)Theymayalsoenlistthehelpofpeopleoutsidethe
ProductOwnerTeamtohelpthemcarryouttheirresponsibilities.ThisincludesaskingtheProduct
DevelopmentTeamtodosomeofthework.Butneitherdelegationnorcollaborationabsolvesthe
ProductOwnerfromultimateresponsibilityforthem.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 35

AgoodProductOwnerhasaclearvisionofwhatdonemeansandsharesthatvisionwiththeProduct
DevelopmentTeamasearlyaspossible.AgoodProductOwneralsorecognizesthatcommunicationis
inherentlyflawedandthatyouneedtoregularlyverifythatwhatyouintendedtocommunicatehas
beenunderstood.Therefore,agoodProductOwnerispreparedtotryoutandprovidefeedbackonthe
softwareunderdevelopmentatfrequentintervals;agoodProductOwnerdoesntsimplythrowa
specificationoverthewallandsaydontbothermeuntilyouarereadyforacceptancetesting.They
takeanactiveinterestintheemergingsoftwareandencouragetheProductDevelopmentTeamto
providefrequentopportunitiesforfeedback.

1.2.2 ResponsibilitiesoftheProductDevelopmentTeam
TheProductDevelopmentTeamisthepersonorpersonswhohaveacceptedthetaskofbuildingthe
productcommissionedbytheProductOwner.TheProductDevelopmentTeamisresponsiblefor
deliveringaworkingproductthatmeetstheProductOwnersacceptancecriteria.Thedetailed
responsibilitiesoftheProductDevelopmentTeaminclude:
Determinehowtherequestedfunctionalityshouldbeachievedincluding:
Hardwarevs.firmwarevs.softwarepartitioning.
Determinewhattechnologieswillbeusedtobuildtheproduct(unlessthesewerepartof
therequirementsorconstraintsprovidedbytheProductOwner)
BuildtheproductandverifythatitmeetstheProductOwnersexpectationstothebestoftheir
ability.Theproductincludes:
Workingsoftwareandhardware
Whateveruserdocumentationisrequired(unlessthisisprovidedbytheProductOwneras
partoftherequirements.)
Whateverartifactsarerequiredtomaintainthesoftwareincludingsoftwareandhardware
designdocumentation,unitandcomponenttests,testtoolsrequiredtotestthe
product,etc.
TheProductDevelopmentTeammaycarryouttheseresponsibilitiesthemselvesortheymayenlistthe
helpofpeopleoutsidetheProductDevelopmentTeamtohelpthemcarryouttheirresponsibilities.But
neitherdelegationnorcollaborationabsolvestheProductDevelopmentTeamfromultimate
responsibilityforthem.

1.2.3 ResponsibilitiesofOtherSpecialities
Thenatureoftheproductbeingbuiltoftendictatesthekindsofspecialiststhatneedtobeinvolvedin
theproject.ThesespecialistsmaybeengagedbyeithertheProductOwnerTeamortheProduct
DevelopmentTeam.Regardlessofwhichpartytheyworkwith,theyneedtounderstandtheirownrole
indeliveringaproductthatsatisfiesthepotentialusers.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 36

1.3 TheBasicAcceptanceProcess
ThebasicacceptanceprocessatitshighestlevelsubsumestheBuildAcceptUsecycledepictedin
Figures1Aand1B.EachofthehighlevelstagesshowninFigure1Aand1Binvolvesmultiplesubstages
oractivitiesandanexitdecision.
TheBuildstageiscomposedoftheproductsconstruction,itsreadinessassessment,andasubsequent
readinessdecision,allperformedbytheproductdevelopmentteam.
TheAcceptstageiscomposedofacceptancetestingoverseenbytheproductowner(andsometimes
performedincollaborationwiththeproductdevelopmentteam),andasubsequentacceptancedecision
madebytheproductowner.Normally,thereadinessdecisionandtheacceptancedecisionaremadeon
thesamesetofcriteria.Theprecedingactivities,readinessassessmentandacceptancetesting,may
involveoverlappingactivities,althoughreadinessassessmentmaybemoreinternallyfocusedthan
acceptancetesting.Howeverthetwodecisionsdifferintheirgoals.Readinessdecisiondetermines
whethertheproductisreadyfortheproductownertoevaluateandtheacceptancedecisiondetermines
whethertheproductmeetsallitsrequirementsandisreadytobeused.Thelaststage,theUsestage,
mayinvolveafinal,userfacingevaluativeactivityineitheraproductionorenduserenvironmentto
determinewhetheritisusableanddeployabletoitsintendedaudience.
Figure2TheAcceptanceProcessdrillsdownintoeachoftheBuild,Accept,andUsestagestoexpress
themintermsoftheirlowerlevelcomponents..Thefigureshowsonlythepathassociatedwiththe
positiveoutcomesoftheunderlyingdecisions.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 37

Figure2
TheAcceptanceProcess
Intheidealworldthissequenceofactivitieswouldbedoneexactlyonceandtheassessmentcouldbe
doneentirelybytheProductOwner;inpractice,thiscouldbearecipefordisasterasuntestedsoftware
cancontainlargenumbersofbugsperthousandlinesofcode.Itusuallytakesmanytriestobuilda
productthattheProductOwnerfindsacceptablethereforetheacceptanceprocessistraversed,atleast
partially,manytimes.Toensurethataqualityproducthasbeenbuiltandtherebyminimizethenumber
oftimestheProductOwnerisaskedtoacceptthesameproduct,mostProductDevelopmentTeamswill
includesomelevelofselfassessmentofeachreleasecandidatebeforeprovidingthesoftwaretothe
ProductOwnerformakingtheacceptancedecision.Inthisbookwecallthetestingandother
verificationactivitiesthatareperformedpriortoaskingtheProductOwnertoacceptthesoftware
readinessassessmentandthedecisiontohandoffthesoftwareiscalledthereadinessdecision.The
testingdonebytheProductOwnerortheirproxyafterreceivingthesoftwarefromtheProduct
DevelopmentTeamiscalledacceptancetestingandthedecisionwhethertoacceptthesoftwareinits
currentstateiscalledtheacceptancedecision.Thedecisioneachusermakeswhetherornottoactually
usetheproductonceitisavailabletothemiscalledtheusagedecision.
Eachdecisioncanresultinapositiveornegativeoutcome.Onlythepositiveoutcomesareshownin
Figure2;thenegativeoutcomesareshowninFigure3PathsthroughtheAcceptanceProcess.
ThetransitionsfromtheBuildstagetotheAcceptstageandfromtheAcceptstagetotheUsestage
typicallyrequirethemovementofsoftware(orsoftwarecontainingproducts)fromoneenvironmentto
another.Forthepurposesofthisdiscussion,thespecificstepsinvolvedinmakingthesoftwareavailable,
thoughimportantforacceptance,arenotconsideredapartoftheacceptanceprocess.

1.3.1 AssessingReleaseCandidatesusingtheAcceptanceProcess
Theversionofthesoftwarethatisputthroughtheacceptanceprocessisoftenreferredtoasarelease
candidate.Itgoesthroughthereadinessdecisionandacceptancedecisionprocessesstepbystepand
decisionbydecisionuntilitmeetsoneofthefollowingrequirements:
Itpassesthroughallthedecisionsandisdeemedreadyforusebyusers.
Itisdeemedinsufficientinsomeway;atwhichpoint,itissentbacktoanearlierphase.

Anegativeacceptancedecisionorreadinessdecisioncancausethereleasecandidatesoftwaretobe
sentbacktoanearlierpointintheprocess.
Figure3PathsthroughtheAcceptanceProcessillustratesthepossiblepathsthroughtheacceptance
processwhenthereadinessoracceptancedecisionscannotbemadeandrequireadditionalcapabilities
orinformation.Inthefigure,QualityDatareferstotheinformationaboutthequalityofthesystem
obtainedfromreadinessassessmentoracceptancetesting.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 38


Fiigure3
PaathsthroughtheAcceptan
nceProcess
Fromthereadinessdecision,areleaasecandidateecanbesentbacktoread dinessassessm mentifthequuality
dataobtainedfromthe ereadinessassessmentisinsufficienttomakeawellinformedreeadinessdeciision.
Ifthedataafromtheaccceptancetestingphaseissufficienttodetermineth hatcriticalfun
nctionalityis
missingortheproductthassevered deficiencies,tthereleasecaandidatecanbesentbacktodevelopment
forrework.
Fromtheacceptanced decision,arelleasecandidaatecanbesenntbacktoaccceptancetesttingifthequaality
macceptanceisinsufficientt.Iftheaccep
datafrom ptancedecisio
onalsodepen ndsondatacollecteddurin ng
readinessassessmentanditisdeem medtobeinsufficient,thereleasecand didatecanbesentbackto
readinessassessment.Ifcriticalfun missingorhasssufficientlyseveredeficieencies,thereelease
nctionalityism
candidateecanbesentbacktodevelopmentforrrework.
Normalpracticeistolooganybugsffoundinthebbugmanagem mentsystemssothatthePrroduct
Developm mentTeamcanstarttheprrocessofrem mediationbuttocontinuettestinguntilaallthetestshave
beenrunortestingisbblockedbyabbugthatprevventsfurthertestingfromoccurring.Attthispointteesting
stopsuntiilanewreleaasecandidateeisreceived.A ormorebugss,anewrelease
Afterremediaationofoneo
candidateeisbuiltandp
passedthrougghtheaccepttanceprocesss.Eachreleassecandidatesshouldbea
distinctlynamedversio onofthesofttware(andshhouldbetaggedassuchinthesourceco odemanagem ment
system.)
Whenalaargenumberofbugsisfouundduringtheacceptanceeprocess,theeProductOwnermayneed dto
decidewh whichrequirefurtherinvesstigationandwhichcanbeedeferredtoa
hichbugsmustbefixed,w
futurerelease,aproceessknownasBugTriage.B Bugtriaginginvolvesassiggningapriorittyandseverittyto
eachbug..Whilethisisacommonppractice,itisttypicallyasym
mptomofdeeeperissuesinnthe
organizationalcultureandstructureeoftheenterrprise.Ratherrthanfocusin ngonbecomingbetteratffixing
thebugs,mostorganizzationswoulddbebetterseervedbyundeerstandingtherootcausessoftheirbug
backlogandchangingh howtheybuildtheirprodu uctstoreducethenumberofbugsfoun nd.Thesectiion
TroubleshootingtheA AcceptanceProcessinChaapter20FineTuningtheAcceptanceProcessofferrs

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 39

somepossiblerootcausesofalargebugbacklogandpossibleavoidancestrategies.RefertotheSidebar
IncrementalAcceptanceandBugTrackingonAgileProjectsinChapter8.

Note:Althoughtheacceptanceprocessoutlinedhereappearstobesequentialwiththesoftware
movingfromonestagetoanother,thisisnotalwaysthecase.Onwellrunprojects,followingthe
practiceofincrementalacceptance,eachfeaturemaytraversethisprocessindividually.Therefore,itis
possibleforonefeaturetobeintheconceptstage,withthedevelopmentteamworkingonanother
setoffeaturesinthedevelopmentstage,andanothersetoffeatureswiththeProductOwnerinthe
acceptancestage.Thatisatleastthreeinstancesoftheprocessrunninginparallel.
Evenonclassicalwaterfallprojectswemayhavemorethanonereleasecandidategoingthroughthe
processatthesametime.Forexample,theAlphareleasemaybeInUse,theBetaisinAcceptance
Testinganddevelopersmaybeworkingonadditionalfunctionalityforthegeneralreleasewhilealso
fixingbugsinthenextreleasecandidateforthisrelease.Thatisfourinstancesoftheprocessrunning
inparallel.Suchparallelismhoweverisnotwithoutimplicationsduetoincreasedintegrationand
coordinationoverhead.Inparticular,sourcecodemanagementbecomescomplexduetobranching
andsubsequentmergingofsourcecodetrunks(forprovenbranchingstrategiesinTFS,see
[MSOnTFSBranchingGuide]).Bugtrackingbecomescomplexduetotheneedtocoordinatetheissues
fromthereleasesunderacceptancetestingwiththefixesandadditionsinthereleasesunder
development.
Amorecompletediscussionofoverlappingreleasesfollowslaterinthischapterundertheheadings
TheAcceptanceProcessforAlpha&BetaReleases.Forthegeneralprinciplesgoverningincrementality
andparallelism,refertotheSidebarRecastingaSequentialProcessesthroughWorkflowSchedule
IndependenceandParallelization.

WhySeparateReadinessAssessmentfromAcceptanceTesting?
Theprocessofverifyingandvalidatingthesoftwareintensivesystemagainsttherequirementsand
expectedusagecanbeviewedasasinglemonolithicactivityorasveryfinegrainedsteps.Theprimary
reasonforgroupingthesefinegrainedstepsintotwomajorbucketsistoseparatetheactivitiesthat
shouldbecarriedoutbytheProductDevelopmentTeambeforeturningthesystemovertotheProduct
OwnerfromthosethattheProductOwnerdoesaspartofdecidingwhethertoacceptthesystem.
ReadinessassessmentisprimarilyabouttheprofessionalismoftheProductDevelopmentTeamwhile
acceptancetestingisaboutvalidatingthatthesystemasdeliveredwillsuitthepurposesoftheusers
andotherstakeholdersandconfirmingcompliancetocontractualagreements.
Theremaybeotherreasonstodividetheactivitiesintovariouscategories.Wementionthemonly
brieflybecausetheyarebeyondthescopeofthisguide:
DeadlinePressureSomemanagersmightbelievethathavinganearlier,separatemilestonebefore
handingoversoftwaretotheproductownerisaneffectivewaytomotivatedevelopers.
AccountingTheinitialconstructionofthesoftwaremaybetreateddifferentlythanfixingofbugs
fromanaccountingperspective(especiallyworkthatmaybeconsideredasachangerequest
mayaffectprojectaccountingdifferentlythanworkthatmaybeconsideredasabugfix).

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 40

EarlyValidationItisusefultohaverealuserstryusingearlyversionsoftheproducttovalidate
thattheproduct,oncefinished,willfillthenicheforwhichitwastargeted.Thisclearlyisntfull
onacceptanceoftheproductsoitcouldbeconsideredreadinessassessment.Priortodoingthis
typeoftestingwithusers,theProductDevelopmentTeamwouldwanttodotheirowndue
diligencetoensurethesoftwarewasworkingwellenoughtoallowtheuserstotryit.Therefore,
wewouldconsiderthisaformofAlpha/Betareleaseorpossiblyincrementalacceptancetesting
orevenconditionalacceptance.
EndUserTrainingExposingthesoftwaretousersbeforeacceptancecanbeaneffectivewayto
startthetrainingprocess.Ifthequalityofthesystemishighenough,itcanalsobeusefulasa
formofviralmarketing.BothoftheseusesfallintothecategoryofAlpha/Betareleasesrather
thanreadinessassessment.

ReadinessAssessmentorAcceptanceTesting?
Givenaparticulartestingactivity,shoulditbeconsideredpartofreadinessassessmentoracceptance
testing?Thereareseveralfactorsthatplayintothisdecision:
1. Whosdoingtheassessing:AcceptanceisperformedbytheProductOwnerortheirproxy
(someoneactingdirectlyontheProductOwnersbehalf)whilereadinessassessmentcanbe
donebyanyoneinvolvedintheproject.WheretheProductDevelopmentTeamandProduct
Owneraredistinctorganizationalentities,therolesshouldbefairlyclear.Thingsgetmurkier
whenthereisnorealuserorendcustomerinvolved,commoninproductcompanies,orwhen
thereisaseparateorganizationchargedwithtesting.Thesescenarioswillbecoveredinmore
detailinChapter2TheDecisionMakingModel.
2. Formalityofthetesting:AcceptanceisamoreformalactivitybecausetheProductOwneris
involved.Thisimpliesmoreformalrecordkeepingofanyconcernsthatwereidentified.
Readinessassessmentcanbemuchlessformal;itneedonlybeasformalasdictatedbythe
projectcontext.ProductDevelopmentTeamsthatneedtobeabletopassformalauditswill
keepmuchmoreformalrecords.Agileprojectstendtobeveryinformal;duringreadiness
assessmenttheyjustfixthebugsimmediatelyratherthandeferthemforfixinglater.

Thelinescangetsomewhatblurredwhentherearemorethantwodifferentstagesoftesting;these
scenariosaredescribedinAcceptanceinComplexOrganizationsandAcceptingComplexProductsin
Chapter2ElaboratingontheAcceptanceProcess.Intheend,itislessimportanttodecidewhichlabel
toapplytoaparticulartestingactivitythanitistoensurethattherighttestingactivitiesgetdoneand
thateachpartyknowsforwhichactivitiestheyareresponsible.Itcanbeusefultolistallthepotential
testingactivitiesandforeachonedecidewhetheritismandatory,optional,ornotrequired,andto
assignresponsibilityforittoaspecificpersonororganization.Asamplespreadsheetisprovided<online
athttp://testingguidance.codeplex.com>.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 41

1.3.2 TheAccept
T tanceProccessforMu
ultiRelea
aseProduccts
Thusfar,w
wevefocusedontheacceeptanceproceessasitappliiestoasingleereleasecand
didate.Howd
does
theaccep
ptanceprocesssgetappliedwhenaprod
ductwillhaveemultiplereleeases?
mostpart,longglivedmultireleasesystemscanbethoughtofassimplyasequeenceofindiviidual
Forthem
products,whereeachproductisbeeingindividuallyassessedfforreadinessandacceptan
nce.Eachreleease
goesthroughtheentirredecisionmakingprocesss.Figure4TTheAcceptancceProcesswiithMultiple
Releasesiillustratesanexampleofthisprocess.

Figure4
TheAccep ptanceProcessswithMultip
pleReleases
mumMarketaableFunctionality
Thesetofffunctionalityyrequiredforreachreleasee,whichweccalltheMinim
1
orMMF ,isuniquebassedonthego oalsofthereleasedeterminedbythep productowneer.Thesetof
qualitycriiteria,whichw
wecalltheM
MinimumQualityRequirem mentorMQR,,issomewhattmoreconsisstent
fromreleaasetoreleaseebutitmayeevolveastheproductmaturesandprod ductcontextevolves.Thee

1
Theminimumleveloffunctionalityisalsosomeetimescalled dtheMinimum mMarketableeFeatures.TThis
termwascoindedbyM MarkDenneaandJaneClelaandHuanginthecontexto oftheIncrem
mentalFundin ng
Method,aanROIorienttedapproachtoscopingso oftwarereleaases[DenneHuangOnSoftw wareByNumbers],
toreferto
ounitsoffunctionalitythaatcanbedeployedtogetheerandwillstaartgeneratingvtangiblevvalue
afterdeplloyment.Weadoptavariaationofthisterm,Minimu umMarketablleFunctionaliity(MMF)incase
thedeliveeredunitisussable,butdoeesnotcoveraawholefeatuurebyourdeffinition.MMFFisalsoknow
wnin
theindusttryasMMP(MinimumMaarketablePro oduct)andMC CR(Minimum mCredibleRellease).

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 42

specificcriteriawouldbeselectedfromthesetofcriteriaineffectatthetimeoftheproject(whichmay
varyfromthosethatwereineffectforearlierreleases.)AnexampleofthisisthattheSarbanesOxley
Act(SOX)wasenactedin2002,soallsubsequentreleasesrequiredcompliancewiththisactasa
readinessand/oracceptancecriteria.Subsequentreleasesmayalsohavebackwardcompatibility
requirementsthatdidnotexistforearlierreleases.Ideally,theMMFandMQRusedbytheProduct
DevelopmentTeaminmakingtheReadinessDecisionshouldbethesameastheMMFandMQRusedby
theProductOwnertomaketheAcceptanceDecision.Inpractice,thisishardtoguaranteewithout
extensivecollaborationbetweentheProductDevelopmentTeamandtheProductOwnerTeam.

1.3.3 TheAcceptanceProcessforAlpha&BetaReleases
Alphaandbetareleasesarewaystouseendusersasfieldtestersbeforeaproductionqualitygeneral
releasetogathermoredataabouttheproductasitmightbeused"intherealworld."Theendusers
maybeinternaltotheorganization,orexternalbutfriendlier(meaningmorebugtolerant)orhavinga
stakeintheoutcome(wantingtoseetheproductearlierfortheirownbenefit.)Insomeorganizations
internalalphatestingiscalleddogfoodingsonamedbecausewhensomeoneproducesdogfood,they
shouldbepreparedtofeedittotheirowndog.
ThefinaloutcomeofalphaandbetatestingisaslikelytoresultinchangestotheMMFandMQRofthe
product(ineffect,newfunctionalityorqualityrequirements)asitistobeacollectionofbugreports
thatdescribefailuretomeettheexistingMMForMQR.
Eachalphareleaseandbetareleasecanbeconsideredaseparatereleasewithitsownreleasedecision
andacceptancedecision.Thatis,thedevelopmentorganizationneedstodeemittobereadyforan
alpha(orbeta)releaseandtheproductownerneedstoacceptitasbeingreadyforalphareleaseasin:
(Iacceptthisalphareleaseashavingsufficientfunctionalityandqualitytowarrantreleasingtousersto
collectfeedbackThisisillustratedinFigure5TheAcceptanceProcesswithAlphaandBeta
Releases.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 43


Figure5
TheAccep ptanceProcessswithAlphaandBetaRelleases
Notethatteveryrelease,whetherallpha,betaorfinalhasareeadinessdecissionandanacceptance
MQR)foranalphareleasearetypicallyllowerthanth
decision.TThefunctionaality(MMF)aandquality(M hat
neededfoorabetarelease,whichislowerthann neededforaggeneralreleasse.Forexamp ple,theMMFFfor
theAlphareleasemaybeacoresub bsetoffunctiionality;notaallfeaturesneeedbepresent.TheMQRmay
be"nosevverity1bugs"and"respon ndsinunder2 2seconds95%ofthetimee(versusthe1second
responsetimerequired dinproductioon)."Typically,theMQRccriteriafortheeBetareleaseewillbebaseedon
theAlphacriteriawithadditionalcrriteriabasedoonimprovem mentspreviouslyplannedaaswellasfeed dback
fromtheAAlphatestinggasshowninFigure6Alp phaorBetaFFeedback.

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 44


Figure6
Alpha(orBeta)Feedback
Similarly,theMMFandMQRforthegeneralreleasewouldbeaffectedbyfeedbackfromusersofthe
Betarelease.

1.3.4 Soaking,FieldTrialsandPilots
Newproductsareoftentestedwithpilotgroupsbeforebeingrolledouttolargergroupsofusers.A
pilotistypicallythefirstproductionqualityreleasewithfunctionallypickedtosatisfyaparticular
targetaudience.Therefore,theMMFbarislowerbuttheMQRneedstobesufficienttosatisfythe
users.UnlikeanAlphaorBetarelease,thisisrealproductionsoftware.Apilotusergroupwilloften
receiveseveraldotreleasesbasedonissuestheyfindandreport.Theinitialpilotmaybefollowedbya
largerpilotgroup,whomayrequireadditionalfeatures,ortheproductmaygostraighttogeneral
availability.Eachofthepilotreleaseswouldbesubjecttotheacceptanceprocesswithbothreadiness
andacceptancetestingprecedingdeployment.
Asimilarstrategyisknownasfieldtrialorsoaking(thinkofthewashingmetaphor:justlikeclothes
youletthesoftwaresoakforawhile,tomakesureitcomesoutclean).Thesoftwareisdeployedto
friendlycustomers/usersforanextendedperiod(morethanjustatestcycle)toseehowitbehavesina
realcustomer/userenvironment.Aswithapilot,readinessassessmentandacceptancetestingwouldbe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 45

donetoensurethattheMMFandMQRaresatisfied.Theoutcomeinallcasesistogatheruserfeedback
thatmaycausetheproductownertoadjusttheMMFandMQRexpectationsofsubsequentreleases,
nottorevisittheacceptancedecisionofthepilotorfieldtrialrelease.

1.3.5 SoftwareMaintenance
Anytimesoftwareneedstobemaintained(suchaswhensmallchangesaremadetothesoftwareand
thosechangesaredeployed),youare,ineffect,creatingaminorinterim,ordotreleaseofthesoftware
thatneedstogothroughtheentiredecisionmakingcycleyetagain.Itiscommontolookforwaysto
reducethecostofgatheringthedatatosupporttheacceptancedecision.Somewaysofdoingthis
increasetheriskofpossiblymissingnewlycreatedbugs(alsoknownas"regressionbugs")byreducing
theamountoftesting(forexample,riskbasedtestplanning)whileotherssimplyreducetheeffortto
getsimilartestcoverage(forexample,automatedregressiontesting).
Anotheruniqueaspectofsoftwaremaintenancerelatestothewarrantyperiodonasoftwarerelease.
Anychangesthatneedtobemadetothesoftwareshouldbemadeinthesourcecodemanagement
(SCM)system.Whenbuildingmultiplereleases,theremaybeongoingdevelopmentforthenextrelease
thatshouldnot,underanycircumstances,beinsertedintotheproductionsystemalongwiththe
warrantybugfixes.Thisrequiresmanagingseparatecodestreamsorbranchesduringthewarranty
periodandensuringthatallwarrantyfixesarealsoappliedtothenewdevelopmentcodestream.SCM
isalsoneededduringacceptancetestingifthedevelopmentforthenextreleaseismovingforwardin
parallel.Forpracticalstrategiesforusingsourcecodemanagementsystems,see[BerczukOnSCM]and
[MSOnSCM].

1.3.6 ConditionalAcceptance/Readiness
Frequently,theacceptancedecisionmakeracceptsaproductwithconditions.Acceptingaproductwith
conditionsisashorthandwayofsaying,
"Theproductisnotacceptableyet,butitisclosetomeetingourcriteriaforfunctionality(MMF)and
quality(MQR).Ifyouaddressthefollowingconcerns(andwefindnothingnewinthesubsequentround
ofacceptancetesting),weintendtoaccepttheproductinthenextpassthroughthedecisionmaking
process."
Conditionalacceptancebringstheprocessbacktotheconstruction/developmentphaseofthe
acceptanceprocess,butthistimewithamuchbetterideaofexactlywhatmustbedonetomakeit
throughboththereadinessdecisionandtheacceptancedecisiononthenextround.

1.3.7 TheAcceptanceProcessforHighlyIncrementalDevelopment
Theacceptanceprocessasdescribedthusfarlooksverysequentialinnaturebutitcanalsobeusedina
highlyincrementalfashiononagileprojectsbyapplyingworkflowscheduleindependence(seeSidebar
RecastingaSequentialProcessesthroughWorkflowScheduleIndependenceandParallelization).Each
chunkoffunctionality(oftencalledafeatureorauserstory)canbepassedthroughtheacceptance
processindependentlyasshowninFigure7TheAcceptanceProcessAppliedtoIncremental
Development.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 46

Figure7
TheAcceptanceProcessAppliedtoIncrementalDevelopment
TheindividualMMFforeachfeatureoruserstorymaybe:
independentoftheMMFofotherfeaturesoruserstories;or
incrementalwhenitmaybuildontheMMFofpriorfeatures.
TheMQRmaybedifferentfordifferentincrements,althoughitsmoreconsistentacrossincrements.For
example,theMQRforanalphareleaseistypicallylowerthantheMQRforaregular,generalrelease.
WhenMQRrelatestononfunctionalrequirements,itappliesacrossfunctionalfeatures.
Atsomepoint,aproductlevelreadinessandacceptancedecisionsaremadetoensurethateverything
worksproperlytogether.RefertoAcceptingtheOutputofFeatureTeamsinChapter2Elaboratingon
theAcceptanceProcessforamorecompletediscussion.
Inincrementaldevelopment,thedevelopersmaypreexecutetheacceptancetestsWhenallthetests
passforthatfeatureoruserstory,theymayturnoverthefunctionalitytotheProductOwner(orother
customerproxysuchasabusinessanalystoracceptancetester)forimmediate"incrementalacceptance
testing."Therefore,acceptancetestingatthefeaturelevelmaystartimmediatelyaftertheproduct
developmentteamdecidesthatallormostofthefunctionalityforthatfeatureisbuiltandworking
correctly,allowingincrementstooverlap.Thisisaninstanceofincreasingefficiencybyintroducing

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 47

parallelism
mintothewoorkflowwhen nresourcesm maybeseparaate(seeSideb barRecastingaSequential
ProcessessthroughWo orkflowSched duleIndepend denceandParallelization)..Theremayaalsobearoundof
acceptanccetestingperrformedatth heendoftheiteration,asiillustratedinFigure4ofth
heIntroductio
onby
themediu umlengthverticalbarsrep presentingiteerationwidetesting.Theaactivitiescondductedforeaach
featurewwithintheiteraationareillusstratedinFigu
ure8.


Figure8
AgileDeveelopmentwitthIncrementa
alAcceptanceeTesting
Developmmentisdoneo onefeatureaatatimebyeitherasingledeveloperorrasmallteam mofdevelopeers,
documenttationwriterss,testersand duserexperieencepeople,ddependingon nthesizeoftthefeature.((See
thesidebaarWholeTeaamformoreinformation.)InFigure8 AgileDevelopmentwithIncremental
AcceptancceTesting,Deev1andDev2couldeach hbeasingled
developeroraasmallteam..Whenthe
developmmentworkisccomplete,theedeveloper,p possiblyassisttedbyatesteerorbusinesssanalystcond ducts
readinessassessmentonthefeaturrebyrunningalltheknownunit,compo onentandbu usinessaccepttance
ware.Iftheyfiindanyproblems,theyfixthembeforeeproceeding.Thesetestsare
testsagainstthesoftw
oftenautoomatedandaactasaformofmistakeproofingorbu ugrepellent.TTheyareconssistentwith
ShingosaadageInspecctiontofinddefectsiswaste;inspectio
ontoprefentdefectsisesssential
[ShingoDillonOnToyotaaProductionSSystem].
Whentheeyaresatisfie
edthatthesoftwareisworrkingproperlyy,theyasktheProductOw wner(ora
membero oftheProducctOwnerTeam)toruntheeiracceptanceetestsforthaatonefeaturee.IftheProduct
Ownerfin
ndsanyproblems,theysho owtheprobleemstotheteeamwhothen nfixestheprooblems
immediattelyandrepeaatsthereadinnessassessmeentontherevisedcodebeeforegivingitttotheProdu uct
Ownerforfurtheracceeptancetestinng.WhentheeProductOwnerissatisfieed,theyaccep ptthefeatureeand
thedevelo
oper(orteamm)startsworkkingontheirnextfeature.Thispracticee,calledIncreemental
AcceptancceTesting,haastwokeybeenefits.First,aanyconcernffoundbytheProductOwn nerduring
acceptanccetestingcan
nbediscusseddwiththedeeveloperswhiletheystillreemembertheedetailsofho ow
theyimplementedthefunctionalityy.Second,theedefectsord deficienciescaanbeaddresssedimmediattely
beforetheedevelopermmovesontotthenextfeatu ureinsteadofbeingstockpiledfora"bugfixingphase."

1.3.8 TheRoleof
T fBugFixin
ngintheA
Acceptance
eProcess
Findingb
bugsduringaccceptancetesstingisnormaal.Howwereeacttothebu
ugstellsusallotaboutour
overallap
pproachtoproductdeveloopment.Weccanusebugsttomotivateim mprovementstoour

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 48

developmentprocessorwecanjustfixthebugswithoutunderstandingtherootcause.Thelatter
approachoftenresultsinalargenumberofbugsbeingfoundandhavingtobefixed.Wecaneitherfix
thesebugsrightawayorwecanstockpilethemforabugfixingphase.Suchastockpileofbugsis
consideredaformofwasteinleanmethodologiesbecausewearebuildingupaninventoryofpartially
finishedsoftware.AccordingtoTomandMaryPoppendieck,thoughtleadersoftheLeanSoftware
Developmentapproach,itisirresponsibletoorganizetheworkinsuchawaythatTriageisnecessary.
Theyarguethatitisprimarilythedecisiontotestlateratherthantomistakeproofeverystepthat
escalatesthecostofrepairtothepointthatonewoulddeliberatelychoosetoleaveknowndefectsin
theproduct.Thus,theneedforbugtriagecanbeconsideredasaprocesssmellThereare,however,
circumstanceswhenyourdevelopmentisorganizedinawaywhenabugbacklogand,therefore,triage
activitiesarenecessary.Thistendstohappenwithlargeand/ordistributedteamsorwhenproduct
ownerisnotreadilyavailable.Inthiscase,itisimperativetokeepinmindthefollowingpoints:

1) Triageshouldonlylookatnewbugsreexamingoldbugsrepeatedlyiswaste.Thepurposeof
triageshouldbetodecidewhethera)weneedtofixthebugforthesoftwaretobeconsidered
done,theissueisnotreallyabugandthereforerequiresnoaction,orb)thebugrelatesto
futurefunctionalityandcanbeaddressedwhenthatfunctionalityisbuilt.
2) Toavoidwaste,weneedtokeepthedepthofthebugbacklogreasonablysmall.Reasonably
smallisafunctionofourdevelopmentandbugfixingcapacity.Ifonalargeprojectour
developmentteamcanfix200bugsperiterationonaverage,thenthebacklogof200bugsisnot
alot.However,ifourdevelopmentcapacityissuchthatwecanonlyfix10bugsperiteration,
clearlyabacklogofof100shouldbeconsideredasaredflag.
3) Thekeygoalistomakethebugfixlatency(measuredasaMeanTimeToFix)toaminimumby
performingregulartriagesandschedulingbugfixestopreventthebugbacklogfromexploding.
Averagetimetofixbugsshouldbenomorethanafewiterations.Longlivedbugsisasmell
indicatinginventory(aformofwaste.)Thiscanbeavoidedbyensuringwegettodonedone
oneachfeaturebeforemovingontonextfeature.
4) Batchingrelatedbugsforfixingorincludingtheminaplannedfutureuserstory/featurecanbe
consideredasapossiblestrategy(forexample,weknowthatthebugisrelatedtoafeaturethat
wewillbeworkingoninthenextiterationandwhichwillrequireaseriousupdateor
deprecationofsomerelatedfunctionality;inthatcaseitmaymakesensetowaittillthatworkis
completed),weneedtomakesurewearecognizantoftheimpactofsuchbatchingonour
latency.
Foradetaileddiscussionofbugbackloganalysis,bugtriageandtheuseofbugmanagementsystems,
refertoChapter19ManagingtheAcceptanceProcess.

1.3.9 UsingAcceptanceTeststoDriveDevelopment
TheacceptanceprocessworksbestwhentheProductOwnersuppliestheacceptanceteststothe
developmentteamearlyenoughtohelpthemunderstandwhatdonelookslike.Ideally,thisisbefore
developmentevenstartsbutitmustdefinitelybebeforedevelopmentisfinishedattheverylatest.The
bestresultsareachievedwhentheseacceptancetestsarenothandedoff,butratherdeveloped

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 49

together,forthiswaytheacceptancetestsarenotonlyusedforfindingbugsintheproduct,butrather
clarifyrequirementsandimproveconversationsbetweentheProductOwnerandtheproduct
developmentteam[MelnikEtAlOnExecutableAcceptanceTests].Thefocusisoncollaborationnot
handoff.ThispracticeisknownasAcceptanceTestDrivenDevelopment(ATDD)orStorytestDriven
Development(STDD).Whenusedinconjustionwithincrementaldevelopment,theteamwilltypically
automatemuchofthetestingtokeepthecostofrepeatedlydoingreadinessassessmentlow.The
sidebarWhatittakestodoIncrementalAcceptancedescribesothersuccessfactorsfordoinghighly
incrementalacceptancetesting.

Summary
Thischapterintroducestheconceptoftheacceptanceprocessasawaytothinkabouttheactivities
relatedtomakingthedecisiontoacceptsoftware.Theacceptancedecisionistheculminationofa
processthatinvolvesseveralorganizationseachwithspecificresponsibilities.Eachreleasecandidateis
passedthroughthedecisionpointsoftheacceptanceprocessonitswaytomakingthefinalacceptance
decision.TherearethreemajordecisionpointsbetweensoftwareconstructionbytheProduct
DevelopmentTeamworkingwiththeirProductOwnerandsoftwareusagebytheenduser.First,the
ProductDevelopmentTeammustdecidewhetherthesoftwareintensiveproductinitscurrentform,
knownasthereleasecandidate,isreadyforacceptancetesting.Ifitis,theProductOwnermustdecide
whetherthereleasecandidatemeetstheiracceptancecriteriabeforetheproductcanbemadeavailable
totheendusers.Ultimately,eachendusermustdecideforthemselveswhethertheywanttousethe
software.Thisusagedecisionhasverylittleinfluenceonthecurrentacceptancedecisionalthoughit
mayinfluencetheacceptancecriteriaforfuturereleasesoftheproduct.
Whileeachuserpotentiallymakestheirownusagedecision,thereadinessandacceptancedecisionsare
normallyeachasingledecision.Eachdecisionpointor"gate"shouldhavewelldefinedcriteriatoguide
thedecisionmaking.ThesecriteriashouldbeknowninadvancebyboththeProductDevelopment
TeamandtheProductOwner.
Theacceptancedescribedinthischapterappliestobothsequentialandagileprojectsbutinslightly
differentways.Thephasednatureofsequentialprojectsmeansthatalltestingisdonewithinaseparate
testingphaseandtheacceptanceprocessdescribeswhatgoesonwithinthatphase.Agileprojects
traversetheentiredevelopmentlifecycleforeachfeatureoruserstory.Therefore,theacceptance
processisexecutedatseverallevelsofgranularitywiththefinestgrainexecutionbeingattheindividual
featurelevelandthelargestbeingatthewholeproduct(orfeatureintegration)level.

WhatsNext
InChapter2ElaboratingonTHEAcceptanceProcesswegointomoredetailabouthowtheacceptance
decisionisrelatedtotheusagedecisionandhowtheprocessisimplementedincomplexorganizations
andwhenbuildingcomplexproducts.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 50

References
[CooperOnDoingItRight]RobertG.Cooper,DoingitRight:WinningwithNewProducts,Ivey
BusinessJournal,July/August2000.AlsoavailableasaProductDevelopmentInstituteReference
Paper#10asathttp://www.stagegate.com.
[CooperOnStageGateProcess]Robert.G.Cooper,TheStageGateIdeatoLaunchProcessUpdate,
WhatsNewandNexGenSystems,J.ProductInnovationManagement,Volume25,Number3,
May2008.AmodifiedversionalsoavailableasasaProductDevelopmentInstituteReference
Paper#30Perspective:TheStageGateIdeatoLaunchProcessUpdate,WhatsNewand
NexGenSystemsathttp://www.stagegate.com.
[FewsterGrahamOnTestAutomation]MarkFewsterandDorothyGraham,SoftwareTest
Automation,1999,ACMPress
[ShingoDillonOnToyotaProductionSystem]Shingo,ShigeoandDillon,Andrew,AStudyofthe
ToyotaProductionSystem:FromanIndustrialEngineeringViewpoint:ProductivityPress,1989.
[HighsmithOnAgileProjectManagement]Highsmith,JimAgileProjectManagement:Creating
InnovativeProductsAWP2004ISBN13:9780321219770
[BerczukAppletonOnSCM]Berczuk,SteveandBradAppleton,SoftwareConfigurationManagement
Patterns:EffectiveTeamwork,PracticalIntegration,AddisonWesley(2003)ISBN:0201741171
[MSOnSCM]Patterns&PracticesTeamDevelopmentwithMicroosftVisualStudioTeam
FoundationServer,Ch.3,4,6:MicrosoftPress,2008.
[OMGOnSwimlanes]ActivityPartitionNotations.InUMLSuperstructureSpecification,pp.342345
http://www.omg.org/spec/UML/2.2/Superstructure/PDF/,Oct18,2009
[PoppendiecksOnLean]Poppendieck,Mary&TomLeanSoftwareDevelopmentAddisonWesley
(2003)ISBN:0321150783
[DenneHuangOnIFM]MarkDenneandJaneClelandHuang,"TheIncrementalFundingMethod:
DataDrivenSoftwareDevelopment,"IEEESoftware,vol.21,no.3,pp.3947,May/June2004
[LadasOnScrumban]CoreyLadas,Scrumban:EssaysonOSystemsforLeanDevelopment,Modus
CooperandiPress,2008
[HiranabeOnKanban]KenjiHiranabe,KanbanAppliedtoSoftwareDevelopment:fromAgileto
Lean,publishedonlineat<http://www.infoq.com/articles/hiranabeleanagilekanban>,
January14,2008
[MSOnTFSBranchingGuide]TFSBranchingGuide2.0,http://tfsbranchingguideii.codeplex.com/,Aug
15,2009
[PrimaveraOnScrum]PrimaveraScrumCasestudy.
http://controlchaos.com/download/Primavera%20White%20Paper.pdf,Oct19,2009

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 51

[KnibergOnScrumFromTrenches]HenrikKniberg,ScrumandXPfromtheTrenches,
http://www.infoq.com/minibooks/scrumxpfromthetrenches,Oct19,2009
[DenneHuangOnSoftwareByNumbers]MarkDenneandJaneClelandHuang,SoftwarebyNumbers:
LowRisk,HighReturnDevelopment,PrenticeHall,2003
[MelnikEtAlOnExecutableAcceptanceTests]GrigoriMelnik,FrankMaurer,MichaelChiasson,
"ExecutableAcceptanceTestsforCommunicatingBusinessRequirements:Customer
Perspective",Proc.Agile2006,IEEEComputerSocietyPress:3546,2006

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 52

Chapter2
2 Elabor
E ratinggonth
he
AccceptaanceProceess
Chapter1 1introducedttheacceptancceprocessandhowitfitsiintotheovera
allproductdeevelopment
lifecycle.TThedescriptio
onwaswasd
deliberatelysimple.Thischapterintrodu ucessomewa aystovarythee
acceptancceprocesstodealwithmo orecomplexssituationsinclludingcompleexorganizatio onsandprodu ucts.

2.1 TheRoleoftheUssageDecission
Thusfarw
wehavefocussedalmosten ntirelyontheereadinessan
ndacceptanceedecisions.W
Whataboutth he
usagedeccision?Itturn
nsoutthattheeusagedecissiondoesntd
directlyimpaccttheacceptaancedecision
n
becauseitthappensafttertheproductisalreadyaaccepted.

2.1.1 Separating
gtheAccep
ptanceDeccisionfrom
mtheUsag
geDecisio
on
Whenasooftwareproductisbeingsselectedorbu uctOwnerbyanotherparty,the
uiltspecificallyforaProdu
ProductO
Ownermayde ecidetoacceptthesoftwaareassuitableebeforeofferingittotheuserswhowiill
thenmaketheusaged
decisionindivviduallyasillustratedinFiggure9.


Figure9
AcceptancceandUsageeDecisionsan
ndCriteria
Almostallproductshavemanyuserrsandeachisslikelytohavvetheirownu usagecriteriawhichwhileit
mayhaveeinfluencedtheacceptanccecriteriauseedduringthereadinessandacceptanceedecisions,m mayin
factbequ
uitedifferentfromit.Inso
omecases,theeuseofthepproductisnottoptional,suchwhensoftware
isusedtocarryoutaspecificjobsu orairlinereseervationssystem.Inthesecases
uchasacallceenteragento
theusageedecisionism
madeonbehaalfoftheuserrsbytheirmaanagement;thisdecisionn neednotfollo
ow

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 53

immediatelyaftertheacceptancedecisionastheremaybeotherprerequisitesforusagesuchasuser
training.
Whileausercould,intheory,choosenottousethesoftware,theconsequenceoflosingtheirjobwould
makethisdecisionhighlyunlikely.Theusersfeedback,however,mightinfluencesubsequentreleasesof
thesoftwareaswellseeinFeedbackfromtheUsageDecisions.

2.1.2 DeterminingtheAcceptanceCriteria
Theacceptancedecisionismadebasedonsomecriteriaincludingbothfunctionalityandqualityrelated
elements.Theminimumleveloffunctionality(representedbyMMF)andtheminimumlevelofquality
(MQR)shouldbedeterminedwellinadvanceoftheacceptancedecisiontogivetheProduct
DevelopmentTeamofthesoftwareareasonablechanceatbeingabletosatisfythem.Therefore,the
decisionsabouttheMMFandMQRcomebeforetheAcceptanceDecisionasshowninFigure9
AcceptanceandUsageDecisionsandCriteria.

2.1.3 FeedbackfromtheUsageDecisions
Notethattheusagedecisionsmadebyindividualusersaftertheacceptancedecisiondonotdirectly
affecttheacceptancedecisionbutitmay,throughsalesdata,postsalessupportdataorfocusgroups
affectthedefinitionofMMForMQRforsubsequentreleasesoftheproduct.ThisisillustratedinFigure
10UsageDecisionFeedback.


Figure10
UsageDecisionFeedback

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 54

WhiletheMQRtendstobefairlyconstantfromonereleasetothenext,theProductOwnermayrevise
thequalitycriteriaforasubsequentreleasebasedonthefeedbackreceivedfromusers.Forexample,
usertestingofanalphareleasemightrevealthattheresponsetimesspecifiedintheMQRarenotfast
enoughinpracticeandmightcausetheproductownertomakethemmorestringentforthebetaor
finalrelease.
TheMMFforeachreleasetendstobeuniquebecauseitisthelistoffunctionalitythatneedstobe
includedtowarranthavingtherelease.FeedbackfromuserscaninfluencetheMMFbasedonrequests
formissingfunctionality,suggestionsforhowexistingfunctionalityshouldbechanged,andthroughbug
reports.
FeedbackfromusersofAlphaandBetareleasesoftenresultsinchangestotheMMForMQRofthefinal
release.

2.2 AcceptanceinComplexOrganizations
Thesimpleacceptanceprocessdescribedpreviouslywillsufficeinmanyorganizations.Butthereare
someorganizationswhosestructureisverycomplexasaretheproductstheydevelop.These
organizationsrequireadecisionmakingprocessthatmirrorsthecomplexityoftheorganizationor
product.Thedecisionsmaybemadeinseriesaseachdecisionmakerexaminestheproductbutthis
leadstolongerelapsedtimesbeforethevalueoftheproductcanberealized.Thecommonsolutionisto
haveeitherthereadinessoracceptancetestingactivitiesanddecisionbemadeinparallelbyseveral
organizationseachwithvetopowerovertheacceptabilityoftheproduct.

Notetoreaders:Ifthesescenariosdontapplytoyourorganization,feelfreetoskipaheadto2.3
AcceptingComplexProducts.2.3

2.2.1 ParallelAcceptanceDecisions
Manyorganizationshaveotherdepartmentswhoareresponsibleforoperatingandsupportingthe
systemonceitisdeployed.Thesedepartmentsneedtohaveasayintheacceptancedecision.Thisis
reflectedinFigure11ParallelAcceptanceDecisions;itshowsseveralacceptancedecisionsbeingmade
inparallel,eachbasedonadifferentsetofacceptancecriteria.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 55


Figure11
ParallelA
AcceptanceDeecisions
Whentheereareseveraalorganizatioonsinvolvedinntheacceptaancedecisioneachwithth heirownareaof
specialty,eachmayinssistinconducctingitsownaactivitiesindeependentlyofeachother.Eachmakesttheir
owndecissionregardinggthereleasecandidatean ndanyoneoffthemcanefffectivelyvetootheothers.
Examplesofsuchstake
eholders,and
dtheirareao
ofinterest,co
ouldinclude:

B
BusinessUnitFunctionaliityisacceptab
ble.

O
OperationsDe
epartmentTTheproductccanbemanaggedandsupportedinprod
duction.

C
CorporateBra
andingProductcomplieswithbrandin
ngrequirements.

C
CorporateSec
curityProdu
ucthascomplliedwithsecu
urityrequirem
ments.

U
UsabilityGrou
upProductccomplieswith
husabilityreq
quirements.
ThisapprooachintroduccesriskiftheMMFandMQRusedbyeeachstakeholderarenotin ncludedinthee
MMFand dMQRusedb bytheProductDevelopmentTeam.Ideaally,eachstakeholderwou uldparticipatteat
leastparttimeinthePProductDevelopmentTeamtoensuretthattheteam mtrulyundersstandsthe
acceptanccecriteriaand ddesignstheebestpossibleesolutiontothebusinessneed.Thispaarticipationcoould
befortheeentiredurattionoftheproojectorforoneortwoiterationsfocusedonthatstakeholdersneeds.
Figure12AProductD
DevelopmenttLifecyclewitthParallelElaaborationand
dAcceptancebyProduct
OwneranndOperationssillustratessthescenario
owheretheP
ProductOwneeristhepartyywhohas

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 56

conceived
dthesoftwareasasolutio
ontoabusineessproblemandtheoperaationsdepartm
mentistasked
withsupp
portingthesyystemonceitisputintoprroduction.


Figure12
AProducttDevelopmen
ntLifecyclewiithParallelEllaborationan
ndAcceptanceebyProductO
Ownerand
Operationns

2.2.2 ParallelRe
P eadinessD
Decisions
Insomecasesseveraloorganizationssdotheirown nreadinessassessmentoffthereleaseccandidateand d
makeindeependentreaadinessdecisions.Theprod ductwillnotbehandedovvertothepro oductownerffor
theaccepptancedecisio
onifeitherorrganizationdeeemstheproductnotreaadyuntilanyyadditionalw work
theyidenttifyhasbeencompletedandtheydeem medthesoftw wareready..ThisisillustrratedinFigurre13
ParallelReadinessDeecisions.

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 57

Figure13
ParallelR
ReadinessDeccisions
Examplesofpotentialstakeholderswithparallelreadinessdeecisionsinclud
de:

P
ProductDevel
lopmentTeam
mCodeispaassingallunitttestsandfun
nctionaltestss

S
SecurityTeam
mDesignhasspassedsecu
urityreview

D
DataArchitect
tureTeamD
Dataarchitecturehasbeen
napproved

LegalDepartm
mentAllembeddedtechn
nologieshaveebeensuitablylicensed.
Again,eacchofthesesttakeholdersshouldbecon
nsideredmem
mbersofthew
wholeteam,eevenifonlypart
time,toeensurethatth
heMMF/MQR Rareaddressedbywhatth
heProductDevelopmentTTeambuilds.
Whetheradecisionisddeemedtobeepartofread dinessoracceeptanceissom mewhatarbittrary.Thecho oiceis
usuallybaasedonwhennthedecisionnmakerwanttstogetinvolvedandwhicchsideofalaarger
organizationalboundarythestakeh holdersits.Fo
orexample,in
nsomeorganizationsthessecurity
departmeentmaybeco onsideredabusinessfuncttionwhomusstacceptthereleasecandiidatewhilein n
mayhaveapresenceinorrneartheITd
othersitm departmento orproductdevvelopmentgrroupandtherreby
choosetooparticipatew
whenthereadinessdecisio onismade.O
Ofcourse,itissusuallyprefeerabletoinvo
olve
theseparttiesmuchearlierintheprrojectinamo
orecollaborattivefashionsoothatreadinessassessmeentis
conducted dincrementaallyasthepro
oductisdeveloped.

2.3 Accepting
gComple
exProduccts
Organizattionsthatbuildlarge,complexproductsstypicallyuseeamorecom
mplexversionoftheaccepttance
processth
hatinvolvesm
multistageaccceptance.Laargecomplexproductstyp picallyrequireelargenumbeersof

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 58

people,to
oomanytow workinasingleteam.Whilethelargeco
omplexprodu uctsaretypicallycomposeedof
components,theteam msthatbuildtthemmaybeorganizedinseveralwayss.Eachcompo onentmaybee
builtandownedbyaccomponentteeamorteamssmaybeorgaanizedaround dthefunctionnalityprovideedto
theusers..Thelatterap
pproachiscalledfeaturecrewsandisswidelyused
dbytheDeveloperDivision nof
Microsoftt[MillerCarterOnLargeScalleAgile].Themannerinw whichthesepeeopleareorgganizedimpaccts
theaccepptanceprocesss.Thekindso ofproblemsffoundduringtheacceptannceprocesscaanbeagood
indication
noftheappro opriatenessofthewaywehaveorganizzedourteams.

2.3.1 Acceptingt
A theOutputofCompo
onentTea
ams
Acommonwayofsubdividingworkkinlargeorgaanizationsisttodecomposeetheproducttinto
componentsthatareb builtandveriffiedinparalleelbyseparateecomponentteamsthateeachspecializeein
theirresp
ptivetechnolo
ogiesandtech hnicaloruserrdomains.Whenthisorgaanizationalsolutionisused
d,
eachcomponentshouldhaveitsow wnreadinessassessmentp priortocondu
uctingreadinessassessmeenton
theintegrratedproductt.ThisisillusttratedinFigure14ComponentTeamsswithSerialDDecisions.


Figure14
ComponeentTeamswitthSerialRead
dinessDecisio
ons
Eachcomponentmusttgothroughaacomponent readinessdeecisionbeforeebeingintegrratedintothee
largerpro
oductforsubssequentprod ductlevelread dinessdecisio
on.Theintegratedproductisthenturn
ned
overtoth
heProductOw wnerfortheaacceptancedecisions.Thereadinessasssessmentacttivitiesmaygo oby
variousnaamesincludinngcomponenttestingorrsystemtesttingfortheccomponentsandsystem
testingo
orintegrationntestingforrtheentireprroduct.EachccomponenthhasitsownM
MMFandMQR R;
hopefully,,thesearedirectlytracabllefromtheMMMF/MQRforthereadinessandaccepttancedecisions
madeabo outtheentireeproduct.

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 59

Thisreveaalsseveraloftheweaknesssesofthisap pproachtocomplexproducts.First,beccausethe
componentscannotbe edirectlyundderstoodandthereforeaccceptedbytheeProductOw wner,some
intermediiary,usuallyaanarchitecthhastoberespponsiblefortrranslatingProoductOwnerrequirementsinto
componentrequireme ents.Thiseffeectivelytransffersproducto ownershipoff thecomponeentfromthe
ProductOOwnertotheintermediaryytherebyredu ucingthedeggreetowhichthecomponentteamis
accountab bletothereaalProductOw opossibleforthecomponeentrequirementstodriftffrom
wner.Itisalso
whatisacctuallyrequireed.Second,theproductreeadinesscann noteasilybeassesseduntilallofthe
componentshavepasssedtheirresp pectivereadinnessdecisionssandareinteegratedtoforrmtheproduct.
Thisdelayystheproducctreadinessassessmentun ntilafterthebigbangintegrationoftthecomponents,
discoverthatsomeoftheccomponentreequirementswere
whichisratherlateinaaprojecttod
misunderstood.Thisproblemcanb besomewhatovercomethroughtheuseofcontinuo ouscode
on(CI)practiccescombinedwithincremeentalintegrattion[BasLarm
integratio manOnScalinggAgile]and
incrementalacceptanccetestingbuttinpracticeittismuchhard dertodoincrrementaldeliveryusing
componentteams(com mparedtofeaatureteams)andfeaturetteamswilltyp picallybeabletodeliver
productqqualitysoftwaareforindividdualfeaturesearlier.Italssoperpetuateesthetechniccalcomponen nt
etechnicalcomponentdevvelopersfrom
ghettobyyisolatingthe manyundersttandingoftheebusiness.

2.3.2 Acceptingt
A theOutputofFeaturreTeams
Anotherw wayoforganizingthedeveelopmentoflargecomplexxproductsisttousefeatureteams/crew ws
[BasLarmaanOnScalingA Agile,Eckstein
nOnLargeScalleAgileandM MillerCarterOn Agile].Eachfeature
nLargeScaleA
teamisreesponsibleforrbuildingand dverifyingaffeaturebased
dontheProductOwnersaacceptance
criteria.B
Buildingeachfeaturemayinvolvemodifyingseverallcomponentssbuttheteam mincludes
expertiseoneachcom mponentandeensuresthattthecomponeentsarepropeerlyintegrateedaspartofttheir
featurereeadinessassessment.Therrefore,theprroductreadinessassessmeentandtheprroductaccepttance
testingarebothfocuse edonfeatureeintegrationrratherthanooncomponenttintegration..Thisscenario
ois
illustrateddinFigure15FeatureTeeamswithSerrialAcceptancceDecisions.

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 60

Figure15
FeatureTeamswithSerialAcceptanceDecisions
EachfeaturesreadinessdecisionismadebasedontheMMFandMQRspecifictothatfeatureas
determinedbytheProductOwner(ideallywithinputfromtheProductDevelopmentTeam);no
intermediaryisrequiredtotranslatetheProductOwnersrequirementsintocomponentrequirements.
Thefeaturereadinessdecisionisaprerequisiteforthecorrespondingfeaturesacceptancedecision
whichshoulduseroughlythesameacceptancecriteria.Meanwhile,theproductisintegratedanda
productlevelreadinessdecisionismadebasedontheproductsMMFandMQR.Theproductlevel
MMFshouldsimplybeanaggregateofthefeaturelevelMMFsThereforetheinteractionsbetween
featuresneedtobeincludedaspartoftheMMFforeachaffectedfeatureorasseparatefeature
interactionfeatures.Whentheproductlevelreadinessandallfeaturelevelacceptancedecisionsare
positive,theproductlevelacceptancedecisioncanbemade.
Thisstyleoforganizationandacceptanceisusuallypreferabletocomponentteamsbecauseitensures
eachteamisworkingwitharealendProductOwnerandisresponsiblefordeliveringawholeproduct,
notjustaninternalcomponent.SeethesidebarFeatureCrewsonMicrosoftOfficeforanexampleof
howthisworksinpracticeinChapter9ProductManagersPerspective.
IftheProductOwnerisahighlyconstrainedresource,havingalargenumberoffeatureteams
contendingfortheirparticipationmayreducetheeffectivenessofthisapproach.Iftheinteractions
betweenfeaturesarealargerriskfactorthantheriskoftechnicalintegrationwiththetechnical
components,itmaybemoreappropriatetoreducethenumberoffeatureteamsbydelegatingmoreof
theworktocomponentteamssothatfeatureinteractionscanbeaddressedwithsinglefeatureteams
ratherthanbetweenthem.

2.3.3 AcceptingCustomizableProducts
Manyproductsarebuiltbyoneorganizationforsaletoanotherorganization,theircustomer.Whenthe
productneedstobecustomizedforeachcustomertherearetwodistinctacceptancephasesforthetwo
differentproductsinquestion.Thefirstproductisthegenericproductbuiltbythesellingorganization
totherequirementsspecifiedbytheproductmanager.Thesecondproductistheonebuilttothe
specificationofthepurchasingcustomer.Theconstructionofthelatterproductmayincludeeither
configurationorcustomcodinginadditiontothegenericcustomizableproduct.
Thefirstacceptancedecisioniswhentheproductmanageracceptsthecustomizableproductassaleable
tothecustomers.Thisacceptanceprocesscouldbeanyofthevariantsdescribedpreviouslyincluding
AcceptingComplexProducts.
Thesecondacceptancedecisioniswhentheacquirerofthecustomizableproductacceptstheproduct
ascustomizedtotheirownsituation.Thereisasingleacceptancedecisionforeachversionofthe
customizablegenericproductandoneacceptancedecisionperversionofthecustomizedproductas
showninFigure16Acceptingcustomizableproducts.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 61


Figure16
Acceptingcustomizableproducts

Sidebar:AcceptingConfigurableERPProducts.

Manysoftwareproductsaresoldasgenericimplementationsofabusinessprocessthatcanbe
configuredtosuitabusinessspecificneeds.TwogoodexamplesofthisareERPsystemssuchasSAPs
ERPproductandMicrosoftsDynamics.Bothprovidegenericimplementationsofcorebusiness
processessuchastimereportingandpayroll,supplyorderingandaccountspayable,productorder
processingandaccountsreceivable.Theoutoftheboximplementationsoftheseprocessescanbe
modifiedthoughconfigurationdataorextendedbypluggingincustomwrittenprocedures.Fora
particularcustomersinstallationoftheERPsystem,itgoesthroughtwocompletelydistinct
acceptanceprocesses.First,theproductownerattheconfigurableproductvendordefinesthegoals
ofthenextreleasebysynthesizingnewfeatureideasfromtheproductdefinitionteam,bugreports
frompreviousreleasesandnewfeaturerequestsfromnewandexistingcustomers.Thisgetsturned
intothefunctional(MMF)andnonfunctional(MQR)requirementsfortherelease.TheERPproductis
built,goesthroughreadinessassessmentbytheproductdevelopmentteam,acceptancetestingbythe
productmanagementteamandprofessionaltestersbeforebeingdeclaredreadytoshipto
customers.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 62

Meanwhile,thecustomersaredefiningtheirowngoalsandobjectivesfortheinstallationbasedonthe
expectedfeaturesetofthenewreleaseandtheirspecificbusinesscontext.Eachcustomershould
haveitsownproductownerforthecustomizedproduct.Theymaydobusinessprocessmodellingand
gapanalysisbetweentheirbusinessprocessandtheoutoftheboxproducttodefinetheirown
MMFandMQR.Whentheproductships,theirownproductdvelopmentteaminstallsitintotheirown
developmentsystemanddoesthecustomizationand/orconfiguration.Ifthisistheirfirstinstallation
oftheproductthentheyadjustthevariousconfigurationparametersandimplementnewplugin
procedures.IfthisisanupgradefromapreviousversionoftheERPsystemthenmuchofthework
consistsofreapplyingtheconfigurationfromtheirprevioussystemanddoingregressiontestingofthe
functionality.Inbothcases,thereleasecandidate(s)shouldtraversethefullacceptanceprocesswith
thecustomersproductdevelopmentteammakingtheReadinessDecisionandthecustomersproduct
ownermakingtheacceptancedecisionwhichisbasedontheMMFandMQRdefinedbythe
customersproductownerfortheirspecificinstallationoftheERPsystem,nottheMMFandMQR
definedbythevendorsproductownerforthegenericproduct.Inaninterestingparadox,ERPsystems
wouldseemtobeimminentlysuitabletoagilecustomizationandincrementalacceptancetesting
becausetheyworkrightoutoftheboxandcustomizationevolvestheproductfromagenericonetoa
customizedone.ButmanyERPsystemslackthetoolstoautomatethebuildandregressiontesting.
SeethesidebarWhatittakestodoIncrementalAcceptanceandthepapers[MeszarosOnAgileERP]
and[AstonOnAgileERP]formoreinformationonwhatneedstobeavailableandmightbemissingin
yourERPsystem.

2.4 Exitvs.EntryCriteria
Manyofthedecisionsintheacceptanceprocessinvolvehandoffsbetweenparties.Eachhandoff
involvestwosetsofcriteria.Thepartymakingthehandoffmayhavesomeexitcriteriathatmustbe
satisfiedbeforetheyarepreparedtomakethehandoff.Thepartyreceivingthehandoffmayhavesome
criteriatobesatisfiedbeforetheyarepreparedtoreceivethehandoff.Thesecriteriacanbeassessedat
separateexitandentrydecisionsasillustratedinFigure17SeparateExitvs.EntryDecisions&Criteria.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 63


Figure17
SeparateExitandEntryyDecisionsan
ndCriteria
Thiseffecctivelyresultsinasituation
nwherethereeisnonomaanslandbeetweentheBu
uildandAccept
phasesassshowninFiggure18:


Figure18
AcceptancceProcesswiithSeparateEExit/EntryCritteriaforAcceeptance
WhentheeProductDevvelopmentTeeamandprod ductownerpaartieshaveagoodworkinggrelationship pthey
shouldbeeabletoagreeuponasingglesetofcriteeriathatsatissfiesboththeirneedsandhaveasinglee
decisionp
pointbasedoontheagreeduponsetofccriteriaasillu ustratedinFiggure19SinggleDecisionP
Point
withCombinedExitan ndEntryCriteria.

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 64


Figure19
SingleDeccisionwithCo
ombinedExitandEntryCriiteria
Thetwoppartieswouldagreeonwh hichpartydoeestheassessmmentforeach
hcriterionandwhomakessthe
decisionb
basedontheassessment.Typicallyitisthepartydoingthehando offthatwilld
dotheassessm
ment
andthereeceivingpartyymaybeinteerestedinseeeingtheresultts.
Themainentrycriterio onfortheaccceptancephaseisthattheProductDevelopmentTeamhasdeem med
thesoftwarereadyforracceptancettesting.SeconndarycriteriaamayincludeewhethertheeProductOwneris
sufficientllypreparedtooconducttheeacceptancetesting.Thedecisiontoentertheacceeptancephasee
shouldtakkeallthecriteriaintoacco
ount.
ple,aProductDevelopmentTeamsReeadinessDecissionshouldb
Forexamp bemadebasedona
combinationof:

E
Entrycriteriaf
foracceptanccetestingpro
ovidedbytheProductOwn
nerincluding::

AllfeaturesinMMFFfunctionalw
withnoknownseverity1&
&2bugsopen
n.

Stresstestedat110
0%ofratedcaapacityfor48
8hourswithllessthan1failedtransactiion
000.
per10

Regresssiontestofp
previousreleaasefeatureseexecutedwith
h100%passrate.

E
ExitcriteriaforconstructionprovidedbyytheProducttDevelopmen
ntTeamsuch
has:

odecoverageebyunittestss.
90%co

Dataaarchitecturereeviewedandapproved.

Staticanalysishasidentifiednodeficiencies.

Securittyreviewcom
mpletedwithnohighpriorrityissuesopeen.

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 65

InpracticeitishighlypreferableforthereadinesscriteriatoincludealltheProductOwnersacceptance
criteria.Thisavoidsfindingbugsduringtheacceptancetestingphase.

Sidebar:RepresentingtheAcceptanceProcesswithStagesandGates
Manycompaniesemployaprocessinvolvingstagesandgatestomanagetheirproductdevelopment
pipeline.Theseprocessesweredescribedinsection1.X(ProcesseswithStagesandGates).Thegates
serveasdecisionpointsbetweenstages.Themaingoalistomanagetheallocationofresources
(fundingbeingthemostcommon)amongandwithinprojects.Thegatesattheexitofstagestypically
involvego/nogodecisionswiththegodecisionsleadingtoprogressivelylargeredresource
commitmentsandinthesubsequentstages.
AtypicalsequentialStageGateTMprocessstartswithanewideaevaluatedataninitialgatetodecide
whethertheideaisworthpursuing.AsampleprocessisillustratedinFigureS0.
Attheideastageusuallyafixedamountofresourcesareallocatedtotheproducttodeveloptheidea
furtherandbuildabusinesscaseinthenextstage.Thenextgatedetermineswhethertheidea
warrantsadvancingtofullscaledevelopment.Ifso,resourcesareassignedfordevelopingtheproduct;
ifnot,theprojectiskilled.Thenextgatetypicallyinvolvesdecidingwhetherdevelopmentofthe
productiscomplete.Ifso,theproductentersthetestingandvalidationstage;ifnot,itmaystayin
developmentuntilitrunsoutofresources,inwhichcasetheprojectmayrevertbacktotheprevious
stagetodeterminewhetheradditionalresourceswouldbewarranted.Theexitgateofthetestingand
validationdetermineswhethertheproductisreadyforlaunch,thatistobesoldorused.


FigureS0.AsampleStageGateProcessTM[source:http://www.stagegate.comisfairlytypical].
Manysoftwaredevelopmentprocessescanbecastintermsofstagesandgates.TheRationalUnified
Process[KruchtenOnRUP]describes4phasesInception,Elaboration,Construction,Transition.Each
phasehasanassociatedmilestone.Theemphasiswithinaphaseisplacedonaspecificsetofactivities,
buttheprocessdoesnotprecludeperformingotheractivitiesoutsidethatcore.Thusthephasescan
beviewedasstagesinwhichmanykindsofactivitiestakeplaceandthesameactivitiesmaybe
repeatedoriteratedacrossmanydifferentstages.Themilestonesattheendofeachphasecouldbe
treatedasgates.
Agilesoftwaredevelopmentprocessesmayalsobecastintermsofstagesandgates,butwitha
differentspintocapturetheiriterativeandincrementalnature.Whenaniterativeandincremental
processisunwound,itcanstillbeviewedasastreamofgatesandstages,withfundingincrementally
allocatedatreleaseends.Thiscanespeciallybeveryeffectiveifhighvaluefeaturesaredeployedin

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 66

earlyreleasessothattheprojectstartsgeneratingvalueassoonaspossibleandcanbecomepartially
selfsustaining[DenneHuangOnSoftwareByNumbers].Inthisvariation,ratherthanstageshaving
distinctpurposesandgateshavingdifferenttypesofdecisions,thestages,activitieswithinstages,and
gatesarereplicated.Eachstagegatepairrepresentsaniterationforimplementingthenextsetof
featurestobedeveloped,togetherwithacontinuationorfundingdecision.Thegatesareusedto
approveadditionalfundsforthenextiterationbasedonwhatwaslearnedinpreviousiterations
[HighsmithOnAgileProjectManagement,ErdogmusOnEconomicsOfAgile].(Thisisaninstanceofthe
applicationoftheprinciplesdiscussedintheChapter1SidebarRecastingaSequentialProcesses
throughWorkflowScheduleIndependenceandParallelization.)
WereorganizepartofStage3,Gate4,Stage4,andGate5,intoacustomprocessforsoftware
products:theAcceptanceProcess.WhileitcouldbearguedthatGate4,GoToTesting,assumesthe
ProductDevelopmentTeamhasinfacttestedthesoftwareextensively,inmostorganizationsthe
majorityoftestingoccursinStage4,TestingandValidation.Weproposethatmuchofthetesting
activitybelowthelevelofbusiness,user,orcustomerrequirementsoftheproductbepushedback
intoStage3,Development,andbecomepartofthereadinessassessmentactivitywithinthatstage.
ThiseffectivelymakesthereadinessdecisionpartofGate4,GotoTesting,andleavesStage4focussed
oncustomeroruseracceptancetesting.ThenGate5,GotoLaunch,wouldrepresenttheAcceptance
decision.ThevariationsonwhomakeswhichdecisionandwhenaredescribedinChapter3The
DecisionMakingModel.
HowistheAcceptanceProcessforreleasecandidatesofaproductexpressedintermsofstagesand
gatesandhowdoestheresultrelatetothetypicalStageGateprocessTM?Werelaxthenormal
requirementofhavingagatebetweeneachpairofsubsequentstagesandseparatethepreparation
foradecisionassociatedwithagateintoadistinctstagethatimmediatelypreceedsthegate.Wealso
allowstagestobenestedtohaveinnerstagesandgates.FigureS1TheAcceptanceProcessas
NestedStagesandGatesillustratesthisthroughanexample.ThestagesBuild,Accept,andUsein
FigureS0correspondtothestagesDevelopment,ValidationandTesting,andLaunchinFigure0.it.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 67


FigureS1
TheAcceptanceProcessasNestedStagesandGates
InsidetheBuildstageonthetop,Build1isthefirstreleasecandidatethatdevelopmentaskstheir
readinessassessorstoevaluate.Ithasseveralseverity1bugsthatmustbefixedandismissingsome
keyfeatures.Developmentfixesthesebugsandaddsthemissingfunctionality.Afterseveralmore
tries,BuildNfinallymakesitallthewaythroughReadinessAssessmentandisreleasedtotheproduct
ownerforacceptancetesting
ApositivereadinessdecisioninsidetheBuildstageautomaticallysatisfiesthecriteriaforGate4
thereforetheprojectentersintotheAcceptstage.Theproductownerconductsacceptancetesting
onlytodiscoversomekeyissues.TheseareprovidedtotheProductDevelopmentTeamwho
addressestheminBuildN+1afterseveralintermediatebuilds(notshown)thatfailtheirreadiness
assessment.NotethatatthispointtheprojectisstillconsideredtobeintheAcceptstagedespitethe
factthattheballisinTheProductDevelopmentscourttofixthebugs.Theproductownerfindsno
significantissueswiththisbuildandacceptsthesoftwareasfinished,satisfyingGate5scriteria.
TheprojectnowmovestotheUsestage.Afterafewweeksofusage,auserexperiencesaseverecrash
thatistracedbacktoamissedscenario.TheproductowneraskstheProductDevelopmentTeamfora
fix.DevelopmentcomesupwithafixinBuildN+2,whichpassesreadinessassessmentandacceptance
testing(mostlyregressiontestingplusanewtestcasecoveringthisbug)andBuildEisputinto
production.TheprojectstaysintheUsestage(oftenreferredtoasWarranty)untilthecriteriato
exittheUsestagearemet;thismaybedecidedbythepassageofanagreeduponperiodoftime
duringwhichtheproductiscoveredunderTheProductDevleopmentTeamswarranty.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 68

AttheendoftheBuildstageandduringtheAcceptstage,eachreleasecandidateisassessedfor
whetheritsatisfiesfunctionalandnonfunctionalcriteria.Howevertypically,thetransitionsfromthe
BuildstagetotheAcceptandUsestagesinvolvealsobusinessandpoliticalfactorsthatmayhavelittle
todowithwhetherthesoftwaremeetsthefunctionalandnonfunctionalrequirements(SeeChapter
XSystemRequirementsModel).Examplesofsuchfactorsinclude:
Aneedtofreeuptheproductdevelopmentteamforotherwork
Changingwhopaysformodifications:beforeacceptanceitisusuallytheproductdevelopment
teamwhileafteracceptanceitistypicallytheproductowner.
Changingthefundingsourcefromcapitaltoexpense.
Managementpressuretoshortcutagateduetoamisalignmentofprojectgoalswithincentives.
Whilethesefactorsmayplayacriticalroleingatedecisions,theyareoutsidethescopeofthisguide.

Sidebar:QualityGatesandtheAcceptanceProcess.(MicrosoftQualityGatesexamplesasMMF/MQR
criteriaforreadinessandacceptancedecisions.)
<TBA>

2.5 Summary
Thischapterintroducestheconceptoftheacceptanceprocessasawaytothinkabouttheactivities
relatedtomakingthedecisiontoacceptsoftware.Theacceptancedecisionistheculminationofa
processthatinvolvesseveralorganizationseachwithspecificresponsibilities.Eachreleasecandidateis
passedthroughthedecisionpointsoftheacceptanceprocessonitswaytothemakingthefinal
acceptancedecision.Therearethreemajordecisionpointsbetweensoftwareconstructionbythe
ProductDevelopmentTeamworkingwiththeirProductOwnerandsoftwareusagebytheenduser.
First,theProductDevelopmentTeammustdecidewhetherthesoftwareintensiveproductinitscurrent
form,knownasthereleasecandidate,isreadyforacceptancetesting.Ifitis,theProductOwnermust
decidewhetherthereleasecandidatemeetstheiracceptancecriteriabeforetheproductcanbemade
availabletotheendusers.Ultimately,eachendusermustdecideforthemselveswhethertheywantto
usethesoftware.Thisusagedecisionhasverylittleinfluenceonthecurrentacceptancedecision
althoughitmayinfluencetheacceptancecriteriaforfuturereleasesoftheproduct.
Whileeachuserpotentiallymakestheirownusagedecision,thereadinessandacceptancedecisionsare
normallyeachasingledecision.Eachdecisionpointor"gate"shouldhavewelldefinedcriteriatoguide
thedecisionmaking.ThesecriteriashouldbeknowninadvancebyboththeProductDevelopment
Teamandtheproductowner.Complexorganizationsmayhaveseveralreadinessoracceptance
decisionsbeingmadeinparallel,eachbasedontheirowncriteria(forexample,operationalacceptance);
thereleasecandidatesprogressionthroughtheprocessmaybevetoedbyfailingtosatisfythecriteria

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 69

ofanyoftheparalleldecisions.ComplexproductsmayrequiremultipleProductDevelopmentTeam
teamseachwiththeirownreadiness,andpotentiallyacceptancedecisionspriortoanoverallreadiness
andacceptancedecisionfortheintegratedproduct;thereleasecandidatecanbesentbackforfurther
workfromanyofthedecisionpoints.
Theacceptancedescribedinthischapterappliestobothsequentialandagileprojectsbutinslightly
differentways.Thephasednatureofsequentialprojectsmeansthatalltestingisdonewithinaseparate
testingphaseandtheacceptanceprocessdescribeswhatgoesonwithinthatphase.Agileprojects
traversetheentiredevelopmentlifecycleforeachfeatureoruserstory.Therefore,theacceptance
processisexecutedatseverallevelsofgranularitywiththefinestgrainexecutionbeingattheindividual
featurelevelandthelargestbeingatthewholeproduct(orfeatureintegration)level.

2.6 WhatsNext?
InChapter3DecisionMakingModelweexaminetherolesandresponsibilitiesofthevariousparties
involvedinmakingthedecisionswhichmakeuptheacceptanceprocess.Wealsodescribetheroles
playedbythepartieswhoprovidethedataonwhichthedecisionsarebased.

2.7 References
[BasLarmanOnScalingAgile]Vodde,Bas.andLarman,CraigScalingLean&AgileDevelopment:Thinking
andOrganizationalToolsforLargeScaleScrum,AddisonWesleyProfessional,AgileSoftware
DevelopmentSeriesseries.,2008
[HighsmithOnAgileProjectManagement]Highsmith,JimAgileProjectManagement:CreatingInnovative
ProductsAWP2004ISBN13:9780321219770
[MeszarosOnAgileERP]Meszaros,Gerard,etalAgileERPYouDontKnowWhatYouveGotTillits
Gone,http://agileerppaper.gerardmeszaros.com/
[AstonOnAgileERP]Aston,Janice,etalAgileERPOvercomingtheChallenges,
http://asug2008agileerpchallenges.gerardm.com/
[EcksteinonLargeScaleAgile]JuttaEckstein,AgileSoftwareDevelopmentintheLarge:DivingIntothe
Deep,DorsetHouse,2004
[ErdogmusOnEconomicsOfAgile]HakanErdogmus,TheEconomicImpactofLearningandFlexibilityon
ProcessDecisions,IEEESoftware,Vol22,No6(Nov.2005),pp.7683
[DenneHuangOnSoftwareByNumbers]MarkDenneandJaneClelandHuang,SoftwarebyNumbers:
LowRisk,HighReturnDevelopment,PrenticeHall,2003
[KruchtenOnRUP]PhilippeKruchten,TheRationalUnifiedProcess:AnIntroduction,AddisonWesley,
2003
[MillerKarterOnLargeScaleAgile]AdeMillerandEricCarter,AgileandtheInconceivablyLarge,Proc.
Agile2007,IEEEPress,2007.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 70

Chapter3
3 D
Decisio
onMakinggMod
del
Chapter1
1introducedttheacceptancceprocessanndthethreekkeydecisionsthatneedtobemadeas
oracceptabilitybywhoeveermakestheacceptanced
softwareisassessedfo decision:theReadiness
Decision,theAcceptannceDecisionaandtheUsageeDecision.
Figure1illlustratesthisssimplifiedveersionofthedecisionmakkingmodel.

Figure1
Simplified
ddecisionma
akingmodel
onelaboratessonhowthefirsttwodecisionsaremaadeandwhomakestheminavarietyof
Thissectio
businessmmodels.Thed decisionsarenotmadeinavacuum;th heyrequireinformationthatmustbem
made
availabletthroughactivvities.Figure2 orasingledecision:
2illustratesthisprocessfo

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 71

Figure2
Decisionmmakingmodeelsampleactiivities
Thediamo ondontherigghtsideofFiggure2repressentsthedecisiontobemadebasedon nthetestresuults
(thedecissioncanbeeithertheread dinessdecisioonortheacceeptancedecision).Thetesttresultsareb
based
onthetesstingandasse
essmentactivvities,whichaassessesthessystemunderrtestagainstttheexpectattions.
Theexpecctationsofthesystemund dertestwereedefinedbaseedontheuseersrequirements.Allofth hese
activitiesareexecutedwithintheco ontextofateestplan.
ManyoftthepracticesinVolumeIId describehowwtodotheassessactivity,andotherpracticesinVollume
IIdescribeewaystodeffinetheexpecctationsbasedontheneed ds.Thatison neofthereasonsthisguideehas
anumberrofrequireme entsrelatedp
practicesitisnotaboutttesting,itisaaboutacceptaance,and
acceptancceisbasedon nexpectationnsandrequireements.

3.1 TheSixA
AbstractR
Roles
Thejobtittlesofthede
ecisionmakerrsvarygreatlyyfrombusineessmodeltob
businessmod delandacrosss
businessd
domainsandorganizations,sothisguid deusesabstraactrolenameestodescribeetheroleswithin
thedecisionmakingm model.Thisguidealsoprovidesalistofccommonaliasses.Howeverr,beawareth hat
manyofthenamesare ehighlyoverloadedandth hatyourcusttomer(topickjustoneexxample)maybe
anentirelydifferentro onementioneedasanaliashere.Toseehowtheabsttractrolenam
olethantheo mes
maptojobtitleswithinnorganizationsinspecificbusinessmodels,seethesidebarDeccisionMakingg
ModelSteereotypes.

3.1.1 Readiness
R DecisionM
Maker
Thereadinessdecisionnmakermakeesthefinalreeadinessdecissionbasedon ninputfromoothers.When
na
singleperrsonperformssthisrole,thejobtitlemigghtbesometthinglikeChieefEngineer,P
ProjectManagger,

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 72

DevelopmentManager,orVPofEngineering.Thisrolecouldalsobeplayedbyacommittee,asis
commoninenterprisedeploymentwithmanystakeholders,ormadebyseveralpeopleorcommitteesin
parallelasdescribedinComplexOrganizationsinthepreviouschapter.

3.1.2 ProductDevelopmentTeam
Theproductdevelopmentteambuildsthesoftware.Generally,thisteammayincludeuserexperience
researchersanddesigners,graphicartists,requirements/businessanalysts,softwarearchitects,
softwaredevelopers,middlewareandintegrationspecialists,systemadministrators,DBAs,
release/deploymentspecialists,testersandtechnicalwriters.Inotherwords,thisteamincludesanyone
whoisinvolvedinanywayintheactualconstruction,customization,orintegrationofthesoftware.

3.1.3 ReadinessAssessors
Thereadinessassessors,astheirnamesuggests,assessthereadinessofthesoftwareforacceptance
testing.Theyprovideinformationthatisusedtomakethereadinessdecision.Thejobtitlesinvolved
dependsverymuchonthenatureoftheprojectandtheorganization,butittypicallyincludesrolessuch
asdevelopers,testers,anddocumentationwriters.Ineffect,areadinessassessorcanbeanyonewho
mightbeaskedtoprovideanopiniononwhetherthesoftwareisready.Insomecases,thisopinionis
basedonformaltestingactivities,butitmightalsobebasedontechnicalreviewsorevenqualitative
inputs.

3.1.4 AcceptanceDecisionMaker
Theacceptancedecisionmakeristhepersonorcommitteewhodecideswhethertoacceptthe
software.Inaproductcompany,ajobtitleforthisrolemightbeProductManager,butinaninformation
technology(IT)environment,thisroleistypicallyfilledbyacustomer,productowner,businesslead,or
businesssponsor.

3.1.5 AcceptanceTesters
Acceptancetestersprovidedataonacceptabilityoftheproduct.Theyperformactivitiestoassessto
whatdegreetheproductmeetstheexpectationsofthecustomerorenduser.Acceptancetestersmay
includetwoteamsonefocusingonfunctionalacceptanceofthesystem,whileanotherfocusingon
operationalacceptance.Theyprovideinformationtotheacceptancedecisionmaker.Theymaybe
dedicatedtestingstaff,endusersaskedtodotesting,oranywhereinbetweeninskillset.

3.1.6 Users
Usersmakeindividualusagedecisions.Eachuserdecideswhethertousetheproductasitiswhenitis
shippedordeployed.Theirfeedbackmightbeusedtoadjusttherequirementsforthenextreleaseorto
dousabilitytestingofthebetaversionsofthecurrentrelease.Peoplewhoareusersmayalsobe
involvedasacceptancetestersorbetatesters.Similarvitalfeedbackmightcomefromoperationsstaff,
supportstaff,trainers,fieldengineers,whoareexposedtoissuesofconsumabilityandusability.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 73

3.2 MakingtheThreeDecisions
Thissectiondescribeshowtheprecedingsixabstractrolesareinvolvedinmakingthethreedecisions.

3.2.1 MakingtheReadinessDecision
Thereadinessdecisionismadebythereadinessdecisionmaker(s).Thereadinessdecisionisanexitgate
withadecisionaboutwhethertolettheproductbeseenbeyondtheboundariesofthesupplier
organization.Thedecisionisbasedonreadinessassessment(whichisbasedonthefeaturesincluded
andthequalityofthosefeatures)donebythereadinessassessors.Thedecisioncanbemadebyasingle
person(suchasaChiefEngineer)orbyacommittee(suchasengineers,architects,orotherproject
stakeholders).WhenitisnotasingledecisionasdescribedinChapter1TheAcceptanceProcess,each
decisionmaybemadebyasinglepersonoracommittee.Fromthepointofeachdecisionmakerthe
softwaresystemiseitherreadyoritisnotready.Ifitisnotready,theremaybealistofconcernsthat
needtobeaddressedbeforeitwillbeconsideredready.Formoreinformation,seeHowWillWe
ManageConcernssectioninChapter16PlanningforAcceptance.
Theremayhavebeenanumberofearlierdecisionmakingcheckpointsaspartofthedevelopment
process(suchas"requirementscomplete,""designcomplete,"or"codecomplete").Thesearebeyond
thescopeofthisguidebecausetheyareneitherdirectlypartofthereadinessdecisionnorarethey
easilytested.

3.2.2 MakingtheAcceptanceDecision
Theacceptancedecisionismadebytheperson(orpersons)playingtheAcceptanceDecisionMakerrole.
ThedecisionissummarizedbythequestionShouldweacceptthesoftwareandputitintouse
deliveringvaluetoourorganization?Theremaybeadditionalcontractualconsequencesformakingthe
acceptancedecision,suchasacommitmenttopaythesupplier,thestartofapredefinedwarranty
period,andsoon.Whileintheorytheseshouldnotbetheprimaryconsiderationswhenmakingthe
decision,inpracticetheyoftenare.Thedecisionshouldbewhetherthesoftwareiscompleteor
doneenoughtobedeployedorshipped.Formoreinformationaboutthedefinitionof"done,"see
Chapter7DonenessModel.Formoreinformationaboutthecompletedefinitionofthesystem
attributesthatmaybeconsideredwhenmakingtheacceptancedecision,seeChapter5System
Model.
Thedefinitionof"done"isinfluencedbyseveralfactors,includingthefollowing:
MinimumMarketableFunctionality(MMF)fortheproduct.Whatfeaturesorfunctionsmustthe
productsupporttobeworthreleasing?ThisisbasedonwhatevercriteriatheProductOwner
decidesareimportant,derivedfromproductplans,marketsurveys,competitiveanalysis,or
economicanalysis.WhiletheProductOwnerisaccountableforthedecisionthisdoesnot
removetheneedfortheProductDevelopmentTeamtounderstandtheproblemdomainin
generalandtheneedsofpotentialusers.Whereverpossible,theyshouldassisttheProduct
Ownerindefiningtheproduct.ForadefinitionoftheresponsibilitiesoftheProductOwner,see
Chapter1.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 74

Minimumqualityrequirement(MQR)fortheproduct.Whatisthelevelofqualitythatmustbe
achievedbeforetheproductcanbereleased?Qualityhasseveraldimensions.Thepresenceor
absenceofbugs/defectsinfunctionalityisjustonedimensionofquality.Nonfunctional
requirements,alsoknownasqualityattributesisanotherdimension.TheMQRencompasses
thelatterwhileMMFencompassestheformer.
Harddeadlines.Bywhatdatemustaparticularversionoftheproductbereleasedtohaveany
value.Thesecanincludetradeshowdates,regulatorydeadlines,orcontractualobligations.
Eachdeadlinecouldbesatisfiedbyadifferentversion(orbuild")oftheproduct.Formore
information,see"ProjectContextModel."

Theacceptancedecisionismadebasedondataacquiredfromanumberofsourcesandactivities.
Acceptancetestinggeneratesmuchofthedataneededtomaketheacceptancedecision.Thisdata
includesthefollowing:
Pass/failresultsofallteststhatwereperformedaspartoftheacceptancetesting.Thiscouldverify
bothfunctionalrequirementsandparafunctionalrequirements.
FeaturecompletenessAsimpliedbythepass/failresultsoffunctionaltests

Theacceptancedecisionmayalsousedatagatheredduringreadinessassessment.Themostcommon
exampleofthisisdatarelatedtosystemperformance(capacity,availability,etc.)whichmanycustomer
organizationswouldnotbecapableofgatheringthemselvesbutwhichtheyarecapableof
understandingonceithasbeengathered.
Theacceptancedecisionisallaboutmaximizingvaluederivedfromtheproductbythecustomer
(representedbyProductOwner)andminimizingrisk.Timehasadirectvalueinthattimespent
collectingmoredatathroughtestinghasadirectcost(thecostofresourcesconsumedingatheringthe
data)andanindirectcost(thedeferralofbenefitthatcanonlyberealizedoncethesystemisaccepted).
Riskhascostthatcouldbecalculatedasthesumofthecostofallpossiblenegativeeventsmultipliedby
theprobabilityoftheiroccurrence.Thoughthiskindofliteralcalculationisnotfrequentlydoneour
perceptionsofriskareinherentlybasedonanintuitiveinterpretationofthecircumstancesalongthese
lines.Theintentisfortheacceptancedecisionmakertounderstandthetradeoffsanddecideon
whetherweneedmoredata,orwehavedoneenoughandwecanmakethedecision.Indoingthisitis
importanttobeawareofthetwoextremenegativepossibleoutcomes,oneintime,theotherinrisk.
Examplesofcostsofunmitigatedriskmightincludethefollowing:
costofpatchingsoftwareinthefield;
costofmanualworkaroundsforbugs;
costofmaintainingspecializedresourcesforsoftwaremaintenance;
losingcustomersthatneedspecificfeaturesthataremissing;
opportunitycostofunrealizedordelayedbusinessbenefitsifproductfailsusage
assessmentorendupnotbeingusedafterdeployment.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 75

Thecostoftimecanbenonlinearinthatifadeadlineorneedismissed,allbenefitsmaybelostand
punitiveactiontaken.Theriskthecalculationisdifferentfor'blackswan'[TalebOnImpactOfImprobable]
eventsofextremelylowprobabilitybutextremelyhighimpactsuchthatagainallvaluemaybelost.
Thesecannotbeignoredintheacceptanceofnewsoftware.Inconcept,whenthecostofriskexceeds
thecostofdelay,moretestingshouldbeperformed.Whenthecostofmoretestingexceedstherisk
relatedcostthatwouldbereduced(byreducingprobabilityofoneormoreeventsoccurringorby
reducingtheexpectedcostgiventheeventdoesoccur),youcandecidetoaccepttheproductwithout
furthertesting.

3.2.3 MakingtheUsageDecision
Eachpotentialuserofthesystemhastomakeapersonaldecisionaboutwhethertousethesoftware.
Thisdecisionisdifferentfromtheacceptancedecisioninthatitismademanytimesbydifferentpeople
ororganizations.Infact,theremaybeseveraltiersofthesedecisionsascompaniesdecidewhetherto
adoptaproduct(oranewversionthereof)anddepartmentsorindividualsdecidewhethertocomply
withtheorganizationaldecision.Theimportantconsiderationfromtheperspectiveofthisguideisthat
thesedecisionshappenaftertheacceptancedecisionanddonotdirectlyinfluencetheacceptance
decision.Theymayindirectlyinfluenceitinoneofthefollowingtwoways:
Proactively.Usagedecisionsmayindirectlyinfluencetheacceptancedecisionforanupcoming
releasebyfutureuserscommunicatingtheindividualacceptancecriteriatotheproductowner
inresponsetomarketresearchorsurveys.Thistypeofcriteriamayalsobesubmittedtothe
productownerthroughunsolicitedinputs,suchasfeaturerequestsorbugreports.Also,user
feedback(includinglackofusage)fromalphaandbetareleasesmayinfluencetheacceptance
criteriaforthegeneralrelease.Seesection1.2.3TheAcceptanceProcessforAlpha&Beta
ReleasesinChapter1IntroducingtheAcceptanceProcessforamoredetaileddiscussion.
Retroactively.Usagedecisionsmayindirectlyinfluencetheacceptancedecisionbyproviding
feedbackonthereleasedproductindicatingalackofsatisfactionineitherfunctionalityor
quality.Thismayinfluencetheacceptancedecisioncriteriaofafuturerelease,butitrarely
causestheacceptancedecisionalreadymadetoberevisited.Thenotableexceptionwouldbe
thediscoveryof"severity1"bugsincriticalfunctionalitythatmightresultinarecallofthe
releasesoftware.

Theusagedecisionofteninvolvesdecidingwhetherornotthebusinessoruserisreadytostartusing
thesoftware.Thisissometimescalledbusinessreadinessanditfallsintheareaofchange
management.Oninternalinformationtechnologyprojects,theProductOwneristypicalresponsiblefor
businessreadiness.Thismightincludetrainingofusers,transforminglegacybusinessdataintoformats
compatiblewiththenewsystemorpreparationofnewdatareadytobeloadedwhenthenewsystem
becomesavailable.Whileitisanimportanttopic,itis,strictlyspeaking,notpartofacceptingsoftware
andisbeyondthescopeofthisguide.

Sidebar:ThePerilsofIgnoringtheUsageDecision

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 76

Acompanyoneoftheauthorsworkedforimplementedapopulartoolformanagingleads.The
ProductOwnerTeamconsultedwiththekeystakeholderswhowouldbeconsumingthereportson
potentialfuturebusinessinthesalespipelineandconfiguredthesystemwiththefieldneededto
generatingthereports.Mostofthefieldsweremarkedasmandatory.
Whenthesystemwasreadyforacceptancetesting,theProductOwnersteamtestedthesystemand
foundittobeworkingtotheirsatisfaction.Theyconductedextensiveusertraingingofthesalesforce
toensurethateveryoneknewhowtousetheirwonderfulnewtool.Theyacceptedanddeployedthe
system.Whenthesystemwentlive,theuseruptakewaspoor.Severalyearsafterdeployingthe
system,mostuserswerestillchoosingtonotusethetoolbecauseitwasprovidingthemwithvery
littlevaluewhileimposingconsiderableoverheadontheirdaytodaywork.Mostofthemdeveloped
theirquotesinspreadsheetsorinacustombuiltpricingapplicationthatdirectlyaddressedtheir
needs.Theleadmanagementtooldidntprovidetheequivalentfunctionalitythereforeitadded
additionalstepsreenteringdatabutindifferentformats.Tomakemattersworse,itforcedtheusers
topickthroughalonglistofpossiblecustomerseachtimeauserneededtocreateaquote.Asaresult,
thesystemlanguishedandmuchofthepotentialvaluewasalostopportunitycost.
TheProductOwnerofasubsequentprojecttoreplacethecustompricingtooltookthelessonsto
heartandincludedfunctionalitytoaddresstheshortcomingsbyfeedingthepricequotesintothelead
managementsystemandsynchronizethecustomersautomatically.Thissatisfiedtheneedsofthe
reportgenerationstakeholdersbyprovidingthemwithallthedataneededtocalculatethepresent
valueofleadsinthesalespipelinewhilenotburdeningtheuserswithduplicatedataentry.Asaresult,
usageoftheleadmanagementtoolsoared.Ahappythoughbelatedending.

3.3 Rolesvs.Organizations
Therolesdescribedinthisdecisionmakingmodelmaybeplayedbypeopleinseveraldifferent
organizations.Theprimaryvalueofdiscussingorganizationhereisinmakingiteasiertomap
terminologyfromvariousorganizationmodelstobetterunderstandwhoplayswhichdecisionmaking
role.Iftheorganizationalmodeldoesnothelpinthisendeavour,itcanbeignored.
Whenthesoftwareisbeingbuiltbyadifferentorganizationthantheonewhocommissionedits
construction,theorganizationthatcommissionedthesoftwareisoftenreferredtoasthecustomer,and
theorganizationthatisbuildingthesoftwareisthesupplier.Thisistruewhethertheorganizationsin
questionareseparate,unrelatedcompaniesorsimplydepartmentswithinasinglecompany.For
example,theITdepartmentistypicallyasupplierofsystemstothecorebusinessdepartments(suchas
TransportationorManufacturing)andsupportingdepartments(suchasHumanResourcesorFinance.)
Whenacceptancetestingisoutsourcedtoathirdpartytestorganization,itisoftenreferredtoasthe
(thirdparty)testlab.Thetestlabisasupplierofservicesasdistinguishedfromthesupplierofthe
software.
Anorganizationthatbuysanddeploysshrinkwrappedsoftwarecanalsobereferredtoasacustomer,
andtheorganizationtheybuyitfrommaybereferredtoasthevendororsupplier.Thefactthatthe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 77

vendorcontractstheworktoanoutsourcer(anothervendorofwhichtheyarethecustomer)illustrates
theproblemwithusingtheterm"customer"todescribetheacceptancedecisionmaker(theproduct
owner)asadvocatedinextremeprogrammingwhichcustomerisitreferringto?
FigureXillustratesthisproblem.ThePurchaser(intheroleofcustomer)buysshrinkwrapfromS/W
Vendor(inroleofsupplier).S/WVendor(intheroleofcustomer)outsourcesdevelopmenttoS/W
Developer(thesupplier).S/WDeveloper(ascustomer)outsourcesreadinessassessmenttoTestLab(as
supplier).Sowehavethreeseparatecustomersupplierrelationshipsinthisscenariomakingithardto
tellwhichpartywearetalkingaboutwhenwerefertothecustomer.


FigureX
MultipleCustomersandSuppliers

3.3.1 WhoPlaysWhichRoles?
Thusfarthediscussionshavecenteredontheabstractrolesinvolvedinthedecisionmakingprocess.But
whoactuallyplaystheroles.

WhoPlaystheReadinessDecisionMakingRole?
TheReadinessDecisionMakerroleistypicallyperformedbysomeoneintheproductdevelopment
organization.Theyarethepersonwhoultimatelydecideswhetherthesoftwareisreadytobeshownto
theacceptancetesters.TypicaljobtitlesincludeDevelopmentLead,ProjectManager,Directorof
Development,ChiefEngineer,etc..Onagileprojectswherethisdecisionismadeseparatelyforeach

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 78

feature,thefeaturelevelReadinessDecisionMakerisoftenthedeveloperwhosignsupforthe
responsibilitytoimplementthestory/feature.Intheterminologyusedinthisguide,theReadiness
DecisionMakerisamemberoftheProductDevelopmentTeam.

WhoPlaystheAcceptanceDecisionMakingRole?
TheAcceptanceDecisionMakerroleistypicallyperformedbysomeoneinthebusinesssideofthe
organizationwhendevelopmentisdoneinternallyorbysomeoneinthecustomerorganizationwhen
developmentisoutsourced.Theyrethepersonwhoultimatelydecideswhetherthesoftwareisreadyto
beshowntotheacceptancetesters.TypicaljobtitlesforinternallydevelopedsoftwareincludeBusiness
Lead,ProductManager,orsomekindofbusinesstitle.Onagileprojectswherethisdecisionismade
separatelyforeachfeature,thefeaturelevelAcceptanceDecisionMakerisoftenthespecificbusiness
subjectmatterexpertwhoprovidedthedetaileddescriptionofthefunctionalitytobebuilt.Inthe
terminologyusedinthisguide,theProductOwneristheAcceptanceDecisionMaker.

WhoDoesWhatTesting?
ReadinessassessmentistheresponsibilityoftheProductDevelopmentTeamButdoesthismeanthat
developersaretheonlyonesdoingreadinessassessment?Bythedefinitionusedinthisguide,the
ProductDevelopmentTeamoftenincludespeoplewhowoulddescribethemselvesastesters.
Developerswoulddoataminimum,unittesting.Eithertestersand/ordeveloperswoulddowhole
systemtesting(functionalandnonfunctional).Otherpartieswhoarealsoconsideredpartofthe
ProductDevelopmentTeammaydootherformsofreadinessassessment.Forexample,dataarchitects
mightdodataschemadesignreviews.Securityarchitectsmightdosecuritycodereviews.
Ideally,acceptancetestingshouldbedonebyrealusersofthesoftwaresincetheyaretheoneswhoare
bestsuitedtodecidingwhetheritisacceptable.Butwhatiftherealusersareanonymous?Whocan
andshoulddoacceptancetestingontheirbehalf?Thisisacasewheretestersoftenactastheproxyfor
theenduser.InthissituationatleastsomeofthetesterswouldbepartoftheProductOwnerTeam
ratherthantheProductDevelopmentTeam.
InFigureX2WhoDoesWhatTestingbyTypeofOrganization,eachcelldescribeswhodoeswhich
testinginacommonprojectcontext.Thetoplefttriangleinthecellindicateswhoisdoingreadiness
assessmentandthebottomrighttriangleindicateswhoisinvolvedinacceptancetesting.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 79


FigureX2
WhoDoessWhatTestin
ngbyTypeof
f Organization
n
Whensofftwareisbeingbuiltbyanorganizationforitsownuse,theusersshouldbeavvailablefor
participattingintheaccceptancetestting.Ifthereisnotestingffunction,thebusinesstypicallyexpects
developmmenttotestth hesoftwarethoroughlybeeforehandinggitover.Iftheereisatestin
ngfunction,th
hey
nd
wouldtyppicallyparticippateina2 pphaseofreaddinessassessm ment,oftencalledsystem
mtestingand d
integrationtestingbeforehandingthesoftwarreovertotheeusersforuseracceptancetesting(U UAT)
orbusineessacceptanccetesting(B BAT).
Whensofftwareisbeingbuiltforsalletoanonymouscustomers,realuserstypicallydon nttakepartinn
acceptanccetesting(exxceptperhapssasusersofaalphaorbetareleases.)Insstead,thetesstingorganizaation
doesacceeptancetestinngontheirbeehalfactingassaproxy(orsurrogate)usser.Ifthereissnospecializeed
testingfunction,theaccceptancetesstingcouldbeedonebytheeproductspeecifiers(membersofthe
ProductOOwnerTeam)ortheaccepttancetestinggcouldbeouttsourcedtoa3rdpartytesttlab.
Intheterm
minologyuse
edinthisguid
de,testersdoingReadinesssAssessmenttarememberrsoftheProd
duct
Developm mentTeamwh hiletestersdo
oingacceptanncetestingarrememberso
oftheProducctOwnerTeamm.

3.4 Summary
y
TheReadiinessDecision nismadebysomeoneorssomecommittteefromtheesupplierplayyingtheroleof
ReadinesssDecisionMaakerbasedon ninformation ngatheredbyyReadinessAsssessors.ReaadinessAssesssors
typicallyincludedeveloopers,architeectsandsometimestesterrs.Thegoalo
oftheReadinessDecision
Makeristtoensurethaattheproductthasenoughfunctionalityyandisofgooodenoughqu ualitytoexpo
oseto
theProdu uctOwner.ThheRDMistyp picallyasenio
orpersonintheproductdevelopmenttteams
organization.

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 80

TheAcceptanceDecisionismadebytheProductOwner(whichmaybeasinglepersonoracommittee
ofseveralkeystakeholders)playingtheroleofAcceptanceDecisionMakerbasedoninformation
gatheredbypeopleplayingtheroleofAcceptanceTesteraswellasanyReadinessAssessment
informationsharedbythesupplier.TheADMisusuallyakeypersoninthecustomerorganization.The
acceptancetestersmaybeendusers,representativesoftheenduserssuchasthespecifiersofthe
requirements,ortesters)whenendusersarenotavailabletodoacceptancetesting.Whenthe
customerorganizationshascommissionedtheproductionofaproductandtheyhavenointernaltest
capability,theymaychoosetooutsourcesomeoftheacceptancetestingtoathirdpartytestlab.

3.5 References
[TalebOnImpactOfImprobable] Taleb, Nassim Nicholas.TheBlackSwan:TheImpactoftheHighly
Improbable.RandomHouse.2007

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 81

Chapter4 ProjectContextModel
TheProjectContextModelisawaytobetterunderstandthegoalsoftheprojectandtheconstraints
underwhichitmustoperate.Itisnotaformalmodel;instead,itisasetofinformationtobecollected
andfactoredintootheractivities.Itincludesinformationsuchasthefollowing:
UsageContext.Whatistheoverallcontextinwhichthesoftwarewillexist?
Businessgoals.Anexamplequestiontogatherinformationfortheseis,"Whatisthebusiness
valuetobeprovidedbythesystemandhowisittobeachieved(strategy)?"
Scope.Anexamplequestiontogatherinformationforthisis"Whatfunctionalityisinscopeand
whatisoutofscopeforthisproject?"
Stakeholdersandusers.Anexamplequestiontogatherinformationfortheseis"Whoarethe
intendedusersandwhoaretheotherprojectandsystemstakeholders?"
Budget.Anexamplequestiontogatherinformationforthisis"Howmuchmoneyisavailableto
achievethebusinessgoal?"
Harddeadlines.Anexamplequestiontogatherinformationfortheseis"Whatdeadlinesmust
bemetfortheprojecttobeconsideredasuccess?"Examplesofharddeadlinesinclude
tradeshows,contractualdeadlines,andregulatorydeadlines.
Constraints.Anexamplequestiontogatherinformationfortheseis"Whatresources(suchas
people,space,andequipment)areavailabletotheproject?"Afollowupquestionmight
include"Whichresourcesarenegotiableandwhicharehardconstraints?"

Thesectionsofthischapterprovidemoreinformationabouteachofthesefactors.Othercontextual
factorsnotdiscussedinthischapteralsohaveaninfluenceonsoftwareacceptance.Thesefactors
include:
Criticalityofthesystem,whichaffectsdefecttolerancelevels,howcomprehensiveacceptance
testingshouldbe,andwhatconstitutesgoodenoughquality(Cockburn[Cockburn],for
example,definesfourzonesofcriticalityofthesystembasedonthefailureimpact:Lossof
Life(clearlythehighest),LossofEssentialMoney,LossofDiscretionaryMoney,orLossof
Comfort.Boehm&TurnerscriticalityscaleissimilarbutdistinguishesLossofManyLives
fromLossofSingleLife[BoehmTurner]);
ProcessesandToolsavailableandusedintheproject,suchascontinuousintegration,
automatedbuilds,andsourcecodemanagementtools,whichaffectthetiming,style,
amount,andfrequencyofacceptancetesting;and
Trust,whichaffectshowmuchandwhatkindofretestingtoperformbasedontestingdataand
resultsfrompreviousstagesorfromotherteams(e.g.,howmuchdoestheproductowner

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 82

performingacceptancetestingtrusttheproductdevelopmentteamperformingreadiness
assessment).

4.1 UsageContext
Softwarecanbebuiltformanydifferentkindsofusageandtheorganizationthatbuildsorcommissions
itaffectshowthesoftwareisbuiltandaccepted.Figure1TheProductContextillustratesonewayof
classifyingthesoftwareintensivesystemswebuild.


Figure1
TheProductContext(courtesyofJeffPatton;usedwithpermission)
Someorganizationsbuildsoftwarefortheirowninternalusagetoreducecostswhileothersbuild
productstoselltootherstogeneraterevenue.Theleftsideofthequadrantdiagramdescribeswhatis
commonlycalledenterprisesoftware;softwarethatisbuiltbytheenterpriseforitsownuse.Theright
sideofthediagramrepresentssoftwarethatisbuiltforsaletootherpartieseitherassoftwareinits
ownright(whichmaybeinstalledonsiteorsoftwareasaservice)orassoftwareembeddedina
hardwareproduct.Someoftheproductsbuiltontherightsideofthediagramshowupastools,
components,frameworksorproductsusedintheconstructionoftheenterpriseproductsontheleft
sideofthediagram.Independentlyofthis,theusersofthesoftwaremaybecaptiveuserswhoare

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 83

compelled dtousethep
productonceitisavailableeortheymayychoosetoussetheproducctoracompeeting
product.TTheprocessoofacceptingssoftwarecanvvarybasedon nthiscontextt.

4.1.1 Enterprise
E Systems
Enterpriseesystemsare ecommissionedbyanenteerpriseforitssowninternaaluse.Thesysstemsautomaate
allorparttofvariousbu
usinessproceesses.Theuseecasesofeacchsystemimp plementoneormorestep psofa
businessp process,eitheerinwholeorrinpartasillustratedbytheleftsideofFigure2.Th
heymaybebu uilt
fromscratchbywritinggcode,comp posedofpurchasedcompo onentsorcreaatedbyconfigguringapackkaged
productp purchasedfromavendor(ontherightssideofFigure2.)Theworkkmaybedoneebyapartoffthe
enterprisee,oftencalleddtheInformaationTechnollogy(IT)orInformationSyystems(IS)deepartment,orrthe
developm mentmaybeo outsourcedtooathirdpartyy.Manyofth hesesystemstendtobeintegratedwith h
othersysttemswithintheenterprisee(whichoften nincludelegaacysystems)andalargepartofthe
challengeofbuildingandacceptinggthesesystem msisensuringgthattheinteegrationworkkscorrectlyas
dbytherightsideofFigure2.
illustrated


Figure2
BusinessPProcessvs.SyystemsIntegra
ationView
Theintegrationrequireementsmayttakemanyformsincludinggdatareplicaation/synchroonizationand
transactio
onalinteractio
onstonamejjustafew,bu
utallshouldb
bedrivenfrom
mthebusinesssrequiremen
nts.
Acceptancceofenterprisesystemsinnvolvesdeterrminingwhettherornotthesystemwillmeetits
businesseenablemento orcostreductiontargets.
Figure2aalsopointsou
utthecauseo
ofmanypotentialissues:aacceptanceissdonefromabusinesspro
ocess
perspectivvewhileman nyProductDeevelomentTeamsarefocusedexclusiveelyonasubseet(oftenjustone)
ofthesystemsthatimplementtheprocess.Thisscanleadtom mismatchesb betweenbusinessexpectattions

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 84

andwhattheITdepartmentdelivers.ThisisespeciallytruewhenITisorganizedaroundthesystems(as
componentteams)ratherthanbusinessprocesses(asfeatureteams)(seediscussiononcomponent
teamsvsfeatureteamsintheAcceptingComplexProductsectionofChapter2)orwhendevelopmentof
thesystemisoutsourced.Thelatterisparticularlyfraughtwithperilastheoutsourcermaynot
understandanythingaboutthebusiness.
Becausethesoftwareiscommissionedbytheenterpriseitself,thereisthepotentialforalargedegree
ofbusinessinvolvementinthedevelopmentprocess.Thiscanresultinsoftwarethatisaverygoodfit
forthebusiness.Unfortunately,manybusinessesdelegatethehardworkofdefiningtheirrequirements
totheproductdevelopmentteam,whetherinternaloroutsourced,andeffectivelyabdicatecontrolover
whatisbuilt.Thisoftenresultsinbarelyusablesystemsthathavelong,painfulacceptancephases.A
goodwaytoavoidthisistohavestrongbusinessownershipandstewardshipofthedevelopmentof
thesesystemsintheformofaninformedproductownerwhomakesthecriticaldecisionsaboutwhat
thesystemsshouldsupportandwhatitshouldnot.Thispositiongoesbyvariousformaltitlessuchas
businesssponsor,businessleadorproductowner.Ifacriticaldecisionrequirestechnicalknowledgeand
theproductownerisnottechnicallysavvy,theproductownerneedsassistancefromtheproduct
developmentteamoroutsideadvicetomakethedecision.
ThesoftwareinthebottomleftquadrantofFigure1istypicallyembeddedinabusinessworkflow
implementingoneormanystepsofthebusinessprocess.Theusersdonothaveachoiceofwhetheror
nottousethesoftwarebecauseusingitispartoftheirjob.Usersarehighlyaccessiblethereforecanbe
usedforusabilitytestingandacceptancetestingbutbecauseusageisnonoptionalthereisan
unfortunatetendencyoftheproductownertoignoretheusabilityattributesofthesystem.Theusers
areofteninvolvedintheacceptancetestingprocessinanactivityoftencalleduseracceptancetesting
(UAT).
ThesoftwareinthetopleftquadrantofFigure1isdifferentinthatusersarenotcompelledtousethe
producteventhoughitwasbuiltforinternaluse.Theproductownertriestorepresenttheneedsofthe
realuserwhomayormaynotbeaccessible.Forexample,thesoftwaremaybeusedprimarilybyin
houseuserswhoareaccessible,oritmaybedesignedfortheuseofpeopleoutsidethecompany
includingcustomersorbusinesspartners.Thesystemmaybestandalonebutismorelikelytobe
integratedbehindthesceneswithothersoftwarewithintheenterprise.

4.1.2 SoftwareIntensiveProducts
Manyproductsinthemodernworld,fromtoystoautomobiles,haveasoftwarecomponentandthe
amountofsuchembeddedcomponentsissteadilyincreasing,especiallywithregardstotheportionof
thevalueaddedtothefinalproduct.Thissoftwareiswhatprovidesthecomplexbehaviorsthat
characterizeproductsinthe21stcentury.Theseproductshavebeenalargepartofthegrowthof
moderneconomies.Acceptanceoftheseproductsinvolvesdecidingwhethertheproductasbuiltwill
meetitsrevenuetargets,contributeordetractfromthebrandimage,andpossiblyotherfactors.The
productrequirementsaretypicallydrivenbyaproductmanager.
ThesoftwareinthebottomrightquadrantofFigure1isbuiltbyonecompanyforsaletoanother
company.Thissoftwaremaybepurchasedandusedasis.Oritmaybedesignedtobeconfiguredor

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 85

customizedbyorforthecustomertofitwithintheirenterpriseenvironment(ontheleftsideofthe
diagram.)Thenotionofacceptancethenhasaninterestingtwistinthatthegenericproductmustfirst
beacceptedbytheproductmanagerbeforebeingmadeavailabletothecustomerswhomaythenhave
itconfiguredorcustomizedbeforeacceptingitfordeploymentwithintheirorganizations.Thisis
describedinmoredetailinsectionAcceptingCustomizableProductsinChapter1TheAcceptance
Process.
Thesoftwareinthetoprightquadrantmaybepurelysoftwareforusebyconsumers(socalledshrink
wrappedsoftware)oritmaybeembeddedinahardwareproduct.Ineithercasetheenduseristoo
numerous(andisoftennotavailable)toinvolveintheacceptanceprocess.Notethattheremaybe
additionalspecializedtechniquesusedforbuildingandassessingembeddedproductsandthese
specializedtechniquesareoutofscopeforthisguidebutmanyofthetechniquesdescribedheredo
applytoembeddedproducts.

4.2 BusinessGoals
Businessgoalsaffecthowthesoftwarewillbeaccepted.Toomanyprojectsarerunwithamajorityof
thestaffnotunderstanding(orevencaringabout)thebusinessgoalsoftheproject.Sometimesthisis
theresultofdeliberatedecisionsbymanagementtowithholdinformation,andsometimesitisjustan
oversight.Eitherway,weshouldexpecttogetsuboptimalresultsifeachteammemberisfocusedon
optimizinghisorherownjobassignmentinsteadofensuringthatbusinessgoalsareachieved.
Asabareminimum,weshouldensurethateveryoneontheprojecthasaclearunderstandingofwhat
theprojectisexpectedtodeliverandhowthatwillprovidevaluetothebusiness.Exampleofdifferent
waystoaddvaluetothebusinessincludereducedcosts,increasedsales,moresatisfiedusers,and
improvedmarketperception/branding.

4.3 Scope
Scopedetermineswhatneedstobeincludedbeforetheproductwillbeaccepted.Thescopeofthe
projectshouldrealizethebusinessgoals.Atthebroadestlevels,thescopecanbedefinedintermson
thetypesofusersweexpecttosupportandthetypesoffunctionalitythattheproductwillsupplyto
them.Itisasimportanttoexplicitlyindicatewhatisoutsidethescopeasitiswhatisinside;otherwise
weriskwastingtimeandenergydiscussing,andpossiblybuildingandtestingfunctionality,thatis
supposedtobeexcluded.Whentherequirementschangeorevolveafterdevelopmentstartsandthe
scopeneedstoberevisited,thedecisiontochangealreadyscopedfunctionalityorexpandittoinclude
anypreviouslyexcludedfunctionalityshouldbeintentional,rational,andexplicit.Animportantquestion
theProductDevelopmentTeamshouldasktheProductOwnerinthatcaseiswhatexisting
stories/featuresintheproductbacklogshouldbereplacedwithanewstory.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 86

4.4 StakeholdersandUsers
Stakeholdersandusersaffectwhoplayswhatroleintheacceptanceprocess.Acceptingsoftware
requirestheinvolvementofvariousstakeholders.Somearedirectlyinvolvedintheacceptancetesting
anddecisionmakingprocess,whileothersneedtohavetheirinterestsprotectedeventhoughtheyare
notinvolved.

4.4.1 Users
Themostobvioususagegoalholdersarethepeople(orsystems)whoaretousethecorefunctionality
ofthesystem.Typically,thesepeoplearereferredtoastheusers.TheyaretheoneswhouseaWeb
sitefordoingtheirbanking,entertainment,oronlineshopping;useanapplicationforexecutingoneor
morestepsofabusinessprocess;oroperateacombinedsoftware/hardwareproductsuchasamedical
imagingsystem.Butthesearenottheonlypeopleinteractingwiththesystem.Otherusersincludethe
peoplewhoadministerthesystembypopulatingthecatalogoftheonlinestoreorsetupthecontentin
anonlineentertainmentsystem;thepeoplewhorundiagnosticsonandmaintainthesystem.Thereare
alsopeople(orsystems)whoinstallthesystem,startit,monitoritsstatus,andshutitdownwhenthe
serversneedmaintenance.Theyallhaverequirementswithrespecttohowtheyusethesystem.For
moreinformationaboutusersandtheirgoals,seetheUserModelingactivityinVolume2ofthisseries.

4.4.2 SystemStakeholders
Therearealsostakeholders[ACUC]whowillnotdirectlyinteractwiththesystemwhenitisinoperation
butwhoexpectthesystemtolookoutfortheirinterestsasitisused(orabused)byothers.For
example,asystemthatcontainspersonalinformationaboutsomeonehasthatpersonasastakeholder
eveniftheythemselvesdonotdirectlyinteractwiththesystem.Thisisanexampleofanonfunctional
requirement.

4.4.3 ProjectStakeholders
ProductsandITsystemsareusuallydevelopedwithinaproject.Theremaybemanydefinedroleswithin
theproject,someofwhichmaybeplayedbyusersorsystemstakeholdersandsomebyuniqueparties.
Theseprojectstakeholdersmaybeinvolvedintheacceptancetestingoracceptancedecisionmaking
processwithoutbeingadirectsystemstakeholderoruser.Forexample,theproductmanageror
productownermayneverusethesystem,norbeasystemstakeholder,buttheydefinitelyhaveastake
intheacceptancedecisionprocess.ThebusinesssponsorofanITprojectmayneverusetheproductin
thefieldorevenviewareportitgenerates,butthepersoninthisrolehasaclearstakeintheprojects
outcomeintheformoftheexpectedbusinessbenefitsinreturnfortheinvestmentoftimeandmoney.

4.4.4 CommunicationBetweenStakeholders
Thelikelihoodofacceptanceofasoftwareintensivesystemisdirectlyproportionaltotheeffectiveness
ofthecommunicationbetweenthosewhocommissionedthesoftware(theproductownerorthe
productownerteam)andtheteambuildingthesoftware(theproductdevelopmentteam).The

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 87

requirementsandthetestsarepartofthiscommunication.Thecommunicationismademuchmore
effectiveifthereisacommonlanguagethateveryoneagreestouseconsistently.
Thecommonlanguagethateverybodyintheproductdevelopmentteamandtheproductownerteam
sharesisoftenreferredtoasaUbiquitousLanguage[EvansOnDomainDrivenDesign].Themembersof
theproductownerteamarethedomainexpertsandarticulatetherequirementsinthelanguageofthe
domain.Howeverthislanguageshouldbeaccessibletoandunderstoodbytheproductdevelopment
team.Conversely,thetechnicaljargonthatthedevelopmentteamusestoexpressgeneraldevelopment
conceptsrelatingtothecustomerfacingaspectsoftheproject(inthedesign,architecture,testing,and
acceptanceoftheproduct)shouldnotneedtobeaccessibletoandunderstoodbytheproductowner
team.TheUbiquitousLanguageofthewholeteamshouldbedevelopedthroughcloseinteractionacross
andwithintheproductowneranddevelopmentteamsearlyintheprojecttobeeffectiveinreducing
thecommunicationoverheadandavoidingcostly,latemisunderstandings.

4.5 Budget
Mostprojectshaveafixedbudget.Thistendstoconstrainthenumberofpeoplewhowillworkonthe
projectbothinnumberandinrateofpay.Oncetheteamsizeandcompositionisdefined,theteamwill
exhibitawelldefinedburnrateandanyextensionoftheprojecttimelinesduetolatedeliveryofthe
softwarewilltranslateintobudgetoverrunsunlessasignificantbufferisheldbackwhenplanningthe
project.
Anincrementalfundingapproach[DenneClelandHuang]isaviablealternativetoafixedbudget,and
providesflexibilityincostmanagement,scheduling,resourceallocation,teamcomposition,andscoping.
Incrementalfundingreliesonprioritizingandschedulinggroupsofusable,highvaluefeaturesearlyin
theprocessandcommittingresourcesforsuchchunksoffunctionalityinapiecemealmanner.
Incrementalfundingisparticularlysuitableforprojectsthatexhibithighcost,schedule,businessand
technical,uncertainty.

4.6 HardDeadlines
Allprojectsfacedeadlinesofonesortoranother.Someprojectsroutinelymissmanyofthesedeadlines.
Whenthebusinessconsequenceofmissingadeadlineissignificant,weshouldtaketheappropriate
measurestoensurethatthedeadlinesarenotmissed,suchasaplanningcontingencybufferorcutting
scope.Therefore,itisessentialtounderstandwhichdeadlinesarearbitrary(suchasin"Itwouldbe
reallygreatifwecouldhavethatfunctionalitybySeptember")orcritical(suchasin"Weneedto
demonstratethisfunctionalityatthetradeshowinSeptemberandfailuretodosocouldsignificantly
affectourfourthquartersalesandourshareprice.")Intheformercase,missingthedeadlinemayresult
inslightorevennoinconvenience.Inthelattercase,thevalueofthedeliverycoulddiminishtozero
makingtheproductandtheprojectthatdeliversitcompletelyirrelevant.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 88

4.6.1 RecoveryfromDevelopmentScheduleSlippage
Amajorissuewithharddeliverydeadlinesrelatestotheperceivedcompressibilityoftheacceptance
testingphase(andtestingingeneral.)Thisisparticularlyanissuewhenthedeliverydateisfixedand
immovable.Whenthedevelopmentphaseisrunninglate,asitoftendoesonwaterfallproject,thereis
greatpressureontheentireteamtorecoverthescheduleslippagesomehow.Therearetwotechniques
commonlyemployed,eachwiththeirowndifficulties.

CompressingtheTestCycle
Readinessassessmentandacceptancetestingwilltypicallystartlatebutisoftenexpectedtomakeup
thescheduleslippagebyreducingthelengthofthetestcycle.Thisresultsinlesstestingthanplanned
andthereforelikelymorebugsslipthroughtoproduction.Inotherwords,theacceptancedecisionis
madebasedonincompletedataaboutthequalityoftheproduct.

OverlappingTestCyclewithDevelopment
Theotherschedulerecoverytechniquethatisoftenusedwithharddeadlinesistoaskthetestersto
starttestingtheproductbeforeitiscompletelyfinished.Whilethispractice,IncrementalAcceptance
Testing,isconsiderednormalonanagileproject,aprojectplannedasastrictlysequentialproject
cannotadoptthispracticeeasilymidproject.Forexample,manyfeaturesmaybe80%doneratherthan
80%ofthefeaturesbeingcompletelydoneandtestable.Thereforeitishardforthetesterstoknow
whattotestbecausethereisntaclearlistofwhatshouldbeworkingnow.Anotherissueisthat
IncrementalAcceptanceTestingusuallyincludesextensiveautomatedregressiontestingsothatthe
testerscanfocusonthenewlycompletedfunctionalityratherthanhavingtomanuallyretesteverything.
Theautomatedregressiontestingisalsoakeyformofbugrepellentusedbythedeveloperstoavoid
introducingregressionbugsbydetectingthesebugsaspartofreadinessassessmentsotheycanbefixed
beforethenewreleasecandidateispassedontothetesters.

4.7 Constraints
Mostprojectsoperateundersomekindofconstraints.Constraintscanincludepeopleandskillsor
facilitiesandequipment.Sometimestheseconstraintscanbeloosened,whiletheyarestrictlylimitedat
othertimes.Acompanymaynotbeabletohireadditionalstafforrecruitpeoplewithspecificskills.For
example,theskilllevelofthestaffperformingtestingmayhaveasignificantinfluenceonthetesting
approachesused,theamountoftestingdone,andthetimeittakestotest.Inthesecasesweneedto
createthebestpossibleplanthatallowsustomeetourgoalswithouthiringadditionalstaff.Thismay
resultinverydifferentplansthanifwewereabletohireadditionalstaff.

4.8 Resources

[ACUC]AlistairCockburnWritingEffectiveUsesCasesAddisonWesley

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 89

[BoehmTurner]BarryBoehmandRichardTurner.BalancingAgilityandDisciplineAddisonWesley

[Cockburn]AlistariCockburnMethodologyperproject,Figure9,
http://alistair.cockburn.us/Methodology+per+project,Oct19,2009

[DenneClelandHuang][DenneClelandHuang0]MarkDenneandJaneClelandHuang,"TheIncremental
FundingMethod:DataDrivenSoftwareDevelopment,"IEEESoftware,vol.21,no.3,pp.3947,
May/June2004

[EvansOnDomainDrivenDesign]EricEvans,DomainDrivenDesign,AddisonWesley,2003.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 90

Chapter5
5 System
mReq
quirem
mentssMod
del
Theproceessofacceptinngasoftwareeintensivesyystemisinherrentlydepend dentonthesyystemtestedaand
thetestca
asesused.Yo
ourtestcasesaredesigned dtotestforsp
pecificrequireements.Thereeforebefore
designinggyourtestcassesitisimperrativeyoukno
owtherequirrementsthattheyareinten ndedtotest.

5.1 Requirem
mentsand
dAccepta
ance
Theaccepptanceofaso
oftwareintennsivesystemisclearlyrelattedtowhetheeritmeetsth
herequirements.
Assessingwhetheritmmeetstherequirementsinvvolvestheprocessoftestiingthesoftwareusingtestt
casesthattareinsomewayrelatedtotherequirements.Therrefore,requirrementsareaanimportant
componentoftheacce eptanceprocessevenifth
heyarenotdirectlyatestin
ngrelatedarttifact.
Thetermrequiremen ntsissomewwhatcontentio ous.Somepeeoplebelievethatyoumerrelyneedtotalkto
potentialusersandaskkthemfortheirrequiremeents.Frequen ntly,thisisrefferredtoasrrequirements
elicitation
norrequirem
mentsgatherinng.Thisisillu
ustratedinFiggure1.


Figure1
Gathering gRequiremen
ntsdirectlyfro
omusers
1weseetherequirementsfromtheussersbeingtranslateddirecctlyintoprodu
InFigure1 uctbythepro
oduct
developmmentteam.Ineffect,thereequirementsbecometheccoarsegraineedworkitemssfortheprodduct
developmmentteam.Initerativeand dincrementalprocesses,th quirementata
heproductisbuiltonereq
time,indicatedherebyythelineswithintheprod
duct.Therem
maybeanopp userstoprovide
portunityforu
feedbackononeversioonoftheproductbeforeaadditionalreq
quirementsarreimplementted.
Somepeo oplebelievethatrequirem mentscannotbegatheredllikestrawberriesormushrrooms;insteaad,
theybelieeverequireme entsmustbebasedontheedefinitionofaproductth hatisdesigneedtomeetthe
potentialusers'needs..Thisguidesu ummarizesthhisprocessassproductdesignwhichacttsasaplaceholder
forawideerangeofacttivitiesthatm
mayinvolvesp
pecializedpro
oductdesignsskills.Thisprocessisillusttrated

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 91

inFigure2.Foraserioustreatmentofthetopicsofproductdesignandexperiencedesignsee[HSPK]as
wellaseffectivelowfidelityproductexperiencedesigntechniquesby[BuxtonOnDesignForUsability].

Figure2
GettingRequirementsviaProductDesign
InFigure2weseetheuserrequirementsbeingaugmentedwithobservationsfromresearchers.These
observationsandrequirementsareinputintotheproductdesignprocessexecutedbytheproduct
designteamwhichmaybethedevelopmentteamoraseparategroupofspecialists.Theresulting
productdesign(oftenexternalizedintheformofprototypes)isthentestedwithpotentialuserstofind
outhowwellitsatisfiestheusersneeds.Thetestresultsarethenincorporatedintotheproductdesign.
Severalrefinementcyclesmaytakeplace.Theactualconstructionoftheproductisbasedontheresults
ofthisfeedback.Initerativeandincrementalprocesses,theproductisbuiltonerequirementatatime,
indicatedherebythelineswithintheproduct.Theremaybeanopportunityforuserstoprovide
feedbackononeversionoftheproductbeforeadditionalrequirementsareimplemented.
Theproductdesignapproachinvolvesexplicitactivitiestovalidatethattherequirementstrulysatisfy
theneedsoftheusers.Thisvalidationcanbedoneaspartofanacceptancetestingphase,butthatis
ratherlatetodiscoverthatwhatyoubuiltisgoingtorequiresignificantchangesbeforeitwillmeetthe
users'needs.Therefore,weadvocateacceptancetestingoftheproposedproductdesign(notthe
softwaredesign)beforethesoftwareisbuilt.Techniquessuchasexperiencesketching
[BuxtonOnDesignForUsability],paperprototypingandWizardofOztesting

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 92

[GouldLew wisOnDesignFForUsability]canbeusedttoverifythatyouare"buildingtherigh htsystem"very


earlyinth hilethereissttilltimetoadjjusttheproductdesign.Fo
heprojectwh orinformatio
onaboutthesse
techniquees,seetheUssabilityTestin
ngthumbnailinVolumeIV.

5.2 TypesofRequirem
ments
Therequirements,how weverderivedd,aretypicalllydividedinto
otwobroadccategories,fu
unctional
requiremeentsandnonfunctionalreequirements(alsoknownaasparafuncttionalrequireements,extra
functionalrequirements,systematttributesorqu nalrequiremeentsdescribethe
ualityattributtes.)Function
functionalitytobeprovidedtouserrsoradministtratorsoftheesoftwareinttensivesystemm.Nonfunctional
requiremeentstranscen ndthefunctio
onality.Thereelationshipbeetweenthetwwokindsofreequirementsis
illustrated
dinFigure3.


Figure3
Functionaalvs.nonfuncctionalrequirrements
mentsandforverifyingthat
Differenttechniquesareusedfordescribingthevariouskindssofrequirem
thoserequirementsarremet.

5.2.1 Functional
F Requirem
ments
Functionaalrequiremenntsdescribeh
howvarioustyypesofuserssexpectthesystemtohelp
pthemdotheeir
jobs.Thefunctionalrequirements,wwhetherexplloredandgatthereddirectllyorderivedfromaproduuct
design,caanbeorganize
edandcommmunicatedannumberofdiffferentways,includingtheefollowing:

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 93

Businessprocesses

Usecases

Userstories

Businessrules

Subsystems

Featurelists

Scenarios

Protocolspecifications

Functionalspecifications

Statemodels

Figure4illustratesaveryhighlevelsystemrequirementmodelthatincludesbusinessworkflowtying
togetherseveralusecases.TherequirementsincludetheneedtointeractwithSystem4aspartof
someoftheusecases.ItalsorequiresSystems1,2and3toexchangeinformationcreatedormodified
duringtheworkflow.


Figure4
RequirementsinvolvingUsers,UseCasesandUserStories
Thisguidehighlightsasmallsetofpopularrequirementspractices(seeVolumeII).Theyareusedto
illustratehowtherequirementspracticesandartifactsarerelatedtothetestsyoumayuseduring

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 94

acceptanceofasoftwareintensivesystem.TheUserModelingactivityisusedtocapturesalient
informationabouttheusersofthesoftware.Itcanhelptheproductdevelopmentteamunderstandthe
requirementsbetterbyunderstandinghowtheusersthinkandwhatmotivatestheirbehavior.UseCase
Modelingisusedtocapturewhattheuserswantthesystemtohelpthemachieve.Eachusecase
describesonekindofinteraction(e.g.transaction)betweenauserandthesystem.Ausecasetypically
requiresmanytestcasesbutoftencannotbetestedwithouttestingotherusecases.UserStoriesare
usedtobreakthefunctionalityofthesystemintoverysmallbutstillusefulandtestablechunksthatcan
bebuiltinjustafewdaysorweeks.Theyaretypicallymuchsmallerthanusecasesbutmayspan2or
moreusecases.Theymightdescribeabusinessruleorevenjustonescenarioofanintersystem
communication.Userstoriesmaptooneormoretestcases.
Theserequirementsactivitiesandartifactshelpusunderstandthetypesoftestswemayneedto
executetogatherthedataonwhichtheacceptancedecisionismade.

5.2.2 NonfunctionaRequirements
Nonfunctionalrequirementsarerequirementsthatdescribegeneralqualitiesorbehaviorsofthe
systemthatspanspecificusagescenarios.Frequently,theserequirementsaretracedbacktoprotection
ofvariousstakeholderswhomayormaynotbeusersofthesysteminquestion.
Unlikefunctionalrequirements,whichvarygreatlyfromsystemtosystem,thereisafairlystandardlist
oftypesofnonfunctionalrequirements.Manyoftheserequirementsendwiththesuffix"ility"because
theydescribecharacteristicsthewholesystemneedstosupporteitherwhileitisinuseoraspartofits
overallsystemlifecycle.
Requirementsthatthesystemneedstosatisfywhileitisoperatinginclude:

AvailabilityWhendoesthesystemneedtobeavailableforuse?

DataIntegrityThedataneedstobestoredandprocessedinawaythatitcanbereliably
retrievedwithoutchangingitsoriginalvalue.

SafetyThesystemshouldnotcausephysicaloremotionalharmtoauserorstakeholder.

RecoverabilityWhenitfails,thesystemshouldrestoreitspreviousstatewithoutundue
hardshiptotheusersorsupportersofplatform.

AccessibilityPeoplewithdiverselimitationsshouldbeabletousethesystem.Itmayneedto
conformtostandardslikethosestatedbytheADAorotherregulatoryorstandardsbodies.

SupportabilityItshouldbeeconomicaltoprovidesupporttousersoftheproduct.

ReliabilityItshouldworkwellandresistfailureinallsituations.

RobustnessItshouldtakeabuseandstillfunction(faulttolerance).

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 95

PerformanceWhatspeedorthroughputbenchmarksdoesitneedtomeet?Performance
includesresponsetime:thesystemshouldrespondtouseractionsandrequestsinatimely
manner.

UsabilityItshouldbeeasytousebyitsintendedusers.

InstallabilityTheproductshouldbeeasytoinstallontoatargetplatform.Thisissometimes
partofusability.

SecurityTheproductshouldprotectagainstunauthorizeduseorintrusion.

ScalabilityTheproductshouldbecapableofbeingscaledupordowntoaccommodatemore
usersortransactions.

CompatibilityWithwhatexternalcomponentsandconfigurationsdoesitneedtowork?
Requirementsrelatedtothetotalcostofownershiporproductlifecycleinclude:

TestabilityInwhatwaysshouldtheproductbetestable?

MaintainabilityHoweasyshoulditbetoevolve,fix,orenhancetheproduct?

PortabilityHoweasyshoulditbetoportorreusethetechnologyelsewhere?

LocalizabilityHoweconomicalwillitbetopublishtheproductinanotherlanguage?

ReusabilityHoweasilycansourcecode(e.g.classes,functions,etc.)beusedinother
circumstances?

ExtensibilityHoweconomicalshoulditbetoenhancetheproductwithotherfeaturesand
addons?

ConfigurabilityHoweasyshoulditbetopreparetheproducttobeusedonavarietyof
differentplatformsorforslightlydifferentusesandoperations?
Theprecedinglistoftypesofnonfunctionalrequirementsisnotintendedtobeexhaustive.Italsoisnot
intendedtobeuniversal;someoftheserequirementsmaybeirrelevantforsomesoftwareintensive
systems.Partoftheartofrequirementsgatheringorengineeringisdecidingwhichonesareimportant
andwhichonesarenot.Thosethatareimportantneedtobemadeexplicitandincorporatedintothe
testplans;thosethatarenotimportantcanprobablybeignored(buttheriskofignoringthemshouldbe
assessedbeforedoingso.)
Mostofthenonfunctionalrequirementscutacrosstheusecasesofthesystem.Thatis,theyapplyto
many,ifnotall,ofthediscretechunksoffunctionalitydescribedinthefunctionalrequirements.Note
thatsomeformsofnonfunctionalrequirementscanbedescribedatleastpartiallyinfunctionalterms;
securityisagoodexample.YoucansaythatUserRoleXshouldbepreventedfromchangingthevalueof
fieldFonscreenS.
Thekeytotestingconformancewithnonfunctionalrequirementsistheclassificationofeachofthese
requirementsindicatingtowhatdegreetheprojectstakeholderscareabouttherequirement.For
example,foranapplicationonebuildsforonespersonaluse,onemaynotcareaboutscalability

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 96

becausetherewillbeonlyoneuser,butonalargeecommerceapplication,scalabilityisveryimportant.
Itisworthreviewingthislistofnonfunctionalrequirementsandconsciouslydecidinghowimportant
eachoneistothesuccessofyourproductorproject.Thefollowingtableliststhegoals,importance,and
rationaleofavarietyofnonfunctionalattributesforaspecifichypotheticalsystem.

Attribute Goal Importance Rationale

Web site performance under Less than 500 milliseconds High Major source of
load response time revenue

Web server capacity under At least 300 transactions per Medium Large number of users
load second.
Graceful degradation under
load

Reliability/availability 7x24x52 Critical Users require instant


satisfaction when
worried about their
money

Usability Easily discoverable High Most users will use


infrequently

AppendixXinVolumeIVTestingActivities&Responsibilitiesisanexampleofachecklistthatcanbe
usedtorecordthesedecisions.

5.3 Summary
Inordertobeabletodoagoodjobofreadinessassessmentandacceptancetesting,thetargets
(requirements)mustbeclear.Readinessassessmentandacceptancetestingaredrivenbythequestion
"howwillweknowwearedone?"Assuch,theyshouldideallystartearlyintheproject.
Thereadinessassessmentandacceptancetestingactivitiesgatherdataabouthowwellthesystem
satisfiestherequirements.Therequirementsincludebothfunctionalandnonfunctionalrequirements.
Thereareanumberofalternativewaystodecomposetherequirementsintomanageablechunks.Some
formsofrequirementsmapmoredirectlytotestcasesthanothers.

5.4 WhatsNext?
Thingscananddogowrongwhenbuildingsoftwareintensivesystems.Commonlyoccurringproblems
includemisunderstoodrequirementsandfailuretomeetnonfunctionalrequirements.Thenextchapter
describestechniquesforidentifyingwhatmightgowrongandplanningreadinessassessmentand
acceptancetestingactivitiesthatwoulddiscoverthemasearlyaspossible.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 97

5.5 References
[HSPK]HendrikSchifferstein,PaulHekkert(Eds.)ProductExperience.Elsevier,2008.
[BuxtonOnDesignForUsability]BillBuxton.SketchingUserExperiences:GettingtheDesignRightandthe
RightDesign.MogranKauffman,2007.
[GouldLewisonDesignForUsability]Gould,J.D.andC.Lewis,DesigningforUsabilityKeyPrinciplesand
WhatDesignersThink,inProc.ofACMCHI83ConferenceonHumanFactorsinComputingSystems.p.
5053,1983.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 98

Chapter6 RiskModel
Everyprojectinherentriskswhetherornotwerecognizethem.Determiningwhatthoserisksarecanbe
achallenge.Projectrisksofproducingsoftwareintensivesystemsrelatetowhethertheproduct
developmentteamunderstandswhattheProductOwnerwantsbuiltandhowtobuilditandinabilityto
constructanddelploytheproductasintended,inatimelymanner,andcosteffectively.Opeationalrisks
relatetotheproductnotfunctioningasexceptedorfailingduringoperation.Ariskmodelcanhelpus
understandalltherisksandcomeupwithriskmitigationplanstoreducetheirlikelihoodorimpact.

6.1 WhatCouldPossiblyGoWrong?RiskAssessment
Onewaytodefineriskisbyaskingwhatkeepsusawakeatnight?Morespecifically,whatmighthappen
andwhatwouldbetheconsequencesifitdidhappen?
Wecanmakethediscussionofriskmoremeaningfulbytranslatingnebulousconcernsintoconcrete
eventsthatcouldhappenandtalkingaboutthelikelihoodthatitmighthappenandtheconsequencesif
itdoeshappen.
Forexample,supposeweorderedsomecriticalhardwarewhichwerequiretoconductcertaintypesof
acceptancetestingwithoutwhichwearenotpreparedtomaketheacceptancedecision.Whatcould
possiblygowrong?Thefollowingaresomeexamples:

Thehardwarecouldbedestroyedintransit.

Thewronghardwarecouldbeshippedeitherthroughanorderingerrororafulfillmenterror.

Thehardwarecouldbedefective.

Foreachoftheprecedingeventswecanestimatethelikelihoodthatitwilloccurandassesstheimpact
onourprojectifitdidoccur.Performingthesetwocalculationsseparatelyhelpsustobetterunderstand
theriskasshownintheriskmatrixinFigure1.Astheprobabilityofaneventincreases,therisk
increases.Astheconsequencesofaneventincreases,theriskincreases.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 99

LowMediumHigh

Probability

Low Medium High


Consequence
Figure1
RiskMatrix
InFigure1RiskMatrix,thetwoareasoneithersideofthediagonalandthediagonalitselfrepresent
threedegreesofrisk.Thebottomleft(green)riskregimerepresentslowrisk,thetopright(red)risk
regimerepresentshighrisk,andthetoplefttobottomrightdiagonal(yellow)riskregimerepresents
moderaterisk.Ingeneral,risksthatfallinthesameriskregimeareequallyimportanttomitigate.

6.2 UnquantifiableRisks
Itisimportanttorecognizethatitisnottheknownandquantifaiablerisksthatcausethebiggest
problems,butthecompletelyunexpectedthingsthatdoso.Inhistreatmentoftheissue,PeteMcBreen
emphasizesthatflexibilityisthekeytosurvivingtheunexpected[McBreenOnRiskManagement].

6.3 WhoShouldIdentifytheRisks
Everyoneontheprojecthasadifferentandvaluableperspectiveonthepossiblerisksinvolvedin
buildinganddeliveringaproductwhetheritisforsaletocustomersorforusebyinternalusers.Risk
identificationshouldnotbeamonopolyoftheprojectmanagersbutshouldincludeproductdesigners,
productdevelopers,testersandallstakeholdersincludingtheProductOwner.Ultimately,itisthe
ProductOwnerwhoneedstounderstandtherisksanddecidewhichonestheycanacceptasisand
whichonestheywanttospendmoneytoreducethereforetheProductOwnermustbeintimately
familiarwithatleasttherisksinthered(topright)andyellow(diagonal)regionsoftheriskmatrix.
Ultimately,asTomGilbstatedinhisAskingPrinciple:Ifyoudontaskforriskinformation,youare
askingfortrouble[GilbOnSEManagement].

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 100

6.4 ShouldWeDoSomethingAboutIt?RiskManagement
Afterweunderstandtherisksofourproject,whatcanwedoaboutthem?Therearethreepossible
coursesofaction:

Wecanaccepttheprobabilityandconsequencethataparticulareventmighthappen.

Wecanperformactivitiestoreducethelikelihoodofithappening.

Wecanperformactivitiestoreducetheconsequenceifitdoeshappen.

Thecoursewechoosedependsonanumberoffactors,includingthefollowing:

Thefactorswehavecontrolover,suchasthefollowing:

Iftherearenocoursesofactionthatcouldreducethelikelihoodofsomething
happeningwemaybeforcedtofocusontryingtoreducetheconsequence.For
example,theonlywaytoavoidanextremeweathereventmightbetomovetoa
differentarea,whichmaysimplyexchangeonesetofextremeweathereventsfora
differentset.Butwecanpreparefortheoccuranceoftheextremeweatherbychoosing
appropriatefacilitiesorotherpreparations.

Ifthereisnowaytoreducetheconsequenceofanevent,weneedtofocusonreducing
thelikelihoodofitoccurring.Forexample,mostpeoplewouldagreethatitisbetter
(andeconomicallymorefeasible)totrytoreducethelikelihoodofaheartattackby
exercisingandeatingwellthantotrytoimprovetheprobabilityofsurvivingitbyhiring
aheartspecialisttobeatyoursideatalltimes.

Therelativecostoftheoptionsavailabletouse.Ifitismuchcheapertoreducethelikelihood
thantheconsequence,weshouldfirstfocusonloweringthelikelihoodandviceversa.Notethat
thecostistypicallynonlinearandgetsmoreexpensivetheclosertozerowetrytodrivethe
likelihoodorconsequence.Thereforeitmaybemoreappropriatetotrytopartiallyreduceboth
likelihoodandconsequenceratherthanreducejustonetoalargerdegree.

Thecostofriskreductionrelativetothecostwewouldincuriftheriskoccurred.Forexample,if
aparkingticketcoststwiceasmuchaspayingfortheparkingandthereisonlya20percent
chanceofgettingcaughtandticketed,wemaychoosetotakethechancebynotpayingfor
parking.

6.5 HowCanTestingHelp?RiskMitigationStrategies
Ifwedecidetomitigatearisk,howwegoaboutitdependsonthenatureoftherisk.Risksthatrelateto
thepossibilityofdeliveringadefectiveproductareamenabletoriskmitigationthroughsomeformof
testing.Risksthatrelatetodiscoveringsomethingtoolateforatimelyfixcanbemitigatedbyactivities
thatmovethediscoveryearlierintheproject.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 101

6.5.1 DoingSomethingEarlier
Manyrisksonprojectsarerelatedtotime.Willsomethinghappenintime?Ifithappenstoolate,willwe
havetimetoreactwithoutaffectingtheprojecttimeline?
Agoodexampleofthisisthelatediscoveryofmissedormisunderstoodrequirements.Whenthis
discoveryoccursduringtheacceptancetestingphaseofaprojectshortlybeforetheproductisexpected
tobeturnedovertousers,theimpact(ofthediscovery)maybeasignificantdelayinachievingthe
businessbenefitsexpectedfromthesystem.Inthiscase,wecanreducetheimpactofthediscoveryby
doingtheacceptancetestingactivitiesearlierintheproject.
Theincrementalacceptancetestingpracticeusedonmanyagileprojectsisonewaytomovediscovery
ofproblemssuchasmisunderstoodrequirementsearlierintheprojectsothereisplentyoftimeto
addressthem.Sequentialprojectscanalsoreapthebenefitsofincrementalacceptancetestingby
movingtoanincrementaldeliverymodelwheretheproductisbuiltinfunctionalmodulesthatcanbe
acceptancetestedastheybecomeavailable.

6.5.2 DoingSomethingDifferent
Anextremeformof"toolate"discoveryiswhenwedonotdiscoveritatallandaproblemoccurswhen
theproductisbeingused.Iftheproblemissevereenoughtohaveseriousrepercussions,the
consequencescanbedisastrous.Thehighprofilelossesortheftofcustomers'privateinformationisjust
oneexampleofsomethingdiscovered"toolate."Thesetypesofriskmayrequireadditionalactivitiesto
reducethelikelihoodoftheiroccurrence.Thesolutionoftenliesindoingadditionalkindsoftestingto
improvethelikelihoodthatacertainclassofdefect,ifitexists,isfoundintime.Manytestauthoring
practicesarefocusedonwaystodefineadditionalteststhatimprovethetestcoverage(fromarisk
coverageinsteadofacodecoverageperspective).

6.6 Summary
Ariskmanagementmodelandawaytotrackrisksandriskmitigationisimportantonalltypesof
projects.Makingrisksexplicitallowsustodevisewaystochoosetherightriskstotakeandtomitigate
theriskseitherbyreducingtheprobabilityortheconsequence.Forrisksrelatedtomisunderstandingof
therequirements,earliertestingthroughincrementalacceptancecanreducetheconsequencebygiving
usmoretimetoreworkanyissues.Prioritizingacceptancetestingactivitiesshouldbepartoftherisk
modeltoensurethathighlycriticalfeaturesarethoroughlyaddressedduringsoftwareacceptance.The
ProductOwnerneedstobeinvolvedinidentifyingandunderstandingrisksandindeterminingwhich
risksneedtobeaddressed.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 102

6.7 WhatsNext?

Asignificantriskonmostprojectsislatediscoverythattheproductismuchfartherfromdonethan
thought.Thenextchapterintroducesamodelforthinkingaboutthedegreetowhichtheproductis
done.

6.8 Reference

[McBreenOnRiskManagement]PeteMcBreen,TheMythofRiskManagement,BetterSoftware
Magazine:3034,June,2008.

[GilbOnSEManagement]TomGilb,PrinciplesofSoftwareEngineeringManagement,AddisonWesley,
1988.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 103

Chapter7 DonenessModel
Determiningwhethertheproductisdoneistheultimategoalofsoftwareacceptance.Knowinghow
closetodoneweare,isakeyfunctionofprojectmanagementdiscipline.Thissectionintroducesamodel
fordetermininghowcompleteaproductisrelativetoitsgoalsfrombothfunctionalandnonfunctional
perspectives.

Akeypartofmakingtheacceptancedecisionrevolvesarounddecidingwhethertheproductisdone.
Comingtoconsensusondonenessrequiresagreementonwhatisbeingassessedaquestionof
targetgranularityandwhatitisbeingassessedagainstthedonenesscriteria.Initsmostbasicform,
theacceptanceprocessdefineshowwedecidewhetheraparticularreleaseoffunctionalityisdone.In
itsmoreadvancedformswecandeterminedonenessatthelevelofindividualfeaturesorworkitems.
Eachofthesedifferenttargetshasasetofdonenesscriteriathatmustbeagreedupon.Thefollowing
aresomeexamplequestionstoconsider:
Isasoftwareintensivesystem(forexample,asoftwareproduct)readyforanalphatestwitha
friendlyusercommunity?
Isanindividualfeatureoruserstoryreadyforacceptancetestingbyabusinesstester?
Isasoftwareintensivesystemreadyforthedesignclosemilestone?

Thedefinitionof"done"isdifferentforeachoftheseexamples.

7.1 ReleaseCriteriaDonenessofEntireProduct
Whendecidingtheacceptanceofanentiresoftwareintensivesystemforreleasetousers,donenessisa
binarystate.EithertheproductisacceptedbytheProductOwnerandripeforreleasetoitsintended
audienceorfortheusagedecision,oritsnot.Ifitsnot,thenwecantalkabouthowfartheproductis
fromthestateofbeingdonefortheunderlyingdecision.Thesameprinciplesapplytothereadiness
decisionthatprecedestheacceptancedecision:theoneaboutwhethertoreleasetheproducttothe
ProductOwnerfortheacceptancedecision.Thustherearetwomaincriteriafordeterminingwhethera
productisdone:
Isthereenoughhighvalue,ProductOwnerdefinedfeaturesincludedtomaketherelease
worthwhile?
Isthequalityofthefeatureimplementationshighenoughforthefeaturestobeusableintheir
expectedusagecontext?

Thefirstcriterion,knowninthisguideasMinimumMarketableFnctionality(MMF),istypically
determinedwhileplanningtherelease.Thecriterionmayberevisitedastheprojectisbeingexecuted

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 104

andmoreislearnedaboutthesystemcontext(suchasbusinessrequirements)andthetechnical
capabilitiesoftheProductDevelopmentTeam.
Whenreviewingacceptancetestresultsforeachfeature,itisfairlysimpletodeterminethepercentage
offeaturesthatisdone.ThisisthenumberoffeaturesthattheProductOwnerhasdecidedpasstheir
criticalacceptancetestsdividedbythetotalnumberoffeaturesfortherelease.Itisalsopossibleto
expressthisratioinstorypointstoaccountforfeaturesofdifferentsizes.
Thesecondcriterion,knowninthisguidanceasMinimumQualityRequirement(MQR),iswhatwe
constantlytestagainstwhilebuildingandassessingthesoftware.Tobeabletosaywhetherafeature
hasmettheMQR,weneedtohavetheacceptancetestsdefinedforthatfeature;inthisguidance,thisis
theperfeaturedefinitionof"Whatdonelookslike."Figure1illustratesgraphicallythedegreeof
donenessusingthesetwocriteria.Eachchartrepresentingthedonenessononeoftwoseparatedates
inthesameproject.


Figure1
DonenessrelativetoMMFandMQRattwopointsintime
InFigure1,thechartontheleftshowsthecompletenessofeachfeatureatpointXintime;thecharton
therightshowsthecompletenesstwoweekslater.Eachcolumnrepresentsafeaturewiththewidthof
eachcolumnindicatingtheestimatedefforttobuildthefeature.ThelinelabeledRAT(Readyfor
Acceptance)marksthelevelatwhichthefeatureisdeemedreadyforacceptancetestingbyvirtueof
havingconductedthereadinessassessment.Itistheperfeatureequivalentofthereadinessdecision
thatismadeatthesystemlevel.ThespacebetweentheRATlineandthelinelabeledMQRiswhen
acceptancetestingisdone.ThelinelabeledMMFisthedemarcationbetweenthefeaturesthatmustbe
present(leftoftheline)andthosethatareoptional(rightoftheline;omittedinthesediagrams)forthis
release.
WecanseeprogressovertimebycomparingtheleftandrightchartsinFigure1.Features1through5
(numberingfromtheleftstartingfrom1)werealreadycompletedatthepointintimerepresentedby
theleftchart.Features5through7werealreadystartedattimexandcompleted(deemeddone)in
theperiodendingattimex+2weeks.Features8through10werealreadyinprogressattimexand
werenotcompletedattimex+2weeks.Features11and12werestartedafterthedatedepictedbythe
leftchartandnotfinishedatthetimedepictedbytherightchart.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 105

Theproductisdeemedacceptablewhenallfeaturespassalltheiracceptancetests.Thisistheupper
rightcornerofthechartwherethelineslabeledMQRandMMFintersect.Whentherectangletothe
lowerleftofthispointisentirelycoloredin,theproductisaccepted.Tosimplifythediscussion,thenon
functionalityrequirementsaredeliberatelyignored,buteachsetofnonfunctionaltestscouldbe
treatedasanother"featurebar"fromtheperspectiveofmeasuringdoneness.

7.1.1 GoodEnoughSoftware
ItisveryeasytosayItallhastobethere!whenaskedwhatfunctionalityismustbepresenttosatisfy
theMMF.Likewise,itiseasytoreplyTherecanbenobugs!whenaskedwhatMQRneedstobe.But
theseanswersaretakingtheeasywayout.Theproductownerhastomakeaconsciousdecisionabout
thevalueofqualityandfeatures.Thedogmaticanswer(allthefeaturesandabsolutelynobugs)may
delaydeliveryoftheproductbymonthsorevenyears.Meanwhileacompetitorsproductmaybe
earningmoneyandmarketsharebecausethatcompetitorhadamorerealisticgoalforMMFandMQR.
ThechoiceofMMFandMQRshouldbeaconsciousbusinessdecisionmadeinfullknowledgeofthecost
ofincreasedqualityandfunctionalityintermsofactualcosts(additionaldevelopment,testing)and
opportunitycosts(delayedincomeandotherproducts).
ForadditionaltreatmentofthetopicofGoodEnoughSoftware,see[BachOnGoodEnoughSoftware]and
[YourdonOnGoodEnoughSoftware]

7.2 DefiningWhatDoneMeans
Foreachchunkoffunctionalitywehavedecidedtodelivera"feature"thatneedstobedefinedforthe
minimumqualityrequirement(MQR)intheformofasetofacceptanceteststhatmustpassbeforethe
ProductOwnerwillacceptthefeature.Thesetofacceptancetestsforareleaseismerelytheaggregate
oftheacceptancetestsforallthefeatures("functionaltests")plustheacceptancetestsforeachofthe
nonfunctionalrequirements(the"nonfunctionaltests")thataredeemedmandatory.

7.2.1 Determining"Readiness"
"Readiness"iswhentheProductDevelopmentTeambelievestheproductis"doneenough"toaskthe
ProductOwnertoconsideracceptingtheproduct.ThisimpliesthattheProductDevelopmentTeamhas
areasonablyaccurateunderstandingofhowtheProductOwnerwillconducttheacceptancetesting.(In
somecases,theProductDevelopmentTeam's"readinesstests"maybemuchmorestringentthanthe
acceptanceteststheProductOwnerwillrun.)Thisunderstandingisknownasthe"acceptancecriteria"
andisusuallycapturedintheformofacceptancetests.Ideally,theacceptancetestsarejointly
developedandagreeduponbetweentheProductDevelopmentTeamandtheProductOwnerbefore
thesoftwareisbuilttoavoidplaying"battleship"or"blindman'sbluff"andtheconsequentrework
whentheProductDevelopmentTeamguesseswrong.Itisalsoidealforreadinesstestsandtestresults
tobesuppliedtotheProductOwnerbytheProductDevelopmentTeam.Thiscanassistinauditingthe
readinesstesting,streamliningacceptancetesting,andperformingamoreinformedstyleoftesting,
suchasexploratoryacceptancetesting.Thiskindoftestingcanbethoughtofablackhattesting
referringtoEdwarddeBonosoneofsixviewpointsforthinkingaboutaproblem

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 106

[DeBonoOnSixThinkingHats].Ablackhatapproachlooksataproblemfromacautiousandprotective
perspective,concentratingonwhatcangowrong.

7.2.2 DonenessofIndividualFeatures
Foranindividualfeaturewecandescribethedegreetowhichitisdoneinseveralways.Oneishowfar
alongitisinthesoftwaredevelopmentlifecycle.Forexample,Thedesignisdoneandcodingis80%
finished.Weexpectthefeaturetobereadyfortestingintwoweeksaspartofthetestingphase.This
formofprogressreportingisoftenusedinsequentialprojectsbecausethedesignandcodingcantake
manyweeksorevenmonths.Unfortunately,itisnotaveryusefulbecauseofitssubjectivity,low
transparency,andcoarsegranularity.Inprojectsthatadoptatoocoarselevelofschedulingand
clumpingwork,codingmayremain80%finishedformanyweeksormonthsevenifprogressisbeing
made.Itcanthereforebeverydifficulttotellwhenprogressisnotbeingmade.
Oncethecodinganddebuggingisfinished,itismoreusefultodefinedonenessintermsofwhichtest
casesarepassingandwhicharenot.Thisalternativewayofmeasuringprogressisobjective,
transparent,andfinegrained.Inthesequentialapproach,thepercentageoftestcasespassingstaysat
zeroformostofthelifecycleandthenrapidlyrisestosomewherenear100%.Initerativeand
incrementalprojects,featuresaresometimesimplementedonetestcaseatatimetherebyallowingthe
percentageoftestcasespassingtobeusedasanobjectiveprogressmetricatthefeaturelevelinstead
ofattheproductlevel.
Thenumberoftestcasesforaparticularfeaturemaychangeovertimeasthefeatureisunderstood
better.Thismayalsobeasignofworkcreep(moreefforttoimplementduetomorecomplexity)or
scopecreep(additionalfunctionalitythatwasntoriginallyintended).Theadditionaltestsmayresult
fromincreasedclaritycausedbyattemptingtowritetestsoritmaycomeasaresultoffeedbackfrom
otherformsoftestingincludingusabilitytesting,exploratorytestingoracceptancetestingofanAlpha
orBetarelease.Regardlessofthesourceoftheadditionaltests,thepercentdonemaytakeastep
backwardsifthenumberoftestsincreasesmorethanthenumberofadditionaltestspassing.

7.2.3 DonenessofNonfunctionalAttributes
Nonfunctionalattributesofthesystemareabitdifferentfromindividualfeaturesinthattheytendto
applyacrossallfeatures;thisiswhysomepeopleprefertocallthemparafunctional.Nonfunctional
attributes,whenquantifiedandobjectivelymeasurable,maybebinaryinthattheproducteither
satisfiestherequirementornot.Anexampleisscalability:eithertheproductcansupportamillion
transactionspersecondornot.Wemight,however,beabletobreakdownthefunctionalityofthe
systemintofunctionalareasanddeterminetheusabilityofeachareaindependently.Forexample,we
mayrequireahighdegreeofusabilityforcustomerselfservicefunctionswhileadministrativefunctions
canhaveamuchlowereaseofuse.Othernonfunctionalrequirementsarequantifiable.Forexample,
howmanyuserscanwesupportbeforeresponsetimeexceedsourresponsetimethreshold?Itcouldbe
1user,10users,or1000users.
Wecanchoosewhetherwewanttovisualizeprogressonnonfunctionalcompletenessaseither
additionalfeatures(support10users,support100users,support1000users)asillustratedinFigureZor

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 107

asasinglefeature(supportmanysimultaneoususers)withaminimumqualityrequirement(e.g.1000
users)asillustratedinFigureY.Ourchoicemaybeinfluencedbyhowweorganizetheworktodevelop
thefunctionality.Agileprojectsoftenoptforthefeatureperthresholdapproachbecausesupportfor
additionaluserscanberolledoutoveraseriesofiterations.Sequentialprojectsmayprefertotreat
themasdifferentlevelsofqualityforasinglefeatureasillustratedinFigureYorasadifferentlevelof
qualityacrossallfeaturesasillustratedinFigureX.(SeethediscussionofvaryingMMFforAlpha/Beta
releasesinsectionTheAcceptanceProcessforAlpha&BetaReleasesinChapter1foracounter
argument.)


FigureZ
IncrementalNonfunctionalFeatures
InFigureZtherequirementforsupportingmultipleusersisdividedintotwofeatures.Thefeatureto
supportatleast10usersisrepresentedbycolumn6(withhighlightedborder)andisreadyfor
acceptancetestingwhilethefeaturetosupport100users,representedbycolumn14(withhighlighted
border),hasnotyetbeenstarted.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 108

FigureY
SingleNonfunctionalityFeature
InFigureYtherequirementforsupportingmultipleusersistreatedasasinglefeaturerepresentedby
column10.Itcurrentlysupportsatleast10usersbutisnotreadyforacceptancetestingatthefull
requirementof100users.


FigureX
BigBangNonfunctionality
InFigureXverificationoftherequirementforsupportingmultipleusersistreatedasanadditionallevel
ofqualitylabeledNFacrossallfeatures.Ithasnotyetbeenstarted.

7.3 Communicating"PercentDoneness"
Itcanbesaidyouareeither"done"or"notdone(yet)."Butinpractice,itisimportanttobeableto
clearlycommunicate"howclosetodone"youare.Morespecifically,itisimportanttobeableto
communicate"whatremainstodobeforewecansayweare'done'".Thisistheamountofworkleftfor
eachfeaturethathasnotyetpassedallitsacceptancetestssummedoverallthefeaturesthatarepart
oftheMMF.WhenlookingateitherdiagraminFigure1,wecanask"Whatpercentageoftherectangle
below/leftofMQR/MMFiscoloredin?"Thisgivesusasenseofhowmuchworkisremaining.Alotof
whiteimpliesalotofwork;verylittlewhitemeanswearenearlydone.Wherethewhiteislocatedtells
uswhatkindofworkislefttodo.Whiteacrossthetopmeanswehavelotsofpartiallyfinishedfeatures;
whiteprimarilyontherightimpliesthatentirefeatureshaventbeenbuiltyetbutmostfeaturesare
eithernotstartedorfullyfinished.Thesepatternsofprogressiondependprimarilyontheproject
managementmethodologyweareusing.ThediagraminFigure2showssnapshotsofcompletenessfor
threedifferentprojectstyles:

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 109

Figure2
Completenesscharts
InFigure2,thefirstrowofchartsrepresentsaclassicsequentialorphasedordocumentdrivenstyleof
projectmanagementwithwhiteacrossthetopindicatingthatallfeaturesareatsimilarstagesof
completion.ThebottomrowrepresentsaclassicExtremeProgrammingprojectwithwhiteprimarilyon
therightindicatingthatthereareveryfewfeaturespartiallydonesincemostareeitherdoneandnot
started.Themiddlerowrepresentsaprojectusinganincrementalstyleofdevelopmentwithlonger
featurecyclesthantheExtremeProgrammingproject.Noticethedifferenceinhowthecoloringinof
thechartadvancestowardtheupperrightcornerwheretheprojectwouldbeconsideredfullydone
basedon100%ofMMFand100%ofMQR.Thenextfewsectionsdelvemoredeeplyintohowwe
calculateandcommunicatedonenessonthesedifferentstylesofprojects.

7.3.1 CommunicatingPercentDoneonAgileProjects
Anagileprojectcansimplydividethenumberoffeaturesthatareacceptedbytheproductowneras
donebythetotalnumberoffeaturesscheduledfortherelease.Thisprovidesthepercentageof
featuresthataredone.Wecanexpressthismeasureinmonetarytermsbyweightingeachfeatures
percentcompletenessbythefeaturesestimatedcost.Theestimatedcostisrepresentedbythewidth
ofeachfeaturecolumnintheprogresschartsofFigure2.Howeverthemonetaryequivalentofafeature
calculatedthiswayisnotthesameasthefeaturesbusinessvalue.Itsessentiallyanexpressionofthe
featurescompletenessincostterms.Themonetaryequivalentiswhatisreferredtoasearnedvaluein
EarnedValueManagement(EVM).SeetheSidebartitledEarnedValueManagementandAgileProjects.

Sidebar:EarnedValueManagementandAgileProjects
EarnedValueManagement(EVM)isabudgetandscheduletrackingtechniquedevelopedbythe
ProjectManagementInstitute.Earnedvaluetrackstherelativeprogressofaworkunitasa
percentageoftheallocatedbudgetthathasbeenspentforthatworkunit.Ifaworkunitis80%
completeandwasallocateda$100budget,thenitscurrentearnedvalueis$80.Thetotalvalue
earnedbytheprojectisthesumoftheearnedvaluesofalltheworkunitsscheduledandbudgeted

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 110

fortheeproject.Whentheprojecctiscompletee,itstotalearrnedvalueeq
qualstheprojjectstotalbu
udget.
Thereffore,unliketh
henamesugggests,earnedvalueisessentiallyaneffo orttrackingm
metric.
InEVM
M,actualcostsarealsotracked.Ifatanypointintheeproject,theactualaccum
mulatedcost
exceeddstheearned dvalueoftheprojectatthhattime,thep
projectiscon
nsideredoverbudget,withhthe
shortfaallrepresentingthebudgeetoverrun(inEVMterms,costvariancee).Iftheaccumulatedearn ned
valueaatthatpointintimeisbelo
owtheestimatedtotalexpendituresacccordingtothheplanned
budget,thentheprrojectisconsiideredoversschedulewith hthedifferencerepresentiingthemoneetary
expresssionoftheprojectssched duleshortfall(inEVMterm
ms,schedulevariance)attthatpoint.
Schedu ulevarianceccanalsobecaalculatedincaalendartime.
FigureWillustratessschedulevarriance(inbotthmonetaryaandtemporallterms)andccostvariancefora
projectwithanestiimatedtotalb budgetof$5millionandaanestimatedscheduleof1 1year.InJulyy
2010,aatanearnedvalueof$2.5 5million,theprojecthasatotalearnedvaluethatisonly50%ofits
planneedtotalcost,butitshouldhaveaccumu ulated75%offitaccordingtotheprojecctplan.Therefore
itsoveerschedulebby$1.75millio
onor50%inmonetaryterrms.Sincetheeprojectshouldhave
accum mulatedthatm muchearnedvaluebackin nApril2010,incalendartim
meterms,itiis3months
behind d.Theprojecctisalso120%
%overbudgeetonJuly2010becauseith hasaccumulaatedanactualcost
of$5.55millionatth
hatdatecomp paredtothe$ $2.5millionitthasearned.

FigureW
Calcula
ationofsched
duleandcosttvarianceinEEVM.
AnappproachtoporrtingEVMtoaanagileconteextisdiscusseedbyGriffithhsandCabri
[CabriGGriffithsOnAggileEVM].GrifffithsandCabbriadvocateaapplyingEVM Monanincrem mentalbasis
(iterationbyiteratio
onorreleaseebyrelease).Inanagileprroject,agrand
dplanfortheeprojectisno
ot
meaningfulforproggresstrackinggpurposessin ncethegrand dplanwouldsubjecttoconstantchangge
duringgthecourseooftheproject.Howevermminiplansinteermsofworkkscheduledin nthecurrent
scope(featuresorsstories)andeestimatedcossts(resourcessallocatedtothescope)areusually

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 111

availableandreliableenoughtoserveasareferencepointonaperiterationorperreleasebasis.
Budgetandscheduleoverrunscanthenbecalculatedwithineachincrement,whetherforasingle
iterationorrelease,inthesamewayasinEVM.Effortoraproxyforeffortsuchasstory/feature
pointscanbesubstitutedforcost.
Inagileprojects,customerorbusinessvalueismoreimportantthanearnedvalue.Thereforeearned
valueinitselfmaynotbeverymeaningful.Thecustomervalueorbusinessvalueequivalentof
earnedvaluecanalsobeeasilytrackedprovidedthateachfeaturehasavalueorutilityassignedby
theproductowner.Thebusinessvalueorcustomervaluecanbeexpressedinabsolute,monetary
terms(incurrency)orinrelative,utilityterms(inartificialutils).Thebusinessvalueisanestimate
basedontheproductownersassessment.Thereforeitissubjective,butitcanstillbeinformedand
backedupbyseriouseconomicanalysis.Actualbusinessvalueisstilloftenhardtomeasure,
especiallyatthefeaturelevel.Unlikeinplainearnedvaluecalculation,onlydeliveredfeaturesshould
earntheirrespectivebusinessvalue:eachfeatureearnsitsfullestimatedbusinessvalueonce
delivered.However,ifafeatureisonlyusefulinthefieldwhendeliveredtogetherwithanother
feature,itshouldntearnbusinessvalueuntilallthefeaturesonwhichitdependsarealsodelivered.
Percentweightingstoadjustforafeaturesstatusofcompletenessdontmakemuchsensefor
trackingbusinessvalue.Consequently,individualfeaturesdontearnpartialbusinessvalue.Partial
orpercentagebusinessvalueearnedisonlymeaningfulattheprojectlevel.

ThediagraminFigure3illustratessnapshotsofhow"done"eachfeatureisatvariouspointsintime.
Eachminichartrepresentsapointintime.Theheightofthecoloredinportionofeachfeaturebar
representswhatdegreethatfeatureisdone.Asimplewaytocalculatethisisdividingthenumberof
acceptancetestspassingbythetotalnumberofacceptancetestsforthatfeature.Notethatthenumber
oftestscouldincreaseovertimeastheproductownersunderstandingofthefeatureimprove.

Figure3
Donenesschartsforanagileproject
Notehowagileprojectsfocusonbothmakingthefeaturessmallenoughthattheycanbedoneina
cycleandnottakingontoomanyfeaturesatthesametimetherebyminimizingthelengthoftimethata
particularfeatureisindevelopment.(Thegoalistocompleteeachfeatureinthesameiterationitwas
startedin,oratworstcase,theverynextiteration.)ThisallowstheProductOwnertodoincremental
acceptancetestingaseachfeatureisdelivered.Anybugsfoundcanbescheduledforfixingatthe
appropriatetime(whichmayberightawayorinsubsequentiterations.)
Figure4illustratesa"burndownchart"basedonplottingthenumberoffeatureslefttobe"done"
againsttime.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 112

F
e
a
t
u
r
e
s

l
e
f
t

Time

Figure4
Burndownchartforanagileproject
Thesimplicityoftheburndowncharthelpsquicklygrasptheprogressoftherelease.However,itmay
masksomeotherdynamicsthattakesplaceontheproject.Forexample,when
Forsuchsituations,MichaelCohnintroducedanalternativeburndownchart(Figure4B),inwhichthe
progress,asusually,isshownabovethebaseline,andchangesinscopearereflectedbelowthebaseline.
[CohnOnBurndownChartAlternative]

<TBA>


Figure4B
Alternativeburndownchart
Insteadofhaving100percentofthefeatures50percentdoneatthehalfwaypointoftheproject,agile
projectsstrivetohave50percentofthefeatures100percentdone.ThisgivestheProductOwner
optionsifspecificationanddevelopmentofthefunctionalitytakeslongerthanexpected,whichisnot
uncommon.Theycandecidewhethertoadjust(reduce)theproductscopetodeliverontimeorto
adjust(delay)thedeliverydatetoincludeallfunctionality.Italsomeansthattheworkofreadiness
assessmentandacceptancetestingarespreadoutmoreorlessevenlyacrosstheproject.Thisis
discussedinChapter17PlanningAcceptance.
Figure5illustrateschartsforalessagileproject.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 113


Figure5
Donenesschartsforalessagileproject
Onthisexampleproject,mostfeaturesaretakingseveraliterationstocompleteandacceptancetesting
onlystartsafterallfeaturesaredeemedready.Becausetheyarefoundverylateintheproject,thereis
lesstimetofixdeficienciesfoundduringacceptancetesting(suchasmissedrequirements)sotheyneed
tobefixedmuchmorequicklyortheytendtostayunfixedforthisrelease,withworkarounds,
restrictions,andspecialcases..

7.3.2 CommunicatingPercentDoneonSequentialProjects
Sequential(a.k.a.waterfall)projectshavemoreofachallengebecausethephases/milestones
synchronizedevelopmentinsuchawayastoensurethatallfunctionalityisavailablefortestingat
roughlythesametime.Thispreventsusing"percentoffunctionalityaccepted"asameaningful
predictivemeasureofprogress.Instead,Sequentialprojectsusuallyasksomeonetodeclarewhat
percentageeachfeatureisdone.Forexample,thedevelopermaysaytheyare80percentdonecoding
anddebugging(thoughthisnumberisoftenstuckat80formanyweeksinarow!).Consideringthe
subjectivenatureofestimationtechniques,sequentialprojectsoftenchoosetousetechniquessuchas
"earnedvalue"todeterminea"degreeofdoneness"metric.Unfortunately,thesetechniquesareprone
toerrorandfudging,andtheyarebothdifficultandtimeconsumingtoproduceandmaintain.
Figure6containsatypicalsequenceofdonenesschartsforthisapproach.

Figure6
Donenesschartsforasequentialproject
Inthissequentialversionofthecharts,wecanseehowphased/sequentialdevelopmentencouragesus
toworkinparallelonmanyfeaturesbecauseeachfeatureissynchronizedbygatingmechanismssuchas
themilestonesRequirementsFrozen,DesignComplete,andCodingComplete.Thismeansthatallthe
featuresareavailableforacceptancetestingatroughlythesametimeandmustbefinishedacceptance
testinginaveryshortperiodoftimebecauseitisonthecriticalpathoftheproject.Thishas
implicationsforthestaffinglevelsrequiredforthereadinessassessmentandacceptancetestingroles.
Whendevelopmentislate,theperiodforreadinessassessmentandacceptancetestingisfurther
shortenedandtheresourcesfurtherstressed.Italsohasimplicationsontheimpactoffindingbugs
duringthetestingbecausethefixesareonthecriticalpathtodelivery.
Figure7illustratesa"burndownchart"basedonplottingthenumberoffeatureslefttobe"done"
againsttime.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 114

F
e
a
t
u
r
e
s

l
e
f
t
Time

Figure7
Burndownchartforasequentialproject

Sidebar:WhatittakestodoIncrementalAcceptance
InprojectsthatroutinelyapplyIncrementalAcceptanceTesting;theProductOwner(ortheonsite
customerproxy)isshowntheworkingfeatureassoonasitisdeemedreadybythedevelopers,often
justafewdaysintoadevelopmentiteration.Projectsthatdonotemploythispractice,inparticular
thoseemployingasequentialprocess,encountermanyobstacleswhentheytrytooverlaptestingwith
development.Theproblemismoreseverewhentheoverlapisintroducedtorecovertheschedule
whendevelopmentisrunninglate.Whythedifference?
Thereareanumberoffactorsatplayhere,eachofwhichmaycontributetothesituation.The
problemsincludedifficultycommunicatingwhatistestableandwhatisnot;thisresultsinfrustration
amongsttestersthatthetestingresultsinmanybugstowhichthedeveloperrespondsthatfeature
isntdoneyet;youshouldnotbetestingityet.
Crossfunctionalteamsadvocatedbyagileprocessesaddressthecommunicationproblem.The
productowner,requirementsanalysts,usabilityexperts,developers,andtestersalltouchbasedaily.
Theyparticipateinthesameiterationplanningmeetingsandendofiterationreviews.Thismeansthat
thereismuchbetterawarenessamongsttesters,whethertheybereadinessassessorsoracceptance
testers,ofwhatcanbetestedandwhatshouldbeleftforlater.
Adisciplinedsourcecodemanagementapproachalsohelps.Applyingtestdrivendevelopment
practicesandusingacontinuousintegrationserverthatrebuildstheentirecodebaseafterevery
checkinarepracticesthatsupportdisciplinedsourcecodemanagement.Inanenvironmentusing
thesepractices,thebuildprocessinvolvesrunningalltheautomatedunittests.Anytestfailuresare
treatedasastopthelineeventwherethetopprioritybecomesgettingthebuildworkingagain.This
ensuresastablecodebaseatalltimes;theteamsimplywonttoleratemembersdumpinghalffinished
codeintothebuild.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 115

Anotherformofdisciplineisthedecompositionoflargefeaturesintosmallerfeaturesandeven
smallertaskswhichcanbecompletedinunderaday.Thisallowsworking,testedcodetobechecked
inseveraltimesadayavoidingtheoverheadofbranchingandmerging.
Controllingthenumberoffeaturesinprogressisyetanothermeasurethatsupportsincremental
acceptancetesting.Thispractice,borrowedfromtheworldofleanmanufacturingandleanproduct
design,minimizestheamountofmultitaskingpeopleneedtodotherebyimprovingfocus,flowand
productivity.Averyusefulsideeffectisthatthelengthoftimefromdecidingonwhatisrequiredto
whenthefunctionalityisreadytobeacceptancetestediskepttoaminimum;oftenjustafewdays.
Thisreducesthelikelihoodoftherequirementschangingwhiledevelopmentisinprogressorthe
ProductOwnerforgettingwhatitwastheymeantbythatrequirement.
Highlyincrementalacceptancetestingimpliesthattheproductwillbetestedmanytimes.Thecostof
testingwouldrisewitheveryiterationasmorefunctionalityisadded.Agileteamsavoidthisbyhaving
extensiveautomatedregressiontestsuites(atleastunittestsbutusuallyfunctionaltestsaswell)that
theyrunbeforedeclaringthatthecodeisreadyfortesting.Theseregressiontestsactasalarge
changedetectorwhichinformstheteamwheneverthebehaviouroftheproductchanges
unexpectedly.Forexample,itmaybeanunexpectedsideeffectofafeaturebeingintroduced.
Acceptancetestersaretypicallyinvolvedinpreparationoftheautomatedfunctionaltestssothey
knowwhatisalreadytestedandfocusonareasnotcoveredbytheautomationsuchasuserinterface
behaviour.
Ateamthatisencounteringdifficultyindoingincrementalacceptancetestingmaybeexhibiting
symptomsofmissingoneormoreofthesepractices.Thisisespeciallyprevalentonteamsnewtoagile
andteamsthathavetodealwithlargelegacycodebases.

7.4 Summary
Itmayseemlikeasimpleconcepttodetermineifafeatureorproductisdone,butinpracticeitcan
provetobesomewhatslippery.Wemustknowhowtodetermineifwearedone,whatcriteriatouseto
establishwhetherwearedoneandhowtotestthosecriteria.Itinvolvesdeterminingiftheproper
featuresorfunctionalityispresent,ifitworksasexpectedorrequiredanddoesitmeettheexpectations
oftheProductOwner.Havingaprocessinplacewillincreaseourchanceofsuccessfullydeliveringa
productwearehappywith.

7.5 WhatsNext?
Thisfinishesourintroductiontothemodelswecanuseastoolsforthinkingaboutacceptance.Thenext
fewchaptersdescribeperspectivesonacceptancefromanumberofdifferentplayers.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 116

7.6 References
[BachOnGoodEnoughSoftware]Bach,J.TheChallengeofGoodEnoughSoftware.
http://www.satisfice.com/articles/gooden2.pdf
[YourdonOnGoodEnoughSoftware]Yourdon,E."WhenGoodEnoughSoftwareIsBest,"IEEESoftware,
vol.12,no.3,pp.7981,May1995.
[CabriGriffithsOnAgileEVM]Cabri,A.andGriffiths,M.Earnedvalueandagilereporting,Proc.ofthe
Agile2006Conference,2328July2006,pp.1722.
[DeBonoOnSixThinkingHats]EdwarddeBono,SixThinkingHats,BackBayBooks,1999.
[CohnOnBurndownChartAlternative]MikeCohn,AnAlternativeReleaseBurndownChart,
http://www.mountaingoatsoftware.com/altreleaseburndown,Oct19,2009

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 117

PartIIPerspectivesonAcceptance
Theprocessleadinguptotheacceptancedecisioninvolvesanumberofdifferentplayersandeachhas
theirownperspectiveontheprocess.Thechaptersinthispartdescribeacceptancefromanumberof
theseperspectivesincludingseveralpartiesthatmayplaytheroleofacceptancedecisionmaker
(BusinessLead,ProductManager,EnterpriseArchitect,LegalDepartmentandTestManager)andother
partiesthatmayplaytheroleofreadinessdecisionmaker(UserExperienceSpecialist,Development
Manager,SolutionArchitectandTestManager.)Eachoftheperspectivesthatfollowfocusonaspectsof
acceptancethatareparticularlysalienttotheplayer.Itisworthreadingalltheperspectivesevenifone
seemstofitthereaderlikeaglove.
[Notetoeditors:ThetextinPart2deliberatelyspeaksdirectlytothepersonineachoftheroles.Thatis,
itaddressesthepersonasthoughwewereinconversationwiththem.Itusestheyoupronounon
purposeasopposedtothePart1chapterswhichtypicallyusedwemustorwedo.]

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 118

Chapter8 TheBusinessLeads
Perspective
Anorganizationmaycommissiontheconstructionorcustomizationofsoftwareforitsownuse.The
constructionwouldbedonebyanotherorganizationwhichmaybeaninternaldepartment(suchasthe
InformationTechnologydepartment)oranothercompany(anoutsourceddevelopmentcompany.)In
suchanorganization,theBusinessLeadisthepersonwhobringstotheprojectteamthevisionofwhat
thesoftwareisintendedtodo.TheBusinessLeadmayalsobeknownastheProjectSponsor,Business
Sponsor,BusinessOwner,orBusinessPartner,andactsinthecapacityoftheProductOwner.
SeetheBusinessPersonintheMarketingDivisionpersonainAppendixAOrganizationStereotypesand
ReaderPersonas.

Thedecisiontocommissioncustom(orbespokeasitiscalledinsomelocales)softwareisabusiness
decisionmadebasedontheexpectedbusinessbenefitsoftheinvestmentthereturnoninvestment
(ROI).Thebenefitsmaybeincreasedincomethroughadditionalsalesorservices,costreductionthrough
fewerstaff,moreefficientworkprocessesorreducederrorsessentially,thesebenefitsareintendedto
contributetotheeconomicbottomlineoftheorganization.Sonowthatyouvemadethedecisionto
commissionconstructionofthesoftware,nowwhat?Beforeyoucanrealizethebenefitsyouwillbe
askedtomakethedecisiontoacceptthesoftwareasbuiltorprovidealistofdeficienciesthatmustbe
addressedbeforeyouwillacceptit.Acceptanceisasignificantstageofanycontractualprocess(See
LegalPerspectivesonAcceptance).Howyouapproachthisacceptancedecisioncanhaveabiginfluence
onhowpainfulorpainlesstheprocesswillbe.Itwillalsoimpactyourpostacceptanceactivities
(includingactualusage,operationalsupport,maintenanceetc.)
ThewrongapproachcanbesummarizedwiththefamousquoteIcannotdescribegoodartbutIll
recognizeitwhenIseeit.Takingthisapproachtotheacceptancedecisionissuretoresultinlate
delivery,costoverruns,deliveryofsoftwarethatdoesnotbringvaluetoyourorganizationanda
strainedrelationshipwithyourProductDevelopmentTeambecausethetheywillhaveapoorchanceof
meetingyourneedsiftheydontknowwhattheyare.Whentheydeliverthesoftwareyouwanted,
youllbesuretofindallsortsofissueswithhowitworks.YoullreportthesebugstotheProduct
DevelopmentTeamandexpectmostofthemtobefixedbeforeyouacceptthesoftwareandputitinto
production.Thatwilltakeadditionaltimeandresourcestherebyincreasingcostsanddeferringthe
benefits.Intheworstcase,thevendormaybeunabletofixallthedefectstoyoursatisfactionandyoull
eitherhavetoacceptsoftwareyouarenothappywithorwriteoffthesystementirely.
ToquoteLudwigWittgenstein,oneoftheleadingphilosophersofthe200thcentury,thereisagulf
betweenanorderanditsexecution.Ithastobefilledbytheactofunderstanding
[WittgensteinOnPhilosophicalInvestigations].Tobesuccessful,donotunderestimatethisactof
understanding.SuccessfulprojectstypicallyhaveaclearagreementbetweentheProductOwnerand

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 119

theProductDevelopmentTeamonwhatconstitutesasuccessfuloutcomefortheproject.Thisproject
charterprovidesahighlevelviewofthebusinessleadsexpectationsoftheprojectandhoweachparty
willbehavethroughouttheproject.Italsolaysoutanykeyconstraintsregardingschedule,costandany
otherrelevantfactors.Dontfocusthecharter(orcontract,StatementofWork,etc.)onwhatremedies
youwillhaveiftheProductDevelopmentTeamfailstodeliver;bythentheprojecthasalreadyfailed!
Instead,focusonthebehaviorsthatwillensureprojectsuccess.Allpartiesshouldtreattheprojectasa
miniaturejointventureandstriveforawinwinoutcome.Ifanyofthepartiesistryingtowinatthe
othersexpense,allpartieswilllose.Dysfunctionalrelationshipsleadtodysfunctionalproducts.Seethe
sidebarTargetcostcontractingfordescriptionofanalternativetotraditionalfixedpriceortime&
materialscontracts.RefertoSectionResponsbilitiesoftheProductOwnerinChapter1forasummaryof
theresponsibilities.

Sidebar:TargetCostContracting:fromadversarialcontractstowardspartneringcontracts
LeanSoftwareDevelopmentvisionariesTomandMaryPoppendieckadvocatethepartneringor
relationalapproachtocontracting,inwhichpartiesagreetoworkinacollaborativewaytocarryout
theproject[PoppendiecksOnLean,Ch.9].TargetCostContractsareatypeofincentivescontractsthat
canbeusedtosupportsharingofbothfinancialrisksandrewards.Thishelpstoalignthebestinterest
ofeachpartywiththebestinterestsofthejointventure.ThePoppendiecksencourageforging
synergisticrelationshipacrosscompanyboundaries.Theyenvisionthatthosewhodowillseeatleast
the25percentto30percentcostreductionofthecostofdoingbusinesspredictedbyPeterDruckerin
[DruckerOnManagementChallenges].
TargetCostContractsareanapproachtoidentifyingandconstraininguncertainty.Itsdoneinthe
followingway.Atargetcostfortheprojectisagreedatthebeginningandbothpartiesareproactively
workingonachievingthetargetcost.Uponprojectscompletion,theactualcostismeasuredagainst
theoriginallysettargetcost.Iftheactualcostisgreatthanthetargetcost,thesupplierandthe
customereachbearanagreedelementofthatcostoverrun.Iftheactualcostislowerthanthetarget
cost,thesuppliersharesthesavingswiththecustomer.Theshareofthecostoverrun/underrunmay
beaconstantproportionormayvarydependingonthesizeofthedelta.
Note:TheanalysisofPerry&Barnesrevealsastrongcaseforsettingthesuppliersshareofcost
overrunorunderrunatavaluethatisnotlessthan50%[PerryEtAlOnTargetCostContracts].
- ThekeycharacteristicsoftheTargetCostContractsareasfollows:Targetcostincludesallchanges
- Targetisthejointresponsibilityofbothparties
- Targetcostisclearlycommunicatedtoworkers
- Negotiationsoccuriftargetcostisexceededwithneitherpartybenefiting.
- Thisenticesworkersatallleveltoworkcollaboratively,compromise,andmeetthetarget.
ForadditionalcasestudyofusingTargetCostContractsinsoftwaredevelopment,seethisexperience
report[EckfeldtEtAlOnTargetCostContracts].

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 120

8.1 TestPlanning
Acceptingsoftwaresoundslikeadiscreteeventbutthereismuchtobedoneleadinguptoit.Asa
ProductOwneryouneedtobepreparedtoparticipateinthedevelopmentofyoursoftware.Anyone
whohashadahousebuiltspecificallyforthemknowsthattherearealotofdecisionstobemadeand
inspectionstobedonethroughouttheentireconstructionprocess.Softwareshouldnotbeany
different.Expecttogetasmuchoutofyoursoftwareasyouarepreparedtoputin.Theabsentee
ProductOwnergetstheresultstheydeserve;anunsuitablesystemdesignedforsomeoneelse.

8.1.1 CommunicatingExpectations
Toavoidthisfateyoucertainlymusthaveaplanformakingtheacceptancedecision.Butyoumustalso
planforregularcommunicationwiththeProductDevelopmentTeamaboutthecriteriaforthat
decision.Oneofthemajoracceptancecriteriawillbesuccessfulimplementationoftherequirements
youhavespecified.Theserequirementsmaytakemanydifferentformsincluding:
Functionalspecificationswritteninplainlanguage(e.g.Thesystemshall)
Usecasemodelsanddocuments
Protocolspecificationsforinterfacestoothersystems
Userinterfacemockups,prototypes,cognitivewalkthroughs,orexistingsystemstoemulate.
Descriptionsofbusinessrulesandalgorithms
Theserequirementsdocumentsformanimportantbasisbothforthevendorbuildingthesystemand
youracceptancetestingplans.Butrecognizethatthesedocumentsandmodelswillneveranswerallthe
questionsthevendormayhave.Norwilltheyexhaustivelydescribeallyourexpectations.Infact,you
maynotevenbeawareofsomeofyourexpectationsuntiltheyfailtobesatisfiedbythesoftware.No
amountofupfrontrequirementsplanningwillflushoutallyourexpectationsthereforeitiscrucialto
interactwiththesoftwareearlyandoftentogiveyourselftimetodiscovertheseunrealized
expectationsearlyenoughtoaffordtheProductDevelopmentTeamtimetoimplementthem.
ExpecttheProductDevelopmentTeamtodiscovermanyambiguitiesandunforeseencircumstancesin
whateverrequirementsandexpectationsyouhavedescribedtothem.Plantobeavailabletothe
ProductDevelopmentTeamonaregularbasis,themorefrequentthebetter.Ifyoucannotparticipate
personally,assignsomeonetoactasyourProductOwnerproxyandensurethattheyhavefull
understandingofwhatyouwanttoachieveandaclearmandateonwhattheycandecidethemselves
andwhatneedsyourapproval.Onagileprojects,expecttoprovideatleastonepersonfulltimeforthe
entiredurationoftheprojecttoclarifyrequirementsanddocontinuousincrementalacceptancetesting.

8.1.2 PlanningWhentoTest
AkeypartoftheagreementwiththeProductDevelopmentTeamshouldbethefrequencywithwhich
youaresuppliedworkingsoftwareonwhichyoucanperformacceptancetesting.Inthepuresequential
orclassicwaterfallmodel,youwilllikelyneedtowaituntilneartheendoftheprojectbeforeyou
receiveanysoftware.Thismaybeasinglefinalreleasethatthevendorbelievesiscomplete,high

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 121

qualitycode,ortheremaybeaseriesofreleaseswithnamessuchasAlpha,BetaorReleaseCandidate
thatareprovidedearlyenoughtoallowrevisionofhowthesoftwareworksbasedontheresultsof
acceptancetestingofeachrelease.Inmostcases,thepreliminaryreleasesareeitheroflowerqualityor
containlessfunctionalitythanthefinalrelease.Makeyourincrementalacceptancetestingplans
accordingly;dontexpecteverythingtobeavailableintheearlyreleases.Whatistobeavailableineach
releaseshouldbethesubjectofdiscussionswiththeProductDevelopmentTeam.
Intheagilemodel,youshouldexpecttoreceiveworking,potentiallydeployablesoftwareonaregular
schedulethroughouttheentireprojecttimeline.Youllbeaskedtodecidewhatfunctionalitythe
ProductDevelopmentTeamshouldbuildineachiteration(akasprint)andyoullneedtobepreparedto
providethedetailedacceptancecriteriaforeachfeatureoruserstorytotheProductDevelopment
Teamearlyintheiteration.ThisprocessisknownasAcceptanceTestDrivenDevelopment.Whenthe
softwarefeatureisfinished,youllbeaskedtotryoutthenewlybuiltfunctionalityanddecidewhetherit
workstoyoursatisfactionorrequireschangesbeforeyouwillacceptit.Inthisincrementalacceptance
testingmodelyouwillneedtotestadditionalfunctionalityeachiteration.Youmayalsoneedto
dedicateoneormoreiterationsattheendoftheprojecttothefinalendtoendacceptancetestingand
releaseactivities.
RegardlessoftheprocessmodelyoushouldexpecttheProductDevelopmentTeamtodoadequate
testingaspartoftheirreadinessassessment.Theexpectationshouldbeclearlyunderstoodbyall
partiesandyoushouldplantoprovidetheacceptancecriteriaanddetailedacceptanceteststothe
ProductDevelopmentTeamsothattheycanrunthetestsduringtheirreadinessassessmentactivities.
Youwillalsoneedtohaveplansinplaceforhowyouwillassessthesoftwarewhenitbecomesavailable.
Ataminimum,youwillneedtoknowhowmanypeopleandhowmuchtimewillberequiredtodothe
assessment.Thedetailsofwhatteststheywouldrunmayalsobeincludedorthismaybedetermined
nearertotheactualtesting.

8.1.3 RegressionTesting
Assumingyouwillreceiveanewversionofthesoftwaremorethanonce,andthisisaverysafe
assumptionregardlessofwhetheryouuseasequentialoragileprocessmodel,youllneedtoretestnot
onlythenewlyintroducedorchangedfunctionalitybutalsoallthepreviouslybuiltfunctionality.This
canturnouttobealotofeffort.Onestrategyistochoosetheteststoberunbasedonwhatwas
changedbutthisincreasestheriskofmissingnewlyintroducedbugs.Anotherstrategyistohaveasuite
ofautomatedtests(see
AutomatedTestinginCh.16:PlanningforAcceptance)thatcanberunquicklyaftereverychangetothe
software.Thiscangreatlyreducetheeffortinvolvedinretestingorregressiontesting.Thereforemake
suretoaskyourProductDevelopmentTeamtoincludeautomatedtestsaspartoftheircostingto
ensureyoudonthaveaneverincreasingworkloadasmoreandmorefunctionalityisdelivered.The
automatedtestsarereallyfortheProductDevelopmentTeamsbenefitasithelpsthemensurethat
theyarenotintroducingdefectsintotheexistingsoftware.Also,youmaywanttorequestthe
regressiontestssuitestobeshippedaspartoftheprojectdeliverables.Testsareassetsandtherefore
youshouldplanonkeepingthemcurrentforaslongasyouplantousetheproduct.Havingthe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 122

acceptancetestsinplacemayreducethecostofyourfuturemaintenance/extensionprojects(forwhich
youmaychoosetouseadifferentsupplier).Buttheacceptancetestsmaythemselvesrequiretest
maintenanceorrefactoringasthebehaviorofthesystemundertestevolves.

8.1.4 TestResourcingandOutsourcing
ThetestingyouplantodomaybeinfluencedbyyourknowledgeofwhattestingtheProduct
DevelopmentTeamwillhavedonepriortogivingyouaccesstothesoftwareandbytheskillsand
experienceofthepeopleyouhaveavailabletotest.Whenyoudonthavethenecessaryexpertiseand
youcannotorchoosenottobringitinhouseyoucanelecttouseathirdpartyorganizationtodoyour
acceptancetestingonyourbehalf.Beaware,however,thatveryfewtestoutsourcershaveperfected
mindreading!Youwillneedtoprovidethemwithdetaileddirectionregardingthenatureandobjectives
ofthetestingyouwantthemtodoandtheexpectedbehavior(i.e.requirements)ofthesystemtobe
tested.Youllalsowanttobeveryclearabouttheacceptancecriteriaandwhethertheyaremakingthe
acceptancedecisiononyourbehalforprovidingyouwithdatatosupportyouracceptancedecision.Test
outsourcingiscommonforspecializedtypeoftesting(e.g.securitytesting).Testoutsourcingmayalso
bemandatedbyacertificationbody/regulatoryauthoritythatrequiresanindependent
verification/audittobeperformed.Forexample,alltaxpreparationsoftwarepackagesmayneedtobe
testedforcompliancetothetaxauthoritiesstandardsfortaxreturninformation.

8.1.5 TestEstimation
Estimatingtheamountoftimeandefforttoallocatetoacceptancetestingisdifficultwithoutagood
understandingofthenatureofthefunctionalityinvolvedandthethoroughnessofthereadiness
assessmentthatwillbedonebytheProductDevelopmentTeam.
Acommonstrategyistotimeboxthefinalacceptancetestphaseandguessthenumberofcyclesof
acceptancetestingplusbugfixingthatwillberequired.Thisrequirespastexperiencetocomeupwith
reasonableguesses.Ifindoubt,guesshighandallowforseveralcyclesoftestingtogivethevendortime
tofixbugs.Warning:Thereisalwaysalotofpressuretoguesslowtoallowearlierdeliveryofthe
softwareandthelowercostthatthisimplies.Butguessingtoolowwilltypicallyresultinhavingto
choosebetweenacceptingthesoftwareonschedulewithmanyknowndeficienciesordelayingthe
releasetoallowadditionaltest&fixcyclestobeperformeduntilallimportantdeficienciesarefixedand
retested.ThetestingphaseisoftensqueezedbylatedeliveryofthesoftwarebytheProduct
DevelopmentTeamandthetestersareoftenaskedtomakeupthedelaybyreducingtheelapsedtime
fortesting.
Thealternativestrategyinvolvesincrementalacceptancetesting;acceptancetestingthatisdone
frequentlythroughouttheproject.Thisimprovesthepredictabilityofthedeliverydateintwoways.
First,itfindsbugsearlierallowingthemtobefixedlongbeforethefinalacceptancetestcycle.Second,it
providesyoutheopportunitytolearnhowmuchtestingisreallyrequiredandhowmuchtimeandeffort
itwilltaketodoit.Thislearninghappensearlyenoughintheprojectthatyoucanadjustyourplansfor
lateriterationsandthefinaltestcyclesothattheprojectcanbedeliveredontime.Adjustmentscould

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 123

includeincreasingordecreasingtheplannednumberoftestcycles,addstafforskills,increasingor
decreasingtheamountoftestautomation,contractingathirdpartyforspecializedtestingandsoon.

8.2 WhatDoYouNeedtoTest?
EnsurethatyouhaveaclearunderstandingofwhatwillbetestedbytheProductDevelopmentTeam
andwhatyouareexpectedtotest.Typically,theProductDevelopmentTeamisexpectedtodo
thoroughtestingofallcapabilitiesandqualitycharacteristics(suchasperformance,reliability,usability
etc.)beforehandingoffsystemforacceptancetesting.Ofcourse,thisisbasedontheProduct
DevelopmentTeamsunderstandingofyourexpectationswhichmayvaryfromyoursifthe
communicationislessthanperfectwhichisalmostalwaysthecase.Yourresponsibilityduring
acceptancetestingisensuringthesystemissuitableforyourownpurpose.
Thetestcasesyourunduringacceptancetestingshouldcovereverythingyoucareabout.Thiscan
includebusinessfunctionality,operationsfunctionalityandnonfunctionalcharacteristicsofthesystem.
Eachoftheseareasmayrequiredifferentexpertisetotestandpossiblydifferentstakeholdersignoffs.
Decidingwhowillsignoffbasedonwhattesting,donebywhom,isanimportantpartofacceptancetest
planning.Thebusinessleadmayneedtonegotiatewithotherpartiestodeterminetheacceptance
decisionprocessandthetestingthatprovidestheinformation.

8.2.1 FunctionalTesting
Functionaltestingshouldincludebusinessworkflowteststhatverifythatallreasonablebusiness
scenariosorworkflowsfunctionproperly.Moredetailedusecasetestscanbeutilizedtoverifythat
eachbusinesstransactionorusecaseisimplementedcorrectly.Thehappypathormainsuccess
scenarioofeachusecaseisnotwherebugswilltypicallybefound;theytendtolurkinthedarkcorners
ofthelesscommonlyexercisedscenarios.Businessruletestscanfocusonverifyingthatthebusiness
rulesandalgorithmsareimplementedcorrectly.Applicationswithrichuserinterfacestypicallyrequire
extensivetestingoftheuserinterfaceitself.Thistestingcaneitherbebundledwiththeusecasetestor
youcanhaveseparatetestsessionsdedicatedtoexploratorytestingthebehavioroftheuserinterface.
Thedegreetowhichyouneedtotestshouldbebasedonthelevelofconfidenceyouhaveinyour
ProductDevelopmentTeamandthecomplexityofthedeliverable.Thiscanbebasedonpastexperience
onpreviousprojectsoronwhattheyhavedemonstratedonthisproject.AProductDevelopmentTeam
thathasprovidedfullvisibilityoftheirinternalprocessesandespeciallytheirreadinessassessment
activitiesmayinspiremuchmoreconfidenceinthequalityoftheproduct.Thiscouldchangethefocusof
youracceptancetestingactivitiesfromlookingforobviousmisimplementationsofthebusinesslogicor
rulestohighervalueaddingactivitieslikeusabilitytestingandbusinessworkflowtestingbothofwhich
improvetheproductdesignandthereforethefitnessforpurposeratherthanjustverifyingthequalityof
theconstruction.Inparticular,thepresenceofanextensivesuiteofbusinessreadableautomated
workflow,usecaseandbusinessruleacceptancetestscouldchangetheemphasisfromconducting
extensivemanualtestingofthisfunctionalitytospotchecking.SeeAutomatedExecutionofFunctional
TestsinVolumeIIIforhowtoautomatethesetests.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 124

8.2.2 NonFunctionalTesting
Nonfunctionaltesting(akaextrafunctionaltesting)isoftendonebytheProductDevelopmentTeam
duringthereadinessassessmentorbyaseparatetestorganization.Itshouldincludeverifyinganynon
functionalcharacteristicsofthesystemthataredeemedcriticalforacceptanceaspertheSystem
RequirementsModeldescribedinChaper5,suchas:

Performance(capacityandresponsetime)andstress(behaviorwhenoverloaded)tests,

Securitytesting(enforcementofprivacypolicies,resistancetoattackbyhackers,etc.)

Scalability(theabilitytohandleadditionalusersortransactionloadsinthefuture)

Usability(theefficiency,effectivenessandsatisfactionwithwhichuserscanemploythe
softwaresysteminordertoachievetheirparticulargoals)
andothers.
EnsurethatyouhaveacommonunderstandingwiththeProductDevelopmentTeamastowhatProduct
DevelopmentTeamfunctionaltestingtheywilldoandhowmuchinformationyouexpecttoreceiveas
inputintotheacceptancedecision.
Thepeoplechargedwithoperatingandsustainingthesystem(hereincalledtheoperationsdepartment)
willhaveveryspecificrequirementsthatneedtobeaddressed.Youllneedtobeawareoftheseasyou
workwiththeProductDevelopmentTeamexploringanddefiningtherequirementsforthesystem.Ina
largerorganizationyoumayhaveacounterpartintheoperationsdepartmentwhocanprovidethe
requirementsandactastheacceptancedecisionmakerfromtheoperationsperspective.See
OperationsManagersPerspectiveinPartII.
Ataminimumyoullneedtobeawareofthetimeandresourcescomplyingwiththeserequirementswill
entail.Insomecasesyoullneedtogiveupnewfunctionalityinordertoensurethattheoperational
requirementsaremet.Someexamplesofoperationalrequirementsmightinclude:

Thesystemneedstointegratewiththespecificsystemsmonitoringsoftwareinuseatyour
company.

Dataneedstobemigratedfromapreviousversionofthesystemorfromasystemthisone
replaces.

Thesystemneedstohaveupgrade,startupandshutdownscriptsthatcanbeusedduring
automatedservermaintenancewindows.

Thesystemneedstobebuiltusinganapprovedtechnologystack.Theoperations
departmentmaynothavetheskillstosupportsometechnologies.E.g.A.NETshopmaynot
havetheskillstosupportanapplicationbuiltinJavaorPHP.Sometechnologiesmaybe
incompatiblewithtechnologiesusedatyourcompany.

Theoperationsgroupmayhavespecificwindowsduringwhichtheycannotinstallortest
newversionsofsoftware.Thismayaffectthereleasescheduleyouhavedevisedanditmay

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 125

makeontimedeliveryofsoftwarebytheProductDevelopmentTeamevenmorecriticalas
smallscheduleslipscouldresultinlonginservicedelays.

Therecouldberequirementsforperformingsuchserviceoperationswithoutdegrading
thecurrentsystem

Reliabilityrequirementsdictatethatthesystemmeetstheagreeduponservicelevel
agreements(SLAs)suchasuninterrupteduptime,transactionlatency,.

Thesystemmustprovideselfmonitoring,errordetection,andsystemloggingcapabilities.

Thesystemmustprovideforrobustrecoverability(includingtheabilitytoverifymultiple
failoversandrecoveries),disasterrecovery,databackup&recovery,andallotheraspectsof
systemrecovery.

Afullrangeofdocumentationmustbeprovidedincludingsupportdocumentation,bothpre
andpostrollout,trainingmanuals,andotherreferencedocumentation.

Thesystemmustbemigratedtothenewversionoftheplatform.

Multipleversionsofthesystemmustrunsidebyside.

8.2.3 OtherPartiesInvolvedinAcceptanceTesting
Ifthereareotherpartiesinvolvedintheacceptancedecisionensurethatyouhaveacommon
understandingaboutwhattheywilltestandhowthatfeedsintotheacceptancedecision.Common
partiesincludetheoperationsdepartmentwhowouldhaveasayinwhetherthesystemcanbeinstalled
andmanaged,usabilitypractitionerswhomightneedtoensurethatthesystemcomplieswithany
usability(includingaccessibility)standards,andcorporatesecuritydepartmentswhoneedtoensurethe
systemissecurebeforeitcanbeaccepted.

8.3 WhereWillYouTest?
Akeypartoftestplanningisdecidingwhatenvironmentswillbeusedforwhatkindsofactivitiesand
howthesoftwareismovedbetweenenvironments.

8.3.1 TestEnvironments
Expecttohavemanyactivitieshappeningconcurrentlyasyoudoacceptancetesting.TheProduct
DevelopmentTeamwilllikelybefixingthebugsthatyoureportwhileyoucontinuetolookforfurther
issues.Thisrequiresaseparateenvironment.WhentheProductDevelopmentTeamhasfixedenough
bugstowarrantdeliveringanewversionofthesoftwaretheywillneedtodoreadinessassessment.This
mayrequireyetanotherenvironmentifbugfixingistocontinueinparallel.Ifatestorganizationneeds
totestthesoftwarebeforepassingitontoyouforacceptancetestingtheywillneedtohavetheirown
testenvironmentaswell.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 126

8.3.2 SoftwareDeliveryProcess
Giventhatdifferentgroupswillbeworkinginseparateenvironments,howdoesanewversionofthe
softwaregetinstalledintheenvironment?Forexample,youvefinishedacceptancetestingonrelease
candidate1(RC1)ofthesoftware,reportedanumberofbugs,andtheProductDevelopmentTeamhas
fixedenoughbugstoproposedeliveringanewversion,RC2,toyourtestenvironment.Whatisthe
processforreplacingtheRC1softwareintheacceptancetestingenvironmentwithRC2?Doesthe
ProductDevelopmentTeampushthenewversionwithorwithoutanyadvancenotice?Doesthe
ProductDevelopmentTeamnotifyyouoftheavailabilityofRC2andwaitforyoutoaskforittobe
installedintheacceptancetestingenvironmentwhenyouarereadytoreceiveit?Andhowlongwillthe
acceptancetestingenvironmentbeunavailablewhiletheupgradeisbeinginstalled?

8.4 TestExecution
Whenthesoftwareisdeliveredforacceptancetesting,acceptancetestexecutionmustbegininearnest.
Itisimportanttoknowwhereyoustandatalltimeswithrespecttotheprogressoftestingandyour
currentunderstandingofthesuitabilityofthesoftware.

8.4.1 ProgressMonitoring
Asacceptancetestingisinvariablytimeboxedbyeithercontractsorcustomerexpectations,itis
importanttoensurethatthetestingactivitiesareprogressinginatimefashion.Thenatureofthetest
plandeterminesthenatureofprogressmonitoring.Agileprojectshaveacceptancetestingspreadout
overthedurationoftheprojectandtheresultsoftestingarehighlyvisibleintheformoffeedbackto
theProductDevelopmentteam.Thereisplentyoftimefortheacceptancetesterstolearnfromtheir
mistakesandmisassumptionsintimeforthenextroundofacceptancetesting.Traditionalsequential
projects,ontheotherhand,havealltestinghighlyconcentratedattheendoftheproject.Itiscritical
thatthesetestactivitiesbeverycloselymonitoredtoensurethatanydeviationsfromthetestplanare
quicklyidentifiedandrectified.Thereisntmuchtimeforlearningbetterapproachestotestingso
everyonemustbehighlytrainedbeforetheacceptancetestperiod.Thisapproachtotestingisfraught
withmanydangersandshouldbeavoidedatallcosts.

8.4.2 BugTracking&RecordKeeping
Thenatureoftheproductbeingbuiltandthecontextinwhichitwillbeusedhasalargebearingonthe
natureanddegreeofrecordkeepingforbothtestexecutionandanybugsfound.Itmaybesufficientto
simplyruntestsandtrackanybugsfoundinaninformalmannerasdescribedinthesidebarIncremental
AcceptanceTestingandBugTrackingonAgileProjects.Inotherenvironments,recordsneedtobekept
aboutwhattestswererun,againstwhichversionofthesoftware,when,bywhomandtheactualresults
thatwereobserved.
Ataminimum,anydefectsthatwerefoundduringacceptancetestingneedtoberecordedandtriaged.
Eachbugshouldincludewhichversionofthesoftwareitwasfoundin,whetheritwillprevent
acceptanceofthesoftware(severitylevel)andwhofoundit.Bugsthatmustbefixedbeforethe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 127

softwarewillbeacceptedmustbeclearlycommunicatedtotheProductDevelopmentTeam.Lesssevere
bugsshouldalsoberecordedbutshouldalsobeclearlylabeledasnotgatingacceptance.Keepinmind
thatinmanycontextsanythinglabelnongatingwilllikelynevergetfixedsotherealdecisioniscanyou
livewiththeproductexhibitingthisbug,onnot?

8.4.3 RegressionTesting
Eachtimethesoftwareischanged,whethertoaddnewfunctionalityortofixbugsfoundduring
acceptancetesting,newbugsmayhavebeenintroduced.Therefore,anytestsperformedbeforethe
softwarewaschangedmayneedtobeexecutedagainonthenewversionofthesoftware.Thiscanbe
verytimeconsumingiftheregressiontestsarenotautomated.
SomeProductDevelopmentTeamsautomatetheirunittestsaswellasthefunctionalteststhatyouwill
usetomaketheacceptancedecision.OtherProductDevelopmentTeamsdontknowhowtodothisor
dontfeeljustifiedinincludingthecostofautomatedtestsintheircostestimates.Theymayneedtobe
pushedintodoingthisbyyou,theProductOwner,insistingthatautomatedregressiontestsasjustas
importantasusermanualsandbuiltinhelp.Ifyouplantousethisproductforanylengthoftime,
expecttheproducttoneedtoevolve.Theevolutionwillbemucheasierifthetestsareautomatedand
willsaveyoulargeamountsofeffortandwillhelpavoidhavingthequalitydegradeastheproduct
evolves.
Animportantconsiderationinregressiontestingisthechoiceofdatatobeusedinthetests.Regression
testscanbeexecutedwiththesamesnapshotofproductiondataforgenuinerepeatability.Anapproach
thatmaybemorerepresentativeofareal,evolvingproductionenvironmentistorefreshtheregression
testingdataovertime.Thismoredynamicapproachtoregressiontestingsacrificesrepeatabilityin
returnforthepotentialtofindnewproblems.

8.4.4 MaintainingMultipleVersionsoftheSoftware
Therewillbetimesthatyoumaychoosetomaintainseveralversionsofthesoftwareatthesametime.
Acommonsituationisafterrelease,duringthewarrantyperiod,ifyouarealsodevelopingasubsequent
releaseatthesametime.Anothersituationiswhenyouchoosetosupportmultipleversionofsoftware
inproductionandneedtoapplyanybugfixestoallsupportedversions.Beforewarnedthatthereisan
extracosttomaintainingversionsinparallelasanybugfixesdoneinthesupport/warrantystreamwill
needtobepropagatedintothedevelopmentstreamatsomepointandretestedthere.

Sidebar:IncrementalAcceptanceandBugTrackingonAgileProjectsCaseStudy
ArecentITprojectIworkedonwasprettytypicalofhowbugsaredealtwithonagileprojects.The
developerswouldconsultwiththebusinesslead(actingastheProductOwner)astheydevelopedthe
functionalitytomakesuretheyunderstoodwhatshe(thebusinesslead)wantedbuilt.Theywould
jointlydefinetheaccetancetestsforthefunctionalitybeforeorwhilethefunctionalitywasdeveloped.
Whenthedeveloperhadfinishedwritingthecodeandhadfullyunittestedit,theywouldrunallthe
previouslydefinedacceptancetests.Assumingthetestsallpassed,theywoulddeemtheuserstory
readyforacceptancetestingbythebusinesslead.Theywouldwalkovertheher(the15personteam

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 128

allsatinthesameteamroom)andaskforafewminutesofhertime.Theywoulddemonstratethe
functionalitytoherbyexecutingthemainacceptancetestsinfrontofher.Shewouldtryafewmore
thingsthatoccurredtoher.Ifeverythingworkedasexpectedthusfar,shewouldsaysomethinglike
Thislooksgoodthusfar.Letmedosomemoredetailedtestingandgetbacktoyouwiththeresults
laterintheday.Thedeveloperswouldthengobacktotheirdeskstocatchontheiremailorpickand
startresearchingthenextuserstorywhiletheywaitedforthefinalacceptancedecision.
Ifthebusinessleadfoundsomethingshedidntlikeduringthispreliminarytesting,shewoulddiscuss
theissuewiththedevelopersandtheywouldjointlycomeupwithoptionsanddecidedhowto
proceed.Ifthismeantchangingthesoftware,thedeveloperswouldgobacktotheirdeskstomakethe
changesandthiswholeprocesswouldberepeatwhentheyweredone.
Ifthebusinessleadfoundanybugswhiledoingthemoredetailedacceptancetestingshewouldgo
overtothedevelopersdesktodiscussthem.Theywouldthenagreediscusswhattodo.Thebusiness
leadhadseveralchoices.Somebugswereseriousenoughthatshewantedthemfixedbeforeshe
wouldaccepttheuserstory;thedeveloperswouldaddresstheseproblemsrightawaydropping
anythingelsetheywereworkingontodoso.Thekeyincentivewasthatanyuserstoriesthatwere
almostfinishedbutwhichhadntbeenacceptedbytheProductOwnerwouldntcounttowardsthe
teamsvelocityfortheiteration.(Theteamwasveryproudofitsvelocityandwasalwayslookingfor
waystoimproveit.)
Anotheroptionshehadwastoaddressthebuginanalreadyplanneduserstorythatwouldbe
implementedinafutureiteration.Shewouldchoosethisoptionifthefunctionalitywasotherwise
workingandtheparticulartestcasewasborderlineastowhichstorysscopeittrulybelongedto.
Athirdoptionwastoturnthebugintoafullblownuserstory.Shechosethisoptionifitdidntfitinto
anyoftheotherstoriesalreadyintheproductbacklog.
Thefinaloptionwastotrackthebugonaseparatebuglist.Eachbugwaswrittenona5x7cardand
addedtothebugmanagementsystemwhichwassimplyadesignatedareaofthewhiteboard.
Becausemostbugswerefixedwithinafewhoursofbeingfound,thislistofbugsonthewhiteboard
rarelyexceededadozenorso.Duringourbiweeklyiterationplanningmeetingsthebusinesslead
wouldpickthestoriesfromtheproductbacklogshewantedbuiltandthebugsfromthebuglistshe
wantedaddressed.Sometimesdeveloperswouldsuggestthataparticularbugbeincludedintheplans
becausedevelopmentofauserstorywouldaffectthesamecode.
Thisapproachtobugmanagementworkedverywellforourteambecausewewerecolocated.We
didntneedadatabaseorfancyreportsbecauseweneverhadmorebugsontheboardthantheteam
couldkeeptrackofatanyonetime.Anditensuredthatthequalityoftheproductalwaysmetour
MQRbecausewedidnthaveaplacetohidethebugs.
Thiswhiteboardbasedapproachwontworkonlarge,distributedprojectsbutthephilosophyoffixing
bugsrightawayratherthanlettingthembuilduptounmanageablenumberswill.
GerardMeszaros

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 129

8.5 Summary
TheBusinessLeadisthepersonnominatedbythebusinesstoplaytheroleofProductOwnerforthe
durationofaprojecttodevelopasoftwareintensivesolutiontoabusinessproblem.Theyare
responsibleformakingthedecisionabouthowtobestutilizetheallocatedmoneyorresourcesin
buildingorextendingthesystemfunctionality.
AsummaryoftheresponsibilitiesoftheProductOwnercanbefoundinChapter1TheBasicAcceptance
Process.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 130

Chapter9 ProductManagers
Perspective
AproductmanagerinaproductcompanyplaysaverysimilarroletothatoftheBusinessLeadonanIT
projectwithafewexceptions.Whilethebusinessleadistypicallyengagedonlyforthedurationofthe
projecttodeliverthenewfunctionality,aproductmanageristypicallyresponsiblefortheprofitabilityof
theproductoveritsentirelifetime.Theyaremoreinclinedtothinkintermsofmultiplereleasesofthe
software/product.Theirstrategymayalsobefocusedonaproductline,notindividualproducts,inwhich
casetheintegrationandinteroperabilityamongtheseproductswouldbeofprimaryconcern.Companies
thatbuildproductsarealsomorelikelytohavededicatedtestingresourcesinaseparatetest
department.ManyofthefactorsdiscussedinTheBusinessLeadsPerspectivealsoapplytotheproduct
managersincetheProductManageeralsoplaystheroleofaProductOwner,butinaproductcompany.
Thissectionfocusesonthedifferences.
SeetheProductManagerpersonainAppendixXOrganizationStereotypesandReaderPersonas.

AsaProductManager,youshouldfeeltotalownershipofthesuccessoftheproductinthemarketplace.
Yousurveythemarketforwhatitneedsanddefinetheproducttobebuilt.Youdecidehowmuch
revenuetheproductislikelytogenerateandhowmuchmoneyitisworthinvestingtobuildthe
product.Itisyourjobtoensurethatthesupplierteamunderstandswhatyouwantbuiltandtherelative
prioritiesofthepiecesoffunctionality.

9.1 DefiningDone
Akeypartoftherelationshipbetweentheproductmanagerandtheproductdevelopmentteamis
havingacommonlyunderstooddefinitionofwhatconstitutesdone.Thiscomesintwodimensions:
functionalityandquality.

9.1.1 DefiningMMF
MMFstandsforMinimumMarketableFunctionality(alsoknownasMinimumMarketableProduct
(MMP)orMinimumMarketableFeatures(MMF));itisthesmallestamountoffunctionalitythatyou
coulddelivertoyourcustomerswithoutlosingcredibility.AnythingoverandabovetheMMFaddsvalue
butshouldnotdelaythereleaseifitcannotbecompletedbythedeliverydeadline.Ittakesself
disciplineandagoodunderstandingofthemarkettodefinetheMMF;itissomucheasiertojustthrow
ineverythinganycustomerhaseveryaskedforandcallitareleasebutthisisalmostsuretoresultin
spendingtoomuchandtakingtoolongtodelivertheproduct.
Youmayneedhelptotranslatethehighlevelrequirementsintoadetailedproductdefinition.Business
analysisexpertisemaybeneededtounderstandthebusinessprocessesusedbythepotentialusers.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 131

Productdesignexpertise(e.g.interactiondesignersandgraphicdesigners)maybeneededtohelp
designthefunctionality.Productdevelopmentexpertisemaybeneededtounderstandwhatis
technicallypossibleandconsistentwithhowtheproductalreadyworks.Youmaywanttoinclude
peoplewiththeseskillsaspartoftheproductmanagementteamoryoucanenlistthehelpofother
organizations.Whicheverwayyouchoosetodealwiththeissueyoullneedtoensurethateveryone
understandsyourvisionfortheproductandhowitsatisfiesspecificmarketneeds.

9.1.2 DefiningMQR
Youdontlikereceivingunsatisfactorysoftwareandthesupplierteamhatesbeingtoldthattheir
carefullycraftedsoftwareisnotverygood.Establishingtheminimumqualityrequirement(MQR)isan
importantstepinbuildingatrusting,productiverelationship.AjointlycraftedDoneDoneChecklist
postedinthesupplierorganizationsofficeisagreatwayforthesupplierteamtokeepthefocuson
qualityfrontandcenter.AreleaselevelchecklistasshowninFigure1isagoodwaytosettheoverall
expectationsofwhatthesuppliershouldbeprovidingbeforeacceptancetestingcanstart.
SampleDoneDoneChecklistforRelease1.0
- AllMMFfeaturesareincludedintheRCbuild.
- Asecurityreviewhasbeenconductedandasignoffobtained.
- Testteamisconfidentthatnoneoftheincludedfeatureshasasignificantriskofcausingproblems
intheproductionenvironment,i.e.MQRmet,including:
- configtesting,sidebyside
- performance
- localization
- globalization
- userexperience.
- Businesscomplianceachieved.
- Thereareclear,concisedeploymentandrollbackinstructionsfortheoperationsteam.
- Therearecleartroubleshootingscriptsandknowledgebasearticlesforusebythehelpdesk
representatives.
- Allincludedfeatureshavebeendemoedtoandacceptedbythecustomer.

Figure1
SampleDoneDoneChecklistforaRelease
ThequalitycriteriadescribedinareleaseleveldonedonechecklistasshowninFigure1mayrequire
severalreadinessassessmentcyclesbeforethecriteriaaresatisfied.Toreducethenumberofreadiness
assessmenttest&fixcyclesweneedtoinfluencethelevelofqualitydeliveredbyindividualdevelopers

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 132

tothereadinessassessors.Thiscanbeaffectedbyafeatureleveldonedonechecklistasshownin
Figure2.

Theacceptancecriteriaarespecifiedandagreedupon

Theteamhasatest/setoftests(preferablyautomated)thatprovetheacceptancecriteriaaremet

Thecodetomaketheacceptancetestspassiswritten

Theunittestsandcodearecheckedin

TheCIservercansuccessfullybuildthecodebase

TheacceptancetestspassonthebitstheCIservercreates

Nootheracceptancetestsorunittestsarebroken

Userdocumentationisupdated

Userdocumentationisreviewed

Thefeatureisdemoedtothecustomerproxy

Thecustomerproxysignsoffonthestory

Figure2
SampleDoneDoneChecklistforindividualfeatures
Donedonechecklistssuchasthesearecommonlyusedonagileprojects.

9.1.3 ManagingYourOwnExpectations
Themosteffectivewaytoensurethatpoorqualitysoftwaregetsdeliveredistoaskthesuppliertoover
commit.Softwarerequirementsarenotoriouslyhardtopindownandjustashardtoestimate.Forcinga
suppliertodelivertoanunrealisticscheduleissuretobackfireasyouwilllikelygetalatedeliveryof
poorqualitysoftwarethatyouthenneedtoinvestextratimeintobringingthequalityuptobarely
sufficientlevel.Thisisnotarecipeforsuccess!
Itisfarbettertodefinethelevelofqualityrequiredandmanagethescopeoffunctionalitytoensureit
canallbefinishedinthetimeandmoneyallotted.Timeboxedincrementaldevelopmentisaproven
techniqueforachievingontimedeliveryofqualitysoftware.Itistoyouradvantagetoworkcloselywith
thesupplierorganizationtoselectthefunctionalitytobeimplementedduringeachiteration.Youllhave
muchbettervisibilityofprogresstowardsthereleaseandmuchearlierwarningifthefullslateof
functionalitycannotbecompletedbywhateverdeliverydateyouhavechosen.Thisgivesyoutimeto
prunethefunctionalitytofitintothetimeavailableratherthantryingtocramintoomuchfunctionality
andtherebysacrificingquality.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 133

9.2 WhoMakestheAcceptanceDecision?
Asproductowner,youneedtoberesponsibleformakingthebusinessdecisionaboutwhethertoship
theproductasisortodelaydeliverytoallowmorefunctionalitytobeaddedortoimprovethequality
byfixingknownbugs.Youwilllikelyrelyondatafromotherpartiestomakethisdecisionbutdelegating
thisdecisiontooneofthosepartiesisatyourperilbecausetheyareunlikelytounderstandthebusiness
tradeoffsnearlyaswellasyoudo.
Itisreasonableforyoutoexpectthatthepartiesprovidinginformationaboutthequalityoftherelease
candidatedosointermsthatyouunderstand.Theimpactofeverymissingfeatureorknownbugshould
bedescribedinbusinessterms,nottechnicaljargon.Ideally,thisshouldbeeasilytranslatedinto
monetaryimpact(lostordelayedrevenue)andprobability(likelihoodofoccurrence).

9.3 OperationalRequirements
Mostsoftwareproductshaveoperationalrequirements.Thesearetherequirementsofwhoeverwill
havetoprovisiontheruntimeenvironment,installthesoftware,monitoritsperformance,install
patchesandupgrades,startupandshutdownthesoftwareduringservermaintenancewindows,andso
on.Eventhesimplestshrinkwrapproducthasoperationalrequirements(suchasautomaticupdating)
andcomplexserverproductsoftenhaveveryextensiveoperationalrequirements.Ensurethatyouhave
theappropriatepeopleengagedduringtheproductdefinitionandproductacceptanceprocessto
ensurethattheserequirementsarenotmissed.Makesureyouhavetherightpeopleinvolvedinthe
decisionmakingprocesstoensurethattherequirementsaresatisfied.Operationalrequirementsmay
requirespecializedtestingandtesttools.
SomeexamplesofoperationalrequirementsforSoftwareasaServicearesimilartothoseofanITshop:

Thesystemneedstointegratewiththespecificsystemsmonitoringsoftwareinuseatyour
company.

Thesystemneedstohaveupgrade,startupandshutdownscriptsthatcanbeusedduring
automatedservermaintenancewindows.

Thesystemneedstobebuiltusinganapprovedtechnologystack.Theoperations
departmentmaynothavetheskillstosupportsometechnologies.E.g.a.NETshopmaynot
havetheskillstosupportanapplicationbuiltinJavaorPHP.Sometechnologiesmaybe
incompatiblewithtechnologiesusedatyourcompany.

Theoperationsgroupmayhavespecificwindowsduringwhichtheycannotinstallortest
newversionsofsoftware.Thismayaffectthereleasescheduleyouhavedevisedanditmay
makeontimedeliveryofsoftwarebythesupplierevenmorecriticalassmallscheduleslips
couldresultinlonginservicedelays.

Theoperationalrequirementsofasoftwareproductinclude:

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 134

Thesoftwarewillneedtointegratewithavarietyofsystemsmonitoringsoftware
frameworks.

Thesoftwaremayneedtoworkinavarietyofhardwareplussoftwareconfigurationsfor
bothserverandclient(possiblybrowser)components.Thismayrequireextensive
compatibilitytesting.Itmayalsoconstrainthefeaturestobedeliveredduetocrossplatform
issuesthatrestrictthefunctionalitytoalowestcommondenominator.

9.4 DealingwithLargeProducts
Whenaproductistoolarge,complexorincludestoomanydifferenttechnologies,itcanbedifficultto
commissionasingleteamtodevelopitforyou.Anotherchallengeistheintegrationofseveralexisting
productseachofwhichhasitsownsupplierorganization.Ifyouhavethisissueyouhaveseveralwaysto
dealwithit.Themoreobvioussolutionistobreakdowntheproductrequirementsintoseparate
requirementsforeachsubcomponentandcommissioncomponentteamstodothework.Themain
issuewiththisfromaproductmanagementperspectiveisthatitmakesproductarchitectureandthe
subsequentsubcomponentintegrationyourproblem.SeethesidebarComponentTeamsforamore
detaileddiscussion.
Theotheralternativeistocommissionfeatureteamsthatworkacrosscomponentsanddividethe
productrequirementsintosubsetofproductrequirementsforeachfeatureteam.Thishasthe
advantageofdealingwithanyintegrationissueswithinthefeatureteam.Thesupplierorganizationmay
needtoprovideamechanismtoensureconsistencyofapproachacrossthefeatureteamsuchasa
virtualarchitectureteamorstandardsteam.SeethesidebarFeatureTeamsonMicrosoftOfficefora
moredetaileddiscussion.Regardlessofwhichapproachyouchoose,youllwanttohavesome
mechanismforcoordinatingtheactionsoftheteams.Twocommonapproachesaretheuseofproject
reviewsandtheScrumofScrumsapproach.

Sidebar:ComponentTeams
Alargeclienthadorganizedtheirteamsaroundthecomponentsoftheirarchitecturewhichconsisted
ofmanycomponentsorganizedinseverallayers.ThebottomlayeriscalledThePlatform.Builtatop
ThePlatformaremanyreusablecomponentsandserviceswecansimplyrefertoasA,BandC.These
componentsareusedbythecustomerfacingcomponentsX,YandZ.Whenmarketingreceivesanew
featurerequesttheproductmanagerdecomposestherequestedfunctionalityintorequirementsfor
eachofthecustomerfacingcomponentsX,YandZ.Thearchitectsofthecorrespondingteamsare
eachaskedtodoafeasibilitystudyofthefunctionalityandtoprovideanestimateoftheeffort.This
mayrequirerequestingnewcapabilitiesfromoneormoreofcomponentsA,BorCwhichmayinturn
requirenewcapabilitiesfromThePlatform.Oncethefeasibilitystudywascompletedandanestimate
wasavailablefromeachoftheaffectedteams,marketingwoulddecidewhethertobuildthefeature
andifso,whichreleasetoincludeitin.Intheappropriaterelease,eachoftheteamsinvolvedinthe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 135

designwoulddotheworkrequiredintheirindividualcomponents.Whenallthecomponentswere
finished,bigbangintegrationwouldstart.Thiswouldoftendiscoverseriousmismatchesbetween
expectationsandwhatwasbuilt.
Otherissuesassociatedwiththecomponentteamapproachincludebalancingworkloadamongthe
teamsanddifficultiesimplementingmultiplefeaturessimultaneously.LarmanandVoddestressthat
theseissuesleadtodelaysandincreasedoverheadduetofrequenthandoffs.Analternativeto
componentteamsisorganizingdevelopmentaroundfeatures,discussedinthesidebarFeatureTeams
atMicrosoft.See[VoddeLarmanOnScalingAgile]and[LeffingwellOnFeatureVsComponentTeams]fora
comparisonbetweencomponentandfeatureteams

Sidebar:FeatureTeamsatMicrosoft
Afeatureisanindependentlytestableunitoffunctionalitythatiseitherdirectlyvisibletoacustomer
(customerfacingfeature)orisapieceofinfrastructurethatwillbeconsumedbyanotherfeature.
Featureteamsaresmallinterdisciplinaryteams(512people)thatfocusondeliveringspecificproduct
features.AsamplefeatureteamcouldbetheoneinchargeofRibboninMSWord.Anotheroneisthe
oneinchargeofSpellchecker.AthirdoneisinchargeofAddressLabels.
MicrosoftimplementstheFeatureTeamsstrategyusingamethodologycalledFeatureCrews.It
extendsthegeneralstrategywithspecificguidelinesonhowtomanagethecodebaseissuchawayas
toavoidpartiallyfinishedfeaturesfromaffectingthestabilityofthemainbranch.Theideaisforall
disciplines(development,PM,UxD,testing)toworkcloselytogetheronprivatebuilds(intheirown,
isolatedprivatefeaturebranches)andonlyaddthefeaturetotheproduct(themainbranch)whenitis
FeatureComplete(similarlytoDoneDonedescribedearlier).ThefeatureinFeatureComplete
stateisexpectedtobefullyimplemented,sufficientlytestedandstableenoughforeithera)dog
fooding(usingonesownproductinhouse)andsharingwithcustomersinaCTP(Community
TechnologyPreview)forcustomerfacingfeatures,orb)codingagainstbyanotherfeaturecrewfor
featuresthatdealwithunderlyinginfrastructure.
Thetypicaldurationforfeaturedeliveryisbetweenthreetosixweekstheideaistoensureasteady
flowoffeaturesandregular,frequentfeedbackfromthecustomers.Importantly,eachfeatureteamis
freetodecidewhatdevelopmentapproach/processtouse,aslongastheiroutputmeetsallquality
gates(commonforallfeaturecrews)anddoesntdestabilizetheproduct.(Foracasestudyontheuse
oftheFeatureCrewmethodologyatMicrosoftVisualStudioToolsforOfficeproductunitanddetails
onintegrationoffeaturesintoparentandmainbranches,see[MillerCarterOnLargeScaleAgile])

9.5 BugTrackingandFixing
Usersandtesterswillreportbugsnomatterhowgoodajobyoudobuildingthesoftware.Someof
thesebugswillbelegitimatedefectsintroducedintothesoftwareaccidently.Otherswillberequests
fornewfunctionalityyouconsciouslychosetoleaveoutoftherelease.Exceptfromacontractual(who
pays)perspective,bugsneednotbetreatedanydifferentlyfromfeaturerequests.Eitherway,youneed

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 136

todecidewhattimeframeyouwantthefunctionalityofthesystemchangedandthereforeinwhich
streamofsoftwareyouwanttoincludeit.TheConcernResolutionModelprovidesacommonwayof
thinkingaboutbugs,featurerequestsandotherprojectissues.

9.6 MaintainingMultipleVersionsoftheSoftware
Therewillbetimesthatyoumaychoosetomaintainseveralversionsofthesoftwareatthesametime.
Acommonsituationisafterrelease,duringthewarrantyperiod,ifyouarealsodevelopingasubsequent
releaseatthesametime.Anothersituationiswhenyouchoosetosupportmultipleversionofsoftware
inproductionandneedtoapplyanybugfixestoallsupportedversions.Beforewarnedthatthereisan
extracosttomaintainingversionsinparallelasanybugfixesdoneinthesupport/warrantystreamwill
needtobepropagatedintothedevelopmentstreamatsomepointandretestedthere.

9.7 Summary
TheProductManageristypicallytheProductOwnerofasoftwareintensiveproductandisresponsible
formakingthedecisionabouthowmuchmoneyorresourcestoinvestinbuildingorextendingthe
productfunctionality.
AsummaryoftheresponsibilitiesoftheProductOwnercanbefoundinChapter1TheBasicAcceptance
Process.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 137

Chapter10 TestManagersPerspective
SomeorganizationshaveaseparateTestManagerrolewhileothershavethetestersreportingdirectlyto
theDevelopmentManageroreventheBusinessLeadorProductManager.WhentheTestManagerrole
exists,thetestmanagerisusuallyresponsibleforplanningthebulkofthetestingactivitiesand
managing/coordinatingtheirexecution.Thetestmanagermayormaynotactasthegatekeeperthat
is,maketheacceptancedecisiononbehalfofotherstakeholders.Itisimportantforallpartiesto
understandtheroleoftheTestManagerinthisregard.
SeetheTestManagerpersonainthecompanystereotypecalledAProductCompanyinAppendixX
ReaderPersonasforanexample.

TheTestManagersrolecanbeadifficultonetodowellbecausesomanyprojectfactorsareoutside
yourcontrol.Testplanningisessentialforallbutthemosttrivialprojects;whatthetestplanspecifies
dependheavilyontheroleoftestingasitrelatestotheproductdevelopmentteamandtheproduct
manager.

10.1 TestManagersRoleinAcceptanceDecision
Theroleoftestingseemssimpleenough:usetheproductinvariouswaysandreportanybugsthatyou
find.Butthisisnottheonlyresponsibilityofthetestorganizationinmanyenterprises.Makesureyou
understandwhetheryouandthetestingteamaredoingreadinessassessment,acceptancetestingor
makingtheacceptancedecision.

10.1.1 TestingasAcceptanceDecisionMaker
Insomecircumstances.Thetestorganizationisexpectedtocollectdatatohelptheproductowner
decidewhetherornotthesoftwareisreadytobereleasedtousers.Makesureyouhaveaclear
understandingofwhetheryouarethegatekeeper(acceptancedecisionmaker)oraresupplying
informationtosomeoneelsewhowillbemakingthedecision(preferred.)Recognizethatthe
acceptancedecisionisabusinessdecision.Ifyouaregoingtomakeit,youhadbetterunderstandthe
businessfactorswellenoughtomakeagooddecisionthatyoucanexplaintoeveryone.Otherwise,
youlljustbethebadguyholdinguptheshow.
IfyouaretoactastheAcceptanceDecisionMaker,youmusthaveaccesstoallthebusinessrelevant
informationtomakeagooddecision.Thatis,youmustunderstandthebusinessconsequencesof:

acceptingthesoftwaregiventhenatureofthebugsthatareknowntoexistandthelevelof
confidencethatallseriousbugshavealreadybeenfound,and,

notacceptingthesoftwaretherebydelayingthedelivery.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 138

Eachofthesechoicescanhaveseriousnegativebusinessconsequencesandtheacceptancedecision
cannotbemadewithoutafullunderstandingofthem.TheTestManagershouldnotaccept
responsibilityformakingtheacceptancedecisionlightly.Inmostcasesitisbetterforthisbusiness
decisiontobemadebyabusinessperson,typicallytheproductmanager,businessleadoroperations
managerbasedoninformationprovidedbythetestmanager.

10.1.2 TestingasAcceptanceTesters
Whenyouareprovidinginformationtoanotherpartytomaketheacceptancedecision,makesureyou
havecommonunderstandingofwhatinformationthatparty(orparties)willrequiretomakethe
decision.Ensurethetestersunderstandtheusersforwhomtheyareactingasaproxysotheycandefine
testthatreflectrealistic(notnecessarilyalwaystypical)usageofthesoftware.UserModels(eitheruser
rolesorpersonas)canbeaneffectivewaytogainunderstandingofrealusers.

10.1.3 TestingasReadinessAssessors
IfyouaredoingreadinessassessmenttohelptheDevManagerdecidewhetherthesoftwareisreadyfor
acceptancetestingbythebusinessusers,youllwanttobuildagoodrelationshipwiththedevteamso
thatyoucanhelpthemunderstandthequalityofwhattheyhaveproducedandhowtheycanimprove
it.Embeddingtesterswiththedevelopmentteam(oftenusedinconjunctionwithatechniquecalled
pairtesting)canbeagoodwaytohelpthemlearnhowtobuildqualityinbytestingcontinuouslyrather
thantossinguntestedsoftwareoverthewalltothetestteam.(Kohldescribesbenefitsofpairtestingin
thisexperiencereport[KohlOnPairTesting])

10.2 TestPlanning
Youwillprobablybeexpectedtocoordinatealltestingactivitiesandkeeptrackofthetest
environmentsused,bugslogged,andversioningrelatedtothefixesperformedonthesourcecode,all
thewayfromreadinessassessmentthroughtoacceptancetesting.Thisworktypicallygoesbythename
TestPlanning.Youllneedtocommunicatethisplantoallinterestedparties;especiallythosewhoare
expectedtoexecutepartsoftheplanandthosewhoareinterestedinknowingwhatkindsoftestingwill
bedonebywhom,whereandwhen.ThiscommunicationoftentakestheformofaTestPlandocument
butitcanalsobedoneorcomplementedorallythroughtargetedpresentations,discussionsor
workshops.Manyofthebesttestplanscomeaboutbyinvitingtheinterestedpartiestoparticipatein
thetestplanningprocesseitherinoneononesessionsorviaworkshops.

10.2.1 TestStrategy
Youwilllikelyendupbeingthedefactoowneroftheoverallteststrategy.Theonlyothercontender
wouldbethesupplierorganizationandiftheywanttobeinvolved,probablyonthetestautomation
side,byallmeansencouragetheirinvolvementbecauseitwillmaketestautomationmucheasier.
Theteststrategyincludesthemajordecisionssuchaswhethertestingwillbedoneinafinaltestphase
orincrementally,whethertestingwillbeprimarilyscriptbasedtestingorexploratorytestingora

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 139

combinationofthetwo.Exploratorytestingisaveryeffectivewayoffindingbugsfastbutitisnt
intendedtoberepeatable.Scriptbasedtestingisbetterforensuringaknowndegreeoftestcoverage
andshouldbeautomatedtoallowrapidregressiontestexecution.
Anotherkeydecisionisregardingtheroleanddegreeofuseofautomatedtoolsincludingautomated
executionoffunctionalandparafunctionaltests,theuseofmodelbasedtestcasegeneratorsandthe
useofautomationaspowertools(includingtheuseofautomatedcomparatorsorverifiers)tosupport
manualtestactivities.Someofthesedecisionsrequirelongleadtimestodecidewhethertochoose&
acquireorbuildtoolsortohiretheappropriateresourcesandthereforemustbemaderelativelyearly
intheproject.Thisdoesntimplythatyouwontrefinethemovertimeasmoredetailedinformation
comestolight.
Theteststrategywillbeheavilyinfluencedbythenatureofyourrelationshipwiththeorganization
developingthesoftware.Theidealcaseiswhentheyarepreparedtocollaboratewithyoubydoing
incrementaldeliveryoffunctionalitytosupportincrementalacceptancetesting.Collaborationontest
automationanddesignfortestabilityisalsoakeyfactorinreducingtestexecutiontimeandcost.Ifthis
levelofcollaborationisnotavailableyoumayneedtohiretestautomationengineersandtest
toolsmithswithstrongprogrammingskillstosupportyourtestautomationactivities.

10.2.2 TestAutomation
Anotherareathatgreatlybenefitsfromcollaborationwiththeproductdevelopmentteamistheareaof
testautomation.Akeysuccessfactorisdesignfortestabilityofthesystem.Onlythesupplier
organizationcandothissoyouneedtoworkwiththemtobuilditin.Agoodwaytoincentthemto
collaborateistoprovidethemwithaccesstothe1buttonautomatedexecutionoftheacceptancetests
youdefine.Thisprovidesthesupplierorganizationwithaninstantscorecardthattellsthemhowthey
aredoingatdeliveringqualitysoftware.Everyonehatessurprisesandnothingannoysadevelopment
teammorethanworkinghardtodeliverqualitysoftwareandbeingtoldafterthefactthatitwasnt
goodenough.
Whilethedevelopmentteamneedstomakethesystemamenabletotesting,thetestautomationtools
canbebuiltbyeitherthesupplierorganizationorthetestteam.Thelatterwillrequiretechnicaltesters,
sometimescalledtestautomationengineers.Eitherway,thetestsshouldbedevelopedprimarilybythe
testersorbusinesspeople/analyststoensuretheyreflectrequirementsratherthandesigndecisions.
Testautomationisnotappropriateinallsituations,whereformsofmanualtestingaremoreeffective
oreconomical.SeeSection17.2inChapter17(PlanningforAcceptnace)inPartIIIforadiscussionof
whereautomationisappropriate.Itsthejobofthetestmanagertodeterminethetestautomation
strategy.

10.2.3 AgreeingonExpectations/Requirements
Onepurposeoftestingistoverifythesystemmeetstherequirements.Unfortunately,therequirements
areoftenvagueorambiguousandthismakesverifyingthemadifficultproposition.Onepersonsbug
canbeanotherpersonsfeature.Testersthinkdifferentlyfromanalystsorbusinesspeopleandthisis
bothablessingandacurse.Theyllcomeupwithallmanneroftestscenariosthattherequirement

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 140

specifiersneverimagined.Thisresultsinadefactodivergencebetweentherequirementsandthe
specificationsthatneedtobemanaged.Theproductowner(ProductManager,BusinessLead,etc.)
needstoagreethattheseadditionaltestscenariosareinfactrequirements.
Therequirementsneedtoaddresstheneedsofallstakeholders,notjusttheusers.Operational
requirementsneedtobetested.Ifthetestteamdoesnthavetheskillstoexecutetheparafunctional
teststhenmakesurethattheproductdevelopmentteamexecutesthemandprovidestestplansto
reviewaheadortimeandtestresultsaspartofthehandovertotesting.Theacceptancedecisionmaker
willtoknowthattheparafunctionaltestingwasdoneandmayevenwanttoseetheresults.
Thepreferredsolutionistotreattestscriptsasanextensionoftherequirementsbyusingthemto
illustratehowspecificrequirementsplayoutintheproduct.Thisimpliesthattheproductmanageror
businesslead(ortheirteammembers)needtoagreethatthetestsinterprettherequirementscorrectly.
Ideally,thetestcaseswouldbearticulatedbeforethesoftwareisbuiltsothatthedevelopmentteams
canusethetestcasestounderstandhowthesystemshouldbehave.ThisisknownasAcceptanceTest
DrivenDevelopment.

10.2.4 AgreeingonDone
Arewedoneyet?Thatisaquestionthatneedstobeaskedandansweredcontinuously.Anddone
doesntalwaysmeanthesamething.Onepersonsdefinitionofdonemaybeanotherpersons
definitionofnotevenclosetodone!Itiscriticaltogetagreementonhowdonesoftwareneedstobe
beforeitgoesthrougheachofthequalitygatesofthegatingmodel.Whenissoftwareconsidered
readyforacceptancetesting?Whenisitconsideredacceptable?Theseneedtobeagreedupon
betweenthevariouspartiesinvolved.Thesupplierandtestingneedtoagreeontheminimumquality
requirementofsoftwarebeforeitisreadyfortesting.Thisbecomestheexitcriteriaforthereadiness
acceptancephasethatthesupplierteammustensureismet.AprominentlypostedDoneDone
Checklistisagoodwayforthesupplierteamtokeepthisqualitycriteriafrontandcenter.

10.2.5 EstimatingTestEffort
Estimatingtheamountoftimeitwilltaketogetthroughonecompletecycleoftestingisachallengebut
onethatcanbeovercomewithexperience.Therealchallenge,however,isguessinghowmanytest&fix
cycleswillberequiredbeforethesoftwareisofgoodenoughqualitytoreleasetoacceptancetestingor
tothemarket.Theneedtoguesscanbeavoidedbydeliveringsoftwareincrementallyandtestingeach
incrementassoonasitisavailable.Thishelpsinseveralways:
Itreducestheamountofsoftwarethathasntgonethroughthetest&fixcycleatleastoncewhen
thefinalroundoftestingoccurs.Hereuntestedsoftwarecanbethoughtofasinventoryand
incrementaltestinganddeliveryasinventoryreduction,whereinventoryisconsidereda
formofwaste(see[PoppendiecksOnLean])
Itprovidesdataonhowmanytest&fixcyclesarerequiredforsoftwaredeliveredbythissupplier
organizationearlyenoughtoallowthetestplanstobeadjustedtoensureontimedelivery
ofsoftware.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 141

Itprovidesfeedbacktothesupplierteamonthequalityofthesoftwaretheyaredeliveringearly
enoughtoallowthesupplierteamtimetolearnhowtodeliverbetterqualitysoftwarethat
requiresfewertest&fixcycles.TheDoneDoneChecklistmayneedtobeupdatedbasedon
whatwaslearned.

10.3 ConcernResolution
Testingwillinvariablyresultinanumberofconcernsbeingraised.Someoftheseconcernswillbebased
onfailedtestcasesandothersmaybebasedongeneralimpressionsabouttheproduct.Allconcerns,
however,arebasedonexpectationsanditisimportanttoensurethattheexpectationsarecorrect.Test
casesthatdontmaptoactualrequirementscanresultinbugsthatwillbeclosedasByDesign.Some
failedtestcasesmaypointoutinconsistenciesbetweenhowthesystemoperatesandhowthetester
expectsthesystemtooperate.Thesemaypointtolegitimateusabilityissuesifthetestersmental
modelofthesystemmatchesthatofrealusers.
Partoftheroleofmanytestorganizationsistoactasthesecondopinion,thehouseofsobersecond
thought,aboutthesoftwarebeingbuilt.Itisimportanttobalancethisformofrequirementselaboration
withtheneedtogettheproductoutinatimely,andevenmoreimportantly,predictablefashion.
Discoveryofsuchrequirementsissuesduringafinaltestingphasemakestheproductmanagersjob
harderbecausetheinformationcomestoolatetoaddresswithoutcompromisingtheschedule.
Therefore,thetestmanagershouldstrivetofindsuchissuesasearlyaspossiblesothattheproduct
managercanmakedecisionswithouttheirbacktothewall.
TheConcernResolutionModeldescribesagenericwaytothinkaboutallmannerofconcernsincluding
bugs,requirementsissues,projectissuesandothermismatchesbetweenexpectationsandactualor
perceivedbehaviors.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 142

Chapter11 DevelopmentManagers
Perspective
Thedevelopmentmanagermaygobymanyofficialtitlesbutforourpurposes,thisisthepersoninthe
supplierorganizationwhoisaccountablefordeliveringsoftwaretotheproductowner(PO)whetherthe
POistheinternalbusinesssponsorofanITprojectortheproductmanagerinaproductcompany.Where
aseparatetestorganization(oftencalledQualityAssurance,IndependentVerificationorsomevariation
onthese)existsforthepurposeofdoingacceptancetesting,thedevelopmentmanagerisresponsiblefor
makingthereadinessdecision,thatis,thedecisiontodeliverthesoftwaretowhoeverisresponsiblefor
makingtheacceptancedecision.Forlargerproductsthedevelopmentmanagermaybeassistedbya
solutionarchitect;forsmallerproductsthedevelopmentmanagershouldunderstandthesolution
architectsresponsibilitiesandeithercarrythemoutthemselfordelegatetosomeinthedevelopment
team.
SeetheDevelopmentManagerpersonainthetwocompanystereotypesinAppendixXReader
Personasforanexample.

AstheDevManageryourkeyresponsibilityistomanagethedevelopmentofthesoftwareinsuchaway
thattheproductowner(productmanager,businesslead,etc.)willacceptthesoftware.Youshoulddo
thisinawaythatmaximizesthevalue(orutility)thatthesoftwarewillprovidetotheproductowner(by
meansofvalueprovidedtotheirusersorcustomer)whileminimizingthecost.Thebestwayto
minimizecostistobuildtherightsoftwareandtobuildittherightwaythefirsttimeanddeliveritto
theacceptancetestingorganizationand/orproductownerassoonaspossible.Thisavoidsexpensive
andtimeconsumingtest&fixcyclesinwhichsomeoneelseteststhesoftware,findsbugswhichthey
thenaskyoutofix.Thisdelay(aformofwaste)contributessignificantlytochurnandcost.Whetherthis
istheproductownerorsomeonelookingoutfortheirinterests,thefewerbugstheyfind,theless
reworkyouneedtodo,thelowertheoverallcostandthemorepredictabletheschedulebecomes.

11.1 TheRoleofReadinessAssessment
Mostdevelopmentteamswanttodogoodwork.Theydontwanttodeliversubstandardsoftwareto
theircustomer.Mostcustomerswanttoreceivegoodqualitysoftware.
Deliveringgoodqualitysoftwareiswhatthecustomerexpectsfromthesupplier.Thisisareasonable
expectation,onethataprofessionalsoftwaredevelopmentorganizationshouldbepreparedtomeet.
Deliveringgoodqualityworkingsoftwareisnteasyortrivial;ifitwere,theproductownerprobably
wouldntneedadevelopmentmanager!

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 143

So,whatdoesittaketodelivergoodqualitysoftware,firsttime,everytime?Whatdoesittaketobea
professionalsoftwaredevelopers?Thecompletedetailsofasoundsoftwareengineeringprocessare
beyondthescopeofthisbook.But,therelationshipbetweenthesoftwaredevelopmentorganization
andthepartiesinvolvedinacceptancethesoftwarearenot.Partofthedevelopmentprocessneedsto
beanhonestselfassessmentofthesoftwaretheteamhasproduced.Iftheteamsaysitisready,giveit
tothetestteam.Iftheteamsaysitisntready,askthemwhatstillneedstobedonetomakeitready.A
gooddevelopmentteamwillalwaysbeabletosaysomethingisreadyandshouldbeabletoclearly
articulatetothetestteamwhatisreadyandworthtesting.
RobertC.Martinarguesthatitisirresponsibleforadevelopertoshipasinglelineofcodenotcovered
byatestandtestsmustbekepttothesamelevelofcodequalityastheproductioncode
[MartinOnProfessionalismAndTDD].Althoughthisargumentappliestodifferentdegreesdependingon
whatisbeingbuilt,thedevelopmentmanagerisresponsibleformakingsurethatthecodeisadequately
securedbysolidsuiteoftestsbeforetheproductownerevaluatesit.

11.2 EffectiveReadinessAssessment
11.2.1 WhosJobisQuality?
Howwouldyoufeelaboutbeingthefirstpersontotryabrandnewproductthatnoone,noteventhe
builderhadtriedbefore.Howmanypeopleyouknowwouldbecomfortableinthatsituation?Ifwe
applythesamelogictothesoftwarewebuild,howmanypeopledoyouknowthatwouldwanttobe
thefirstonetotryapieceofsoftwarethatnooneelsehastriedbefore?Yetthisisexactlywhatmany
developmentteamsdowhentheythrowuntestedorpoorlytestedsoftwareoverthewalltothetest
organizationorthecustomer.Manypeoplewouldarguethatthisisjustplainunprofessional.
Allsoftwareshouldbetestedbythedevelopmentorganizationbeforebeinghandedtoanyoneelsefor
testing.

11.2.2 BuildingQualityInStartwiththeEndinMind
Buildingsoftwareshouldnotfeellikeaguessinggamewheredevelopersguesswhatsrequiredand
testersorusersshoutoutwrong!Thedevelopmentorganizationneedstohaveaclearunderstanding
ofwhatsuccesslookslikebeforeitstartsbuildingthesoftware.Partoftheproblemisthatdevelopment
teamsoftendonthaveagoodsenseofhowthesoftwarewillactuallybeused.Developersdonotthink
likeauserbecausemostdevelopersthriveoncomplexitywhilemostusersabhorit.Thismakesit
challengingfordeveloperstodoagoodjobofreadinessassessmentwithoutoutsidehelp.Acceptance
testsprovidedbytheproductownerortestingdonebytesterscanbridgethisgap.Theformeris
proactive,positiveinputthathelpsdevelopmentteamsunderstandwhatdonelookslikebeforethey
buildthesoftware.Thelatterisnegative,reactivefeedbackthattellsthedevelopmentteamyou
haventdoneagoodjob.Whatkindofguidancewouldyouprefer?
Aprojectcharterisonewaytostarttheprocessofdevelopingthiscommonunderstanding.
Requirementsdocuments,usermodels,usecasesanduserstoriesareallwaysthatwetrytodevelopa

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 144

commonunderstandingbetweenthesupplierandthecustomer.Butthesestaticdocumentswrittenin
naturallanguagetypicallycontainmanyambiguousstatementsandinmostcasesdonotprovide
enoughtosufficientlydescribewhatsuccesslookslike.Thedesignsandworkestimatesprovidedbythe
developmentteamthereforelikelycontainassumptions,manyofwhichareeitherwrongorwillresultin
aproductthatismorecomplexandexpensivetobuild.Itiscriticaltoclarifythesepotential
misunderstandingsasquicklyaspossible.
Asthedevelopmentmanageryoushouldworkwiththeproductownerandacceptancetestteamto
ensurethateveryoneontheteamhasaccesstothedefinitionofsuccessintheformofacceptancetests
beforethesoftwareisbuilt.Thisshouldincludeanythingthatisusedasanacceptancecriteriaincluding
bothfunctionalteststhatverifythebehaviorofthesystemfeaturebyfeatureandparafunctional
qualitycriteriasuchassecurity,availability,usability,operationalcriteriaetc.Thisprocessisknownas
AcceptanceTestDrivenDevelopment(akaExampleDrivenDevelopmentorStorytestDriven
Development).Thismayrequirethatproductownerortestersprepareacceptancetestsearlierinthe
projectthantheymighthavetraditionallydone.Itmayalsorequirethemtobemoreinvolvedforthe
durationoftheprojectratherthanjustatthebeginningandtheendsothattheycananswerany
questionsthatcomeupduringtheinterpretationoftherequirementsandalsodoincremental
acceptance.

11.2.3 DefectPreventionbeforeDefectDetection
Traditionalapproachestotestingfocusondefectdetection.Thatis,theemphasisisonfindingbugs
ratherthanpreventingtheiroccurrenceinthefirstplace.Howcantheemphasisbechangedto
prevention?Itisntenoughtodefinethetestsaheadoftime;wemustalsorunthemfrequently.By
doingso,wealwaysknowthescore.Thetestresultstellushowfarfromdoneweare.Theyprovidea
veryclearindicationoftheprogresstowardsdone,onethatdoesntrequirealotofextraworkto
calculateandonethatishardtofudge.Thetestresults,availablethroughouttheproject,providethe
stakeholderswithvisibilityintotheprojectinamuchmoretransparentfashionthantraditionalmetrics
measuringprogressagainstaphasedactivityprojectplan.
Topreventdefectswedefinethetestsbeforebuildingthesoftware,automatethem,andrunthem
frequentlywhilebuildingthesoftware.Weinstitutesimpleteamnormstoavoidregressing:teststhat
havepassedbeforemustcontinuetopassasnewfunctionalityisadded.Noexceptions!Eitherthetest
hastobechanged(changestofunctionalacceptancetestsmayrequirethecustomersagreement)or
thecodehastobechangedtoreturnthetesttopassingstatus.
Whatroledothevariouskindsoftestsplayinthisprocess?Functionaltestsdefinedbytheacceptance
testersdefinewhatdonelookslikeandthesoftwarecannotbedeliveredtothemwhenanyacceptance
testsarefailing.Automatedunitandcomponenttestsdefinethedesignintentofthesoftware;writing
thesefirstandwritingjustenoughcodetomakethempassensureswedontbuildunnecessary
softwareandthatthesoftwarewedowritesatisfiesthedesignintent.Italsoensuresthatthesoftware
youbuildisdesignedfortestability,acriticalsuccessfactorfortestautomation.Anotherbenefitisthat
theyactasalargechangedetectorthatwillinformtheteamofanyunexpectedchangesinbehaviorof
thesoftware.Thishelpstheteamcatchregressionbugsbeforetheycansneakthroughtotheusers.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 145

11.2.4 ReduceUntestedSoftware
InleanthinkingsuchasexemplifiedbytheleanmanufacturingparadigmusedbyToyota
[ShingoOnToyotaProductionSystem],unfinishedwork(inventory)isconsideredaformofwaste.In
softwaredevelopment,softwarethathasbeenwrittenbuthasnotbeenacceptedisunfinishedwork.
Weshouldstrivetofinishthisworkassoonaspossiblebydoingacceptancetestingassoonaspossible
afterthesoftwareiswrittenandreadinessassessmentiscompleted.Thisincrementalacceptance
testingrequirescollaborationbetweendevelopment,testingandtheproductownerasallpartiesmust
bepreparedtoworkfeaturebyfeatureratherthanwaitingforthewholesystemtobeavailablebefore
anyacceptancetestingisstarted.
Theminimumqualityrequirement(MQR)shouldbeagreeduponaheadoftimewiththetesting
organizationortheproductowner.Ideally,anyandallteststhatwillberunaspartoftheacceptance
testingshouldberunbythesupplierorganizationbeforemakingthesoftwareavailableforacceptance
testing.Knowndefectsshouldbeidentifiedaheadoftimetoavoidwastingpeoplestimetestingbroken
functionality.

11.2.5 WhatKindsofTestsAreRequired?
Functionaltestsshouldberunassoonasthecorrespondingfunctionalityisbuilt.Thisshouldinclude
businessworkflowteststhatverifyendtoendbusinessprocesses,usecaseteststhatverifythevarious
scenariosofasingleusecaseorbusinesstransaction,andbusinessruleteststhatensurethatbusiness
rulesareimplementedcorrectly.
Operationalrequirementsalsoneedtobeverifiedbyensuringthatacceptancetestsprovidedbythe
operationsstakeholdersarerunandresultsareanalyzedregularly.
Parafunctionaltestingshouldbedoneonaregularbasisassoonasenoughsoftwareisbuilttoallow
themtoberun.Theearliertheyarerun,themoretimethesupplierhastocorrectanydeficienciesthat
arediscovered.Thisisespeciallyimportantwithparafunctionaltestsbecausechangingthepara
functionalattributesofthesystemmayrequireachangeinarchitecture,apropositionthatmayget
moreexpensivethelaterthechangeismade.
Manyofthesetestscanbedonemuchearlierintheprojectiftheearlyfocusoftheprojectistobuilda
walkingskeletonoftheapplication.Thewalkingskeletonimplementsthefullarchitectureinavery
minimalistway.Forexample,allthelogicmightbehardcodedtherebysupportingonlyasinglehighly
simplifiedbusinesstransactionorworkflow.Butallthemajorarchitecturalcomponentswouldbe
presenttoensuretheruntimecharacteristicsweretrulyrepresentativeofthefinishedproductevenif
thefunctionalbehaviorwasnotimplemented.

11.2.6 SharpeningtheSaw
Anybugsthatarefoundduringacceptancetestingshouldbeasurpriseandshouldpromptthequestion
howdidthatslipthroughourreadinessassessment?Clearly,thesoftwarewasnttrulyready.Ifthe
answerisbecausetheyuseditadifferentwayfromwhatweexpectedthenthesupplierorganization
hastodoabetterjobunderstandingtheusersofthesoftware.Therefore,everybugfoundbecomesa

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 146

learningopportunityfortheteambypromptingthemtolookforwaystoimprovehowtheybuild
software.ForexampleatMircrosoftspatterns&practicesgroup,duringtheWebServiceSoftware
Factory:ModelingEditionproject,theteamdidjustthatonseveraloccasions,modifyingtheirCIsystem
topreventissuesfromreoccurring.
Asmanagerofthedevelopmentorganizationyouneedtocreatetheconditionsinwhichthe
developmentteamcanproducetherightsoftwarebuilttherightway.Thismayrequireovercoming
organizationalandculturalhurdles.Youneedtoworkwithyourcounterpartsinthetestorganizationto
breakdownthetraditionaladversarialrelationship(ifoneexists)andworktogethertocreateamore
collaborativerelationship.AsAdeMiller,patterns&practicesdevelopmentlead,eloquentlyputit:I
thinkoneofthekeythingsaDevManagercandohereistrytosendtheclearmessagethattestingand
thetestorganizationareimportant.Infartoomanycasestestersareseenassomesortoflesser
functionbecausetheylackthetechnicalskillsofdevelopers.Thetestorganizationhastheskillsto
helpyourteambuildbetterqualitysoftware,notjusttotellyouwhenyouhavent.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 147

Chapter12 UserExperienceSpecialists
Perspective
Someorganizations,especiallythosethatbuildconsumerproducts,haveaseparaterolethatis
responsibleforensuringagoodandconsistentuserexperience.Whatistheroleofpeoplewiththiskind
ofskillsetinthemakingoftheacceptancedecision?
SeetheUserExperienceSpecialistpersonainthecompanystereotypecalledAProductCompanyin
AppendixXReaderPersonasforanexample.

Asapersonwiththeresponsibilityofuserexperienceyoumayberesponsiblefordesigningtheuser
experienceassociatedwiththeproduct.Thismayinvolvevariouskindsofactivitiestobetterunderstand
andcharacterizethepotentialusersincludingethnographicresearch,usermodeling,taskmodeling,etc.
Itlikelyalsoincludesverificationactivitiessuchasusabilitytestingatvariouspointsthroughoutthe
designanddevelopmentprocess.

12.1 TheRoleofUserExperienceintheAcceptanceDecision
Itisimportanttohaveaclearunderstandingwiththeproductowner(productmanagerorbusiness
lead)astowhatyourroleshallbeinmakingtheacceptancedecision.Itcouldbeanyof:
1. Beinganinputintotheproductdefinitionprocessbutnodirectinvolvementinthe
acceptancedecision.Thatis,helpingtheproductownerdefinetheproductbutleavingitto
otherpartiestodefinetheacceptancecriteria.
2. Beinganinputintothesoftwaredevelopmentprocesswithnoinvolvementinthe
acceptancedecision.Thisissimilartothefirstpointbutworkingmorecloselywiththe
softwaredevelopmentteamratherthantheproductownership/definitionteam.
3. Asinpoint1butalsoprovidingusabilitytestingdatafortheacceptancedecisionmadeby
theproductowner.
4. Asinpoint1and3butalsohavingasayintheacceptancedecisioneitheraspartofthe
productownersteamorasapartofaseparate,parallelacceptancedecision.

Forproductswhereusabilityisakeydifferentiator,thepreferredrolewouldbeeither3or4.For
productswithcaptiveuserssuchassystemsdevelopedforinternalusersitismorelikelytobe1or2.
Inanyofthesemodelsitisimportanttoworkwiththebusinessleadorproductmanagertodefinethe
usabilitycomponentoftheacceptancecriteriaasearlyaspossibletogivethedevelopmentteamas
muchguidanceaspossible.Earlyintheprojectthismayconsistofdescriptionsofthekindsoftesting

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 148

proceduresthatwillbeusedtoverifyusability;laterintheproject,itmaybespecificfunctionalitytobe
testedsuchasthenoviceuserstop5transactions,specificusabilityguidelinestobecompliedwith,and
soon.
Whereusabilityisimportantandhowtoachieveitisunclear,itiscrucialtoaddresstheusabilityrisksas
earlyaspossible.Someoftheserisksmaybeaddressedthroughtheuseoflowcostusabilitytesting
techniquessuchassketchingandWizardofOztesting,aformoflowfidelityusabilitytestingconducted
withpaperproductsandmockups.Visualcommunicationisverypowerfultopresentandvalidate
ideas.Abilitytogatherfeedbackfromtheuserseffectivelyisasimportantasproductionofthe
prototypes.Figure1presentsasamplelowfielectronicprototypebuiltwithMicrosoftSketchFlow
[SketchFlow]thatauserisabletopreviewinabrowserandproviderapidfeedback.


Figure1.
UseraddingfeedbackonaSketchFlowprototype
Othersituationmayrequiretheconstructionofelectronicprototypesforhigherfidelityusabilitytesting.
Constructionofspecialpurposehighfidelityprototypesmaybeavoidediftheactualsystemcanbebuilt

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 149

anddeliveredincrementallysequencedbyusabilityrisk.Thismayrequireworkingwiththeproduct
ownertohelpthemunderstandtherisksandconsequencesandtousethesetoinfluencetheorderin
whichfunctionalityisbuilt.Usabilityrelatedriskscanbemitigatedbydefininganinternalmilestoneat
whichsufficientfunctionalityisavailabletoallowusabilitytestingoftheactualproductwithrealor
proxyusers.Thisrequirescollaborationbetweentheproductownerandthesupplierorganizationtodo
incrementaldevelopmentanddeliveryofthesoftwarebasedontheusabilitytestschedule.Theareas
withhighestimpactofusabilityissuesshouldbebuiltandtestedfirsttoensureenoughtimeforany
usabilityrelatedissues,whichmaybepervasiveenoughtohaveawiderangingimpact,tobeaddressed
beforethecostofchangehasbecometoohigh.
Wheretheproductownerhasnotspecifiedincrementaldeliveryitmaybepossibletoworkdirectlywith
thedevelopmentmanagertoarrangeforopportunitiestodousabilitytestingusingearlyversionsofthe
actualsoftwaresystem.

Sidebar:MarryingUsabilityPracticeswithIncrementalDevelopmentMethods
AcommonlyheldopinionisthatInteractionDesignandUsabilityTestingneedtobefinishedbefore
softwaredevelopmentcanbestarted.Thisisindirectcontradictionwiththeagileapproachof
incrementaldevelopment.Modernproductdevelopmenttimescalesoftenprecludeaseparatedesign
phasesotheworkofusabilityspecialistsneedstobedonewithinthedevelopmentiterationsasthe
projectunfoldsratherthanbeforetheirstart.
AnumberofcasestudiesdemonstratethatInteractionDesignandagilemethodsdonotneedtobeat
oddswitheachother.In[AstonAndMeszarosOnAgileUsability]JaniceAstonandGerardMeszaros
comparetwophasesofanagilewherelightweightusabilitypracticeswereincorportatedintothe
secondrelease.TheyusedpaperprototypingandWizardofOztestingtovalidatetheuserinterface
designandsawsignificantimprovementsinusabilityanduseracceptance.Thebenefitswereso
significantthatinasubsequentprojectthebusinesspartnersinitiatedtheprototypingandusertesting
activitieswithoutpromptingfromtheITdepartment.
In[MillerOnCustomerInput]LynnMillerconcludes:TheUsabilityEngineeringteamatAliashasbeen
gatheringcustomerinputformanyyears,butneveraseffectivelyaswhenweworkwithanagile
developmentteam.ForSketchBookPro,wewereabletomaximizethequantityandimpactof
customerinputbyhavingtheinteractiondesignersworkinaparallelandhighlyconnectedtrack
alongsideofthedevelopers.Dailyinteractionbetweenthedevelopersandinteractiondesignerswas
essentialtothesuccessofthisprocess.
In[SyOnAgileUCD],DesireeSydescribeshowAliasSoftware(nowAutodesk)adaptedtheirUCD
processtosupportagiledevelopment.Sheconcludes:
*Agileusercentereddesignresultedinbetterdesignedsoftwarethanwaterfallusercentered
design.Agilecommunicationmodesnarrowedthegapbetweengatheringusabilitydataandactingon
it.
*BecauseAgiledevelopmentishighlyfeedbackdriven,productteamsmayrelyonuseropinionin
situationswhereuserobservationismoreappropriate.Usabilitypractitionerscanbethebestsuited

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 150

membersofanAgileteamtomitigatethisbiasbecauseoftheirskillsingatheringandanalyzinguser
experiencedata.
*ItispossibletousethefamiliararsenalofusabilityinvestigationmethodsonAgileprojects,
includingformativeusabilitytesting,userandtaskanalysis,interviews,andevenfieldbasedworklike
contextualinquiry.Thisisachievedbychangingthetimingandgranularityoftheinvestigations,and
howresultsarereported.
*usabilityanddesignactivitiescanbescopedasincrementalminidesigns.Differentvalidationand
elicitationactivitiescanbeblendedwithinsinglesessionsconductedatausabilitylaborinthefield.
DesignactivitiesoccuratleastoneAgilecycleorsprintaheadofthedevelopmentteaminan
InteractionDesignerTrackseparatefromtheDeveloperTrack.Developersreceivevalidateddesigns.
*Prototypedemonstrationsanddailyconversationhavelargelyreplaceddetaileddocumentssuch
asusabilitytestreportsandUIspecificationswhencommunicatingwiththeproductteam.Documents
arenowwrittenforinteractiondesigners,torecordahistoryofdesigndecisions.
In[McInerneyMaurerOnAgileAndUCD],PaulMcInerneyandFrankMaurerdescribethreecasestudies
ofhowuserexperiencespecialistsintegratedwithagileteamsandreportvarioustechniquesfrompair
observationstoinformalusabilitytestingintheteamroomwiththelatestbuildinvolvingboth
developersandcustomerproxies.
Manynotableadvocatesofusabilitypracticesdescribetechniquesforadaptingthemintoagile
methods.JeffPattonhasawebsite[PattonOnAgileProductDesign]andaYahoo!Groupdedicatedto
thetopicofagileusabilityandproductdesign.
In[MeszarosOnAgileProductDesign]GerardMeszaroswritesthatonmostprojectsthereisplentyof
timetounderstandtheusersandtheirgoalsandtodotheoverallproductdesignactivitiescalledfor
byUxDmethodsbeforeiterativedevelopmentstartsbecausethebusinesscaseneedstobemadeand
fundingneedstobesecuredbeforetheprogrammerscanstartdevelopoment.
LarryConstantinehaspointedoutthatevenwhenthisisntthecase,thereisusuallysomebackend
(ortechnicaldevelopment)thatcanbestartedimmediatelythatdoesnotdependontheinteraction
designortheuserinterface[ConstantineOnAgileUsability].)
In[NielsenOnAgileUsability],JakobNielsen(knownasthekingofusability)writes:Therearegood
reasonstobelievethatusabilityandAgiledevelopmentmethodscanworktogetherandimproveuser
experiencequality:
*Agileoffersmanyopportunitiesforovercomingproblemswithtraditionaldevelopmentmethods
thathavelongimpededusability.
*ApproachingAgilenarrowly,asaprogrammingmethodologyratherthanasystemdevelopment
methodology,threatenstodestroythelastdecade'sprogressinintegratingusabilityanddevelopment.
But,asoutlinedabove,therearewaysaroundeachofthesethreats.Solongasteamsrecognizethe
threatsasexplicitissues,theyneednotharmproductquality.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 151

*Finally,weknowfromourresearchthatmanycompanieshavemadethingsworkswimmingly
oncetheyadaptedtheAgilemethodologytosuitqualityfocusedsystemdevelopment.
ForuserexperiencepractitionerswhosupportAgileteams,themainchangeisinmindset.Having
good,generaluserexperienceknowledgewillhelpyouunderstandhowtochangetraditionaldesign
andevaluationmethodstomeetyourAgileteam'sdifferentfocus.Ultimately,however,youmust
bothbelieveinyourselfandembraceAgiledevelopmentconceptsifyouwanttosucceed.

12.2 RecommendationsforSuccess
HerewesummarizealistofrecommendationsbyJeffPatton,anexpertinUserCenteredDesignand
agilemethods[PattonOnAgileProductDesign]andothers:

Helpproductmanagersunderstandandbalancebusinessandusergoals;Useendusertesting
asawaytogivevoicetoenduserneedstobalancethemwithbusinessgoals[SyOnAgileUCD].
Helpproductdevelopmentteamunderstandtheunderlyingproductexperienceanddefine
interactionandvisualdesignpatterns
Dojustenoughuserresearchanddesigntogetstartedbutplanforongoingusercontactfor
researchandvalidation.
Involvethedevelopmentteamintheinteractiondesign;becomeadesignfacilitatorratherthan
purelyabehindcloseddoorsdesigner.
Focusonbigpicture,helptheproductdevelopmentteamsetthecontext
Validateusabilitybeforedevelopmentthroughtheuseoflowfiprototypingandtesting
[AstonMeszarosOnAgileUsability]
Validateusabilityafterdevelopment(throughlightweightusabilitytestingondeveloped
features)[AstonMeszarosOnAgileUsability]
Userparalleltrackdevelopmenttoworkaheadandfollowbehind[MillerOnCustomerInput]and
allowdevelopmenttoproceedonbackendfunctionalityinparallelwithinteractiondesign
[ConstantineOnParallelUx]toavoidseparateinteractiondesignphase.
Increasethefrequencyandtimingofenduserinvolvement:buildareadysupplyofusersand
usersurrogatesinsideandoutsideofyourorganizationtoleverageforcontinuoususertesting.
Considerschedulingregularusertestsessionsanddeferdecidingwhattotestjustintime.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 152

Chapter13 OperationsManagers
Perspective
Someorganizationsbuildorcommissionsoftwarethattheyinstallandruntoprovideaservicetotheir
customersorinternalusers.Intheseorganizationstheacceptabilityofthesoftwaremaybedetermined
bytwoentirelyseparatecommunities,thosethatbenefitfromthefunctionalitythatthesoftware
providesandthoseforwhomoperatingthesoftwareisacost.Intheseorganizationstheacceptance
decisionmustincludeoperationalcriteriaandtheroleoftheoperationsgroupinthemakingofthe
acceptancedecisionmustbeclear.Thiscategorycouldalsobeexpandedtoencompassothergroups
suchassecurity,dataarchitecture,ITarchitectureandinfrastructurethatprovidestewardshipof
corporateintereststocounterbalancepotentialprojectpressureinducedshortcutssuchassecurity,data
architecture,ITarchitectureandinfrastructure.(SeeChapter14EnterpriseArchitectPerspective.)
SeetheOperationsManagerpersonainthecompanystereotypecalledITBuildingSoftwareforthe
BusinessinAppendixXReaderPersonasforanexample.

Asanoperationsmanageryoulikelyhaveacompletelydifferentperspectiveonthedeploymentofa
neworupgradedsoftwaresystem.Otherpartiescareaboutwhetherthenewfunctionalityworks
properlyandwhetheritwillprovidesufficientbusinessbenefitinthenearfuture;youcareaboutwhatit
willdotoyoursupportcostsandwhetheritwillbesustainableinthelongerterm.Therefore,youridea
ofacceptabilitywillbequitedifferentanditshouldinfluencetheacceptancedecision.

13.1 RoleinAcceptanceDecision
Itiscrucialthatyourroleinthemakingoftheacceptancedecisioniscleartoeveryoneconcerned.Do
youprovideinformationtotheacceptancedecisionmakerorareyouoneofseveralacceptancedecision
makerswhooperatebyconsensus,votingorveto?Orperhapsyouparticipateinthereadinessdecision
beforethesoftwareismadeavailableforacceptancetesting?
Aswithotherformsofacceptancecriteriaitishighlydesirableforthesecriteriatobeclearlydefined
beforethesoftwareisbuiltsoastoavoidunnecessarytest&fixcyclesbeforethesoftwarecanbe
deployed.Itisalsodesirabletobeingabletoconductthetestingincrementallythroughouttheproject
ratherthanjustattheend.IncrementalAcceptanceTestingdecreasesprojectrisksbyuncoveringissues
earlyenoughtohavethemfixedbeforethefinalacceptancetestphasewhichisthenmuchmorelikely
topasswithoutuncoveringanyshowstopperproblems.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 153

13.2 OperationalAcceptanceCriteria
Theactualcriteriathatanoperationsgroupwillusetodetermineacceptabilitywillbedependentonthe
natureoftheorganizationandtheservicesitprovidesaswellastheservicelevelagreementforthe
softwareapplicationinquestion.Thebaselevelrequirementstypicallyinclude:

Installeranduninstallertesting

Autoupdateorpatchinstallation

CompatibilitywithyourSystemsmonitoringframework

Performanceunderexpectedloads

Systemsmanagementscriptsforstartup,restartandshutdown.
Additionalacceptancecriteriamayinclude:

Frequentlyaskedquestionsandanswers

Troubleshootingscriptsforthehelpdesk

Failoverandrecoverytestingforhighavailabilityapplications(includingapplicationshutdown,
OSfailure,hardwarefailure,occasionalconnectivityetc.)

Testingofhowsystemhandlesbad(illformated)data

Penetrationtestingforpublicfacingapplications

Stresstestingforapplicationswithpotentiallylargenumbersofusers

Datamigrationfromapreviousversionofthesystemorfromasystemthisonereplaces

Transactionaldatapreservationandarchiving

ITArchitectureapprovalofthetechnologystack.

Simultaneousmaintenance(performingserviceoperationswithconstrainedornoperformance
degradationofthecurrentsystm)Reliability,availability,meetingcertainSLAs(uptimeetc.)

Selfmonitoring,errordetection,systemlogging

Recoverability(mustverifymultiplefailoverandrecovery),disasterrecovery,databackup&
recoveryetc

Supportdocumentation(preandpostrollout),training

Thesystemmustbemigratedtothenewversionoftheplatform

Multipleversionsofthesystemmustrunsidebyside.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 154

13.3 OtherConsiderations
Asanoperationsteam:

Weshouldcontributetotherequirementsdiscussionsasearlyaspossibletoensurethat
realistictargetsarebeingproposedfortheproduct.Weshouldhelptheproductownerandthe
productdevelopmentteamunderstandoperationalcontexts.

Werecognizecriticalityofoursystemundertestandunderstandhowsignificantlyoperational
outagesaffectourbusinessandreputation.

Weunderstandusagepatternsandfluctuationsbasedonthetimeofday,season,special
events.Theprojectedusagewillmostlikelychangeovertime.

Managingtestenvironmentisimportant.Ourtesterswouldlikelyneedtheflexibilitytomake
tuningandconfigurationchanges.Forexample,manyoperabilityscenariosoftenrequiretesters
topurposefullyconfigurethesystemwrongtoobservetheoutcome.

Thechoiceswemakefortestenvironmentwillreflectacostbenefitanalysisbasedonyourrisk
tolerance.

Testlabproceduresarecrucial.Itisimperativetokeepalogofallchangesthataremadetothe
environmentinorderforustobeconfidentthatwhenacceptancetestingisdone,the
configurationandstateoftheproductinthetestenvironmentisalignedwiththeintended
productionconfiguration.

Wecommunicateregularlywiththedeploymentteamandtheproductdevelopmentteam.
OperationalacceptancepracticesarediscussedinVolumeIVofthisGuide.Foraserioustreatmentand
discussionofthepatternsforperformanceandoperability,see[PnPonPerfGuide]and
[FordOnOperabilityPatterns].

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 155

Chapter14 SolutionArchitects
Perspective
Thesolutionarchitectistypicallypartoftheproductdevelopmentteam.Theirroleintheacceptance
processusuallyinvolvesapprovingdesignandtechnologydecisions,designingdatamodelsanddomain
models,andensuringthattheproductasbuiltsatisfiestheneedsoftheendusers.Theroletranscends
differentpartsoftheproductdevelopmentteamsorganizationasthesolutionarchitectisresponsible
forthewholesolutionwhereasindividualcomponentteamsorfeatureteamsareeachonlyresponsible
forapart.

14.1 ResponsibilitiesOverview
Asasolutionarchitect,yourjobistoensuretheentiresolutionsatisfiestheproductownerandthe
potentialusersofthesystem.Youresponsibilitiesinclude:
conductingarchitecturalreviewsandaudits;evaluatingarchitecturalalternatives;
ensuringtheuseofagreeduponarchitectureanddesignpractices;
ensuringthatarchitecturaldocumentsareuseful,uptodate,andsufficientlycomplete;
makingdocumentationtradeoffdecisions;
planningarchitecturalmigration;
andcoordinatingandcommunicatingwitharchitectsandotherstakeholdersregarding
architecturalgovernance,compliance,andpolicymatters.
Yourroleintheacceptanceprocesswillprimarilyfocusonthereadinessassessmentactivitiesbutyou
shouldalsobeinvolvedinhelpingtheproductownerunderstandwhattheywanttheproductowner
teamtobuildandfacilitatethecommunicationofthatinformationtotheproductownerteam.Your
involvementinacceptancetestingmaybeminimalbutyoumaybecalledupontointeractwith
acceptancetesterswhichmayincludeenterprisearchitectsassessingcompliancetoarchitectural
guidelinesandstandards.

14.2 UnderstandingtheSolution
Youneedtoworkwiththeproductownertounderstandtheirvisionoftheproductandtohelpthem
understandwhatoptionsareavailabletothemwithrespecttobuildingthatproduct.Lesstechnical
productownersmayneedyourhelptodefinetheproductbasedonmoregeneralneeds.Youwilllikely
wanttoengageuserexperienceexpertsifgoodusabilityisarequirement.Youwillalsoneedtohelpthe
productownerunderstandtherelativecostsofvariousoptionspresentedtothemsotheycanmakethe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 156

righttradeoffsbasedonreturnoninvestment.Thismayrequireunbundlingofsomefunctionalityinto
smaller,moreatomic,featuresthatcanbeprioritizedseparately.
Beawarethattheproductownermaynotbeawareofalltherequirementsfortheproduct;itmaybe
uptoyoutohelpthemengagetheoperationsmanager,securityarchitects,andotherstakeholdersso
thatalltherequirementsfortheproductbecomeknown.Discoveringsecurityrequirementsor
mandatoryservicelevelagreementslateinaproductdevelopmentcycleissuretoresultinalate
product.
Forlargerproductsyoumaybecalleduptohelpmaketheprojectmanagerorproductownerdecide
whethertodividetheproductownerteamintofeatureteamsorcomponentteamsandtosubdividethe
featurebacklogintofeatureteambacklogsordecomposethefeaturebacklogintocomponentbacklogs.
Whereamixoftechnologiesisinvolved(includingmixedhardware/softwareproducts),youmustbe
preparedtobeinvolvedinnegotiatingtheinterfacesbetweenthevarioustechnologies.
Regardlessofthesizeoftheproductownerteam,itisimportantforeveryonetounderstandtheoverall
solutionandhowtheirpartfitsintoit.Thiscanhelpavoidsituationswhereeachcomponentworks
accordingtospecificationbutwhenintegratedfailtoprovidetheseamlessexperiencetheproduct
ownerwouldexpect.

14.3 EnsuringReadiness
Assolutionarchitect,theonusisonyoutodoeverythingyoucantoensurethattheproductisready
beforethereadinessdecisionneedstobemade.Thingswillgoalotsmootherifeveryoneontheteam
understandswhatdonelookslike.Thisisespeciallytrueforcomponentteamswhoseworkneedstobe
integratedbeforeacceptancetestingcanbedone.Withcomponentteams,youmayneedtodoaform
ofacceptancetestingofthecomponentsfollowedbyintegrationtestingoftheintegratedcomponents
beforeyoucanconductreadinessassessment.Thiswillbeeasiertodoifyouhavebeendoingdesign
reviewsofeachofthecomponentsandhavedefinedclearinterfacesandacceptancecriteria.
Someoftheacceptancecriteriamaybederivedfromenterpriserequirements.Whereenterprise
architectsexist,itisimportanttoengagethemearlyintheproductdevelopmentlifecycletoensureyou
understandwhatstandardstheproductneedstocomplyto,whatreusablecomponentsareavailable
and/orareexpectedtobeusedandwhatenterprisedataimpactstheproduct(andviceversa.)Thiswill
avoidnastysurprisesduringreadinessassessmentandacceptancetesting.

14.4 AssessingReadiness
Assolutionarchitect,youwilllikelybeaskedforyouropinionastowhethertheproductis
readyforacceptancetesting.Youllwanttomakethisrecommendationbasedonharddata
andwillthereforewanttoconductarchitecturereviewsandsecuritycodeinspectionsandto
runstaticcodeanalysisandcoveragetools.Youwillalsowanttounderstandhowwellthe
nonfunctionalcharacteristicsofthesystemcomplywiththerequirements;thatwillrequire
testingofthenonfunctionalcharacteristics.Youmaybeinvolvedintheactualtesting,either

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 157

inpersonorthroughplanninganddelegating,oryoumayreviewresultsprovidedby
someoneelsesuchasthetestorganization.Eitherway,youwillwanttoensurethatthe
systemmeetsanyoperationalrequirements,includingSLAs,inadditiontothefunctional
requirements.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 158

Chapter15 EnterpriseArchitects
Perspective
Theenterprisearchitectisresponsibleforensuringconsistencybetweenproductsemployedinorbuiltby
anenterprise.Theirgoalistoreducetotallifecyclecosts(includingoperatingcosts,technologylicensing
costs,etc.)andtomaximizetheROIofanycommoninfrastructure.Theenterprisearchitectoftenhas
vetopowersoverindividualproductsthatfailtocomplywithenterprisearchitectureprinciples.This
sometimemakesthempartoftheacceptancedecisionwhentheproductisbuiltbyanexternalsupplier
andsometimespartofthereadinessdecisionwhentheproductisbeingbuiltinhousebyanIT
department.

Asanenterprisearchitect,yourjobistoensurethealltheproductsusedbytheenterpriseworkwell
togetherandprovideacceptableperformanceatanacceptablecost.Dependingonthenatureofyour
organizationyoumaybecalledupontoprovidedataforeitherthereadinessdecisionortheacceptance
decision.Insomeorganizationsyoumightevenbeaskedtobeinvolvedinmakingtheacceptance
decision.Theroleyouplaywilldependonwhereintheorganizationyoureportrelativetothesupplier
organization.Inoutsourceddevelopmentsituationsyouaremorelikelytobeinvolvedinacceptance
testingandtheacceptancedecision;forinhousesoftwaredevelopmentandproductcompaniesyou
maybemoreinvolvedinreadinessassessment.Whoyouinteractwithwillalsodependonthenatureof
theorganization.Ifthereisasolutionarchitectforeachproduct,muchofyourinteractionwiththe
supplierteamwillbewiththesolutionarchitect.Inothercasesyoumightbeinteractingprimarilywitha
developmentleadoraprojectmanageroreventheproductowner.

15.1 UnderstandingtheSolution
Whileitisprimarilythesupplierteamsresponsibilitytoworkwiththeproductownertounderstandthe
desiredproduct,youmayneedtobeinvolvedinthesediscussionifthesupplierteamdoesnthavea
solutionarchitect.Otherwise,youwouldlikelyworkwiththesolutionarchitecttoundersandhowthe
requirementsofthespecificproductrelatetotherestoftheenterprisearchitecture.Youwould
determine(alongwithSolutionArchitect)whatformsofintegrationarerequiredandhowthat
integrationshouldbeimplementedfromatechnicalpointofview.Youwouldalsocollaborateto
understandhowthenewproductimpactsanycommoninfrastructuresuchasnetwork,applicationand
databaseservers,etc..Iftheproducthasanyimpactonthelongtermvisionoftheenterprise
architecture,eitherbychangingthevisionorbyimplementinganykeycomponents,thatinformation
needstobecommunicatedtoallpartiesconcernedasthelattermaybecomepartofthereadiness
and/oracceptancecriteria.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 159

15.2 EnsuringReadiness
Insomeorganizationsyouwillbeaskedtoprovidereadinessassessmentinformationtothereadiness
decisionmaker.Youwillneedtoengagethesolutionarchitectordevelopmentleadstoensurethey
understandwhatstandardsorguidelinesareapplicabletotheproductandwhatreusablecomponents
shouldbeused.Youmayalsobeinvolvedindefiningextensionstothecorporatedatamodelsto
supportthenewproduct.Itiscrucialthatyoucommunicateanyexpectationsofhowthesupplierteam
wouldengagetheenterprisearchitectsduringthereadinessassessmentand/oracceptancetesting
(whicheverisapplicable.)

15.3 AssessingReadinessorAcceptability
Thenatureofyourorganizationwillinfluencewhetheryouareformallyinvolvedinreadiness
assessmentoracceptancetestingbutthenatureoftheactivitiesyoudowillprobablybethesame
eitherway.Unfortunately,thelateryouareinvolvedthebiggeranimpactanoncompliancediscovery
willhaveontheoveralldeliveryschedule.Thereforeitispreferabletoengagethesupplierteamsearlier,
beforeevenreadinessassessment,toensurethatthereadinessassessmentoracceptance
testing/reviewsarejustaformalsignoffofinformationthatwaspreviouslydiscoveredratherthanan
unpleasantsurpriseforeveryoneinvolved.
Someoftheactivitiesyoumightwanttostartdoingevenwhiledevelopmentisinprogressinclude
designandarchitecturereviewsthatfocusonstandardscomplianceparafunctionalattributessuchas
security,capacity,scalability,andinfrastructureimpact.Insomecasesyoumayevenbecalleduponto
arrangefortestingofparafunctionalcharacteristicsofthesystemsuchaspenetrationtesting.
Ifyoufindyourselfonanacceptancedecisioncommittee,youwillneedtobepreparedtogatherand
reviewtheresultsoftheparafunctionalreviewsandtestingandcastyourvoteontheacceptabilityof
theproductfromatechnicalperspective.Lettheproductownervotebasedonthefunctionality
providedandtheoperationsmanagerontheirsatisfactionwiththeoperationalrequirements.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 160

Chapter16 Legalperspective
Thedecisiontoacceptsoftwareisoftengovernedbytermsinalegallybindingcontract.Theacceptance
decisionmaythereforehavelegalramificationswhichmayoverrideconsiderationsrelatedtofunctional
andnonfunctionalacceptancecriteria.

Disclaimer:Thischapterisnotmeanttobeusedaslegaladvice.

Sinceacceptanceisamajorlegalmilestoneofcommercialsoftwaresystemdevelopment,itisimportant
tokeepinmindthefollowing:

16.1 AcceptanceMayHaveLegalRamifications
Aspartofthecontractualprocess,acceptancetypicallyresultsinrenderingapaymentaswellasit
affectstheapplicationofanywarrantyprovisionsandpotentiallyanyremediesavailabletothe
customer.Whendefiningthecontract,makesurethattheappropriatefactorsaretakeninto
consideration.Therelationshipbetweentheacceptanceoftheproductintoaproductionenvironment
andthecontractualacceptanceoftheproductascompleteshouldbeclearlyspelledout.
Acceptancecanbeapproachedfromtwopositions:explicitprovisionsspecifiedinthesoftware
developmentcontractandthetermsimpliedintothecontractbylaw(whichalsodependsonthe
geographicjurisdiction).Implicitprovisionsmayalsodifferbasedonwhetherthesoftwareisrendered
asaproductorasaservice.Toseewhatprovisions(explicitandimplicit)doapplytoyourcase,consult
yourlegalcounselor.

16.2 ContractShouldStipulateAcceptanceProcess
Whensoftwaredevelopmentisoutsourced,thecontractshouldclearlystipulatetheacceptanceprocess
andhowtheacceptancecriteriawillbedetermined.Itisimportanttoalsospecifyanyexpectations
abouthowmuchreadinessassessmentwillbeconductedbythesupplierandhowthecriteriafor
readinessassessmentwillbedetermined.Forexample,willthecustomerbeprovidingthesupplierwith
theacceptancetests,arepresentativesampleoftheacceptancetestsorleavingthesupplierto
determinewhatteststorun(theBattleshipapproach.)
Thecontractshouldnotjuststipulatewhenthesuppliermustdeliverthesystembutalsohowlongthe
customercantakeforacceptancetesting.Acceptancetestingisusuallytimeboxed.Thecustomeris
givensomuchtimetoacceptsoftwareandreportdeficiencies,afterwhichthesoftwareisdeemed
completeiftherearenoshowstopperbugs.Thecontractmuststipulatethecriteriausedto
determinewhetherabugisindeedashowstopperorgatingbug.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 161

Thecontractshouldalsostipulatewherethedifferentstepsoftheacceptanceprocesswillbecarried
out.Incaseofcustombuildsoftwaresystem,preliminaryacceptancetestingmaybeperformedatthe
locationofsupplier,whilefinalacceptancetesting(bothfunctionalandoperational)isperformedwhen
thesoftwareisdeployedtothecustomer.Thecontractmaystipulatethat,aspartoftheacceptance
procedure,thesoftwaresystembesuccessfullyrunwithcurrentdatainaliveoperatingenvironmentfor
aperiodoftime,withminimumbugs,beforethesystemiscontractuallyaccepted.
Thecontractshouldnotbeonesided;itshouldencouragebothpartiestogettoamutuallyagreeable
acceptanceasquicklyaspossible.Contractsthatpenalizeonepartyoftenleadtodysfunctionalbehavior
thatresultsinloseloseoutcomes.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 162

PartIIReferences
[AstonMeszarosOnAgileUsability]Aston,JaniceandGerardMeszarosAddingUsabilityTestingtoan
AgileProject,Proc.OfAgile2006.

[CecilOnAgileAndUCD]RichardF.Cecil,"ClashoftheTitans:AgileandUCD"
http://www.uxmatters.com/mt/archives/2006/12/clashofthetitansagileanducd.php,Sep20,2009

[ConstantineOnAgileUsability]Constantine,Larry,LucyLockwood,Processagilityandsoftwareusability:
Towardlightweightusagecentereddesignhttp://www.foruse.com/articles/agiledesign.pdf

[ConstantineOnParallelUx]Constantine,Larry,TutorialonAgileMethodsandUsabilityIntegrating
UsageCenteredDesign(slide19),presentedatSoftwareDevelopmentBestPracticesSeptember2005

[DruckerOnManagementChallenges]Drucker,P.ManagementChallengesforthe21stCentury,Harper
Business,2001,p.33.

[EckfeldtEtAlOnTargetCostContracts]Eckfeldt,B.;Madden,R.;Horowitz,J.SellingAgile:TargetCost
Contracts.InProc.Agile
2005.IEEEComputerSociety:160166,2005.

[FordOnOperabilityPatterns]ChrisFordetal.PatternsforPerformanceandOperability:Buildingand
TestingEnterpriseSoftware,Auerbach:2008.

[KohlOnPairTesting]Kohl.J.PairTesting.BetterSoftwareMagazine.
www.kohl.ca/articles/pairtesting.pdf

[LeffingwellOnFeatureVsComponentTeams]DeanLeffingwell,ScalingSoftwareAgility,TheBlog:Best
PracticesforLargeEnterprises,
http://scalingsoftwareagility.wordpress.com/2009/07/15/organizingagileatscalefeatureteams
versuscomponentteamspart1/

[MartinOnProfessionalismAndTDD]Martin,C.ProfessionalismandTestDrivenDevelopment.IEEE
Software24(3):32362007.

[McInerneyMaurerOnAgileAndUCD]PaulMcInerneyandFrankMaurer."UCDinAgileProjects:Dream
TeamorOddCouple?",ACMInteractions,12(6):1923,2005.

[MeszarosOnAgileProductDesign]Meszaros,GerardConcepttoBackWhatHappensBeforeIteration
Zero,http://concept2backlog.gerardm.com.

[MillerOnCustomerInput]LynnMiller,"CaseStudyofCustomerInputForaSuccessfulProduct,"Proc.
AgileDevelopmentConference(ADC'05):225234,2005.

[MillerCarterOnLargeScaleAgile]Miller,A.andCarter,E.AgileandtheInconceivablyLarge,inProc.
Agile2007,IEEEComputerSociety:304308,2007.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 163

[NielsonOnAgileUsability]Nielson,Jakob,AgileDevelopmentProjectsandUsability,
http://www.useit.com/alertbox/agilemethods.html

[PattonOnAgileProductDesign]JeffPatton,"AgileProductDesign",http://www.agileproductdesign.com,
Oct19,2009

[PerryEtAlOnTargetCostContracts]Perry,J.,Barnes,M.Targetcostcontracts:ananalysisofthe
interplaybetweenfee,target,
shareandprice.InJ.Engineering,ConstructionandArchitecturalManagement,7(2):202208,2000.

[PnPOnPerfGuide]patterns&practices:PerformanceTestingGuidanceforWebApplications,Microsoft
Press:2007.Alsoonline:http://www.codeplex.com/PerfTestingGuide

[PoppendiecksOnLean]Poppendieck,M.,Poppendieck,T.ImplementingLeanSoftware
Development:FromConcepttoCash.AddisonWesley,2007

[ShingoOnToyotaProductionSystem]Shingo,S.AStudyoftheToyotaProductionSystem,Productivity
Press,1989.
[SyOnAgileUCD]Sy,DesireeAdaptingUsabilityInvestigationsforAgileUserCenteredDesignJournalof
UsabilityStudies,Volume2,Issue3,May2007,pp.112132
http://www.upassoc.org/upa_publications/jus/2007may/agileucd.html
[VoddeLarmanOnScalingAgile]Larman,CraigandBasVodde.ScalingLeanandAgileDevelopment.
AddisonWesley,2009.
[WittgensteinOnPhilosophicalInvestigations]Wittgenstein,L.Philosophicalinvestigations,3rdEd.,
Oxford:Blackwell,1967.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 164

PartIIIAcceptingSoftware
PartIintroducedseveralmodelstohelpusthinkabouttheroleofacceptancetestingintheoverall
contextofthesoftwaredevelopmentlifecycle.PartIIdescribedtheacceptanceprocessfromthe
perspectivesofcommonlyoccurringrolesplayedbythepeopleinvolvedinmakingtheacceptanceand
readinessdecisionsontypicalprojects.PartIIIaddressestheprocessofacceptingsoftware,applying
theconceptsintroducedinPartsIandII.InPartIII,weoutlinethemainactivitiesthatleadtothe
acceptancedecisionanddescribestrategiesfortheireffectivemanagement.

Chapter17PlanningforAcceptanceintroducesthepracticesusedbyoneormoreoftheroles
describedinPartItoplantheactivitiesthatwillprovidethedecisionmakerswiththedatathatthey
needtomakeanacceptancedecision.
Chapter18AssessingSoftwareintroducestheacceptancetestingandverificationtechniquesusedin
softwareassessment,withparticularemphasisontestdefinition,executionandmaintenance.It
addressesbothfunctionalandnonfunctionalacceptancetesting.
Chapter19ManagingtheAcceptanceProcessintroducesthepracticesusedtomanageacceptance
testexecutionandacceptancedecisionmakingandaddressesthetradeoffsbetweendifferent
approachestotheseactivities.
Chapter20StreamliningtheAcceptanceProcessdescribestheorganizationoftheacceptanceprocess
tooptimizetheacceptancescheduleandtheresourcesusedintheacceptancedecisionbyfocusingon
threemainstrategies:incrementaldevelopment,wastereductionbyavoidingoverproductionand
concurrentexecutionoftheactivitiesunderlyingsoftwareacceptance.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 165

Chapter17 PlanningforAcceptance
PartIintroducedseveralmodelstohelpusthinkabouttheroleofacceptancetestingintheoverall
contextofthesoftwaredevelopmentlifecycle.Italsointroducedtheabstractrolesofthepeopleinvolved
inmakingtheacceptanceandreadinessdecisions.Thischapterintroducespracticesusedbyoneormore
ofthoserolestoplantheactivitiesthatwillprovidethedataonwhichtheacceptancedecisionmaker(s)
canbasetheirdecision.
Acceptanceisanimportantpartofthelifecycleofaproduct;itisimportantenoughthatitshouldbe
theresultofacarefullythoughtthroughprocess.Thetestplanistheendresultofallthisthinking.Like
mostsuchdocuments,itcanserveanimportantroleincommunicatingtheplansfortesting,butthereal
valueliesinthethinkingthatwentintoproducingit.
Testplanningbuildsontheworkdoneduringprojectchartering,whichdefinestheinitialprojectscope.
Intestplanning,youwedefinethescopeofthetestingthatwillbedone,selecttheteststrategy,and
drilldowntodetailedtestingplansthatdefinewhowilldowhat,when,andwhere.
Mostprojectsprepareatestplanthatlaysoutthefollowing,amongotherthings:
Thescopeoftheacceptanceprocessandthebreakdownintoreadinessassessmentandacceptance
testingphases
Theoverallteststrategyincludingbothmanualandautomatedtesting
Theactivities,testingandotherwise,thatwillbeperformedineachphase(thereadinessphaseand
theacceptancephase)
Theskillsthatarerequiredtoperformtheactivities
Theotherresourcesthatwillbeutilizedtocarryouttheactivities(suchasfacilitiesandequipment)
Thetimeframeandsequence,ifrelevant,inwhicheachactivitywillbeperformed

17.1 DefiningTestObjectives
Weneedtohaveaclearunderstandingofthescopeoftheprojectandthesoftwarebeforewestart
thinkingabouthowwewillacceptthesoftware.Mostorganizationshavesomekindofproject
charteringactivitythatdefinestheproductvisionorscope.Itmayalsoincludeariskassessmentactivity.
Oneformofriskassessmentactivityinvolvesbrainstormingallthepotentiallynegativeeventsthat
couldcausegrieffortheproject.Someofthecommonlyoccurringrisksthatwemayaddressthrough
testingare:
TheProductDevelopmentTeammisinterpretedtheProductOwnersdescriptionresultingina
productthatbehavesdifferentlyfromexpected.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 166

TheProductOwnersdescriptionoftheproductleftoutimportantbehaviorsresultinginmissing
functionality.
TheProductDevelopmentTeamimplementedthebehaviorspoorlyresultinginfrequentcrashesof
theproduct.
Theproductexhibitstherightbehaviorwhenusedbysmallnumbersofusersbutbehaveserratically
orrespondsslowlywhenthenumberofusersortransactionsincreasestoexpectedlevels.
Foreachpossibleeventweclassifyeachofthelikelihoodandtheimpactaslow,medium,orhigh.
Anythingrankedmedium/highorhigh/highneedstobeaddressed.Somerisksmaycauseustochange
thewayweplanourproject,butotherrisksmaycauseustotakeonspecifictestplanningactivities.
Together,thevision/scopeandriskassessmenthelpdrivetheteststrategydefinitionandtestplanning.

17.1.1 TraditionalApproachestoTesting
Traditionally,mostproductsaretestedfromtheusersprespective.Thatis,oncetheproductisfinished
theproductsbehaviorisverifiedviawhateverinterfacestheuserswilltypicallyuse.Often,verylittleif
anytestingisdoneofthepartsofthesystembeforeitisassembledintothefinalproduct.Thisresultsin
adistributionofteststhatlookslikeFigureAInvertedTestPyramid.


FigureA
InvertedTestPyramid

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 167

Thisfigureshowsahighlevelviewofhowvariouskindsoftests(andconsequently,testeffort)are
typicallydistributed(basedontheleveloftestgranularityfromunittocomponent/subsystemto
system/wholeproduct)whentestingisdoneprimarilybytestersusingthesameinterfacesasendusers.
Systemtestingisdoneatthelevelofthewholeproductandcanincludearangeofsystemsizes:
individualsystemsorintegratedworkflowsthroughmultiplesystemsthathavetointeract.Similarly,
componenttestsareusedfortestingpackages,individualexecutablesorsubsystems;whilethe
granularityofunittestingcanbeonlittlethingslikemethods/functionsorbiggerthingslikemodules
andclasses.SeethesidebardTestingTerminologyPitfallsforadiscussionaboutwhatwecallvarious
kindsoftests.
TheproblemwiththeapproachpresentedinFigureA(wheremosteffortisfocusedonsystemtesting
andverylittleonunittesting)isthatitishardtoverifythatthesoftwarebehavesinallpossible
circumstancesbecausemanyofthecircumstancesaretriggeredinsidethesoftware.Evenwhenthisis
notthecase,muchoflogicwithinthesystemiscumbersometoverifyviatheenduserinterface.This
makesverifyingthebehaviorandfindinganybugsveryexpensiveandtimeconsuming.Mostofthe
behaviorcanbeverifiedandmanyofthebugscanbefoundmuchmorecheaplythroughappropriate
unittesting,componenttestingandsubcutaneoussystemtesting.Thesesortsoftestscanandshouldbe
automatedaswillbediscussedinsection17.2.3AutomatedTestingbelow.

Sidebar:TestingTerminologyPitfalls
WhenpreparingFigureA(andFigure4laterinthischapter)westruggledwithwhattocallthetestsin
thetoplayeroftheupsidedownpyramid.Someofthealternativesweconsideredwere:Acceptance
Tests,WholeProductTests,CustomerTests,FunctionalTests,andIntegrationTests.Whilethese
namesareincommonusetodescribetheteststhatoccupythetoplayer,wewerenthappywithany
ofthesenamesbecausetheydidntclearlyconveythedifferencebetweenthetoplayerandthelayers
belowit.Theproblemwitheachofthesehistoricalnamesisthattheyoftenexisttohelpdifferentiate
onekindoftestsfromanotherkind.Forexample,FunctionalTestsasopposedtowhat?Theobvious
answerisNonfunctionalTests,butwefeelnonfunctionaltestsalsofitintothetoplayerofthe
(inverted)pyramid.IfnotAcceptanceTeststhenwhat?ThelogicalalternativeisnotRejectionTests
butRegressionTests!Acceptancetestshelpeusverifynewfunctionaltyworkswhileregressiontests
verifythatexisitingfunctionalitystillworks.Theacceptancetestsforthisreleasearelikelyaddedto
theregressiontestsuiteforthenextrelease.Sothetermsacceptanceandregressionrefertopointsof
differentiationonthetimehorizon.
Examiningthepotentialnameswiththiscriticallenshelpeduseliminatemanypotentialnamesleaving
usonlywiththosenamesthathelpedusfocusonwhetherweweretestingthefullyassembled
productoritsconstituentparts.ThisledustoWholeProductTest,SystemTestandIntegrationTest.
Unfortunately,thetermIntegrationTestcanbeusedatvariouslevelsdependingonwhetherweare
integratingunits,components,subsystemsorsystems(intoasystemofsystems.)Wholeproductis
probablytheclearestnamebutmanyteamsthinkofthemselvesasbuildingsolutiostobusiness
problemsratherthanproductsforsale.ThisledustofocusonSystemTestsandthecontrastwith
ComponentTestsandUnitTests.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 168

Therearresimilarissu
uesintheterm
minologyaroundhowtesttsareprepareedandexecuted(Explorattory
vsScripttedtesting)andhowtestssareautomatted(Handscrriptedorproggrammedvs.record/playb back,
scriptedvs.datadrivvenvs.keywoorddriven,etcc.)Themoralofthissidebaristhatweneedtobecaareful
tomakeesureweund derstandwhatsomeonem meanswhenth heyuseoneo oftheseworddstodescribee
testing.Theymaywe ellmeansomeethingentirelydifferentfrromwhatwewouldhavemeanthadw we
usedtheesameword..

17.1.2 Flipp
pingthePy
yramidRig
ghtSideU
Up
Theupsid
dedowntestp
pyramidcanbeturnedrigghtsideupbyymovingmucchofthetestingactivityto
othe
unitandccomponentle
evel.Butwhaatkindsoftesstingcanbep
pushdown?
Theagilesoftwaredevvelopmentcommunityhassshownthatitispossiblettoproduceco onsistentlyhigh
qualityso
oftwarewitho outsignificanttlyincreasingtheeffortbyyintegratingttestingthrougghoutthe
developm mentlifecycle
e.Thishasled
dtoarethinkiingoftheroleeoftesting(ttheactivity)andoftestteaams.
BrianMarrick,hasdefin
nedamodel[MarickOnTeestingDimensions](seeFiggure1)thath
helpsus
understan
ndthepurpossebehindtheedifferentkin
ndsoftestsw
wecouldexecute.


Figure1
PurposeoofTests
oftestingwecandoalongtwokeydimensions:
ThediagraaminFigure1classifiesvaarioustypeso
Whettherthetestsarebusinessfacingortecchnologyfacin
ng
Whettherthetestsareintended development(byhelpingth
dtosupportd hemgetitrigght)ortocritiique
thheproductaftteritisbuilt

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 169

17.1.3 TestsThatSupportDevelopment
TestscansupportdevelopmentbyhelpingtheProductDevelopmentTeamunderstandwhatthe
productissupposedtodobeforetheybuildit.Thesearetheteststhatcanbepreparedinadvanceand
runaswebuildtheproduct.Aspartofthereadinessassessment,theProductDevelopmentTeamcan
runtheseteststoselfassesswhethertheproductimplementsthenecessaryfunctionality.
Thetestsinthiscolumnfallintotwocategories:thebusinessfacingteststhatdescribewhattheproduct
shoulddointermsunderstandablebythebusinessorProductOwnerandthetechnologyfacingtests
thatdescribehowthesoftwareshouldworkbeneaththecovers.

BusinessFacingTests
Thebusinessfacingteststhatdrivedevelopmentarethesystemaccepatancetests(alsoknownas
customertests).Thesetestselaborateontherequirementsandtheveryactofwritingthesetestscan
exposemissingorambiguousrequirements.Whenwe(ProductOwner,oftentogetherwithsomeone
fromtheProductDevelopmentTeam)preparethesetestsbeforedevelopmentstarts,wecanbesure
thattheProductDevelopmentTeamunderstandswhattheyneedtobuild.ThisisknownasAcceptance
TestDrivenDevelopment.
Ifwepreparethesystemacceptancetestsafterdevelopmentiscompleteorwepreparetheminparallel
withdevelopmentanddonotsharethem(thisisreferredtoas"independentverification"),thetestsdo
nothelpusbuildtherightproduct;instead,thetestsactasanalternativeinterpretationofthe
requirements.Iftheyfailwhenwefinallyrunthem,weneedtodeterminewhichinterpretationofthe
requirementsismoreaccurate:theoneimplementedbythedevelopmentteaminthecodebaseorthe
oneimplementedinthefunctionaltestsbythetestteam.Ifthelatter,timewillbeconsumedwhilethe
developmentteamreworksthecodetosatisfythisinterpretation,reworkthatcouldhavebeenavoided
ifwehadsharedthetests.Eitherway,wehavesetupanadversarialrelationshipbetweendevelopment
andtesting.Itishighlypreferabletopreparethetestsbeforethesoftwareisbuiltsothattestingcan
helpdevelopmentunderstandwhatneedstobebuiltratherthansimplycriticizewhattheyhavebuilt.
Thesetestsmayberunmanuallyortheymaybeautomated.ThelatterallowstheProductDevelopment
Teamtorunthemthroughoutthedevelopmentcycletoensurethatallspecifiedfunctionalityis
correctlyimplemented.TheProductOwnerwillwanttorunadditionalacceptanceteststomakethe
finalacceptancedecision,butsupplyingasetofteststotheProductDevelopmentTeamearlysothey
candrivedevelopmentgoesalongwaytowardbuildingthecorrectproduct.Thisismuchmorelikelyto
happenwhenthetestsareeasyandcheaptorunandthatrequiresautomatedexecution(formore
information,seetheAutomatedFunctionalTestExecutionthumbnail).Thesetestsmaybeimplemented
asprogrammatictests,buttheyaremoretypicallyimplementedusingKeyworddrivenTestAutomation.

TechnologyFacingTests
Therearemanytestsusedbyproductdevelopmentthatarenotbusinessfacing.Developersmay
prepareunitteststoverifythatthecodetheywrotehassuccessfullyachievedthedesignintent.Thisis
howtheydeterminethattheycorrectlybuiltthecode(asopposedtobuildingthecorrectproduct).
Testdrivendevelopment(TDD)iswhendevelopersimplementautomatedunittestsbeforetheybuild

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 170

thecodethetestsverify.Thisdevelopmentprocesshasbeenshowntosignificantlyimprovethequality
ofthesoftwareinseveralwaysincludingbettersoftwarestructure,reducedsoftwarecomplexity,and
fewerdefectsfoundduringacceptancetesting[JeffriesMelnikOnTDDEffectiveness].Thesetestsareever
morefrequentlyautomatedusingmembersofthexUnittestingframework.Formoreinformation,see
[MeszarosOnUnitTestPatterns]

17.1.4 TestsThatCritiquetheProduct
Assumingthattheproducthasimplementedthecorrectfunctionality,weneedtoknowwhetherthe
productmeetsthenonfunctionalrequirements.Thesetestssupporttheacceptancedecision.Wedo
thisbyassessingthenonfunctionalattributesoftheproductafterithasbeen(atleastpartially)built.
Thesetestscritiquetheproductinsteadofdrivingthedevelopmentprocess.Theytelluswhetheritis
goodenoughfromanonfunctionalperspective.Wecandividethesetestsintothetwocategories:
businessfacingandtechnologyfacing.

TechnologyFacingTests
Technologyfacingteststhatcritiquetheproductmeasurehowwelltheproductmeetstechnically
orientedqualityattributes(scalability,availability,selfconsistency,etc.)
Thesetestsprovidemetricswecanusewhendecidingwhethertheproductisreadytobeshipped.In
mostcases,thesetestswillberunaspartofreadinessassessmentbecauseoftheirtechnicalnature.
However,aProductOwnerchargedwithdecidingwhethertoacceptaproductmaybeinterestedin
seeingtheresultsandcomparingthemwiththeminimumrequirement.Theymayevenhireathird
partytestlabtoconductthetestingontheirbehalf.Formoreinformation,seetheTestOutsourcing
thumbnail.

BusinessFacingTests
Ifsystemacceptancetestsareusedtodrivedevelopmenttobuildtheproductpertherequirements,
howdowemakesurewearebuildingtherightproduct?Thebusinessfacingteststhatcritiquethe
productfulfillthisrole.Thesetestsassesstheproduct(eitherasbuiltorasproposed)forfitnessfor
purpose.Usabilitytestsareexamplesofteststhatcritiquetheproductfromabusinessperspective
Typically,thesetestscannotbeautomatedbecausetheyarehighlysubjectiveandsomeevenrequireus
toobservepeopletryingtousetheproducttoachievetheirgoals.

17.2 WhatTestingWillWeDo?AndWhy?
Nowthatwehavebeenintroducedtoawayofreasoningaboutthekindsoftests,wecandecidethe
typesoftestsweneedtorunandwhichteststoautomate.Thisisanoverallteststrategythathelpsus
determinehowtobestaddressourtestingneedsatthelowestcost.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 171

17.2.1 TestStrategy
Definingtheteststrategymaybeconsideredtobepartofthetestplanningprocessoradistinctactivity.
Eitherway,thepurposeofdefiningateststrategyistomakesomehighleveldecisionsaboutwhat
kindsoftestingneedtobedoneandhowtheywillbeexecuted.Teststrategyistightlylinkedtothetest
objectives.Forexample,youwouldhaveadifferentteststrategyifyourobjectivesaretoverifyand
acceptfunctionalityasopposedtoverifycompliance.
Oneofthekeydecisionsiswhatkindsoftestsshouldbeautomatedandwhichapproachtotesting
shouldbeusedformanualtests.Thegoalofthesedecisionsistotrytominimizeprojectriskwhilealso
minimizingthetimeandeffortspenttestingthesoftware.
Previoussectionsintroducedtheconceptsoffunctionalandnonfunctionalrequirements.Aspartofthe
teststrategyweneedtodecidewheretofocus.Clearly,testingcannotprovethatsoftwareworks
correctly;itcanonlyprovethatitdoesnotworkcorrectly.Therefore,wecouldspendaninfiniteamount
oftimetestingandstillnotprovethesoftwareisperfect.Theteststrategyisaboutmaximizingthe
returnoninvestment(ROI)oftestingbyidentifyingthetestingactivitiesthatwillmitigatetherisksmost
effectively.Thisimpliesthatsomerequirementsmaybetestedlessthoroughly,bychoice.
Wealsoneedtodecidewhetherwewilldoalltheacceptancetestingattheendoftheproject(ina
separatetestingphase,orinatestlastorbigbangtestingstyle)orincrementallyasfunctionality
becomesavailable.Thelatterapproach,incrementalacceptancerequireschangesinhowtheprojectis
plannedandhowthesoftwareisdevelopedtoensureacontinuousstreamoffunctionalityisdelivered
startingfairlyearlyintheproject.Thisisthesytlethatwestronglyadvocatesincebigbangtesting
carriestoomanyrisksandisavoidableinmanycontexts.Thepaybackofincrementalacceptanceisearly
discoveryofmisunderstoodandmissedrequirements,therebyallowingtimeforremediationoffthe
criticalpathoftheproject.
Anotherstrategicdecisionmayrelatetotestoracles;oursourceoftruth.Howwillwedefinewhata
correctoutcomelookslike?Isthereacomparablesystemthatwecanuseasanoracle?(Formore
information,seeComparableSystemTestOracle.)Canwehandcraftexpectedresults?(Formore
information,seeHandCraftedTestOracle.)OrwillweneedtouseaHumanTestOracle?Ifso,whatcan
wedofromadesignfortestabilityperspectivetoreducethedependencyonhumantestoracles?

17.2.2 ManualTestingFreedom
Forfunctionaltesting,thekeystrategydecisionsrelatetohowwewillexecutethetests.Whenwe
manuallyexecutethetests,weneedtodecidehowmuchfreedomtograntthetesters.Figure2
containsthe"freedom"scaleusedbyJonandJamesBachtodescribethechoiceswehave.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 172


Figure2
"Freedom"scalefortesters(JonBach,JamesBachusedwithpermission)
Atoneextremeofthetestfreedomscale,thereisfreestyleexploratorytesting,inwhichthetesterscan
testwhatevertheythinkisimportant.Attheotherendofthescaleisscriptedtesting,inwhichtesters
attempttofollowawelldefinedtestscript.Inbetween,thereischarteredexploratorytesting,which
haschartersofvaryingdegreesoffreedomincludingscenarios,userroles/personas,andcharters.
Scriptedtestinginvolveshavinganexpertpreparedetailedtestscriptstobeexecutedmuchlaterby
someoneelse(oracomputerwhenautomated).Thereisverylittleopportunityfortestdesignduring
testexecution.
Exploratorytestingisaneffectiveandefficientapproachtotestingthatleveragestheintelligenceofthe
testertomaximizethebugsfoundinafixedamountoftime.Unlikewithscriptedtesting,testersare
encouragedtoconceivenewthingstotrywhiletheyareexecutingtests.Somepeopledescribeitas
concurrenttestcasedesignandexecutionwithanemphasisonlearning.Basedonasurveyof
companiesusingexploratorytesting,ItkonenandRautiainen[ItkonenRautiainenOnExploratoryTesting,
[ItkonenEtAlOnDefectDetection]reportseveralreasonsforapplyingthisapproach:
whentestingaspectsthatrequiretheusersperspective;
whenwritingsufficientlycomprehensiveautomatedtestsareimpractical;
forleveragingthetestersexperienceandcreativitytodiscoverunusualbehavior;
whenfastfeedbackonproductsbehaviorisrequiredintheabsenceofscriptedtests;
whenrequirementsandfeatureschangerapidly;and
tolearnabouttheproductsbehaviorwhentherequirementsarevague,incomplete,orpossibly
misunderstood.
Formoreinformationonexploratorytesting,see[KanerOnExploratoryTesting]and
[BachOnExploratoryTesting].

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 173

17.2.3 AutomatedTesting
Automatedtestingcoversawiderangeoftopics:fromautomatedexecutiontoautomatedtestcase
generation.Somekindsofnonfunctionaltestsrequireautomatedexecutionbecauseofthenatureof
thetestingbeingperformed.Acommonlyoverlookedareaforautomationistheuseof"powertools"
whileperformingmanualtesting.Toolscanalsobeusedtogeneratetestdataortododatainspection.
Thevarioususesoftestautomationneedtobedeterminedonaprojectbyprojectbasis.Formore
informationaboutthisprocess,seethePlanningTestAutomationthumbnailofthisguideand
[FewsterGrahamOnSoftwareTestAutomation].

MaximizingAutomationROI
AneffectivetestautomationstrategystrivestomaximizetheROIoftheinvestmentinautomation.
Therefore,thetestsweautomateshouldcostless,atleastinthelongrun,thanwewouldhavespent
manuallyexecutingthecomparabletests.Sometestsaresoexpensivetoautomatethatwewillnever
recouptheinvestment.Thesetestsshouldberunmanually.

AutomatingtheRightTests
ToensurethatwegetthebestpossibleROIforourtestautomationinvestment,weneedtofocusour
energiesonthefollowing:
Teststhathavetobeautomatedbytheirverynature
Teststhatareinherentlyeasiertoexecuteusingacomputerthanahuman
Teststhatneedtoberunmanytimes
Tasks(nottests)thatcanmakemanual(orautomated)testingfasterandmoreeffective

17.2.4 AutomatedExecutionofFunctionalTests
Automatedfunctionaltestexecutionisapowerfulwaytogetrapidfeedbackonthequalityofthe
softwareweproduce.Whenusedcorrectly,itcanactuallypreventdefectsfrombeingbuiltintothe
product;whenusedincorrectly,itcanrapidlyturnintoablackholeintowhichtimeandeffortare
sucked.Whenautomatedregressiontestsarerunfrequently,suchasbeforeeverycodecheckin,they
canpreventnewdefectsfrombeinginsertedintotheproductduringenhancementormaintenance
activities.EnsuringtheProductDevelopmentTeamunderstandstheacceptancetestsaheadoftimecan
ensuretheProductDevelopmentTeambuildsthecorrectproductthefirsttimeinsteadofasaresultof
testandfixcycles.Forinformationabouthowthisworks,seetheAcceptanceTestDrivenDevelopment
thumbnailinVolumeII.
Acommonstrategyonprojectsthathaveanextensivesuiteofautomatedtestsistorunthesetestsfirst
asaformofregressiontestasthefirstactivityinatestcycle;thisisaformofextendedsmoketest.This
ensuresthatthesoftwarefunctionsproperly(totheextentoftheautomatedtestcoverage)beforea
humantesterspendsanytimedoingmanualtesting.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 174

Thekeytoeffectiveautomatedfunctionaltestingistouseanappropriatetoolforeachtypeoftest
onesizedoesnotfitall.Thetwomostcommonapproachestoautomatedtestpreparationaretest
recording(seetheRecordedTestthumbnail)andtestscripting.Recordedtestsareeasytoproduce,but
theyareoftenhardtomaintain.Scriptedtestscaneitherbeprogrammatictestautomation,which
involvestechnicalpeoplewritingcodetotestthecode,orkeyworddriventestautomation,whichnon
technicalpeoplecanusetowritetestsusingamuchmoreconstrainedtestingvocabulary.Because
keyworddriventestsaretypicallywrittenintheubiquitouslanguagedefinedfortheproduct,theyare
alsomucheasiertounderstandthanmostprogrammatictests.Whateverapproachweuse,weshould
thinkbeyondtheinitialtestauthoringandalsoconsiderthelifecyclecostsofthetests.Recordedtest
toolsdohavesomevaluableuses.Theycanbeusedtoquicklyrecordthrowawaytestsuitestosupport
thedevelopmentteamwhiletheyrefactortestabilityintotheproductundertest.Theycanalsobeused
inarecordandrefactorstyleasawayofquicklybuildingupacollectionofkeywordsortestutility
methodstobeusedinkeyworddriventestsorprogrammatictests,respectively.
Keyworddriventestinginvolvesspecifyingtestscriptsinanonprogrammaticstyle.Thestepsofthetest
aredatainterpretedbyakeywordlanguageinterpreter.Anotherstyleofdatadriventestautomationis
thereuseofatestscriptwithmultipledatasets.Thisisparticularlyeffectivewhenwecangeneratetest
data,includinginputsandexpectedoutputs,usingacomparablesystemtestoracle.Thenwerunthe
datadriventestonetimeforeachsetofinputs/outputs.Commercialrecordedtesttoolstypically
providesupportforthisstyleoftestingandoftenincludeminimalsupportforrefactoringofthe
recordedtestscriptsintoparameterizedscriptsbyreplacingtheconstantvaluesfromtherecordedtest
withvariablesorplaceholderstobereplacedbyvaluesfromthedatafile.

TestAutomationPyramid
Thetestautomationpyramidisagoodwaytovisualizetheimpactofdifferentapproachestotest
automation.Whentestautomationisanafterthought,thebestwecanusuallydoistousegraphical
userinterface(GUI)basedtestautomationtoolstodrivetheproductundertest.Thisresultsina
distributionoftestsasshowninFigureAInvertedTestPyramid.
Frequently,thesetestsareverydifficulttoautomateandverysensitivetoanychangesinthe
application.BecausetheyrunthroughtheGUI,theyalsotendtotakealongtimetoexecute.Soiftest
automationisanafterthought,weendupwithalargenumberofslow,fragiletests.
Animportantprinciplewhenautomatingtestsistousethesimplestpossibleinterfacetoaccessthe
logicwewanttoverify.Projectsthatusetestdrivendevelopmenttechniquesattackthisproblemat
multiplelevels.Theydodetailedunittestingofindividualmethodsandclasses.Theydoautomated
testingoflargergrainedcomponentstoverifythattheindividualunitswereintegratedproperly.They
augmentthiswithsystemtests(whichincludeusecaseorworkflowtests)atthewholeproductlevel.At
eachhigherlevel,theytrytofocusontestingthosethingsthatcouldnotbetestedatthelowerlevels.
Thisleavesthemwithmuchfewersystemteststoautomate.ThisisillustratedinFigure4ProperTest
Pyramid

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 175


Figure4
ProperTestPyramid
Onewaytoreducetheeffortinvolvedistominimizetheoverlapbetweentheunittestsandcomponent
testswiththesystemtests.Aspecificexampleofthisistheuseofbusinessunitteststotestbusiness
logicwithouthavingtogothroughtheuserinterface.Anothertechniqueistheuseofsubcutaneous
(literally,undertheskinmeaningbehindtheGUI)workflowteststotestbusinessworkflowswithout
beingforcedtoaccessthefunctionalitythroughtheuserinterface.Bothoftheseapproachesrequire
theproducttobedesignedfortestability.

17.2.5 AutomatedTestingofNonfunctionalRequirements
Manytypesofnonfunctionaltestsrequiretheuseofautomatedtesttools.Manyofthesetoolsare
speciallycraftedforthespecificpurposeofassessingtheproductwithrespecttoaparticularkindof
nonfunctionalrequirement.Commonexamplesincludeperformancetestingtoolsthatgenerateloadto
seehowtheproductcopeswithhightransactionratesorusabilitytestingtoolsthatrecordtestsessions
andanalyzedataonusersperformance(e.g.successfultaskcompletion,timeontask,navigationpaths,
errorratesetc.)

17.2.6 AutomationasPowerToolsforManualTesters
Automatedtestsprovideahighdegreeofrepeatability.Thisworksveryeffectivelyasachangedetector,
butitprobablywillnotfindbugsthathavealwaysbeenthere.Forthatweneedhumantesterswhoare

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 176

continuallylookingforwaystobreakthesoftware.Forhumantesterstobeeffective,theymustbeable
tofocusonthecreativetaskofdreamingupandexecutingnewtestscenarios,notthemundanetasksof
settinguptestenvironments,comparingoutputfiles,orgeneratingorcleansinglargeamountsoftest
data.Alotofthesetaskscanbemadefairlypainlessthroughappropriateuseofautomation.
Wecanuseautomatedscriptsandpowertoolsforthefollowing:
Tosetuptestenvironments
Togeneratetestdata
Tocompareactualoutputfilesordatabaseswithtestoracles
Togenerateandanalyzetestreports
Toteardowntestenvironments

Ifweneedalargedataset,wecanwriteaprogramtogenerateonewithknowncharacteristics.Ifwe
needtotesthowparticulartransactionsbehavewhentheproductisstressed,wecanwriteaprogram
thatusesupallthememoryordiskspaceorCPUoncommand.Theseareallexamplesofpowertools
thatmakehumantestersmoreeffective.ForspecificexamplesseeaseriesofVisualStudioTeamTest
walkthroughs[SterlingOnTestTools].

17.2.7 AutomatedTestGeneration
Oneofthegrandobjectivesofsoftwaretestingisautomatedtestgeneration.Theindustryisstillsome
distancefrombeingabletopushonebuttonandhaveatoolgenerateandrunallthetestswewillever
need,buttherearesomeselectedsituationswhereautomatedtestgenerationispractical.Oneexample
iscombinatorialtestoptimization.Forexample,ifwehaveamodulewearetestingthattakesfive
differentparameters,eachofwhichcouldbeanyoneoffourvalues,eachofthevaluescausesthe
moduletobehavesomewhatdifferently,butindifferentway.Totestthiseffectively,wewouldhaveto
test1024(4*4*4*4*4)differentcombinations,whichisnotverypractical.Wecanuseatoolthat
analysesthefivedimensionsandgeneratesaminimalsetoffivevaluetuplesthatwillverifyeach
interactionofaparticularpairofvaluesatleastonce(seeallpairstestgenerationandstressfulinput
testgenerationin[BachOnTestTools]includesall).
AnotherformoftestgenerationisModelBasedTesting(MBT).InMBT,webuildamodelofthesalient
aspectsoftheproductundertestanduseittoderivetests.Thetestscanbederivedmanuallyorwecan
buildatestgenerationtoolthatanalysesthemodelandgeneratesthetestsneedtoachievefulltest
coverage.Thetoolcansimplygeneratetestscriptsthatwillbeexecutedbyothertoolsorhuman
testers,oritcanexecuteeachtestasitisgenerated.

17.2.8 ReadinessvsAcceptance
AsdescribedChapter1TheAcceptanceProcessandChapter2DecisionMakingModel,the
acceptanceofsoftwarecanbedivided,atleastlogically,intotwoseparatedecisions.Thereadiness

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 177

decisionismadebytheProductDevelopmentTeambeforegivingthesoftwaretotheProductOwner
whomakestheacceptancedecision.Akeydecisionisdeterminingwhichtestsarerunaspartof
readinessassessmentandwhicharerunaspartofacceptancetesting.Inmostcases,thereadiness
assessmentismuchmoreextensivethantheacceptancetesting.Whenfunctionaltestsareautomated,
itislikelytheyruninreadinessassessment,andthesoftwareisnotreleasedtoacceptancetestinguntil
allthetestspass.ThisresultsinabetterqualityproductbeingpresentedtotheProductOwnerfor
acceptancetesting.

17.2.9 ManagingTestDevelopment&Automation
Itisusefultohaveacommonunderstandingofhowthedevelopmentoftestsrelatestothe
developmentoftheproduct.Insequentialprocesses,thedevelopmentofthetestsstartswellafter
productdevelopmentanddoesntinfluenceit.Weprefertothinkoftestdevelopmentasoccurringin
parallelwithproductdevelopmentspecificallysothatitcaninfluencewhatisbuilt.Itisalsousefulto
thinkabouthowthetestsaredefinedandhowthatrelatestotestautomation.Ifautomationisfocused
onentiretestcases,theautomationcannotbeginuntilatleastsomeofthetestcasesarefullydefined.
Analternativeistothinkoftestautomationattheteststeplevel.Thenautomationdevelopmentcan
beginassoonassomeofthestepsrequiredbytestcasesaredefined.Thisalsoallowsthetest
automationdevelopmenttobecarriedoutbypeoplewithdifferentskills(testautomation)thanthose
doingthetestdesign(testing).Theautomationworkcanbecarriedoutinparallelwiththetest
definitionandtheproductdevelopment.Thismakesitpossibletohaveautomatedtestsavailableas
soonastheproductsoftwareisavailableratherthanatsomelatertime.
TheteststepsoractionsshouldbedefinedusingtheUbiquitousLanguageadoptedbyeveryoneinthe
project.ThetestscriptsdefinedusingthesetermscanbeexecutedbyhumantestersoriftheROIis
sufficient,automated.Ifallthestepsneededbyatestscripthaveautomatedimplementations,the
entiretestscriptcanbeexecutedautomatically.Ifonlysomeofthestepsareautomated,ahuman
testermaystillbeneededbutthetimerequiredtoexecutethetestmaybereducedsignificantly.

17.3 WhoWillAccepttheProduct?
Ultimately,theacceptancedecisionbelongstotheProductOwner.Insomecases,theProductOwner
maynotbeasingleperson.Inthesecases,wemayhaveaProductOwnerCommitteethatmakesthe
acceptancedecisionusingsomesortofdemocraticorconsensusbasedprocess.Or,wemayhaveaset
ofacceptancedecisionmakersthateachcanvetoacceptance(seeChapter1TheAcceptanceProcess
inPartI)Inothercases,theProductOwnermaybeunavailable.Inthesecases,wemayneedaProduct
Ownerproxytoactasthe"goaldonor"whobothprovidesrequirementsandmakestheacceptance
decision.TheproxymaybeeitheradelegateselectedbytheProductOwner,amediatorbetweena
groupofcustomers,orasurrogatewhoactsonbehalfofalargegroupofanonymouscustomers.The
latterroleisoftenreferredtoastheproductmanager.Formoreinformationaboutthisprocess,seethe
CustomerProxySelectionthumbnailinVolumeII.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 178

Arelatedquestioniswhowilldotheacceptancetesting?Andbyextension,whowilldothereadiness
assessment.Thisverymuchdependsonthebusinessmodelandthecapabilitiesandskillsetofthe
partiesinvolved.Thesidebar"DecisionMakingModelStereotypes"enumeratesseveralcommon
scenarios.WheneithertheProductDevelopmentTeamortheProductOwnerfeelstheyneedassistance
conductingthereadinessassessmentoracceptancetesting,theymayresorttoaTestOutsourcing
model.Thethirdpartytestlabwoulddotheassessment,butthereadinessdecisionandacceptance
decisionstillbelongtotheProductDevelopmentTeamandProductOwnerrespectively.

17.4 WhenWillWeDotheTesting?
Thetestplanneedstoaddresswhenthetestingwillbedone.Sometestingactivitieswillbedonebythe
ProductDevelopmentTeamaspartofreadinessassessment,whileothersaretheresponsibilityofthe
ProductOwnerwhowillbedecidingwhethertoaccepttheproduct.Thetestplanneedstoincludethis
infurtherdetailtothepointwherewehaveanunderstandingofhowmuchtimeweneedforreadiness
assessmentandacceptancetestingandwhatwewillusethattimefor.Therearetwomainstrategiesfor
whentestingisdoneonaproject.Thesequentialapproachtoproductdevelopmentlifecycleplaces
mosttestingactivitiesintoaseparatetestphaseafteralldevelopmentiscompleted.Theagileapproach
istotesteachincrementoffunctionalityasitiscompleted.

17.4.1 TestPhase
Onewaytoplantestingistodefineatestingphaseoftheprojectafterallnewfunctionality
developmentiscomplete.ThisisillustratedinFigure5TestingPhaseinaSequentialProduct
DevelopmentLifecycle.


Figure5
TestingPhaseinaSequentialProductDevelopmentLifecycle
Thistestingphasetypicallyconsistsofseveraltimeboxedtestcycles,eachofwhichcontainsboth
readinessassessmentandacceptancetestingactivitiesasillustratedinFigure6.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 179


Figure6
MultipleTTestCyclesinAcceptPhaseeofProject
Eachtestcyclewouldb befocusedon nonereleaseecandidate(R RC)versionoffthesoftwaree.Withinthetest
cycletheProductDeve elopmentTeaammakesareeadinessdeciisionbeforeinvolvingtheProductOwn ner
Teamintheacceptanccetesting.Inttheearlytesttcycles,thisrreadinessdeccisionismadeebyanswerin ngthe
question,"Isitgoodennoughtoboth herhavingtheProductOw wnerTeamteestit?"Itisvaaluabletoget
ProductO Ownerfeedbaackonthesofftwareevenifweknowth herearesomeedefects.Theetestcycleseeach
resultincconcernsthattneedtobein
nvestigated.AAnyconcernssthatneedso oftwarechanggesarethen
addressed dbytheProductDevelopm mentTeam,aandanewreleeasecandidateisbuilt.Thissetsthestaage
fortheneexttestcycle.Werepeatthhisprocessuntilthereleassecandidateisacceptedb bytheacceptaance
decisionm maker(s).
predefinedseetoftestingaactivities;thessemaybelaidout
Withineaachtestcycle,,itislikelyweewillhaveap
asaPertcchartoraGanttcharttoreeflecttimingandinterdep pendencies.W Wemaydecid detoreuseth
he
sameplan nforeachoftthetestcyclees,orwecoulddefineaun niqueplanforreachcycle.Inpractice,wwith
goodauto omatedregre essiontestinginplace,weshouldfindfeewerandfew werdefectseaachtestcyclee.
Becauseo ofthis,ariskbasedapproaachtoplannin ngthesubseq quentcyclesccouldresultinshortercycle
timesand dfastertimettomarket.Wemayalsodeeliberatelych hoosetodefeersomekindsoftesting
activitiesttolatertestccyclesortodo
otheminearrliertestcyclees.
Thetestresourcesonaasequentialp projectaretyypicallyinvolvvedmostlyatthebackend doftheprojeect.
Evenwheentheyareinvvolvedinflesshingouttherequirementsatthebegin nningofthepproject,their
involvemeentwhilethesoftwareisb beingbuiltten
ndstobemin nimal.Thestaaffingprofileofahypothettical
10weekpprojectisillusstratedinFigu
ure7SequeentialTestSpeecialistStaffin
ngProfile.

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 180

12

10

8 BugFix
Regression
6
TestExecution
4 TestDesign
Requirements
2

0
1 2 3 4 5 6 7 8 9 10

Figure7
SequentialTestSpecialistStaffingProfile
Thisprojectinvolvestenfeaturesandeachfeaturerequiresonepersonweekofrequirements,one
personweekoftestdesignandonepersonweekoftestexecutionfollowedbyapersonweekofbug
fixingbydevelopmentandonepersonweekofregressiontesting.Notehowthebulkoftheactivity
occursatthebackendoftheprojectandrequirestentestspecialistsforweeks8and10(week9
involvesprimarilydevelopmentresources.)Thisburstynatureoftestingisusuallyaccommodatedby
testerssplittingtheirtimeacrossseveralprojects.Thismakesithardforthemtokeepuptodateon
whatisbeingbuiltandhowtherequirementsareevolvingbecausemultiplexing(taskswitching)isa
formofwaste.(SeethesidebaronMultiplexingvs.UndividedAttention).Also,thisrequires
developmenttobefinishedinweek7toleavetimefortwo1weektestcycleswithaweekofbugfixing
inbetween.

Sidebar:Mutliplexingvs.UndividedAttention
Havingthebulkoftestingactivitydoneattheendoftheproductdevelopmentlifecyclegivesthe
managerstheillusionthattestersexperienceslackintheirworkloadsatdifferenttimesandtherefore
canbeassignedtomultipleprojectsthatwouldenablethemtousethetimemoreefficientlyand
effectively.Notonlydoesthisapproachplaceconstraintsonwhendevelopersreceivefeedbackon
theproduct,butalsoonhowmuchtimetheyhavetoactonthatfeedback.
Multiplexingrequirescontextswitiching.Thecostofcontextswitchingincludestheeffortfor
understandingeachprojectsstatusandreimmersingonselfintheotherprojectcontextaswellas
findingpropertestenvironments,resettingthem,relearningcertainrequirementsoftheproduct,re
establishingcontactwiththerightpeople,andoftenregainingfocusandrethinkingissuesthathave
alreadybeensettled.AllofthesearereferredtobyTomDeMarcoasimmersion[DeMarcoOnSlack].
Multiplexingmayalsorequirecatchingupontheworkdoneinonesabsence.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 181

GeraldWeinbergestimatesthatthecontextswitchingcostofassigninganadditionalprojecttoone
personisamassive20%[WeinbergOnContextSwitching].Forthreeprojects,itsa40%dropin
productivity!
JoelSpolskystestimonyindicatesthatwithsoftwareengineersthecostofcontextswitchingeven
greater75%[SpolskyOnContextSwitching].Spolskyarguesthatsoftwaredevelopmentisthekindof
intellectualactivitywherewehavetokeepalotofthingsinourheadatonce.Themore[information]
you[can]keepinmind,themoreproductiveyouare.Suchinformationincludesvariablenames,APIs,
datastructures,requirements,helperfunctions,testcases,detailsofstacktraces,debugging
information,therepositorystructure,configurationparametersandwheretheyareset.
Researchersincognitivepsychologyandorganizationalbehaviourwhospecificallystudiedsoftware
engineersalsocametothesameconclusion:multiplexingisharmful.Itsoneofthecontributing
factorstotimefamine,whichisdestructivetoindividualslivesandnotinthebestinterestoftheir
employerseither[PerlowOnTimeFamine]and[OcasioOnWorkFragmentation].


Ooopsthepieceofcodeaboveclearlydoesntbelonghere.Itcrawledinfromadifferentproject.
Whennoticedit,Irealizedwhatatravestythiswas!Iwasworkingonademoofthefluent
configurationinterfaceforfortheEnterpriseLibrary5.0projectwhilealsoworkingontheAcceptance
TestEngineeringGuide.
Thiswassuchawonderfulillustrationofanotherpointhowthebugscaneasilycrawlinwhile
multiplexingandnotgivingthefullattentiontooneprojectthatIvedecidedtoleavethisinhereas
acaseinpoint!
Thebottomlineisthis:fulltimeinvolvementinasingleprojectimprovesindividualperformance.

17.4.2 IncrementalTesting
Thealternativetoleavingallthetestingtotheendofaprojectistodoincrementaltestingassoftwareis
developed.ThisrequiresthattheProductDevelopmentTeamorganizesitsworksuchthatthereisa
steadystreamoftestablesoftwareavailablestartingearlyintheproject.ThisisillustratedinFigure8
IncrementalProductDevelopmentLifecycle.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 182


Figure8
TestinginanIncrementalProductDevelopmentLifecycle
Thisincrementalproductdevelopmentlifecycleallowsacceptancetestingtobespreadoutovermostof
theprojectdurationinsteadofbeingjammedintoamuchshortertestingphaseonthecriticalpathnear
theendoftheproject.(Thisisratherdifferentthanthesequentialapproachusedinmanyorganizations;
thesidebarWhatitTakestodoIncrementalAcceptancedescribessomeofthechallengesand
solutions.)Aseachfeatureoriterationisfinished,itisassessedforreadinessimmediatelybythe
ProductDevelopmentTeamandturnedovertotheProductOwnerforacceptancetesting.Ideally,any
bugsfoundarefixedimmediatelyasdescribedinthesidebarIncrementalAcceptanceandBugTracking
onAgileProjectsinChapter8ratherthanbeingstockpiledfortheendoftheproject.
Ifthesametenfeaturesdescribedinthesequentialexamplearebuiltbyanagileteamusing
IncrementalAcceptanceTesting,thetestingstaffprofilelooksmorelikeFigure9AgileTestSpecialist
StaffingProfile.

5
4
Regression
3
TestExecution
2
1 TestDesign

0 Requirements
1 2 3 4 5 6 7 8 9 10

Figure9
AgileTestSpecialistStaffingProfile.
Onthisprojectwestartoutbyspecifyingtwofeaturesinthefirstweek,designthetestsforthosetwo
featuresinthesecondweekaswellasspecifyingathirdfeature.Inthethirdweekwetestthefirst
featureanddotestdesignforthethirdfeature.Notehowthestaffingrampsuptofourpersonweeksof

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 183

workandstaystherefortherestoftheduration.Towardstheendasnewfeaturetestingrampsdown
theslackistakenupbyregressiontestingoftheearlyfeatures.Becausethetestersarepartofthe
developmentteamtheyareabletokeepabreastofhowtherequirementsandproductdesignare
evolving.Sincethesamepeoplewillbedesigningandexecutingtheteststheneedtowritedetailedtest
scriptsarereducedandmoreeffortcanbespentonexploratorytestingandtestautomation.Sincethe
testingisdonesoonafterthecodeisdevelopedthedeveloperscanfixthebugsimmediatelysothereis
noseparatetestingandbugfixphaseattheendthereforedevelopmentcancontinueclosertothe
deliverymilestone.

17.4.3 SessionBasedTestManagement
Analternativetousingaplandrivenapproachwithinthetestcycleistouseamoreiterativestyle
knownasSessionBasedTestManagement.Wecreateaprioritizedbacklogoftestingactivitiesthatwe
addressinaseriesoftestsessions.Asnewconcernsareidentifiedintestsessions,wemayadd
additionaltestactivitiestothetestbacklog.Thekeyistokeepthebacklogprioritizedbythevalueofthe
testing.Thisvalueistypicallybasedontheexpecteddegreeofriskreduction.Thedepthofthebacklog
givesusanideaofhowmuchtestingworkwehaveleft(forexample,bycreatingatestingburndown
chart)andifwearemakingheadwaybyaddressingconcernsorifyouarelosingground(ifthisisthe
case,thebacklogisincreasingindepth).SessionBasedTestManagementiscommonlyusedwith
exploratorytesting.

17.5 WhereWillWeDotheTesting?
Thetestplanneedstoidentifywherethetestingwillbeperformed.Whenallthetestingwillbe
performedinhouse,theprimaryconsiderationiswhichphysical(orvirtual)environmentswillbeused.
Thisisparticularlyimportantwhennewenvironmentsneedtobecreatedorsharedenvironmentsneed
tobebooked.Ifwelackphysicalresourcesortheskillstodothetesting,wemaychoosetodotest
outsourcingtoathirdpartytestlab.
Wealsoneedtodefinethecriteriaformovingthesoftwarebetweentheenvironments.Thetransition
fromthereadinessassessmentenvironmenttotheacceptanceenvironmentisgovernedbythe
readinessassessmentcriteria.Whendevelopershavetheirownindividualdevelopmentenvironments,
wealsoneedcriteriaforwhensoftwarecanbesubmittedintotheteam'sintegrationenvironment
wherereadinessassessmentwilloccur.ThesecriteriaareoftenreferredtoastheDoneDoneChecklist
becausethedefinitionof"done"ismorestringentthanwhatadevelopertypicallyreferstoasdone.

17.6 HowLongWilltheTestingTake?
Becausetestingisusuallyonthecriticalpathtodeliveryofsoftwareintensiveproducts,project
sponsorsandprojectmanagersusuallywanttoknowhowlongtestingwilltake.Themostcommon

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 184

answeris"Howmuchttimedoweh have?"Frequeently,thetimeavailableisnotlongeno oughtogatheer


enoughdatatomakeaahighconfideencereadinesssoracceptancedecision.Thisanswerisnotquiteaas
flippantasitsoundsbe ecauseoftheenatureoftesting.Wecannnotproveso oftwareworksscorrectlyw we
canonlyd disproveitbyyfindingbugs.Evenifweh
hadaninfiniteeamountofttimetotest,aafterapoint,
diminishin ngreturnswillkickinandwewillnotfindmanymorredefects.So o,wehaveto odeterminewwhat
isbarelyssufficienttoggetenoughcoonfidenceabo outthequalittylevel.Thisrrequiresatleastaminimal
levelofteestestimation oundofthetimeandefforrtweneedtoexpend.
ntoestablishthelowerbo
Sometesttsmayhaveddependenciesstherestrictwhentheycaanberun.Theesecouldbedependencieeson
othertesttenvironmen
ntssuchaswh
hendoinginteegrationtestingwithotheersystemsortheymaybe
dependen nciesondate.
needtoknowhowlongweellneedtowaitforanydeefectsweneeedaddressedtobefixedbythe
Wealson
ProductD
DevelopmentTeam.Willth herebedead dtimebetweentestcycleeswhilewew
waitforfixed
softwareasillustratedinFigureYAlternatingTTest&Fix?


FigureY
Alternatin
ngTest&Fix
Thedeadperiodsoccuuriftheaccep
ptancedecisioonisthefirstpointatwhicchtheProducctDevelopmeent
TeamisprovidedthelistofmustfixbugsafterrwhichtheProductDevelopmentTeam mstartsthe
processofcharacterizingandfixingthebugs.Thismaytaked daysorweeksstodevelopaanewreleasee
candidateewhichmustthenundergo oreadinessassessmentbeeforebeingpresentedtottheProduct
Ownerforthenextrouundofaccepttancetesting..
Thedurattionoftheacceptancephaase(consistingofseveraltestcycles)caanbereduced dsignificantlyyif
theProdu uctDevelopmmentTeamisnnotifiedofbu
ugsassoonasstheyarefouund.Thisallow
wsthemtobee
fixingdefeectscontinuo
ouslyanddeliveringanew
wreleasecanddidateonareegularschedu uleasshowniin
FigureXContinuousTest&Fix.

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 185


FigureX
Continuou usTest&Fix
ThisdepeendsonthePrroductOwner(orproxy)d
doingbugtriageonallnew
wlyfoundconcernssothattthe
ProductDDevelopmentTeamismadeawareofalllgatingbugs(bugsthatm mustbefixedb
beforeacceptting
thesoftware)assoonaspracticablee.

planningtoautomatetestts,weshouldhaveaneffortestimatefo
Ifwearep ortheautomation.Inmosst
cases,wewantseparateestimatesfortheconsttructionoftheeautomation ninfrastructureandthe
preparatioonofthetesttsbecauseoffthedifferenttskillsandkn
nowledgeneeededtodotheetwojobs.Fo
or
moreinfoormation,seetheTestAutomationPlan nningthumbn nail.

17.7 HowWilllWeDete
ermineOu
urTestEfffectivene
ess?
Alearninggorganization
nisonethatiisconstantlystrivingtoim
mprovehowittworks.Thisiinvolves
understanndinghowwe ellwearedoingtodayand dtryingnewaapproachestooseewhetheertheymakeus
moreeffeective.Measu uringeffectiveenessrequireesTestMetriccs.Thesemettricsmeasuretwokeyareaasof
performance:
Theymeasurehow wfaralongareweinexecu
utingourtesttplan.Thatis,howmuchw
workisleftbeefore
w
weknowenou ughtomaketthereadinessoracceptanccedecision?FFormoreinfo
ormation,seeethe
TeestStatusReportingthum
mbnail.
Theymeasuretheeffectivenesssofourtestin
ng.Formoreinformation,seetheAssessingTest
Efffectivenesstthumbnail.

17.8 HowWilllWeMana
ageConce
erns?
Thepurpo
oseoftestinggistoidentifyyanyconcern
nswiththeso
oftwareoftho
osewhomattter.Manyoftthese
concernswillrequirecchangestotheesoftwareeitherbecauseesomethingw wasincorrectllyimplementted(a

AcceptancceTestEngineeringGuideeVolumeI,ReleaseCand
didate1 186

bug)orbecausetheProductOwnerrealizedthatwhattheyhadrequestedwillnotsatisfythebusiness
need(anenhancementorchangerequest).Thetestplanneedstolayouthowtheseconcernswillbe
managedandtracked;italsoneedstoincludetheprocessfordecidingwhatneedstobechangedand
whatisacceptableasis.
Asweconductthevariousreadinessassessmentandacceptanceactivities,wenoteanyconcernsthat
comeup.Investigatingtheseconcernsmorecloselyrevealsthat,eachconcernfallsintooneofthe
followingcategories:
Bugordefect
Requirementschange
Projectissue
Nonconcern

Figure3illustratesthelifecycleforeachofthemajortypesofconcerns.


Figure3
ConcernResolutionModel
Partoftheadvantageofcategorizingtheseconcernsistohelpidentifyhowtheywillbeaddressed.Each
categoryofconcernshasitsownresolutionprocessandeachmustbeaddresseddifferently.

17.8.1 BugsorDefects
Bugsaredefectsfoundinthesoftwarethatrequireasoftwarechange.Thebugsneedtobeunderstood
wellenoughtomakedecisionsaboutwhattodoaboutthem.Acommonprocessfordoingthisisknown
asBugTriage,whichdividesthebugsintothreecategorieswithrespecttothenextmilestoneorrelease:
MustFix,WouldLiketoFix(ifwehavetime),andWillNotFix.Ofcourse,thesoftwaremustberetested

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 187

afterthefixesaremade,whichiswhytherearetypicallymultipletestcycles.Fixesmayalsoneedtobe
propagatedintootherbranchesofthesoftware.Ifthebugisfoundtoexistinpreviousversionsthatare
currentlyinuse,apatchmayneedtobepreparedifthebugisseriousenough.Securityrelatedbugsare
anexampleofbugsthataretypicallypatchedbackintoallpreviousversionsstillinuse.Likewise,abug
fixedintheacceptancetestbranchmayneedtobepropagatedintothecurrentdevelopmentbranchif
newdevelopmenthasalreadystartedinaseparatedevelopmentbranch.

17.8.2 RequirementsChanges
TheProductOwnermayhaverealizedthateventhoughtheProductDevelopmentTeamdeliveredwhat
theProductOwneraskedfor,itwillnotprovidetheexpectedvalue.TheProductOwnershouldhavethe
rightandresponsibilityformakingthebusinessdecisionaboutwhethertodelaythereleasetomakethe
changeorcontinuewiththelessusefulfunctionality.Oncewevedecidedtoincludeachangeinthis
releasewecantreatitmoreorlessthesameasabugfromatrackingandretestperspective.

17.8.3 OtherIssues
Someconcernsthatareexposeddonotrequirechangestothesoftware.Theymaybeprojectissues
thatneedtobetrackedtoresolution,additionalthingsthatshouldbetested,andsoon.Thesetypically
donotgettrackedinthebugmanagementsystembecausemostprojectshaveothermeansfortracking
them.Otherconcernsmaybenotedbutdeemedtonotbeconcernsatall.

17.9 Summary
Thischapterintroducedtheactivitiesandpracticesinvolvedinplanningatestingeffort.Akeyactivityis
thedefinitionofateststrategybecausethisiswhatguidesusaswestrivetomaximizetheROIofour
efforts.Thereisaplaceforbothautomatedtestingandmanualtestingonmostprojectsbecausethe
twoapproachesarecomplementary.Automatedfunctionaltestsarehighlyeffectivechangedetectors
thatgoalongwaytowardpreventingnewbugsfrombeingintroducedduringsoftwaremaintenance
activities.Theyarealsorepeatable.Carehastobetakentousetheappropriatefunctionaltest
automationtoolstoavoidtheslow,fragiletestsquagmire.Exploratorytestingmanualtestingcanbe
effectiveandefficientwaysoffindingbugs,especiallyunusualones,thosethatarehardtoautomate,
andthosethatstemfromincomplete,misunderstood,orvaguerequirements.Theuseofpowertoolsby
humantesterscansignificantlyincreasetheeffectivenessofmanualtesting.

17.10 WhatsNext?
Inthischapterwehavedescribedthepracticesinvolvedinplanningfortheacceptanceoftheproduct.
Inthenextchapterwewillexaminethepracticesweusewhileexecutingtheacceptanceprocess.

17.11 References
[MarickOnTestingDimensions]Marick,Brianhttp://www.exampler.com/oldblog/2003/08/21/

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 188

[CrispinGregoryOnAgileTesting]Crispin,LisaandJanetGregory.AgileTesting,AddisonWesley,2008.
[MeszarosOnUnitTestPatterns]Meszaros,Gerard.xUnitTestPatterns:RefactoringTestCode.Addison
WesleyProfessional.2007.
[KanerOnExploratoryTesting]Kaner,Cem,JackFalk,andHungQ.Nguyen.TestingComputerSoftware,
2ndEdition.Wiley.1999.
[BachOnExploratoryTesting]Bach,James.WhatisExploratoryTesting?
http://www.satisfice.com/articles/etarticle.pdf
[FewsterGrahamOnTestAutomation]Fewster,M.,Graham,D.SoftwareTestAutomation.Addison
Wesley,1999.
[ItkonenRautiainenOnExploratoryTesting]JuhaItkonenandKristianRautiainen,ExploratoryTesting:A
MultipleCaseStudy,InternationalSymposiumonEmpiricalSoftwareEngineering,ISESE2005,1718
Nov.2005,pp.8494.
[ItkonenEtAlOnDefectDetection]Itkonen,J.;Mantyla,M.V.;Lassenius,C.DefectDetectionEfficiency:
TestCaseBasedvs.ExploratoryTesting,inProc.ofFirstInternationalSymposiumontheEmpirical
SoftwareEngineeringandMeasurement,ESEM2007.2021Sept.2007,pp6170.
[JeffriesMelnikOnTDDEffectiveness]Jeffries,R.andMelnik,G.,TDD:TheArtofFearlessProgramming,
GuestEditorsIntroduction,IEEESoftwareSpecialIssueonTestDrivenDevelopment,MayJune2007,
pp.2430.
[SterlingOnTestTools]CharlesSterling,VisualStudioTeamSystem2010TestFeatureswalkthrough
withscreenshots.http://blogs.msdn.com/charles_sterling/archive/2008/11/05/visualstudioteam
system2010testfeatureswalkthroughwithscreenshots.aspx,October20,2009
[BachOnTestTools]JamesBach.RepositoryofTestTools.http://www.satisfice.com/tools.shtml,October
20,2009
[WeinbergOnContextSwitching]GeraldWeinberg,QualitySofwtareManagement,Vol.1:Systems
Thinking,NewYork,NY:DorsetHousePublishing,1992.
[SpolskyOnContextSwitching]JoelSpolsky.HumanTaskSwitchesConsideredHarmful
http://www.joelonsoftware.com/articles/fog0000000022.html,2001,visitedOctober20,2009
[DeMarcoOnSlack]TomDeMarco,Slack,Broadway,2001.
[OcasioOnWorkFragmentation].WilliamOcasio.Towardsanattentionbasedviewofthefirm.
StrategicManagementJournal,18:187206,1997.
[PerlowOnTimeFamine]LesliePerlow.Thetimefamine:Towardasociologyofworktime,
AdministrativeScienceQuaterly,44(1):5781,1999.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 189

Chapter18 AssessingSoftware
Inthepreviouschapterswehaveintroducedmanyoftheconceptsaroundhowweplantheassessment
oftheproductagainsttheMinimumMarketableFunctionality(MMF)andminimumqualityrequirement
set(MQR)towhichwehaveagreed.Inthischapterwewillintroducethevarioustechniquesthatweuse
aswedotheassessmentincludingtestconception,testdesignandtestexecution.
Assessmentisagenerictermwecanusetodescribetheactivities,testingorotherwise,thatweuseto
evaluatethesystemundertest.Someoftheseactivitiesarefocussedonpreparingandexecutingtests
whileothersmaybereviewactivities.Someoftheactivitiesaredoneaspartofreadinessassessmentby
theproductdevelopmentteamwhileothersmaybedonebyorfortheproductownerunderthebanner
ofacceptancetesting.Wherethesamepracticeisusedinbothformsoftesting,themechanicsofthe
practicestypicallydontchangeverymuchalthoughtheemphasisofhowmucheachpracticeisdone
andtheobjectivesofapplyingthatpracticemightvary.
Westartthediscussionwithanoverviewofthelifecycleofanindividualtests,somethingthatboth
functionalandnonfunctionaltestsdoshareandthenmoveintoadiscussionofthepracticesusedin
eachstateoftheindividualtestlifecyclewhereweintroducethetechniquesthatareuniquetospecific
ofkindsofrequirements.

18.1 TheLifecycleofanIndividualTest
Everysingletest,howeversimpleorcomplex,whethermanualorautomated,goesthroughanumberof
stagesduringitslifetime.ThislifecycleisillustratedinFigure1.


Figure1
IndividualTestLifecycle

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 190

Thestatesofthelifecycleare:
Conception Anacceptancetestisconceivedtoaddressaparticularriskand/orachievea
specificgoal.
Authoring/Design Thetestiswritteneitherindetailedstepformorsomekindofoutlineofwhat
needstobedone,bywhom,when,where,andhow.
Scheduling Theexecutionofthetestisplannedorscheduledforaspecifictimeframeand
resources(people,testenvironment(s),etc.)
Execution Thetestisexecutedagainstthesystemundertest.
ResultAssessment Theresultsofthetestareassessedagainsttheexpectations.(Thismayoccuras
partofexecutionorseparately.)
Reporting Theassessedtestresultsareaggregatedandreported.
Actioning Thetestresultsmayresultineitherfurthertestingbeingidentifiedand/orbug
reportsbeingcreatedandtriaged.
Maintenance Eachtestisanassetandmostmustbemaintainedsothatitcanprovidevaluein
thefuture.Sometests,however,e.g.whendoingexploratorytesting,,arenot
meanttobemaintainedorrepeated.
EndofLife Thetesthasoutliveditsusefulness.Itisabandonedandnolongerrunor
maintained.
Notethattestconceptionandtestauthoringareoftenlumpedtogetherunderthelabeloftestdesign.
Someformsoftesting,oftencalledstatictesting,involveinspectingartefactsthatdescribethesystem
undertestratherthanrunningtheactualcode.Theterminologyusedfortheseformsoftestingis
somewhatdifferent(forexample.theyareoftencalledreviewsratherthantests)butforthepurposeof
discussingtestlifecycle,weshallusethetestterminology.
Letsexamineeachofthesestatesinabitmoredetail.

18.1.1 TestConception
Atsomepoint,someonedecidesthatweneedtoverifyoneormoreaspectsofthesystemundertests
behavior;wecallthesethingstotesttestconditions.Atthistimethetestisjustafigmentof
someonesimagination.Itstartsitstransitionfromanimplicitrequirementtoonethatismuchmore
explicitwhenitgetswrittendownorcapturedinadocument.Itmightappearinalistoftestconditions
associatedwithafeature,requirementoruserstory.Typically,itwilljustbeaphraseornamewithno
associateddetail.Nowthatthetestexistsinconcept,wecanstartmovingitthroughitslifecycle.

18.1.2 TestAuthoring/TestDesign
Testauthoringortestdesignisthetransformationofthetestortestconditionfrombeingjustanamed
itemonalistintoconcreteactions.Itmayalsoinvolvemakingdecisionsaroundhowtoorganizetest

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 191

conditionsintotestcaseswhicharethesequencesofstepsweexecutetoverifythem.Notethattest
authoring/designmayhappenlongbeforetestexecutionorconcurrentlywithtestexecution.

18.1.3 TestScheduling
Onceatestcasehasbeenidentifiedandauthoredoratestcharterdefined,wemustdecidewhenitwill
beexecuted.Theschedulemayindicatetheonetimethetestisrunorthefrequencywithwhichitisrun
andthetriggeringmechanism.Itmayalsoidentifywhoorwhatisexecutingthetestandinwhichtest
environment(s).

18.1.4 TestExecution
Onceauthoredandscheduled,weneedtoactuallyexecutethetests.Fordynamicteststhisinvolves
runningthesystemundertest;statictestsinvolveinspectingvariousartefactsthatdescribethe
systemundertestbutdonotinvolvedexecutingthecode.Dependingonthekindoftestinquestion
thetestmaybeexecutedmanuallybyaperson,byanautomatedtestingtool,orbyapersonusingtools
thatimprovetesterproductivitythroughautomation.

18.1.5 ResultAssessment
Dependingonthetoolsinvolved,thepass/failstatusofthetestsmaybedeterminedasthetestsare
executedortheremaybeaseparatesteptodeterminethetestresultsafterthetestexecutionhasbeen
completed.Wedeterminewhetheratestpassedorfailedbyinspectingtheactualresultsobservedand
determiningwhethertheyareacceptable.

18.1.6 TestReporting
Onceasuiteoftestshasbeenexecutedandassessed,wecanreportonthetestresults.Agoodtest
reporthelpsalltheprojectstakeholdersunderstandwheretheprojectstandsrelativetotherelease
gate.Chapter1TheAcceptanceProcessprovidesdetailsonwhatinformationmightaffectthis
decision.Testreportingincludesbothteststatusreportingtoindicatehowmuchtesteffortremainsand
testeffectivenessreportingwhichdescribesourlevelofconfidenceinourtests.

18.1.7 TestActioning
Thepurposeofexecutingtestsistolearnaboutthequalityofourproductsothatwecanmake
intelligentdecisionsaboutwhetheritisreadyforuseorrequiresfurtherdevelopmentortesting.The
AcceptanceProcessdescribestheprocessfordecidingwhetherornottoacceptthesoftwarebutbefore
wecanmakethatdecisionwemayneedtofixsomeofthedefectswehavefound.TheBugTriaging
processisusedtomaketheIsitgoodenough?decisionbydeterminingwhichbugsneedtobefixed
beforewecanrelease.(SeetheDonenessModelformoredetails.)

18.1.8 TestMaintenance
Sometestsareonlyrunoncewhileothersmayneedtoberunmanytimesoverlongperiodsoftime.
Teststhatwillberunmorethanoncemayrequiremaintenancebetweenrunsasaresultofchangesto

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 192

thepartsofthesystemundertestthattheyinteractwith(forexample,thedatabasestate).These
kindsoftestsmaywarrantmoreofanupfrontinvestmenttoensurethattheyarerepeatableand
robust.Testsintendedformanualexecutionwillneedtobeupdatedwheneverthepartsofthesystem
beingtestedundergosignificantchangesinfunctionality.Whereashumantesterscanusuallywork
aroundminorchanges,fullyautomatedtestswilltypicallybeimpactedbyeventhesmallestchanges
(withteststhatinteractwiththesystemundertestthroughtheGUIbeingthemostfragile)and
thereforemayrequiresignificantlymorefrequentmaintenance.

18.1.9 EndofLife
Soonerorlateratestmaynolongerbeworthexecuting.Perhapsthefunctionalityitverifieshasbeen
removedfromthesystemundertestormaybewehavedeterminedthatthefunctionalityiscovered
sufficientlywellbyothertestssothatwenolongergetmuchadditionalvaluefromrunningthistest.At
thispointthetesthasreacheditsendoflifeandnolongerwarrantseitherexecutionormaintenance.

18.2 VariationsinTestLifecycleTraversal
Sometestsspendalotoftimeineachstateofthetestlifecyclewhileothersmaypassthroughthe
statesveryquickly.Forexample,inautomatedfunctionaltestingwemightspendsweekspreparinga
complextest,waitseveralweeksbeforewecanfirstexecuteit,andthenrunitseveraltimesadayfor
manyyears.Incontrast,duringasingleonehourexploratorymanualtestingsession,thetestermay
conceiveofseveraltestconditions,designatesttoexplorethem,learnsomethingaboutthesystem
undertest,conceiveseveralmoretestconditionsanddesignteststoexplorethemalso,allinthe
spaceofafewminutes.Theautomatedtestswillspendmostoftheirlifetimeinthemaintenancestate
whileexploratorytestsareveryethereal;thereisntaconcreterepresentationthatneedstobe
maintained.

18.2.1 HighlyCompressedTestLifecycleExploratoryTesting
ExploratorytestingissummarizedbyCemKanerasSimultaneoustestdesignandexecutionwithan
emphasisonlearning.Fromthisdescriptionitshouldbeclearthatthereisnoclearseparationofthe
variousstagesofthetestlifecyclewhendoingexploratorytesting.Thetesterlearnsaboutthesystem
undertestbyusingitandforminghypothesisabouthowitshouldbehave.Basedonthesehypotheses
thetesterconceivesofoneormoretestconditionstowhichtheymightsubjectthesystemundertest.
Theyrapidlydesign,intheirmindseye,atestcasetheycouldusetoachievethis.Theyexercisethetest
caseandobservetheresulttherebyformingmorehypotheseswhichinturnleadtomoretest
conditions.Thereisnotanattemptmadetohavethetestpersistbeyondthetestsessionunlessit
revealedabug.Thisremovestheneedtodocumentandmaintainthetests(thoughagoodexploratory
testerkeepsasetofnotes/journalofkeypointsanddiscoveriesduringhisorhertesting);theobvious
consequenceisthatexploratorytestingisnotintendedtobeveryrepeatable.
Thisprocessisverylightweightwithverylittleoverheadgettinginthewayofthetesterinteractingwith
thesoftware.Thismakesitpossibleforexploratorytesterstoformulatealotofhypotheses,testthem

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 193

andfindalotofbugsinaveryshortperiodoftime.Thetestertakesnotesastheygofocussedprimarily
onthefollowingoutputs:
1. Whatfunctionalitytheyhavetestedandtheirgeneralimpressionsofwhattheyhaveseen.
2. Whatbugstheyfoundandwhattheyhaddonetocausethem.
3. Whattestconditionstheyhadconceivedthattheywerenotabletogetto.Thesemaybe
usedasthecharterforasubsequentexploratorytestsession.
4. Howmuchtimewasspentactuallytestingversushowmuchwasspentgettingready.This
informationisusefulwhendecidingwhatkindofpowertoolswouldmaketheexploratory
testermoreefficientinthefuture.

Despiteitssomewhatchaoticappearanceexploratorytestingcanbequitedisciplinedandmethodical
eventhoughitisnotveryrepeatable.Exploratorytestingcanrangefromcompletelyunstructuredto
highlydisciplined.Themoredisciplinedformsofexploratorytestinguseasequenceoftimeboxedtest
sessionstostructurethetestingactivities.
Planningofexploratorytestingconsistsofdefininganinitiallistofhighleveltestconditions(asinkinds
ofthingsweshouldtest)foruseastestsessionchartersanddecidinghowmanytestsessionstobudget
forexecutingthecharters.Examplesofchartersmightincludethefollowing:

Pretendthatyouareahackerandtrybreakingintothesystem(apersonabasedcharter)

Tryoutvariationsoftheinvoicingworkflowfocussingonrejecteditems(ascenariobased
charter)

Tryusingtheuserinterfaceusingonlythekeyboard(adevicebasedorpersonabasedcharter).

Tryscenarioswhereseveraluserstryaccessingthesameaccountatthesametime.(ascenario
basedcharteroftencalledtugofwar.)
Thetestchartersareprioritizedbeforebeingscheduledviaassignmenttoatesterexecutingaspecific
testsession.Uponcompletionofthetestsession,thetestermayrecommendadditionalchartersbe
addedtothebacklogofchartersbasedonwhattheyhadlearnedaboutthesystemundertest,business
domain,usersneedsetc.Exploratorytestingisoftendoneinaniterativestylewithmanyofthetest
chartersforlateriterationsbeingdiscoveredduringexecutionofthetestsessionsinearlieriterations.
Thisallowsexploratorytestingtofocusontheareasofthesoftwarethathavebeenfoundtobethe
mostsuspiciousratherthanprovidingthesameamountofeffortforallareasregardlessofthequality
levelactuallyobserved.This,combinedwiththelowoverheadnatureofthesimultaneoustestdesign
andexecution,iswhatallowsexploratorytestingtobesuchaneffectivewayoffindingthebugswe
knowmustlurkinthesoftware.

18.2.2 ASpreadOutTestLifecycleScriptedTesting
Inscriptedtesting,theaveragetestlifecycleismuchlongerthatinexploratorytesting.Thetestsmaybe
conceivedaspartofthetestplanningexerciseorinmoredetailedtestdesignactivities.Theactualtests

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 194

arethendocumentedorprogrammed,andpotentiallyreviewed,oftenbeforethesoftwareisavailable
fortesting.Thetestsmayrequiremaintenanceevenbeforetheyarefirstexecutedagainstsystem
undertestifthedesignofthesoftwarehasevolvedsincethetestsweredesigned.Eventually,we
determinethescheduleforexecutingthetestsandthetestsareexecutedattheappropriatetime
(whichmaybeweeksorevenmonthslater.)Anybugswefindareloggedandthetestresultsare
reportedtothestakeholders.Thebugsareactioned,oftenweeksorevenmonthsaftertheywere
found.Ifthetestsaretoberepeatedatalatertime,usuallyagainstasubsequentversionofthesystem
undertest,thetestsmayrequiremaintenancetotrackchangesinthesystemtheytest.Eventually,
someonedecidesthatthisparticulartestisnolongeraddinganyvalueandthetestisabandoned.
Thiscyclecouldtakeanywhereforseveraldaysorweektomanyyears.ThetestteamforMicrosoft
Officepreparesextensiveautomatedtestscriptstoverifythebehavioroffeaturesinapplicationslike
MSWord.Thetestsforaspecificgenerationoftheproduct(e.g.Word2003)havealifetimeofover10
yearsbecauseofMicrosoftscommitmentof5yearsofmainstreamsupportandafurther5yearsof
limitedextendedsupportforeachbusinessanddevelopmentsoftwareproduct,followedbyaminimum
of1yearofselfhelponlinesupportviatheknowledgebase[Ref
http://support.microsoft.com/lifecycle/].Newbuildsaretypicallycreatedeveryweekthroughthis
productlifetimeandthetestsarerunagainsteachnewbuild.Inthisexamplethemaintenancephaseof
thetestdominatestheindividualtestlifecycle.Microsoftpatternsandpracticesteamusecontinuous
integrationthatpotentiallyproducesseveralbuildsadaywithcontinuousautomatedtestexecution.

18.2.3 IntermediateTestLifecyclesHybridTestApproaches
Thetwoprevioussectionsdescribedthetwoextremesoftestlifecycleduration.Inpractice,thetest
lifecyclescanfitanywherebetweenthesetwoextremes.Therecouldalsobeamixoftestlifecycle
durationseveninthesametestingsession.Forexample,amanualtestercouldbefollowingadetailed
testscriptthatwaswrittenmonthsago.Theynoticesomethingoddthatisntspecificallyrelatedtothe
scriptanddecidetogooffscripttoexploretheoddity.Inthisoffscriptexcursiontheyaredoing
simultaneoustestdesignandexecution(inotherwords,exploratorytesting).Atsomepointtheymay
returntotheoriginalscriptaftereitherconfirmingthatthesystemisworkingproperlyorloggingthe
bugsthattheyhavefound.

18.3 PracticesforAssessingSoftware
InChapter16PlanningforAcceptanceweintroducedmanypracticesinthecontextofplanningthe
readinessassessmentandacceptancetestingactivities.Nowitistimetolookatthepracticesthatwe
usewhiledesigning,executingandactioningtheindividualtests.Atthispointwefocusonthepractices
andnotonwhodoesthem;itreallydoesntmatterwhethertheyaredoneaspartofreadiness
assessmentoracceptancetestingasthepracticesthemselvesarenotchangedbywhenandwhodoes
them.VolumeIIcontainsacollectionofthumbnailsandjobaidesforeachpracticedescribedherewith
VolumeIIIpresentingsampleartefactsofapplyingthosepracticesinaproject.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 195

18.3.1 TestConceptionPractices
Therearequiteafewpracticesforconceivingtestsortestconditions.Somearemorestructuredor
formalthanothers.Theyallsharethegoalofcreatinganextensivetodolistforoursubsequenttest
designandexecutionefforts.Moststartwitheitherrequirements,whetherfunctionalornonfunctional,
orrisks(concernsaboutsomethingthatmightgowrong.)Someofthemorecommontechniques
includethefollowing:

Riskbasedtestidentification

Threatmodeling

Heuristicsorchecklists

UsecasebasedtestingDefinetestsbasedonspecificusecasesofthesystemundertest(see
theFunctionalTestingthumbnail.)

BusinessruletestingDefinetestsforvariouscombinationsofvaluesusedasinputstobusiness
rulesorbusinessalgorithms.

InterfacebasedtestingDefinetestsbasedonthecharacteristicsoftheuserinterface(human
computerinterface)orcomputercomputerinterfaceprotocol.

Scenariobasedtestingusingrealworldusagescenariostoinspirethedesignoftestcases.

SoapoperatestingUsingexaggeratedrealworldusagescenariostoinspirethedesignoftest
cases.

ModelbasedtestgenerationBuildingoneormoremodelsofkeycharacteristicsofthe
systemundertestandgeneratingtestsfromthemodel(s).

GroupBrainstorming.

Paired/collaborativetestingWorkingtogethertodesignbettertestcases.

RiskBasedTestIdentification
Inriskbasedtestidentificationwedoriskmodelingtoidentifyareasrelatingtofunctionalornon
functionalaspectsthatweareconcernedmightnotbeimplementedcorrectlyormighthavebeen
adverselyaffectedbychangestothefunctionality.Weusethisinformationtoidentifytestconditions
wewanttoensureareverifiedbytests.Agoodexampleofakindoftestthatmightbeidentified
throughriskanalysisisthefaultinsertiontest.Forexample,theriskweidentifiedwasThenetwork
connectionfails.WeaskourselvesWhywouldthenetworkconnectionfail?andcomeupwith3
differentpossiblecauses:Unpluggedcable,networkcardfailure,networkcardoutofservicedueto
maintenanceactivityinprogress.Thesearethreetestconditionswewouldwanttoexerciseagainstour
systemundertest.Otherformsofriskmightrelatetopotentialmistakesduringsoftwaredevelopment.
E.g.Thisshippingchargealgorithmisverycomplex.Thismightcauseustodefinealargenumberof
testconditionstoverifyvariousaspectsofthealgorithmbasedonthekindsofmistakesadeveloper
wouldbelikelytomake.Forexampleapplyingthevarioussurchargesinthewrongorder.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 196

ThreatModeling
Anotherformofriskmodelingisthreatmodelingforsecurity.Thepotentialthreatsidentifiedbythe
threatmodelcanleadustochoosefromasetofsecurityassurancepractices.Wemightdefinespecific
penetrationtestscenariostoensurethesoftwarerepelsattemptsatpenetration.Wemightconduct
securityreviewsofthecodebasetoensuresafecodingpracticeshavebeenfollowed.Wecandecideto
doFuzzTestingtoverifythatthesoftwarecannotbecompromisedbyinjectingspeciallychosendata
valuesviauserinputfields.

UseCaseBasedTestIdentification
Whenwehavebusinessrequirementsdefinedintheformofusecaseswecanidentifytestconditions
byenumeratingallthepossiblepathsthroughtheusecaseanddeterminingforeachpathwhatinput
value(s)wouldcausethatpathtobeexecuted.Eachpathconstitutesatleastonetestcondition
dependingonhowmanydistinctcombinationsofinputsshouldcausethatpathtobeexecuted.

InterfaceBasedTestIdentification
Anothersourceoftestconditionsisthedesignoftheinterfacethoughwhichtheusecaseisexercised
whetheritbeauserinterfaceusedbyahumanoranapplicationprogramminginterface(API)or
messagingprotocol(suchasawebservice)usedbyacomputer.Theinterfacemayhaveverydetailed
designintricaciesinadditiontotheelementsrequiredtoexercisetheusecase.Theseintricaciesarea
richsourceoftestconditions.Forexample,auserinterfacemayhavepulldownlistsforsomeinput
fields.Testconditionsforthesepulldownlistswouldincludecaseswheretherearenovalidentries
(emptylist),asinglevalidentry(listof1item)andmanyvalidentries(longlistsofitems.)Eachofthese
testconditionswarrantsverification.

BusinessRulesandAlgorithms
Businessrulesandbusinessalgorithmsareanotherrichsourcefortestconditions.Forarulethat
validatesausersinputsweshouldidentifyatestconditionforeachkindofinputthatshouldbe
rejected.Rulesthatdescribehowthesystemmakesdecisionsaboutwhattodoshouldresultinatleast
onetestconditionforeachpossibleoutcome.Rulesthatdescribehowcalculationsshouldbedone
shouldresultinatleastonetestconditionforeachformofcalculation.Forexample,whencalculatinga
graduatedshippingchargewiththreedifferentresultsbasedonthevalueoftheshipment,wewould
identifyatleastonetestconditionforeachofthegraduatedvalues.

ScenarioBasedTestIdentification
Scenariobasedtestingistheuseofreallifescenariostoderivetests.Therearevariouskindsof
scenarios.Thescenariobasedtestingthumbnaildescribesawiderangeofscenariotypesthis
introductiontouchesononlyafewofthem.Acommontypeofscenariobasedtestisthebusiness
workflowtest.Thesetestsexercisethesystemundertestbyidentifyingcommonandnotsocommon
endtoendsequencesofactionsbythevarioususersofthesystemasaparticularworkitemispassed
frompersontopersonasitprogressestowardssuccessfulcompletionorrejection.Wecanexaminethe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 197

workflowdefinitionslookingforpointsintheworkflowwheredecisionsaremade,anddefinesufficient
workflowscenariostoensurethateachpathoutofeachdecisioniscovered.
Aparticularlyinterestingformofscenariotestisthesoapoperatestinwhichthetesterdreamsupa
particularlytorturousscenariothattakesthesystemundertestfromextremesituationtoextreme
situation.Thenamecomesfromitssimilaritytoasoapoperatelevisionprogramwhichcondensesmany
days,monthsandpotentiallyyearsofextraordinaryeventsinpeopleslivesintoshortmelodramatic
episodes.Thisformoftestidentificationisgoodforthinkingoutsidethebox.
Scenariosabouthowsoftwareisinstalledbyapurchasercouldleadtoidentificationofpotential
compatibilityissuesandtheneedforcompatibilitytesting.

ModelBasedTestGeneration
Inmodelbasedtestgenerationwebuildadomainorenvironmentalmodelofthesystemundertests
desiredbehaviour(expressedinmathematicaltermsorinsomeotherabstractnotation)anduseitto
generatealltherelevanttestcases.Forexample,whentestingafunctionthattakes4parameterseach
with3possiblevalues,wecouldgenerate81testconditions(3*3*3*3)byiteratingthrougheachvalue
foreachparameter.Forlargeandcomplexsystems,thenumberofsuchcaseswillbehuge.Various
guidingorselectiontechniquesareusedtoreducethetestcasespace.Tofullydefinetheexpected
result,wewouldneedtohaveanindependentwaytodeterminetheexpectedreturnvalue,perhapsa
ComparableSystemTestOracleoraHeuristicTestOracle.Generationofthetestsfromthemodelmay
befullyautomatedormanual.

IdentifyingNonfunctionalTests
Thetestcasesusedtoassesscompliancetothenonfunctionalrequirementscanbeidentifiedinmuch
thesamewaysasthoseforfunctionalityrequirementswiththemaindifferencebeinghowthe
requirementsarediscoveredandenumerated.

OtherTestIdentificationPractices
Allofthesepracticescanbeappliedbyasinglepersonworkingaloneattheirdesk.Butasingleperson
canbebiasedorblindtocertainkindsoftestconditions;therefore,itisbeneficialtoinvolveseveral
peopleinanytestidentificationactivities.Onewaytodothisistoreviewthetestconditionswithat
leastoneotherperson,aformoftestreview.Anotherpracticeispairedtestinginwhichtwopeople
worktogethertoidentifythetestconditions(ortowriteorexecutetests.)Wecanalsousegroup
techniqueslikelisting,brainstormingorcardstormingtoinvolvelargergroupsofpeople
[TabakaOnCollaboration].
Allthesepracticescanbenefitfromthejudicioususeofchecklistsandheuristics.Thesecanbeusedto
triggerthoughtprocessesthatcanidentifyadditionaltestsorentirecategoriesoftests.Theyarealso
usefulwhendoingexploratorytesting.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 198

18.3.2 TestAuthoring/TestDesignPractices
Nowthatwehaveoflistofthingswewanttotest,ourtodolist,wecangetdowntodesigningthetest
cases.Akeytestdesigndecisioniswhetherwewillprepareaseparatetestcaseforeachtestcondition
oraddressmanytestconditionsinasingletestcase.Thereisnosinglebestwayasitdependsverymuch
onhowthetestswillbeexecuted.Whenhumantesterswillbeexecutingthetestcasesitmakesalotof
sensetoavoidexcessivetestenvironmentsetupoverheadbytestingmanytestconditionsinasingle
testcase.Thehumantesterhastheintelligencetoanalysetheimpactofafailedteststepanddecide
whethertocontinueexecutingthetestscript,abandontestexecutionortoworkaroundthefailedstep.
Automatedtestsarerarelythisintelligentthereforeitisadvisabletotestfewertestconditionspertest
case.Inthemostextremecase,typicalwhenautomatingunittests,eachtestcaseincludesasingletest
condition.

TestsasAssets
Testsareassets(notliabilities)thatneedtobeprotectedfromlossorcorruption.Theyshouldbe
managedwiththesamelevelofcareanddisciplinethatisusedformanagingtheproductcodebase.
Thismeanstheyshouldbestoredinaversioncontrolledrepositorysuchasasourcecodemanagement
(SCM)system.Thelineupofteststhatcorrespondtoaparticularlineupofproductcodeneedstobe
labelledortaggedinthesamewaytheproductcodeislabelledsothattheteststhecorrespondtoa
specificproductcodebuildcanberetrievedeasily.

TestsasDocumentation
Regardlessofwhetheratestcasewillbeexecutedbyapersonoracomputer,thetestcaseshouldbe
writteninsuchawaythatapersoncanunderstanditeasily.Thisbecomescriticalforautomatedtests
whenthetestsneedtobemaintainedeitherbecausetheyhavefailedorbecausewearechangingthe
expectedbehaviorofthesystemundertestandweneedtomodifyallthetestsforthechanged
functionality.
Manyofthepracticesusedforidentifyingtestconditionscarrythroughtotestcasedesignbutthe
emphasischangestoenumeratingthestepsofthetestcaseanddeterminingwhattheexpected
outcomeshouldbeforeachtest.Forusecasetestswemustenumeratetheuseractionsthatcausethe
particularpathtobeexercised.Wealsoneedtoincludestepstoverifyanyobservableoutcomesaswe
executethestepsandawayofassessingwhethertheoutcomematchesourexpectations.(Seethe
section17.3.5onResultAssessmentPractices.)

PickinganAppropriateLevelofDetailforTestScripts
Aswedefinethestepsofourtestsitisveryimportanttoensurethatthelevelofdetailisappropriate
forthekindoftestwearewriting.Forexample,inabusinessworkflowtest,eachstepofthetestcase
shouldcorrespondtoanentireusecase.Ifweweretousethesametestvocabularyandlevelofdetail
asinausecasetest,ourworkflowtestswouldbecomeexceedinglylongandreadersofthetestwould
haveahardtimeunderstandingthetest.Thisisaclassicexampleofnotbeingabletoseetheforestfor
thetrees.Soapoperatestsarewrittenmuchlikeaworkflowtestexceptthatthestepsand
circumstancearemoreextreme.Again,eachstepinthetestshouldcorrespondtoanentireusecase.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 199

BusinessRuleTesting
Businessrulestestscanbedesignedmuchlikeusecasetestsbyinteractingwiththesystemundertest
viatheuserinterface.Iftherearealotoftestconditions,itmaymaketestingproceedmuchmore
quicklyifweuseadatadriventestapproachwhereinweenumerateeachtestconditionasarowina
tablewhereeachcolumnrepresentsoneoftheinputsoroutputs.Thenwecansimplywritea
parameterizedtestcasethatreadstherowsfromthetableoneatatimeandexercisesthesystem
undertestwiththosevalues.Anevenmoreeffectiveapproachrequiresmorecooperationwiththe
productdevelopmentteambecauseitinvolvesinterfacingthetestcasethatreadstherowsofthetable
directlytotheinternalcomponentthatimplementsthebusinessrule(wecallthemsubcutaneous
tests).Thisapproachresultsinautomatedteststhatexecutemuchfasterthanteststhatexercisethe
softwarethroughtheuserinterface.Thetestscanusuallyberunmuchearlierinthedesignphaseofthe
projectbecausetheydontevenrequiretheuserinterfacetoexist.Thesetestsaremuchmorerobust
becausechangestotheuserinterfacedontaffectthem.Thesetestsareparticularlywellsuitedtotest
drivendevelopment.

ModelBasedTesting
Amoresophisticatedwayofusingmodelsistogenerateexecutabletestcasesthatincludeinputvalues,
sequencingofcallsandoracleinformationtochecktheresults.Inordertodothat,themodelmust
describetheexpectedbehaviourofthesystemundertest.Modelbuildingiscomplexbutisthekey.
Oncethemodelisbuilt,atool(basedonsomemethodornotation)istypicallyusedtogenerateabstract
testcases,whichlatergettransformedintoexecutabletestscripts.Manysuchtoolsallowthetesterto
guidethetestgenerationprocesstocontrolthenumberoftestcasesproducedortofocusonspecific
areasofthemodel.

UsabilityTesting
Thedesignofusabilitytestingrequiresanunderstandingofthegoalsofuserswhowillbeusingthe
systemundertestaswellasthegoalsoftheusabilitytestingitselfandthepracticestobeused.The
goalsoftestingwillchangefromtestsessiontotestsessionandthepracticeswillevolveastheproject
progresses.Earlyroundsofusabilitytestingmaybefocusedongettingtheoveralldesignrightandwill
involvepaperprototypes,storyboards,andWizardofOztesting.Laterroundsoftestingaremore
likelytoinvolvetestingrealsoftwarewiththepurposebeingtofinetunethedetailsoftheuser
interfaceanduserinteraction.Allofthesetests,however,shouldbebasedontheusagemodelsdefined
aspartofUserModelingandProductDesignpractices.

OperationalTesting
Functionalrequirementstendtofocusontheneedsoftheendusersbutthereareotherstakeholders
whohaverequirementsforthesystem.Theneedsoftheoperationsdepartment,thepeoplewho

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 200

supportthesoftwareasitisbeingused,needtobeverifiedaspartoftheacceptanceprocess.The
specificneedsmayvaryfromcasetocasebutcommonformsofoperationalacceptancetestinginclude:

Testingofinstallers,uninstallersandsoftwareupdates.

Testingofbatchjobsforinitialdataloading

Testingofdatareformattingforupdatedsoftware.

Testingofdatarepairfunctionality.

Testingofstartupscriptsandshutdownscripts.

Testingofintegrationwithsystemmonitoringframeworks.

Testingofadministrativefunctionalitysuchasusermanagement.

Testingdocumentationcontent.

ReducingtheNumberofTestsNeeded
Ifwehavetoomanytestconditionstobeabletesteachone,wecanusevariousreductiontechniques
toreducethenumberoftestconditionswemustverify.Whenwehavealargesetofpossibleinputsto
verify,wecanreducethenumberoftestcasesweneedtoexecutebygroupingtheinputvaluesinto
equivalenceclassesbasedontheexpectedbehaviorofthesystemundertest.Thatis,anequivalence
classincludesalltheinputsforwhichthesystemundertestshouldexhibitthesameorequivalent
behavior(includingbothendstateandoutputs.)Wethenselectafewrepresentativeinputvaluesfrom
eachequivalenceclassforuseinourtests.Whenthevaluesarenumericandordered(e.g.,integersor
reals)wepickthevaluesrightattheboundariesbetweenthedifferentbehaviors,atechniqueknownas
boundaryvaluesanalysis(BVA).Whentheyarenominal,i.e.,theyrepresentclassificationofbehaviours
withnonaturalordering(e.g.,afinitesetofartbitrarystringsorenumerationtypeswithnomeaningful
orderinginthecontext),wecandesignasingledatadriventestforeachequivalenceclassandrunthe
testforeachinputvalueintheclass.
Ifwehaveseveralinputsthatcaneachvaryandwesuspectthatthebehaviorofthesystembasedon
theindividualinputsisnotindependent,weavoidtestingallcombinationsofinputvaluesbyusing
combinatorialtestoptimizationtoreducethenumberofdistinctcombinationswetest.Examples
include:

Manyindependentvariationsorexceptionpathsinausecase.

Manydifferentpathsthroughastatemodel.

Algorithmsthattakemanyindependentinputvaluesthateachaffecttheexpectedoutcome.

Manysystemconfigurationsthatshouldallbehavethesameway.
ThemostcommonvariationofcombinatorialtestoptimizationisknownasPairwiseorAllPairstesting;
itinvolvespickingthesmallestsetofvaluesthatensurethateveryinputvalueispairedwitheveryother
valueatleastonce.Thistechniqueisusedsofrequentlythatthereisa
website[PairWiseOnAllPairsTesting]dedicatedtolistingthemanyopensourceandcommercial

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 201

programsthatexisttohelpuspickthevaluestouse.Ithasbeenshownthatpairwisetestingprovides
bettercoverageandresultsinfewerteststhanrandomtesting.

18.3.3 TestExecutionPractices
Thedetailsofhowthetestsareexecutedvarygreatlyacrossthedifferentkindsoftestsbutseveral
thingsarecommon.Akeyoutcomeoftestexecutionisthedatathatwillbeusedasinputtothe
subsequentreadinessandacceptancedecisions.Thoughvariousprojectcontextswillrequiremuch
moreextensiverecordkeepingthanothers,itisreasonabletoexpectaminimumlevelofrecord
keeping.Thisrecordkeepingconsistsofthefollowingkeypiecesofinformation:

Whattestshavebeenrun,bywhom,whenandwhere?

Whatresultswereobserved?

Howdidtheycomparetowhatwasexpected?

Whatbugshavebeenfoundandlogged?
ThecomparisonwithexpectationsisdescribedinthesectionResultAssessmentPractices.

FunctionalTestExecutionPractices
Thenatureofatestdetermineshowweexecuteit.Dynamictestsinvolverunningthesystemundertest
whilestatictestsinvolveinspectingvariousartefactsthatdescribethesystem.Dynamicteststypically
fallintooneofthefollowingcategories:

AutomatedFunctionalTestExecutionThisinvolvesusingcomputerprogramstorunthetests
withoutanyhumanintervention.Thetestautomationtoolsetsupthetestenvironment,runs
thetests,assessestheresultsandreportsthem.Itmayevenincludeloggingofanybugs
detected.Thetestsmaybestartedbyahumanorbyanautomatedtestschedulerorstarted
automaticallywhencertainconditionsaresatisfiedsuchaschangestothecodebase.

ManualtestexecutionThisinvolvesapersonexecutingthetest.Thepersonmaydoallsteps
manuallyormayusesomeautomationaspowertoolstomakethetestinggofaster.Thehuman
mayadjustthenatureofthetestastheyexecuteitbasedonobservedbehaviorortheymay
executethestepsofatestcaseexactlyasdescribed.

ExploratorytestexecutionThisisaformofmanualtestexecutionthatgivesthetestermuch
morediscretionregardingexactlywhatstepstocarryoutwhiletesting.Eachtestsessionis
usuallyscopedusingatestcharter.

NonfunctionalTestExecutionPractices
Nonfunctionaltestsaresomewhatdifferentfromfunctionaltestsinseveralways.Asintimatedbytheir
name,parafuncationaltestsspanspecificfunctionsofthesystem.Staticnonfunctionaltestsmay
involverunningtoolsthatanalysethesoftwareinquestionortheymanyinvolvereviewerswholookat

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 202

thecodeorotherartefactstofindpotentialdesignorcodingdefects.Dynamicnonfunctionaltestsmay
involverunningtoolsthatinteractwiththesystemundertesttodetermineitsbehaviorinvarious
circumstances.

StaticNonfunctionalTestPractices
Staticanalysisisdonebyexaminingthecodeorhigherlevelmodelsofthesystemtounderstandcertain
characteristics.Specificformsofstaticanalysisincludethefollowing:
ADesign/ArchitectureReviewisusedtoexaminehigherlevelmodelsofthesystemtounderstand
certaincharacteristics.Themostcommoncharacteristicsofinterestincludecapacity/scalability,
responsetimeandcompliancewithstandards(internalandexternal.)
ASecurityReviewisaspecializedformofDesignorArchiecturereviewthatinvolvesexaminingthe
architectureandthecodelookingforwaysamalicioususerorprogrammightbeabletobreakintothe
system.
StaticCodeAnalysisinvolvesexaminingthesourcecodeeithermanuallyorusingtoolstoascertain
certaincharacteristicsincluding:

Reachabilityofcodesegments

Correctusageofkeylanguageconstructssuchastypesafety

DynamicNonfunctionalTestPractices
Performanceandstresstestsaregoodexamplesofnonfunctionalteststhatrequirespecializedtools.
Sometimeswehavecomplexorlongrunningtestproceduresthatexercisethesystemundertestjustto
seewhatwillhappen;thereneednotbeenumeratedexpectationsperse.

18.3.4 ResultAssessmentPractices
Thevalueinexecutingatestcaseistodeterminewhetherornotsomerequirementhasbeensatisfied.
Whileasingletestcannotprovetherequirementhasbeenmet,asinglefailingtestcancertainlyprove
thatithasnotbeenmetcompletely.Therefore,mosttestcasesincludesomeformofassessmentor
checkingofactualresultsagainstwhatweexpect.Thisassessmentcanhappeninrealtimeasthetestis
executedoritmaybedoneafterthefact.Thischoiceisapurelypragmaticdecisionbasedonthe
relativeeaseofoneapproachversustheother.Therearethefollowingthreebasicapproachesto
verifyingwhethertheactualresultisacceptable.

Comparetheactualresultwithapredeterminedexpectedresultusingacomparatorwhichmay
eitherbeapersonoracomputerizedalgorithm.

Examinetheactualresultforcertainpropertiesthatavalidresultmustexhibit.Thisisdone
usingaverifier.

Justrunthetestsandnotchecktheresultsatall.Thismaybeappropriatewhenthetestingis
beingconductedexpresslytogatherdata.Forexample,thepurposeofusabilitytestingisto
findoutwhatkindsofissuespotentialusershaveusingtheproduct.Wewouldntreportan

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 203

individualusabilitytestsessionashavingfailedorsucceeded.Rather,weaggregatethefindings
ofalltheusabilitytestsessionsforaspecificpieceoffunctionalitytodeterminewhetherthe
designofthesystemundertestneedstobechangedtoimproveusability.

UsingComparatorstoDetermineTestResults
Themostconvenientformofassessmentiswhenwecanpredictwhattheactualresultsshouldbeina
highlydeterministicfashion.Themechanismthatgeneratestheexpectedresultissometimescalleda
trueoracleandthereareseveralwaysthetestresultscanbegenerated.Teststhathaveatrueoracle
areveryusefulwhendoingAcceptanceTestDrivenDevelopmentbecausethereisacleardefinitionof
whatdonelookslike.
Whenthereisanexistingsystemwithsimilarfunctionalityandweexpectthenewsystemtoproduce
exactlythesameresults,itmaybeconvenienttousetheexistingsystemasaComparableSystemTest
Oracle.
Whenwehaveanewsystemforwhichnocomparablesystemexists,weoftenhavetodefinethe
expectedresultsmanually.ThisisknownasaHandcraftedTestOracle.Oncethesystemisupand
runningwemayalsohavetheoptionofcomparingsubsequentreleasesofthesoftwarewithprevious
versions,anapproachwecallPreviousResultTestOracle.Thisistheapproachthatmostrecordand
playbackorcapture/replaytesttoolsuse.InsomecasesitmaybeappropriatetoforgotheHand
CraftedTestOracleandwaitforthesystemtogenerateresultswhicharetheninspectedbyaperson,a
HumanTestOracle,beforebeingusedasaPreviousResultTestOracle.Thisapproachforgoesthe
benefitsofAcceptanceTestDrivenDevelopment.

UsingVerifierstoDetermineTestResults
Whenwecannotpredetermineexactlywhattheexpectedresultshouldbe,wecaninsteadexaminethe
actualresultforcertaincharacteristics.Ratherthanhaveanoracledescribewhattheresultshouldbe,
weasktheoracletomakeajudgementastowhethertheresultisreasonablegiventheinput(s).This
approachhastheadvantageofnotrequiringthatwepredeterminetheexpectedresultforeach
potentialinput.Themaindisadvantageisthatwemayacceptsomeresultsthatsatisfytheinvariantbut
whicharenotactuallycorrect.
Forexample,aHumanTestOraclecouldexamineageneratedgraphictodeterminewhethertheycan
recognizeitasanacceptablerenderingofanunderlyingmodel.Oracomputeralgorithmcouldverify
thatwhentheactualresultisfedintoanotheralgorithmtheoriginalinputvalueisrecovered.The
programthatimplementsthisalgorithmissometimescalledaHeuristicTestOracle.HeuristicTest
Oraclesmaybeabletoverifysomeresultsareexactlycorrectwhileforotherresultsitmayonlybeable
toverifytheyareapproximatelycorrect.
Forexample,wecouldwriteatestscriptthatstepsthroughallintegerstoverifythatthesquareroot
functionreturnstherightvalue.Ratherthaninspecttheactualvaluesreturnedbyeachfunctioncalland
comparethemtoahandcraftedtestoracleoracomparablesystemoracle,wecouldinsteadmultiply
theresultbyitselfandverifythatwegetbacktheoriginalnumberwithinaspecifiedtolerance,say,+/
.001.Inthisexample,weshouldalsotestagainstanotherinvarianttoensurethattheactualresultisnot

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 204

negativewhichwouldclearlybeatestfailure.Acomputerizedheuristicoracleissometimescalleda
verifier.

LoggingBugs
Oneofthekeyreasonsfortestingandreviewsistofinddifferencesbetweenexpectationsandreality.
Whenwedofindsuchadifferenceitisimportantthatitbeloggedsothatitcanbefurtherinvestigated,
prioritizedandtheappropriateactiondetermined.Theinvestigationcouldrevealittobeabug,a
misunderstandingbytestersabouthowthesystemwasintendedtobeused,missingdocumentation,or
anyofamyriadofotherreasons.Toavoidpresumingtheoutcome,weprefertocalltheseconcerns.To
allowtheinvestigationtobeconductedefficiently,itisimportanttologthekeycharacteristicsofeach
concern.Ataminimum,weneedtologthefollowing:
1. Theexactstepsrequiredtoreproducetheproblem.Thismayrequirererunningthetesta
numberoftimesuntilwecanconfidentlydescribeexactlywhatittakestocausetheproblem
tooccur.
2. Whatactuallyoccurred.
3. Whatweexpectedtohappen.Weshouldprovideasmuchdetailaswouldbenecessaryfor
thereadertounderstand.Weshouldnotjustrefertoarequirementbutratherdescribe
exactlywhatweexpected,whathappened,andhowwhatactuallyoccurredwasdifferent
fromwhatweexpected.

Everypotentialbugreportshouldbegivenacleartitlethatdescribesspecificallywhatwastested;this
avoidsconfusionbetweenbugswithverysimilartitlesyetcompletelydifferentexpectedandactual
behaviours.
Referto[Kaneretal,Chapter.Reporting and Analyzing Bugs.] on bug reporting guidance and
http://blogs.msdn.com/productfeedback/archive/2004/09/27/235003.aspxforanadditionalexample.

18.3.5 TestMaintenancePractices
Sometestsareonlyintendedtoberunoncewhilesomeareintendedtoberunmanytimesoveralong
periodoftime.Somekindsoftestsholdtheirvaluelongerthanothers;somekindsoftestsdeteriorate
veryquicklybecausetheyaresotightlycoupledtotheSUTthatevensmallchangestotheSUTmake
themobsolete.Teststhatareexpectedtobeusedmorethanoncemaywarrantanupfrontinvestment
toensurethattheyarerepeatableandrobust.
Usefultechniquesformakingtestsmorerobustincludethefollowing:

Buildmaintainabilityintothetests.Writethetestsatappropriatelevelofabstraction.Dont
couplethetesttoanypartofthesystemitisnttesting.Dontprovideanyunnecessaryor
irrelevantdetailinanyofthestepsofthetest.Strivetodescribetheteststepsinbusiness
ratherthantechnicalterms.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 205

Designthesystemundertestfortestability.Designingfortestabilityiscommonpracticein
computerhardwarebutistoooftenneglectedinsoftwaredesign.Designthesystemtomakeit
easytoputitintoaspecificstate.Makeiteasyfortestprogramstointerfacewiththesystem
throughapplicationprogramminginterfaces(API)ratherthanforcingprogramstouse
interfacesintendedforhumans.Writingthetestsbeforethesystemisdesignedisagoodway
toinfluencethedesigntosupporttestability.

Refactortheteststoimprovemaintainability.Testsshouldbeassets,notliabilities.Teststhat
arehardtounderstandarehardtomaintainwhenthesystemundertestismodifiedtomeet
changingrequirements.Automatedtestsinparticularshouldberefactoredtoavoid
unnecessaryduplicationandirrelevantinformation.SeetheTestEvolution,Refactoringand
Maintenancethumbnail.

18.4 Summary
Thischapterintroducedthepracticesweusewhiledefining,executingandmaintainingindividualtests.
Whilesomeofthesepracticesarespecifictofunctionaltestingandothersarespecifictononfunctional
testing,theoveralllifecycleofatestisconsistentforbothcategoriesoftests.Thepracticesusedfor
readinessassessmentaremoreorlessthesameasthoseusedforacceptancetestingalthoughthe
specifictestsproducedforeachwilldependontheoveralltestplanasdescribedinChapter16
PlanningforAcceptance.

18.5 WhatsNext?
Thenextchapterdescribespracticesrelatedtomanagingtheacceptanceprocess,especiallyhowit
relatestomonitoringandreportingontestprogressandtheresultsandmanagingtheprocessof
decidingwhichbugstofix.

18.6 Resources
[PairWiseOnAllPairsTesting]Pairwise.OrgAwebsitededicatedtocatalogingallthetoolsavailablefor
allpairstesting.http://www.pairwise.org/tools.asp
[TabakaOnCollaboration]Tabaka,JeanCollaborationExplainedAddisonWesley.NJAddisonWesley.
NJ[Kaneretal]Kaner,C.etal.TestingComputerSoftware,2/e.Wiley,1999.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 206

Chapter19 ManagingtheAcceptance
Process
Thepreviouschapterintroducedtheindividualtestlifecycleandthepracticestheassessorsusefor
identifyingtestconditions,designingthetests,executingthetestsandassessingtheoutcomes,and
maintainingthetests.Thischapterintroducesthemanagementpracticesweusewhileexecutingthe
testsandwhilemakingtheacceptancedecision.
Theprocessofacceptingsoftwareinvolvesmanyactivitiesthatgeneratethedataweuseasinputinto
theacceptancedecision.Eachoftheseactivitiestakestimeandconsumesvaluablehumanandother
resources.Awellmanagedacceptanceprocesswillusetheseresourceswiselywhileapoormanaged
processhasthepotentialtowastetheseresources,delaytheacceptancedecisionandevencompromise
thequalityofthedecisionmade.
Whileweareexecutingourtestplans,wewanttoknowtheanswerstotwoimportantandinterrelated
questions:

First,howistestingprogressing?Whenwillwehaveahighconfidenceassessmentofthe
productquality?Willitbeintimetomakeourreadinessoracceptancedecision?

Second,whatisourcurrentassessmentoftheproductquality?Isitgoodenoughtorelease
whethertocustomersortotheacceptancetesters?Whatactionsdoweneedtotaketomakeit
goodenough?

19.1 TestSchedulingPractices
Theoverallschedulethatdefineswhichtestswillberunandwhenshouldhavealreadybeendefinedas
partofthetestplan.Thatincludesthestrategicdecisionsaboutwhetheralltestingisdoneduringafinal
testphaseorincrementallythroughouttheproject.Inthissectionwefocusonthetechniquesweuse
forschedulingoftestexecution.Asareminder,undertestexecutionweincludebothdynamictesting
whentestcasesarerunthatinvolveexecutingsoftwareinthesystemundertestandstatictestingthat
isperformedintheformofreviewsorinspectionswithoutactuallyrunningcodeinthesystemunder
test.

19.1.1 PlanDrivenTestScheduling
Thetraditionalapproachtotestschedulinginvolvesdefiningaperiodoftime,sometimescalledatest
cycle,inwhichonecompleteroundoftestingcanbecompleted.Thetestplandefineshowmanytest
cyclesareplanned.Thedetailsofwhathappensineachtestcycleoftenemergeastheprojectunfolds.
Oneapproachistodefineadetailedprojectplanforthefirsttestcycle,onewhichdefinesthesetof
testingactivitiesthatwillbedoneonadaybydaybasis.ProjectmanagementtoolssuchasPertorGantt
chartsmaybeusedtodocumentthisdetailedtestexecutionplan.(SeePlanDrivenTestManagement.)

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 207

19.1.2 SessionBasedTestScheduling
Thealternativetodefiningadetailedprojectplanoftestactivitiesistoscheduleaseriesoftestsessions
andcreateabacklog(prioritizedlist)oftestsessionchartersthatareexecutedinthesetestsessions.
Eachsessionwouldbeofastandardduration,say90minutes,andmostchartersshouldbeexecutable
withinonesession.ThisSessionBasedTestManagementismosttypicallyusedformanaging
exploratorytestingbutcouldalsobeusedforexecutionofscriptedtesting.Wheneachtestsessionis
completedthetestermarksthetestcharterasoneofcompletedorneedsoneormoreadditional
sessions,orsuggestsadditionaltestchartersforfuturetestsession.Thisprovidesagoodindicationof
howtestingisprogressingasillustratedinfigureK(TestCharterBurndown)whichshowsthenumberof
testchartersdroppingovertimebutwiththeoccasionalupwardsspikeasnewtestchartersare
identified.


FigureK
TestCharterBurndown
Inthischart,thebarsrepresentingtheoriginalchartersshowthechartersthatweredefinedbeforetest
executionbegan.ThebarslabelledAddedChartersstackedontopofthemrepresentnewchartersthat
wereidentifiedwhiletestexecutionwasoccurring.TheTotalLeftlineindicateshowmanytestcharters
remaintobefullfiledwhiletheChartersCompletedindicatesthenumberofchartersalreadyfulfilled.
Wecangetagoodideaatanypointduringtestexecutionofwhentestingwillbecompletedby
projectingtheTotalLeftlineouttotherightuntilitintersectszerocharters.Thatistheearliestprobably
completiondate.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 208

19.1.3 SelfOrganizedTestScheduling
Athirdapproachisoftenusedbyagileorselforganizingteams.Allthetestingtasksarepostedonawall
chart,variouslycalledabigvisiblechart[JeffriesOnBigVisibleCharts]oraninformationradiator
[CokburnOnAgileDevelopment],andteammemberssignuptodospecifictestingtasks.Astheyfinish
onetask,theymarkitdoneandpickanothertasktoperform.ThisiscalledSelfOrganizedTest
Scheduling.

19.1.4 EventTriggeredTestScheduling
Someformsofautomatedtestingcanbesetuptorunautomaticallyatcertaintimes(suchasevery
nightat2am)orwhencertaineventsoccur(suchaswhensomeonechecksinachangetopartofthe
codebase.)Theresultscanbepostedonawebsite,automaticallyemailed/IMdtoallteammembers,
orcommunicatedbyamulticoloredlightdisplayorlavalampsintheteamworkspaceorbyaniconin
everyonescomputerssystemtrayorsidegadget(foronesuchtoolseeTeamBuildMonitor
[LambOnTeamBuildMonitor]).ThispracticeisoftencalledContinuousIntegration
[MartinFowlerOnContinuousIntegration]orCIforshort.Teamsthatusecontinuousintegrationtypically
adoptastopthelinementalitytofailingtests.Thegoalistopromotecodehealthearly.Whenevera
testfailureisreportedbytheautomatedtestharness,fixingthefailedtestbecomesthetoppriorityfor
everyoneontheteam.Thisisthesoftwareequivalentofthestopthelinepracticeusedinlean
manufacturing.Oncethefailureisfixed,everyonecangobacktoworkingontheirrespectivetasks.This
focusonkeepingtheproductcodehealthyimprovesqualityandandreducesthedurationofthe
acceptancetestcycle.
Anotherapproachistopreventchangesthatbreakthebuildfrombeingcheckedin.TheGatedCheckin
featureoftheTeamFoundationServerdefersadeveloperscheckinuntilitcanbemergedand
validatedbyanautomatedbuild.Passingautomatedtestsorvalidatingresultsofthestaticanalysis
couldareexamplesofthevalidationpolicies.FormoreinformationonTeamFoundationBuild,see
[MSDNOnTeamFoundationBuild].

19.2 TestProgressReportingPractices
Whenwefirststartexecutingthetestswedontknowwhetherthequalityishighorlowbutwedo
knowthatwedonthaveahighdegreeofconfidenceyet.Astestingprogresses,weshouldbegettinga
betterideaofwhatthequalitylevelisandhowmuchlongeritwilltakeustogettotherequiredlevelof
confidence.Thisisverysimilartotheconeofuncertaintyconceptthatpredictsthecompletiondate
and/orcostofdevelopingthesoftware.FigureXillustratestheconeofuncertaintyforqualityfora
typicalproject.Theinitialguestimatewasanywherebetween10and100personmonths.Asthe
requirementswerebetterunderstood,therangereducedsomewhatbutasuddendiscoveryof
additionalscoperaisedtheestimatesonceagain.Aneffortwasmadetodescopetheprojecttorecover
theoriginaltimelines.Workcreepslowlyraisedboththeupperandlowerlimitandtheestimatesfinally
convergewhentheproductisacceptedasis.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 209

140 Initial Scope NeededExtra


Guess Increases Test&FixCycle
120 Descoped
Detailed
Release1
Estimate Acceptance!
100

80
Latest(90%)
60
MostLikely(50%)
Earliest(10%)
40

20 Work
Creep
0


FigureX
ConeofUncertaintyforQuality
Wecan,ofcourse,assumethatwewillhaveareasonablyaccurateassessmentofqualitywhenwehave
finishedallourplannedtesting.Thisassumptionmayormaynotbecorrectbecauseitdependsonhow
effectivelyourplannedtestactivitieswillfindalltheimportantbugs.Thisalsoimpliesaclear
understandingofwhatisimportanttotheproductowner.Thisisverydifficulttoassessaheadoftime.
Wecertainlyneedtomonitorhowmuchtestingworkremainstobeexecutedinthecurrenttestcycle
andwhenweexpecttohaveitallcompleted.Thisappliestoeachtestcycleweexecute.
Sessionbasedtestingintroducesafeedbackmechanismthatexplicitlyallowsustoadjustthetestplans
aswelearnnewinformationbothabouttheproductandabouttheproductownerandtheproduct
ownersneeds.Aseachtestsessioniscompletedthetestersaddanynewlyidentifiedareasofconcern
tothebacklogoftestchartersyettobeexecuted.Developersmayalsosuggestadditionaltestcharters
toaddresstheriskassociatedwiththemodificationstheymaketothesoftwareasaresultofchange
requestsandbugfixesidentifiedbytests.Wemonitorthesizeofthebacklogoftestcharterstogeta
senseofwhetherwearemakingheadway.Astheconfidenceofthetestersanddevelopersimproves
theywillsuggestfewerandfewernewtestchartersandthereforethesizeofthecharterbacklogwould
dropfaster.
Predictingwhenthesoftwarewillbeofgoodenoughqualitytodeliverisdifficultbecausethatinvolves
predictinghowmanytestcycleswewillrequireandhowmuchtimetheproductdevelopmentteamwill
requirebetweenthetestcyclestofixthebugs.(SeeChapter16PlanningforAcceptanceforamore
detaileddiscussiononthistopic.)Thisrequiresmonitoringthebugbacklogandthearrivalrateofnew
bugstoassessandadjustthepredicteddeliverydate.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 210

19.3 AssessingTestEffectiveness
Oneofthechallengesoftestingisassessinghoweffectiveourtestsreallyaresothatwecanknowhow
confidentweshouldbeinourassessmentofthequalitywehave.Thereareseveralwaystocalculate
theeffectivenessofthetestsincludingtheuseofcoveragemetricsandusingthefindrateof
intentionallyseededbugs.

19.3.1 CoverageMetrics
Wecanusemetricsliketestcoverageandcodecoveragetocalculatethetheoreticaleffectivenessof
ourtests.Thesemetricscantelluswhatpercentageoftherequirementshasbeentestedandwhat
percentageofthecodehasbeenexecutedbythetestsbutneitheroftheseisadirectmeasureofwhat
percentageofthebugswehavefound.Ofcourse,theprimaryissueisthatwehavenoideaofhow
manybugsreallyexistsoitisprettydifficulttosaywithanycertaintywhatpercentagewehavefound.
Foracautiousapproachtousingcodecoverage,seeBrianMaricksessay
[MarickOnCodeCoverageMisuse].

19.3.2 DefectSeeding
Onetechniquethecanprovideamoredirectmeasureisdefectseeding.Itinvolvesdeliberatelyplacinga
knownsetofdefectsinthesoftware.Asbugsarefoundduringtestingwecanestimatethepercentage
ofdefectsfoundbydividingthenumberofseededdefectsfoundbythetotalnumberofseededdefects
asshowninFigureY.
SeededDefectsFound
PercentDefectsFound
TotalSeededDefects

FigureY
PercentageofBugsFound
WecanprojectthenumberofdefectsyettobediscoveredusingtheformulainFigureZ.
FoundDefects SeededDefectsFound

TotalDefects TotalSeededDefects
Or,
TotalSeededDefects
TotalDefects FoundDefects
SeededDefectsFound
FigureZ
TotalBugsCalculation
ThesecalculationsaredescribedinmoredetailintheTestStatusReportingthumbnail.

19.4 BugManagementandConcernResolution
Akeyoutputoftestingandreviewsisalistofknowngapsbetweenwhattheproductownerexpectsof
theproductandhowitactuallybehaves.Whiletheprogressoftestingisusuallyreportedintermsof

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 211

whichtestshavebeenrunandwhichhavent,thegapbetweenexpectationsandrealityisusually
expressedasalistofknownbugsorissuesthatmayormaynothavetobeaddressedbeforethe
productwillbeaccepted.InChapter16PlanningforAcceptance,weintroducedtheConcern
ResolutionModelwhichdescribeshowconcerns,includingsoftwarebugs,changerequests,issuesand
documentationbugs,canbemanaged.Concernsthatfallintoanyofthesecategoriescouldbe
consideredgating(alsoknownasblockingorblockers),thatis,wouldpreventthesystemfrombeing
accepted.Oncewehavefinishedourfirstfullpassoftesting,presumablyattheendofourfirsttest
cycle,wecanthinkofthesetofgatingbugsasbeingtheoutstandingworklistfortheproduct
developmentteam.Ourgoalistodrivethelistofgatingbugsdowntozerosothatwecanconsider
acceptingandreleasingtheproduct.(Notethatjustbecauseitreacheszerodoesnotimplythatthere
arenogatingbugs,justnonewecurrentlyknowabout.)Theproductownercaninfluencethegating
countintwoways:theproductownercanclassifynewlyfoundbugsasgatingortheycanreclassify
existinggatingbugsasnongatingiftheydecidethatthebugcanbetoleratedbecausethereisalow
enoughlikelihood(reducedprobabilityasdescribedinChapter5RiskModel)itwillbeencounteredor
thereareacceptableworkaroundsintheeventthatitisencountered(reducedimpactpertherisk
model.)

19.4.1 BugTriage
Itiscommonpracticetoclassifytheseverityofbugsbasedontheirbusinessimpact.UsuallyaSeverity1
(orSev1asitiscommonlyabbreviated)bugisacompleteshowstopperwhileaSev5bugismerely
cosmeticandwontimpacttheabilityofuserstousetheproducteffectively.Manyproductowners
insistthatallSev1&2bugsbefixedbeforetheywillaccepttheproduct.Someproductownersconsider
Sev3bugscriticalenoughtoinsisttheyareresolvedbeforetheproductcanbeaccepted.Notethatthe
interpretationoftheseverityscaleismerelyavocabularyforcommunicationoftheimportanceofbugs
betweentheproductownerandtheproductdevelopmentteam;itisentirelyuptotheproductowner
tomakethedecisionwhetherthebugneedstobefixed.Theproductdevelopmentteammayinfluence
thatdecisionbypointingoutsimilaritiesordifferencewithotherbugsthathadthesameseverityrating
orbypointingoutthepotentialimpactaspartofanargumenttoincreaseordecreasetheseverity.They
mightalsopointoutpotentialworkaroundsorpartialfixesthattheyfeelmightjustifyreducingthe
severity.Butultimately,itistheproductownerdecisionsastowhethertheproductisacceptablewith
thebugstillpresent.
Theproductownerneedstobereasonableaboutwhatbugsshouldbeclassifiedasgating.Ifthereare
1,000bugsandtheteamcanfix20bugsperweek,itwilltaketheteam50weekstofixalltheexisting
bugsassumingthatnonewbugsarefoundandnoregressionbugsareintroducedbythebugfixes.Both
theseassumptionsarehighlyoptimistic.Therefore,thecustomerneedstoaskWhichofthesebugs
trulyneedtobefixedbeforeIcanaccepttheproduct?Thisispurelyabusinessdecisionbecauseevery
bughasabusinessimpact.Somemayhaveaninfinitesimallysmallbusinessimpactwhileothersmay
havealargebusinessimpact.Theproductownerneedstobeopinionatedaboutthisandtobeprepared
tolivewiththeconsequencesoftheirdecisionwhetherthatistodelayacceptance,deploymentand
usageoftheproductuntilaparticularbugisfixedortoputthesoftwareintousewiththeknownbug
present.Thereisnopointininsistingthatallbugsmustbefixedbeforeacceptancesimplytobeableto

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 212

saythattherearenoknownbugs.Doingsowilllikelydelayaccruinganybenefitsofthesystem
unnecessarily.Theprocessofdecidingtheseverityofeachbugandwhetherornotitshouldeverbe
fixedissometimescalledBugTriage.

19.4.2 BugBacklogAnalysis
Thislistofknownbugsisacollectionofusefulknowledgeaboutthestateoftheproduct.Mostbug
managementtoolsincludeasetofstandardreportsthatcanbeusedtobetterunderstandthebug
backlog.Theseinclude:

BugFixRateReportDescribestherateatwhichbugsarebeingfixed.

BugArrivalRateReportDescribestherateatwhichnewbugsarebeingreported.A
decreasingarrivalratemaybeanindicationthatthereturnoninvestmentoftestinghas
reachedthepointofdiminishingreturns.Oritcouldjustmeanthatlesstestingisbeingdoneor
thatthetestingisrepeatingitself.Longlivedproductsthatreleaseonaregularcycletypically
findthattheshapeofthecurveisfairlyconstantfromreleasetoreleaseandthiscanbeusedto
predictthereadytodeliverdatequiteaccurately.SeeFigureDBugArrivalRateandFigureE
CumulativeBugsFound.

BugDebtorBugBurnDownReportDescribestherateatwhichthenumberofbugsisbeing
reducedorisincreasing.Theburndownreportaggregatesthebugarrivalrateandthebug
fixingratetoallowustopredictwhenthenumberofgatingbugswillbelowenoughtoallow
acceptingtheproductandreleasingittousers.

BugAgingReportClassifiesbugsbyhowlongithastakentofixthemandhowlongithasbeen
sinceunresolvedbugswerefirstreported.Thelatterwillgiveanindicationofpotentiallevelof
customerdissatisfactioniftheaverageageofbugsislarge.

BugCorrelationReportClassifiesbugsbasedontheirrelationshipwithattributesofthe
systemundertest.Theproductowneristypicallymostinterestedinwhichfeatureshavethe
mostbugsbecausethishelpsthemunderstandthebusinessimpactofacceptingthesystem
withoutresolvingthem.Theproductdevelopmentteam,ontheotherhand,istypically
interestedinwhichcomponents,subsystems,ordevelopmentteamshavethemostbugs
associatedwiththembecausethishelpsthemunderstandwheretheirowninternalprocesses
needtobeimprovedmost.FigureZisanexampleofaBugsbyAreaorBugsbyFeatureTeam
report.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 213

80
70 NumberofBugsFoundbyDate
60
50
40
30
20
Num
10
berof
0 Bug


FigureD
BugArrivalRate

CumulativeBugsFoundbyDate

1400
1200
1000
800
Cumulative
600
BugsFoundby
400 Date
200
0
01/03/2008 01/04/2008 01/05/2008 01/06/2008


FigureE
CumulativeBugsFound.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 214

250

200 Usability
Installation
150 Documentation
Localization
100
Globalization
50 Logging/Tracing
Performance
0
Functionality
Security


FigureZ
BugCorrelationReport


FigureW
BugDebt/BurnDownReport

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 215

19.5 TestAssetManagementPractices
Theartefactsproducedwhileplanningandexecutingtheassessmentprocessareassets;thesetest
assetsneedtobemanaged.Ifrepeatabilityoftestingisconsideredimportant,suchaswhenthesame
testsneedtobeusedforregressiontestingsubsequentreleases,testscriptsandthecorrespondingtest
datasetsneedtobestoredinaversioncontrolledtestassetrepository.Thismaybeadocument
repositorysuchasSharePointoracoderepositorysuchastheTeamFoundationServerrepository,
SubversionorCVS.ThisisdescribedinmoredetailintheTestAssetManagementthumbnail.
Whentestassetsareexpectedtobelonglivedsuchaswhenthesametestswillbeusedtoregression
testseveralreleasesofaproduct,itisimportanttohaveastrategyfortheevolvingthetestassetsto
ensuremaintainability.TheTestEvolution,RefactoringandMaintenancethumbnaildescribeshowwe
cankeepourtestassetsfromdegradingovertimeastheproducttheyverifyevolves.

19.6 AcceptanceProcessImprovement
Theacceptanceprocessconsumesasignificantportionofaprojectsresources.Therefore,itisagood
placetolookwhentryingtoreducecostsandimprovetheeffectivenessofonesprocesses.Twogood
candidatesforprocessimprovementareimprovingtheeffectivenessofthetestpracticesand
streamliningtheacceptanceprocesstoreducetheelapsedtime.Thelatteristhesubjectofthenext
chapter.

19.6.1 ImprovingTestEffectiveness
Multireleaseprojectsandlonglivedproductswillgothroughtheacceptanceprocessmanytimesin
theirlifetime.Eachreleasecanbenefitfromlessonslearnedinthepreviousrelease,ifwecaretoapply
thelessons.Itisworthconductingoneormoreretrospectivesaftereachreleasetobetterunderstand
whichreadinessassessmentandacceptancetestingactivitieshadthemostimpactonproductquality
andwhichhadtheleast.Thehighlyeffectiveactivitiesshouldbecontinuedinfuturereleasesandthe
ineffectiveonesshouldbereplacedwithsomethingelse.Someteamsmakeitapointtotryatleastone
newpracticeeachreleasetoseeifitwilldetectbugsthatpreviouslyslippedthroughthereadiness
assessmentandacceptancetestingsafetynets.Itisalsoworthanalysingthelistofdefectfoundinthe
usagephasetoidentifyanyshortcomingsinthesafetynet.Bugcorrelationreportscancomeinhandyin
thisexercise.SeeFigureQBugsbyHowFoundReport.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 216

NumberofBugsbyHowFound
350
300
250
200
150
100
50
NumberofBugs
0


FigureQ
BugsbyHowFoundReport.

19.7 Summary
Likeanyprocess,executionoftheacceptanceprocessneedstobemanaged.Thisincludesmonitoring
theexecutionoftheplannedtestingandensuringitresultsindatathatallowsforahighconfidence
acceptancedecision.Akeyaspectofmanagingacceptanceisthemanagingthebugbacklogtogetthe
bestpossiblequalityproductattheearliestpossibletime.Thesetwogoalsareatoddswitheachother
anddecidingwhichtakesprecedenceshouldbeabusinessdecision.Whenbuildingproductisan
ongoinggoaloftheproductownerorganization,continuousimprovementoftheacceptanceprocess
shouldalsobemanagedtofurtherreducetimetousageandimproveproductquality.

19.8 WhatsNext
Theacceptanceprocesstypicallytakesupasignificantportionofaprojectsresourcesandelapsedtime.
Thenextchapterlooksatwaysbothelapsedtimeandresourceusagecanbereduced.

19.9 Resources
[MartinFowlerOnContinuousIntegration]Fowler,M.ContinuousIntegration.
http://martinfowler.com/articles/continuousIntegration.html
[MSDNOnTeamFoundationBuild]http://msdn.microsoft.com/enus/library/bb668958.aspx
[CockburnOnAgileDevelopment]Cockburn,AlistairAgileSoftwareDevelopment:TheCooperative
GameAddisonWesleyProfessional

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 217

[JeffiresOnBigVisibleCharts]RonJeffries,BigVisibleCharts,
http://xprogramming.com/xpmag/bigvisiblecharts/
[LambOnTeamBuildMonitor]http://blogs.msdn.com/jimlamb/attachment/3467297.ashx
[MarickOnCodeCoverageMisuse]BrianMarick,HowtoMisuseCodeCoverage
http://www.exampler.com/testingcom/writings/coverage.pdf

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 218

Chapter20 FineTuningtheAcceptance
Process
ThepreviouschaptersofPartIIIintroducedthepracticesinvolvedinplanningandexecutingthe
acceptanceprocess.Byreadingthosechaptersthereadershouldbeabletogetabasic
understandingofwhatisinvolvedinacceptingsoftware.Thischapterdescribeshowwecanuse
theinformationweobtainwhileexecutingtheacceptanceprocessascluesforhowtoimprove
theproductdevelopmentprocessesofourorganizationtoreducedefectlevelsandtominimize
theelapsedtimeandresourcesconsumedwhilemakingtheacceptancedecision.
Theacceptanceprocessisanecessarybutpotentiallywastefulexercise.Weshouldstrivetokeepitas
simpleandvalueaddingaspossible.Thelongerittakes,themoreitcostsinwastedeffortanddelayed
deliveryofbusinessvalue.Wecantakestepstostreamlinetheacceptanceprocessbyreorganizinghow
wedoreadinessassessmentandacceptancetesting.Buttheacceptanceprocessisjustonepartofthe
productdevelopmentprocess.Theissueswefindwhileexecutingtheacceptanceprocessare
consequencesofthechoiceswehavemadeinhowwestructureourproducts,markets,andworkflows.
Wecanuserthesesymptomstomotivateabetterunderstandingoftheunderlyingproblemsinhowwe
work.

20.1 DebuggingtheAcceptanceProcess
Issuesencounteredduringtheacceptanceprocesscanbevaluablecluesaboutproblemsinhowour
organizationandworkflowsarestructured.Thesecluesprovidehintsabouthowweshouldrestructure
ourorganizationanditsprocessestoworkmoreefficiently.Thefollowingarealistofcommon
symptomsandpossiblesolutions.

20.1.1 OverlyLongDurationofAcceptanceProcess
Anacceptanceprocessthattakesalongperiodoftimehintsatissueswithhowtheorganizationis
structured.Itcanbecausedby:
Toomanygroupseachneedingtotheirownspecializedworkandtoomanyhandoffsbetween
them.Considerusingvaluestreammappingtoidentifywhichstepsaddrealvalueandwhich
couldbeeliminatedentirely(preferable)ordoneinparallel(secondchoice.)Therootcauseis
likelythewaytheorganizationisstructured;changingthestructuremaybeanecessarythough
highrisk/effortendeaverthoughonewithapotentiallyhugepayback.
Toomanytest&fixcyclesbeingneeded.Thisisdescribedbelow.

20.1.2 TooManyDefectsfoundduringAcceptanceProcess
Iftheacceptanceprocessfindsmanydefects,thisislikelyasignthatTheProductDevelopmentTeams():

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 219

Dontknowhowthefinishedproductshouldbehave.
ArenotheldaccountablefordeliveringafinishedproducttotheProductOwneracceptancetesting.
AlackofunderstandingofwhatdonelookslikeisasignthattheProductOwnerhasnotdonean
effectivejobofcommunicatingthenatureoftheproducttotheProductDevelopmentTeam.Thiscould
bebecausetheproductownerdoesntknoweither,oritcouldbeineffectivecommunicationsuchas
occurswhentheprimarymeansofcommunicationiswrittenrequirementspecifications.TheProduct
OwnercanlearnmorequicklywhattheyreallywantbydoingIncrementalAcceptanceTesting.This
givesthePDTmoretimetoaddressanychangesthaniftheacceptancetestingweretobedoneinafinal
AcceptanceTestPhase.Ineithercase,thejointunderstandingoftheacceptancecriteriacanbe
improvedbyincludingacceptancetestsaspartoftherequirementsprocess.Thesetestscanbe
providedbytheProductOwnerordevelopedjointlythroughcollaborationbetweentheProductOwner
andtheProductDevelopmentTeam.
Lackofaccountabilityoccursforseveralreasons.Oneiswhentheproductisdecomposedinto
componentsthatarefarremovedfromtheenduserfunctionality.TheProductDevelopmentTeam
oftendoesntunderstandhowtheircomponentsupportsthebusinessgoalsandthereforebuilds
functionalitythatmayactualbeatcrosspurposestoit.Thisoftenresultsinbugsfoundonlyafterthe
componentsareintegrated.ThisisoneofthedownfallsoftheComponentTeamapproachtoorganizing
thework.
Anotherreasonforlackofaccountabilityiswhenmanagementvaluesscheduleoverquality.Ofcourse
wesayqualityisimportantbutitsmanagementsactionsthatreallycount.Whenmanagement
pressuresdeveloperstomeetunrealistictimelineswearesayingscheduletrumpsquality.Aclear
definitionoftheMinimumQualityRequirement(MQR)iscrucial.WeneedtoencouragetheProduct
DevelopmentTeamtoimproveitsprocessestodeliverbetterqualitybymistakeproofingtheprocesses
asmuchaspossible.TestDrivenDevelomentandautomatedtestsarebothexamplesofhowtodothis.
Oncetheycanbuildsoftwarewithfewerdefects,theywillbeabletodeliverfasteraswell;theinverseis
nottrue.

20.1.3 TooManyTest&FixCyclesNeeded
WhentheproductrequirementshavebeenclearlycommunicatedandtheProductDevelopmentTeam
hasdoneaneffectivereadinessassessment,theacceptancetestingdoesnotfindmanydefects.Those
defectsthatarefoundshouldbefairlyminorandeasilyfixed.Butwhatiftherearealotofdefects?
Thentheproductwillneedtogobackthroughtheconstruction,readinessassessmentandacceptance
testingprocessanothertime.Whenthiscyclehastoberepeatedseveraltimesbeforetheproductis
goodenoughtoconsiderreleasing,itmaybeduetooneofseveralrootcauses:
Newbugsarebeingintroducedbymanyofthefixesfortheexistingbugs.Thiscouldbebecausethe
softwarehasbecomebrittleduetoageandexcessiveinternalcoupling(ormaybeitwasntdesigned
verywellevenwhenitwasnew.)Oritcouldbeduetorushingtheworkandalackofasafetynetto
catchthedefectsbeingintroduced.Thelattercanbeaddressedbyincorporatingautomatedunit
testingandpossiblytestdrivendevelopmenttoavoidinsertingthedefects.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 220

TheproductDevelopmentTeamisnotdoingeffectivereadinessassessmentofeachreleasecandidate.
Thismaybebecausetheyarebeingrushedorbecausetheyarerelyingprimarilyonmanual
regressiontestingandcannothopetoretestalltheaffectedsoftware.Considerintroduceing
automatedregressiontestingatvariouslevels(unit,component,system)toreducethelengthof
timeittakestodoaneffectiveregressiontestcycle.

20.1.4 LotsofDebateBetweenBugsvs.ChangeRequests
Whenisabugadefectandwhenisitachangerequest?Ifthisdiscussionisoccurringalotduringthe
acceptanceprocessitmaybeanindicationthattheProductOwnerandtheproductDevelopmentTeam
arenotstrivingtosolvethesameproblem.Somepossiblerootcausesare:
FixedpricecontractsencouragetheproductDevelopmentTeamtoclassifyeverythingasaCRevenifit
islegimatelyabug.Solutionistoavoidcreatingdysfunctionalrelationshipsthatcomeoutoffixed
pricecontracts.ConsiderusingTargetPricecontracts[PoppendiecksOnLean]instead;these
motivatebothcustomerandsuppliertominimizethechangeswhilemaximizingvalue.Thereisno
distinctionbetweenabugandaCR;theonlydistinctionisbetweenchangesthatareworthmaking
andthosethatthecustomercanlivewithout.
VaugerequirementscausedbyalackofunderstandingoftheProductOwnerastowhattheywanted.
See
VaguerequirementsleadingtolackofclarityaboutwhatPOreallyaskedfor.Thegapbetweenwhatthe
POthoughttheyaskedforandwhattheproductDevelopmentTeaminterpretedtherequestcaused
thebugandthesubsequentdebate.ThesolutionistoimprovethecommunicationbetweenthePO
andtheproductDevelopmentTeam.Thisisbestachievedthroughcollaborationratherthansimply
writingmorecopiousrequirementdocumentation.Thecommunicationcanbesupportedby
detailedexampleswhichcanbeusedassampleacceptancetests.Considerincludingreadiness
assessorsandacceptancetestersintheproductdesigndiscussionsasthiswillusuallyunearth
interestingscenariousthattheProductOwnermayneedtoconsider.

DebuggingtheAcceptanceProcess
Issuesencounteredduringtheacceptanceprocesscanbevaluablecluesaboutproblemsinhowour
organizationandworkflowsarestructured.Thesecluesprovidehintsabouthowweshouldrestructure
ourorganizationanditsprocessestoworkmoreefficiently.Thefollowingflowchartsuggestpossible
solutionsbasedoncommonsymptoms:
Psuedocodeforaflowchart:
IfAcceptanceTakingTooLong
o IftoomanyTest&fixcycles
Iftoomanyregressionbugsintroduced

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 221

See0UseAutomatedTestExecutiontoReduceRegressionBugs
else
o Elseiftoolongpercycle
ConsiderstreamliningtheprocessusingValueStreamMapping
Else(nottakingtoolong)
o Iftoomannewbugsfound
Iffoundbyendusers

IfUsersfindsystemhardtouse
o See0UseIncrementalUsabilityTestingtoDiscoverDesign
DefectsEarlier

Ifbugsfoundinafewspecificareas
o See0IncreaseBreadthofAcceptanceTesting

Elseifbugsfoundeverywhere
o See0IncreaseDepthofAcceptanceTesting
Iffoundbyacceptancetesters

IfPDTisnotdoingthroughreadinessassessment
o IfPDTisorganizedaroundcomponents
See0UseFeatureTeamstoImproveAccountabilityof
ProductDevelopmentTeamforEndUserFunctionality
o ElseIfPDThasaccesstoacceptancetests
Runthemmorefrequently
See0UseAutomatedTestExecutiontoReduce
RegressionBugs
o Else
See0UseAcceptanceTestDrivenDevelpmenttoHelp
ProductDevelopmentTeamUnderstandRequirements
Better

isbeingrushedbymanagementorPOordeadlines
o See0FocusonQualitytoGetSpeedofDelivery

Elseifa

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 222

IffoundbyReadinessAssessors

Iffoundbyindependenttesters
o UseATTDtoimprovecommunicationofdone

SeealsoIffoundbyacceptancetesters
o ElseiftoomanyaccumulatedbugstofixtherebyrequiringBugTriage
Ifduetohighbugarrivalrates

seeIftoomanynewbugsfound
ElseifBugsarentbeingfixedduetolackoftime

EducateProductownerondecidingbetweenBugsvsNewFeatures
Ifduetofearofintroducingnewbugsduetofragilityofcodebase

Findwaystogetcodebaseunderautomatedtestandrefactor
o ElseifdisagreementsbetweenBugsandChangeRequests
IfFixedPriceContracts

See0UseTargetPriceContractstoAlignInterestsofProductOwner
and
Else

See0UseAcceptanceTestDrivenDevelpmenttoHelpProduct
DevelopmentTeamUnderstandRequirementsBetter

o Else
Isthereaproblem?
Thesolutionsreferencedinthisflowchartareelaboratedbelow.

PossibleRemedies
Basedontheresultsoftheanalysisofacceptanceprocessissuesencountered,oneormoreofthe
followingremediesmaybeuseful:

UseTargetPriceContractstoAlignInterestsofProductOwnerandProductDevelopment
Team

UseIncrementalUsabilityTestingtoDiscoverDesignDefectsEarlier
Incorporateusabilitytestingofearlyversionorprototype(includingpaperprototypes)

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 223

UseIncrementalAcceptanceTestingtoDiscoverDefectsEarlier

UseIncrementalAcceptanceTestingtoHelpProductOwnerDiscoverRequirementsEarlier

InvolveTestersDuringRequirementsDefiintiontoEnsureCompletenessofRequirements

UseAcceptanceTestDrivenDevelpmenttoHelpProductDevelopmentTeamUnderstand
RequirementsBetter
ProductOwnershouldprovideacceptanceteststoProductDevelopmentTeam
o CollaboratewithProductDevelopmenttodevelopthem

UseAutomatedTestExecutiontoReduceRegressionBugs

UseFeatureTeamstoImproveAccountabilityofProductDevelopmentTeamforEndUser
Functionality

IncreaseBreadthofAcceptanceTesting
Includemorevariedfunctionality/scopewithinthescopeoftesting.

IncreaseDepthofAcceptanceTesting
Usemore/differenttestdesign(e.g.Scenariobasedtesting)andexecutiontechniques(e.g.
ExploratoryTesting)togetbettertestcoverage.

FocusonQualitytoGetSpeedofDelivery
FocusonQuality;speedwillfollowduetolesstimespentfindingandfixingbugs.

StreamliningtheAcceptanceProcess
Theacceptanceprocessisanecessarybutpotentiallywastefulexercise.Weshouldstrivetokeepit
assimpleandvalueaddingaspossible.Thelongerittakes,themoreitcostsinwastedeffort
anddelayeddeliveryofbusinessvalue.itItcanbecomeaseriousimpedimenttobeing
responsivetoourusers(theproductownerscustomers.)Beforewecantakestepsto
streamlineitwemustfirstunderstandit.

20.1.5 UseValueStreamMappingtoUnderstandtheAcceptanceProcess
Wecanimproveourunderstandingofanexistingorproposedacceptanceprocessthroughan
exercisecalledValueStreamMapping.Thisisaformofbusinessprocessmodelingthatfocuses
ourattentionontheratioofelapsedtimeandtheamountofvalueadded.Figure1isavalue
streammapofahypotheticalacceptanceprocess.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 224

Legend:
Value in days
Time in days
Severity 1 & 2 bugs found Avg 3 x Severity 1 & 2 bugs found Avg 0.5 x Valueadding
Bugs found Avg 4 x activities=
FixBugs
Insufficient Quality Data Insufficient Quality Data 60days
25% 25%

0 days
total=211days
Feature Feature Integration Integration
2 days Readiness Readiness Acceptance Acceptance Acceptance Acceptance
Assessment Decision Testing Decision Testing Decision
Construct
Software 2.5 day 5 days 5 days 28%efficiency
5 days 0.5 day 10 days 0.5 day 10 days 0.5 day
40* days 0.5 day 0.5 day 0.5 day
40 days

Testers50% Testers50% Testers50%

Inadequate design 25%

Insufficient information 30% Insufficient information 30%


Insufficient Quality Data 10%

Prepare Security Operations Prepare CMB


Operations
Security Queue Acceptance Acceptance CMB Queue Acceptance
Decision Assessment Decision
Document Decision Documents
1 day Avg 10 2.5 1 day Avg 10
10 days days 0.05 day days 0.5 day 10 days days 0.05 day
0.25 day 5 days 0.5 day 0.25 day

MonthlyReview MonthlyReview
Testers50%
(every20days) (every20days)

Figure1
AsIsAcceptanceProcess
Thisprocesstakesanaverageof211daystoexecuteandprovidesonly60daysofactualvalueresulting
inacycleefficiencyofonly28%.Someofthefactorsthatmakethisprocesstakesolongtoexecuteare:
Sixtydaysofsoftwaredevelopmentoutputissentthroughtheprocessinasinglebatch.Therefore,
thereisalargeinventoryofuntestedsoftwarewhichresultsinmanybugsbeingfoundduring
thereadinessassessmentandacceptancetestingactivities.Thefixingofthesebugsisdone
entirelyonthecriticalpathoftheproject.
Thereareseveralcyclesinprocesseachofwhicharetypicallyexecutedseveraltimes.Forexample,
ReadinessAssessmentsendsthecodeback,onaverage4timesforanadditional5dayselapsed
timewhichaddszerodaysofvaluebecauseitispurerework.Eachcycletakes7.5days(2.5days
fixingand5daysreadinessassessment)thereforethisfeedbackloopadds30daysofelapsed
time.
Sometaskstakelongerthannecessarybecausetheresourcesarenotdedicated.Forexample,the
readinessassessorshaveotherresponsibilitiesandittakesthem5daystodo2.5daysoftesting.
Customeracceptanceisdoneintwoseparatephasesandisfollowedbythreeotherformsof
acceptancedecisionmaking,twoofwhichcansendthesoftwarebackallthewaytobugfixing.
Thesoftwareneedstoberetestedeachtimethesoftwareissentbacktofixbugs.
Thesecurityteamisoverworkedresultinginanaveragewaittimeof10daysforthesecurityreview.
Thepreparationofthesecuritydocumentstakesanaverageof10daysasdevelopersdiscover
whatisrequiredbutonly1dayofthateffortaddsrealvalue.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 225

TheChangeManagementBoard(CMB)meetsmonthlyresultinginanaveragewaitof10business
days.

Note:Thecalculationsassumethatnovalueisprovidedbythebugfixingpassesthroughthesame
activitiesasthisisunplannedrework.Ofcourse,thiswholecalculationisbasedontheassumptionthat
wecanaddvaluebyreviewingwork;anargumentcanbemadethatthewholeacceptanceprocessisa
formofwastecausedbythecustomersinabilitytoaccuratelydescribewhattheywantbuiltandthe
suppliersinabilitytofollowthespecificationadequately.

20.1.6 StreamlinetheAcceptanceProcess
Thisacceptanceprocesscanbestreamlinedbychanginghowtheworkpassesthroughthevarious
readinessandacceptanceactivities.Figure2illustratestheresultofhavingappliedanumber
transformationsontheprocess.ThesetransformationsareinspiredbyLeanThinkingwhichfocuseson
eliminatingwaste.SeethesidebarWasteinSoftwareDevelopmentforadescriptionoftheseven
commonformsofwaste.Whichformsofwastewetrytoeliminatefirstdependsonwhatisimportant
tous.Iftimetomarketisthekeyconsideration,wemaywanttotacklethequeuingdelaysfirstand
thenlookforwaystoreducetheamountofprocessing.Ifcost/resourceconstraintsareholdingusback,
wemaywanttolookforwaystoreducetheamountofprocessingfirst.

Legend:
Value in days PerIterationActivities
Time in days
Severity 1 & 2 bugs found Avg 1 x

Note:AllTesters
Insufficient Quality Data
100%Dedicated
4 Iterations +
25% Repeatedonceper
1 bug fix Feature
iterationplusonefinal
Feature
Acceptance Acceptance bugfixcycle
Testing Decision
Construct
Readiness
Software/
Fixbugs
10 days
Assessment
Readiness
Decision
1 day
1 day 0.1 day
0.1 day
2.5 days
0.5 day
10 days 2.5 days
0.5 day
Insufficient Quality Data
25%
Valueadding
Integration
Acceptance
Integration
Acceptance
activities=
Testing Decision
1 day
58days
0.1 day
1 day
0.1 day Inadequate design
25%
total=110days
Insufficient Quality Data
10%
Operations
Operations
Assessment
Acceptance
Decision
53%efficiency
1 day
1 day 0.1 day
0.1 day

Insufficient information Insufficient information 30%


< 10% CMB
Prepare 1on1 Security PrepareCMB
Security Security Security Queue Acceptance
Readiness Documents
Consultation Document Review Decision Decision
0.5 day 1 day 0.5 day 1 day WeeklyReview Avg 2.5
0.5 day 2 days 0.05 day 10 days days 0.05 day
0.5 day (every5days)
0.05 day 0.25 day

Consultations&
Interviewsprebooked
Figure2
StreamlinedAcceptanceProcess
Thisstreamlinedversionoftheprocessistheresultofthefollowingchanges:

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 226

Wehavebrokentheprojectinto4twoweek(10businessday)iterations.Eachiterationissentthrough
readinessassessment,featureacceptance,integrationacceptanceandoperationalacceptance.Bugs
foundinthefirstthreeroundsoftestingarefixedinthesubsequentiteration.Afterthefourth
iteration,thesoftwareisretestedandtheresultingbugsarefixedinasinglebugfixingiteration.
Contrastthiswiththe4roundsofbugfixasaresultofreadinessassessmentandthreeroundsdue
tofeatureacceptancetesting.Eachroundoftestingtakesroughly30%ofthetimeittookintheAs
Isprocessbecausethebacklogofuntestedsoftwareismuchsmaller.Thisisanexampleofreducing
wasteassociatedwithinventory.
Theacceptancetestsareprovidedtothedevelopmentteamalongwiththerequirements.Thisallows
thedevelopersandthereadinessassessmenttesterstoverifythefunctionalityisworkingcorrectly
beforehandingitofftotheacceptancetesterstherebyreducingthenumberofbugsfoundin
acceptancetesting.Thereadinessassessmentshownasoccurringaftertheiterationactually
happenscontinuously(assoonasthedevelopersaysthefeatureisready)thereforeanybugs
foundinreadinessassessmentarefixedbeforethedeveloperhasshiftedtheirfocusonthenext
fixture.Thisreducesthecostoffixingthefewbugsthatarefoundinreadinessassessmentbya
factorof4.Thisisanexampleofreducingthewasteassociatedwithdefects.
Theacceptancereviewforsecurityhasbeenchangedfromanacceptanceactivitytoareadinessactivity.
Thisallowsittobedoneinparallelwithdevelopment.Ithasalsobeenchangedfrombeingapush
modelwheretheteampreparesadocumenttobereviewedtoapullmodel.Thisstartswitha
consultationwiththesecurityspecialistwhohelpsthedevelopmentteamunderstandthe
appropriatesecurityrequirementsanddesignandwhatneedstobeinthesecuritydocument.This
providesinputintothesoftwareconstructionprocessresultinginasecuritycompliantdesignaswell
asmoreappropriatecontentinthesecuritydocument.Thisreducestheturnbackratefrom30%to
10%andthetotaleffortfrom10daysto3dayswithoutanyreductioninvalue.Theeffortto
producethedocumentisreducedandtheefforttoreadthedocumentisalsoreduced;definitelya
winwinsolution.Afinalsecurityreviewisheldpriortothefinaliterationtoreviewthefinished
documentaswellasanylatebreakingsecurityrelateddesignconsiderations.Thisisanexampleof
eliminatingwastebyreducingdelaysandbyreducingextraprocessing.
Thefeatureacceptancetesting,integrationacceptancetestingandoperationsacceptancetestingare
doneinparallelwithanunderstandingthatanyshowstopperbugsfoundinanyoftheparallel
testingactivitiescanresultinthesoftwarebeingrejected.Thisreducestheelapsedtimetothe
longestofthethreeinsteadofthesumofthethreeactivities.Thiseliminateswasteintheformof
delaysincurredwhileonegroupwaitsforanothergrouptofinishtheirwork.
Becauseonly25%ofthefunctionalityisbeingtestedanewineachiteration,theacceptancetestingcan
befinishedin1day.Thisisashortenoughperiodthatthetesterscanfocusontheoneprojectand
finishit1dayelapsedtime.Thisisntaformofwastereductionasmuchasitisanexampleof
improvingflowbymakingtheoutputoftheprocesslessburstyviasmallerbatchsizes.Achieving
flowisanotherofthekeyprinciplesofleandevelopment.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 227

TheChangeManagementBoarddocumentationispreparedinparallelwiththefinalbugfixingiteration
andcanbesubmittedtotheCMBassoonasthethreeparallelacceptancedecisionsarepositive.To
furtherreducethedelay,theCMBnowmeetsweeklyfor1hourratherthanmonthlyforahalfday.
Thisreducestheaveragewaitfrom10businessdaysto2.5.Thisisanexampleofreducingwaste
associatedwithwaiting.
Thecollectedimpactofthesechangestotheacceptanceprocesshasreducetheaverageelapsedtime
to94daystoexecute110daysofvalueaddedprocessingthatdelivers58daysofvalueforacycle
efficiencyof53%.

Note:Thecalculationsassumethatvalueisprovidedbytheplanneddevelopmentiterationsbutno
valueisprovidedbythefinalbugfixingiteration.Thecycleefficiencywasbasedonthecumulative
durationincludingactivitiesdoneinparalleleventhoughtheydidnotaddanyelapsedtime.

Summary
Asignificantportionoftheelapsedtimebetweenthefinishofdevelopmentofsoftwareandwhenthe
softwarecanstartprovidingvaluetothecustomerisconsumedbytheacceptanceprocess.Theelapsed
timecanbereducedsignificantlybybuildingsoftwareincrementallyanddoingincrementalreadiness
assessmentandacceptancetestingaseachincrementofsoftwareisfinished.Differentkindsof
inspectionandtestingactivitiescanbedoneinparallelwithdevelopmentofthelaterincrements
reducingtheamountofworkthatneedstobedoneonthecriticalpathbetweencompletionof
developmentandfinalacceptance.Thetypesoftheissuesdiscoveredduringacceptanceprocesscanbe
usedtodiagnoseorganizational,culturalandprocessissuesintheorganizationsinvolved.Changingthe
processestoavoidtheissuesisusuallymorebeneficialthanimprovingtheefficiencyofaddressingthe
issuesoncefound.

WhatsNext?
VolumeIhasintroducedanumberoftoolsyoucanuseforreasoningabouthowyouacceptsoftware.It
hasdescribedwhentousealargenumberofacceptancerelatedplanning,requirements,inspection,
reviewandtestingpractices.Youmaywanttoresearchsomeofthesepracticesinmoredetail.Volume
2inthisseriesdescribesmanyofthesepracticesinmoredetailwhileVolume3providesexamplesof
theartefactsthatmighthavebeenproducedonafictionalproject.

Sidebar:FormsofWasteinSoftwareDevelopment
ThesevencommonwastesofmanufacturingcanberememberedusingtheacronymTIM
WOOD.Insoftware,thereisanequivalentofeachformwasteasexemplifiedbythelist
providedbyMary&TomPoppendieckintheirbookImplementingLeanSoftware

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 228

Development[PoppendiecksOnImplementingLean].Unfortunately,thesoftwarespecific
namesdontformaniceacronymsoweveincludedbothnameshere.Thesoftwarespecific
nameshavea*nexttothem.

T=ThewasteofTransport(Handoffs*)
Transportationiswastebecauseitdoesntaddanyvaluetotheendproductbutitincreases
thecostandtheelapsedtime.
Insoftware,transportcorrespondstohandoffsbetweenpartiesusuallyviadocuments.The
requirementsdocumenthandedbyspecifierstosoftwaredevelopersisoneexample,the
designdocumenthandedbyarchitectstodevelopersisanother.Thepreparationofthese
documentstakeslargeamountsofeffortoftenmuchmorethancommunicatingthesame
informationverbally.Handoffsusuallyresultinlossofinformationandthisistypicallyworse
whenthehandoffisasynchronous(e.g.documents)ratherthanfacetoface.Seethesidebar
UsingWholeTeamstoBuildProductsinChapter1forwaystoreducethenumberofhandoffs.

I=Inventory(PartiallyDoneWork*)
Inventoryisbadbecauseitcostsmoneytoproducetheinventoryandoftencostsmoneyto
storeormanagetheinventory.Inventoryalsomasksissuesbydelayingwhendefectiveparts
arediscovered.Justintimemanufacturingisallaboutreducinginventorytothelowestlevels
possible.
Insoftware,inventoryisanyartifactthathastakenefforttoproducebutwhichisnotyet
providingthecustomerwiththevalueexpectedtobeprovidedbyhavingthesoftwareinuse.
Somecommonformsofinventoryinclude:
UntestedsoftwareSoftwarethathasbeenwrittenbutnotyettested
Softwarewithalonglistofbugsuntriagedortriagedbutnotfixed
RequirementsdocumentsDocumentsthatprovidedetaileddescriptionsoffunctionality
thatwontbebuiltrightaway.

M=Movement(TaskSwitching*)
Movementisbadbecausewhileaworkerismovingtheycantbeproducing.Thisaddsno
valuetotheproductbutreducesproductivityoftheworkertherebyincreasingcost.
Insoftwaredevelopment,theequivalentofmovementistaskswitching.Thisiscausedby
askingpeopletoworkonseveralthingsatthesametime.Everytimetheyswitchbetweenone
taskandanothertask,timeiswastedwhiletheyreestablishtheirworkingcontext.

W=Waiting(Delays*orQueuing)
Wheneverworkstopswhilewaitingforsomethingtohappen,itisaformofwaste.
Commoncausesofwaitinginsoftwareincludewaitingforapprovals,waitingforclarificationof
requirementsandwaitingforslowcomputersortoolstofinishtheirprocessing.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 229

O=Overprocessing(LostKnowledge*orProcessInefficiency)
Overprocessingiswastecausedbydoingunnecessarystepsordoingasteplongerthan
necessary.Thisaddsnoadditionalvaluebutitdoesaddcost.
Insoftware,overprocessingisanystepsintheproductionprocessthatarerequiredtoproduce
highqualitysoftware.Themostcommonexampleistheproductionofanydocumentation
thatnoonewilleverread.Anothercommonformishavingtorediscoverinformationthatwas
lostsomewherebeforeitcouldbeused.

O=Overproduction(ExtraFeatures*)
Overproductionisproducingtoomuch.Itislikeinventoryexceptthatitisfinishedproduct
whileinventoryisworkinprogress.
Insoftware,overproductionisthedevelopmentofunnecessaryfeatures.Ithasbeenreported
[REF]thatonaverage80%offeaturesdevelopedarerarelyorneverused.Thisisclearly
overproduction!

D=Defects
Defects,bugs,problemreports,usabilityissuesallrequireextraworktoanalyse,understand
andaddress.Thisextraworkispurewaste.Itisexactlythesameinsoftware.

Resources
[PoppendiecksOnLean]Poppendieck,M.LeanSoftwareDevelopmentAnAgileToolkit,AddisonWesley
Professional#ISBN10:0321150783ISBN13:9780321150783,
[PoppendiecksOnImplementingLean]Poppendieck,M.,Poppendieck,T.ImplementingLeanSoftware
DevelopmentFromConcepttoCash,AddisonWesley,.(Seerefinanotherchapter.)

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 230

AppendixAOrganizationStereotypes
andReaderPersonas
Thisappendixdescribestwostereotypicalorganizationsandthepeopleinvolvedinmakingorproviding
dataforthereadinessandacceptancedecisions.Thepeoplearedescribedaspersonas.
Notetoproduction
Reviewers:Thesetwostereotypesareincludedtoactasabusinesscontextforthepersonas.The
personasareincludedfortworeasons:
1. Toremindtheauthorsandreviewersofourprimarytargetaudience.
2. Togivereadersafiguretoempathizeorassociatethemselveswith.Thisshouldhelpthem
pickthepersonathatmostcloselyresemblestheirownjobresponsibilities.

Contents Primarypersonas Secondarypersonas

JosephKrawczak
LisaAndrews
Thinkingabout YvetteKirwan
VolumeI EricAndersen
Acceptance ChattyCathy
FredSpook
YvetteKirwan
Managing JosephKrawczak
VolumeII ChattyCathy
Acceptance LisaAndrews
FredSpook
EricAndersen
LisaAndrews
Functional JosephKrawczak
EricAndersen
VolumeIII Acceptance YvetteKirwan
Testing ChattyCathy
FredSpook
Operational LisaAndrews
ChattyCathy
VolumeIV Acceptance EricAndersen
FredSpook
Tetsing YvetteKirwan

OrganizationStereotypes
ITOrganizationofaBusiness
XYZCorpisalargetransportationcompany.Computersoftwarehasbeenintegraltothecompanys
businessforseveraldecades.Thecompanyhasseveralthousandapplications.Atthehighendare

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 231

systemsbuiltbytheITdepartmentrangingfromlargemainframesystemswritteninCobolandPL1to
WebbasedapplicationswritteninASP.NET.Atthelowendthereareenduserapplicationsbuiltin
VisualBasicandAccessandspreadsheetapplicationsbuiltbyendusersscatteredthroughoutthe
businessareas.
TheITdepartmentwillbuildthesoftwarebasedonthespecificationsprovidedbythebusinessand,
afterdoingitsownsystemtesting,expectsthebusinesstoconductuseracceptancetesting.Thereisno
separateQAdepartmentbuteachdevelopmentteamhasatleastonetestingspecialistwhohelpsthe
teamtesttheirownsoftware.TheITdepartmentsqualitygateprocessalsorequiressignoffsfromtheIT
ArchitecturegroupandtheOperationalsupportgroupbeforesoftwarecanbedeployedintothe
Acceptanceenvironment.Asaresultofrecentlyintroducedcorporategovernanceinitiatives(suchas
SOX),companypolicyrequiresapprovalbytheCorporateSecuritydepartmentbeforeanynew
applicationcanbedeployedintoproduction.
TheITArchitecturedepartmentincludesateamofdataarchitectswhoseprimaryresponsibilityisto
ownthecorporatedatamodel.Thecompanyviewsthedataintheapplicationsasacorporateassetand
thedataarchitectshavebeenworkinghardtomanagetheduplicationofdataacrossthem.TheIT
departmentalsomanagesthelifecyclesofthevariousITtechnologiesusedinthecompany.New
technologieshavetobeapprovedbytheITArchitecturegroupbeforetheycanbeusedandold
technologiesmaybemarkedforreplacement.TheCorporateSecuritydepartmenttakesanactive
interestinallnewapplicationsandmustapprovethesecuritymeasuresbeforeanynewapplicationcan
bedeployed.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 232

AProductCompany
AdventureWorksisaproductcompanythatsellsbusinessproductivityimprovementproductsthat
includebothhardwareandsoftware.Someoftheproductsaresoldashardwarewithembedded
softwareandsomeassoftwarepackagesthatleverageexistinghardwareproducts.Thecompanyhasa
separateUxDandQAdepartments.TheUxDdepartmentisinvolvedinpartoftherequirements
definitionactivitiesassociatedwithdesigningtheproductaswellasensuringthattheproductshavea
consistentlook&fell.TheQAdepartmentteststhesoftwarebuiltbytheProductDevelopmentgroup
andadvisestheProductManageronwhetherthesoftwarecanbereleasedtothemarket.

ReaderPersonas
PersonasOftenFoundintheITOrganizationofaBusiness
ProductManagerorProductOwner
EricAndersenisaproductmanagerwhohasbeenaskedtoworkwithanagiledevelopmentteamasthe
ProductOwnerforoneofthesoftwareonlyproducts(anewsoftwarereleasethatworkswithexisting
hardwaresuppliedbythecompanyandthirdparties.)Becausethecompanysproductsareconsidered
hightechnology,Ericisfairlytechsavvy.Hehasagoodunderstandingofthecustomersrequirements
butdoesgetabitlostwhenthediscussiondescendstothelevelofnetworkingprotocols.

KeyquestionsEricneedstofindanswersto:
WhatkindsofinformationdoIneedfromtheQAdepartmenttodecidewhetherornotto
acceptthesoftware?Whatqualitycriteriamustbemet?HowdoIknowtheyhavebeen
met?

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 233

Whoisresponsibleforverifyingthesecriteriaaremet?Whoisresponsibleforensuring
thesecriteriaaremet?
HowdoestheUxDteamparticipateinacceptancetestingofthesoftware?
Howcanthereleasestrategy(multiplereleases,alphas,betas,etc.)helpme?
E.g.Improveusability,fitnessforpurpose
Getrevenueearlier
Viralmarketing
Whatistheprocessforaddressingdeficienciespriortorelease?

BusinessPersonintheMarketingDivision
LisaAndrewsisabusinesspersoninthemarketingdivisionofXYZCorp.Lisaisinherlatetwenties
(relativelyyoungforherpositioninthesomewhatconservativecompany)soshehasgoodcomputer
usageskillsbutisnotbyanystretchoftheimaginationconsideredtechnical.ThisisLisasfirstIT
project.Lisahasbeenaskedtobethebusinessleadoftheprojecttoreplaceseveralexistingsystems
usedbythestaffinthemarketing,salesandcontractsdepartmentswithasingleintegratedsystem
usingthelatesttechnologies.Inthisroleshewillhavethefinalsayonwhetherthesystemisacceptable
tobedeployed.Shewillalsoberesponsibleforensuringthatthebusinessbenefitsusedtojustifythe
projectareactuallyrealized.Shewillbeassistedonthisprojectbyseveralusersoftheexisting
applicationsaswellasabusinessanalystandatestprofessional.Shewillneedtooverseethisteamasit
definesthedetailedrequirementsforthenewsystem.

KeyquestionsLisaneedstoanswer:
Whatkindsoftestingwillmyteamneedtodotohelpmedecidewhetherornottoacceptthe
software?
Howcanthereleasestrategy(multiplereleases,alphas,betas,communitytechnologypreviews,
etc.)helpme?
E.g.Improveusability,fitnessforpurpose,userbuyin
Getcostsavingsearlier
WhenshouldIplanondoingAT?HowmuchtimeshouldIallowforAT?Howmucheffortwillbe
involved?HowmuchofstaffdoIneedtodoit?
Testlastorincremental?
AmIinvolvedinATordoIdelegateittosomebodyelse?Frommyorg?athirdpartylab?
WhoelsecanhelpmewithATifIdonthavetheskill/confidence
WhatkindsoftestingdoIexpecttheITdepartmenttodobeforeaskingmyteamtodoacceptance
testing?

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 234

WhatinformationdoIneedITtoprovidetomeaboutthekindsoftestingtheyhavedoneandhow
willIusethatinformationindecidingwhetherornottoacceptthesoftware(orevenstart
acceptancetesting?)
WhatinformationdoIneedtoprovidetoITpriortomedoingAT?

OperationsDepartmentManager
ChattyCathyisthemanageroftheoperationsdepartment.Sheisresponsiblefordeployingallserver
basedapplicationsandkeepingthemrunningaccordingtoeachapplicationsservicelevelagreement
(SLA).Shesbeenwiththecompany30yearshavingworkedherwayupfrombeinganadministrative
assistantinoneofthebusinessunits.Herformaltrainingwassecretarialschoolandsheslearned
whateversheneededtoknowonthejob.WhenshewasfirstmovedintoI.T.itwasasatesterfor
desktopapplicationsbutshequicklymovedtoupmanagethedesktopsupportteam.Whenthat
functionwasoutsourced,shewasaskedtotakeovertheoperationsgroup.Thatwastwoyearsagoand
shehasonlyrecentlybecomecomfortablewithherunderstandingofhowthegroupfunctions.Shehas
yettomakeanychangestothegroupsprocesses.

ITSecuritySpecialist
FredSpookistheITSecuritySpecialistintheCorporateSecuritydepartment.Hesaformerspy(IfItold
youwhichagencyIworkedforIwouldhavetokillyou,hesayswithawinkwheneverhesasked.)who
movedintotheprivatesectorwhenthefirstmajorsecuritybreecheswerepublicized.Hisbackground
hasmadehimratherparanoidandhedelightsinskeweringthedevelopmentteamswithhisarsenalof
nastysecurityscenarios.Asaresult,mostapplicationsfailtheirfirstsecurityreviewandrequire
significantarchitecturalrefactoringbeforepassing.Thebetterdevelopmentteamshavelearnedto
approachhimforanearlyofftherecordsecurityauditassoonastheirdesignhasstabilized.

PersonasOftenFoundinaProductCompany
TestManager
JosephKrawczakisthemanageroftheproductverificationgroupatAdventureWorks.Histeamof
professionaltestersverifiesthattheproductsbuiltbytheproductdevelopmentgroupworkasspecified
bytheproductmanagementteam.Hisbackgroundisintestingtelecomsystemswherehemanageda
30persontestingdepartment.Whenthetelecomindustrydownsizedhewasgivenasizablepackage
andwashiredbyABCCorptoheadupitsnewlycreatedIndependentTestdepartment.Hismandate
wastoimprovethequalityoftheproductsproducedbyAdventureWorks.Histeamofprofessional
testersisexpectedtoprovidehimwiththedatarequiredtodeterminewhethertheproductis
acceptable.Heprovidesthisdata,alongwithaaccept/rejectrecommendationtotheProductManager
whomakesthefinaldecision.Heisalwayspreparedtobackuphisrecommendationwithaclear

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 235

descriptionofthebusinessimpactofthebugsthathefeelsmustbefixedbeforehewouldbe
comfortableadvisingtheproductbeaccepted.

UserExperienceSpecialist
YvetteKirwanisaFineArtmajorwhogotajobasagraphicartistinamediacompany.When
conventionaladvertisingworktookadownturn,shewasmovedintotheproductdesigngroupwhere
shedidgraphicdesignforwebbasedsoftwareapplications.Shequicklyrealizedthatthegraphicswere
mainlyeyecandyandthattherealvaluewasprovidedbytheinteractiondesignsoshewentbackto
schoolparttimetogettrainedasaninteractiondesigner.Withthistrainingunderherbeltshequickly
becameakeyplayerinthedesignofsoftwareintensiveproductsatthecompany.Shewashiredby
AdventureWorks.specificallytoworkonthenewproduct.Herroleistodefinetheinteractiondesign
basedonmarketrequirements,usermodelsandfeedbackfromtheproductmanagerandthe
development.Thedesignsneedtobeimplementableinacostefficientmannerusingthetechnology
stackusedbytheproductdevelopmentteam.WhilesheistechnicallypartoftheUxDTeam,shesitsin
thedevelopmentareawiththedevelopmentteamasshehasfoundthistobeamoreeffectivewayto
transitiondesignknowledgetothedevelopersandtheirmanager.Shehasalsofoundittobeusefulto
enlistfriendsandpeopleoffthestreettodoinformalusabilitytestingofpaperprototypesaswellasthe
actualsoftware.Thisgivesherrealdatashecanusetohelpinfluencethetestdepartments
understandingtheusers.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 236

AppendixBKeyPoints
ThechaptersinPartIIdescribedtheprocessleadinguptotheacceptancedecisionfromthe
perspectivesoftheBusinessLead,ProductManager,TestManager,DevelopmentManager,User
ExperienceSpecialist,SolutionsArchitect,EnterpriseArchitect,OperationsManager,andLegal.For
convenience,wesummarizedthekeypointshere.ThereareKeyPointscommontoallrolesandothers
specifictoeachindividualrole.
ThefollowingtablelistsKeyPointsthatarerolespecificandwhatrolestheyarespecificto.
<Thetablewillbeupdatedandavailableontestingguidance.codeplex.com>


Figure1
KeyPointsspecifictoindividualroles

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 237

KeyPointsApplicabletoAllRoles
Treatacceptanceasaprocessnotanevent:Appreciatethefactthatacceptancetestingwillnot
beasinglediscreteevent,butaprocesswithadistinctstartandend.

Definethescopeoftesting:Haveaclearunderstandingofwhatwillbetestedbythesupplier
andwhatyouareexpectedtotest.Typically,supplierisexpectedtodothorough(e.g.)testing
ofallcapabilitiesandqualitycharacteristics(suchasperformance,reliability,usabilityetc.)
beforehandingoffsystemforacceptancetesting.Doesthesupplierhavetheenvironmentsfor
integrationtestingwithoutinhousesystems?

Delegatedecisions:Ifyoudonthaveenoughtimeorunderstandingtotakepartintheproject,
assignaproxytorepresentyouandhaveaclearunderstandingofthekindsofdecisionsthey
canmakewith/withoutconsultingwithyou.

Delegatework/testing:Haveaclearunderstandingofyour(organizations)abilitiesw.r.t.
testingskills.Contractotherpartiestohelpyoutestifnecessary.Butdontunderestimatethe
effortinvolvedinmanagingthetestingoutsourcerastheywillneedtolearnyourbusiness
domainandrequirementsbeforetheycanbeeffective.

Understandthetestenvironments:Understandwhereyouwilltest.Beclearonhowyoullget
thenewreleasecandidatesdelivered

Viewtestsasrequirements:Acceptancecriteria/testsshouldbearticulatedandprepared
beforethesoftwareisbuiltandsharedwithvendorasawaytofleshoutthedetailsofthe
requirements.

Estimateacceptanceeffort&durationwhilerealizingitisdifficulttoestimate:Estimatingthe
effort&durationofacceptancetestingishard;timeboxingthefinalacceptancetestingcycleis
commonbutrequirespastexperiencetocomeupwithreasonableduration/effort.Ifindoubt,
guesshighandallowforseveralcyclesofacceptancetestingtogivethevendortimetofixbugs.

Useincrementalacceptancetestingtoreduceuncertaintyofestimating:Canavoidsomeof
theuncertaintyandmostoftheriskbydoingincrementalacceptancetesting.Bugsfound
earlieridentifyambiguityinthespecwhilethereisstilltimetoaddressitandcanbefixedlong
beforethefinalacceptancetestphase

Understandthedecisionmakingprocess:Makesureyouhaveaclearunderstandingofthe
decisionmakingprocess.OtherpotentialdecisionmakersincludeTestManagers,Security
Manager,UserExperience/UsabilityManager,Architect,LegalDepartment,etc.Ensureyou
haveagreementonwhethertheymakeadecisionorprovidedataorarecommendationtoyou.

Communicateinformationflowclearly:Ensureanyonewhoyouexpecttoprovideyouwith
dataonwhichtobaseyourdecisionunderstandswhatdatatheyareprovidingyouandwhen.
E.g.TheTestManagerprovidestestresultstoyouandyoumaketheacceptancedecision.

Understandtheoperationalrequirements:Makesureyouunderstandtheoperational
requirementsforyoursystem.Whetheryouaresellingasoftwarepackagetocustomersor

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 238

sellingSoftwareasaService(SaaS),beawarethatwhoeveroperatesthesoftwarewillhave
operationalrequirementsoverandabovethefunctionalrequirementsoftheendusers.

Clarifydecisionmakingvsdecisionsupporting:Makesureyouhaveaclearunderstandingof
whetheryouarethegatekeeper(acceptancedecisionmaker)orasupplierofinformationto
theProductManagerforhertomakethedecision(preferred)

Maximizelearning:Workwiththeproductownertoensurethateveryoneontheteamhas
accesstothedefinitionofsuccessintheformofacceptancetestsbeforethesoftwareisbuilt

KeyPointsforBusinesslead
3. Treatacceptanceasaprocessnotanevent:Appreciatethefactthatacceptancetestingwill
notbeasinglediscreteevent,butaprocesswithadistinctstartandend.
4. Recognizetheremaybemanydecisionmakers:Theprocessmayrequiredecisionsbymany
partieseachsupportedbydatacollectionactivities.Eachpartylikelyhasverydifferent
requirementsandinterests;youneedtobeawareofthemandcommunicatethemtothe
supplier(s)toensuretheyaddressthem.
5. Definethescopeoftesting:Haveaclearunderstandingofwhatwillbetestedbythe
supplierandwhatyouareexpectedtotest.Typically,supplierisexpectedtodothorough
(e.g.)testingofallcapabilitiesandqualitycharacteristics(suchasperformance,reliability,
usabilityetc.)beforehandingoffsystemforAT.Doesthesupplierhavetheenvironmentsfor
integrationtestingwithoutinhousesystems?
6. Delegatedecisions:Ifyoudonthaveenoughtimeorunderstandingtotakepartinthe
project,assignaproxytorepresentyouandhaveaclearunderstandingofthekindsof
decisionstheycanmakewith/withoutconsultingwithyou.
7. Delegatework/testing:Haveaclearunderstandingofyour(organizations)abilitiesw.r.t.
testingskills.Contractotherpartiestohelpyoutestifnecessary.Butdontunderestimate
theeffortinvolvedinmanagingthetestingoutsourcerastheywillneedtolearnyour
businessdomainandrequirementsbeforetheycanbeeffective.
8. Understandthetestenvironments:Understandwhereyouwilltest.Beclearonhowyoull
getthenewreleasecandidatesdelivered
9. Viewtestsasrequirements:Acceptancecriteria/testsshouldbearticulated/prepared
beforethesoftwareisbuiltandsharedwithvendorasawaytofleshoutthedetailsofthe
requirements.
10. Recognizethechoiceofprocessimpactseverything:TwokeystylesofAT:FinalTestPhase
andIncrementalAT.Theformerconcentratesmostacceptanceactivitiesattheendofthe
project.Incrementalacceptancetestingspreadsouttheworkandreducesriskbutrequires
allpartiestoworkindifferentways.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 239

11. Estimateacceptanceeffort&durationwhilerealizingitisdifficulttoestimate:Estimating
theeffort&durationofacceptancetestingishard;timeboxingthefinalacceptancetesting
cycleiscommonbutrequirespastexperiencetocomeupwithreasonableduration/effort.If
indoubt,guesshighandallowforseveralcyclesofacceptancetestingtogivethevendor
timetofixbugs.
12. Useincrementalacceptancetestingtoreduceuncertaintyofestimating:Canavoidsomeof
theuncertaintyandmostoftheriskbydoingincrementalacceptancetesting.Bugsfound
earlieridentifyambiguityinthespecwhilethereisstilltimetoaddressitandcanbefixed
longbeforethefinalacceptancetestphase.

KeyPointsforProductManager
AllBusinessLeadsKeyPointsplus:

Understandthedecisionmakingprocess:Makesureyouhaveaclearunderstandingofthe
decisionmakingprocess.OtherpotentialdecisionmakersincludeTestManagers,Security
Manager,UserExperience/UsabilityManager,Architect,LegalDepartment,etc.Ensureyou
haveagreementonwhethertheymakeadecisionorprovidedataorarecommendationto
you.

Recognizethatcomplexproductsandorganizationshavecomplexdecisionmaking
processes:Thereareseveraldifferentwaystoorganizetheteamstodealwithverylarge
productsorproductsthataretheamalgamationofseveralproducts/productlines(e.g
SharePoint=Server+Office+IE).Thechoiceshouldconsidertheresultingdecisionmaking
model(e.g.componentteams,featureteams.)

Communicateinformationflowclearly:Ensureanyonewhoyouexpecttoprovideyouwith
dataonwhichtobaseyourdecisionunderstandswhatdatatheyareprovidingyouand
when.E.g.TheTestManagerprovidestestresultstoyouandyoumaketheacceptance
decision.

Ensureallsuppliersunderstandusersandusagemodel:Makesureyouunderstandthe
usersandcommunicatethistobothsupplierandtestorganizations.

Understandtheoperationalrequirements:Makesureyouunderstandtheoperational
requirementsforyoursystem.Whetheryouaresellingasoftwarepackagetocustomersor
sellingSoftwareasaService(SaaS),beawarethatwhoeveroperatesthesoftwarewill
haveoperationalrequirementsoverandabovethefunctionalrequirementsoftheend
users.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 240

KeyPointsforTestManager
Determineyourtestmandate:Makesureyouunderstandwhetheryouaredoingreadiness
assessmentoracceptancetestingorboth.Foracceptancetestingbesurethetestershave
realistictestsfortheuserstheyproxy.Considerusingpersonas.Forreadinessassessmentit
isimportanttohaveagoodrelationshipwiththedevelopmentteamtohelpthem
understandthequalityoftheproductandwhatwillimproveit.

Clarifydecisionmakingvsdecisionsupporting:Makesureyouhaveaclearunderstanding
ofwhetheryouarethegatekeeper(acceptancedecisionmaker)orasupplierofinformation
totheProductManagerforhertomakethedecision(preferred)

Getagreementfromtheproductownerthatthetestcasesreflecttherequirements:Aim
towritetheacceptancetestcasesfirstandprovidethemtothedevelopmentteamsbefore
thesoftwareisbuilttobeusedasaidestoassistinunderstandinghowthesystemshould
behave.ThisisknownasAcceptanceTestDrivenDevelopment.

Planamultiprongedteststrategy:Planforamultiprongedteststrategythatincludes
incrementaltestingandbothscriptedandexploratorytests.

Collaboratewiththedevelopmentteam:Aimforanincrementaldeliveryoffunctionalityto
supportincrementalacceptancetesting.Workwiththedevelopmentteamtodesignfor
testabilitytofacilitateautomatedtestingandenablingthemtobeabletoruntestson
demand.

Dotestautomationwhenfeasibleandinthewayitsupports(notdictates!)meetingyour
testobjectives.Considerhavingadedicatedtestautomationengineer.

Agreeonadefinitionofwhatdoneis:Getthevariouspartiesinvolvedtoagreeona
definitionofwhatdoneisforsoftwarebeforeitgoesthrougheachofthequalitygatesofthe
gatingmodelandincludeachecklistofrequirements.

KeyPointsforDevelopmentManager
Performreadinessassessments:Allsoftwareshouldbeadequatelytestedbythe
developmentteambeforepresentingittothecustomer(s).

Agreeontheminimumcrediblerequirement(MCR)aheadoftime:Theproductowner,test
organizationandthesuppliershouldallagreeontheMCRaheadoftime,thatis,beforethe
softwareisbuilt.

Useacceptanceteststodefinesuccess:UseacceptancetestsbasedontheMQRtodefine
successbeforethesoftwareisbuiltandensurethateveryoneonthedevelopmentteamhas
accesstothemandrunsthetestsastheybuildthesoftware.

Runthetestsaspartofreadinessacceptance:Runalltheteststhattheproductownerwill
runandsomeyouvedefined.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 241

Minimizeuntestedcode:Testasyougo,ideallyincrementaltesting.

Usetestingtopreventbugs:Testscanhelppreventbugsbeingintroducedduringthe
developmentcycle.Includefunctionaltesting,component/unittests,andparafunction
testingasearlyaspossible.

KeyPointsforUserExperienceSpecialist
Understandyourrole:Haveaclearunderstandingwiththeproductowner,product
managerorbusinessleadwhatyourroleshallbeinmakingtheacceptancedecision.

Maximizelearning:Workwiththeproductownertoensurethateveryoneontheteamhas
accesstothedefinitionofsuccessintheformofacceptancetestsbeforethesoftwareis
built.

Definetheusabilitycomponentoftheacceptancecriteriaasearlyaspossible:Workwith
thebusinessleadorproductmanagertodefinetheusabilitycomponentoftheacceptance
criteriaasearlyaspossible.

Addresstheusabilityrisksasearlyaspossible:Theareaswithhighestusabilityimpact
shouldbebuiltandtestedfirstsothatanyusabilityrelatedissuescanbeaddressedasearly
aspossibleinthedevelopmentcycle.

KeyPointsforOperationsManager
Understandroleinacceptancedecision:Youandeveryoneconcernedshouldbeclearon
yourroleinthemakingoftheacceptancedecision.Areyouadataprovidertothe
acceptancedecisionmakeroranacceptancedecisionmaker?

Identifywaystoreducethenumberoftest&fixcycles:Workwiththedevelopment
managertoidentifywaystoreducethenumberoftest&fixcyclesthroughearlydeliveryof
testablesoftwareandincrementalacceptancetesting.

Defineacceptancefromtheoperationalperspectiveasearlyaspossible:Ideallyyouhave
clearlydefinedacceptancecriteriabeforethesoftwareisbuilt.

KeyPointsforSolutionArchitect
Describethewholesolutiontoeveryone:Communicatethesolutioninitsentiretytothe
wholesupplierteamtoavoidtheMypartworksjustfine(evenifthesolutiondoesnt!)
mentality.

Ensureavailabilityofacceptancetests:Workwiththerequirementsanalystsandtestersto
ensurethattheacceptancetestsareavailabletothesupplierteambeforethesoftwareis
built.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 242

Educateabouttradeoffs:Workwiththeproductownertohelpthemunderstandwhatkinds
offunctionalityiseasytobuildandwhatishard.Helpthemgetthemostbangfortheir
money.

Unbundlefunctionality:Workwiththeproductownertohelpthemidentifythelowervalue
functionalitythatcouldpotentiallybedeferredincasethereisaneedtotradeoff
functionalityforqualityordeliverydate.

DefineSLAmetrics

Engageenterprisearchitects:Engagetheenterprisewidearchitects(responsiblefor
infrastructure,standards,technologydecisions,longtermvision,security,dataarchitecture,
etc.)earlysotheycanhelpyouunderstandwhatrequirestheirapprovalbeforeitbecomes
partofthecriticalpath.

KeyPointsforEnterpriseArchitect
Publisharchitecturalacceptancecriteria:Publishgeneralacceptancecriteriasothatall
supplierteamsunderstandtherulesofthegameandwhatkindsofapprovalstheyneedto
getfromyou.

ArticulatecrosssystemintegrationrequirementsandSLAs:Identifyapplicableintegration
requirementsandservicelevelagreements.Makesurethesupplierteamisawareofthem
andthattheyareacceptancecriteria.

Ensurestandardscompliance:Verifythatproductscomplywithapplicablestandards.

Encouragereuse:Verifythatproductteamsareawareofandcapitalizeonopportunitiesto
reusecomponentsanddata.

Engagesupplierteamsearly:Engagesupplierteamsearlytoensurethatinspectionsprevent
defectsratherthanfindthem.

Delegatenoncriticaldecisions:Ifitisnotimportanttobeinvolvedinadecisionthenbow
outandlettheprojectteammakethedecision.

Keepapprovalprocesslightweight:Maketheprocessforseekingguidanceandapprovalas
lightweightaspossible.Emphasizeeffectivenessofcommunicationovercomprehensive
documentation.Avoidoffloadingwork(e.g.moredocumentation)totheprojectteamsto
saveyoueffortifthatresultsinincreasingthetotalwork.

Bepragmatic:Bepragmaticastowhatprogressindividualprojectorproductteamsare
expectedtomaketowardslongtermarchitecturalobjectives.Theyareresponsibletothe
productowner,nottoyou.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 243

KeyPointsforLegal
Includeclearstipulationsofanycontractualimplications:Acceptancetypicallyisa
conditionforrenderingapaymentandaffectstheapplicationofanywarrantyprovisionsand
potentiallyanyremediesavailabletothecustomer.

Discourageonesidedcontracts:Thecontractshouldencouragebothpartiestogettoa
mutuallyagreeableacceptanceasquicklyaspossible.Contractsthatpenalizeoneparty
oftenleadtodysfunctionalbehaviorthatresultsinloseloseoutcomes.

Clearlydefinehowmuchtimeisallowedforacceptancetests:Acceptancetestingisalways
timeboxed.Thereisalimitedamountoftimeforacceptanceandreportingdeficiencies.

Includeacleardefinitionofacceptancecriteria:Besidestestingperformedatthelocationof
supplier,andfunctionalandoperationaltestsperformedwhenthesoftwareisdeployedto
thecustomer,softwaresystemoftenarerequiredtosuccessfullyrunwithcurrentdataina
liveoperatingenvironmentforaperiodoftime,withminimumbugs,beforethesystemis
formallyaccepted.

Knowtheexplicitandimplicitcontractualconditions:Acceptancecanbeviewedintermsof
explicitprovisionsspecifiedinthesoftwaredevelopmentcontractandthetermsimpliedinto
thecontractbylaw(whichalsodependsonthegeographicjurisdiction).

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 244

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 245