AcceptanceTest

EngineeringGuide
VolumeI:ThinkingaboutAcceptance
HowtoDecideifSoftwareisReadyforYou
andYourCustomers

GrigoriMelnik,GerardMeszaros,JonBach

ForewordbyKentBeck

ReleaseCandidate1October26,2009

COMMUNITYPREVIEWLICENSE
Thisdocumentisapreliminaryreleasethatmaybechangedsubstantiallypriortofinalcommercial
release.ThisdocumentisprovidedforinformationalpurposesonlyandMicrosoftmakesnowarranties,
eitherexpressorimplied,inthisdocument.Informationinthisdocument,includingURLandother
InternetWebsitereferences,issubjecttochangewithoutnotice.Theentireriskoftheuseorthe
resultsfromtheuseofthisdocumentremainswiththeuser.Unlessotherwisenoted,thecompanies,
organizations,products,domainnames,emailaddresses,logos,people,places,andeventsdepictedin
exampleshereinarefictitious.Noassociationwithanyrealcompany,organization,product,domain
name,emailaddress,logo,person,place,oreventisintendedorshouldbeinferred.Complyingwithall
applicablecopyrightlawsistheresponsibilityoftheuser.Withoutlimitingtherightsundercopyright,
nopartofthisdocumentmaybereproduced,storedinorintroducedintoaretrievalsystem,or
transmittedinanyformorbyanymeans(electronic,mechanical,photocopying,recording,or
otherwise),orforanypurpose,withouttheexpresswrittenpermissionofMicrosoftCorporation.

Microsoftmayhavepatents,patentapplications,trademarks,copyrights,orotherintellectualproperty
rightscoveringsubjectmatterinthisdocument.Exceptasexpresslyprovidedinanywrittenlicense
agreementfromMicrosoft,thefurnishingofthisdocumentdoesnotgiveyouanylicensetothese
patents,trademarks,copyrights,orotherintellectualproperty.

2009MicrosoftCorporation.Allrightsreserved.

MicrosoftaretrademarksoftheMicrosoftgroupofcompanies.
Allothertrademarksarepropertyoftheirrespectiveowners.

TableofContents

ForewordbyKentBeck 1
Preface:AcceptanceTestEngineeringGuide 3
Introduction 10
ACautionaryTale 20
PARTI.ThinkingAboutAcceptance
Chapter1 TheAcceptanceProcess 27
Chapter2 ElaboratingontheAcceptanceProcess 53
Chapter3 DecisionMakingModel 71
Chapter4 ProjectContextModel 82
Chapter5 SystemsRequirementsModel 91
Chapter6 RiskModel 99
Chapter7 DonenessModel 104
PARTII.PerspectivesonAcceptance
Chapter8 BusinessLeadsPerspective 119
Chapter9 ProductManagersPerspective 131
Chapter10 TestManagersPerspective 138
Chapter11 DevelopmentManagersPerspective 143
Chapter12 UserExperienceSpecialistsPerspective 148
Chapter13 OperationsManagersPerspective 153
Chapter14 SolutionArchitectsPerspective 156
Chapter15 EnterpriseArchitectsPerspective 159
Chapter16 LegalPerspective 161
PARTIII.AcceptingSoftware
Chapter17 PlanningforAcceptance 166
Chapter18 AssessingSoftware 190
Chapter19 ManagingtheAcceptanceProcess 207
Chapter20 FinetuningtheAcceptanceProcess 219
Appendices
AppendixA OrganizationStereotypesandReaderPersonas 231
AppendixB KeyPoints 237

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 1
Foreword

Ifigureit'smyjobinaforewordtotellyouwhyIthinkabookisworthreading.Oftenthisinvolves
settingcontext,placingthebookwithrespecttohistoricalorsocialortechnicaltrends.SometimesI
pointouthowtheauthorshavebeentoohumbleintheirpresentation.SometimesItalkaboutwhatI've
learnedfromabook.InthiscaseIamabletouseallthreetechniques.
Technologistshavespentthelasthalfcenturyswingingonapendulumbetweenbeinglowlyserfs
obeyingthecommandsoftheirbusinessmastersandactingashighpriestsatthealtarofComputing.
Thelastdecadehasseenthefirsttentativestepstowardsamorerealisticandsustainablepositionof
partnershipwithbusinesssponsorsandusers.Thenewemphasisonacceptancetestingisevidencethat
thistrendcontinues.
Isay"emphasis"becauseacceptancetestingisnotnew.Eversincethefirstusertoldthefirst
programmer"That'snotquiteright,"acceptancetestinghasbeenpartofthedevelopmentworkflow.
Sincethosefocusedontechnologyandthosefocusedonbusinessbringdifferentvalues,expectations,
andvocabulary,somemisunderstandingisinevitable.What'sthatProfessorHeisenberg?Oh,yes,thanks
forthereminder.Theappearanceofasystemautomatically(itseems)changesitsrequirements.
Betweenthesefactors,acceptancetestingispartofeffectivedevelopment.
Legitimatelyshiftingrequirementshaveaprofoundeffectonthetiming,role,andtechniquesof
acceptancetesting.Oneofthetrendsindevelopmentoverthepastdecadeisthattestingtakesplace
soonerandmorefrequentlyindevelopment.Evenwiththebulkofacceptancetestingshiftedtojustin
advanceofdevelopment(AcceptanceTestDrivenDevelopment),someonestillneedstobelisteningfor
"Thanksforthat,butwhatIreallywantis..."
Oneimplicationofthegrowingfrequencyofacceptancetestingistheneedforautomationofrepetitive
testingtasks.Thisisnotajobfortestersalone.Toincreasetherateoffeedback,programmersmust
maketheirsystemsmoretestable.Automationfreesresourcesforhigherleveragedmanualtesting.
OnethingIappreciatedinthisbookwastheconsistentconnectionbetweenacceptancetestingandrisk
management.Acceptancetestingisworthwhileforthevalueitaddstodevelopment:increased
probabilityofcustomersatisfaction,reducedriskoferrors,andbettertrackingofprogress.Theimageof
riskmanagementasawaytobuyreactiontimewillstickwithme.Earlyincrementalacceptancetests
canbethescoutssentouttoinformthearmyoftroubletocome.
Oneofthestrengthsofthisseriesofbooksisthevarietyofformatsfortheinformation.Thefirst
volumepresentstheintellectualfoundationsofacceptancetesting.Remainingvolumespresent
concretepracticesthatworkwithinthatframework,amenuof"Here'stechniquesthathaveworked"as
wellasaworkedexampleforthoselookingtounderstandacceptancetestingincontext.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 2

I'llclosebypointingoutwhereIthinkGrigori,Gerard,andJonshumilitygotthebetterofthem.Whatif
softwaredevelopmentwasturnedonitshead?Whatifthewholeprocessworkedbackwardsfrom"The
customerisdelightedwiththesystem"?Beforeyoucouldexpecttohearastatementlikethat,you'd
needtospendalotoftimetestingthesysteminrealisticenvironments.You'dneedtomakesureallthe
easilyunderstoodfeaturesworkedasexpected.Acceptancetestingandacceptancetestdriven
developmentcanencouragethis.Novelfeaturesandinteractionsoffeatureswouldalsoneedtoworkin
asensibleway,anothergoalacceptancetestingcancontributeto.Whenthose"Great,butwhatI'd
reallyliketoseeis..."momentshappen,incrementalacceptancetestingcanhelpthemoccurearly
enoughthatthewholeteamcanrespond.
Astheconcretemanifestationofthedialogbetweenbusinessandtechnology,acceptancetestingcan
playacentralroleinsoftwaredevelopment.Hereinyourhandsisabookthatdistillsyearsof
experiencewithacceptancetesting,abagofideasyoucanreachintoagainandagainasyoutravel
towards:"Thecustomerisdelightedwiththesystem..."
KentBeck,June2009

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 3
Preface:AcceptanceTestEngineering
Guide

WhyWeWroteThisGuide
Acceptingsoftwareisoneofthemostimportantdecisionsforsoftwaretotakeflightandtostart
deliveringvalue.Yet,thereexistslittleguidanceaboutwhatthisdecisionshouldbebasedonandhowto
makeiteffectively.
Overmanyyearsinindustryandappliedresearch,itwasourobservationthatpeopleoften
misunderstoodthechallengesofacceptingsoftware,orworse,didnotevenhaveaclearnotionofwho
isinchargeofacceptance.Manyappearnottobeawareofthearsenalofpracticesandtoolsavailable
tothemtoaddressthosechallenges.Thisledtodisappointments,failedopportunities,brokenbusiness
relationships,andlitigations.Thereexistsarichbodyofknowledgeonsoftwaretesting(withnotable
worksbyBorisBezier,BertrandMeyer,GeraldWeinberg,CemKaner,JamesBach,BrettPettichord,Lee
Copeland,MarkFewster,DorothyGraham,RexBlack,BrianMarick,JamesWhittakertonameafew),
butsoftwareacceptanceingeneral,andacceptancetesting,inparticular,asaprocess,receivedno
serious,wellreasonedorcomprehensivetreatment.Wesetouttofillthisgapwithtwokeyobjectives:
devisementalmodelsforthinkingaboutsoftwareacceptanceandofferactionableguidanceon
achievingsoftwareacceptance.
WhoShouldReadThisGuide
Thisguideisintendedforanyonewhoisinvolvedintheprocessofdecidingthedegreetowhicha
softwareintensiveproductmeetstheacceptancecriteriaofthosewhocommissioneditsconstruction.
Specifically,thisguidewillhelpyouifanyofthefollowingapplytoyou:
Youareinvolvedindecidingwhethertoacceptthesoftwareasithasbeenbuilt.Thisisthe
acceptancedecision.
Youareinvolvedindecidingonwhatacceptancecriteriashallbeusedtomaketheacceptance
decision.
Youareinvolvedincollectingdatathatthepersonmakingtheacceptancedecisionrequiresto
makethatdecision.Thisisacceptancetesting.
Youareinvolvedindecidingwhethertheproductisreadytobeseenbythepeopleinvolvedin
theacceptancedecisionoracceptancetesting.Thisisthereadinessdecision.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 4
Youareinvolvedincollectingdatathatthepersonmakingthereadinessdecisionrequiresto
makethatdecision.Thisisreadinessassessment.
Youareinvolvedindefiningtheexpectationsagainstwhichthereadinessassessmentor
acceptancetestingactivitieswillbeconducted.Thisisacombinationofrequirements
gathering,testplanningandtestdesign.
Youareinvolvedinmanaginganyoftheprecedingactivities.

Thisguidedescribesthepracticesusedbypeopleintheprecedingroles,regardlesswhatyourjobtitle
is.Ifanyofthemdescribesyourrole,youshouldfindsomethingofinterestinthisguide.Appendix
ReaderPersonasdescribesourtargetaudienceinmoredetail.Thesepersonasareusedto:
remindtheauthorsandreviewersofourprimarytargetaudience.
giveyou,thereaderafiguretoempathizeorassociateyourselfwith.Thisshouldhelpyoupick
thepersonathatmostcloselyresemblesyourownjobresponsibilities.
IfanyofthegoalsdepictedinFigure1areapplicabletoyou,theguideandtheonlineknowledgebase
companion(TestingGuidance.codeplex.com)areforyou.

Figure1.
Readergoalsandpathsthroughtheguide.
GuideStructure
Thisguideisstructuredasafourvolumeseries(Figure2):
VolumeIprovidesanoverviewoftheacceptanceprocessandhowacceptancetestingandother
keypracticesfitintotheprocess.Thisvolumeisintendedtobereadfrombeginningtoend.
Itissubdividedintothreemainparts:
VolumeIIcoversAcceptancePlanningandManagementpracticesandpatterns.
VolumeIIIcoversFunctionalAcceptanceTestpracticesandpatterns.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 5
VolumeIVcoversOperationalAcceptanceTestpracticesandpatterns.Operationalacceptance
includesnotonlyperformance,recoverability,availability,securitybutalsotestingofother
nonfunctionalrequirementsandqualities,suchaslearnability,usability,supportability,
upgradeabilityetc.

Figure2.
GuideStructure.
VolumeIisintendedtobereadasanintroductiontotheconcepts,termsandpracticesdescribedin
VolumesIIIV.VolumesIIIVareintendedtobeusedasareference;mostreaderswillnotreadthem
frombeginningtoend.
VolumeIThinkingAboutAcceptanceconsistsof3parts:
PartIThinkingaboutAcceptanceexplainssixmentalmodelsthatareusefulwhen
thinkingabouttheacceptanceprocess.
PartIIPerspectivesonAcceptancedescribestheacceptanceprocessfromthe
perspectivesofkeystakeholdersintwodifferentkindsoforganizations:theInformation
TechnologyDepartmentinabusinessandtheProductDevelopmentCompany.Most

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 6
readersinvolvedintheacceptanceprocessshouldfindsomecommonalitywithatleast
oneoftherolesdescribes.
PartIIIAcceptingSoftwareintroducesthepracticesthatarenecessaryforplanningthe
acceptanceprocess,forperformingacceptancetestingandforimprovingthe
acceptanceprocess.

VolumesIIthroughIVcontainacollectionofthumbnailsandsamples:
AThumbnailisashortoverviewofapracticethatexplainswhatitis,whenyoumaywanttouse
it,therisksthatitmitigates,andanoverviewofhowtoperformthepractice.Thumbnails
alsoincludealistofreferencestopapers,books,andotherresourcesthatprovidemore
completedescriptionsofthepracticesinquestion.Themainpurposeofathumbnailisto
describeatopicwellenoughtoprovideanoverview,serveasamentalreminderfor
someonewhohasusedthepracticeonhowtodoit,andgivesomeoneunfamiliarwiththe
practiceenoughinformationaboutthepracticeanditsapplicabilitytodetermineifthey
wanttolearnmoreaboutit.Someofthesetopicsandpracticeshaveentirebookswritten
aboutthemthatdescribetheconceptsingreaterdetailanddepththanthisguidecould
possiblydo.
ASampleisaanartifactgeneratedbyapplyingdifferentpracticesinafictionalbutrealistic
situationforthefictionalcompanyGlobalBank.Theseartifactsareembeddedinaseriesof
casestudiesofwhattheGlobalBankteammayhaveproducedwhilebuildingthe
application.Thecasestudiesprovidesomecontexttotheindividualartifacts.Theyalso
providecrossreferencestothethumbnails.Theartifactsareintendedtobeusedaswayto
learnmoreabouthowtoperformapractice;theycanalsobeusedastemplatesforyour
ownartifacts.

HowtoReadThisGuide
Thewayyouapproachthisguidewilldependonwhatroleyouhaveandwhatyouwanttolearnabout
theprocessofacceptingsoftware.Dependingonwhatyouwanttodo,youwillwanttoapplydifferent
strategies.
GetanOverviewofAcceptancePracticesandProcesses
StartbyreadingVolumeIifyouwanttodoanyorallofthefollowing:
Learngeneralinformationaboutacceptancetesting.
Findacceptancetestingpractices.
Createaprojectplanthatincorporatessoftwareacceptance.
Justifyaprojectplanthatincorporatessoftwareacceptance.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 7
Justifyanapproachusedforacceptancetesting.
Validatethatyouareontrackwithyouracceptancetestingstrategyorapproach.
Getyourprojectunstuckwhensoftwareisunacceptable.
Determinewheretheremaybegapsinyouracceptancetestingapproachorstrategy.

AfterreadingVolumeI,youmaywanttoskimparticularpracticesofinterestandthecorresponding
samplesinVolumesIIthroughIV.
DecideWhichAcceptancePracticestoUseonYourProject
StartbyreadingVolumeItogetanoverviewofpossiblepractices,andthenrefertothethumbnailsin
VolumesIIIVforspecificpracticesyouareconsidering.Eachthumbnailincludesasectiontitled"When
toUseIt,"whichincludesadviceaboutwhenthepracticeshouldbeused,andasectiontitled
"Limitations,"whichprovidesguidelinesaboutwhenthepracticeshouldnotbeapplied.
LearnHowtoPerformaSpecificAcceptancePractice
StartbyfindingathumbnailinVolumesIIIVifyouwanttodoanyofthefollowing:
Learnaspecificacceptancetestingpracticeorstrategy.
Teachaspecificacceptancetestingpracticeorstrategytosomeoneelse.
Reviewaspecificacceptancetestingpractice.
Findmoreinformationandrelatedresourcestoconsultaboutaparticularpractice.

Afteryoulocatethethumbnailforthespecificpracticeyouwanttolearnabout,readitandanyrelated
samples.Ifyouneedmoredetailedinformationaboutthepractice,seethe"References"sectioninthe
thumbnail.
GetaTemplateforaSpecificArtifact
StartbyfindingansampleinVolumesIIIVifyouwanttodoanyofthefollowing:
Findatemplateforaspecificartifact.
Learnhowtofillinaspecificartifact.

FindtheexampleyouwantinVolumesIIIV,removethesampleinformation,andpopulateit
appropriately.Ifyouneedtoreviewthepracticethatgeneratedtheexample,theexamplelistsallthe
appropriatethumbnailsinVolumesIIIV.
PlantheExecutionofthePracticesonYourProject
StartbyreadingVolumeItogetanoverviewofhowthepracticesfittogetherandsupporteachother.In
particular,thesectionsontheAcceptanceProcess,theDecisionMakingModel,andtheDoneness
Modelmaybeofparticularinterest.Afterthat,reviewthespecificthumbnailsinVolumesIIIV,paying

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 8
particularattentiontothesubsection,"TestLifeCycleApplicability"inthesection,"WhentoUseIt."
Eachsampleartifactisaccompaniedbyanotationthatindicatesatwhatpointinthehypothetical
projecttheartifactwasproduced.Notethatsomeartifactsappearatseveralpointsintheproject
timelinebecausetheyevolveovertime.Ifyoufindyouracceptanceprocesstakestoolongyoumayfind
theStreamliningAcceptanceProcesschapterofVolumeItobehelpful.
FindToolsforAcceptanceTesting
Ifyouarelookingfortoolstoperformacceptancetesting,youmayusetheguidetoexplorethe
availablespace.Althoughsomeofthecasestudiesillustrateusingspecifictools,rememberthatthe
primaryfocusofthisguideisondescribingpractices.
Thechoiceoftoolsusedwhileproducingthisguideshouldnotbeinterpretedasanendorsementofany
tool,norshoulditbeinterpretedasanindicationthatanytoolusedisthebestoneforthejob.Bythe
timeyoureadthisguide,thetoolsillustratedinthesamplesintheguidemayhavebeensupplantedby
newerandbetterones.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 9
TheTeamWhoBroughtYouThisGuide
Mainauthors:
GrigoriMelnik,GerardMeszaros,JonBach
Contributors:
HakanErdogmus,MichaelPuleio,RohitSharma
AdvisoryBoardMembers:
MakAgashe,MohammadAlSabt,LoisPhilippeCarignan,SreekumarChandra,BartHsu,ChrisShaffer,
EngChongLim,DonnaMcLeod,KarlMetivier,TracyMonteith,NoelNyman,KenPerilman,BobSnead,
KeithStobie,JamesWhittaker
Reviewers:
JennittaAndrea,MarkAustin,DanBarnes,KentBeck,LarryBrader,HansBuwalda,JanetGregory,Sam
Guckenheimer,MihaKralj,JamesLyndsay,DanielPiessens,MaxPoliashenko,TomPoppendieck,BJ
Rollison,DonSmith,BasVodde,ThomasWoisetchlger
SpecialThanksto:
JamesBachforconstantremindertotestourtestingguidance
JeffPattonfortheProductContextchart(VolumeI,Ch.4)
BobBrumfieldfortheMultiplexingsidebarsuggestion(VolumeI,Ch.17)andmanyinsightful
conversations
patterns&practicesleadsforsupportingthisendeavor
Production:
RichardBurte,RoAnnCorbisier,DennisDeWitt,TinaBurdenMcGrayne,VeronicaRuiz
Community:
AttendeesatSTARWEST,patterns&practicessummitsandTechEdconferenceswhoprovidedinformal
feedbackonthis.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 10
Introduction
Thisguideisaboutacceptingsoftware.Acceptingsoftwareinvolvesacceptancetesting,butitismuch
morethanthat.Theconceptofacceptancetestingmeansdifferentthingstodifferentpeople.Insimple
terms,acceptancetestingisthesetofactivitiesweperformtogathertheinformationweneedtomake
thedecisionIsthissoftwarereadyforme(ormycustomer)anddoesitfullfillmy(andmycustomers)
requirements?Thisdecisionisusuallycomposedofseveraldecisions,eachwithsupportingactivities.
Therefore,todefineacceptancetesting,itmaybeusefultounderstandtheprocessbywhichthe
decision(s)aremade.Thisprocessmayinvolveseveralorganizationalentities,eachwithoneormore
decisionmakers.Thesoftwareistypicallypassedbetweentheorganizationalentitiesforthemtodecide
whetherthesoftwareisreadytogothroughthenextstep.Thisprocessisintroducedinmoredetailin
thesection"AcceptanceProcessModel"andfollowedupwithamoredetaileddescriptionofthe
decisionmakingprocessinthesectioncalled"DecisionMakingModel."
SoftwareAcceptanceandAcceptanceTesting
Acceptancereferstotheactofdeterminingwhetherapieceofsoftwareorasystemmeetstheproduct
ownersexpectations.Itincludesboththefinaldecisiontoacceptthesoftwareandanyactivities,
includingacceptancetesting,requiredtocollectthedataonwhichtheacceptancedecisionisbased.
Boththeacceptancetestingandtheacceptancedecisioncanberelegatedtoaseparateacceptance
phaseoftheprojectortheycanbedonethroughouttheproject,whichisknownasIncremental
AcceptanceTesting.
MentalModelsforAcceptanceTesting
Whilewritingthisguide,westruggledtodetermineasuitabledefinitionofacceptancetestingthat
wouldmakesensetoabroadrangeofreaders.Itseemedlikethereweremanydifferentvocabulariesin
usebydifferentcommunitiessuchasconsumerproductcompanies,informationtechnology
departmentsoflargebusinesses,anddataprocessingdivisionsoftelecommunicationserviceproviders,
tonamejustafew.Toassistusindescribingacceptance,wecameupwithseveralmentalmodelsof
variousaspectsofacceptancetesting.Wetestedthemodelsagainstnumerousexamplesfromour
collectiveprojectexperiencesatMicrosoftpatterns&practices,telecommunicationproductcompanies,
ITdepartmentsandbeyond.ThenwetestedthemodelswiththepeopleontheAdvisoryBoardforthe
project.Thiswasaniterativeprocess.Wealsotestedthesethroughthepublicreviewprocessby
releasingearlydraftsofthisguidetothecommunityandsolicitingfeedback.
Itisimportanttonotethatourearlymodelsfailedtheiracceptancetests!Thatwasagreatlessonabout
theneedtogetfeedbackincrementally,apracticeweadvocateforacceptancetesting.Basedon
feedbackfromouradvisorswerefactoredthemodelsandcameupwithadditionalmodelstofillthe
gaps.ThekeybreakthroughwaswhenwecameupwiththeDecisionMakingModel,whichtiestogether

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 11
mostoftheconceptsaroundacceptingasystem.ItbuildsontheAcceptanceProcessModelwhich
describesthekeystepsandactivitiesasthesystemundertestmovesfromrequirements,through
developmentandintotestingandfinallyproduction;italsodescribeshowthedecisiontoacceptthe
systemismade.TheDecisionMakingmodeldescribeswhomakesthedecisionsandwhoprovidesthe
dataforthosedecisions.
Thedecisionsarenotmadeinavacuum;thereareanumberofinputs.Theseincludetheproject
context,thenatureofthesystembeingbuiltandtheprocessbeingusedtobuildit.Thelatteris
importantbecauseitaffectshowwedefinedone.
Figure1illustratestherelationshipsbetweenthekeymodels.

Figure1

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 12
TheKeyMentalModelsofAcceptance

ThesemodelsarethefocusofPartIThinkingaboutAcceptancebutheresashortintroductiontoeach
modeltogetusstarted:
TheAcceptanceProcessModel.Thismodeldefinestheoverallstagesofsoftwaredevelopment
andthe"gates"thatmustbepassedthroughonthejourneyfromconstructiontosoftware
inuse.
DecisionMakingModel.Thismodeldescribeshowtodecidewhethersoftwarecangothrougha
gatetothenextstageandwhomakesthedecision.Italsodefinesthesupportingrolesthat
mayhelpthedecisionmakergathertheinformationneededtomakethedecision.
ProjectContextModel.Thismodeldescribesthebusinessandprojectfactorsthatinfluencethe
decision,includingtimeframesanddeadlines,resourceavailability,budget,andanything
contributingtoprojectrisks.
SystemModel.Thismodeldescribestheattributesofthesoftwareintensivesystemthatmay
factorintothedecision.Thisincludesbothfunctionalandnonfunctionalattributes.The
systemmodelmayariseoutofrequirementsgatheringactivitiesoramoreformalproduct
designprocess.Techniquesforcapturingfunctionalrequirementsincludesimpleprose,use
casemodels,protocolspecificationsandfeaturelists.Thenonfunctionalrequirementsare
typicallycapturedindocumentsorchecklists.
RiskModel.Thismodelintroducestheconceptsofevents,likelihood/probability,and
consequence/impact.Ithelpseveryoneinvolvedunderstandwhatcouldgowrongand,
thereby,prioritizetheacceptancecriteriaandthekindsofinformationtogathertohelp
maketheacceptancedecision.Italsodescribesseveraldifferentriskmitigationstrategies,
includingthefollowing:
Dosomethingearliertobuyreactiontime.
Doadditionalactivitiestoreducelikelihoodofsomethingoccurring.
DonenessModel.Thismodeldescribesdifferentdimensionsofdonenessthatneedtobe
consideredinareleasedecisionandhowtheyareaffectedbytheprocessmodel.

ThechaptersinPartIIIAcceptingSoftwareintroduceothermodelsthatbuildonthiscoremodel:
TestLifecycleModel.Thisdescribesthestagesthatanindividualtestcasegoesthroughhowto
gatherinformationformakingreadinessandacceptancedecisions.
ConcernResolutionModel.Thisdescribeshowtohandleanyconcernsthatareraisedduringthe
acceptancetestingprocess.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 13
DevelopmentProcesses
Thesoftwaredevelopmentprocesshasasignificantimpactonhowacceptancetestingisperformed.
ThroughouttherestofthisvolumeandtheotherstofollowwefoundourselvessayingOnsequential
projectsandOnagileprojectsbutmanypeoplehavetheirowndefinitionsofwhattheseterms
mean.Wewantedtomakesureallreadersunderstoodwhatwemeantbytheseterms.Wefeelthat
thesenamesrefertopointsonaprocesscontinuumwithotherlabelledpointspossible.Thissection
describestheprocesscontinuumwithtwodistinctprocessstereotypesontheoppositeendsofthescale.
SequentialProcesses
Sequentialprocessesorganizeworkbasedonthekindsofactivitiesinvolvedandtheinterdepencies
betweentheactivities.Theclassicwaterfallapproachinvolvesasinglereleasewhileincremental
waterfallprojectshavemultiplereleases.
ClassicWaterfall
Thewaterfallapproach(sonamedafterthediagramsusedinapaperbyWinstonRoyce[Royce])
involvesorganizingtheprojectintoaseriesofdistinctphases.Eachphasecontainsaspecifictypeof
work(suchasrequirementsanalysis)andhasspecificentryandexitcriteria.Intheclassicorpure
waterfallapproachthephasesdonotoverlap.Theentryandexitcriteriarequiretheoutcomeofa
previousphasetobecompleteandcorrect(validated)beforethenextphasecanstart.Thispushesthe
deliveryoftheproductsfunctionalitytotheend:theproductasawholeisdeployedinabigbang
approach.Figure2illustratesthemajorphasesofawaterfallproject.

Figure2
AClassicalWaterfallProject
Thisprocessisusuallyimplementedbybreakingdowneachphaseintohierarchicallyorganizedunitsof
workappropriatetothetypeofworkinvolved.Forexample,withintherequirementsphase,thework
maybedividedbetweenanalystsbyrequirementtopic,butduringtheconstructionphase,workmaybe
dividedamongthedevelopersbymodule.Thehandoffsbetweenphasesareusuallyintheformof
documents,exceptthatthehandofffromconstructiontotestingalsoinvolvesthecodebase.Readiness
assessmentisdonebythesupplierorganization,whichwerefertoastheProductDevelopmentTeam,
afteralltheconstructioniscompleted;acceptancetestingisperformedbytheproductownerafterthe
softwareisdeemedtobeready.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 14
IncrementalWaterfall
Itiscommonlyacceptedthatthelongeraprojectgoesbeforedeliveringsoftware,thehigherthe
probabilityoffailure.Ifthecontextorrequirementschangebeforetheprojectiscompleted,thepure
waterfallcannotsucceed.Onewaytocombatthisistobreaktheprojectintoincrementsof
functionality.Theseincrementscanbetestedandinsomecasesevendeployed.Figure3aillustratesa
waterfallprojectwithtwoincrementsofindependentfunctionalityeachofwhichistestedand
deployed.Thistypeofprojectisalsocalledcheckpointedwaterfall).

Figure3a
Multireleasewaterfallwithindependentfunctionality
Inthisapproach,theplanningphase,requirementsanalysisphase,anddesignphaseareperformed
onceearlyintheprojectwhiletheconstructionphase,testphase,anddeploymentphasearerepeated
severaltimes.Theworkwithineachphaseisdecomposedthesamewayasforsinglereleaseprojects.If
thefunctionalitybuiltinthesecondreleaseoverlapsthefunctionalityinthefirstrelease,thetestingand
deploymentmustencompasstheentirefunctionality.Figure3billustratesmultiplereleaseswith
overlappingfunctionality.Notehowthetestactivitymustspanthefunctionalityofbothreleases.

Figure3b
Multireleasewaterfallwithoverlappingfunctionality
Inthemultireleasewaterfallprocess,sometimesthetestphaseoftheareleasemayoverlapwiththe
constructionphaseofthesubsequentreleaseiftheconstructionandtestingteamsareseparateand
featuresacrossthethetworeleasesaresufficientlyindependent.
H
i
g
h
-
L
e
v
e
l
P
l
a
n
n
i
n
g
R
e
q
u
i
r
e
m
e
n
t
s
A
n
a
l
y
s
i
s
A
r
c
h
i
e
c
t
u
r
e
&

D
e
s
i
g
nConstruction
T
e
s
t
D
p
l
y
Construction
T
e
s
t
D
p
l
y
time
F
u
n
c
t
i
o
n
a
l
i
t
y
Construction
T
e
s
t
D
p
l
y
t
Construction
T
e
s
t
D
p
l
y
t

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 15

AgileProcesses
Mostagilemethodsuseaniterativeandincrementalapproachtodevelopment.Afteraninitialplanning
period,theprojectdurationisbrokenintodevelopmentiterationsthatdeliverincrementsofworking
software.Figure4illustratesaprojectwithtwoiterations;mostprojectswouldhavemanymore
iterationsthanthis.

Figure4
Iterative&IncrementalDevelopment
Figure4illustratestwoiterations,eachofwhichstartswithaniterationplanningsessionandendswith
someacceptancetesting.Ineachiterationtheworkisbrokendownintofeaturesoruserstories,each
ofwhichindependentlygoesthroughtheentiresoftwaredevelopmentlifecycle.Theproductowner,
onsitecustomerorcustomerproxy,whoisreadilyaccessibletotheProductDevelopmentTeam,is
responsiblefordescribingthedetailsoftherequirementstothedevelopers.Itisalsotheproduct
ownersresponsibilitytodefinetheacceptancetestsforeachfeatureoruserstory.Theymayprepare
theteststhemselves,delegatethejobtorequirementsortestspecialistswithinthProductOwnerTeam
orpreparethemincollaborationwiththeProductDevelopmentTeam.Thetestsactasamoredetailed
versionoftherequirementsdescriptioninaprocessknownasAcceptanceTestDrivenDevelopment
orStorytestDrivenDevelopment.Thisallowsthedeveloperstoexecutetheacceptancetestsduring
thedevelopmentcycle.
Whenallthetestspassforaparticularfeatureoruserstory,thedevelopersturnoverthefunctionality
totheproductowner(orproxy)forimmediate"incrementalacceptancetesting."Iftheproductowner
findsanybugs,thedeveloperfixesitassoonaspossible.ItistheProductOwnersdiscretiontoinsist
thatitbefixedrightawayortoconsideritpartofanotherfeaturetobeimplementedinalater
iteration.Iftheywantitfixedandthedeveloperhasalreadystartedworkingonanotherfeature,the
developertypicallyputstheotherfeatureonholdwhiletheyaddresstheProductOwnersconcerns;the

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 16
featureisntconsidereddoneuntilalltheconcernsareaddressed.Theproductownerconcernsare
notstockpiledinabugdatabaseforfixingduringabugfixingphaseoftheproject.
MultiReleaseAgileProjects
Mostagilemethodsadvocate"deliverearly,deliveroften."Intheory,theresultofanydevelopment
iterationcouldbedetermined,afterthefact,tobesufficienttobeputintoproduction.Thiswouldlead
directlytothedeploymentactivities.Inpractice,mostagileprojectsplanonmorethanonereleaseto
productionandtheiterationsarethenplannedtodeliverthenecessaryfunctionality.Figure6Multi
ReleaseAgileProjectillustratesanagileprojectwithtworeleases.

Figure5
MultiReleaseAgileProject.
Notehowthereisatestingcycleforthesecondreleasewhichincludesregressiontestingofthe
functionalitydeliveredinthefirstrelease.Mostagilemethodsemphasizetestautomationsothe
regressiontestingcostisminimized.
KanbanbasedAgileProcess
Someagilemethodologiesdispensewithiterationsinfavourofallowingafixednumberoffeaturesin
progressatanytime.Thisisdesignedtoemphasizetheconceptofacontinuousflowofworkingcode
fortheonsiteproductowner(orproxy)toaccept.Fromanacceptancetestingperspective,these
Kanbanbasedmethods[Kanban]stilldoincrementalacceptancetestingatthefeatureleveland
formal/finalacceptancetestingbeforeeachrelease,butthereisnologicalpointatwhichtotriggerthe
interimacceptancetestingthatwouldhavebeendoneatiteration'sendiniterationbasedagile
methods.Figure6KanbanbasedAgileProjectillustratesthisstyleofdevelopment.Notethelackof
iterations.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 17

Figure6
KanbanbasedAgileProcess
Intheexample,itisimportanttonoteherethattherearenevermorethanthreefeatures(inthis
example,oneudergoingdesign,asecondundergoingconstruction,andathirdundergoingtesting)in
progressatanyonetime.Inotherwords,thereareonlythreedevelopment"slots,"andaslotbecomes
availableforanotherfeatureonlyafterithasfinisheditsincrementalacceptancetesting.Thisissimilar
tohowKanbanareusedtocontroltheinventoryinleanfactoryproductionlines.Intheory,theproduct
ownercandecideatanytimethatthereisenoughfunctionalitytowarrantdeployingtheproduct
followingashortperiodofregressiontesting.
Kanbanbasedsoftwareproceses[Scrumban]areimplementationsofamoregeneralphilosophyknown
asLeansoftwaredevelopment[LSD].
ProcessasaContinuum
"Agile"and"waterfall"areexamplesoftwohighlevelprojectstreotypesconsistingofcertain
combinationsofcharacteristics.Itiseasytoimaginethedecisiononeachofthesecharacteristicsas
beingthesettingofaprocessslidercontrol.Forexample,theNumberofreleasesslidermighthave
stopsat1,2,3,andsoon.TheNumberofiterationsslidercouldhavevaluesof1andsoon,which
indicatewhetherthereareintermediatecheckpointswithinarelease.TheMaximumnumberof
featuresinprogressslidersimilarlymaytakeonvaluesdependingonthenumberofdevelopmentslots
availableinaKanbanbasedsystem.AnotherdimensionmightbeIntegrationfrequency,withsettings
ofBigBang,MajorMilestone,Quarterly,Monthly,Biweekly,Weekly,Daily,andContinuous.
Thefollowingtablesummarizesthepositionsoftheseslidersforwhatisconsideredtobeastereotypical
projectofeachkind.Thesepositionsarenotdefinitiveorcomplete,buttheychallengeyoutocreate
yourownslidersandsettingsforyourcontext.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 18

Type of Process

Project Attributes
Classic
Waterfall
Incremental
Waterfall
Agile (Iteration) Agile (Kanban)
Number of releases 1 1 2 or more 2 or more
Number of iterations 1 26 4 or more 1
Iteration length Not applicable Many months 1-4 weeks Not applicable
Maximum number of
features in progress
No maximum No maximum 1 iterations worth Less than the
number of
team members
Integration frequency Big Bang Quarterly Daily or hourly Daily or hourly
Requirement-to-test
duration
Months or years Months Days Days
Test timing Separate phase Separate phase Mostly incremental Mostly
incremental
Release criteria Scope-based Scope-based Time-boxed Time-boxed
Average Requirement task
effort
Person months Person months Person days Person days
Average development task
effort
Person days or
weeks
Person days or
weeks
Person hours Person hours
Culture Hierarchical Hierarchical Collaborative Collaborative
Skills Highly
specialized
Highly
specialized
Generalists Generalists or
Specialists
Determining progress Work
completed
relative to plan
Work completed
relative to plan
Delivery of working
code
Delivery of
working code
Work remaining Estimate
duration of
remaining tasks
Estimate
duration of
remaining tasks
Estimated time for
remaining features
Estimated time
for remaining
features

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 19
References
[Royce]WinstonW.Royce,ManagingtheDevelopmentofLargeSoftwareSystems,Proceedingsof
IEEEWESCON,Aughust1970,pp19.
[Kanban]KenjiHiranabe,KanbanAppliedtoSoftwareDevelopment:fromAgiletoLean,published
onlineat<http://www.infoq.com/articles/hiranabeleanagilekanban>,January14,2008
[LSD]Poppendieck,Mary&TomLeanSoftwareDevelopmentAddisonWesley(2003)ISBN:032
1150783
[Scrumban]CoreyLadas,Scrumban:EssaysonKanbanSystemsforLeanDevelopment,Modus
CooperandiPress,2008

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 20
ACautionaryTale
Thefollowingisthestoryofaprojectwherethepersoninchargeofdefiningandacceptingaproduct
failstorisetothechallenge.Itdescribeswhatgoeswrongandwhy.Italsoprovidesanalternate
outcomeofhowtheprojectcouldhaveworkedoutifproperacceptancepracticeshadbeenused.
BobKellyisamidlevelmanagerinthemarketingdepartmentatAcmeProducts.Heisinchargeofthe
MidSizedMarketsproductgroupandistheproductmanagerofitscoreproduct,theXCelRator.Hes
justfinishedabunchofmarketresearchandhascomeupwithagreatideaforanewmoduleforthe
productthathebelieveswilldoubletherevenueoftheproduct.TheProductDevelopmentdivisionhas
ateamthatisjustwindingdownitscurrentprojectandBobiskeentogettheteamstartedonbuilding
thenewmodule.HecallsameetingandrunsthroughthePowerPointslidedeckhesbeenusingwhen
talkingwithfocusgroupsandpotentialclients.Heconcludesthepresentationbylayingoutthekey
deliverydateswhichmustbemettoallowthecompanytoshowcasetheproductattheannualtrade
show.Heaskstheprojectmanagertodrawuptheprojectplanandgettheteamworkingonthenew
module.
TheDevManagermeetswithhisteamtocollectsomeinformationandthenhedefinestheprojectplan.
BasedonthedeliverydatehedefinesintermediatemilestonesforRequirementsComplete,Design
Complete,CodeCompleteandTestComplete.
Teamstartswritingrequirementsdocuments.AtRequirementsComplete,devmanagerdeclares
requirementscompleteontime.Theproductdevelopmentteam(devteam)knowsfullwellthatsome
areasoftherequirementsarestilltoovaguetoimplement.Theywillhavetogetclarificationonthe
detailsastheydothedesign.Hopefullytheproductmanager,Bob,willbemoreavailabletoanswer
questionsabouttherequirementsashislackofavailabilityisatleastpartiallytoblameforthe
requirementsstillbeingvague.
AtDesignComplete,ditto.Devteamknowsithasntdesignedsomepartsofthefunctionalityexceptata
verycursorylevel.Theywillhavetofillinthedetailsastheycode.Meanwhile,Bob,theproduct
managerthinkseverythingisproceedingaccordingtoplanandthathellhaveagreatproducttodemo
atthetradeshow.Hestartsthinkingabouthowhellspendthebonushessuretogetforcompleting
thisprojectontime.Hecallshisarchitectandaskshertostartdrawingupplansfortheextensiontohis
house,completewithindoor/outdoorswimmingpool.
Atcodecompletetheteamrequestsanextramonthtofinishcodingpartsofthefunctionalitythey
haventhadtimetodoyet.TheTestManageraskswhatpartsofthesoftwarearecompletesothat
testerscanstarttestingandtheteamrespondsNone,weareeachworkingonadifferentpartofthe
systemanditwillallbefinishedataboutthesametime.
Toensuretheproductisreadyforthetradeshow,Bobasksthetestmanagertomakeuptheschedule
bycompressingthedurationofthetestphase.Hereducesthenumberofplannedtestcyclestoreduce
theelapsedtimebasedonassurancesthatthecodewillbeingreatshapeduetotheextratimethedev
teamisbeinggiven.Bobstillbelievestheproductcanbereadyforthetradeshow(butnonethelesshe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 21
asksthearchitecttoscalebacktheextensiontohishousebyremovingtheenclosureforthepool
thinkingtohimselfIllstillbeabletouseitmostoftheyearandIcanalwaysaddtheenclosurewithmy
release2bonus.)
AsthenewCodeCompletedeadlineapproaches,theteamasksforanothermonth.Bobreluctantly
givesthem2weeks.Twoweekslaterwillmakeittightforthetradeshowbutwecandemothebeta,
hehopes.
Thedevteamfinallydeliversthecodetwomonthslate.Thetestteamstartstestingbutfindssignificant
problemsthatpreventcompletionofthefirstpassofthetestcases.Testinghaltsafterlessthanaweek.
Thedevteamtakesamonthtofixallthemajorshowstopperbugsbeforehandingthecodebackto
test.Testteammakesitthroughthefulltestcyclethistimebutfindsseveralhundreddefects.
Developmentstartsfixingthebugsastheyarefound.
ThetestteamfinallyreceivesanewbuildthatthedevteamsayshasalltheSev1and2bugfixes.
AlmostimmediatelytheyfindseveralnewSev1regressionbugs.Developmentdisagreesthatsomeof
thebugsareevenvalidrequirements.TherequirementsneversaidanythingaboutandWhywould
anyoneeverdothat?theyexclaim.Bob,theproductmanager,hastobebroughtintosettlethe
dispute.Heagreesthatsomeofthebugsarentrealrequirementsbutmostarevalidscenariosthathe
neverconsideredbutwhichneedtobesupported.TesttellsBobthatthereisnowayinthatthe
originalschedulecanbemet.
After4test&fixcyclesthattook50%longerthantheoriginalschedule(letalonemakingupthe
schedule),mostoftheSev1and2bugsappeartohavebeenfixed.ThereisstillseveralhundredSev3
and4bugsandtestteamhasstoppedbotheringtologtheSev5s(poorlywordedmessages,fieldlabels
orbuttonnames,etc.).Testteamsaystheproductisnotreadytoshipandwillrequireatleast2more
test&fixcyclesbeforetheywillagreetoreleaseit.
Theproductisnowseveralmonthslateandwontbereadyforthetradeshoweveninalpharelease.
Wecanstillshowtheboxanddosomeverycontrolleddemos.Bobassureshisboss.Bobisseeinghis
bonusdiminishwitheachweekofreducedsalesinthisfiscalyear.Hedecidestoshiptheproduct,
overrulingtheTestdepartment.Herevisesthesalesforecastsbasedonthelatelaunchcausedbythe
underperformanceofthedevelopmentandtestteams.
ThemanageroftheoperationsdepartmenthearsaboutthistoolateandcomesstormingintoBobs
office.Ihearyouareplanningtoreleasethenewmodulewith300knownSeverity3defects.Doyou
haveanyideawhatthiswilldotooursupportcosts?Acounterproductiveargumentensuesbecause
thereisnoturningbackatthispoint;theannouncementshavebeenmadeandtochangeplansnow
wouldbehugelyembarrassingtothecompany.IsurehopethisworksoutOK.thinksBobtohimself;
atthispointhedoesntfeellikehesinchargeofhisowndestiny.
Bobsmarketingmachinehasbeeninhighgearforquiteawhileandhasgeneratedquiteabitofpent
updemand,especiallysincesomeuserswerehopingtobeusingtheproductmonthsago.Userstry
usingthenewproductindroves.Theyrunintoallmannerofproblemsandcallthehelplinewhichis
overwhelmedbythecallvolumes.Manyusersgetbusysignals;theluckyoneswaitlisteningtorecorded
announcementsforlongperiodsoftime.Thehelpdeskhastobringonextrastaffwhoneedtobe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 22
trainedveryhastilyandthereforecannotprovideverygoodservicetothecustomers.Somecustomers
giveuptryingtousetheproductbecauseoflongwaitsorpoorsupport.
Manyoftheuserproblemsarerelatedtousabilityissuesthatwerenotdetectedduringtherushed
testingbecauseitwassofocusedontheSev1&2bugs(andtheusabilitystuffwasusuallyrated3or
below,manyofwhichwerentevenloggedduetothefocusonthe1sand2s.)Atpeakusagetimesthe
systemslowstoacrawl;thedevteamiscalledintofigureoutwhyitissoslowdistractingthemfrom
workingonsomeoftheimprovementsidentifiedduringtesting.Usershavetroubleimportingdatafrom
priorversionsofthesoftwareorfromcompetitorsproducts.
Alargepercentageoftheusersabandontheproductafterthefreetrialisover;theconversionrateis
lessthanhalfoftheprojectedrate.Revenuesarerunningat40%oftherevisedprojectionsandless
than20%oftheoriginalprojections.Supportcostsare50%overoriginalprojectionsduetothehiring
bingeintheusersupportcentre.
Thecapitalcostis35%overtheoriginalbudgetandhaseatenintotheplannedbudgetforasecond
modulewithadditionalmusthavefunctionality.Instead,the2
nd
releasehastofocusonimprovingthe
quality.Thenewmodulewillhavetowaituntil2
nd
halfofnextyear.Theproductmanagerrevisesthe
revenueprojectionsyetagain(andcallsthecontractortocanceltheadditiontohishouse).Shortlyafter
sendinginhismonthlystatusreporthisphonerings.ItistheVP,hisboss,requestinghecometohis
officeimmediately...
WhatWentWrong
1. ProductmanagerBobdecidedscope,timeframesandresourcestherebyleavingonlyqualityas
thederivablevariable.Thisresultedinthedevteambeingovercommittedattheproduct
managersinsistence.(Sometimesthedevteamwillovercommitoutofoptimismbutinthis
casetheproductmanagerdidittothem.)
2. Sequentialprocessisinherentlyopaquefromaprogressperspective.Thefirstmilestonethat
isnteasytofudgeisCodeComplete.Thefirstrealisticassessmentofprogressiswellintothe
testcycle.
3. Productmanagerwasntavailabletoclarifyvagueandmissingrequirements.Testerswerenot
involvedsoonenoughtoidentifythetestscenariosthatwouldhavehighlightedthemissing
requirements.ButnoonecouldprovetherequirementswereincompletesoRCwasdeclared
ontime.
4. Productdevelopmentteamcouldntprovedesignwasntdone(becauseitisamatterof
opinionastohowdetailedthedesignneedstobe)soDesignCompletewasdeclaredontime.
Thesequentialprocessdoesntgivethedevelopmentteamagoodwaytoestimateitstrue
velocityuntillateintheproject,whenthecodeismostlywrittenanddebugged.
5. Devteamcutcornerstomakethenew(alreadylate)CodeCompletedeadline.Thecodewas
writtenbutmuchofitwasntproperlyunittested.Theteamwasdeepinunittestdebt,they
knewit,andwouldhavetoldanyonewhoaskedbutnoonewantedtoheartheanswer.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 23
6. ThequalitywasawfulwhendeliveredtoTest.Soithadtoberedone(weneverhavetimetodo
itrightbutwealwaysmaketimetodoitover!)
7. Testwasaskedtorecovertheschedule(typical!)buttestingtooklongerbecausethepoor
qualitycoderequiredmoretest&fixcyclestogetitintoshape.
8. Nocleardefinitionofdonesodecisionismadeontheflywhenemotionsgetinthewayof
clearthinking.Theproductmanagerlethisattachementtohiscommitmentsoverrideany
sensibilityaboutquality.
9. Theoperationsacceptancecriteriawereneversolicitedandbythetimetheywereknownitwas
toolatetoaddressthem.
10. Sequentialprocesshidtrueprogress(velocity)untilitwastoolatetorecover.Therewasnoway
toscalebackfunctionalitybythetimeitbecameundeniablethatitwontallbedoneontime.
Therewasnowaytoreducescopetofitthetimelinebecausethesequentialstyleapproachto
theprojectplan(RequirementsComplete,DesignComplete,CodeComplete,TestingComplete
milestones)causedallfeaturestobeatroughlythesamestageofdevelopment.Therefore
cuttinganyscopewouldresultinalargewasteofeffortandverylittlesavingsofelapsedtime.
(Onlytheeffortofbugfixingwouldhavebeensaved.)
11. DevelopmentwasrushedinfixingtheSev1problemssothattestingcouldgetthroughatleast
onefulltestcycle.Thiscausedthemtomakemistakesandintroduceregressionbugs.Ittook
severaltest&fixcyclesjusttofixtheoriginalSev1&2sandtheresultingregressionbugs.Inthe
meantimetheSev3spiledupandupandup.Thisresultedinseveralmoretest&fixcyclestofix
andeventhenmorethanhalfwerestilloutstanding.
12. Lackofplanningfortheusagephaseoftheprojectresultedinapoorcustomersupport
experiencewhichexacerbatedthepoorproductquality.
HowitCouldHaveGone
Bob,theproductmanagercomestoproductdevelopmentteamwithrequirementsthatare
representativeofthecustomersexpectations.
Devteamestimatesrequirementsasbeing50%overteamscapabilitybasedondemonstrated
developmentvelocity.Theproductmanagerisnothappyaboutthis.Devteamproposesincremental
development&acceptancetesting.Insteadof4sequentialmilestonesbasedonphasesofdevelopment
theysuggest4incrementalinternalreleasesoffunctionalitywhereeachincrementcanbetested
properly.Thiswillallowthemtodeterminetheirdevelopmentcapacity(orvelocity)whichwillhelp
theproductmanagerplansubsequentmilestonesmoreaccurately.
Bobselectsfirstincrementoffunctionalitytodevelop.Withthehelpofthedevteamhedefinesthe
usagemodelconsistingofuserpersonasandtasks.TheDevteamwhipsupsomesketchesorpaper
prototypesandhelpsBobrunsomeWizardofOztestsonthepaperprototypesthatrevealsome
usabilityissues.BobadjuststherequirementsanddevteamadjuststheUIdesign.Bob,theproduct

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 24
managerworkswithdevteamandthetesterstodefinetheacceptancetests.Thedevteamautomates
thetestssotheycanberunondemand.Theyalsobreakdownthefeaturesintouserstoriesthateach
takejustafewdaystodesignandtest.
Teamdesignssoftwareandwritescodeusingtestdrivendevelopment.Allcodeisproperlyunittested
asitiswritten.Allpreviouslydefinedautomatedtestsarererunseveraltimesadaytomakesureno
regressionbugsareintroducedasthenewfeatureisimplemented.Aseachfeatureisfinished,as
definedbythefeatureleveldonedonechecklist,thedeveloperdemostoproductmanagerand
testerwhocanpointoutanyobviouslymissingfunctionalitythatneedstobefixedbeforethesoftware
isconsideredreadyforacceptancetesting.
Aspartofincrementalacceptancetestingtheydoidentifyafewnewusagescenariosthatwerenotpart
oftherequirementsandprovidethesetoBobassuggestionsforpotentialinclusioninthesubsequent
increments.Bobadjuststhecontentofthenextincrementbyincludingafewofthemorecriticalitems
andremovinganequivalentamountoffunctionality.Healsocontactstheoperationsmanagerto
validatesomeoftheoperationalusagescenariosidentifiedbythedevteamandtesters.Theoperations
managersuggestsafewadditionaloperationaluserstorieswhichBob,theproductmanager,addsto
thefeaturebacklogforthenextincrement.
Attheendoffirstincrementoffunctionality(whichtook5iterationstodevelop,onemorethan
expected)devteamrunsthenonfunctionalqualitiesteststoverifythesoftwareperformsupto
expectationsevenwith110%oftheratednumbersofusers.Thefirsttestresultsindicateitcanonly
handleroughly50%oftheexpectedusersandgetsevenslowerasthedatabasestartstoaccumulate
records.Theyaddsomeworkitemstothescheduleforthesecondincrementtoaddresstheseissues
andwarntestingaboutthelimitationstoavoidtheirwastingtimestumblingontothem.Testingfinds
onlyafewminorbugsduringexecutionofthefunctionaltestscripts(themajorstuffwasallcaught
duringincrementalacceptancetesting.)Theymoveontodoingsomeexploratorytestingusingsoap
operasandscenariosastheirtestsessioncharters.ThesetestsidentifyseveralpotentialscenariosBob
neverthoughtof;headdsthemtothefeaturebacklog.
Bob,astheproductmanagerarrangestodosomeusabilitytestingofthefirstincrementwithsome
friendlyusersbasedonsomeoftheusagescenariosidentifiedintheusermodel.Theresultsofthe
testingidentifyseveralenhancementsthatwouldimprovetheusersatisfaction.Bobaddsthesetothe
thingstodointhenextincrementoffunctionality.
Bobcalculatesthatdemonstrateddevelopmentvelocityis25%lessthanoriginalestimates.Basedon
thisheadjustshisexpectationsforthefunctionalitytobedeliveredinthereleasebyremovingsomeof
thelesscriticalfeaturesandthinningsomeofthecriticalfeaturesbyremovingsomenicetohave
glitz.Bettertodelivergoodquality,ontimethantotrytocraminextrafunctionalityandrisk
everythinghethinkstohimself.
Thedevelopmentteamdeliversthe2
nd
incrementoffunctionalitywithsimilarresultsasthefirst.The
worktheydidtoimproveperformanceresultsintheperformancetestspassingwithflyingcolorswith
acceptableresponsetimesat120%ofratedcapacityandnodegradationasthedatabasefillsupwith
transactionaldata.Theyaddsometestsforpenetrationtestingandscheduleasecurityreviewwiththe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 25
securitydepartment.Bobmakessomeadjustmentstothefunctionalityplannedforthe3
rd
incrementof
functionality.Heincludessomefunctionalitytoaddressoperationalrequirementssuchasmigrating
datafromearlierversionsofthesoftwareandimportingdatafromcompetitorsproducts;hewantsto
makeitrealeasyforuserstoadopthisproduct.Heshappywithhowtheprojectisprogressingand
confidentthattheywillbeabletodeliverexcellentqualityandonschedule.
Inthethirdincrement,thedevelopmentteamdelivers20%morefunctionalitythanoriginallyplanned.
Theproductmanagerhadtoscrambletoprovidetheacceptancetestsfortheextrafeaturesbrought
forwardfromthefourthincrement.Basedonthis,theproductmanagerisabletoplanforsomeextra
functionalityforthefourthincrement.Heconsidersrevivingsomeofthefunctionalitythathedcutafter
thefirstincrementbutdecidesitreallywasntthatuseful;amuchbetteruseofthedevteamsefforts
wouldbesomeoftheusabilityenhancementssuggestedbythelastroundofusabilitytesting.Healso
addsfunctionalitytomakeiteasytoupgradetothenext(yetunplanned)releasewithouthavingtotake
downtheserver.Thatwillhelpreducetheoperationalcostsofthesoftware.Yes,thisisgoingtoa
greatproduct!hesaystohimself.
AsthedevelopmentteamisworkingonIncrement4,theproductmanagerdiscussestheAcceptance
TestPhaseoftheproject.Wehadoriginallyplanned3fulltest&fixcycleseachof2weeksdurationwith
aweekforfixesinbetweenforatotalof8weeksoftesting.recountstheTestManager.Butbasedon
theresultsoftestingIncrements1,2and3Impredictingthatwellonlyneedonefulltestcycleof2
weeksplusa1weekminicycleofregressiontestingforanyfixesthatneedtobedone(andImnot
expectingmany.)Theautomatedregressiontestingthedevteamisdoingaspartofreadiness
assessmenthasbeenpreventingtheintroductionofmanyregressionbugsandtheautomatedstory
testswevebeencoauthoringwithyouandthedevteamhaspreventedmisunderstandingsofthe
requirements.ThisisthemostconfidentIveeverfeltaboutaproductatthispointintheproject!
Summary
Therearegoodwaystoeffectivelydefine,buildandacceptproductsandtherearewaysthatmayresult
indisappointment.Therestofthisguideisdedicatedtohelpingyouunderstandwhichiswhich.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 26
PartIThinkingAboutAcceptance
Akeypartofunderstandinganysystem,whetherwearedesigningit,buildingit,testingit,acceptingor
usingitistobuildmentalmodelsabouthowweexpectittowork.Systemsthatareeasytousemakeit
easyforuserstodevelopsimplementalmodelsoratleastfamiliarmentalmodelsthatallowthemto
reuseknowledgeabouthowothersystemswork.Thispartoftheguidedescribessixsimplemodelsthat
helpusreasonabouttheprocessofacceptingsoftware.
Theultimategoalofanydevelopmentprocessistobuildtherightproductofhighqualityandtobuildit
withminimumwaste.Theprimarydecisionswemustmakearetheworkflowandtheamountofwork
thatwillbemovedthroughtheworkflowatatime.Theworkflowconsistsofthesetofactivitiesthe
teammustpursueandtheorderingofthoseactivities,thatis,whethertheactivitieswillbecarried
sequentiallyorasneeded.Ourorganizationschoicesonthesetwocriticaldimensionsaffecthowwewill
goaboutmakingtheacceptancedecisions.Inaddition,howaprojectchoosestomeasuretherightness
andthequalityoftheproductwillmakeadifferenceindecidinghowtheproductdevelopmentteam
andtheproductownerwillknowifitsdoneandreadytodeploy.Themodelshelpaddresstheimpact
ofthechoicesthatthestakeholdersmakeonthesedimensions.

Chapter1TheAcceptanceProcessintroducestheoverallacceptanceprocess.Itdescribesthedecisions
thatmustbemadeandhowacceptancerelatestotheoverallsoftwaredevelopmentlifecycle,the
processeswithstagesandgates,andtheclassicVmodelofsoftwaredevelopment.
Chapter2TheDecisionMakingModeldescribestheprocessformakingthedecisionsintroducedin
thepreviouschapter.Italsoprovidesguidanceonwho(whichroles)shouldmakethedecisionsandwho
shouldcollecttheinformationonwhichthedecisionsarebased.
Chapter3TheProjectContextModeldescribesthevariousprojectfactorsthatmightinfluencethe
acceptanceprocess.
Chapter4TheSystemRequirementsModeldescribesvariouswaysrequirementscanbepartitioned
anddescribedandhowthatrelatestotheacceptanceprocess.
Chapter5TheRiskModeldescribesasimplewayofthinkingaboutpotentiallyundesirableeventsand
howtheymightinfluencetheacceptanceprocess.
Chapter6TheDonenessModeldescribeswaystodescribetowhatdegreetheproductisfinished.It
givesuswaystoaskthequestionwhatdoesdonemean?asawaytodefineouracceptancecriteriafor
individualfeaturesandforentirereleasesofproducts.Italsointroducestheconceptofincremental
acceptance.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 27

Chapter1 TheAcceptanceProcess
ThischapterdefinestheprocessbywhichsoftwareisdeemedacceptablebytheProductOwner.It
introducesthethreemajorconstituencieswhomustmakedecisionsatkeypointsintheprocessandhow
thesedecisionsrelateto,andinfluence,eachother.WestartbydrillingintotheAcceptphaseofthe
softwaredevelopmentlifecycletoexaminethekeydecisionsandhowreleasecandidatesflowthrough
theprocess.Thenweexaminehowmorecomplexscenariossuchasmultiplereleases,complex
organizationsandcomplexproductsinfluencetheprocess.Sidebarsexaminehowthedecisionprocess
relatestoStageGateprocessesandQualityGates.Wefinishthischapterwithtechniquesfor
streamliningtheacceptanceprocess.Subsequentchaptersdescribetherolesandresponsibilitiesof
variouspartiesinvolvedinmakingthedecisionsandhowthisprocessportsfromsequentialtohighly
iterativeandincrementalagileprojects.

1.1 AcceptanceasPartoftheProductDevelopmentLifecycle
Softwareintensivesystemsgothroughanumberofstagesontheirjourneyfromconcept,ahalfformed
ideainsomeonesmind,toprovidingvaluetoitsusers.ThisprocessisillustratedinFigure1ASequential
ProductDevelopmentLifecyclewiththestagesplacedintoswimlanes[OMGOnSwimlanes]basedon
whichpartyhasprimaryresponsibilityforthestage.

Acceptanc

Fi
A
Acommo
software;
conceptt
Developm
specificat
TheProdu
acceptsth
userswho
businessp
warranty
Agileprod
Elaborate
AnAgileP

ceTestEngin
igure1A
SequentialP
nscenariois
thisistheco
oproduceas
mentTeambu
ioninsteado
uctOwnerthe
hesoftware.W
othenusethe
problem.The
period.
ductdevelopm
e,BuildandAc
ProductDeve
eeringGuide
ProductDevelo
forabusines
oncept.Theb
specification
uildssoftware
fcustomersa
enassessesh
Whenthesof
esoftwareto
eProductDev
mentprocess
cceptactivitie
lopmentLifec
eVolumeI,
opmentLifecy
sspersontoh
usinessperso
oftheproduc
etosatisfythe
atisfactionma
howwellitsa
ftwareisdee
orealizethein
velopmentTe
sblurrthelin
esoftheprod
cycle.
ReleaseCand
ycle
haveanideaf
on(hereincal
ctthatonceb
especificatio
ayoftenlead
tisfiesthene
medaccepta
ntendedvalu
eamisusually
esbetweent
ductdevelop
didate1
forhowtoso
lledtheProdu
builtwilldeliv
neventhoug
tothewrong
edsoftheint
ble,thesoftw
ethatsolves
yobligatedto
hetimingand
mentprocess
lveabusines
uctOwner)el
vervalue.The
ghfocussingo
gproductbui
tendedusers
wareismade
theoriginally
providesupp
dresponsibili
sasillustrated

sproblemwi
laborateonth
eProduct
onthe
iltphenomen
andifsatisfie
availabletot
yidentified
portfora
itiesofthe
dinFigure1B
28
th
he
non.
ed,
the
B

Acceptanc
Fi
A
Agileprod
andtheP
artifactsb
involvedi
toassess
Elaboratio
Asaresul
therebys
end.Incre
aboutwh
purpose.
Thereare
lifecycleo
TheSt
TheV
ceTestEngin
igure1B
nAgileProdu
ductdevelopm
roductDevel
butrathercol
ntheelabora
theproducta
on,Buildinga
t,testingthe
preadingthe
ementaldeve
atistrulynee
Seethesideb
eseveraldiffe
ofasoftware
tageGate
TM
VModel.
eeringGuide
uctDevelopm
mentischara
opmentTeam
laborateonp
ationofthere
asitisbeingb
andAcceptan
productforb
testingactivi
elopmentand
ededandtrul
barUsingWh
erentwaysto
intensivepro
Process.
eVolumeI,
entLifecycle
acterizedbyfa
m.Whileeach
producingthe
equirements
builtratherth
ceisdoneinc
bothaccepta
itythroughou
dincremental
lypossiblean
holeTeamsto
describethe
oduct.Someo
ReleaseCand
acetofacec
hhastheirow
em.Thedeve
andproduct
hanwaitingfo
crementallyin
nceandqual
utthedevelop
,earlytesting
ndtypicallyre
oBuildProdu
roleoftestin
ofthewellkn
didate1

ollaboration
wnresponsibi
lopmentteam
designandth
orafinalacce
nsmallchunk
ityassurance
pmentlifecyc
gallowalotm
esultsinprodu
ctsformoreo
ngandaccept
nownmodels
betweenthe
ilities,theydo
mismorelike
heproductow
eptancephas
ksratherthan
estartsearlyi
cleratherpus
moreopportu
uctswithbet
onthemotiva
tanceinthed
are:
ProductOwn
onthandoff
elytobedirec
wnerisavaila
eoftheproje
ninasinglep
intheprocess
shingittothe
unityforlearn
terfitnessfo
ationbehind
development
29
ner
f
ctly
able
ect.
pass.
s,
every
ning
r
this.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 30
Thefirstisaproductdevelopmentprocessthatisnotspecifictosoftwareintensivesystems.Itdescribes
howtransitionsbetweenthedifferentstagesoftheproductdevelopmentlifecyclecanbemanaged.The
latterisaclassicalmodelspecifictosoftwaredevelopment.Itemphasizesequallyboththefrontend
andbackendoftheproductdevelopmentlifecyclebyexplainingtherelationshipsbetweendifferent
kindsofstagesandartefactscommonlyassociatedwithfrontendofthelifecycleanddifferenttypesof
testingandvalidationactivitiesthattraditionallytakeplaceatitsbackend.
Sidebar:UsingWholeTeamstoBuildProducts
Oneofthekeyformsofwasteinsoftwaredevelopmentishandoffsbetweenhighlyspecializedroles.A
keypracticeinagileandleanproductdevelopmentistheeliminationofhandoffswheneverpossible
throughtheuseoftheWholeTeamapproach.TheWholeTeamapproachbringseveryskillneededto
developtheproductontoasingleProductDevelopmentTeamwhichworkswiththeProductOwnerto
designthemostsuitableproductbasedontheneedsandconstraints.Theteamcollectivelycommits
todeliveringincrementsofpotentiallyshippableproductfunctionality.Thiseliminatesdependencies
betweenteamsandallowstheProductOwnertoworkdirectlywiththeteamtodesignthebest
possiblesolutiontothebusinessneeds.
Asanexample,theScrummethodadvocatestheuseoftheWholeTeamapproach.Scrumproponents
claimtypicalimprovementinproductivityof5xto10x(500%to1000%)[PrimaveraOnScrum,
KnibergOnScrumFromTrenches].Includingeveryoneneededtodelivertheproductonasingleteam
allowstheteamtocontinuallyimproveitsprocessbyeliminatingwaste.Organizationalboundariesdo
notgetinthewayofthisprocessstreamliningbecauseeveryoneisonthesameteamstrivingtoreach
thesamegoal:deliveringvaluableproducttotheProductOwner.
ThereisanongoingdebateintheagilecommunitywhethertheProductOwnerispartoftheteamor
separatefromit.Thereisnosingleanswertothisquestionasitdependsonthenatureofthe
organization(s)andpeopleinvolved.Thepragmaticansweristhatthereareoftentwolevelsofteam
atplay:
1. TheWholeTeamincludingtheproductownerandanyonehelpingthemdefine,buildandverify
theproduct.
2. TheProductDevelopment(sub)TeamwhichbuildstheproductandtheProductOwner(sub)Team
whichacceptstheproduct.Thesetwosubteamsmustworkcloselytogetherandoftensitinthe
sameteamroom.
<maybeagraphicshowingthetwosubteamsandtheskillswhichmightliveineachandwherethey
overlap>
Theexactbreakdownofwhichskillsareoneachsubteamvariesfromcasetocase.Developersusually
belongtotheProductDevelopmentTeamandanalystsusuallybelongtotheProductOwnerTeam.
Testers,documentationwriters,interactiondesignsmayallbelongtoeithersubteam.Theimportant
thingisthatalltheskillsarepresentandaccountableforworkingtogethertodeliverthebestpossible
productwithfewifanyexternaldependencies.

Acceptanc
1.1.1 P
Manypro
processto
decisions
streamof
[CooperO
associated
exclusive:
mightbe
suchapro
Figure1c
ASequen

Stagegat
andevery
Incremen
activities
andsimila
areexecu
example,
maybeac
ceTestEngin
Processesw
oductdevelop
oevolveanid
thatcontrol
falternatings
OnDoingItRigh
dresourceco
:theprojectc
happeningat
ocess.
tialProcessw
eprocessesn
ygatebeingd
talFundingM
anddecisions
artotheIFM,
utedacrossst
theproductd
ccepted.The
eeringGuide
withStage
pmentproject
deaintoadep
progressionf
stagesandga
ht].Thegate
ommitmentim
canonlybein
tthesametim
withStagesan
neednotbes
distinct.Anex
Method(IFM)
sthatarerep
,theacceptan
agesanddec
development
projectcould
eVolumeI,
esandGat
tsgothrough
ployedprodu
fromonestag
tes,aspropo
sareassociat
mplications.
nonestageat
me.:Figure1
ndGates
strictlysequen
xampleofan
[DenneHuan
peated.Incon
nceprocesst
cisionsthatar
tteammayb
dpartiallybe
ReleaseCand
tes
htheirlifecycl
ct.Ifthepro
getoanother
osedintheSta
tedwithgo/n
Inastrictlyse
tatimealtho
cASequent
ntial[Cooper
iterativeproc
ngOnIFM]wh
ntrasttothe
thatwedescr
remademany
uildmanyrel
intheDeploy
didate1
lebyadoptin
ocesssdistinc
r,theprocess
ageGatePro
nogoorscop
equentialpro
oughwithina
tialProcessw
OnStageGate
cesswithstag
osestagesan
strictlyseque
ribeinthisgu
ytimesovert
easecandida
ystage(apre
ganessentia
ctstagesareg
scanberepre
cess
pechangedec
ocess,thestag
astagemany
withStagesan
e]inthesense
gesandgates
ndgatesinvol
entialdepictio
idehasthesa
thelifetimeo
atesfortestin
eviousrelease
allysequentia
guardedby
esentedasa
cisionswith
gesasmutua
kindsofactiv
ndGatesde
eofeverysta
sisthe
lvethesame
oninFigure1
ameactivities
ofaproject.F
ngbutonlyaf
ehavingpasse
31
l
lly
vities
picts

age
1c
sthat
For
few
ed

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 32
throughthegateaftertheAcceptstageGate5inFigure1c)eventhoughtheproductdevelopment
teamisworkingonprovidinganotherreleasecandidateforacceptancetesting.Theresultofsuch
dynamicsistheintroductionofmanystaggeredparallelprocessstreamswiththeirownstagesand
gates,interactionsbetweentheseparallelstreams,andloopswithinandacrossthestreams(seethe
SidebartitledRecastingaSequentialProcessesthroughWorkflowScheduleIndependenceand
Parallelization).Thedivergencefromthetraditionalsinglesequentialflowisinlinewiththemore
modernandflexibleinterpretationsandadaptationsadvocatedby[CooperOnStageGate].Howeverthe
acceptanceprocesswillfeatureintermediatestagesthatdirectlyfeedintosubsequentstageswithout
explicitgatesinbetween.Inaddition,thegates,ratherthanexplicitlycontrollingfundingdecisionsfor
subsequentstages,primarilydictatecontrolflow.Thesevariationsmaybeconsideredasdepartures
fromtheoriginalStageGateProcess[CooperOnDoingItRight].Thestagegatelikemodelunderlying
theacceptanceprocessisexpoundedinanothersidebartitledRepresentingtheAcceptanceProcess
withStagesandGatesinChapter2.

Sidebar:RecastingaSequentialProcessesthroughWorkflowScheduleIndependenceand
Parallelization
CoreyLadas[LadasOnScrumban]providestwopowerfulinsightsonsoftwareprocessesbasedon
Kanbansystems[HiranabeOnKanban]andleandevelopment[PoppendiecksOnLean]thatallowsusto
seetheminanewlight.Thefirstinsightexplainshowworkflow,theorderandinterdependenceof
stepsinherentinasoftwareprocess,canbeseparatedfromhowtheworkthatflowsthroughthe
processisscheduled.Thisiscalledworkflowscheduleindependence.Thesecondinsightallows
reorganizingasequentialworkflowasparallelstreamswithmergingpointswhereworkfromthe
streamscanbeintegrated.Whenbothideasarecombined,softwareprocesses,regardlessofwhether
theiressentialworkflowissequentialcanbemadeiterative,incremental,andparallel.Forany
process,theessentialworkflowisthesequencingofsteps,frombeginningtoend,appliedtoaworking
system,toimplementanewimprovement,whetheranewpieceoffunctionalityoranewnon
functionalrequirement.
Thiskeyideabehindworkflowscheduleindependenceisindependenceofrequirements.
Independenceofrequirementsinturnresultsfromhowworkisdividedupinsmallerchunksinthe
beginningofaworkflow,oftenbyanelaborationorrequirementsanalysisactivity.Ifthechunks,the
resultinglowlevelrequirementsthattheproductdevelopmentteamtransformsintoworking
functionality,aresmallandcanindividuallybecompletedallthewaytointegration,acceptance
testing,anddeployment,theyareindependent.Theextenttowhichthisconditionissatisfieddictates
whetherthechunkscanbescheduledtoflowthroughtheprocessindividually,ingroupsofsmaller
batches,orasasinglebigbatch.Themoreindependentthechunksare,thesmallerthebatchescan
get,downtotheleveloftheindividualrequirements.Iftheyformindependentgroupsof
interdependentchunks,thenthegroupscanbescheduledindependently,butnotthechunks
themselves.Regardlessofthegranularityofthebatchesthatflowthoughtheprocess,theunderlying
essentialworkflowstaysthesame,buttheworkflowexecutedagainforeachbatch.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 33
Thesecondinsight,whichLadasreferstoaspipelining,leveragesanyindependenceofthestepsofthe
essentialworkflowitselfinsteadofthechunksofworkthatflowthroughit.Thestepsthatdontshare
resourcescanbeparallelized.Theresultofsuchparallelizationisworkflowwithstaggeredbranches
andminimalunusedcapacity.Parellelizationincreasestheefficiencyoftheprocessprovidedthatthe
outputsofparallelstreamscanbeintegratedinsuchawaythecapacitygainintroducedbythe
parallelizationexceedstheextraoverheadofintegration.
Thusworkflowscheduleindependenceandparallelizationtogethermayallowaseeminglystrictly
sequentialprocesstobeexecutedinahighlyiterative,incremental,andparallelmanner,makingit
moreefficient,flexibleandresponsivetochange.Theyapplytovariousdegreestomanysoftware
processesandlifecyclemodelswithsequentialdepictions.Inparticularworkflowschedule
independenceandparallelizationexplainhowthevariousdevelopmentprocessesontheprocess
continuumdiscussedinIntroductioncanbederivedfromtheessentiallysequentialworkflow
underlyingtheclassicalwaterfallmodel.
Ladassinsightsalsoapplytotwoadditionalrelatedprocessmodelsthatwediscussinthischapter.
BothofthesemodelsprocessesexpressedintermsofstagesandgatesandtheVModelhave
sequentialdepictionsnottoodissimilartothatoftheclassicalwaterfallprocessdiscussedin
Introduction.Andbothofthesemodels,liketheclassicalwaterfallprocess,areamenabletorecasting
inaniterative,incremental,andparallelmannerbyapplyingworkflowscheduleindependenceand
parallelization.Suchrecastingmayoccuratdifferentgranularities:atthelevelofwholereleases,
iterationswithinreleases,featuresets,orindividualfeatures.

1.1.2 TheVModelofSoftwareDevelopment
InthedevelopmentofsoftwareintensivesystemstheVModeliscommonlyusedtodescribethe
relationshipsbetweenactivitiesfocusingonbuildingthesystemandactivitiesforsystemverification].A
variantoftheVModeladaptedfrom[FewsterGrahamOnTestAutomation]isshowninFigure1dTheV
ModelofSoftwareDevelopment.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 34

Figure1dTheVModelofSoftwareDevelopment

InFigure1d,activitiesandartefactsthatareassociatedwithbuildingthesystemareshownontheleft
sideoftheVshape.Testingactivitiesareassociatedwithsystemverification,andtheseareshownon
therightsideofthefigure.Testsspecifictoalayertietogethertheactivitiesandartefactsofthatlayer
shownontherightsideofthelayertotheverificationrelatedactivitiesofthesamelayershownonthe
left.Testsforeachlayercanbedefinedearlyduringtheactivitiesthatareassociatedwithbuildingthe
system,buttheycanbeexecutedonlyafterdevelopmenthasprogressedsufficientlytotherightsideof
theVshapeintothecorrespondingverificationactivity.NotethatthisinterpretationoftheVModelcan
accommodatebothsequentialortestlaststyleverificationandincrementalortestasyougostyle
verification.TheprinciplesdescribedintheSidebartitledRecastingaSequentialProcessesthrough
WorkflowScheduleIndependenceandParallelizationapplytotheVModel.Thereforetheunderlying
workflowcanbeexecutediterativelyatdifferentlevelsofgranularity,onafeaturebyfeature,iteration
byiteration,orreleasebyreleasebasis.
Aftercodinghasbegunandtheunittestsfortheimplementedpiecesareinplace,unittestingmay
start.Asacomponentsunitsareimplemented,certainAPIsofthecomponentmaybecomefunctional.
IfthecorrespondingcomponenttestsforthoseAPIsareinplacethencomponenttestingforthatmay
begin.Continuingthisprogression,weclimbuptherightsideoftheVshape:theunitsarerolledinto
componentsandultimatelycomponentsarecomposedintoproductfeaturessuchthatthebusiness
requirementsarereadytobeexercised,thusreachingtheacceptancetestingactivityatthepinnacleof
therightside.Theacceptanceprocessdescribeswhathappensatthisfinalactivityofthetoplayer.We
makethedecisiontoacceptorrejectthesoftwarebasedonhowwellitmeetsthebusiness
requirementsshownonthetopleftsideoftheVModel.
Whenappliedinastrictlysequentialmanner,weassociatetheAcceptingTestingactivitywithadistinct
AcceptanceTestPhase.ContrastedwithpracticessuchasAcceptanceTestDrivenDevelopmentand
IncrementalAcceptanceTestingadvocatedinagilesoftwaredevelopment,theAcceptanceTestPhase
undertakentowardtheendofthedevelopmentlifecyclerepresentsthetestlastapproachto
acceptancetesting..Thespecificpracticescommonlyusedintheagilesoftwaredevelopmentcontext
Build Verify
Busi ness
Requi rements
Uni ts
Uni t
Testi ng
Acceptance
Testi ng
Components
Component
Testi ng
System
Testi ng
Product
Requi rements
Tests
Tests
Tests
Tests
Desi gn
Archi tecti ng
Codi ng
Anal ysi s

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 35
aredescribedlaterinthischapterinsectionTheAcceptanceProcessforHighlyIncremental
Development.
1.2 PartiesoftheAcceptanceProcess
Theacceptanceprocessdescribestheinteractionsbetweentwokeyparties:theProductOwnerandthe
ProductDevelopmentTeam.Eachhasveryspecificresponsibilitiesintheproductdevelopmentprocess.
Whilethenamesofthesepartiesmaycoincidewithnamesusedinspecificmethods(suchasScrum),we
provideourowndefinitionsforthepurposesofthisbook.
1.2.1 ResponsibilitiesoftheProductOwner
TheProductOwneristhepartywhohascommissionedtheconstructionoftheproductbutdoesnthave
thetimeorskillstobuilditthemselves.TheProductOwnerisresponsibleforclearlycommunicating
theirexpectationsoftheproducttotheProductDevelopmentTeamandfordecidingwhetherthose
expectationshavebeensatisfiedbytheproductdeliveredtothem.Thedetailedresponsibilitiesofthe
ProductOwnerinclude:
Understandthepotentialusersoftheproductoridea.
Determinewhatcapabilitiestheproductneedstohavetoaddresstheusersneeds.
Determinetheamountofresourcestheyarepreparedtoinvestintobuildingtheproduct.
Specifythedesignoftheproduct(notthedesignofthesoftwareinsidetheproduct).
HelptheProductDevelopmentTeamunderstandthepotentialusersandtheirneedsandhowthe
productwilladdressthem.
Definetheacceptancecriteriathatmustbemetbeforetheproductcanbeaccepted.Theseinclude:
Examplesofhowtheproductwillbeusedbyusersandhowtheproductshouldreactin
eachexample.
Criteriaregardingnonfunctionalpropertiesoftheproductincludingresponsetime,
availability,accessability,configurability,etc.
Defineanyconstraintsontheproductincludingtechnicalconstrainsts(whattechnologycanormust
beused).
AssessthecostanddeliverydateestimatesprovidedbytheProductDevelopmentTeam.
Makethedecisionastowhethertheproductisworthbuilding.
TheProductOwnermaycarryouttheseresponsibilitiesthemselvesortheymaydelegatetomembersof
theirteam(knownastheProductOwnerTeam.)Theymayalsoenlistthehelpofpeopleoutsidethe
ProductOwnerTeamtohelpthemcarryouttheirresponsibilities.ThisincludesaskingtheProduct
DevelopmentTeamtodosomeofthework.Butneitherdelegationnorcollaborationabsolvesthe
ProductOwnerfromultimateresponsibilityforthem.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 36
AgoodProductOwnerhasaclearvisionofwhatdonemeansandsharesthatvisionwiththeProduct
DevelopmentTeamasearlyaspossible.AgoodProductOwneralsorecognizesthatcommunicationis
inherentlyflawedandthatyouneedtoregularlyverifythatwhatyouintendedtocommunicatehas
beenunderstood.Therefore,agoodProductOwnerispreparedtotryoutandprovidefeedbackonthe
softwareunderdevelopmentatfrequentintervals;agoodProductOwnerdoesntsimplythrowa
specificationoverthewallandsaydontbothermeuntilyouarereadyforacceptancetesting.They
takeanactiveinterestintheemergingsoftwareandencouragetheProductDevelopmentTeamto
providefrequentopportunitiesforfeedback.
1.2.2 ResponsibilitiesoftheProductDevelopmentTeam
TheProductDevelopmentTeamisthepersonorpersonswhohaveacceptedthetaskofbuildingthe
productcommissionedbytheProductOwner.TheProductDevelopmentTeamisresponsiblefor
deliveringaworkingproductthatmeetstheProductOwnersacceptancecriteria.Thedetailed
responsibilitiesoftheProductDevelopmentTeaminclude:
Determinehowtherequestedfunctionalityshouldbeachievedincluding:
Hardwarevs.firmwarevs.softwarepartitioning.
Determinewhattechnologieswillbeusedtobuildtheproduct(unlessthesewerepartof
therequirementsorconstraintsprovidedbytheProductOwner)
BuildtheproductandverifythatitmeetstheProductOwnersexpectationstothebestoftheir
ability.Theproductincludes:
Workingsoftwareandhardware
Whateveruserdocumentationisrequired(unlessthisisprovidedbytheProductOwneras
partoftherequirements.)
Whateverartifactsarerequiredtomaintainthesoftwareincludingsoftwareandhardware
designdocumentation,unitandcomponenttests,testtoolsrequiredtotestthe
product,etc.
TheProductDevelopmentTeammaycarryouttheseresponsibilitiesthemselvesortheymayenlistthe
helpofpeopleoutsidetheProductDevelopmentTeamtohelpthemcarryouttheirresponsibilities.But
neitherdelegationnorcollaborationabsolvestheProductDevelopmentTeamfromultimate
responsibilityforthem.
1.2.3 ResponsibilitiesofOtherSpecialities
Thenatureoftheproductbeingbuiltoftendictatesthekindsofspecialiststhatneedtobeinvolvedin
theproject.ThesespecialistsmaybeengagedbyeithertheProductOwnerTeamortheProduct
DevelopmentTeam.Regardlessofwhichpartytheyworkwith,theyneedtounderstandtheirownrole
indeliveringaproductthatsatisfiesthepotentialusers.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 37
1.3 TheBasicAcceptanceProcess
ThebasicacceptanceprocessatitshighestlevelsubsumestheBuildAcceptUsecycledepictedin
Figures1Aand1B.EachofthehighlevelstagesshowninFigure1Aand1Binvolvesmultiplesubstages
oractivitiesandanexitdecision.
TheBuildstageiscomposedoftheproductsconstruction,itsreadinessassessment,andasubsequent
readinessdecision,allperformedbytheproductdevelopmentteam.
TheAcceptstageiscomposedofacceptancetestingoverseenbytheproductowner(andsometimes
performedincollaborationwiththeproductdevelopmentteam),andasubsequentacceptancedecision
madebytheproductowner.Normally,thereadinessdecisionandtheacceptancedecisionaremadeon
thesamesetofcriteria.Theprecedingactivities,readinessassessmentandacceptancetesting,may
involveoverlappingactivities,althoughreadinessassessmentmaybemoreinternallyfocusedthan
acceptancetesting.Howeverthetwodecisionsdifferintheirgoals.Readinessdecisiondetermines
whethertheproductisreadyfortheproductownertoevaluateandtheacceptancedecisiondetermines
whethertheproductmeetsallitsrequirementsandisreadytobeused.Thelaststage,theUsestage,
mayinvolveafinal,userfacingevaluativeactivityineitheraproductionorenduserenvironmentto
determinewhetheritisusableanddeployabletoitsintendedaudience.
Figure2TheAcceptanceProcessdrillsdownintoeachoftheBuild,Accept,andUsestagestoexpress
themintermsoftheirlowerlevelcomponents..Thefigureshowsonlythepathassociatedwiththe
positiveoutcomesoftheunderlyingdecisions.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 38
Figure2
TheAcceptanceProcess
Intheidealworldthissequenceofactivitieswouldbedoneexactlyonceandtheassessmentcouldbe
doneentirelybytheProductOwner;inpractice,thiscouldbearecipefordisasterasuntestedsoftware
cancontainlargenumbersofbugsperthousandlinesofcode.Itusuallytakesmanytriestobuilda
productthattheProductOwnerfindsacceptablethereforetheacceptanceprocessistraversed,atleast
partially,manytimes.Toensurethataqualityproducthasbeenbuiltandtherebyminimizethenumber
oftimestheProductOwnerisaskedtoacceptthesameproduct,mostProductDevelopmentTeamswill
includesomelevelofselfassessmentofeachreleasecandidatebeforeprovidingthesoftwaretothe
ProductOwnerformakingtheacceptancedecision.Inthisbookwecallthetestingandother
verificationactivitiesthatareperformedpriortoaskingtheProductOwnertoacceptthesoftware
readinessassessmentandthedecisiontohandoffthesoftwareiscalledthereadinessdecision.The
testingdonebytheProductOwnerortheirproxyafterreceivingthesoftwarefromtheProduct
DevelopmentTeamiscalledacceptancetestingandthedecisionwhethertoacceptthesoftwareinits
currentstateiscalledtheacceptancedecision.Thedecisioneachusermakeswhetherornottoactually
usetheproductonceitisavailabletothemiscalledtheusagedecision.
Eachdecisioncanresultinapositiveornegativeoutcome.Onlythepositiveoutcomesareshownin
Figure2;thenegativeoutcomesareshowninFigure3PathsthroughtheAcceptanceProcess.
ThetransitionsfromtheBuildstagetotheAcceptstageandfromtheAcceptstagetotheUsestage
typicallyrequirethemovementofsoftware(orsoftwarecontainingproducts)fromoneenvironmentto
another.Forthepurposesofthisdiscussion,thespecificstepsinvolvedinmakingthesoftwareavailable,
thoughimportantforacceptance,arenotconsideredapartoftheacceptanceprocess.
1.3.1 AssessingReleaseCandidatesusingtheAcceptanceProcess
Theversionofthesoftwarethatisputthroughtheacceptanceprocessisoftenreferredtoasarelease
candidate.Itgoesthroughthereadinessdecisionandacceptancedecisionprocessesstepbystepand
decisionbydecisionuntilitmeetsoneofthefollowingrequirements:
Itpassesthroughallthedecisionsandisdeemedreadyforusebyusers.
Itisdeemedinsufficientinsomeway;atwhichpoint,itissentbacktoanearlierphase.

Anegativeacceptancedecisionorreadinessdecisioncancausethereleasecandidatesoftwaretobe
sentbacktoanearlierpointintheprocess.
Figure3PathsthroughtheAcceptanceProcessillustratesthepossiblepathsthroughtheacceptance
processwhenthereadinessoracceptancedecisionscannotbemadeandrequireadditionalcapabilities
orinformation.Inthefigure,QualityDatareferstotheinformationaboutthequalityofthesystem
obtainedfromreadinessassessmentoracceptancetesting.

Acceptanc
Fi
Pa
Fromthe
dataobta
Ifthedata
missingo
forrewor
Fromthe
datafrom
readiness
readiness
candidate
Normalp
Developm
beenrun
stopsunti
candidate
distinctly
system.)
Whenala
decidewh
futurerel
eachbug.
organizat
thebugs,
backloga
Troubles
ceTestEngin
igure3
athsthrough
readinessde
inedfromthe
afromtheac
rtheproduct
k.
acceptanced
macceptance
assessment
assessment.
ecanbesent
racticeistolo
mentTeamca
ortestingisb
ilanewrelea
eisbuiltandp
namedversio
argenumber
hichbugsmu
ease,aproce
.Whilethisis
ionalculture
mostorganiz
ndchangingh
hootingtheA
eeringGuide
theAcceptan
cision,arelea
ereadinessa
cceptancetes
thassevered
decision,arel
isinsufficient
anditisdeem
Ifcriticalfun
backtodeve
oganybugsf
nstartthepr
blockedbyab
asecandidate
passedthroug
onofthesoft
ofbugsisfou
stbefixed,w
essknownas
acommonp
andstructure
zationswould
howtheybui
AcceptanceP
eVolumeI,
nceProcess
asecandidate
ssessmentis
tingphaseis
deficiencies,t
leasecandida
t.Iftheaccep
medtobeins
nctionalityism
lopmentforr
foundintheb
rocessofrem
bugthatprev
eisreceived.A
ghtheaccept
tware(andsh
undduringth
whichrequire
BugTriage.B
practice,itist
eoftheenter
dbebetterse
ldtheirprodu
rocessinCha
ReleaseCand
ecanbesent
insufficientt
sufficientto
thereleaseca
atecanbesen
ptancedecisio
ufficient,the
missingorhas
rework.
bugmanagem
mediationbut
ventsfurther
Afterremedia
tanceprocess
houldbetagg
eacceptance
furtherinves
Bugtriagingi
typicallyasym
rprise.Rather
ervedbyunde
uctstoreduc
apter20Fin
didate1
backtoread
omakeawe
determineth
andidatecan
ntbacktoacc
onalsodepen
releasecand
ssufficiently
mentsystems
tocontinuet
testingfrom
ationofoneo
s.Eachreleas
edassuchin
eprocess,the
stigationand
nvolvesassig
mptomofdee
rthanfocusin
erstandingth
ethenumbe
eTuningthe
dinessassessm
llinformedre
hatcriticalfun
besentback
ceptancetest
ndsondatac
didatecanbe
severedeficie
sothatthePr
testinguntila
occurring.At
ormorebugs
secandidates
thesourceco
eProductOw
whichcanbe
gningapriorit
eperissuesin
ngonbecomi
erootcauses
rofbugsfoun
Acceptance

mentifthequ
eadinessdeci
nctionalityis
todevelopm
tingifthequa
ollecteddurin
sentbackto
encies,there
roduct
allthetestsh
tthispointte
s,anewrelea
shouldbea
odemanagem
nermayneed
edeferredto
tyandseverit
nthe
ngbetteratf
softheirbug
nd.Thesecti
Processoffer
39
uality
ision.
ent
ality
ng
elease
ave
esting
se
ment
dto
a
tyto
fixing
ion
rs

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 40
somepossiblerootcausesofalargebugbacklogandpossibleavoidancestrategies.RefertotheSidebar
IncrementalAcceptanceandBugTrackingonAgileProjectsinChapter8.
Note:Althoughtheacceptanceprocessoutlinedhereappearstobesequentialwiththesoftware
movingfromonestagetoanother,thisisnotalwaysthecase.Onwellrunprojects,followingthe
practiceofincrementalacceptance,eachfeaturemaytraversethisprocessindividually.Therefore,itis
possibleforonefeaturetobeintheconceptstage,withthedevelopmentteamworkingonanother
setoffeaturesinthedevelopmentstage,andanothersetoffeatureswiththeProductOwnerinthe
acceptancestage.Thatisatleastthreeinstancesoftheprocessrunninginparallel.
Evenonclassicalwaterfallprojectswemayhavemorethanonereleasecandidategoingthroughthe
processatthesametime.Forexample,theAlphareleasemaybeInUse,theBetaisinAcceptance
Testinganddevelopersmaybeworkingonadditionalfunctionalityforthegeneralreleasewhilealso
fixingbugsinthenextreleasecandidateforthisrelease.Thatisfourinstancesoftheprocessrunning
inparallel.Suchparallelismhoweverisnotwithoutimplicationsduetoincreasedintegrationand
coordinationoverhead.Inparticular,sourcecodemanagementbecomescomplexduetobranching
andsubsequentmergingofsourcecodetrunks(forprovenbranchingstrategiesinTFS,see
[MSOnTFSBranchingGuide]).Bugtrackingbecomescomplexduetotheneedtocoordinatetheissues
fromthereleasesunderacceptancetestingwiththefixesandadditionsinthereleasesunder
development.
Amorecompletediscussionofoverlappingreleasesfollowslaterinthischapterundertheheadings
TheAcceptanceProcessforAlpha&BetaReleases.Forthegeneralprinciplesgoverningincrementality
andparallelism,refertotheSidebarRecastingaSequentialProcessesthroughWorkflowSchedule
IndependenceandParallelization.
WhySeparateReadinessAssessmentfromAcceptanceTesting?
Theprocessofverifyingandvalidatingthesoftwareintensivesystemagainsttherequirementsand
expectedusagecanbeviewedasasinglemonolithicactivityorasveryfinegrainedsteps.Theprimary
reasonforgroupingthesefinegrainedstepsintotwomajorbucketsistoseparatetheactivitiesthat
shouldbecarriedoutbytheProductDevelopmentTeambeforeturningthesystemovertotheProduct
OwnerfromthosethattheProductOwnerdoesaspartofdecidingwhethertoacceptthesystem.
ReadinessassessmentisprimarilyabouttheprofessionalismoftheProductDevelopmentTeamwhile
acceptancetestingisaboutvalidatingthatthesystemasdeliveredwillsuitthepurposesoftheusers
andotherstakeholdersandconfirmingcompliancetocontractualagreements.
Theremaybeotherreasonstodividetheactivitiesintovariouscategories.Wementionthemonly
brieflybecausetheyarebeyondthescopeofthisguide:
DeadlinePressureSomemanagersmightbelievethathavinganearlier,separatemilestonebefore
handingoversoftwaretotheproductownerisaneffectivewaytomotivatedevelopers.
AccountingTheinitialconstructionofthesoftwaremaybetreateddifferentlythanfixingofbugs
fromanaccountingperspective(especiallyworkthatmaybeconsideredasachangerequest
mayaffectprojectaccountingdifferentlythanworkthatmaybeconsideredasabugfix).

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 41
EarlyValidationItisusefultohaverealuserstryusingearlyversionsoftheproducttovalidate
thattheproduct,oncefinished,willfillthenicheforwhichitwastargeted.Thisclearlyisntfull
onacceptanceoftheproductsoitcouldbeconsideredreadinessassessment.Priortodoingthis
typeoftestingwithusers,theProductDevelopmentTeamwouldwanttodotheirowndue
diligencetoensurethesoftwarewasworkingwellenoughtoallowtheuserstotryit.Therefore,
wewouldconsiderthisaformofAlpha/Betareleaseorpossiblyincrementalacceptancetesting
orevenconditionalacceptance.
EndUserTrainingExposingthesoftwaretousersbeforeacceptancecanbeaneffectivewayto
startthetrainingprocess.Ifthequalityofthesystemishighenough,itcanalsobeusefulasa
formofviralmarketing.BothoftheseusesfallintothecategoryofAlpha/Betareleasesrather
thanreadinessassessment.
ReadinessAssessmentorAcceptanceTesting?
Givenaparticulartestingactivity,shoulditbeconsideredpartofreadinessassessmentoracceptance
testing?Thereareseveralfactorsthatplayintothisdecision:
1. Whosdoingtheassessing:AcceptanceisperformedbytheProductOwnerortheirproxy
(someoneactingdirectlyontheProductOwnersbehalf)whilereadinessassessmentcanbe
donebyanyoneinvolvedintheproject.WheretheProductDevelopmentTeamandProduct
Owneraredistinctorganizationalentities,therolesshouldbefairlyclear.Thingsgetmurkier
whenthereisnorealuserorendcustomerinvolved,commoninproductcompanies,orwhen
thereisaseparateorganizationchargedwithtesting.Thesescenarioswillbecoveredinmore
detailinChapter2TheDecisionMakingModel.
2. Formalityofthetesting:AcceptanceisamoreformalactivitybecausetheProductOwneris
involved.Thisimpliesmoreformalrecordkeepingofanyconcernsthatwereidentified.
Readinessassessmentcanbemuchlessformal;itneedonlybeasformalasdictatedbythe
projectcontext.ProductDevelopmentTeamsthatneedtobeabletopassformalauditswill
keepmuchmoreformalrecords.Agileprojectstendtobeveryinformal;duringreadiness
assessmenttheyjustfixthebugsimmediatelyratherthandeferthemforfixinglater.

Thelinescangetsomewhatblurredwhentherearemorethantwodifferentstagesoftesting;these
scenariosaredescribedinAcceptanceinComplexOrganizationsandAcceptingComplexProductsin
Chapter2ElaboratingontheAcceptanceProcess.Intheend,itislessimportanttodecidewhichlabel
toapplytoaparticulartestingactivitythanitistoensurethattherighttestingactivitiesgetdoneand
thateachpartyknowsforwhichactivitiestheyareresponsible.Itcanbeusefultolistallthepotential
testingactivitiesandforeachonedecidewhetheritismandatory,optional,ornotrequired,andto
assignresponsibilityforittoaspecificpersonororganization.Asamplespreadsheetisprovided<online
athttp://testingguidance.codeplex.com>.

Acceptanc
1.3.2 T
Thusfar,w
theaccep
Forthem
products,
goesthro
Releasesi
Figure4
TheAccep
Thesetof
orMMF
1
,
qualitycri
fromrelea

1
Themin
termwas
Method,a
toreferto
afterdepl
thedelive
theindust
ceTestEngin
TheAccept
wevefocuse
ptanceproces
mostpart,long
whereeach
ughtheentir
illustratesan
ptanceProces
ffunctionality
isuniquebas
iteria,whichw
asetorelease

imumlevelo
coindedbyM
anROIorient
ounitsoffun
loyment.We
eredunitisus
tryasMMP(
eeringGuide
tanceProc
dontheacce
ssgetapplied
glivedmulti
productisbe
redecisionm
exampleoft
sswithMultip
yrequiredfor
sedonthego
wecalltheM
ebutitmaye

ffunctionalit
MarkDennea
tedapproach
ctionalitytha
adoptavaria
sable,butdoe
MinimumMa
eVolumeI,
cessforMu
eptanceproce
whenaprod
releasesyste
eingindividua
akingprocess
hisprocess.
pleReleases
reachrelease
oalsoftherel
MinimumQua
evolveasthe
yisalsosome
andJaneClela
toscopingso
atcanbedep
ationofthist
esnotcovera
arketablePro
ReleaseCand
ultiRelea
essasitappli
ductwillhave
mscanbeth
llyassessedf
s.Figure4T
e,whichwec
easedeterm
lityRequirem
productmat
etimescalled
andHuangin
oftwarerelea
loyedtogethe
erm,Minimu
awholefeatu
oduct)andMC
didate1
aseProduc
iestoasingle
emultiplerele
oughtofass
forreadiness
TheAcceptanc
calltheMinim
inedbythep
mentorMQR,
uresandprod
dtheMinimum
thecontexto
ases[DenneH
erandwillsta
umMarketabl
urebyourdef
CR(Minimum
cts
ereleasecand
eases?
implyaseque
andacceptan
ceProcesswi
mumMarketa
productowne
,issomewhat
ductcontext
mMarketable
oftheIncrem
uangOnSoftw
artgeneratin
leFunctionali
finition.MMF
mCredibleRel
didate.Howd
enceofindivi
nce.Eachrele
ithMultiple
ableFunction
er.Thesetof
tmoreconsis
evolves.The
eFeatures.T
mentalFundin
wareByNumb
gvtangiblev
ity(MMF)in
Fisalsoknow
lease).
42
does
idual
ease
ality
stent
e
This
ng
ers],
value
case
wnin

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 43
specificcriteriawouldbeselectedfromthesetofcriteriaineffectatthetimeoftheproject(whichmay
varyfromthosethatwereineffectforearlierreleases.)AnexampleofthisisthattheSarbanesOxley
Act(SOX)wasenactedin2002,soallsubsequentreleasesrequiredcompliancewiththisactasa
readinessand/oracceptancecriteria.Subsequentreleasesmayalsohavebackwardcompatibility
requirementsthatdidnotexistforearlierreleases.Ideally,theMMFandMQRusedbytheProduct
DevelopmentTeaminmakingtheReadinessDecisionshouldbethesameastheMMFandMQRusedby
theProductOwnertomaketheAcceptanceDecision.Inpractice,thisishardtoguaranteewithout
extensivecollaborationbetweentheProductDevelopmentTeamandtheProductOwnerTeam.
1.3.3 TheAcceptanceProcessforAlpha&BetaReleases
Alphaandbetareleasesarewaystouseendusersasfieldtestersbeforeaproductionqualitygeneral
releasetogathermoredataabouttheproductasitmightbeused"intherealworld."Theendusers
maybeinternaltotheorganization,orexternalbutfriendlier(meaningmorebugtolerant)orhavinga
stakeintheoutcome(wantingtoseetheproductearlierfortheirownbenefit.)Insomeorganizations
internalalphatestingiscalleddogfoodingsonamedbecausewhensomeoneproducesdogfood,they
shouldbepreparedtofeedittotheirowndog.
ThefinaloutcomeofalphaandbetatestingisaslikelytoresultinchangestotheMMFandMQRofthe
product(ineffect,newfunctionalityorqualityrequirements)asitistobeacollectionofbugreports
thatdescribefailuretomeettheexistingMMForMQR.
Eachalphareleaseandbetareleasecanbeconsideredaseparatereleasewithitsownreleasedecision
andacceptancedecision.Thatis,thedevelopmentorganizationneedstodeemittobereadyforan
alpha(orbeta)releaseandtheproductownerneedstoacceptitasbeingreadyforalphareleaseasin:
(Iacceptthisalphareleaseashavingsufficientfunctionalityandqualitytowarrantreleasingtousersto
collectfeedbackThisisillustratedinFigure5TheAcceptanceProcesswithAlphaandBeta
Releases.

Acceptanc
Figure5
TheAccep
Notethat
decision.T
neededfo
theAlpha
be"nosev
response
theAlpha
fromtheA
ceTestEngin
ptanceProces
teveryreleas
Thefunctiona
orabetarele
releasemay
verity1bugs
timerequired
criteriawith
Alphatesting
eeringGuide
sswithAlpha
e,whetheral
ality(MMF)a
ase,whichis
beacoresub
"and"respon
dinproductio
additionalcr
gasshownin
eVolumeI,
andBetaRel
lpha,betaor
andquality(M
lowerthann
bsetoffuncti
ndsinunder2
on)."Typicall
riteriabasedo
Figure6Alp
ReleaseCand
leases
finalhasare
MQR)forana
neededforag
ionality;nota
2seconds95
y,theMQRc
onimprovem
phaorBetaF
didate1
eadinessdecis
lpharelease
generalreleas
allfeaturesne
%ofthetime
criteriaforthe
mentspreviou
Feedback.
sionandana
aretypicallyl
se.Forexamp
eedbeprese
e(versusthe
eBetarelease
slyplanneda
cceptance
lowerthanth
ple,theMMF
nt.TheMQR
1second
ewillbebase
aswellasfeed
44

hat
Ffor
may
edon
dback

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 45

Figure6
Alpha(orBeta)Feedback
Similarly,theMMFandMQRforthegeneralreleasewouldbeaffectedbyfeedbackfromusersofthe
Betarelease.
1.3.4 Soaking,FieldTrialsandPilots
Newproductsareoftentestedwithpilotgroupsbeforebeingrolledouttolargergroupsofusers.A
pilotistypicallythefirstproductionqualityreleasewithfunctionallypickedtosatisfyaparticular
targetaudience.Therefore,theMMFbarislowerbuttheMQRneedstobesufficienttosatisfythe
users.UnlikeanAlphaorBetarelease,thisisrealproductionsoftware.Apilotusergroupwilloften
receiveseveraldotreleasesbasedonissuestheyfindandreport.Theinitialpilotmaybefollowedbya
largerpilotgroup,whomayrequireadditionalfeatures,ortheproductmaygostraighttogeneral
availability.Eachofthepilotreleaseswouldbesubjecttotheacceptanceprocesswithbothreadiness
andacceptancetestingprecedingdeployment.
Asimilarstrategyisknownasfieldtrialorsoaking(thinkofthewashingmetaphor:justlikeclothes
youletthesoftwaresoakforawhile,tomakesureitcomesoutclean).Thesoftwareisdeployedto
friendlycustomers/usersforanextendedperiod(morethanjustatestcycle)toseehowitbehavesina
realcustomer/userenvironment.Aswithapilot,readinessassessmentandacceptancetestingwouldbe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 46
donetoensurethattheMMFandMQRaresatisfied.Theoutcomeinallcasesistogatheruserfeedback
thatmaycausetheproductownertoadjusttheMMFandMQRexpectationsofsubsequentreleases,
nottorevisittheacceptancedecisionofthepilotorfieldtrialrelease.
1.3.5 SoftwareMaintenance
Anytimesoftwareneedstobemaintained(suchaswhensmallchangesaremadetothesoftwareand
thosechangesaredeployed),youare,ineffect,creatingaminorinterim,ordotreleaseofthesoftware
thatneedstogothroughtheentiredecisionmakingcycleyetagain.Itiscommontolookforwaysto
reducethecostofgatheringthedatatosupporttheacceptancedecision.Somewaysofdoingthis
increasetheriskofpossiblymissingnewlycreatedbugs(alsoknownas"regressionbugs")byreducing
theamountoftesting(forexample,riskbasedtestplanning)whileotherssimplyreducetheeffortto
getsimilartestcoverage(forexample,automatedregressiontesting).
Anotheruniqueaspectofsoftwaremaintenancerelatestothewarrantyperiodonasoftwarerelease.
Anychangesthatneedtobemadetothesoftwareshouldbemadeinthesourcecodemanagement
(SCM)system.Whenbuildingmultiplereleases,theremaybeongoingdevelopmentforthenextrelease
thatshouldnot,underanycircumstances,beinsertedintotheproductionsystemalongwiththe
warrantybugfixes.Thisrequiresmanagingseparatecodestreamsorbranchesduringthewarranty
periodandensuringthatallwarrantyfixesarealsoappliedtothenewdevelopmentcodestream.SCM
isalsoneededduringacceptancetestingifthedevelopmentforthenextreleaseismovingforwardin
parallel.Forpracticalstrategiesforusingsourcecodemanagementsystems,see[BerczukOnSCM]and
[MSOnSCM].
1.3.6 ConditionalAcceptance/Readiness
Frequently,theacceptancedecisionmakeracceptsaproductwithconditions.Acceptingaproductwith
conditionsisashorthandwayofsaying,
"Theproductisnotacceptableyet,butitisclosetomeetingourcriteriaforfunctionality(MMF)and
quality(MQR).Ifyouaddressthefollowingconcerns(andwefindnothingnewinthesubsequentround
ofacceptancetesting),weintendtoaccepttheproductinthenextpassthroughthedecisionmaking
process."
Conditionalacceptancebringstheprocessbacktotheconstruction/developmentphaseofthe
acceptanceprocess,butthistimewithamuchbetterideaofexactlywhatmustbedonetomakeit
throughboththereadinessdecisionandtheacceptancedecisiononthenextround.
1.3.7 TheAcceptanceProcessforHighlyIncrementalDevelopment
Theacceptanceprocessasdescribedthusfarlooksverysequentialinnaturebutitcanalsobeusedina
highlyincrementalfashiononagileprojectsbyapplyingworkflowscheduleindependence(seeSidebar
RecastingaSequentialProcessesthroughWorkflowScheduleIndependenceandParallelization).Each
chunkoffunctionality(oftencalledafeatureorauserstory)canbepassedthroughtheacceptance
processindependentlyasshowninFigure7TheAcceptanceProcessAppliedtoIncremental
Development.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 47

Figure7
TheAcceptanceProcessAppliedtoIncrementalDevelopment
TheindividualMMFforeachfeatureoruserstorymaybe:
independentoftheMMFofotherfeaturesoruserstories;or
incrementalwhenitmaybuildontheMMFofpriorfeatures.
TheMQRmaybedifferentfordifferentincrements,althoughitsmoreconsistentacrossincrements.For
example,theMQRforanalphareleaseistypicallylowerthantheMQRforaregular,generalrelease.
WhenMQRrelatestononfunctionalrequirements,itappliesacrossfunctionalfeatures.
Atsomepoint,aproductlevelreadinessandacceptancedecisionsaremadetoensurethateverything
worksproperlytogether.RefertoAcceptingtheOutputofFeatureTeamsinChapter2Elaboratingon
theAcceptanceProcessforamorecompletediscussion.
Inincrementaldevelopment,thedevelopersmaypreexecutetheacceptancetestsWhenallthetests
passforthatfeatureoruserstory,theymayturnoverthefunctionalitytotheProductOwner(orother
customerproxysuchasabusinessanalystoracceptancetester)forimmediate"incrementalacceptance
testing."Therefore,acceptancetestingatthefeaturelevelmaystartimmediatelyaftertheproduct
developmentteamdecidesthatallormostofthefunctionalityforthatfeatureisbuiltandworking
correctly,allowingincrementstooverlap.Thisisaninstanceofincreasingefficiencybyintroducing

Acceptanc
parallelism
Processes
acceptanc
themediu
featurew
Figure8
AgileDeve
Developm
document
thesideba
Acceptanc
developm
readiness
testsagai
oftenauto
Shingosa
[ShingoDi
Whenthe
membero
Ownerfin
immediat
Ownerfo
thedevelo
Acceptanc
acceptanc
theyimpl
beforethe
1.3.8 T
Findingb
overallap
ceTestEngin
mintothewo
sthroughWo
cetestingper
umlengthve
withintheitera
elopmentwit
mentisdoneo
tationwriters
arWholeTea
ceTesting,De
mentworkisc
assessment
nstthesoftw
omatedanda
adageInspec
llonOnToyota
eyaresatisfie
oftheProduc
ndsanyprobl
telyandrepea
rfurtheracce
oper(orteam
ceTesting,ha
cetestingcan
ementedthe
edeveloperm
TheRoleof
bugsduringac
pproachtopr
eeringGuide
orkflowwhen
orkflowSched
rformedatth
rticalbarsrep
ationareillus
thIncrementa
onefeaturea
s,testersand
amformorei
ev1andDev
complete,the
onthefeatur
ware.Iftheyfi
actasaform
ctiontofind
aProductionS
edthattheso
ctOwnerTea
ems,theysho
atsthereadin
eptancetestin
m)startswork
astwokeybe
nbediscussed
functionality
movesontot
fBugFixin
cceptancetes
oductdevelo
eVolumeI,
nresourcesm
duleIndepend
heendofthe
presentingite
stratedinFigu
alAcceptance
atatimebye
duserexperie
nformation.)
2couldeach
edeveloper,p
rebyrunning
indanyprobl
ofmistakep
defectsiswa
System].
ftwareiswor
m)torunthe
owtheproble
nessassessme
ng.Whenthe
kingontheir
enefits.First,a
dwiththede
y.Second,the
thenextfeatu
ngintheA
stingisnorma
opment.Wec
ReleaseCand
maybesepara
denceandPa
iteration,asi
erationwide
ure8.
eTesting
itherasingle
encepeople,d
InFigure8
hbeasingled
possiblyassist
alltheknow
ems,theyfix
roofingorbu
ste;inspectio
rkingproperly
eiracceptance
emstothete
entonthere
eProductOw
nextfeature.
anyconcernf
eveloperswhi
edefectsord
ureinsteado
Acceptance
al.Howwere
canusebugst
didate1
ate(seeSideb
rallelization).
illustratedin
testing.Thea
developeror
dependingon
AgileDevelo
developerora
tedbyateste
nunit,compo
thembefore
ugrepellent.T
ontoprefent
y,theyaskth
etestsfortha
eamwhothen
visedcodebe
nerissatisfie
Thispractice
foundbythe
letheystillre
deficienciesca
fbeingstock
eProcess
eacttothebu
tomotivateim
barRecasting
.Theremaya
Figure4ofth
activitiescond
rasmallteam
nthesizeoft
pmentwithI
asmallteam.
erorbusiness
onentandbu
eproceeding.
Theyarecons
defectsisess
eProductOw
atonefeature
nfixesthepro
eforegivingit
ed,theyaccep
e,calledIncre
ProductOwn
ememberthe
anbeaddress
piledfora"b
ugstellsusal
mprovement
aSequential
alsobearou
heIntroductio
ductedforea
mofdevelope
thefeature.(
ncremental
.Whenthe
sanalystcond
usinessaccept
Thesetests
sistentwith
sential
wner(ora
e.IftheProd
oblems
ttotheProdu
ptthefeature
emental
nerduring
edetailsofho
sedimmediat
ugfixingpha
lotaboutour
stoour
48
ndof
onby
ach

ers,
(See
ducts
tance
are
uct
uct
eand
ow
tely
se."

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 49
developmentprocessorwecanjustfixthebugswithoutunderstandingtherootcause.Thelatter
approachoftenresultsinalargenumberofbugsbeingfoundandhavingtobefixed.Wecaneitherfix
thesebugsrightawayorwecanstockpilethemforabugfixingphase.Suchastockpileofbugsis
consideredaformofwasteinleanmethodologiesbecausewearebuildingupaninventoryofpartially
finishedsoftware.AccordingtoTomandMaryPoppendieck,thoughtleadersoftheLeanSoftware
Developmentapproach,itisirresponsibletoorganizetheworkinsuchawaythatTriageisnecessary.
Theyarguethatitisprimarilythedecisiontotestlateratherthantomistakeproofeverystepthat
escalatesthecostofrepairtothepointthatonewoulddeliberatelychoosetoleaveknowndefectsin
theproduct.Thus,theneedforbugtriagecanbeconsideredasaprocesssmellThereare,however,
circumstanceswhenyourdevelopmentisorganizedinawaywhenabugbacklogand,therefore,triage
activitiesarenecessary.Thistendstohappenwithlargeand/ordistributedteamsorwhenproduct
ownerisnotreadilyavailable.Inthiscase,itisimperativetokeepinmindthefollowingpoints:
1) Triageshouldonlylookatnewbugsreexamingoldbugsrepeatedlyiswaste.Thepurposeof
triageshouldbetodecidewhethera)weneedtofixthebugforthesoftwaretobeconsidered
done,theissueisnotreallyabugandthereforerequiresnoaction,orb)thebugrelatesto
futurefunctionalityandcanbeaddressedwhenthatfunctionalityisbuilt.
2) Toavoidwaste,weneedtokeepthedepthofthebugbacklogreasonablysmall.Reasonably
smallisafunctionofourdevelopmentandbugfixingcapacity.Ifonalargeprojectour
developmentteamcanfix200bugsperiterationonaverage,thenthebacklogof200bugsisnot
alot.However,ifourdevelopmentcapacityissuchthatwecanonlyfix10bugsperiteration,
clearlyabacklogofof100shouldbeconsideredasaredflag.
3) Thekeygoalistomakethebugfixlatency(measuredasaMeanTimeToFix)toaminimumby
performingregulartriagesandschedulingbugfixestopreventthebugbacklogfromexploding.
Averagetimetofixbugsshouldbenomorethanafewiterations.Longlivedbugsisasmell
indicatinginventory(aformofwaste.)Thiscanbeavoidedbyensuringwegettodonedone
oneachfeaturebeforemovingontonextfeature.
4) Batchingrelatedbugsforfixingorincludingtheminaplannedfutureuserstory/featurecanbe
consideredasapossiblestrategy(forexample,weknowthatthebugisrelatedtoafeaturethat
wewillbeworkingoninthenextiterationandwhichwillrequireaseriousupdateor
deprecationofsomerelatedfunctionality;inthatcaseitmaymakesensetowaittillthatworkis
completed),weneedtomakesurewearecognizantoftheimpactofsuchbatchingonour
latency.
Foradetaileddiscussionofbugbackloganalysis,bugtriageandtheuseofbugmanagementsystems,
refertoChapter19ManagingtheAcceptanceProcess.
1.3.9 UsingAcceptanceTeststoDriveDevelopment
TheacceptanceprocessworksbestwhentheProductOwnersuppliestheacceptanceteststothe
developmentteamearlyenoughtohelpthemunderstandwhatdonelookslike.Ideally,thisisbefore
developmentevenstartsbutitmustdefinitelybebeforedevelopmentisfinishedattheverylatest.The
bestresultsareachievedwhentheseacceptancetestsarenothandedoff,butratherdeveloped

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 50
together,forthiswaytheacceptancetestsarenotonlyusedforfindingbugsintheproduct,butrather
clarifyrequirementsandimproveconversationsbetweentheProductOwnerandtheproduct
developmentteam[MelnikEtAlOnExecutableAcceptanceTests].Thefocusisoncollaborationnot
handoff.ThispracticeisknownasAcceptanceTestDrivenDevelopment(ATDD)orStorytestDriven
Development(STDD).Whenusedinconjustionwithincrementaldevelopment,theteamwilltypically
automatemuchofthetestingtokeepthecostofrepeatedlydoingreadinessassessmentlow.The
sidebarWhatittakestodoIncrementalAcceptancedescribesothersuccessfactorsfordoinghighly
incrementalacceptancetesting.
Summary
Thischapterintroducestheconceptoftheacceptanceprocessasawaytothinkabouttheactivities
relatedtomakingthedecisiontoacceptsoftware.Theacceptancedecisionistheculminationofa
processthatinvolvesseveralorganizationseachwithspecificresponsibilities.Eachreleasecandidateis
passedthroughthedecisionpointsoftheacceptanceprocessonitswaytomakingthefinalacceptance
decision.TherearethreemajordecisionpointsbetweensoftwareconstructionbytheProduct
DevelopmentTeamworkingwiththeirProductOwnerandsoftwareusagebytheenduser.First,the
ProductDevelopmentTeammustdecidewhetherthesoftwareintensiveproductinitscurrentform,
knownasthereleasecandidate,isreadyforacceptancetesting.Ifitis,theProductOwnermustdecide
whetherthereleasecandidatemeetstheiracceptancecriteriabeforetheproductcanbemadeavailable
totheendusers.Ultimately,eachendusermustdecideforthemselveswhethertheywanttousethe
software.Thisusagedecisionhasverylittleinfluenceonthecurrentacceptancedecisionalthoughit
mayinfluencetheacceptancecriteriaforfuturereleasesoftheproduct.
Whileeachuserpotentiallymakestheirownusagedecision,thereadinessandacceptancedecisionsare
normallyeachasingledecision.Eachdecisionpointor"gate"shouldhavewelldefinedcriteriatoguide
thedecisionmaking.ThesecriteriashouldbeknowninadvancebyboththeProductDevelopment
TeamandtheProductOwner.
Theacceptancedescribedinthischapterappliestobothsequentialandagileprojectsbutinslightly
differentways.Thephasednatureofsequentialprojectsmeansthatalltestingisdonewithinaseparate
testingphaseandtheacceptanceprocessdescribeswhatgoesonwithinthatphase.Agileprojects
traversetheentiredevelopmentlifecycleforeachfeatureoruserstory.Therefore,theacceptance
processisexecutedatseverallevelsofgranularitywiththefinestgrainexecutionbeingattheindividual
featurelevelandthelargestbeingatthewholeproduct(orfeatureintegration)level.
WhatsNext
InChapter2ElaboratingonTHEAcceptanceProcesswegointomoredetailabouthowtheacceptance
decisionisrelatedtotheusagedecisionandhowtheprocessisimplementedincomplexorganizations
andwhenbuildingcomplexproducts.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 51
References
[CooperOnDoingItRight]RobertG.Cooper,DoingitRight:WinningwithNewProducts,Ivey
BusinessJournal,July/August2000.AlsoavailableasaProductDevelopmentInstituteReference
Paper#10asathttp://www.stagegate.com.
[CooperOnStageGateProcess]Robert.G.Cooper,TheStageGateIdeatoLaunchProcessUpdate,
WhatsNewandNexGenSystems,J.ProductInnovationManagement,Volume25,Number3,
May2008.AmodifiedversionalsoavailableasasaProductDevelopmentInstituteReference
Paper#30Perspective:TheStageGateIdeatoLaunchProcessUpdate,WhatsNewand
NexGenSystemsathttp://www.stagegate.com.
[FewsterGrahamOnTestAutomation]MarkFewsterandDorothyGraham,SoftwareTest
Automation,1999,ACMPress
[ShingoDillonOnToyotaProductionSystem]Shingo,ShigeoandDillon,Andrew,AStudyofthe
ToyotaProductionSystem:FromanIndustrialEngineeringViewpoint:ProductivityPress,1989.
[HighsmithOnAgileProjectManagement]Highsmith,JimAgileProjectManagement:Creating
InnovativeProductsAWP2004ISBN13:9780321219770
[BerczukAppletonOnSCM]Berczuk,SteveandBradAppleton,SoftwareConfigurationManagement
Patterns:EffectiveTeamwork,PracticalIntegration,AddisonWesley(2003)ISBN:0201741171
[MSOnSCM]Patterns&PracticesTeamDevelopmentwithMicroosftVisualStudioTeam
FoundationServer,Ch.3,4,6:MicrosoftPress,2008.
[OMGOnSwimlanes]ActivityPartitionNotations.InUMLSuperstructureSpecification,pp.342345
http://www.omg.org/spec/UML/2.2/Superstructure/PDF/,Oct18,2009
[PoppendiecksOnLean]Poppendieck,Mary&TomLeanSoftwareDevelopmentAddisonWesley
(2003)ISBN:0321150783
[DenneHuangOnIFM]MarkDenneandJaneClelandHuang,"TheIncrementalFundingMethod:
DataDrivenSoftwareDevelopment,"IEEESoftware,vol.21,no.3,pp.3947,May/June2004
[LadasOnScrumban]CoreyLadas,Scrumban:EssaysonOSystemsforLeanDevelopment,Modus
CooperandiPress,2008
[HiranabeOnKanban]KenjiHiranabe,KanbanAppliedtoSoftwareDevelopment:fromAgileto
Lean,publishedonlineat<http://www.infoq.com/articles/hiranabeleanagilekanban>,
January14,2008
[MSOnTFSBranchingGuide]TFSBranchingGuide2.0,http://tfsbranchingguideii.codeplex.com/,Aug
15,2009
[PrimaveraOnScrum]PrimaveraScrumCasestudy.
http://controlchaos.com/download/Primavera%20White%20Paper.pdf,Oct19,2009

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 52
[KnibergOnScrumFromTrenches]HenrikKniberg,ScrumandXPfromtheTrenches,
http://www.infoq.com/minibooks/scrumxpfromthetrenches,Oct19,2009
[DenneHuangOnSoftwareByNumbers]MarkDenneandJaneClelandHuang,SoftwarebyNumbers:
LowRisk,HighReturnDevelopment,PrenticeHall,2003
[MelnikEtAlOnExecutableAcceptanceTests]GrigoriMelnik,FrankMaurer,MichaelChiasson,
"ExecutableAcceptanceTestsforCommunicatingBusinessRequirements:Customer
Perspective",Proc.Agile2006,IEEEComputerSocietyPress:3546,2006

Acceptanc
Cha
Ac
Chapter1
lifecycle.T
acceptanc
2.1
Thusfarw
usagedec
becauseit
2.1.1 S
Whenaso
ProductO
thenmak
Figure9
Acceptanc
Almostal
mayhave
factbequ
isusedto
theusage
ceTestEngin
pter2
ccepta
1introducedt
Thedescriptio
ceprocessto
TheRole
wehavefocus
cision?Itturn
thappensaft
eparating
oftwareprod
Ownermayde
etheusaged
ceandUsage
lproductsha
einfluencedt
uitedifferent
carryoutas
edecisionism
eeringGuide
2 E
ance
theacceptanc
onwaswasd
dealwithmo
oftheUs
sedalmosten
nsoutthatthe
tertheprodu
gtheAccep
uctisbeings
ecidetoacce
decisionindiv
eDecisionsan
vemanyuser
heacceptanc
fromit.Inso
pecificjobsu
madeonbeha
eVolumeI,
Elabor
Proce
ceprocessan
deliberatelysi
orecomplexs
sageDecis
ntirelyonthe
eusagedecis
ctisalreadya
ptanceDec
selectedorbu
ptthesoftwa
viduallyasillu
ndCriteria
rsandeachis
cecriteriause
omecases,the
uchasacallce
alfoftheuser
ReleaseCand
rating
ess
dhowitfitsi
mple.Thisch
situationsincl
sion
ereadinessan
siondoesntd
accepted.
cisionfrom
uiltspecificall
areassuitable
stratedinFig
slikelytohav
edduringthe
euseofthep
enteragento
rsbytheirma
didate1
gonth
intotheovera
apterintrodu
ludingcomple
ndacceptance
directlyimpac
mtheUsag
yforaProdu
ebeforeoffe
gure9.
vetheirownu
readinessan
productisnot
orairlinerese
anagement;t
he
allproductde
ucessomewa
exorganizatio
edecisions.W
cttheaccepta
geDecisio
uctOwnerby
ringittothe
usagecriteria
dacceptance
toptional,su
ervationssyst
hisdecisionn
evelopment
aystovarythe
onsandprodu
Whataboutth
ancedecision
on
anotherpart
userswhowi
whichwhile
edecisions,m
chwhensoft
em.Inthese
neednotfollo
53
e
ucts.
he
n
y,the
ill

it
mayin
ware
cases
ow

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 54
immediatelyaftertheacceptancedecisionastheremaybeotherprerequisitesforusagesuchasuser
training.
Whileausercould,intheory,choosenottousethesoftware,theconsequenceoflosingtheirjobwould
makethisdecisionhighlyunlikely.Theusersfeedback,however,mightinfluencesubsequentreleasesof
thesoftwareaswellseeinFeedbackfromtheUsageDecisions.
2.1.2 DeterminingtheAcceptanceCriteria
Theacceptancedecisionismadebasedonsomecriteriaincludingbothfunctionalityandqualityrelated
elements.Theminimumleveloffunctionality(representedbyMMF)andtheminimumlevelofquality
(MQR)shouldbedeterminedwellinadvanceoftheacceptancedecisiontogivetheProduct
DevelopmentTeamofthesoftwareareasonablechanceatbeingabletosatisfythem.Therefore,the
decisionsabouttheMMFandMQRcomebeforetheAcceptanceDecisionasshowninFigure9
AcceptanceandUsageDecisionsandCriteria.
2.1.3 FeedbackfromtheUsageDecisions
Notethattheusagedecisionsmadebyindividualusersaftertheacceptancedecisiondonotdirectly
affecttheacceptancedecisionbutitmay,throughsalesdata,postsalessupportdataorfocusgroups
affectthedefinitionofMMForMQRforsubsequentreleasesoftheproduct.ThisisillustratedinFigure
10UsageDecisionFeedback.

Figure10
UsageDecisionFeedback

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 55
WhiletheMQRtendstobefairlyconstantfromonereleasetothenext,theProductOwnermayrevise
thequalitycriteriaforasubsequentreleasebasedonthefeedbackreceivedfromusers.Forexample,
usertestingofanalphareleasemightrevealthattheresponsetimesspecifiedintheMQRarenotfast
enoughinpracticeandmightcausetheproductownertomakethemmorestringentforthebetaor
finalrelease.
TheMMFforeachreleasetendstobeuniquebecauseitisthelistoffunctionalitythatneedstobe
includedtowarranthavingtherelease.FeedbackfromuserscaninfluencetheMMFbasedonrequests
formissingfunctionality,suggestionsforhowexistingfunctionalityshouldbechanged,andthroughbug
reports.
FeedbackfromusersofAlphaandBetareleasesoftenresultsinchangestotheMMForMQRofthefinal
release.
2.2 AcceptanceinComplexOrganizations
Thesimpleacceptanceprocessdescribedpreviouslywillsufficeinmanyorganizations.Butthereare
someorganizationswhosestructureisverycomplexasaretheproductstheydevelop.These
organizationsrequireadecisionmakingprocessthatmirrorsthecomplexityoftheorganizationor
product.Thedecisionsmaybemadeinseriesaseachdecisionmakerexaminestheproductbutthis
leadstolongerelapsedtimesbeforethevalueoftheproductcanberealized.Thecommonsolutionisto
haveeitherthereadinessoracceptancetestingactivitiesanddecisionbemadeinparallelbyseveral
organizationseachwithvetopowerovertheacceptabilityoftheproduct.
Notetoreaders:Ifthesescenariosdontapplytoyourorganization,feelfreetoskipaheadto2.3
AcceptingComplexProducts.2.3
2.2.1 ParallelAcceptanceDecisions
Manyorganizationshaveotherdepartmentswhoareresponsibleforoperatingandsupportingthe
systemonceitisdeployed.Thesedepartmentsneedtohaveasayintheacceptancedecision.Thisis
reflectedinFigure11ParallelAcceptanceDecisions;itshowsseveralacceptancedecisionsbeingmade
inparallel,eachbasedonadifferentsetofacceptancecriteria.

Acceptanc
Figure11
ParallelA
Whenthe
specialty,
owndecis
Examples
B
O
C
C
U
Thisappro
MMFand
leastpart
acceptanc
beforthe
Figure12
Owneran
ceTestEngin
AcceptanceDe
erearesevera
eachmayins
sionregarding
ofsuchstake
BusinessUnit
OperationsDe
CorporateBra
CorporateSec
UsabilityGrou
oachintroduc
dMQRusedb
timeintheP
cecriteriaand
eentiredurat
AProductD
ndOperations
eeringGuide
ecisions
alorganizatio
sistinconduc
gtherelease
eholders,and
Functionali
epartmentT
andingProd
curityProdu
upProductc
cesriskifthe
bytheProduc
ProductDeve
ddesignsthe
tionofthepro
Development
sillustrates
eVolumeI,
onsinvolvedin
ctingitsowna
candidatean
dtheirareao
ityisacceptab
Theproductc
uctcomplies
ucthascompl
complieswith
MMFandM
tDevelopme
lopmentTea
ebestpossible
ojectorforo
tLifecyclewit
sthescenario
ReleaseCand
ntheaccepta
activitiesinde
ndanyoneof
ofinterest,co
ble.
canbemanag
withbrandin
liedwithsecu
husabilityreq
QRusedbye
ntTeam.Idea
mtoensuret
esolutionto
neortwoite
thParallelEla
owheretheP
didate1
ancedecision
ependentlyo
fthemcaneff
ouldinclude:
gedandsupp
ngrequireme
urityrequirem
quirements.
eachstakehol
ally,eachsta
thattheteam
thebusiness
rationsfocus
aborationand
ProductOwne
eachwithth
feachother.
fectivelyveto
ortedinprod
nts.
ments.
derarenotin
keholderwou
mtrulyunders
need.Thispa
edonthatst
dAcceptance
eristheparty
heirownarea
Eachmakest
otheothers.
duction.
ncludedinthe
uldparticipat
standsthe
articipationco
akeholdersn
byProduct
ywhohas
56

of
their
e
teat
ould
eeds.

Acceptanc
conceived
withsupp
Figure12
AProduct
Operation
2.2.2 P
Insomec
makeinde
theaccep
theyident
Parallel

ceTestEngin
dthesoftwar
portingthesy
tDevelopmen
ns
ParallelRe
asesseveralo
ependentrea
ptancedecisio
tifyhasbeen
ReadinessDe
eeringGuide
easasolutio
ystemonceit
ntLifecyclewi
eadinessD
organizations
adinessdecisi
onifeitheror
completeda
ecisions.
eVolumeI,
ontoabusine
isputintopr
ithParallelEl
Decisions
sdotheirown
ons.Theprod
rganizationde
ndtheydeem
ReleaseCand
essproblema
roduction.
laborationan
nreadinessa
ductwillnot
eemsthepro
medthesoftw
didate1
ndtheopera
ndAcceptance
ssessmentof
behandedov
ductnotrea
wareready.
ationsdepartm
ebyProductO
fthereleasec
vertothepro
adyuntilany
.Thisisillustr
mentistaske
Ownerand
candidateand
oductownerf
yadditionalw
ratedinFigur
57
d

d
for
work
re13

Acceptanc
Figure13
ParallelR
Examples
P
S
D
L
Again,eac
time,toe
Whether
usuallyba
organizat
departme
othersitm
chooseto
thesepart
conducted
2.3
Organizat
processth
ceTestEngin
ReadinessDec
ofpotential
ProductDevel
SecurityTeam
DataArchitect
egalDepartm
chofthesest
ensurethatth
adecisionisd
asedonwhen
ionalbounda
entmaybeco
mayhaveap
oparticipatew
tiesmuchea
dincrementa
Accepting
tionsthatbui
hatinvolvesm
eeringGuide
cisions
stakeholders
lopmentTeam
mDesignhas
tureTeamD
mentAllem
takeholderss
heMMF/MQR
deemedtobe
nthedecision
rythestakeh
onsideredab
resenceinor
whentherea
rlierinthepr
allyasthepro
gComple
ldlarge,com
multistageac
eVolumeI,
withparallel
mCodeispa
spassedsecu
Dataarchitec
beddedtechn
houldbecon
Rareaddress
epartofread
nmakerwant
holdersits.Fo
usinessfunct
rneartheITd
dinessdecisio
rojectinamo
oductisdevel
exProduc
plexproducts
cceptance.La
ReleaseCand
readinessde
assingallunit
urityreview
turehasbeen
nologieshave
nsideredmem
edbywhatth
dinessoracce
tstogetinvol
orexample,in
tionwhomus
departmento
onismade.O
orecollaborat
oped.
cts
stypicallyuse
argecomplex
didate1
ecisionsinclud
ttestsandfun
napproved
ebeensuitab
mbersofthew
heProductD
eptanceissom
vedandwhic
nsomeorgan
stacceptthe
orproductdev
Ofcourse,itis
tivefashionso
eamorecom
productstyp
de:
nctionaltests
lylicensed.
wholeteam,e
evelopmentT
mewhatarbit
chsideofala
izationsthes
releasecandi
velopmentgr
susuallyprefe
othatreadin
mplexversion
picallyrequire
s
evenifonlyp
Teambuilds.
trary.Thecho
arger
security
idatewhilein
roupandther
erabletoinvo
essassessme
oftheaccept
elargenumbe
58
art
oiceis
n
reby
olve
entis
tance
ersof

Acceptanc
people,to
compone
builtand
theusers.
Microsoft
theaccep
indication
2.3.1 A
Acommo
compone
theirresp
eachcom
theintegr

Figure14
Compone
Eachcom
largerpro
overtoth
variousna
testingo
hopefully,
madeabo
ceTestEngin
oomanytow
nts,theteam
ownedbyac
.Thelatterap
t[MillerCarte
ptanceproces
noftheappro
Acceptingt
nwayofsub
ntsthatareb
ptivetechnolo
ponentshou
ratedproduct
entTeamswit
ponentmust
oductforsubs
heProductOw
amesincludin
orintegration
,thesearedi
outtheentire
eeringGuide
workinasingl
msthatbuildt
componentte
pproachisca
rOnLargeScal
ss.Thekindso
opriatenesso
theOutpu
dividingwork
builtandverif
ogiesandtech
ldhaveitsow
t.Thisisillust
thSerialRead
tgothrougha
sequentprod
wnerforthea
ngcompone
ntestingfor
rectlytracabl
eproduct.
eVolumeI,
eteam.Whil
themmaybe
eamorteams
lledfeature
leAgile].The
ofproblemsf
fthewaywe
tofCompo
kinlargeorga
fiedinparalle
hnicaloruser
wnreadiness
tratedinFigu
dinessDecisio
acomponent
ductlevelread
acceptanced
nttestingor
rtheentirepr
lefromtheM
ReleaseCand
ethelargeco
organizedin
smaybeorga
crewsandis
mannerinw
foundduring
haveorganiz
onentTea
anizationsist
elbyseparate
rdomains.W
assessmentp
re14Comp
ons
readinessde
dinessdecisio
ecisions.The
rsystemtest
roduct.Eachc
MMF/MQRfo
didate1
omplexprodu
severalways
anizedaround
swidelyused
whichthesepe
theacceptan
zedourteam
ams
todecompose
ecomponent
henthisorga
priortocondu
onentTeams
ecisionbefore
on.Theinteg
readinessas
tingforthec
componenth
rthereadine
uctsaretypic
s.Eachcompo
dthefunction
dbytheDeve
eopleareorg
nceprocessca
s.
etheproduct
teamsthate
anizationalso
uctingreadin
swithSerialD
ebeingintegr
ratedproduc
ssessmentact
components
hasitsownM
ssandaccept
allycompose
onentmaybe
nalityprovide
loperDivision
ganizedimpac
anbeagood
tinto
eachspecialize
lutionisused
essassessme
Decisions.

ratedintothe
tisthenturn
tivitiesmaygo
andsystem
MMFandMQR
tancedecisio
59
edof
e
edto
nof
cts
ein
d,
enton
e
ned
oby
R;
ns

Acceptanc
Thisrevea
compone
intermedi
compone
ProductO
accountab
whatisac
compone
Thisdelay
whichisr
misunder
integratio
incremen
compone
productq
ghettoby

2.3.2 A
Anotherw
[BasLarma
teamisre
criteria.B
expertise
featurere
testingar
illustrated
ceTestEngin
alsseveralof
ntscannotbe
iary,usuallya
ntrequireme
Ownertothe
bletotherea
ctuallyrequire
ntshavepass
ystheproduc
atherlateina
stood.Thisp
on(CI)practic
talacceptanc
ntteams(com
qualitysoftwa
yisolatingthe
Acceptingt
wayoforgani
anOnScalingA
esponsiblefor
Buildingeach
oneachcom
eadinessasse
ebothfocuse
dinFigure15
eeringGuide
theweaknes
edirectlyund
anarchitecth
ents.Thiseffe
intermediary
alProductOw
ed.Second,t
sedtheirresp
ctreadinessa
aprojecttod
roblemcanb
cescombined
cetestingbut
mparedtofea
areforindivid
etechnicalco
theOutpu
zingthedeve
Agile,Eckstein
rbuildingand
featuremay
mponentande
ssment.Ther
edonfeature
FeatureTe
eVolumeI,
ssesofthisap
derstoodand
hastoberesp
ectivelytransf
ytherebyredu
wner.Itisalso
heproductre
pectivereadin
ssessmentun
discoverthat
besomewhat
withincreme
tinpracticeit
atureteams)
dualfeatures
mponentdev
tofFeatur
elopmentofl
nOnLargeScal
dverifyingaf
involvemod
ensuresthatt
refore,thepr
eintegrationr
eamswithSer
ReleaseCand
pproachtoco
thereforeacc
ponsiblefortr
fersproducto
ucingthedeg
opossiblefor
eadinesscann
nessdecisions
ntilafterthe
someofthec
overcometh
entalintegrat
tismuchhard
andfeaturet
earlier.Itals
velopersfrom
reTeams
argecomplex
leAgileandM
featurebased
ifyingseveral
thecompone
roductreadin
ratherthano
rialAcceptanc
didate1
mplexprodu
ceptedbythe
ranslatingPro
ownershipof
greetowhich
thecompone
noteasilybe
sandareinte
bigbangin
componentre
roughtheus
tion[BasLarm
dertodoincr
teamswilltyp
soperpetuate
manyunderst
xproductsist
MillerCarterOn
dontheProd
lcomponents
entsareprope
essassessme
oncomponent
ceDecisions.
cts.First,bec
eProductOw
oductOwner
f thecompone
thecompon
entrequirem
assessedunt
egratedtofor
tegrationoft
equirements
eofcontinuo
manOnScaling
rementaldeli
picallybeabl
esthetechnic
tandingofthe
tousefeatur
nLargeScaleA
uctOwnersa
sbuttheteam
erlyintegrate
entandthepr
tintegration.
causethe
wner,some
requirement
entfromthe
entteamis
entstodriftf
ilallofthe
rmtheprodu
thecompone
were
ouscode
gAgile]and
veryusing
etodeliver
calcomponen
ebusiness.
eteams/crew
Agile].Eachfe
acceptance
mincludes
edaspartoft
roductaccept
.Thisscenario

60
sinto
from
ct.
nts,
nt
ws
ature
their
tance
ois

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 61
Figure15
FeatureTeamswithSerialAcceptanceDecisions
EachfeaturesreadinessdecisionismadebasedontheMMFandMQRspecifictothatfeatureas
determinedbytheProductOwner(ideallywithinputfromtheProductDevelopmentTeam);no
intermediaryisrequiredtotranslatetheProductOwnersrequirementsintocomponentrequirements.
Thefeaturereadinessdecisionisaprerequisiteforthecorrespondingfeaturesacceptancedecision
whichshoulduseroughlythesameacceptancecriteria.Meanwhile,theproductisintegratedanda
productlevelreadinessdecisionismadebasedontheproductsMMFandMQR.Theproductlevel
MMFshouldsimplybeanaggregateofthefeaturelevelMMFs

Thereforetheinteractionsbetween
featuresneedtobeincludedaspartoftheMMFforeachaffectedfeatureorasseparatefeature
interactionfeatures.Whentheproductlevelreadinessandallfeaturelevelacceptancedecisionsare
positive,theproductlevelacceptancedecisioncanbemade.
Thisstyleoforganizationandacceptanceisusuallypreferabletocomponentteamsbecauseitensures
eachteamisworkingwitharealendProductOwnerandisresponsiblefordeliveringawholeproduct,
notjustaninternalcomponent.SeethesidebarFeatureCrewsonMicrosoftOfficeforanexampleof
howthisworksinpracticeinChapter9ProductManagersPerspective.
IftheProductOwnerisahighlyconstrainedresource,havingalargenumberoffeatureteams
contendingfortheirparticipationmayreducetheeffectivenessofthisapproach.Iftheinteractions
betweenfeaturesarealargerriskfactorthantheriskoftechnicalintegrationwiththetechnical
components,itmaybemoreappropriatetoreducethenumberoffeatureteamsbydelegatingmoreof
theworktocomponentteamssothatfeatureinteractionscanbeaddressedwithsinglefeatureteams
ratherthanbetweenthem.

2.3.3 AcceptingCustomizableProducts
Manyproductsarebuiltbyoneorganizationforsaletoanotherorganization,theircustomer.Whenthe
productneedstobecustomizedforeachcustomertherearetwodistinctacceptancephasesforthetwo
differentproductsinquestion.Thefirstproductisthegenericproductbuiltbythesellingorganization
totherequirementsspecifiedbytheproductmanager.Thesecondproductistheonebuilttothe
specificationofthepurchasingcustomer.Theconstructionofthelatterproductmayincludeeither
configurationorcustomcodinginadditiontothegenericcustomizableproduct.
Thefirstacceptancedecisioniswhentheproductmanageracceptsthecustomizableproductassaleable
tothecustomers.Thisacceptanceprocesscouldbeanyofthevariantsdescribedpreviouslyincluding
AcceptingComplexProducts.
Thesecondacceptancedecisioniswhentheacquirerofthecustomizableproductacceptstheproduct
ascustomizedtotheirownsituation.Thereisasingleacceptancedecisionforeachversionofthe
customizablegenericproductandoneacceptancedecisionperversionofthecustomizedproductas
showninFigure16Acceptingcustomizableproducts.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 62

Figure16
Acceptingcustomizableproducts

Sidebar:AcceptingConfigurableERPProducts.
Manysoftwareproductsaresoldasgenericimplementationsofabusinessprocessthatcanbe
configuredtosuitabusinessspecificneeds.TwogoodexamplesofthisareERPsystemssuchasSAPs
ERPproductandMicrosoftsDynamics.Bothprovidegenericimplementationsofcorebusiness
processessuchastimereportingandpayroll,supplyorderingandaccountspayable,productorder
processingandaccountsreceivable.Theoutoftheboximplementationsoftheseprocessescanbe
modifiedthoughconfigurationdataorextendedbypluggingincustomwrittenprocedures.Fora
particularcustomersinstallationoftheERPsystem,itgoesthroughtwocompletelydistinct
acceptanceprocesses.First,theproductownerattheconfigurableproductvendordefinesthegoals
ofthenextreleasebysynthesizingnewfeatureideasfromtheproductdefinitionteam,bugreports
frompreviousreleasesandnewfeaturerequestsfromnewandexistingcustomers.Thisgetsturned
intothefunctional(MMF)andnonfunctional(MQR)requirementsfortherelease.TheERPproductis
built,goesthroughreadinessassessmentbytheproductdevelopmentteam,acceptancetestingbythe
productmanagementteamandprofessionaltestersbeforebeingdeclaredreadytoshipto
customers.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 63
Meanwhile,thecustomersaredefiningtheirowngoalsandobjectivesfortheinstallationbasedonthe
expectedfeaturesetofthenewreleaseandtheirspecificbusinesscontext.Eachcustomershould
haveitsownproductownerforthecustomizedproduct.Theymaydobusinessprocessmodellingand
gapanalysisbetweentheirbusinessprocessandtheoutoftheboxproducttodefinetheirown
MMFandMQR.Whentheproductships,theirownproductdvelopmentteaminstallsitintotheirown
developmentsystemanddoesthecustomizationand/orconfiguration.Ifthisistheirfirstinstallation
oftheproductthentheyadjustthevariousconfigurationparametersandimplementnewplugin
procedures.IfthisisanupgradefromapreviousversionoftheERPsystemthenmuchofthework
consistsofreapplyingtheconfigurationfromtheirprevioussystemanddoingregressiontestingofthe
functionality.Inbothcases,thereleasecandidate(s)shouldtraversethefullacceptanceprocesswith
thecustomersproductdevelopmentteammakingtheReadinessDecisionandthecustomersproduct
ownermakingtheacceptancedecisionwhichisbasedontheMMFandMQRdefinedbythe
customersproductownerfortheirspecificinstallationoftheERPsystem,nottheMMFandMQR
definedbythevendorsproductownerforthegenericproduct.Inaninterestingparadox,ERPsystems
wouldseemtobeimminentlysuitabletoagilecustomizationandincrementalacceptancetesting
becausetheyworkrightoutoftheboxandcustomizationevolvestheproductfromagenericonetoa
customizedone.ButmanyERPsystemslackthetoolstoautomatethebuildandregressiontesting.
SeethesidebarWhatittakestodoIncrementalAcceptanceandthepapers[MeszarosOnAgileERP]
and[AstonOnAgileERP]formoreinformationonwhatneedstobeavailableandmightbemissingin
yourERPsystem.

2.4 Exitvs.EntryCriteria
Manyofthedecisionsintheacceptanceprocessinvolvehandoffsbetweenparties.Eachhandoff
involvestwosetsofcriteria.Thepartymakingthehandoffmayhavesomeexitcriteriathatmustbe
satisfiedbeforetheyarepreparedtomakethehandoff.Thepartyreceivingthehandoffmayhavesome
criteriatobesatisfiedbeforetheyarepreparedtoreceivethehandoff.Thesecriteriacanbeassessedat
separateexitandentrydecisionsasillustratedinFigure17SeparateExitvs.EntryDecisions&Criteria.

Acceptanc
Figure17
Separate
Thiseffec
phasesas
Figure18
Acceptanc
Whenthe
shouldbe
decisionp
withCom
ceTestEngin
ExitandEntry
ctivelyresults
sshowninFig
ceProcesswi
eProductDev
eabletoagre
pointbasedo
binedExitan
eeringGuide
yDecisionsan
inasituation
gure18:
ithSeparateE
velopmentTe
euponasing
ontheagreed
ndEntryCrite
eVolumeI,
ndCriteria
nwherethere
Exit/EntryCrit
eamandprod
glesetofcrite
uponsetofc
ria.
ReleaseCand
eisnonoma
teriaforAcce
ductownerpa
eriathatsatis
criteriaasillu
didate1
anslandbe
eptance
artieshavea
sfiesboththe
ustratedinFig
etweentheBu
goodworking
irneedsand
gure19Sing

uildandAcce

grelationship
haveasingle
gleDecisionP
64
pt
pthey
e
Point

Acceptanc
Figure19
SingleDec
Thetwop
decisionb
andthere
Themain
thesoftw
sufficientl
shouldtak
Forexamp
combinat
E
E
ceTestEngin
cisionwithCo
partieswould
basedonthe
eceivingparty
entrycriterio
arereadyfor
lypreparedto
keallthecrit
ple,aProduc
ionof:
Entrycriteriaf
Allfea
Stress
per10
Regres
Exitcriteriafo
90%co
Dataa
Static
Securit
eeringGuide
ombinedExit
agreeonwh
assessment.
ymaybeinte
onfortheacc
racceptancet
oconductthe
eriaintoacco
tDevelopme
foracceptanc
turesinMMF
testedat110
000.
ssiontestofp
rconstructio
odecoverage
architecturere
analysishasi
tyreviewcom
eVolumeI,
andEntryCri
hichpartydoe
Typicallyitis
erestedinsee
ceptancepha
testing.Secon
eacceptance
ount.
ntTeamsRe
cetestingpro
Ffunctionalw
0%ofratedca
previousrelea
nprovidedby
ebyunittests
eviewedand
dentifiedno
mpletedwith
ReleaseCand
iteria
estheassessm
thepartydo
eingtheresult
seisthatthe
ndarycriteria
testing.The
eadinessDecis
ovidedbythe
withnoknow
apacityfor48
asefeaturese
ytheProduct
s.
approved.
deficiencies.
nohighprior
didate1

mentforeach
ingthehando
ts.
ProductDev
amayinclude
decisiontoe
sionshouldb
ProductOwn
nseverity1&
8hourswithl
executedwith
tDevelopmen
rityissuesope
hcriterionan
offthatwilld
elopmentTe
ewhetherthe
ntertheacce
bemadebase
nerincluding:
&2bugsopen
lessthan1fa
h100%pass
ntTeamsuch
en.
dwhomakes
dotheassessm
amhasdeem
eProductOw
eptancephase
dona
:
n.
iledtransacti
rate.
has:
65
sthe
ment
med
neris
e
ion

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 66
InpracticeitishighlypreferableforthereadinesscriteriatoincludealltheProductOwnersacceptance
criteria.Thisavoidsfindingbugsduringtheacceptancetestingphase.

Sidebar:RepresentingtheAcceptanceProcesswithStagesandGates
Manycompaniesemployaprocessinvolvingstagesandgatestomanagetheirproductdevelopment
pipeline.Theseprocessesweredescribedinsection1.X(ProcesseswithStagesandGates).Thegates
serveasdecisionpointsbetweenstages.Themaingoalistomanagetheallocationofresources
(fundingbeingthemostcommon)amongandwithinprojects.Thegatesattheexitofstagestypically
involvego/nogodecisionswiththegodecisionsleadingtoprogressivelylargeredresource
commitmentsandinthesubsequentstages.
AtypicalsequentialStageGate
TM
processstartswithanewideaevaluatedataninitialgatetodecide
whethertheideaisworthpursuing.AsampleprocessisillustratedinFigureS0.
Attheideastageusuallyafixedamountofresourcesareallocatedtotheproducttodeveloptheidea
furtherandbuildabusinesscaseinthenextstage.Thenextgatedetermineswhethertheidea
warrantsadvancingtofullscaledevelopment.Ifso,resourcesareassignedfordevelopingtheproduct;
ifnot,theprojectiskilled.Thenextgatetypicallyinvolvesdecidingwhetherdevelopmentofthe
productiscomplete.Ifso,theproductentersthetestingandvalidationstage;ifnot,itmaystayin
developmentuntilitrunsoutofresources,inwhichcasetheprojectmayrevertbacktotheprevious
stagetodeterminewhetheradditionalresourceswouldbewarranted.Theexitgateofthetestingand
validationdetermineswhethertheproductisreadyforlaunch,thatistobesoldorused.

FigureS0.AsampleStageGateProcess
TM
[source:http://www.stagegate.comisfairlytypical].
Manysoftwaredevelopmentprocessescanbecastintermsofstagesandgates.TheRationalUnified
Process[KruchtenOnRUP]describes4phasesInception,Elaboration,Construction,Transition.Each
phasehasanassociatedmilestone.Theemphasiswithinaphaseisplacedonaspecificsetofactivities,
buttheprocessdoesnotprecludeperformingotheractivitiesoutsidethatcore.Thusthephasescan
beviewedasstagesinwhichmanykindsofactivitiestakeplaceandthesameactivitiesmaybe
repeatedoriteratedacrossmanydifferentstages.Themilestonesattheendofeachphasecouldbe
treatedasgates.
Agilesoftwaredevelopmentprocessesmayalsobecastintermsofstagesandgates,butwitha
differentspintocapturetheiriterativeandincrementalnature.Whenaniterativeandincremental
processisunwound,itcanstillbeviewedasastreamofgatesandstages,withfundingincrementally
allocatedatreleaseends.Thiscanespeciallybeveryeffectiveifhighvaluefeaturesaredeployedin

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 67
earlyreleasessothattheprojectstartsgeneratingvalueassoonaspossibleandcanbecomepartially
selfsustaining[DenneHuangOnSoftwareByNumbers].Inthisvariation,ratherthanstageshaving
distinctpurposesandgateshavingdifferenttypesofdecisions,thestages,activitieswithinstages,and
gatesarereplicated.Eachstagegatepairrepresentsaniterationforimplementingthenextsetof
featurestobedeveloped,togetherwithacontinuationorfundingdecision.Thegatesareusedto
approveadditionalfundsforthenextiterationbasedonwhatwaslearnedinpreviousiterations
[HighsmithOnAgileProjectManagement,ErdogmusOnEconomicsOfAgile].(Thisisaninstanceofthe
applicationoftheprinciplesdiscussedintheChapter1SidebarRecastingaSequentialProcesses
throughWorkflowScheduleIndependenceandParallelization.)
WereorganizepartofStage3,Gate4,Stage4,andGate5,intoacustomprocessforsoftware
products:theAcceptanceProcess.WhileitcouldbearguedthatGate4,GoToTesting,assumesthe
ProductDevelopmentTeamhasinfacttestedthesoftwareextensively,inmostorganizationsthe
majorityoftestingoccursinStage4,TestingandValidation.Weproposethatmuchofthetesting
activitybelowthelevelofbusiness,user,orcustomerrequirementsoftheproductbepushedback
intoStage3,Development,andbecomepartofthereadinessassessmentactivitywithinthatstage.
ThiseffectivelymakesthereadinessdecisionpartofGate4,GotoTesting,andleavesStage4focussed
oncustomeroruseracceptancetesting.ThenGate5,GotoLaunch,wouldrepresenttheAcceptance
decision.ThevariationsonwhomakeswhichdecisionandwhenaredescribedinChapter3The
DecisionMakingModel.
HowistheAcceptanceProcessforreleasecandidatesofaproductexpressedintermsofstagesand
gatesandhowdoestheresultrelatetothetypicalStageGateprocess
TM
?Werelaxthenormal
requirementofhavingagatebetweeneachpairofsubsequentstagesandseparatethepreparation
foradecisionassociatedwithagateintoadistinctstagethatimmediatelypreceedsthegate.Wealso
allowstagestobenestedtohaveinnerstagesandgates.FigureS1TheAcceptanceProcessas
NestedStagesandGatesillustratesthisthroughanexample.ThestagesBuild,Accept,andUsein
FigureS0correspondtothestagesDevelopment,ValidationandTesting,andLaunchinFigure0.it.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 68

FigureS1
TheAcceptanceProcessasNestedStagesandGates
InsidetheBuildstageonthetop,Build1isthefirstreleasecandidatethatdevelopmentaskstheir
readinessassessorstoevaluate.Ithasseveralseverity1bugsthatmustbefixedandismissingsome
keyfeatures.Developmentfixesthesebugsandaddsthemissingfunctionality.Afterseveralmore
tries,BuildNfinallymakesitallthewaythroughReadinessAssessmentandisreleasedtotheproduct
ownerforacceptancetesting
ApositivereadinessdecisioninsidetheBuildstageautomaticallysatisfiesthecriteriaforGate4
thereforetheprojectentersintotheAcceptstage.Theproductownerconductsacceptancetesting
onlytodiscoversomekeyissues.TheseareprovidedtotheProductDevelopmentTeamwho
addressestheminBuildN+1afterseveralintermediatebuilds(notshown)thatfailtheirreadiness
assessment.NotethatatthispointtheprojectisstillconsideredtobeintheAcceptstagedespitethe
factthattheballisinTheProductDevelopmentscourttofixthebugs.Theproductownerfindsno
significantissueswiththisbuildandacceptsthesoftwareasfinished,satisfyingGate5scriteria.
TheprojectnowmovestotheUsestage.Afterafewweeksofusage,auserexperiencesaseverecrash
thatistracedbacktoamissedscenario.TheproductowneraskstheProductDevelopmentTeamfora
fix.DevelopmentcomesupwithafixinBuildN+2,whichpassesreadinessassessmentandacceptance
testing(mostlyregressiontestingplusanewtestcasecoveringthisbug)andBuildEisputinto
production.TheprojectstaysintheUsestage(oftenreferredtoasWarranty)untilthecriteriato
exittheUsestagearemet;thismaybedecidedbythepassageofanagreeduponperiodoftime
duringwhichtheproductiscoveredunderTheProductDevleopmentTeamswarranty.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 69
AttheendoftheBuildstageandduringtheAcceptstage,eachreleasecandidateisassessedfor
whetheritsatisfiesfunctionalandnonfunctionalcriteria.Howevertypically,thetransitionsfromthe
BuildstagetotheAcceptandUsestagesinvolvealsobusinessandpoliticalfactorsthatmayhavelittle
todowithwhetherthesoftwaremeetsthefunctionalandnonfunctionalrequirements(SeeChapter
XSystemRequirementsModel).Examplesofsuchfactorsinclude:
Aneedtofreeuptheproductdevelopmentteamforotherwork
Changingwhopaysformodifications:beforeacceptanceitisusuallytheproductdevelopment
teamwhileafteracceptanceitistypicallytheproductowner.
Changingthefundingsourcefromcapitaltoexpense.
Managementpressuretoshortcutagateduetoamisalignmentofprojectgoalswithincentives.
Whilethesefactorsmayplayacriticalroleingatedecisions,theyareoutsidethescopeofthisguide.

Sidebar:QualityGatesandtheAcceptanceProcess.(MicrosoftQualityGatesexamplesasMMF/MQR
criteriaforreadinessandacceptancedecisions.)
<TBA>
2.5 Summary
Thischapterintroducestheconceptoftheacceptanceprocessasawaytothinkabouttheactivities
relatedtomakingthedecisiontoacceptsoftware.Theacceptancedecisionistheculminationofa
processthatinvolvesseveralorganizationseachwithspecificresponsibilities.Eachreleasecandidateis
passedthroughthedecisionpointsoftheacceptanceprocessonitswaytothemakingthefinal
acceptancedecision.Therearethreemajordecisionpointsbetweensoftwareconstructionbythe
ProductDevelopmentTeamworkingwiththeirProductOwnerandsoftwareusagebytheenduser.
First,theProductDevelopmentTeammustdecidewhetherthesoftwareintensiveproductinitscurrent
form,knownasthereleasecandidate,isreadyforacceptancetesting.Ifitis,theProductOwnermust
decidewhetherthereleasecandidatemeetstheiracceptancecriteriabeforetheproductcanbemade
availabletotheendusers.Ultimately,eachendusermustdecideforthemselveswhethertheywantto
usethesoftware.Thisusagedecisionhasverylittleinfluenceonthecurrentacceptancedecision
althoughitmayinfluencetheacceptancecriteriaforfuturereleasesoftheproduct.
Whileeachuserpotentiallymakestheirownusagedecision,thereadinessandacceptancedecisionsare
normallyeachasingledecision.Eachdecisionpointor"gate"shouldhavewelldefinedcriteriatoguide
thedecisionmaking.ThesecriteriashouldbeknowninadvancebyboththeProductDevelopment
Teamandtheproductowner.Complexorganizationsmayhaveseveralreadinessoracceptance
decisionsbeingmadeinparallel,eachbasedontheirowncriteria(forexample,operationalacceptance);
thereleasecandidatesprogressionthroughtheprocessmaybevetoedbyfailingtosatisfythecriteria

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 70
ofanyoftheparalleldecisions.ComplexproductsmayrequiremultipleProductDevelopmentTeam
teamseachwiththeirownreadiness,andpotentiallyacceptancedecisionspriortoanoverallreadiness
andacceptancedecisionfortheintegratedproduct;thereleasecandidatecanbesentbackforfurther
workfromanyofthedecisionpoints.
Theacceptancedescribedinthischapterappliestobothsequentialandagileprojectsbutinslightly
differentways.Thephasednatureofsequentialprojectsmeansthatalltestingisdonewithinaseparate
testingphaseandtheacceptanceprocessdescribeswhatgoesonwithinthatphase.Agileprojects
traversetheentiredevelopmentlifecycleforeachfeatureoruserstory.Therefore,theacceptance
processisexecutedatseverallevelsofgranularitywiththefinestgrainexecutionbeingattheindividual
featurelevelandthelargestbeingatthewholeproduct(orfeatureintegration)level.
2.6 WhatsNext?
InChapter3DecisionMakingModelweexaminetherolesandresponsibilitiesofthevariousparties
involvedinmakingthedecisionswhichmakeuptheacceptanceprocess.Wealsodescribetheroles
playedbythepartieswhoprovidethedataonwhichthedecisionsarebased.
2.7 References
[BasLarmanOnScalingAgile]Vodde,Bas.andLarman,CraigScalingLean&AgileDevelopment:Thinking
andOrganizationalToolsforLargeScaleScrum,AddisonWesleyProfessional,AgileSoftware
DevelopmentSeriesseries.,2008
[HighsmithOnAgileProjectManagement]Highsmith,JimAgileProjectManagement:CreatingInnovative
ProductsAWP2004ISBN13:9780321219770
[MeszarosOnAgileERP]Meszaros,Gerard,etalAgileERPYouDontKnowWhatYouveGotTillits
Gone,http://agileerppaper.gerardmeszaros.com/
[AstonOnAgileERP]Aston,Janice,etalAgileERPOvercomingtheChallenges,
http://asug2008agileerpchallenges.gerardm.com/
[EcksteinonLargeScaleAgile]JuttaEckstein,AgileSoftwareDevelopmentintheLarge:DivingIntothe
Deep,DorsetHouse,2004
[ErdogmusOnEconomicsOfAgile]HakanErdogmus,TheEconomicImpactofLearningandFlexibilityon
ProcessDecisions,IEEESoftware,Vol22,No6(Nov.2005),pp.7683
[DenneHuangOnSoftwareByNumbers]MarkDenneandJaneClelandHuang,SoftwarebyNumbers:
LowRisk,HighReturnDevelopment,PrenticeHall,2003
[KruchtenOnRUP]PhilippeKruchten,TheRationalUnifiedProcess:AnIntroduction,AddisonWesley,
2003
[MillerKarterOnLargeScaleAgile]AdeMillerandEricCarter,AgileandtheInconceivablyLarge,Proc.
Agile2007,IEEEPress,2007.

Acceptanc
Cha
Chapter1
software
Decision,
Figure1il
Figure1
Simplified
Thissectio
businessm
availablet
ceTestEngin
pter3
1introducedt
isassessedfo
theAcceptan
llustratesthis
ddecisionma
onelaborates
models.Thed
throughactiv
eeringGuide
3 D
theacceptanc
oracceptabili
nceDecisiona
ssimplifiedve
akingmodel
sonhowthe
decisionsare
vities.Figure2
eVolumeI,
Decisio
ceprocessan
tybywhoeve
andtheUsage
ersionofthe
firsttwodec
notmadein
2illustratest
ReleaseCand
onM
ndthethreek
ermakesthe
eDecision.
decisionmak
isionsarema
avacuum;th
hisprocessfo
didate1
aking
keydecisions
acceptanced
kingmodel.
adeandwho
heyrequirein
orasinglede
gMod
thatneedto
decision:the
makesthem
formationth
cision:
del
bemadeas
Readiness
inavarietyo
atmustbem
71
f
made

Acceptanc
Figure2
Decisionm
Thediamo
(thedecis
onthetes
Theexpec
activities
Manyoft
IIdescribe
anumber
acceptanc
3.1
Thejobtit
businessd
thedecisi
manyoft
anentirel
maptojo
ModelSte
3.1.1 R
Thereadi
singleper
ceTestEngin
makingmode
ondontherig
sioncanbeei
stingandasse
ctationsofth
areexecuted
thepractices
ewaystodef
rofrequireme
ceisbasedon
TheSixA
tlesofthede
domainsand
onmakingm
henamesare
ydifferentro
btitleswithin
ereotypes.
Readiness
nessdecision
rsonperforms
eeringGuide
elsampleacti
ghtsideofFig
thertheread
essmentactiv
esystemund
withintheco
inVolumeIId
finetheexpec
entsrelatedp
nexpectation
AbstractR
ecisionmaker
organization
model.Thisgu
ehighlyoverl
olethantheo
norganizatio
DecisionM
nmakermake
sthisrole,th
eVolumeI,
ivities
gure2repres
dinessdecisio
vities,whicha
dertestwere
ontextofate
describehow
ctationsbase
practicesit
nsandrequire
Roles
rsvarygreatly
s,sothisguid
idealsoprov
oadedandth
onementione
nsinspecific
Maker
esthefinalre
ejobtitlemig
ReleaseCand
sentsthedec
onortheacce
assessesthes
edefinedbase
estplan.
wtodotheas
dontheneed
isnotaboutt
ements.
yfrombusine
deusesabstra
idesalistofc
hatyourcust
edasanalias
businessmo
eadinessdecis
ghtbesomet
didate1
isiontobem
eptancedecis
systemunder
edontheuse
sessactivity,
ds.Thatison
testing,itisa
essmodeltob
actrolename
commonalias
tomer(topi
here.Tosee
dels,seethe
sionbasedon
thinglikeChie
adebasedon
ion).Thetest
rtestagainst
ersrequirem
andotherpr
neofthereas
aboutaccepta
businessmod
estodescribe
ses.However
ckjustoneex
howtheabst
sidebarDec
ninputfromo
efEngineer,P
nthetestresu
tresultsareb
ttheexpectat
ents.Allofth
acticesinVol
onsthisguide
ance,and
delandacross
etheroleswi
r,beawareth
xample)may
tractrolenam
cisionMaking
others.When
ProjectManag
72
ults
based
tions.
hese
lume
ehas
s
thin
hat
be
mes
g
na
ger,

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 73
DevelopmentManager,orVPofEngineering.Thisrolecouldalsobeplayedbyacommittee,asis
commoninenterprisedeploymentwithmanystakeholders,ormadebyseveralpeopleorcommitteesin
parallelasdescribedinComplexOrganizationsinthepreviouschapter.
3.1.2 ProductDevelopmentTeam
Theproductdevelopmentteambuildsthesoftware.Generally,thisteammayincludeuserexperience
researchersanddesigners,graphicartists,requirements/businessanalysts,softwarearchitects,
softwaredevelopers,middlewareandintegrationspecialists,systemadministrators,DBAs,
release/deploymentspecialists,testersandtechnicalwriters.Inotherwords,thisteamincludesanyone
whoisinvolvedinanywayintheactualconstruction,customization,orintegrationofthesoftware.
3.1.3 ReadinessAssessors
Thereadinessassessors,astheirnamesuggests,assessthereadinessofthesoftwareforacceptance
testing.Theyprovideinformationthatisusedtomakethereadinessdecision.Thejobtitlesinvolved
dependsverymuchonthenatureoftheprojectandtheorganization,butittypicallyincludesrolessuch
asdevelopers,testers,anddocumentationwriters.Ineffect,areadinessassessorcanbeanyonewho
mightbeaskedtoprovideanopiniononwhetherthesoftwareisready.Insomecases,thisopinionis
basedonformaltestingactivities,butitmightalsobebasedontechnicalreviewsorevenqualitative
inputs.
3.1.4 AcceptanceDecisionMaker
Theacceptancedecisionmakeristhepersonorcommitteewhodecideswhethertoacceptthe
software.Inaproductcompany,ajobtitleforthisrolemightbeProductManager,butinaninformation
technology(IT)environment,thisroleistypicallyfilledbyacustomer,productowner,businesslead,or
businesssponsor.
3.1.5 AcceptanceTesters
Acceptancetestersprovidedataonacceptabilityoftheproduct.Theyperformactivitiestoassessto
whatdegreetheproductmeetstheexpectationsofthecustomerorenduser.Acceptancetestersmay
includetwoteamsonefocusingonfunctionalacceptanceofthesystem,whileanotherfocusingon
operationalacceptance.Theyprovideinformationtotheacceptancedecisionmaker.Theymaybe
dedicatedtestingstaff,endusersaskedtodotesting,oranywhereinbetweeninskillset.
3.1.6 Users
Usersmakeindividualusagedecisions.Eachuserdecideswhethertousetheproductasitiswhenitis
shippedordeployed.Theirfeedbackmightbeusedtoadjusttherequirementsforthenextreleaseorto
dousabilitytestingofthebetaversionsofthecurrentrelease.Peoplewhoareusersmayalsobe
involvedasacceptancetestersorbetatesters.Similarvitalfeedbackmightcomefromoperationsstaff,
supportstaff,trainers,fieldengineers,whoareexposedtoissuesofconsumabilityandusability.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 74
3.2 MakingtheThreeDecisions
Thissectiondescribeshowtheprecedingsixabstractrolesareinvolvedinmakingthethreedecisions.
3.2.1 MakingtheReadinessDecision
Thereadinessdecisionismadebythereadinessdecisionmaker(s).Thereadinessdecisionisanexitgate
withadecisionaboutwhethertolettheproductbeseenbeyondtheboundariesofthesupplier
organization.Thedecisionisbasedonreadinessassessment(whichisbasedonthefeaturesincluded
andthequalityofthosefeatures)donebythereadinessassessors.Thedecisioncanbemadebyasingle
person(suchasaChiefEngineer)orbyacommittee(suchasengineers,architects,orotherproject
stakeholders).WhenitisnotasingledecisionasdescribedinChapter1TheAcceptanceProcess,each
decisionmaybemadebyasinglepersonoracommittee.Fromthepointofeachdecisionmakerthe
softwaresystemiseitherreadyoritisnotready.Ifitisnotready,theremaybealistofconcernsthat
needtobeaddressedbeforeitwillbeconsideredready.Formoreinformation,seeHowWillWe
ManageConcernssectioninChapter16PlanningforAcceptance.
Theremayhavebeenanumberofearlierdecisionmakingcheckpointsaspartofthedevelopment
process(suchas"requirementscomplete,""designcomplete,"or"codecomplete").Thesearebeyond
thescopeofthisguidebecausetheyareneitherdirectlypartofthereadinessdecisionnorarethey
easilytested.
3.2.2 MakingtheAcceptanceDecision
Theacceptancedecisionismadebytheperson(orpersons)playingtheAcceptanceDecisionMakerrole.
ThedecisionissummarizedbythequestionShouldweacceptthesoftwareandputitintouse
deliveringvaluetoourorganization?Theremaybeadditionalcontractualconsequencesformakingthe
acceptancedecision,suchasacommitmenttopaythesupplier,thestartofapredefinedwarranty
period,andsoon.Whileintheorytheseshouldnotbetheprimaryconsiderationswhenmakingthe
decision,inpracticetheyoftenare.Thedecisionshouldbewhetherthesoftwareiscompleteor
doneenoughtobedeployedorshipped.Formoreinformationaboutthedefinitionof"done,"see
Chapter7DonenessModel.Formoreinformationaboutthecompletedefinitionofthesystem
attributesthatmaybeconsideredwhenmakingtheacceptancedecision,seeChapter5System
Model.
Thedefinitionof"done"isinfluencedbyseveralfactors,includingthefollowing:
MinimumMarketableFunctionality(MMF)fortheproduct.Whatfeaturesorfunctionsmustthe
productsupporttobeworthreleasing?ThisisbasedonwhatevercriteriatheProductOwner
decidesareimportant,derivedfromproductplans,marketsurveys,competitiveanalysis,or
economicanalysis.WhiletheProductOwnerisaccountableforthedecisionthisdoesnot
removetheneedfortheProductDevelopmentTeamtounderstandtheproblemdomainin
generalandtheneedsofpotentialusers.Whereverpossible,theyshouldassisttheProduct
Ownerindefiningtheproduct.ForadefinitionoftheresponsibilitiesoftheProductOwner,see
Chapter1.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 75
Minimumqualityrequirement(MQR)fortheproduct.Whatisthelevelofqualitythatmustbe
achievedbeforetheproductcanbereleased?Qualityhasseveraldimensions.Thepresenceor
absenceofbugs/defectsinfunctionalityisjustonedimensionofquality.Nonfunctional
requirements,alsoknownasqualityattributesisanotherdimension.TheMQRencompasses
thelatterwhileMMFencompassestheformer.
Harddeadlines.Bywhatdatemustaparticularversionoftheproductbereleasedtohaveany
value.Thesecanincludetradeshowdates,regulatorydeadlines,orcontractualobligations.
Eachdeadlinecouldbesatisfiedbyadifferentversion(orbuild")oftheproduct.Formore
information,see"ProjectContextModel."

Theacceptancedecisionismadebasedondataacquiredfromanumberofsourcesandactivities.
Acceptancetestinggeneratesmuchofthedataneededtomaketheacceptancedecision.Thisdata
includesthefollowing:
Pass/failresultsofallteststhatwereperformedaspartoftheacceptancetesting.Thiscouldverify
bothfunctionalrequirementsandparafunctionalrequirements.
FeaturecompletenessAsimpliedbythepass/failresultsoffunctionaltests

Theacceptancedecisionmayalsousedatagatheredduringreadinessassessment.Themostcommon
exampleofthisisdatarelatedtosystemperformance(capacity,availability,etc.)whichmanycustomer
organizationswouldnotbecapableofgatheringthemselvesbutwhichtheyarecapableof
understandingonceithasbeengathered.
Theacceptancedecisionisallaboutmaximizingvaluederivedfromtheproductbythecustomer
(representedbyProductOwner)andminimizingrisk.Timehasadirectvalueinthattimespent
collectingmoredatathroughtestinghasadirectcost(thecostofresourcesconsumedingatheringthe
data)andanindirectcost(thedeferralofbenefitthatcanonlyberealizedoncethesystemisaccepted).
Riskhascostthatcouldbecalculatedasthesumofthecostofallpossiblenegativeeventsmultipliedby
theprobabilityoftheiroccurrence.Thoughthiskindofliteralcalculationisnotfrequentlydoneour
perceptionsofriskareinherentlybasedonanintuitiveinterpretationofthecircumstancesalongthese
lines.Theintentisfortheacceptancedecisionmakertounderstandthetradeoffsanddecideon
whetherweneedmoredata,orwehavedoneenoughandwecanmakethedecision.Indoingthisitis
importanttobeawareofthetwoextremenegativepossibleoutcomes,oneintime,theotherinrisk.
Examplesofcostsofunmitigatedriskmightincludethefollowing:
costofpatchingsoftwareinthefield;
costofmanualworkaroundsforbugs;
costofmaintainingspecializedresourcesforsoftwaremaintenance;
losingcustomersthatneedspecificfeaturesthataremissing;
opportunitycostofunrealizedordelayedbusinessbenefitsifproductfailsusage
assessmentorendupnotbeingusedafterdeployment.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 76

Thecostoftimecanbenonlinearinthatifadeadlineorneedismissed,allbenefitsmaybelostand
punitiveactiontaken.Theriskthecalculationisdifferentfor'blackswan'[TalebOnImpactOfImprobable]
eventsofextremelylowprobabilitybutextremelyhighimpactsuchthatagainallvaluemaybelost.
Thesecannotbeignoredintheacceptanceofnewsoftware.Inconcept,whenthecostofriskexceeds
thecostofdelay,moretestingshouldbeperformed.Whenthecostofmoretestingexceedstherisk
relatedcostthatwouldbereduced(byreducingprobabilityofoneormoreeventsoccurringorby
reducingtheexpectedcostgiventheeventdoesoccur),youcandecidetoaccepttheproductwithout
furthertesting.
3.2.3 MakingtheUsageDecision
Eachpotentialuserofthesystemhastomakeapersonaldecisionaboutwhethertousethesoftware.
Thisdecisionisdifferentfromtheacceptancedecisioninthatitismademanytimesbydifferentpeople
ororganizations.Infact,theremaybeseveraltiersofthesedecisionsascompaniesdecidewhetherto
adoptaproduct(oranewversionthereof)anddepartmentsorindividualsdecidewhethertocomply
withtheorganizationaldecision.Theimportantconsiderationfromtheperspectiveofthisguideisthat
thesedecisionshappenaftertheacceptancedecisionanddonotdirectlyinfluencetheacceptance
decision.Theymayindirectlyinfluenceitinoneofthefollowingtwoways:
Proactively.Usagedecisionsmayindirectlyinfluencetheacceptancedecisionforanupcoming
releasebyfutureuserscommunicatingtheindividualacceptancecriteriatotheproductowner
inresponsetomarketresearchorsurveys.Thistypeofcriteriamayalsobesubmittedtothe
productownerthroughunsolicitedinputs,suchasfeaturerequestsorbugreports.Also,user
feedback(includinglackofusage)fromalphaandbetareleasesmayinfluencetheacceptance
criteriaforthegeneralrelease.Seesection1.2.3TheAcceptanceProcessforAlpha&Beta
ReleasesinChapter1IntroducingtheAcceptanceProcessforamoredetaileddiscussion.
Retroactively.Usagedecisionsmayindirectlyinfluencetheacceptancedecisionbyproviding
feedbackonthereleasedproductindicatingalackofsatisfactionineitherfunctionalityor
quality.Thismayinfluencetheacceptancedecisioncriteriaofafuturerelease,butitrarely
causestheacceptancedecisionalreadymadetoberevisited.Thenotableexceptionwouldbe
thediscoveryof"severity1"bugsincriticalfunctionalitythatmightresultinarecallofthe
releasesoftware.

Theusagedecisionofteninvolvesdecidingwhetherornotthebusinessoruserisreadytostartusing
thesoftware.Thisissometimescalledbusinessreadinessanditfallsintheareaofchange
management.Oninternalinformationtechnologyprojects,theProductOwneristypicalresponsiblefor
businessreadiness.Thismightincludetrainingofusers,transforminglegacybusinessdataintoformats
compatiblewiththenewsystemorpreparationofnewdatareadytobeloadedwhenthenewsystem
becomesavailable.Whileitisanimportanttopic,itis,strictlyspeaking,notpartofacceptingsoftware
andisbeyondthescopeofthisguide.
Sidebar:ThePerilsofIgnoringtheUsageDecision

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 77
Acompanyoneoftheauthorsworkedforimplementedapopulartoolformanagingleads.The
ProductOwnerTeamconsultedwiththekeystakeholderswhowouldbeconsumingthereportson
potentialfuturebusinessinthesalespipelineandconfiguredthesystemwiththefieldneededto
generatingthereports.Mostofthefieldsweremarkedasmandatory.
Whenthesystemwasreadyforacceptancetesting,theProductOwnersteamtestedthesystemand
foundittobeworkingtotheirsatisfaction.Theyconductedextensiveusertraingingofthesalesforce
toensurethateveryoneknewhowtousetheirwonderfulnewtool.Theyacceptedanddeployedthe
system.Whenthesystemwentlive,theuseruptakewaspoor.Severalyearsafterdeployingthe
system,mostuserswerestillchoosingtonotusethetoolbecauseitwasprovidingthemwithvery
littlevaluewhileimposingconsiderableoverheadontheirdaytodaywork.Mostofthemdeveloped
theirquotesinspreadsheetsorinacustombuiltpricingapplicationthatdirectlyaddressedtheir
needs.Theleadmanagementtooldidntprovidetheequivalentfunctionalitythereforeitadded
additionalstepsreenteringdatabutindifferentformats.Tomakemattersworse,itforcedtheusers
topickthroughalonglistofpossiblecustomerseachtimeauserneededtocreateaquote.Asaresult,
thesystemlanguishedandmuchofthepotentialvaluewasalostopportunitycost.
TheProductOwnerofasubsequentprojecttoreplacethecustompricingtooltookthelessonsto
heartandincludedfunctionalitytoaddresstheshortcomingsbyfeedingthepricequotesintothelead
managementsystemandsynchronizethecustomersautomatically.Thissatisfiedtheneedsofthe
reportgenerationstakeholdersbyprovidingthemwithallthedataneededtocalculatethepresent
valueofleadsinthesalespipelinewhilenotburdeningtheuserswithduplicatedataentry.Asaresult,
usageoftheleadmanagementtoolsoared.Ahappythoughbelatedending.
3.3 Rolesvs.Organizations
Therolesdescribedinthisdecisionmakingmodelmaybeplayedbypeopleinseveraldifferent
organizations.Theprimaryvalueofdiscussingorganizationhereisinmakingiteasiertomap
terminologyfromvariousorganizationmodelstobetterunderstandwhoplayswhichdecisionmaking
role.Iftheorganizationalmodeldoesnothelpinthisendeavour,itcanbeignored.
Whenthesoftwareisbeingbuiltbyadifferentorganizationthantheonewhocommissionedits
construction,theorganizationthatcommissionedthesoftwareisoftenreferredtoasthecustomer,and
theorganizationthatisbuildingthesoftwareisthesupplier.Thisistruewhethertheorganizationsin
questionareseparate,unrelatedcompaniesorsimplydepartmentswithinasinglecompany.For
example,theITdepartmentistypicallyasupplierofsystemstothecorebusinessdepartments(suchas
TransportationorManufacturing)andsupportingdepartments(suchasHumanResourcesorFinance.)
Whenacceptancetestingisoutsourcedtoathirdpartytestorganization,itisoftenreferredtoasthe
(thirdparty)testlab.Thetestlabisasupplierofservicesasdistinguishedfromthesupplierofthe
software.
Anorganizationthatbuysanddeploysshrinkwrappedsoftwarecanalsobereferredtoasacustomer,
andtheorganizationtheybuyitfrommaybereferredtoasthevendororsupplier.Thefactthatthe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 78
vendorcontractstheworktoanoutsourcer(anothervendorofwhichtheyarethecustomer)illustrates
theproblemwithusingtheterm"customer"todescribetheacceptancedecisionmaker(theproduct
owner)asadvocatedinextremeprogrammingwhichcustomerisitreferringto?
FigureXillustratesthisproblem.ThePurchaser(intheroleofcustomer)buysshrinkwrapfromS/W
Vendor(inroleofsupplier).S/WVendor(intheroleofcustomer)outsourcesdevelopmenttoS/W
Developer(thesupplier).S/WDeveloper(ascustomer)outsourcesreadinessassessmenttoTestLab(as
supplier).Sowehavethreeseparatecustomersupplierrelationshipsinthisscenariomakingithardto
tellwhichpartywearetalkingaboutwhenwerefertothecustomer.

FigureX
MultipleCustomersandSuppliers
3.3.1 WhoPlaysWhichRoles?
Thusfarthediscussionshavecenteredontheabstractrolesinvolvedinthedecisionmakingprocess.But
whoactuallyplaystheroles.
WhoPlaystheReadinessDecisionMakingRole?
TheReadinessDecisionMakerroleistypicallyperformedbysomeoneintheproductdevelopment
organization.Theyarethepersonwhoultimatelydecideswhetherthesoftwareisreadytobeshownto
theacceptancetesters.TypicaljobtitlesincludeDevelopmentLead,ProjectManager,Directorof
Development,ChiefEngineer,etc..Onagileprojectswherethisdecisionismadeseparatelyforeach

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 79
feature,thefeaturelevelReadinessDecisionMakerisoftenthedeveloperwhosignsupforthe
responsibilitytoimplementthestory/feature.Intheterminologyusedinthisguide,theReadiness
DecisionMakerisamemberoftheProductDevelopmentTeam.
WhoPlaystheAcceptanceDecisionMakingRole?
TheAcceptanceDecisionMakerroleistypicallyperformedbysomeoneinthebusinesssideofthe
organizationwhendevelopmentisdoneinternallyorbysomeoneinthecustomerorganizationwhen
developmentisoutsourced.Theyrethepersonwhoultimatelydecideswhetherthesoftwareisreadyto
beshowntotheacceptancetesters.TypicaljobtitlesforinternallydevelopedsoftwareincludeBusiness
Lead,ProductManager,orsomekindofbusinesstitle.Onagileprojectswherethisdecisionismade
separatelyforeachfeature,thefeaturelevelAcceptanceDecisionMakerisoftenthespecificbusiness
subjectmatterexpertwhoprovidedthedetaileddescriptionofthefunctionalitytobebuilt.Inthe
terminologyusedinthisguide,theProductOwneristheAcceptanceDecisionMaker.
WhoDoesWhatTesting?
ReadinessassessmentistheresponsibilityoftheProductDevelopmentTeamButdoesthismeanthat
developersaretheonlyonesdoingreadinessassessment?Bythedefinitionusedinthisguide,the
ProductDevelopmentTeamoftenincludespeoplewhowoulddescribethemselvesastesters.
Developerswoulddoataminimum,unittesting.Eithertestersand/ordeveloperswoulddowhole
systemtesting(functionalandnonfunctional).Otherpartieswhoarealsoconsideredpartofthe
ProductDevelopmentTeammaydootherformsofreadinessassessment.Forexample,dataarchitects
mightdodataschemadesignreviews.Securityarchitectsmightdosecuritycodereviews.
Ideally,acceptancetestingshouldbedonebyrealusersofthesoftwaresincetheyaretheoneswhoare
bestsuitedtodecidingwhetheritisacceptable.Butwhatiftherealusersareanonymous?Whocan
andshoulddoacceptancetestingontheirbehalf?Thisisacasewheretestersoftenactastheproxyfor
theenduser.InthissituationatleastsomeofthetesterswouldbepartoftheProductOwnerTeam
ratherthantheProductDevelopmentTeam.
InFigureX2WhoDoesWhatTestingbyTypeofOrganization,eachcelldescribeswhodoeswhich
testinginacommonprojectcontext.Thetoplefttriangleinthecellindicateswhoisdoingreadiness
assessmentandthebottomrighttriangleindicateswhoisinvolvedinacceptancetesting.

Acceptanc
FigureX2
WhoDoes
Whensof
participat
developm
wouldtyp
integrati
orbusine
Whensof
acceptanc
doesacce
testingfu
ProductO
Intheterm
Developm
3.4
TheReadi
Readiness
typicallyi
Makerist
theProdu
organizat
ceTestEngin

sWhatTestin
ftwareisbein
tingintheacc
menttotestth
picallyparticip
ontestingb
essacceptanc
ftwareisbein
cetesting(ex
eptancetestin
nction,theac
OwnerTeam)
minologyuse
mentTeamwh
Summary
inessDecision
sDecisionMa
ncludedevelo
toensuretha
uctOwner.Th
ion.
eeringGuide
ngbyTypeof
gbuiltbyan
ceptancetest
hesoftwaret
pateina2
nd
p
eforehandin
cetesting(B
gbuiltforsal
xceptperhaps
ngontheirbe
cceptancetes
ortheaccept
edinthisguid
hiletestersdo
y
nismadeby
akerbasedon
opers,archite
attheproduct
heRDMistyp
eVolumeI,
f Organization
organization
ting.Iftherei
horoughlybe
phaseofread
gthesoftwar
BAT).
letoanonym
sasusersofa
ehalfactingas
stingcouldbe
tancetesting
de,testersdo
oingacceptan
someoneors
ninformation
ectsandsom
thasenough
picallyasenio
ReleaseCand
n
foritsownu
snotestingf
eforehanding
dinessassessm
reovertothe
ouscustome
alphaorbeta
saproxy(or
edonebythe
gcouldbeout
ingReadiness
ncetestingar
somecommit
ngatheredby
etimestester
functionality
orpersonint
didate1
se,theusers
function,the
gitover.Ifthe
ment,oftenc
eusersforu
rs,realusers
releases.)Ins
surrogate)us
eproductspe
tsourcedtoa
sAssessment
rememberso
tteefromthe
yReadinessAs
rs.Thegoalo
yandisofgoo
heproductd
shouldbeav
businesstypi
ereisatestin
alledsystem
seracceptan
typicallydon
stead,thetes
ser.Ifthereis
ecifiers(mem
3
rd
partytest
taremember
oftheProduc
esupplierplay
ssessors.Rea
oftheReadin
odenoughqu
evelopmentt

vailablefor
callyexpects
ngfunction,th
mtestingand
cetesting(U
nttakepartin
stingorganiza
snospecialize
bersofthe
tlab.
rsoftheProd
ctOwnerTeam
yingtherole
adinessAssess
essDecision
ualitytoexpo
teams
80
hey
d
UAT)
n
ation
ed
duct
m.
of
sors
oseto

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 81
TheAcceptanceDecisionismadebytheProductOwner(whichmaybeasinglepersonoracommittee
ofseveralkeystakeholders)playingtheroleofAcceptanceDecisionMakerbasedoninformation
gatheredbypeopleplayingtheroleofAcceptanceTesteraswellasanyReadinessAssessment
informationsharedbythesupplier.TheADMisusuallyakeypersoninthecustomerorganization.The
acceptancetestersmaybeendusers,representativesoftheenduserssuchasthespecifiersofthe
requirements,ortesters)whenendusersarenotavailabletodoacceptancetesting.Whenthe
customerorganizationshascommissionedtheproductionofaproductandtheyhavenointernaltest
capability,theymaychoosetooutsourcesomeoftheacceptancetestingtoathirdpartytestlab.
3.5 References
[TalebOnImpactOfImprobable] Taleb, Nassim Nicholas.TheBlackSwan:TheImpactoftheHighly
Improbable.RandomHouse.2007

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 82
Chapter4 ProjectContextModel
TheProjectContextModelisawaytobetterunderstandthegoalsoftheprojectandtheconstraints
underwhichitmustoperate.Itisnotaformalmodel;instead,itisasetofinformationtobecollected
andfactoredintootheractivities.Itincludesinformationsuchasthefollowing:
UsageContext.Whatistheoverallcontextinwhichthesoftwarewillexist?
Businessgoals.Anexamplequestiontogatherinformationfortheseis,"Whatisthebusiness
valuetobeprovidedbythesystemandhowisittobeachieved(strategy)?"
Scope.Anexamplequestiontogatherinformationforthisis"Whatfunctionalityisinscopeand
whatisoutofscopeforthisproject?"
Stakeholdersandusers.Anexamplequestiontogatherinformationfortheseis"Whoarethe
intendedusersandwhoaretheotherprojectandsystemstakeholders?"
Budget.Anexamplequestiontogatherinformationforthisis"Howmuchmoneyisavailableto
achievethebusinessgoal?"
Harddeadlines.Anexamplequestiontogatherinformationfortheseis"Whatdeadlinesmust
bemetfortheprojecttobeconsideredasuccess?"Examplesofharddeadlinesinclude
tradeshows,contractualdeadlines,andregulatorydeadlines.
Constraints.Anexamplequestiontogatherinformationfortheseis"Whatresources(suchas
people,space,andequipment)areavailabletotheproject?"Afollowupquestionmight
include"Whichresourcesarenegotiableandwhicharehardconstraints?"

Thesectionsofthischapterprovidemoreinformationabouteachofthesefactors.Othercontextual
factorsnotdiscussedinthischapteralsohaveaninfluenceonsoftwareacceptance.Thesefactors
include:
Criticalityofthesystem,whichaffectsdefecttolerancelevels,howcomprehensiveacceptance
testingshouldbe,andwhatconstitutesgoodenoughquality(Cockburn[Cockburn],for
example,definesfourzonesofcriticalityofthesystembasedonthefailureimpact:Lossof
Life(clearlythehighest),LossofEssentialMoney,LossofDiscretionaryMoney,orLossof
Comfort.Boehm&TurnerscriticalityscaleissimilarbutdistinguishesLossofManyLives
fromLossofSingleLife[BoehmTurner]);
ProcessesandToolsavailableandusedintheproject,suchascontinuousintegration,
automatedbuilds,andsourcecodemanagementtools,whichaffectthetiming,style,
amount,andfrequencyofacceptancetesting;and
Trust,whichaffectshowmuchandwhatkindofretestingtoperformbasedontestingdataand
resultsfrompreviousstagesorfromotherteams(e.g.,howmuchdoestheproductowner

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 83
performingacceptancetestingtrusttheproductdevelopmentteamperformingreadiness
assessment).
4.1 UsageContext
Softwarecanbebuiltformanydifferentkindsofusageandtheorganizationthatbuildsorcommissions
itaffectshowthesoftwareisbuiltandaccepted.Figure1TheProductContextillustratesonewayof
classifyingthesoftwareintensivesystemswebuild.

Figure1
TheProductContext(courtesyofJeffPatton;usedwithpermission)
Someorganizationsbuildsoftwarefortheirowninternalusagetoreducecostswhileothersbuild
productstoselltootherstogeneraterevenue.Theleftsideofthequadrantdiagramdescribeswhatis
commonlycalledenterprisesoftware;softwarethatisbuiltbytheenterpriseforitsownuse.Theright
sideofthediagramrepresentssoftwarethatisbuiltforsaletootherpartieseitherassoftwareinits
ownright(whichmaybeinstalledonsiteorsoftwareasaservice)orassoftwareembeddedina
hardwareproduct.Someoftheproductsbuiltontherightsideofthediagramshowupastools,
components,frameworksorproductsusedintheconstructionoftheenterpriseproductsontheleft
sideofthediagram.Independentlyofthis,theusersofthesoftwaremaybecaptiveuserswhoare

Acceptanc
compelled
product.T
4.1.1 E
Enterprise
allorpart
businessp
fromscra
productp
enterprise
developm
othersyst
challenge
illustrated
Figure2
BusinessP
Theinteg
transactio
Acceptanc
businesse
Figure2a
perspectiv
ofthesys
ceTestEngin
dtousethep
Theprocesso
Enterprise
esystemsare
tofvariousbu
process,eithe
tchbywriting
purchasedfro
e,oftencalled
mentmaybeo
temswithint
ofbuildinga
dbytheright
Processvs.Sy
rationrequire
onalinteractio
ceofenterpr
enablemento
alsopointsou
vewhileman
temsthatim
eeringGuide
productonce
ofacceptings
Systems
ecommission
usinessproce
erinwholeor
gcode,comp
mavendor(
dtheInforma
outsourcedto
heenterprise
ndaccepting
sideofFigur
ystemsIntegra
ementsmayt
onstonamej
isesystemsin
orcostreduc
utthecauseo
nyProductDe
plementthe
eVolumeI,
itisavailable
softwarecanv
edbyanente
esses.Theuse
rinpartasill
posedofpurc
ontherights
ationTechnol
oathirdparty
e(whichoften
gthesesystem
e2.
ationView
takemanyfo
justafew,bu
nvolvesdeter
tiontargets.
ofmanypote
evelomentTe
process.This
ReleaseCand
eortheymay
varybasedon
erpriseforits
ecasesofeac
ustratedbyt
hasedcompo
sideofFigure
logy(IT)orIn
y.Manyofth
nincludelega
msisensuring
rmsincluding
utallshouldb
rminingwhet
ntialissues:a
amsarefocu
scanleadtom
didate1
ychoosetous
nthiscontext
sowninterna
chsystemimp
heleftsideo
onentsorcrea
2.)Thework
formationSy
hesesystems
acysystems)
gthattheinte
gdatareplica
bedrivenfrom
therornotth
acceptanceis
sedexclusive
mismatchesb
setheproduc
t.
aluse.Thesys
plementone
fFigure2.Th
atedbyconfig
kmaybedone
ystems(IS)de
tendtobein
andalargep
egrationwork
ation/synchro
mthebusines
esystemwill
sdonefroma
elyonasubse
betweenbusi
ctoracompe
stemsautoma
ormorestep
heymaybebu
guringapack
ebyapartof
epartment,or
tegratedwith
artofthe
kscorrectlya
onizationand
ssrequiremen
meetits
businesspro
et(oftenjust
nessexpectat
84
eting
ate
psofa
uilt
kaged
fthe
rthe
h
s

nts.
ocess
one)
tions

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 85
andwhattheITdepartmentdelivers.ThisisespeciallytruewhenITisorganizedaroundthesystems(as
componentteams)ratherthanbusinessprocesses(asfeatureteams)(seediscussiononcomponent
teamsvsfeatureteamsintheAcceptingComplexProductsectionofChapter2)orwhendevelopmentof
thesystemisoutsourced.Thelatterisparticularlyfraughtwithperilastheoutsourcermaynot
understandanythingaboutthebusiness.
Becausethesoftwareiscommissionedbytheenterpriseitself,thereisthepotentialforalargedegree
ofbusinessinvolvementinthedevelopmentprocess.Thiscanresultinsoftwarethatisaverygoodfit
forthebusiness.Unfortunately,manybusinessesdelegatethehardworkofdefiningtheirrequirements
totheproductdevelopmentteam,whetherinternaloroutsourced,andeffectivelyabdicatecontrolover
whatisbuilt.Thisoftenresultsinbarelyusablesystemsthathavelong,painfulacceptancephases.A
goodwaytoavoidthisistohavestrongbusinessownershipandstewardshipofthedevelopmentof
thesesystemsintheformofaninformedproductownerwhomakesthecriticaldecisionsaboutwhat
thesystemsshouldsupportandwhatitshouldnot.Thispositiongoesbyvariousformaltitlessuchas
businesssponsor,businessleadorproductowner.Ifacriticaldecisionrequirestechnicalknowledgeand
theproductownerisnottechnicallysavvy,theproductownerneedsassistancefromtheproduct
developmentteamoroutsideadvicetomakethedecision.
ThesoftwareinthebottomleftquadrantofFigure1istypicallyembeddedinabusinessworkflow
implementingoneormanystepsofthebusinessprocess.Theusersdonothaveachoiceofwhetheror
nottousethesoftwarebecauseusingitispartoftheirjob.Usersarehighlyaccessiblethereforecanbe
usedforusabilitytestingandacceptancetestingbutbecauseusageisnonoptionalthereisan
unfortunatetendencyoftheproductownertoignoretheusabilityattributesofthesystem.Theusers
areofteninvolvedintheacceptancetestingprocessinanactivityoftencalleduseracceptancetesting
(UAT).
ThesoftwareinthetopleftquadrantofFigure1isdifferentinthatusersarenotcompelledtousethe
producteventhoughitwasbuiltforinternaluse.Theproductownertriestorepresenttheneedsofthe
realuserwhomayormaynotbeaccessible.Forexample,thesoftwaremaybeusedprimarilybyin
houseuserswhoareaccessible,oritmaybedesignedfortheuseofpeopleoutsidethecompany
includingcustomersorbusinesspartners.Thesystemmaybestandalonebutismorelikelytobe
integratedbehindthesceneswithothersoftwarewithintheenterprise.
4.1.2 SoftwareIntensiveProducts
Manyproductsinthemodernworld,fromtoystoautomobiles,haveasoftwarecomponentandthe
amountofsuchembeddedcomponentsissteadilyincreasing,especiallywithregardstotheportionof
thevalueaddedtothefinalproduct.Thissoftwareiswhatprovidesthecomplexbehaviorsthat
characterizeproductsinthe21stcentury.Theseproductshavebeenalargepartofthegrowthof
moderneconomies.Acceptanceoftheseproductsinvolvesdecidingwhethertheproductasbuiltwill
meetitsrevenuetargets,contributeordetractfromthebrandimage,andpossiblyotherfactors.The
productrequirementsaretypicallydrivenbyaproductmanager.
ThesoftwareinthebottomrightquadrantofFigure1isbuiltbyonecompanyforsaletoanother
company.Thissoftwaremaybepurchasedandusedasis.Oritmaybedesignedtobeconfiguredor

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 86
customizedbyorforthecustomertofitwithintheirenterpriseenvironment(ontheleftsideofthe
diagram.)Thenotionofacceptancethenhasaninterestingtwistinthatthegenericproductmustfirst
beacceptedbytheproductmanagerbeforebeingmadeavailabletothecustomerswhomaythenhave
itconfiguredorcustomizedbeforeacceptingitfordeploymentwithintheirorganizations.Thisis
describedinmoredetailinsectionAcceptingCustomizableProductsinChapter1TheAcceptance
Process.
Thesoftwareinthetoprightquadrantmaybepurelysoftwareforusebyconsumers(socalledshrink
wrappedsoftware)oritmaybeembeddedinahardwareproduct.Ineithercasetheenduseristoo
numerous(andisoftennotavailable)toinvolveintheacceptanceprocess.Notethattheremaybe
additionalspecializedtechniquesusedforbuildingandassessingembeddedproductsandthese
specializedtechniquesareoutofscopeforthisguidebutmanyofthetechniquesdescribedheredo
applytoembeddedproducts.
4.2 BusinessGoals
Businessgoalsaffecthowthesoftwarewillbeaccepted.Toomanyprojectsarerunwithamajorityof
thestaffnotunderstanding(orevencaringabout)thebusinessgoalsoftheproject.Sometimesthisis
theresultofdeliberatedecisionsbymanagementtowithholdinformation,andsometimesitisjustan
oversight.Eitherway,weshouldexpecttogetsuboptimalresultsifeachteammemberisfocusedon
optimizinghisorherownjobassignmentinsteadofensuringthatbusinessgoalsareachieved.
Asabareminimum,weshouldensurethateveryoneontheprojecthasaclearunderstandingofwhat
theprojectisexpectedtodeliverandhowthatwillprovidevaluetothebusiness.Exampleofdifferent
waystoaddvaluetothebusinessincludereducedcosts,increasedsales,moresatisfiedusers,and
improvedmarketperception/branding.
4.3 Scope
Scopedetermineswhatneedstobeincludedbeforetheproductwillbeaccepted.Thescopeofthe
projectshouldrealizethebusinessgoals.Atthebroadestlevels,thescopecanbedefinedintermson
thetypesofusersweexpecttosupportandthetypesoffunctionalitythattheproductwillsupplyto
them.Itisasimportanttoexplicitlyindicatewhatisoutsidethescopeasitiswhatisinside;otherwise
weriskwastingtimeandenergydiscussing,andpossiblybuildingandtestingfunctionality,thatis
supposedtobeexcluded.Whentherequirementschangeorevolveafterdevelopmentstartsandthe
scopeneedstoberevisited,thedecisiontochangealreadyscopedfunctionalityorexpandittoinclude
anypreviouslyexcludedfunctionalityshouldbeintentional,rational,andexplicit.Animportantquestion
theProductDevelopmentTeamshouldasktheProductOwnerinthatcaseiswhatexisting
stories/featuresintheproductbacklogshouldbereplacedwithanewstory.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 87
4.4 StakeholdersandUsers
Stakeholdersandusersaffectwhoplayswhatroleintheacceptanceprocess.Acceptingsoftware
requirestheinvolvementofvariousstakeholders.Somearedirectlyinvolvedintheacceptancetesting
anddecisionmakingprocess,whileothersneedtohavetheirinterestsprotectedeventhoughtheyare
notinvolved.
4.4.1 Users
Themostobvioususagegoalholdersarethepeople(orsystems)whoaretousethecorefunctionality
ofthesystem.Typically,thesepeoplearereferredtoastheusers.TheyaretheoneswhouseaWeb
sitefordoingtheirbanking,entertainment,oronlineshopping;useanapplicationforexecutingoneor
morestepsofabusinessprocess;oroperateacombinedsoftware/hardwareproductsuchasamedical
imagingsystem.Butthesearenottheonlypeopleinteractingwiththesystem.Otherusersincludethe
peoplewhoadministerthesystembypopulatingthecatalogoftheonlinestoreorsetupthecontentin
anonlineentertainmentsystem;thepeoplewhorundiagnosticsonandmaintainthesystem.Thereare
alsopeople(orsystems)whoinstallthesystem,startit,monitoritsstatus,andshutitdownwhenthe
serversneedmaintenance.Theyallhaverequirementswithrespecttohowtheyusethesystem.For
moreinformationaboutusersandtheirgoals,seetheUserModelingactivityinVolume2ofthisseries.
4.4.2 SystemStakeholders
Therearealsostakeholders[ACUC]whowillnotdirectlyinteractwiththesystemwhenitisinoperation
butwhoexpectthesystemtolookoutfortheirinterestsasitisused(orabused)byothers.For
example,asystemthatcontainspersonalinformationaboutsomeonehasthatpersonasastakeholder
eveniftheythemselvesdonotdirectlyinteractwiththesystem.Thisisanexampleofanonfunctional
requirement.
4.4.3 ProjectStakeholders
ProductsandITsystemsareusuallydevelopedwithinaproject.Theremaybemanydefinedroleswithin
theproject,someofwhichmaybeplayedbyusersorsystemstakeholdersandsomebyuniqueparties.
Theseprojectstakeholdersmaybeinvolvedintheacceptancetestingoracceptancedecisionmaking
processwithoutbeingadirectsystemstakeholderoruser.Forexample,theproductmanageror
productownermayneverusethesystem,norbeasystemstakeholder,buttheydefinitelyhaveastake
intheacceptancedecisionprocess.ThebusinesssponsorofanITprojectmayneverusetheproductin
thefieldorevenviewareportitgenerates,butthepersoninthisrolehasaclearstakeintheprojects
outcomeintheformoftheexpectedbusinessbenefitsinreturnfortheinvestmentoftimeandmoney.
4.4.4 CommunicationBetweenStakeholders
Thelikelihoodofacceptanceofasoftwareintensivesystemisdirectlyproportionaltotheeffectiveness
ofthecommunicationbetweenthosewhocommissionedthesoftware(theproductownerorthe
productownerteam)andtheteambuildingthesoftware(theproductdevelopmentteam).The

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 88
requirementsandthetestsarepartofthiscommunication.Thecommunicationismademuchmore
effectiveifthereisacommonlanguagethateveryoneagreestouseconsistently.
Thecommonlanguagethateverybodyintheproductdevelopmentteamandtheproductownerteam
sharesisoftenreferredtoasaUbiquitousLanguage[EvansOnDomainDrivenDesign].Themembersof
theproductownerteamarethedomainexpertsandarticulatetherequirementsinthelanguageofthe
domain.Howeverthislanguageshouldbeaccessibletoandunderstoodbytheproductdevelopment
team.Conversely,thetechnicaljargonthatthedevelopmentteamusestoexpressgeneraldevelopment
conceptsrelatingtothecustomerfacingaspectsoftheproject(inthedesign,architecture,testing,and
acceptanceoftheproduct)shouldnotneedtobeaccessibletoandunderstoodbytheproductowner
team.TheUbiquitousLanguageofthewholeteamshouldbedevelopedthroughcloseinteractionacross
andwithintheproductowneranddevelopmentteamsearlyintheprojecttobeeffectiveinreducing
thecommunicationoverheadandavoidingcostly,latemisunderstandings.
4.5 Budget
Mostprojectshaveafixedbudget.Thistendstoconstrainthenumberofpeoplewhowillworkonthe
projectbothinnumberandinrateofpay.Oncetheteamsizeandcompositionisdefined,theteamwill
exhibitawelldefinedburnrateandanyextensionoftheprojecttimelinesduetolatedeliveryofthe
softwarewilltranslateintobudgetoverrunsunlessasignificantbufferisheldbackwhenplanningthe
project.
Anincrementalfundingapproach[DenneClelandHuang]isaviablealternativetoafixedbudget,and
providesflexibilityincostmanagement,scheduling,resourceallocation,teamcomposition,andscoping.
Incrementalfundingreliesonprioritizingandschedulinggroupsofusable,highvaluefeaturesearlyin
theprocessandcommittingresourcesforsuchchunksoffunctionalityinapiecemealmanner.
Incrementalfundingisparticularlysuitableforprojectsthatexhibithighcost,schedule,businessand
technical,uncertainty.
4.6 HardDeadlines
Allprojectsfacedeadlinesofonesortoranother.Someprojectsroutinelymissmanyofthesedeadlines.
Whenthebusinessconsequenceofmissingadeadlineissignificant,weshouldtaketheappropriate
measurestoensurethatthedeadlinesarenotmissed,suchasaplanningcontingencybufferorcutting
scope.Therefore,itisessentialtounderstandwhichdeadlinesarearbitrary(suchasin"Itwouldbe
reallygreatifwecouldhavethatfunctionalitybySeptember")orcritical(suchasin"Weneedto
demonstratethisfunctionalityatthetradeshowinSeptemberandfailuretodosocouldsignificantly
affectourfourthquartersalesandourshareprice.")Intheformercase,missingthedeadlinemayresult
inslightorevennoinconvenience.Inthelattercase,thevalueofthedeliverycoulddiminishtozero
makingtheproductandtheprojectthatdeliversitcompletelyirrelevant.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 89
4.6.1 RecoveryfromDevelopmentScheduleSlippage
Amajorissuewithharddeliverydeadlinesrelatestotheperceivedcompressibilityoftheacceptance
testingphase(andtestingingeneral.)Thisisparticularlyanissuewhenthedeliverydateisfixedand
immovable.Whenthedevelopmentphaseisrunninglate,asitoftendoesonwaterfallproject,thereis
greatpressureontheentireteamtorecoverthescheduleslippagesomehow.Therearetwotechniques
commonlyemployed,eachwiththeirowndifficulties.
CompressingtheTestCycle
Readinessassessmentandacceptancetestingwilltypicallystartlatebutisoftenexpectedtomakeup
thescheduleslippagebyreducingthelengthofthetestcycle.Thisresultsinlesstestingthanplanned
andthereforelikelymorebugsslipthroughtoproduction.Inotherwords,theacceptancedecisionis
madebasedonincompletedataaboutthequalityoftheproduct.
OverlappingTestCyclewithDevelopment
Theotherschedulerecoverytechniquethatisoftenusedwithharddeadlinesistoaskthetestersto
starttestingtheproductbeforeitiscompletelyfinished.Whilethispractice,IncrementalAcceptance
Testing,isconsiderednormalonanagileproject,aprojectplannedasastrictlysequentialproject
cannotadoptthispracticeeasilymidproject.Forexample,manyfeaturesmaybe80%doneratherthan
80%ofthefeaturesbeingcompletelydoneandtestable.Thereforeitishardforthetesterstoknow
whattotestbecausethereisntaclearlistofwhatshouldbeworkingnow.Anotherissueisthat
IncrementalAcceptanceTestingusuallyincludesextensiveautomatedregressiontestingsothatthe
testerscanfocusonthenewlycompletedfunctionalityratherthanhavingtomanuallyretesteverything.
Theautomatedregressiontestingisalsoakeyformofbugrepellentusedbythedeveloperstoavoid
introducingregressionbugsbydetectingthesebugsaspartofreadinessassessmentsotheycanbefixed
beforethenewreleasecandidateispassedontothetesters.
4.7 Constraints
Mostprojectsoperateundersomekindofconstraints.Constraintscanincludepeopleandskillsor
facilitiesandequipment.Sometimestheseconstraintscanbeloosened,whiletheyarestrictlylimitedat
othertimes.Acompanymaynotbeabletohireadditionalstafforrecruitpeoplewithspecificskills.For
example,theskilllevelofthestaffperformingtestingmayhaveasignificantinfluenceonthetesting
approachesused,theamountoftestingdone,andthetimeittakestotest.Inthesecasesweneedto
createthebestpossibleplanthatallowsustomeetourgoalswithouthiringadditionalstaff.Thismay
resultinverydifferentplansthanifwewereabletohireadditionalstaff.
4.8 Resources
[ACUC]AlistairCockburnWritingEffectiveUsesCasesAddisonWesley

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 90
[BoehmTurner]BarryBoehmandRichardTurner.BalancingAgilityandDisciplineAddisonWesley
[Cockburn]AlistariCockburnMethodologyperproject,Figure9,
http://alistair.cockburn.us/Methodology+per+project,Oct19,2009
[DenneClelandHuang][DenneClelandHuang0]MarkDenneandJaneClelandHuang,"TheIncremental
FundingMethod:DataDrivenSoftwareDevelopment,"IEEESoftware,vol.21,no.3,pp.3947,
May/June2004
[EvansOnDomainDrivenDesign]EricEvans,DomainDrivenDesign,AddisonWesley,2003.

Acceptanc
Cha
Theproce
thetestca
designing
5.1
Theaccep
Assessing
casesthat
compone
Theterm
potential
elicitation
Figure1
Gathering
InFigure1
developm
developm
time,indi
feedback
Somepeo
theybelie
potential
forawide
ceTestEngin
pter5
essofacceptin
asesused.Yo
gyourtestcas
Requirem
ptanceofaso
whetheritm
tareinsome
ntoftheacce
requiremen
usersandask
norrequirem
gRequiremen
1weseethe
mentteam.In
mentteam.In
catedhereby
ononeversio
oplebelievet
everequireme
users'needs.
erangeofact
eeringGuide
5 S
ngasoftware
ourtestcases
sesitisimper
mentsand
oftwareinten
meetsthereq
wayrelated
eptanceproc
ntsissomew
kthemforth
mentsgatherin
ntsdirectlyfro
requirement
effect,there
iterativeand
ythelineswi
onofthepro
hatrequirem
entsmustbe
.Thisguidesu
tivitiesthatm
eVolumeI,
ystem
eintensivesy
aredesigned
rativeyoukno
dAccepta
nsivesystemi
uirementsinv
totherequir
essevenifth
whatcontentio
eirrequireme
ng.Thisisillu
omusers
sfromtheus
equirements
dincremental
thintheprod
ductbeforea
mentscannot
basedonthe
ummarizesth
mayinvolvesp
ReleaseCand
mReq
ystemisinher
dtotestforsp
owtherequir
ance
sclearlyrelat
volvesthepr
ements.Ther
heyarenotdi
ous.Somepe
ents.Frequen
ustratedinFig
sersbeingtra
becomethec
processes,th
duct.Therem
additionalreq
begatheredl
edefinitiono
hisprocessas
pecializedpro
didate1
quirem
rentlydepend
pecificrequire
rementsthat
tedtowhethe
ocessoftesti
refore,requir
rectlyatestin
eoplebelieve
ntly,thisisref
gure1.
nslateddirec
coarsegraine
heproductis
maybeanopp
quirementsar
likestrawber
faproductth
sproductdes
oductdesigns
ments
dentonthesy
ements.There
theyareinten
eritmeetsth
ingthesoftw
rementsarea
ngrelatedart
thatyoumer
ferredtoasr
ctlyintoprodu
edworkitems
builtonereq
portunityforu
reimplement
riesormushr
hatisdesigne
ignwhichact
skills.Thispr
sMod
ystemtesteda
eforebefore
ndedtotest.
herequireme
areusingtest
animportant
tifact.
relyneedtot
requirements
uctbythepro
sfortheprod
quirementat
userstoprov
ted.
rooms;instea
edtomeetth
tsasaplaceh
ocessisillust
91
del
and

nts.
t
alkto

oduct
duct
a
ide
ad,
e
older
trated

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 92
inFigure2.Foraserioustreatmentofthetopicsofproductdesignandexperiencedesignsee[HSPK]as
wellaseffectivelowfidelityproductexperiencedesigntechniquesby[BuxtonOnDesignForUsability].
Figure2
GettingRequirementsviaProductDesign
InFigure2weseetheuserrequirementsbeingaugmentedwithobservationsfromresearchers.These
observationsandrequirementsareinputintotheproductdesignprocessexecutedbytheproduct
designteamwhichmaybethedevelopmentteamoraseparategroupofspecialists.Theresulting
productdesign(oftenexternalizedintheformofprototypes)isthentestedwithpotentialuserstofind
outhowwellitsatisfiestheusersneeds.Thetestresultsarethenincorporatedintotheproductdesign.
Severalrefinementcyclesmaytakeplace.Theactualconstructionoftheproductisbasedontheresults
ofthisfeedback.Initerativeandincrementalprocesses,theproductisbuiltonerequirementatatime,
indicatedherebythelineswithintheproduct.Theremaybeanopportunityforuserstoprovide
feedbackononeversionoftheproductbeforeadditionalrequirementsareimplemented.
Theproductdesignapproachinvolvesexplicitactivitiestovalidatethattherequirementstrulysatisfy
theneedsoftheusers.Thisvalidationcanbedoneaspartofanacceptancetestingphase,butthatis
ratherlatetodiscoverthatwhatyoubuiltisgoingtorequiresignificantchangesbeforeitwillmeetthe
users'needs.Therefore,weadvocateacceptancetestingoftheproposedproductdesign(notthe
softwaredesign)beforethesoftwareisbuilt.Techniquessuchasexperiencesketching
[BuxtonOnDesignForUsability],paperprototypingandWizardofOztesting

Acceptanc
[GouldLew
earlyinth
technique
5.2
Therequi
requireme
functiona
functiona
requireme
illustrated
Figure3
Functiona
Different
thosereq
5.2.1 F
Functiona
jobs.The
design,ca
ceTestEngin
wisOnDesignF
heprojectwh
es,seetheUs
Typesof
rements,how
entsandnon
lrequiremen
litytobepro
entstranscen
dinFigure3.
alvs.nonfunc
techniquesa
uirementsar
Functional
alrequiremen
functionalre
anbeorganize
eeringGuide
ForUsability]
hilethereisst
sabilityTestin
Requirem
weverderived
functionalre
ts,systematt
videdtouser
ndthefunctio

ctionalrequir
reusedford
remet.
Requirem
ntsdescribeh
quirements,w
edandcomm
eVolumeI,
canbeusedt
tilltimetoadj
ngthumbnail
ments
d,aretypicall
equirements
tributesorqu
rsoradminist
onality.There
rements
escribingthe
ments
howvariousty
whetherexpl
municatedan
ReleaseCand
toverifythat
justtheprod
inVolumeIV
lydividedinto
(alsoknowna
ualityattribut
tratorsofthe
elationshipbe
variouskinds
ypesofusers
loredandgat
numberofdiff
didate1
youare"bui
uctdesign.Fo
.
otwobroadc
asparafunct
tes.)Function
esoftwareint
etweenthetw
sofrequirem
sexpectthes
thereddirectl
ferentways,
ldingtherigh
orinformatio
categories,fu
tionalrequire
nalrequireme
tensivesystem
wokindsofre
mentsandfor
ystemtohelp
lyorderived
includingthe
htsystem"ve
onaboutthes
unctional
ements,extra
entsdescribe
m.Nonfunct
equirements
verifyingtha
pthemdothe
fromaprodu
efollowing:
93
ry
se

the
ional
is

t
eir
uct

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 94
Businessprocesses
Usecases
Userstories
Businessrules
Subsystems
Featurelists
Scenarios
Protocolspecifications
Functionalspecifications
Statemodels

Figure4illustratesaveryhighlevelsystemrequirementmodelthatincludesbusinessworkflowtying
togetherseveralusecases.TherequirementsincludetheneedtointeractwithSystem4aspartof
someoftheusecases.ItalsorequiresSystems1,2and3toexchangeinformationcreatedormodified
duringtheworkflow.

Figure4
RequirementsinvolvingUsers,UseCasesandUserStories
Thisguidehighlightsasmallsetofpopularrequirementspractices(seeVolumeII).Theyareusedto
illustratehowtherequirementspracticesandartifactsarerelatedtothetestsyoumayuseduring

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 95
acceptanceofasoftwareintensivesystem.TheUserModelingactivityisusedtocapturesalient
informationabouttheusersofthesoftware.Itcanhelptheproductdevelopmentteamunderstandthe
requirementsbetterbyunderstandinghowtheusersthinkandwhatmotivatestheirbehavior.UseCase
Modelingisusedtocapturewhattheuserswantthesystemtohelpthemachieve.Eachusecase
describesonekindofinteraction(e.g.transaction)betweenauserandthesystem.Ausecasetypically
requiresmanytestcasesbutoftencannotbetestedwithouttestingotherusecases.UserStoriesare
usedtobreakthefunctionalityofthesystemintoverysmallbutstillusefulandtestablechunksthatcan
bebuiltinjustafewdaysorweeks.Theyaretypicallymuchsmallerthanusecasesbutmayspan2or
moreusecases.Theymightdescribeabusinessruleorevenjustonescenarioofanintersystem
communication.Userstoriesmaptooneormoretestcases.
Theserequirementsactivitiesandartifactshelpusunderstandthetypesoftestswemayneedto
executetogatherthedataonwhichtheacceptancedecisionismade.

5.2.2 NonfunctionaRequirements
Nonfunctionalrequirementsarerequirementsthatdescribegeneralqualitiesorbehaviorsofthe
systemthatspanspecificusagescenarios.Frequently,theserequirementsaretracedbacktoprotection
ofvariousstakeholderswhomayormaynotbeusersofthesysteminquestion.
Unlikefunctionalrequirements,whichvarygreatlyfromsystemtosystem,thereisafairlystandardlist
oftypesofnonfunctionalrequirements.Manyoftheserequirementsendwiththesuffix"ility"because
theydescribecharacteristicsthewholesystemneedstosupporteitherwhileitisinuseoraspartofits
overallsystemlifecycle.
Requirementsthatthesystemneedstosatisfywhileitisoperatinginclude:
AvailabilityWhendoesthesystemneedtobeavailableforuse?
DataIntegrityThedataneedstobestoredandprocessedinawaythatitcanbereliably
retrievedwithoutchangingitsoriginalvalue.
SafetyThesystemshouldnotcausephysicaloremotionalharmtoauserorstakeholder.
RecoverabilityWhenitfails,thesystemshouldrestoreitspreviousstatewithoutundue
hardshiptotheusersorsupportersofplatform.
AccessibilityPeoplewithdiverselimitationsshouldbeabletousethesystem.Itmayneedto
conformtostandardslikethosestatedbytheADAorotherregulatoryorstandardsbodies.
SupportabilityItshouldbeeconomicaltoprovidesupporttousersoftheproduct.
ReliabilityItshouldworkwellandresistfailureinallsituations.
RobustnessItshouldtakeabuseandstillfunction(faulttolerance).

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 96
PerformanceWhatspeedorthroughputbenchmarksdoesitneedtomeet?Performance
includesresponsetime:thesystemshouldrespondtouseractionsandrequestsinatimely
manner.
UsabilityItshouldbeeasytousebyitsintendedusers.
InstallabilityTheproductshouldbeeasytoinstallontoatargetplatform.Thisissometimes
partofusability.
SecurityTheproductshouldprotectagainstunauthorizeduseorintrusion.
ScalabilityTheproductshouldbecapableofbeingscaledupordowntoaccommodatemore
usersortransactions.
CompatibilityWithwhatexternalcomponentsandconfigurationsdoesitneedtowork?
Requirementsrelatedtothetotalcostofownershiporproductlifecycleinclude:
TestabilityInwhatwaysshouldtheproductbetestable?
MaintainabilityHoweasyshoulditbetoevolve,fix,orenhancetheproduct?
PortabilityHoweasyshoulditbetoportorreusethetechnologyelsewhere?
LocalizabilityHoweconomicalwillitbetopublishtheproductinanotherlanguage?
ReusabilityHoweasilycansourcecode(e.g.classes,functions,etc.)beusedinother
circumstances?
ExtensibilityHoweconomicalshoulditbetoenhancetheproductwithotherfeaturesand
addons?
ConfigurabilityHoweasyshoulditbetopreparetheproducttobeusedonavarietyof
differentplatformsorforslightlydifferentusesandoperations?
Theprecedinglistoftypesofnonfunctionalrequirementsisnotintendedtobeexhaustive.Italsoisnot
intendedtobeuniversal;someoftheserequirementsmaybeirrelevantforsomesoftwareintensive
systems.Partoftheartofrequirementsgatheringorengineeringisdecidingwhichonesareimportant
andwhichonesarenot.Thosethatareimportantneedtobemadeexplicitandincorporatedintothe
testplans;thosethatarenotimportantcanprobablybeignored(buttheriskofignoringthemshouldbe
assessedbeforedoingso.)
Mostofthenonfunctionalrequirementscutacrosstheusecasesofthesystem.Thatis,theyapplyto
many,ifnotall,ofthediscretechunksoffunctionalitydescribedinthefunctionalrequirements.Note
thatsomeformsofnonfunctionalrequirementscanbedescribedatleastpartiallyinfunctionalterms;
securityisagoodexample.YoucansaythatUserRoleXshouldbepreventedfromchangingthevalueof
fieldFonscreenS.
Thekeytotestingconformancewithnonfunctionalrequirementsistheclassificationofeachofthese
requirementsindicatingtowhatdegreetheprojectstakeholderscareabouttherequirement.For
example,foranapplicationonebuildsforonespersonaluse,onemaynotcareaboutscalability

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 97
becausetherewillbeonlyoneuser,butonalargeecommerceapplication,scalabilityisveryimportant.
Itisworthreviewingthislistofnonfunctionalrequirementsandconsciouslydecidinghowimportant
eachoneistothesuccessofyourproductorproject.Thefollowingtableliststhegoals,importance,and
rationaleofavarietyofnonfunctionalattributesforaspecifichypotheticalsystem.
Attribute Goal Importance Rationale
Web site performance under
load
Less than 500 milliseconds
response time
High Major source of
revenue
Web server capacity under
load
At least 300 transactions per
second.
Graceful degradation under
load
Medium Large number of users
Reliability/availability 7x24x52 Critical Users require instant
satisfaction when
worried about their
money
Usability Easily discoverable High Most users will use
infrequently

AppendixXinVolumeIVTestingActivities&Responsibilitiesisanexampleofachecklistthatcanbe
usedtorecordthesedecisions.
5.3 Summary
Inordertobeabletodoagoodjobofreadinessassessmentandacceptancetesting,thetargets
(requirements)mustbeclear.Readinessassessmentandacceptancetestingaredrivenbythequestion
"howwillweknowwearedone?"Assuch,theyshouldideallystartearlyintheproject.
Thereadinessassessmentandacceptancetestingactivitiesgatherdataabouthowwellthesystem
satisfiestherequirements.Therequirementsincludebothfunctionalandnonfunctionalrequirements.
Thereareanumberofalternativewaystodecomposetherequirementsintomanageablechunks.Some
formsofrequirementsmapmoredirectlytotestcasesthanothers.
5.4 WhatsNext?
Thingscananddogowrongwhenbuildingsoftwareintensivesystems.Commonlyoccurringproblems
includemisunderstoodrequirementsandfailuretomeetnonfunctionalrequirements.Thenextchapter
describestechniquesforidentifyingwhatmightgowrongandplanningreadinessassessmentand
acceptancetestingactivitiesthatwoulddiscoverthemasearlyaspossible.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 98
5.5 References
[HSPK]HendrikSchifferstein,PaulHekkert(Eds.)ProductExperience.Elsevier,2008.
[BuxtonOnDesignForUsability]BillBuxton.SketchingUserExperiences:GettingtheDesignRightandthe
RightDesign.MogranKauffman,2007.
[GouldLewisonDesignForUsability]Gould,J.D.andC.Lewis,DesigningforUsabilityKeyPrinciplesand
WhatDesignersThink,inProc.ofACMCHI83ConferenceonHumanFactorsinComputingSystems.p.
5053,1983.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 99
Chapter6 RiskModel
Everyprojectinherentriskswhetherornotwerecognizethem.Determiningwhatthoserisksarecanbe
achallenge.Projectrisksofproducingsoftwareintensivesystemsrelatetowhethertheproduct
developmentteamunderstandswhattheProductOwnerwantsbuiltandhowtobuilditandinabilityto
constructanddelploytheproductasintended,inatimelymanner,andcosteffectively.Opeationalrisks
relatetotheproductnotfunctioningasexceptedorfailingduringoperation.Ariskmodelcanhelpus
understandalltherisksandcomeupwithriskmitigationplanstoreducetheirlikelihoodorimpact.
6.1 WhatCouldPossiblyGoWrong?RiskAssessment
Onewaytodefineriskisbyaskingwhatkeepsusawakeatnight?Morespecifically,whatmighthappen
andwhatwouldbetheconsequencesifitdidhappen?
Wecanmakethediscussionofriskmoremeaningfulbytranslatingnebulousconcernsintoconcrete
eventsthatcouldhappenandtalkingaboutthelikelihoodthatitmighthappenandtheconsequencesif
itdoeshappen.
Forexample,supposeweorderedsomecriticalhardwarewhichwerequiretoconductcertaintypesof
acceptancetestingwithoutwhichwearenotpreparedtomaketheacceptancedecision.Whatcould
possiblygowrong?Thefollowingaresomeexamples:
Thehardwarecouldbedestroyedintransit.
Thewronghardwarecouldbeshippedeitherthroughanorderingerrororafulfillmenterror.
Thehardwarecouldbedefective.

Foreachoftheprecedingeventswecanestimatethelikelihoodthatitwilloccurandassesstheimpact
onourprojectifitdidoccur.Performingthesetwocalculationsseparatelyhelpsustobetterunderstand
theriskasshownintheriskmatrixinFigure1.Astheprobabilityofaneventincreases,therisk
increases.Astheconsequencesofaneventincreases,theriskincreases.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 100

P
r
o
b
a
b
i
l
i
t
y

L
o
w

M
e
d
i
u
m

H
i
g
h

Low Medium High
Consequence
Figure1
RiskMatrix
InFigure1RiskMatrix,thetwoareasoneithersideofthediagonalandthediagonalitselfrepresent
threedegreesofrisk.Thebottomleft(green)riskregimerepresentslowrisk,thetopright(red)risk
regimerepresentshighrisk,andthetoplefttobottomrightdiagonal(yellow)riskregimerepresents
moderaterisk.Ingeneral,risksthatfallinthesameriskregimeareequallyimportanttomitigate.
6.2 UnquantifiableRisks
Itisimportanttorecognizethatitisnottheknownandquantifaiablerisksthatcausethebiggest
problems,butthecompletelyunexpectedthingsthatdoso.Inhistreatmentoftheissue,PeteMcBreen
emphasizesthatflexibilityisthekeytosurvivingtheunexpected[McBreenOnRiskManagement].
6.3 WhoShouldIdentifytheRisks
Everyoneontheprojecthasadifferentandvaluableperspectiveonthepossiblerisksinvolvedin
buildinganddeliveringaproductwhetheritisforsaletocustomersorforusebyinternalusers.Risk
identificationshouldnotbeamonopolyoftheprojectmanagersbutshouldincludeproductdesigners,
productdevelopers,testersandallstakeholdersincludingtheProductOwner.Ultimately,itisthe
ProductOwnerwhoneedstounderstandtherisksanddecidewhichonestheycanacceptasisand
whichonestheywanttospendmoneytoreducethereforetheProductOwnermustbeintimately
familiarwithatleasttherisksinthered(topright)andyellow(diagonal)regionsoftheriskmatrix.
Ultimately,asTomGilbstatedinhisAskingPrinciple:Ifyoudontaskforriskinformation,youare
askingfortrouble[GilbOnSEManagement].

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 101
6.4 ShouldWeDoSomethingAboutIt?RiskManagement
Afterweunderstandtherisksofourproject,whatcanwedoaboutthem?Therearethreepossible
coursesofaction:
Wecanaccepttheprobabilityandconsequencethataparticulareventmighthappen.
Wecanperformactivitiestoreducethelikelihoodofithappening.
Wecanperformactivitiestoreducetheconsequenceifitdoeshappen.

Thecoursewechoosedependsonanumberoffactors,includingthefollowing:
Thefactorswehavecontrolover,suchasthefollowing:
Iftherearenocoursesofactionthatcouldreducethelikelihoodofsomething
happeningwemaybeforcedtofocusontryingtoreducetheconsequence.For
example,theonlywaytoavoidanextremeweathereventmightbetomovetoa
differentarea,whichmaysimplyexchangeonesetofextremeweathereventsfora
differentset.Butwecanpreparefortheoccuranceoftheextremeweatherbychoosing
appropriatefacilitiesorotherpreparations.
Ifthereisnowaytoreducetheconsequenceofanevent,weneedtofocusonreducing
thelikelihoodofitoccurring.Forexample,mostpeoplewouldagreethatitisbetter
(andeconomicallymorefeasible)totrytoreducethelikelihoodofaheartattackby
exercisingandeatingwellthantotrytoimprovetheprobabilityofsurvivingitbyhiring
aheartspecialisttobeatyoursideatalltimes.
Therelativecostoftheoptionsavailabletouse.Ifitismuchcheapertoreducethelikelihood
thantheconsequence,weshouldfirstfocusonloweringthelikelihoodandviceversa.Notethat
thecostistypicallynonlinearandgetsmoreexpensivetheclosertozerowetrytodrivethe
likelihoodorconsequence.Thereforeitmaybemoreappropriatetotrytopartiallyreduceboth
likelihoodandconsequenceratherthanreducejustonetoalargerdegree.
Thecostofriskreductionrelativetothecostwewouldincuriftheriskoccurred.Forexample,if
aparkingticketcoststwiceasmuchaspayingfortheparkingandthereisonlya20percent
chanceofgettingcaughtandticketed,wemaychoosetotakethechancebynotpayingfor
parking.

6.5 HowCanTestingHelp?RiskMitigationStrategies
Ifwedecidetomitigatearisk,howwegoaboutitdependsonthenatureoftherisk.Risksthatrelateto
thepossibilityofdeliveringadefectiveproductareamenabletoriskmitigationthroughsomeformof
testing.Risksthatrelatetodiscoveringsomethingtoolateforatimelyfixcanbemitigatedbyactivities
thatmovethediscoveryearlierintheproject.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 102
6.5.1 DoingSomethingEarlier
Manyrisksonprojectsarerelatedtotime.Willsomethinghappenintime?Ifithappenstoolate,willwe
havetimetoreactwithoutaffectingtheprojecttimeline?
Agoodexampleofthisisthelatediscoveryofmissedormisunderstoodrequirements.Whenthis
discoveryoccursduringtheacceptancetestingphaseofaprojectshortlybeforetheproductisexpected
tobeturnedovertousers,theimpact(ofthediscovery)maybeasignificantdelayinachievingthe
businessbenefitsexpectedfromthesystem.Inthiscase,wecanreducetheimpactofthediscoveryby
doingtheacceptancetestingactivitiesearlierintheproject.
Theincrementalacceptancetestingpracticeusedonmanyagileprojectsisonewaytomovediscovery
ofproblemssuchasmisunderstoodrequirementsearlierintheprojectsothereisplentyoftimeto
addressthem.Sequentialprojectscanalsoreapthebenefitsofincrementalacceptancetestingby
movingtoanincrementaldeliverymodelwheretheproductisbuiltinfunctionalmodulesthatcanbe
acceptancetestedastheybecomeavailable.
6.5.2 DoingSomethingDifferent
Anextremeformof"toolate"discoveryiswhenwedonotdiscoveritatallandaproblemoccurswhen
theproductisbeingused.Iftheproblemissevereenoughtohaveseriousrepercussions,the
consequencescanbedisastrous.Thehighprofilelossesortheftofcustomers'privateinformationisjust
oneexampleofsomethingdiscovered"toolate."Thesetypesofriskmayrequireadditionalactivitiesto
reducethelikelihoodoftheiroccurrence.Thesolutionoftenliesindoingadditionalkindsoftestingto
improvethelikelihoodthatacertainclassofdefect,ifitexists,isfoundintime.Manytestauthoring
practicesarefocusedonwaystodefineadditionalteststhatimprovethetestcoverage(fromarisk
coverageinsteadofacodecoverageperspective).
6.6 Summary
Ariskmanagementmodelandawaytotrackrisksandriskmitigationisimportantonalltypesof
projects.Makingrisksexplicitallowsustodevisewaystochoosetherightriskstotakeandtomitigate
theriskseitherbyreducingtheprobabilityortheconsequence.Forrisksrelatedtomisunderstandingof
therequirements,earliertestingthroughincrementalacceptancecanreducetheconsequencebygiving
usmoretimetoreworkanyissues.Prioritizingacceptancetestingactivitiesshouldbepartoftherisk
modeltoensurethathighlycriticalfeaturesarethoroughlyaddressedduringsoftwareacceptance.The
ProductOwnerneedstobeinvolvedinidentifyingandunderstandingrisksandindeterminingwhich
risksneedtobeaddressed.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 103
6.7 WhatsNext?
Asignificantriskonmostprojectsislatediscoverythattheproductismuchfartherfromdonethan
thought.Thenextchapterintroducesamodelforthinkingaboutthedegreetowhichtheproductis
done.
6.8 Reference
[McBreenOnRiskManagement]PeteMcBreen,TheMythofRiskManagement,BetterSoftware
Magazine:3034,June,2008.
[GilbOnSEManagement]TomGilb,PrinciplesofSoftwareEngineeringManagement,AddisonWesley,
1988.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 104
Chapter7 DonenessModel
Determiningwhethertheproductisdoneistheultimategoalofsoftwareacceptance.Knowinghow
closetodoneweare,isakeyfunctionofprojectmanagementdiscipline.Thissectionintroducesamodel
fordetermininghowcompleteaproductisrelativetoitsgoalsfrombothfunctionalandnonfunctional
perspectives.

Akeypartofmakingtheacceptancedecisionrevolvesarounddecidingwhethertheproductisdone.
Comingtoconsensusondonenessrequiresagreementonwhatisbeingassessedaquestionof
targetgranularityandwhatitisbeingassessedagainstthedonenesscriteria.Initsmostbasicform,
theacceptanceprocessdefineshowwedecidewhetheraparticularreleaseoffunctionalityisdone.In
itsmoreadvancedformswecandeterminedonenessatthelevelofindividualfeaturesorworkitems.
Eachofthesedifferenttargetshasasetofdonenesscriteriathatmustbeagreedupon.Thefollowing
aresomeexamplequestionstoconsider:
Isasoftwareintensivesystem(forexample,asoftwareproduct)readyforanalphatestwitha
friendlyusercommunity?
Isanindividualfeatureoruserstoryreadyforacceptancetestingbyabusinesstester?
Isasoftwareintensivesystemreadyforthedesignclosemilestone?

Thedefinitionof"done"isdifferentforeachoftheseexamples.
7.1 ReleaseCriteriaDonenessofEntireProduct
Whendecidingtheacceptanceofanentiresoftwareintensivesystemforreleasetousers,donenessisa
binarystate.EithertheproductisacceptedbytheProductOwnerandripeforreleasetoitsintended
audienceorfortheusagedecision,oritsnot.Ifitsnot,thenwecantalkabouthowfartheproductis
fromthestateofbeingdonefortheunderlyingdecision.Thesameprinciplesapplytothereadiness
decisionthatprecedestheacceptancedecision:theoneaboutwhethertoreleasetheproducttothe
ProductOwnerfortheacceptancedecision.Thustherearetwomaincriteriafordeterminingwhethera
productisdone:
Isthereenoughhighvalue,ProductOwnerdefinedfeaturesincludedtomaketherelease
worthwhile?
Isthequalityofthefeatureimplementationshighenoughforthefeaturestobeusableintheir
expectedusagecontext?

Thefirstcriterion,knowninthisguideasMinimumMarketableFnctionality(MMF),istypically
determinedwhileplanningtherelease.Thecriterionmayberevisitedastheprojectisbeingexecuted

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 105
andmoreislearnedaboutthesystemcontext(suchasbusinessrequirements)andthetechnical
capabilitiesoftheProductDevelopmentTeam.
Whenreviewingacceptancetestresultsforeachfeature,itisfairlysimpletodeterminethepercentage
offeaturesthatisdone.ThisisthenumberoffeaturesthattheProductOwnerhasdecidedpasstheir
criticalacceptancetestsdividedbythetotalnumberoffeaturesfortherelease.Itisalsopossibleto
expressthisratioinstorypointstoaccountforfeaturesofdifferentsizes.
Thesecondcriterion,knowninthisguidanceasMinimumQualityRequirement(MQR),iswhatwe
constantlytestagainstwhilebuildingandassessingthesoftware.Tobeabletosaywhetherafeature
hasmettheMQR,weneedtohavetheacceptancetestsdefinedforthatfeature;inthisguidance,thisis
theperfeaturedefinitionof"Whatdonelookslike."Figure1illustratesgraphicallythedegreeof
donenessusingthesetwocriteria.Eachchartrepresentingthedonenessononeoftwoseparatedates
inthesameproject.

Figure1
DonenessrelativetoMMFandMQRattwopointsintime
InFigure1,thechartontheleftshowsthecompletenessofeachfeatureatpointXintime;thecharton
therightshowsthecompletenesstwoweekslater.Eachcolumnrepresentsafeaturewiththewidthof
eachcolumnindicatingtheestimatedefforttobuildthefeature.ThelinelabeledRAT(Readyfor
Acceptance)marksthelevelatwhichthefeatureisdeemedreadyforacceptancetestingbyvirtueof
havingconductedthereadinessassessment.Itistheperfeatureequivalentofthereadinessdecision
thatismadeatthesystemlevel.ThespacebetweentheRATlineandthelinelabeledMQRiswhen
acceptancetestingisdone.ThelinelabeledMMFisthedemarcationbetweenthefeaturesthatmustbe
present(leftoftheline)andthosethatareoptional(rightoftheline;omittedinthesediagrams)forthis
release.
WecanseeprogressovertimebycomparingtheleftandrightchartsinFigure1.Features1through5
(numberingfromtheleftstartingfrom1)werealreadycompletedatthepointintimerepresentedby
theleftchart.Features5through7werealreadystartedattimexandcompleted(deemeddone)in
theperiodendingattimex+2weeks.Features8through10werealreadyinprogressattimexand
werenotcompletedattimex+2weeks.Features11and12werestartedafterthedatedepictedbythe
leftchartandnotfinishedatthetimedepictedbytherightchart.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 106
Theproductisdeemedacceptablewhenallfeaturespassalltheiracceptancetests.Thisistheupper
rightcornerofthechartwherethelineslabeledMQRandMMFintersect.Whentherectangletothe
lowerleftofthispointisentirelycoloredin,theproductisaccepted.Tosimplifythediscussion,thenon
functionalityrequirementsaredeliberatelyignored,buteachsetofnonfunctionaltestscouldbe
treatedasanother"featurebar"fromtheperspectiveofmeasuringdoneness.
7.1.1 GoodEnoughSoftware
ItisveryeasytosayItallhastobethere!whenaskedwhatfunctionalityismustbepresenttosatisfy
theMMF.Likewise,itiseasytoreplyTherecanbenobugs!whenaskedwhatMQRneedstobe.But
theseanswersaretakingtheeasywayout.Theproductownerhastomakeaconsciousdecisionabout
thevalueofqualityandfeatures.Thedogmaticanswer(allthefeaturesandabsolutelynobugs)may
delaydeliveryoftheproductbymonthsorevenyears.Meanwhileacompetitorsproductmaybe
earningmoneyandmarketsharebecausethatcompetitorhadamorerealisticgoalforMMFandMQR.
ThechoiceofMMFandMQRshouldbeaconsciousbusinessdecisionmadeinfullknowledgeofthecost
ofincreasedqualityandfunctionalityintermsofactualcosts(additionaldevelopment,testing)and
opportunitycosts(delayedincomeandotherproducts).
ForadditionaltreatmentofthetopicofGoodEnoughSoftware,see[BachOnGoodEnoughSoftware]and
[YourdonOnGoodEnoughSoftware]
7.2 DefiningWhatDoneMeans
Foreachchunkoffunctionalitywehavedecidedtodelivera"feature"thatneedstobedefinedforthe
minimumqualityrequirement(MQR)intheformofasetofacceptanceteststhatmustpassbeforethe
ProductOwnerwillacceptthefeature.Thesetofacceptancetestsforareleaseismerelytheaggregate
oftheacceptancetestsforallthefeatures("functionaltests")plustheacceptancetestsforeachofthe
nonfunctionalrequirements(the"nonfunctionaltests")thataredeemedmandatory.
7.2.1 Determining"Readiness"
"Readiness"iswhentheProductDevelopmentTeambelievestheproductis"doneenough"toaskthe
ProductOwnertoconsideracceptingtheproduct.ThisimpliesthattheProductDevelopmentTeamhas
areasonablyaccurateunderstandingofhowtheProductOwnerwillconducttheacceptancetesting.(In
somecases,theProductDevelopmentTeam's"readinesstests"maybemuchmorestringentthanthe
acceptanceteststheProductOwnerwillrun.)Thisunderstandingisknownasthe"acceptancecriteria"
andisusuallycapturedintheformofacceptancetests.Ideally,theacceptancetestsarejointly
developedandagreeduponbetweentheProductDevelopmentTeamandtheProductOwnerbefore
thesoftwareisbuilttoavoidplaying"battleship"or"blindman'sbluff"andtheconsequentrework
whentheProductDevelopmentTeamguesseswrong.Itisalsoidealforreadinesstestsandtestresults
tobesuppliedtotheProductOwnerbytheProductDevelopmentTeam.Thiscanassistinauditingthe
readinesstesting,streamliningacceptancetesting,andperformingamoreinformedstyleoftesting,
suchasexploratoryacceptancetesting.Thiskindoftestingcanbethoughtofablackhattesting
referringtoEdwarddeBonosoneofsixviewpointsforthinkingaboutaproblem

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 107
[DeBonoOnSixThinkingHats].Ablackhatapproachlooksataproblemfromacautiousandprotective
perspective,concentratingonwhatcangowrong.
7.2.2 DonenessofIndividualFeatures
Foranindividualfeaturewecandescribethedegreetowhichitisdoneinseveralways.Oneishowfar
alongitisinthesoftwaredevelopmentlifecycle.Forexample,Thedesignisdoneandcodingis80%
finished.Weexpectthefeaturetobereadyfortestingintwoweeksaspartofthetestingphase.This
formofprogressreportingisoftenusedinsequentialprojectsbecausethedesignandcodingcantake
manyweeksorevenmonths.Unfortunately,itisnotaveryusefulbecauseofitssubjectivity,low
transparency,andcoarsegranularity.Inprojectsthatadoptatoocoarselevelofschedulingand
clumpingwork,codingmayremain80%finishedformanyweeksormonthsevenifprogressisbeing
made.Itcanthereforebeverydifficulttotellwhenprogressisnotbeingmade.
Oncethecodinganddebuggingisfinished,itismoreusefultodefinedonenessintermsofwhichtest
casesarepassingandwhicharenot.Thisalternativewayofmeasuringprogressisobjective,
transparent,andfinegrained.Inthesequentialapproach,thepercentageoftestcasespassingstaysat
zeroformostofthelifecycleandthenrapidlyrisestosomewherenear100%.Initerativeand
incrementalprojects,featuresaresometimesimplementedonetestcaseatatimetherebyallowingthe
percentageoftestcasespassingtobeusedasanobjectiveprogressmetricatthefeaturelevelinstead
ofattheproductlevel.
Thenumberoftestcasesforaparticularfeaturemaychangeovertimeasthefeatureisunderstood
better.Thismayalsobeasignofworkcreep(moreefforttoimplementduetomorecomplexity)or
scopecreep(additionalfunctionalitythatwasntoriginallyintended).Theadditionaltestsmayresult
fromincreasedclaritycausedbyattemptingtowritetestsoritmaycomeasaresultoffeedbackfrom
otherformsoftestingincludingusabilitytesting,exploratorytestingoracceptancetestingofanAlpha
orBetarelease.Regardlessofthesourceoftheadditionaltests,thepercentdonemaytakeastep
backwardsifthenumberoftestsincreasesmorethanthenumberofadditionaltestspassing.
7.2.3 DonenessofNonfunctionalAttributes
Nonfunctionalattributesofthesystemareabitdifferentfromindividualfeaturesinthattheytendto
applyacrossallfeatures;thisiswhysomepeopleprefertocallthemparafunctional.Nonfunctional
attributes,whenquantifiedandobjectivelymeasurable,maybebinaryinthattheproducteither
satisfiestherequirementornot.Anexampleisscalability:eithertheproductcansupportamillion
transactionspersecondornot.Wemight,however,beabletobreakdownthefunctionalityofthe
systemintofunctionalareasanddeterminetheusabilityofeachareaindependently.Forexample,we
mayrequireahighdegreeofusabilityforcustomerselfservicefunctionswhileadministrativefunctions
canhaveamuchlowereaseofuse.Othernonfunctionalrequirementsarequantifiable.Forexample,
howmanyuserscanwesupportbeforeresponsetimeexceedsourresponsetimethreshold?Itcouldbe
1user,10users,or1000users.
Wecanchoosewhetherwewanttovisualizeprogressonnonfunctionalcompletenessaseither
additionalfeatures(support10users,support100users,support1000users)asillustratedinFigureZor

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 108
asasinglefeature(supportmanysimultaneoususers)withaminimumqualityrequirement(e.g.1000
users)asillustratedinFigureY.Ourchoicemaybeinfluencedbyhowweorganizetheworktodevelop
thefunctionality.Agileprojectsoftenoptforthefeatureperthresholdapproachbecausesupportfor
additionaluserscanberolledoutoveraseriesofiterations.Sequentialprojectsmayprefertotreat
themasdifferentlevelsofqualityforasinglefeatureasillustratedinFigureYorasadifferentlevelof
qualityacrossallfeaturesasillustratedinFigureX.(SeethediscussionofvaryingMMFforAlpha/Beta
releasesinsectionTheAcceptanceProcessforAlpha&BetaReleasesinChapter1foracounter
argument.)

FigureZ
IncrementalNonfunctionalFeatures
InFigureZtherequirementforsupportingmultipleusersisdividedintotwofeatures.Thefeatureto
supportatleast10usersisrepresentedbycolumn6(withhighlightedborder)andisreadyfor
acceptancetestingwhilethefeaturetosupport100users,representedbycolumn14(withhighlighted
border),hasnotyetbeenstarted.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 109
FigureY
SingleNonfunctionalityFeature
InFigureYtherequirementforsupportingmultipleusersistreatedasasinglefeaturerepresentedby
column10.Itcurrentlysupportsatleast10usersbutisnotreadyforacceptancetestingatthefull
requirementof100users.

FigureX
BigBangNonfunctionality
InFigureXverificationoftherequirementforsupportingmultipleusersistreatedasanadditionallevel
ofqualitylabeledNFacrossallfeatures.Ithasnotyetbeenstarted.
7.3 Communicating"PercentDoneness"
Itcanbesaidyouareeither"done"or"notdone(yet)."Butinpractice,itisimportanttobeableto
clearlycommunicate"howclosetodone"youare.Morespecifically,itisimportanttobeableto
communicate"whatremainstodobeforewecansayweare'done'".Thisistheamountofworkleftfor
eachfeaturethathasnotyetpassedallitsacceptancetestssummedoverallthefeaturesthatarepart
oftheMMF.WhenlookingateitherdiagraminFigure1,wecanask"Whatpercentageoftherectangle
below/leftofMQR/MMFiscoloredin?"Thisgivesusasenseofhowmuchworkisremaining.Alotof
whiteimpliesalotofwork;verylittlewhitemeanswearenearlydone.Wherethewhiteislocatedtells
uswhatkindofworkislefttodo.Whiteacrossthetopmeanswehavelotsofpartiallyfinishedfeatures;
whiteprimarilyontherightimpliesthatentirefeatureshaventbeenbuiltyetbutmostfeaturesare
eithernotstartedorfullyfinished.Thesepatternsofprogressiondependprimarilyontheproject
managementmethodologyweareusing.ThediagraminFigure2showssnapshotsofcompletenessfor
threedifferentprojectstyles:

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 110

Figure2
Completenesscharts
InFigure2,thefirstrowofchartsrepresentsaclassicsequentialorphasedordocumentdrivenstyleof
projectmanagementwithwhiteacrossthetopindicatingthatallfeaturesareatsimilarstagesof
completion.ThebottomrowrepresentsaclassicExtremeProgrammingprojectwithwhiteprimarilyon
therightindicatingthatthereareveryfewfeaturespartiallydonesincemostareeitherdoneandnot
started.Themiddlerowrepresentsaprojectusinganincrementalstyleofdevelopmentwithlonger
featurecyclesthantheExtremeProgrammingproject.Noticethedifferenceinhowthecoloringinof
thechartadvancestowardtheupperrightcornerwheretheprojectwouldbeconsideredfullydone
basedon100%ofMMFand100%ofMQR.Thenextfewsectionsdelvemoredeeplyintohowwe
calculateandcommunicatedonenessonthesedifferentstylesofprojects.
7.3.1 CommunicatingPercentDoneonAgileProjects
Anagileprojectcansimplydividethenumberoffeaturesthatareacceptedbytheproductowneras
donebythetotalnumberoffeaturesscheduledfortherelease.Thisprovidesthepercentageof
featuresthataredone.Wecanexpressthismeasureinmonetarytermsbyweightingeachfeatures
percentcompletenessbythefeaturesestimatedcost.Theestimatedcostisrepresentedbythewidth
ofeachfeaturecolumnintheprogresschartsofFigure2.Howeverthemonetaryequivalentofafeature
calculatedthiswayisnotthesameasthefeaturesbusinessvalue.Itsessentiallyanexpressionofthe
featurescompletenessincostterms.Themonetaryequivalentiswhatisreferredtoasearnedvaluein
EarnedValueManagement(EVM).SeetheSidebartitledEarnedValueManagementandAgileProjects.
Sidebar:EarnedValueManagementandAgileProjects
EarnedValueManagement(EVM)isabudgetandscheduletrackingtechniquedevelopedbythe
ProjectManagementInstitute.Earnedvaluetrackstherelativeprogressofaworkunitasa
percentageoftheallocatedbudgetthathasbeenspentforthatworkunit.Ifaworkunitis80%
completeandwasallocateda$100budget,thenitscurrentearnedvalueis$80.Thetotalvalue
earnedbytheprojectisthesumoftheearnedvaluesofalltheworkunitsscheduledandbudgeted

Acceptanc
forthe
Theref
InEVM
exceed
shortfa
valuea
budge
expres
Schedu
Figure
projec
2010,a
planne
itsove
accum
behind
of$5.5
Figure
Calcula
Anapp
[CabriG
(iterat
meani
during
scope
ceTestEngin
eproject.Wh
fore,unliketh
M,actualcost
dstheearned
allrepresenti
atthatpoint
t,thenthepr
ssionofthep
ulevariancec
Willustrates
twithanesti
atanearned
edtotalcost,
erscheduleb
mulatedthatm
d.Theprojec
5millionatth
W
ationofsched
proachtopor
GriffithsOnAg
ionbyiteratio
ngfulforprog
gthecourseo
(featuresors
eeringGuide
entheprojec
henamesugg
sarealsotra
dvalueofthe
ngthebudge
intimeisbelo
rojectisconsi
rojectssched
canalsobeca
sschedulevar
imatedtotalb
valueof$2.5
butitshould
by$1.75millio
muchearned
ctisalso120%
hatdatecomp
duleandcost
rtingEVMtoa
gileEVM].Grif
onorrelease
gresstracking
oftheproject
stories)ande
eVolumeI,
ctiscomplete
gests,earned
cked.Ifatan
projectatth
etoverrun(in
owtheestim
ideredovers
duleshortfall
alculatedinca
riance(inbot
budgetof$5
5million,the
haveaccumu
onor50%in
valuebackin
%overbudge
paredtothe$
tvarianceinE
anagileconte
ffithsandCab
ebyrelease).
gpurposessin
.Howeverm
estimatedcos
ReleaseCand
e,itstotalear
valueisesse
ypointinthe
hattime,thep
EVMterms,
atedtotalex
schedulewith
(inEVMterm
alendartime
thmonetarya
millionanda
projecthasa
ulated75%of
monetaryter
nApril2010,i
etonJuly201
$2.5millionit
EVM.
extisdiscusse
briadvocatea
Inanagilepr
ncethegrand
miniplansinte
sts(resources
didate1
rnedvalueeq
ntiallyaneffo
eproject,the
projectiscon
costvariance
pendituresac
hthedifferen
ms,schedule
.
andtemporal
anestimated
totalearned
fitaccording
rms.Sincethe
ncalendartim
0becauseith
thasearned.
edbyGriffith
applyingEVM
roject,agrand
dplanwould
ermsofwork
sallocatedto
qualstheproj
orttrackingm
actualaccum
nsideredover
e).Iftheaccu
ccordingtoth
cerepresenti
variance)att
lterms)andc
scheduleof1
valuethatis
totheprojec
eprojectsho
meterms,iti
hasaccumula

hsandCabri
Monanincrem
dplanforthe
subjecttoco
kscheduledin
thescope)a
jectstotalbu
metric.
mulatedcost
budget,with
mulatedearn
heplanned
ingthemone
thatpoint.
costvariance
1year.InJuly
only50%of
ctplan.Ther
uldhave
is3months
atedanactua
mentalbasis
eprojectisno
nstantchang
nthecurrent
reusually
111
udget.
hthe
ned
etary
fora
y
its
efore
lcost
ot
ge

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 112
availableandreliableenoughtoserveasareferencepointonaperiterationorperreleasebasis.
Budgetandscheduleoverrunscanthenbecalculatedwithineachincrement,whetherforasingle
iterationorrelease,inthesamewayasinEVM.Effortoraproxyforeffortsuchasstory/feature
pointscanbesubstitutedforcost.
Inagileprojects,customerorbusinessvalueismoreimportantthanearnedvalue.Thereforeearned
valueinitselfmaynotbeverymeaningful.Thecustomervalueorbusinessvalueequivalentof
earnedvaluecanalsobeeasilytrackedprovidedthateachfeaturehasavalueorutilityassignedby
theproductowner.Thebusinessvalueorcustomervaluecanbeexpressedinabsolute,monetary
terms(incurrency)orinrelative,utilityterms(inartificialutils).Thebusinessvalueisanestimate
basedontheproductownersassessment.Thereforeitissubjective,butitcanstillbeinformedand
backedupbyseriouseconomicanalysis.Actualbusinessvalueisstilloftenhardtomeasure,
especiallyatthefeaturelevel.Unlikeinplainearnedvaluecalculation,onlydeliveredfeaturesshould
earntheirrespectivebusinessvalue:eachfeatureearnsitsfullestimatedbusinessvalueonce
delivered.However,ifafeatureisonlyusefulinthefieldwhendeliveredtogetherwithanother
feature,itshouldntearnbusinessvalueuntilallthefeaturesonwhichitdependsarealsodelivered.
Percentweightingstoadjustforafeaturesstatusofcompletenessdontmakemuchsensefor
trackingbusinessvalue.Consequently,individualfeaturesdontearnpartialbusinessvalue.Partial
orpercentagebusinessvalueearnedisonlymeaningfulattheprojectlevel.

ThediagraminFigure3illustratessnapshotsofhow"done"eachfeatureisatvariouspointsintime.
Eachminichartrepresentsapointintime.Theheightofthecoloredinportionofeachfeaturebar
representswhatdegreethatfeatureisdone.Asimplewaytocalculatethisisdividingthenumberof
acceptancetestspassingbythetotalnumberofacceptancetestsforthatfeature.Notethatthenumber
oftestscouldincreaseovertimeastheproductownersunderstandingofthefeatureimprove.
Figure3
Donenesschartsforanagileproject
Notehowagileprojectsfocusonbothmakingthefeaturessmallenoughthattheycanbedoneina
cycleandnottakingontoomanyfeaturesatthesametimetherebyminimizingthelengthoftimethata
particularfeatureisindevelopment.(Thegoalistocompleteeachfeatureinthesameiterationitwas
startedin,oratworstcase,theverynextiteration.)ThisallowstheProductOwnertodoincremental
acceptancetestingaseachfeatureisdelivered.Anybugsfoundcanbescheduledforfixingatthe
appropriatetime(whichmayberightawayorinsubsequentiterations.)
Figure4illustratesa"burndownchart"basedonplottingthenumberoffeatureslefttobe"done"
againsttime.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 113

Figure4
Burndownchartforanagileproject
Thesimplicityoftheburndowncharthelpsquicklygrasptheprogressoftherelease.However,itmay
masksomeotherdynamicsthattakesplaceontheproject.Forexample,when
Forsuchsituations,MichaelCohnintroducedanalternativeburndownchart(Figure4B),inwhichthe
progress,asusually,isshownabovethebaseline,andchangesinscopearereflectedbelowthebaseline.
[CohnOnBurndownChartAlternative]

Figure4B
Alternativeburndownchart
Insteadofhaving100percentofthefeatures50percentdoneatthehalfwaypointoftheproject,agile
projectsstrivetohave50percentofthefeatures100percentdone.ThisgivestheProductOwner
optionsifspecificationanddevelopmentofthefunctionalitytakeslongerthanexpected,whichisnot
uncommon.Theycandecidewhethertoadjust(reduce)theproductscopetodeliverontimeorto
adjust(delay)thedeliverydatetoincludeallfunctionality.Italsomeansthattheworkofreadiness
assessmentandacceptancetestingarespreadoutmoreorlessevenlyacrosstheproject.Thisis
discussedinChapter17PlanningAcceptance.
Figure5illustrateschartsforalessagileproject.
<TBA>
Time
F
e
a
t
u
r
e
s

l
e
f
t

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 114

Figure5
Donenesschartsforalessagileproject
Onthisexampleproject,mostfeaturesaretakingseveraliterationstocompleteandacceptancetesting
onlystartsafterallfeaturesaredeemedready.Becausetheyarefoundverylateintheproject,thereis
lesstimetofixdeficienciesfoundduringacceptancetesting(suchasmissedrequirements)sotheyneed
tobefixedmuchmorequicklyortheytendtostayunfixedforthisrelease,withworkarounds,
restrictions,andspecialcases..
7.3.2 CommunicatingPercentDoneonSequentialProjects
Sequential(a.k.a.waterfall)projectshavemoreofachallengebecausethephases/milestones
synchronizedevelopmentinsuchawayastoensurethatallfunctionalityisavailablefortestingat
roughlythesametime.Thispreventsusing"percentoffunctionalityaccepted"asameaningful
predictivemeasureofprogress.Instead,Sequentialprojectsusuallyasksomeonetodeclarewhat
percentageeachfeatureisdone.Forexample,thedevelopermaysaytheyare80percentdonecoding
anddebugging(thoughthisnumberisoftenstuckat80formanyweeksinarow!).Consideringthe
subjectivenatureofestimationtechniques,sequentialprojectsoftenchoosetousetechniquessuchas
"earnedvalue"todeterminea"degreeofdoneness"metric.Unfortunately,thesetechniquesareprone
toerrorandfudging,andtheyarebothdifficultandtimeconsumingtoproduceandmaintain.
Figure6containsatypicalsequenceofdonenesschartsforthisapproach.
Figure6
Donenesschartsforasequentialproject
Inthissequentialversionofthecharts,wecanseehowphased/sequentialdevelopmentencouragesus
toworkinparallelonmanyfeaturesbecauseeachfeatureissynchronizedbygatingmechanismssuchas
themilestonesRequirementsFrozen,DesignComplete,andCodingComplete.Thismeansthatallthe
featuresareavailableforacceptancetestingatroughlythesametimeandmustbefinishedacceptance
testinginaveryshortperiodoftimebecauseitisonthecriticalpathoftheproject.Thishas
implicationsforthestaffinglevelsrequiredforthereadinessassessmentandacceptancetestingroles.
Whendevelopmentislate,theperiodforreadinessassessmentandacceptancetestingisfurther
shortenedandtheresourcesfurtherstressed.Italsohasimplicationsontheimpactoffindingbugs
duringthetestingbecausethefixesareonthecriticalpathtodelivery.
Figure7illustratesa"burndownchart"basedonplottingthenumberoffeatureslefttobe"done"
againsttime.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 115

Figure7
Burndownchartforasequentialproject

Sidebar:WhatittakestodoIncrementalAcceptance
InprojectsthatroutinelyapplyIncrementalAcceptanceTesting;theProductOwner(ortheonsite
customerproxy)isshowntheworkingfeatureassoonasitisdeemedreadybythedevelopers,often
justafewdaysintoadevelopmentiteration.Projectsthatdonotemploythispractice,inparticular
thoseemployingasequentialprocess,encountermanyobstacleswhentheytrytooverlaptestingwith
development.Theproblemismoreseverewhentheoverlapisintroducedtorecovertheschedule
whendevelopmentisrunninglate.Whythedifference?
Thereareanumberoffactorsatplayhere,eachofwhichmaycontributetothesituation.The
problemsincludedifficultycommunicatingwhatistestableandwhatisnot;thisresultsinfrustration
amongsttestersthatthetestingresultsinmanybugstowhichthedeveloperrespondsthatfeature
isntdoneyet;youshouldnotbetestingityet.
Crossfunctionalteamsadvocatedbyagileprocessesaddressthecommunicationproblem.The
productowner,requirementsanalysts,usabilityexperts,developers,andtestersalltouchbasedaily.
Theyparticipateinthesameiterationplanningmeetingsandendofiterationreviews.Thismeansthat
thereismuchbetterawarenessamongsttesters,whethertheybereadinessassessorsoracceptance
testers,ofwhatcanbetestedandwhatshouldbeleftforlater.
Adisciplinedsourcecodemanagementapproachalsohelps.Applyingtestdrivendevelopment
practicesandusingacontinuousintegrationserverthatrebuildstheentirecodebaseafterevery
checkinarepracticesthatsupportdisciplinedsourcecodemanagement.Inanenvironmentusing
thesepractices,thebuildprocessinvolvesrunningalltheautomatedunittests.Anytestfailuresare
treatedasastopthelineeventwherethetopprioritybecomesgettingthebuildworkingagain.This
ensuresastablecodebaseatalltimes;theteamsimplywonttoleratemembersdumpinghalffinished
codeintothebuild.
Time
F
e
a
t
u
r
e
s

l
e
f
t

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 116
Anotherformofdisciplineisthedecompositionoflargefeaturesintosmallerfeaturesandeven
smallertaskswhichcanbecompletedinunderaday.Thisallowsworking,testedcodetobechecked
inseveraltimesadayavoidingtheoverheadofbranchingandmerging.
Controllingthenumberoffeaturesinprogressisyetanothermeasurethatsupportsincremental
acceptancetesting.Thispractice,borrowedfromtheworldofleanmanufacturingandleanproduct
design,minimizestheamountofmultitaskingpeopleneedtodotherebyimprovingfocus,flowand
productivity.Averyusefulsideeffectisthatthelengthoftimefromdecidingonwhatisrequiredto
whenthefunctionalityisreadytobeacceptancetestediskepttoaminimum;oftenjustafewdays.
Thisreducesthelikelihoodoftherequirementschangingwhiledevelopmentisinprogressorthe
ProductOwnerforgettingwhatitwastheymeantbythatrequirement.
Highlyincrementalacceptancetestingimpliesthattheproductwillbetestedmanytimes.Thecostof
testingwouldrisewitheveryiterationasmorefunctionalityisadded.Agileteamsavoidthisbyhaving
extensiveautomatedregressiontestsuites(atleastunittestsbutusuallyfunctionaltestsaswell)that
theyrunbeforedeclaringthatthecodeisreadyfortesting.Theseregressiontestsactasalarge
changedetectorwhichinformstheteamwheneverthebehaviouroftheproductchanges
unexpectedly.Forexample,itmaybeanunexpectedsideeffectofafeaturebeingintroduced.
Acceptancetestersaretypicallyinvolvedinpreparationoftheautomatedfunctionaltestssothey
knowwhatisalreadytestedandfocusonareasnotcoveredbytheautomationsuchasuserinterface
behaviour.
Ateamthatisencounteringdifficultyindoingincrementalacceptancetestingmaybeexhibiting
symptomsofmissingoneormoreofthesepractices.Thisisespeciallyprevalentonteamsnewtoagile
andteamsthathavetodealwithlargelegacycodebases.

7.4 Summary
Itmayseemlikeasimpleconcepttodetermineifafeatureorproductisdone,butinpracticeitcan
provetobesomewhatslippery.Wemustknowhowtodetermineifwearedone,whatcriteriatouseto
establishwhetherwearedoneandhowtotestthosecriteria.Itinvolvesdeterminingiftheproper
featuresorfunctionalityispresent,ifitworksasexpectedorrequiredanddoesitmeettheexpectations
oftheProductOwner.Havingaprocessinplacewillincreaseourchanceofsuccessfullydeliveringa
productwearehappywith.
7.5 WhatsNext?
Thisfinishesourintroductiontothemodelswecanuseastoolsforthinkingaboutacceptance.Thenext
fewchaptersdescribeperspectivesonacceptancefromanumberofdifferentplayers.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 117
7.6 References
[BachOnGoodEnoughSoftware]Bach,J.TheChallengeofGoodEnoughSoftware.
http://www.satisfice.com/articles/gooden2.pdf
[YourdonOnGoodEnoughSoftware]Yourdon,E."WhenGoodEnoughSoftwareIsBest,"IEEESoftware,
vol.12,no.3,pp.7981,May1995.
[CabriGriffithsOnAgileEVM]Cabri,A.andGriffiths,M.Earnedvalueandagilereporting,Proc.ofthe
Agile2006Conference,2328July2006,pp.1722.
[DeBonoOnSixThinkingHats]EdwarddeBono,SixThinkingHats,BackBayBooks,1999.
[CohnOnBurndownChartAlternative]MikeCohn,AnAlternativeReleaseBurndownChart,
http://www.mountaingoatsoftware.com/altreleaseburndown,Oct19,2009

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 118
PartIIPerspectivesonAcceptance
Theprocessleadinguptotheacceptancedecisioninvolvesanumberofdifferentplayersandeachhas
theirownperspectiveontheprocess.Thechaptersinthispartdescribeacceptancefromanumberof
theseperspectivesincludingseveralpartiesthatmayplaytheroleofacceptancedecisionmaker
(BusinessLead,ProductManager,EnterpriseArchitect,LegalDepartmentandTestManager)andother
partiesthatmayplaytheroleofreadinessdecisionmaker(UserExperienceSpecialist,Development
Manager,SolutionArchitectandTestManager.)Eachoftheperspectivesthatfollowfocusonaspectsof
acceptancethatareparticularlysalienttotheplayer.Itisworthreadingalltheperspectivesevenifone
seemstofitthereaderlikeaglove.
[Notetoeditors:ThetextinPart2deliberatelyspeaksdirectlytothepersonineachoftheroles.Thatis,
itaddressesthepersonasthoughwewereinconversationwiththem.Itusestheyoupronounon
purposeasopposedtothePart1chapterswhichtypicallyusedwemustorwedo.]

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 119
Chapter8 TheBusinessLeads
Perspective
Anorganizationmaycommissiontheconstructionorcustomizationofsoftwareforitsownuse.The
constructionwouldbedonebyanotherorganizationwhichmaybeaninternaldepartment(suchasthe
InformationTechnologydepartment)oranothercompany(anoutsourceddevelopmentcompany.)In
suchanorganization,theBusinessLeadisthepersonwhobringstotheprojectteamthevisionofwhat
thesoftwareisintendedtodo.TheBusinessLeadmayalsobeknownastheProjectSponsor,Business
Sponsor,BusinessOwner,orBusinessPartner,andactsinthecapacityoftheProductOwner.
SeetheBusinessPersonintheMarketingDivisionpersonainAppendixAOrganizationStereotypesand
ReaderPersonas.

Thedecisiontocommissioncustom(orbespokeasitiscalledinsomelocales)softwareisabusiness
decisionmadebasedontheexpectedbusinessbenefitsoftheinvestmentthereturnoninvestment
(ROI).Thebenefitsmaybeincreasedincomethroughadditionalsalesorservices,costreductionthrough
fewerstaff,moreefficientworkprocessesorreducederrorsessentially,thesebenefitsareintendedto
contributetotheeconomicbottomlineoftheorganization.Sonowthatyouvemadethedecisionto
commissionconstructionofthesoftware,nowwhat?Beforeyoucanrealizethebenefitsyouwillbe
askedtomakethedecisiontoacceptthesoftwareasbuiltorprovidealistofdeficienciesthatmustbe
addressedbeforeyouwillacceptit.Acceptanceisasignificantstageofanycontractualprocess(See
LegalPerspectivesonAcceptance).Howyouapproachthisacceptancedecisioncanhaveabiginfluence
onhowpainfulorpainlesstheprocesswillbe.Itwillalsoimpactyourpostacceptanceactivities
(includingactualusage,operationalsupport,maintenanceetc.)
ThewrongapproachcanbesummarizedwiththefamousquoteIcannotdescribegoodartbutIll
recognizeitwhenIseeit.Takingthisapproachtotheacceptancedecisionissuretoresultinlate
delivery,costoverruns,deliveryofsoftwarethatdoesnotbringvaluetoyourorganizationanda
strainedrelationshipwithyourProductDevelopmentTeambecausethetheywillhaveapoorchanceof
meetingyourneedsiftheydontknowwhattheyare.Whentheydeliverthesoftwareyouwanted,
youllbesuretofindallsortsofissueswithhowitworks.YoullreportthesebugstotheProduct
DevelopmentTeamandexpectmostofthemtobefixedbeforeyouacceptthesoftwareandputitinto
production.Thatwilltakeadditionaltimeandresourcestherebyincreasingcostsanddeferringthe
benefits.Intheworstcase,thevendormaybeunabletofixallthedefectstoyoursatisfactionandyoull
eitherhavetoacceptsoftwareyouarenothappywithorwriteoffthesystementirely.
ToquoteLudwigWittgenstein,oneoftheleadingphilosophersofthe20
0th
century,thereisagulf
betweenanorderanditsexecution.Ithastobefilledbytheactofunderstanding
[WittgensteinOnPhilosophicalInvestigations].Tobesuccessful,donotunderestimatethisactof
understanding.Successfulprojectstypicallyhaveaclearagreementbetweenthe

ProductOwnerand

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 120
theProductDevelopmentTeamonwhatconstitutesasuccessfuloutcomefortheproject.Thisproject
charterprovidesahighlevelviewofthebusinessleadsexpectationsoftheprojectandhoweachparty
willbehavethroughouttheproject.Italsolaysoutanykeyconstraintsregardingschedule,costandany
otherrelevantfactors.Dontfocusthecharter(orcontract,StatementofWork,etc.)onwhatremedies
youwillhaveiftheProductDevelopmentTeamfailstodeliver;bythentheprojecthasalreadyfailed!
Instead,focusonthebehaviorsthatwillensureprojectsuccess.Allpartiesshouldtreattheprojectasa
miniaturejointventureandstriveforawinwinoutcome.Ifanyofthepartiesistryingtowinatthe
othersexpense,allpartieswilllose.Dysfunctionalrelationshipsleadtodysfunctionalproducts.Seethe
sidebarTargetcostcontractingfordescriptionofanalternativetotraditionalfixedpriceortime&
materialscontracts.RefertoSectionResponsbilitiesoftheProductOwnerinChapter1forasummaryof
theresponsibilities.
Sidebar:Target

CostContracting:fromadversarialcontractstowardspartneringcontracts
LeanSoftwareDevelopmentvisionariesTomandMaryPoppendieckadvocatethepartneringor
relationalapproachtocontracting,inwhichpartiesagreetoworkinacollaborative

waytocarryout
theproject[PoppendiecksOnLean,Ch.9].TargetCostContractsareatypeofincentivescontractsthat
canbeusedtosupportsharingofbothfinancialrisksandrewards.Thishelpstoalignthebestinterest
ofeachpartywiththebestinterestsofthejointventure.ThePoppendiecksencourageforging
synergisticrelationshipacrosscompanyboundaries.Theyenvisionthatthosewhodowillseeatleast
the25percentto30percentcostreductionofthecostofdoingbusinesspredictedbyPeterDruckerin
[DruckerOnManagementChallenges].
TargetCostContractsareanapproachtoidentifyingandconstraininguncertainty.Itsdoneinthe
followingway.Atargetcostfortheprojectisagreedatthebeginningandbothpartiesareproactively
workingonachievingthetargetcost.Uponprojectscompletion,theactualcostismeasuredagainst
theoriginallysettargetcost.Iftheactualcostisgreatthanthetargetcost,thesupplierandthe
customereachbearanagreedelementofthatcostoverrun.Iftheactualcostislowerthanthetarget
cost,thesuppliersharesthesavingswiththecustomer.Theshareofthecostoverrun/underrunmay
beaconstantproportionormayvarydependingonthesizeofthedelta.
Note:TheanalysisofPerry&Barnesrevealsastrongcaseforsettingthesuppliersshareofcost
overrunorunderrunatavaluethatisnotlessthan50%[PerryEtAlOnTargetCostContracts].
- ThekeycharacteristicsoftheTargetCostContractsareasfollows:Targetcostincludesallchanges
- Targetisthejointresponsibilityofbothparties
- Targetcostisclearlycommunicatedtoworkers
- Negotiationsoccuriftargetcostisexceededwithneitherpartybenefiting.
- Thisenticesworkersatallleveltoworkcollaboratively,compromise,andmeetthetarget.
ForadditionalcasestudyofusingTargetCostContractsinsoftwaredevelopment,seethisexperience
report[EckfeldtEtAlOnTargetCostContracts].

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 121
8.1 TestPlanning
Acceptingsoftwaresoundslikeadiscreteeventbutthereismuchtobedoneleadinguptoit.Asa
ProductOwneryouneedtobepreparedtoparticipateinthedevelopmentofyoursoftware.Anyone
whohashadahousebuiltspecificallyforthemknowsthattherearealotofdecisionstobemadeand
inspectionstobedonethroughouttheentireconstructionprocess.Softwareshouldnotbeany
different.Expecttogetasmuchoutofyoursoftwareasyouarepreparedtoputin.Theabsentee
ProductOwnergetstheresultstheydeserve;anunsuitablesystemdesignedforsomeoneelse.
8.1.1 CommunicatingExpectations
Toavoidthisfateyoucertainlymusthaveaplanformakingtheacceptancedecision.Butyoumustalso
planforregularcommunicationwiththeProductDevelopmentTeamaboutthecriteriaforthat
decision.Oneofthemajoracceptancecriteriawillbesuccessfulimplementationoftherequirements
youhavespecified.Theserequirementsmaytakemanydifferentformsincluding:
Functionalspecificationswritteninplainlanguage(e.g.Thesystemshall)
Usecasemodelsanddocuments
Protocolspecificationsforinterfacestoothersystems
Userinterfacemockups,prototypes,cognitivewalkthroughs,orexistingsystemstoemulate.
Descriptionsofbusinessrulesandalgorithms
Theserequirementsdocumentsformanimportantbasisbothforthevendorbuildingthesystemand
youracceptancetestingplans.Butrecognizethatthesedocumentsandmodelswillneveranswerallthe
questionsthevendormayhave.Norwilltheyexhaustivelydescribeallyourexpectations.Infact,you
maynotevenbeawareofsomeofyourexpectationsuntiltheyfailtobesatisfiedbythesoftware.No
amountofupfrontrequirementsplanningwillflushoutallyourexpectationsthereforeitiscrucialto
interactwiththesoftwareearlyandoftentogiveyourselftimetodiscovertheseunrealized
expectationsearlyenoughtoaffordtheProductDevelopmentTeamtimetoimplementthem.
ExpecttheProductDevelopmentTeamtodiscovermanyambiguitiesandunforeseencircumstancesin
whateverrequirementsandexpectationsyouhavedescribedtothem.Plantobeavailabletothe
ProductDevelopmentTeamonaregularbasis,themorefrequentthebetter.Ifyoucannotparticipate
personally,assignsomeonetoactasyourProductOwnerproxyandensurethattheyhavefull
understandingofwhatyouwanttoachieveandaclearmandateonwhattheycandecidethemselves
andwhatneedsyourapproval.Onagileprojects,expecttoprovideatleastonepersonfulltimeforthe
entiredurationoftheprojecttoclarifyrequirementsanddocontinuousincrementalacceptancetesting.
8.1.2 PlanningWhentoTest
AkeypartoftheagreementwiththeProductDevelopmentTeamshouldbethefrequencywithwhich
youaresuppliedworkingsoftwareonwhichyoucanperformacceptancetesting.Inthepuresequential
orclassicwaterfallmodel,youwilllikelyneedtowaituntilneartheendoftheprojectbeforeyou
receiveanysoftware.Thismaybeasinglefinalreleasethatthevendorbelievesiscomplete,high

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 122
qualitycode,ortheremaybeaseriesofreleaseswithnamessuchasAlpha,BetaorReleaseCandidate
thatareprovidedearlyenoughtoallowrevisionofhowthesoftwareworksbasedontheresultsof
acceptancetestingofeachrelease.Inmostcases,thepreliminaryreleasesareeitheroflowerqualityor
containlessfunctionalitythanthefinalrelease.Makeyourincrementalacceptancetestingplans
accordingly;dontexpecteverythingtobeavailableintheearlyreleases.Whatistobeavailableineach
releaseshouldbethesubjectofdiscussionswiththeProductDevelopmentTeam.
Intheagilemodel,youshouldexpecttoreceiveworking,potentiallydeployablesoftwareonaregular
schedulethroughouttheentireprojecttimeline.Youllbeaskedtodecidewhatfunctionalitythe
ProductDevelopmentTeamshouldbuildineachiteration(akasprint)andyoullneedtobepreparedto
providethedetailedacceptancecriteriaforeachfeatureoruserstorytotheProductDevelopment
Teamearlyintheiteration.ThisprocessisknownasAcceptanceTestDrivenDevelopment.Whenthe
softwarefeatureisfinished,youllbeaskedtotryoutthenewlybuiltfunctionalityanddecidewhetherit
workstoyoursatisfactionorrequireschangesbeforeyouwillacceptit.Inthisincrementalacceptance
testingmodelyouwillneedtotestadditionalfunctionalityeachiteration.Youmayalsoneedto
dedicateoneormoreiterationsattheendoftheprojecttothefinalendtoendacceptancetestingand
releaseactivities.
RegardlessoftheprocessmodelyoushouldexpecttheProductDevelopmentTeamtodoadequate
testingaspartoftheirreadinessassessment.Theexpectationshouldbeclearlyunderstoodbyall
partiesandyoushouldplantoprovidetheacceptancecriteriaanddetailedacceptanceteststothe
ProductDevelopmentTeamsothattheycanrunthetestsduringtheirreadinessassessmentactivities.
Youwillalsoneedtohaveplansinplaceforhowyouwillassessthesoftwarewhenitbecomesavailable.
Ataminimum,youwillneedtoknowhowmanypeopleandhowmuchtimewillberequiredtodothe
assessment.Thedetailsofwhatteststheywouldrunmayalsobeincludedorthismaybedetermined
nearertotheactualtesting.
8.1.3 RegressionTesting
Assumingyouwillreceiveanewversionofthesoftwaremorethanonce,andthisisaverysafe
assumptionregardlessofwhetheryouuseasequentialoragileprocessmodel,youllneedtoretestnot
onlythenewlyintroducedorchangedfunctionalitybutalsoallthepreviouslybuiltfunctionality.This
canturnouttobealotofeffort.Onestrategyistochoosetheteststoberunbasedonwhatwas
changedbutthisincreasestheriskofmissingnewlyintroducedbugs.Anotherstrategyistohaveasuite
ofautomatedtests(see
AutomatedTestinginCh.16:PlanningforAcceptance)thatcanberunquicklyaftereverychangetothe
software.Thiscangreatlyreducetheeffortinvolvedinretestingorregressiontesting.Thereforemake
suretoaskyourProductDevelopmentTeamtoincludeautomatedtestsaspartoftheircostingto
ensureyoudonthaveaneverincreasingworkloadasmoreandmorefunctionalityisdelivered.The
automatedtestsarereallyfortheProductDevelopmentTeamsbenefitasithelpsthemensurethat
theyarenotintroducingdefectsintotheexistingsoftware.Also,youmaywanttorequestthe
regressiontestssuitestobeshippedaspartoftheprojectdeliverables.Testsareassetsandtherefore
youshouldplanonkeepingthemcurrentforaslongasyouplantousetheproduct.Havingthe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 123
acceptancetestsinplacemayreducethecostofyourfuturemaintenance/extensionprojects(forwhich
youmaychoosetouseadifferentsupplier).Buttheacceptancetestsmaythemselvesrequiretest
maintenanceorrefactoringasthebehaviorofthesystemundertestevolves.
8.1.4 TestResourcingandOutsourcing
ThetestingyouplantodomaybeinfluencedbyyourknowledgeofwhattestingtheProduct
DevelopmentTeamwillhavedonepriortogivingyouaccesstothesoftwareandbytheskillsand
experienceofthepeopleyouhaveavailabletotest.Whenyoudonthavethenecessaryexpertiseand
youcannotorchoosenottobringitinhouseyoucanelecttouseathirdpartyorganizationtodoyour
acceptancetestingonyourbehalf.Beaware,however,thatveryfewtestoutsourcershaveperfected
mindreading!Youwillneedtoprovidethemwithdetaileddirectionregardingthenatureandobjectives
ofthetestingyouwantthemtodoandtheexpectedbehavior(i.e.requirements)ofthesystemtobe
tested.Youllalsowanttobeveryclearabouttheacceptancecriteriaandwhethertheyaremakingthe
acceptancedecisiononyourbehalforprovidingyouwithdatatosupportyouracceptancedecision.Test
outsourcingiscommonforspecializedtypeoftesting(e.g.securitytesting).Testoutsourcingmayalso
bemandatedbyacertificationbody/regulatoryauthoritythatrequiresanindependent
verification/audittobeperformed.Forexample,alltaxpreparationsoftwarepackagesmayneedtobe
testedforcompliancetothetaxauthoritiesstandardsfortaxreturninformation.
8.1.5 TestEstimation
Estimatingtheamountoftimeandefforttoallocatetoacceptancetestingisdifficultwithoutagood
understandingofthenatureofthefunctionalityinvolvedandthethoroughnessofthereadiness
assessmentthatwillbedonebytheProductDevelopmentTeam.
Acommonstrategyistotimeboxthefinalacceptancetestphaseandguessthenumberofcyclesof
acceptancetestingplusbugfixingthatwillberequired.Thisrequirespastexperiencetocomeupwith
reasonableguesses.Ifindoubt,guesshighandallowforseveralcyclesoftestingtogivethevendortime
tofixbugs.Warning:Thereisalwaysalotofpressuretoguesslowtoallowearlierdeliveryofthe
softwareandthelowercostthatthisimplies.Butguessingtoolowwilltypicallyresultinhavingto
choosebetweenacceptingthesoftwareonschedulewithmanyknowndeficienciesordelayingthe
releasetoallowadditionaltest&fixcyclestobeperformeduntilallimportantdeficienciesarefixedand
retested.ThetestingphaseisoftensqueezedbylatedeliveryofthesoftwarebytheProduct
DevelopmentTeamandthetestersareoftenaskedtomakeupthedelaybyreducingtheelapsedtime
fortesting.
Thealternativestrategyinvolvesincrementalacceptancetesting;acceptancetestingthatisdone
frequentlythroughouttheproject.Thisimprovesthepredictabilityofthedeliverydateintwoways.
First,itfindsbugsearlierallowingthemtobefixedlongbeforethefinalacceptancetestcycle.Second,it
providesyoutheopportunitytolearnhowmuchtestingisreallyrequiredandhowmuchtimeandeffort
itwilltaketodoit.Thislearninghappensearlyenoughintheprojectthatyoucanadjustyourplansfor
lateriterationsandthefinaltestcyclesothattheprojectcanbedeliveredontime.Adjustmentscould

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 124
includeincreasingordecreasingtheplannednumberoftestcycles,addstafforskills,increasingor
decreasingtheamountoftestautomation,contractingathirdpartyforspecializedtestingandsoon.
8.2 WhatDoYouNeedtoTest?
EnsurethatyouhaveaclearunderstandingofwhatwillbetestedbytheProductDevelopmentTeam
andwhatyouareexpectedtotest.Typically,theProductDevelopmentTeamisexpectedtodo
thoroughtestingofallcapabilitiesandqualitycharacteristics(suchasperformance,reliability,usability
etc.)beforehandingoffsystemforacceptancetesting.Ofcourse,thisisbasedontheProduct
DevelopmentTeamsunderstandingofyourexpectationswhichmayvaryfromyoursifthe
communicationislessthanperfectwhichisalmostalwaysthecase.Yourresponsibilityduring
acceptancetestingisensuringthesystemissuitableforyourownpurpose.
Thetestcasesyourunduringacceptancetestingshouldcovereverythingyoucareabout.Thiscan
includebusinessfunctionality,operationsfunctionalityandnonfunctionalcharacteristicsofthesystem.
Eachoftheseareasmayrequiredifferentexpertisetotestandpossiblydifferentstakeholdersignoffs.
Decidingwhowillsignoffbasedonwhattesting,donebywhom,isanimportantpartofacceptancetest
planning.Thebusinessleadmayneedtonegotiatewithotherpartiestodeterminetheacceptance
decisionprocessandthetestingthatprovidestheinformation.
8.2.1 FunctionalTesting
Functionaltestingshouldincludebusinessworkflowteststhatverifythatallreasonablebusiness
scenariosorworkflowsfunctionproperly.Moredetailedusecasetestscanbeutilizedtoverifythat
eachbusinesstransactionorusecaseisimplementedcorrectly.Thehappypathormainsuccess
scenarioofeachusecaseisnotwherebugswilltypicallybefound;theytendtolurkinthedarkcorners
ofthelesscommonlyexercisedscenarios.Businessruletestscanfocusonverifyingthatthebusiness
rulesandalgorithmsareimplementedcorrectly.Applicationswithrichuserinterfacestypicallyrequire
extensivetestingoftheuserinterfaceitself.Thistestingcaneitherbebundledwiththeusecasetestor
youcanhaveseparatetestsessionsdedicatedtoexploratorytestingthebehavioroftheuserinterface.
Thedegreetowhichyouneedtotestshouldbebasedonthelevelofconfidenceyouhaveinyour
ProductDevelopmentTeamandthecomplexityofthedeliverable.Thiscanbebasedonpastexperience
onpreviousprojectsoronwhattheyhavedemonstratedonthisproject.AProductDevelopmentTeam
thathasprovidedfullvisibilityoftheirinternalprocessesandespeciallytheirreadinessassessment
activitiesmayinspiremuchmoreconfidenceinthequalityoftheproduct.Thiscouldchangethefocusof
youracceptancetestingactivitiesfromlookingforobviousmisimplementationsofthebusinesslogicor
rulestohighervalueaddingactivitieslikeusabilitytestingandbusinessworkflowtestingbothofwhich
improvetheproductdesignandthereforethefitnessforpurposeratherthanjustverifyingthequalityof
theconstruction.Inparticular,thepresenceofanextensivesuiteofbusinessreadableautomated
workflow,usecaseandbusinessruleacceptancetestscouldchangetheemphasisfromconducting
extensivemanualtestingofthisfunctionalitytospotchecking.SeeAutomatedExecutionofFunctional
TestsinVolumeIIIforhowtoautomatethesetests.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 125
8.2.2 NonFunctionalTesting
Nonfunctionaltesting(akaextrafunctionaltesting)isoftendonebytheProductDevelopmentTeam
duringthereadinessassessmentorbyaseparatetestorganization.Itshouldincludeverifyinganynon
functionalcharacteristicsofthesystemthataredeemedcriticalforacceptanceaspertheSystem
RequirementsModeldescribedinChaper5,suchas:
Performance(capacityandresponsetime)andstress(behaviorwhenoverloaded)tests,
Securitytesting(enforcementofprivacypolicies,resistancetoattackbyhackers,etc.)
Scalability(theabilitytohandleadditionalusersortransactionloadsinthefuture)
Usability(theefficiency,effectivenessandsatisfactionwithwhichuserscanemploythe
softwaresysteminordertoachievetheirparticulargoals)
andothers.
EnsurethatyouhaveacommonunderstandingwiththeProductDevelopmentTeamastowhatProduct
DevelopmentTeamfunctionaltestingtheywilldoandhowmuchinformationyouexpecttoreceiveas
inputintotheacceptancedecision.
Thepeoplechargedwithoperatingandsustainingthesystem(hereincalledtheoperationsdepartment)
willhaveveryspecificrequirementsthatneedtobeaddressed.Youllneedtobeawareoftheseasyou
workwiththeProductDevelopmentTeamexploringanddefiningtherequirementsforthesystem.Ina
largerorganizationyoumayhaveacounterpartintheoperationsdepartmentwhocanprovidethe
requirementsandactastheacceptancedecisionmakerfromtheoperationsperspective.See
OperationsManagersPerspectiveinPartII.
Ataminimumyoullneedtobeawareofthetimeandresourcescomplyingwiththeserequirementswill
entail.Insomecasesyoullneedtogiveupnewfunctionalityinordertoensurethattheoperational
requirementsaremet.Someexamplesofoperationalrequirementsmightinclude:
Thesystemneedstointegratewiththespecificsystemsmonitoringsoftwareinuseatyour
company.
Dataneedstobemigratedfromapreviousversionofthesystemorfromasystemthisone
replaces.
Thesystemneedstohaveupgrade,startupandshutdownscriptsthatcanbeusedduring
automatedservermaintenancewindows.
Thesystemneedstobebuiltusinganapprovedtechnologystack.Theoperations
departmentmaynothavetheskillstosupportsometechnologies.E.g.A.NETshopmaynot
havetheskillstosupportanapplicationbuiltinJavaorPHP.Sometechnologiesmaybe
incompatiblewithtechnologiesusedatyourcompany.
Theoperationsgroupmayhavespecificwindowsduringwhichtheycannotinstallortest
newversionsofsoftware.Thismayaffectthereleasescheduleyouhavedevisedanditmay

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 126
makeontimedeliveryofsoftwarebytheProductDevelopmentTeamevenmorecriticalas
smallscheduleslipscouldresultinlonginservicedelays.
Therecouldberequirementsforperformingsuchserviceoperationswithoutdegrading
thecurrentsystem
Reliabilityrequirementsdictatethatthesystemmeetstheagreeduponservicelevel
agreements(SLAs)suchasuninterrupteduptime,transactionlatency,.
Thesystemmustprovideselfmonitoring,errordetection,andsystemloggingcapabilities.
Thesystemmustprovideforrobustrecoverability(includingtheabilitytoverifymultiple
failoversandrecoveries),disasterrecovery,databackup&recovery,andallotheraspectsof
systemrecovery.
Afullrangeofdocumentationmustbeprovidedincludingsupportdocumentation,bothpre
andpostrollout,trainingmanuals,andotherreferencedocumentation.
Thesystemmustbemigratedtothenewversionoftheplatform.
Multipleversionsofthesystemmustrunsidebyside.

8.2.3 OtherPartiesInvolvedinAcceptanceTesting
Ifthereareotherpartiesinvolvedintheacceptancedecisionensurethatyouhaveacommon
understandingaboutwhattheywilltestandhowthatfeedsintotheacceptancedecision.Common
partiesincludetheoperationsdepartmentwhowouldhaveasayinwhetherthesystemcanbeinstalled
andmanaged,usabilitypractitionerswhomightneedtoensurethatthesystemcomplieswithany
usability(includingaccessibility)standards,andcorporatesecuritydepartmentswhoneedtoensurethe
systemissecurebeforeitcanbeaccepted.
8.3 WhereWillYouTest?
Akeypartoftestplanningisdecidingwhatenvironmentswillbeusedforwhatkindsofactivitiesand
howthesoftwareismovedbetweenenvironments.
8.3.1 TestEnvironments
Expecttohavemanyactivitieshappeningconcurrentlyasyoudoacceptancetesting.TheProduct
DevelopmentTeamwilllikelybefixingthebugsthatyoureportwhileyoucontinuetolookforfurther
issues.Thisrequiresaseparateenvironment.WhentheProductDevelopmentTeamhasfixedenough
bugstowarrantdeliveringanewversionofthesoftwaretheywillneedtodoreadinessassessment.This
mayrequireyetanotherenvironmentifbugfixingistocontinueinparallel.Ifatestorganizationneeds
totestthesoftwarebeforepassingitontoyouforacceptancetestingtheywillneedtohavetheirown
testenvironmentaswell.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 127
8.3.2 SoftwareDeliveryProcess
Giventhatdifferentgroupswillbeworkinginseparateenvironments,howdoesanewversionofthe
softwaregetinstalledintheenvironment?Forexample,youvefinishedacceptancetestingonrelease
candidate1(RC1)ofthesoftware,reportedanumberofbugs,andtheProductDevelopmentTeamhas
fixedenoughbugstoproposedeliveringanewversion,RC2,toyourtestenvironment.Whatisthe
processforreplacingtheRC1softwareintheacceptancetestingenvironmentwithRC2?Doesthe
ProductDevelopmentTeampushthenewversionwithorwithoutanyadvancenotice?Doesthe
ProductDevelopmentTeamnotifyyouoftheavailabilityofRC2andwaitforyoutoaskforittobe
installedintheacceptancetestingenvironmentwhenyouarereadytoreceiveit?Andhowlongwillthe
acceptancetestingenvironmentbeunavailablewhiletheupgradeisbeinginstalled?
8.4 TestExecution
Whenthesoftwareisdeliveredforacceptancetesting,acceptancetestexecutionmustbegininearnest.
Itisimportanttoknowwhereyoustandatalltimeswithrespecttotheprogressoftestingandyour
currentunderstandingofthesuitabilityofthesoftware.
8.4.1 ProgressMonitoring
Asacceptancetestingisinvariablytimeboxedbyeithercontractsorcustomerexpectations,itis
importanttoensurethatthetestingactivitiesareprogressinginatimefashion.Thenatureofthetest
plandeterminesthenatureofprogressmonitoring.Agileprojectshaveacceptancetestingspreadout
overthedurationoftheprojectandtheresultsoftestingarehighlyvisibleintheformoffeedbackto
theProductDevelopmentteam.Thereisplentyoftimefortheacceptancetesterstolearnfromtheir
mistakesandmisassumptionsintimeforthenextroundofacceptancetesting.Traditionalsequential
projects,ontheotherhand,havealltestinghighlyconcentratedattheendoftheproject.Itiscritical
thatthesetestactivitiesbeverycloselymonitoredtoensurethatanydeviationsfromthetestplanare
quicklyidentifiedandrectified.Thereisntmuchtimeforlearningbetterapproachestotestingso
everyonemustbehighlytrainedbeforetheacceptancetestperiod.Thisapproachtotestingisfraught
withmanydangersandshouldbeavoidedatallcosts.
8.4.2 BugTracking&RecordKeeping
Thenatureoftheproductbeingbuiltandthecontextinwhichitwillbeusedhasalargebearingonthe
natureanddegreeofrecordkeepingforbothtestexecutionandanybugsfound.Itmaybesufficientto
simplyruntestsandtrackanybugsfoundinaninformalmannerasdescribedinthesidebarIncremental
AcceptanceTestingandBugTrackingonAgileProjects.Inotherenvironments,recordsneedtobekept
aboutwhattestswererun,againstwhichversionofthesoftware,when,bywhomandtheactualresults
thatwereobserved.
Ataminimum,anydefectsthatwerefoundduringacceptancetestingneedtoberecordedandtriaged.
Eachbugshouldincludewhichversionofthesoftwareitwasfoundin,whetheritwillprevent
acceptanceofthesoftware(severitylevel)andwhofoundit.Bugsthatmustbefixedbeforethe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 128
softwarewillbeacceptedmustbeclearlycommunicatedtotheProductDevelopmentTeam.Lesssevere
bugsshouldalsoberecordedbutshouldalsobeclearlylabeledasnotgatingacceptance.Keepinmind
thatinmanycontextsanythinglabelnongatingwilllikelynevergetfixedsotherealdecisioniscanyou
livewiththeproductexhibitingthisbug,onnot?
8.4.3 RegressionTesting
Eachtimethesoftwareischanged,whethertoaddnewfunctionalityortofixbugsfoundduring
acceptancetesting,newbugsmayhavebeenintroduced.Therefore,anytestsperformedbeforethe
softwarewaschangedmayneedtobeexecutedagainonthenewversionofthesoftware.Thiscanbe
verytimeconsumingiftheregressiontestsarenotautomated.
SomeProductDevelopmentTeamsautomatetheirunittestsaswellasthefunctionalteststhatyouwill
usetomaketheacceptancedecision.OtherProductDevelopmentTeamsdontknowhowtodothisor
dontfeeljustifiedinincludingthecostofautomatedtestsintheircostestimates.Theymayneedtobe
pushedintodoingthisbyyou,theProductOwner,insistingthatautomatedregressiontestsasjustas
importantasusermanualsandbuiltinhelp.Ifyouplantousethisproductforanylengthoftime,
expecttheproducttoneedtoevolve.Theevolutionwillbemucheasierifthetestsareautomatedand
willsaveyoulargeamountsofeffortandwillhelpavoidhavingthequalitydegradeastheproduct
evolves.
Animportantconsiderationinregressiontestingisthechoiceofdatatobeusedinthetests.Regression
testscanbeexecutedwiththesamesnapshotofproductiondataforgenuinerepeatability.Anapproach
thatmaybemorerepresentativeofareal,evolvingproductionenvironmentistorefreshtheregression
testingdataovertime.Thismoredynamicapproachtoregressiontestingsacrificesrepeatabilityin
returnforthepotentialtofindnewproblems.
8.4.4 MaintainingMultipleVersionsoftheSoftware
Therewillbetimesthatyoumaychoosetomaintainseveralversionsofthesoftwareatthesametime.
Acommonsituationisafterrelease,duringthewarrantyperiod,ifyouarealsodevelopingasubsequent
releaseatthesametime.Anothersituationiswhenyouchoosetosupportmultipleversionofsoftware
inproductionandneedtoapplyanybugfixestoallsupportedversions.Beforewarnedthatthereisan
extracosttomaintainingversionsinparallelasanybugfixesdoneinthesupport/warrantystreamwill
needtobepropagatedintothedevelopmentstreamatsomepointandretestedthere.
Sidebar:IncrementalAcceptanceandBugTrackingonAgileProjectsCaseStudy
ArecentITprojectIworkedonwasprettytypicalofhowbugsaredealtwithonagileprojects.The
developerswouldconsultwiththebusinesslead(actingastheProductOwner)astheydevelopedthe
functionalitytomakesuretheyunderstoodwhatshe(thebusinesslead)wantedbuilt.Theywould
jointlydefinetheaccetancetestsforthefunctionalitybeforeorwhilethefunctionalitywasdeveloped.
Whenthedeveloperhadfinishedwritingthecodeandhadfullyunittestedit,theywouldrunallthe
previouslydefinedacceptancetests.Assumingthetestsallpassed,theywoulddeemtheuserstory
readyforacceptancetestingbythebusinesslead.Theywouldwalkovertheher(the15personteam

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 129
allsatinthesameteamroom)andaskforafewminutesofhertime.Theywoulddemonstratethe
functionalitytoherbyexecutingthemainacceptancetestsinfrontofher.Shewouldtryafewmore
thingsthatoccurredtoher.Ifeverythingworkedasexpectedthusfar,shewouldsaysomethinglike
Thislooksgoodthusfar.Letmedosomemoredetailedtestingandgetbacktoyouwiththeresults
laterintheday.Thedeveloperswouldthengobacktotheirdeskstocatchontheiremailorpickand
startresearchingthenextuserstorywhiletheywaitedforthefinalacceptancedecision.
Ifthebusinessleadfoundsomethingshedidntlikeduringthispreliminarytesting,shewoulddiscuss
theissuewiththedevelopersandtheywouldjointlycomeupwithoptionsanddecidedhowto
proceed.Ifthismeantchangingthesoftware,thedeveloperswouldgobacktotheirdeskstomakethe
changesandthiswholeprocesswouldberepeatwhentheyweredone.
Ifthebusinessleadfoundanybugswhiledoingthemoredetailedacceptancetestingshewouldgo
overtothedevelopersdesktodiscussthem.Theywouldthenagreediscusswhattodo.Thebusiness
leadhadseveralchoices.Somebugswereseriousenoughthatshewantedthemfixedbeforeshe
wouldaccepttheuserstory;thedeveloperswouldaddresstheseproblemsrightawaydropping
anythingelsetheywereworkingontodoso.Thekeyincentivewasthatanyuserstoriesthatwere
almostfinishedbutwhichhadntbeenacceptedbytheProductOwnerwouldntcounttowardsthe
teamsvelocityfortheiteration.(Theteamwasveryproudofitsvelocityandwasalwayslookingfor
waystoimproveit.)
Anotheroptionshehadwastoaddressthebuginanalreadyplanneduserstorythatwouldbe
implementedinafutureiteration.Shewouldchoosethisoptionifthefunctionalitywasotherwise
workingandtheparticulartestcasewasborderlineastowhichstorysscopeittrulybelongedto.
Athirdoptionwastoturnthebugintoafullblownuserstory.Shechosethisoptionifitdidntfitinto
anyoftheotherstoriesalreadyintheproductbacklog.
Thefinaloptionwastotrackthebugonaseparatebuglist.Eachbugwaswrittenona5x7cardand
addedtothebugmanagementsystemwhichwassimplyadesignatedareaofthewhiteboard.
Becausemostbugswerefixedwithinafewhoursofbeingfound,thislistofbugsonthewhiteboard
rarelyexceededadozenorso.Duringourbiweeklyiterationplanningmeetingsthebusinesslead
wouldpickthestoriesfromtheproductbacklogshewantedbuiltandthebugsfromthebuglistshe
wantedaddressed.Sometimesdeveloperswouldsuggestthataparticularbugbeincludedintheplans
becausedevelopmentofauserstorywouldaffectthesamecode.
Thisapproachtobugmanagementworkedverywellforourteambecausewewerecolocated.We
didntneedadatabaseorfancyreportsbecauseweneverhadmorebugsontheboardthantheteam
couldkeeptrackofatanyonetime.Anditensuredthatthequalityoftheproductalwaysmetour
MQRbecausewedidnthaveaplacetohidethebugs.
Thiswhiteboardbasedapproachwontworkonlarge,distributedprojectsbutthephilosophyoffixing
bugsrightawayratherthanlettingthembuilduptounmanageablenumberswill.
GerardMeszaros

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 130
8.5 Summary
TheBusinessLeadisthepersonnominatedbythebusinesstoplaytheroleofProductOwnerforthe
durationofaprojecttodevelopasoftwareintensivesolutiontoabusinessproblem.Theyare
responsibleformakingthedecisionabouthowtobestutilizetheallocatedmoneyorresourcesin
buildingorextendingthesystemfunctionality.
AsummaryoftheresponsibilitiesoftheProductOwnercanbefoundinChapter1TheBasicAcceptance
Process.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 131
Chapter9 ProductManagers
Perspective
AproductmanagerinaproductcompanyplaysaverysimilarroletothatoftheBusinessLeadonanIT
projectwithafewexceptions.Whilethebusinessleadistypicallyengagedonlyforthedurationofthe
projecttodeliverthenewfunctionality,aproductmanageristypicallyresponsiblefortheprofitabilityof
theproductoveritsentirelifetime.Theyaremoreinclinedtothinkintermsofmultiplereleasesofthe
software/product.Theirstrategymayalsobefocusedonaproductline,notindividualproducts,inwhich
casetheintegrationandinteroperabilityamongtheseproductswouldbeofprimaryconcern.Companies
thatbuildproductsarealsomorelikelytohavededicatedtestingresourcesinaseparatetest
department.ManyofthefactorsdiscussedinTheBusinessLeadsPerspectivealsoapplytotheproduct
managersincetheProductManageeralsoplaystheroleofaProductOwner,butinaproductcompany.
Thissectionfocusesonthedifferences.
SeetheProductManagerpersonainAppendixXOrganizationStereotypesandReaderPersonas.

AsaProductManager,youshouldfeeltotalownershipofthesuccessoftheproductinthemarketplace.
Yousurveythemarketforwhatitneedsanddefinetheproducttobebuilt.Youdecidehowmuch
revenuetheproductislikelytogenerateandhowmuchmoneyitisworthinvestingtobuildthe
product.Itisyourjobtoensurethatthesupplierteamunderstandswhatyouwantbuiltandtherelative
prioritiesofthepiecesoffunctionality.
9.1 DefiningDone
Akeypartoftherelationshipbetweentheproductmanagerandtheproductdevelopmentteamis
havingacommonlyunderstooddefinitionofwhatconstitutesdone.Thiscomesintwodimensions:
functionalityandquality.
9.1.1 DefiningMMF
MMFstandsforMinimumMarketableFunctionality(alsoknownasMinimumMarketableProduct
(MMP)orMinimumMarketableFeatures(MMF));itisthesmallestamountoffunctionalitythatyou
coulddelivertoyourcustomerswithoutlosingcredibility.AnythingoverandabovetheMMFaddsvalue
butshouldnotdelaythereleaseifitcannotbecompletedbythedeliverydeadline.Ittakesself
disciplineandagoodunderstandingofthemarkettodefinetheMMF;itissomucheasiertojustthrow
ineverythinganycustomerhaseveryaskedforandcallitareleasebutthisisalmostsuretoresultin
spendingtoomuchandtakingtoolongtodelivertheproduct.
Youmayneedhelptotranslatethehighlevelrequirementsintoadetailedproductdefinition.Business
analysisexpertisemaybeneededtounderstandthebusinessprocessesusedbythepotentialusers.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 132
Productdesignexpertise(e.g.interactiondesignersandgraphicdesigners)maybeneededtohelp
designthefunctionality.Productdevelopmentexpertisemaybeneededtounderstandwhatis
technicallypossibleandconsistentwithhowtheproductalreadyworks.Youmaywanttoinclude
peoplewiththeseskillsaspartoftheproductmanagementteamoryoucanenlistthehelpofother
organizations.Whicheverwayyouchoosetodealwiththeissueyoullneedtoensurethateveryone
understandsyourvisionfortheproductandhowitsatisfiesspecificmarketneeds.
9.1.2 DefiningMQR
Youdontlikereceivingunsatisfactorysoftwareandthesupplierteamhatesbeingtoldthattheir
carefullycraftedsoftwareisnotverygood.Establishingtheminimumqualityrequirement(MQR)isan
importantstepinbuildingatrusting,productiverelationship.AjointlycraftedDoneDoneChecklist
postedinthesupplierorganizationsofficeisagreatwayforthesupplierteamtokeepthefocuson
qualityfrontandcenter.AreleaselevelchecklistasshowninFigure1isagoodwaytosettheoverall
expectationsofwhatthesuppliershouldbeprovidingbeforeacceptancetestingcanstart.
SampleDoneDoneChecklistforRelease1.0
- AllMMFfeaturesareincludedintheRCbuild.
- Asecurityreviewhasbeenconductedandasignoffobtained.
- Testteamisconfidentthatnoneoftheincludedfeatureshasasignificantriskofcausingproblems
intheproductionenvironment,i.e.MQRmet,including:
- configtesting,sidebyside
- performance
- localization
- globalization
- userexperience.
- Businesscomplianceachieved.
- Thereareclear,concisedeploymentandrollbackinstructionsfortheoperationsteam.
- Therearecleartroubleshootingscriptsandknowledgebasearticlesforusebythehelpdesk
representatives.
- Allincludedfeatureshavebeendemoedtoandacceptedbythecustomer.

Figure1
SampleDoneDoneChecklistforaRelease
ThequalitycriteriadescribedinareleaseleveldonedonechecklistasshowninFigure1mayrequire
severalreadinessassessmentcyclesbeforethecriteriaaresatisfied.Toreducethenumberofreadiness
assessmenttest&fixcyclesweneedtoinfluencethelevelofqualitydeliveredbyindividualdevelopers

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 133
tothereadinessassessors.Thiscanbeaffectedbyafeatureleveldonedonechecklistasshownin
Figure2.
Theacceptancecriteriaarespecifiedandagreedupon
Theteamhasatest/setoftests(preferablyautomated)thatprovetheacceptancecriteriaaremet
Thecodetomaketheacceptancetestspassiswritten
Theunittestsandcodearecheckedin
TheCIservercansuccessfullybuildthecodebase
TheacceptancetestspassonthebitstheCIservercreates
Nootheracceptancetestsorunittestsarebroken
Userdocumentationisupdated
Userdocumentationisreviewed
Thefeatureisdemoedtothecustomerproxy
Thecustomerproxysignsoffonthestory
Figure2
SampleDoneDoneChecklistforindividualfeatures
Donedonechecklistssuchasthesearecommonlyusedonagileprojects.
9.1.3 ManagingYourOwnExpectations
Themosteffectivewaytoensurethatpoorqualitysoftwaregetsdeliveredistoaskthesuppliertoover
commit.Softwarerequirementsarenotoriouslyhardtopindownandjustashardtoestimate.Forcinga
suppliertodelivertoanunrealisticscheduleissuretobackfireasyouwilllikelygetalatedeliveryof
poorqualitysoftwarethatyouthenneedtoinvestextratimeintobringingthequalityuptobarely
sufficientlevel.Thisisnotarecipeforsuccess!
Itisfarbettertodefinethelevelofqualityrequiredandmanagethescopeoffunctionalitytoensureit
canallbefinishedinthetimeandmoneyallotted.Timeboxedincrementaldevelopmentisaproven
techniqueforachievingontimedeliveryofqualitysoftware.Itistoyouradvantagetoworkcloselywith
thesupplierorganizationtoselectthefunctionalitytobeimplementedduringeachiteration.Youllhave
muchbettervisibilityofprogresstowardsthereleaseandmuchearlierwarningifthefullslateof
functionalitycannotbecompletedbywhateverdeliverydateyouhavechosen.Thisgivesyoutimeto
prunethefunctionalitytofitintothetimeavailableratherthantryingtocramintoomuchfunctionality
andtherebysacrificingquality.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 134
9.2 WhoMakestheAcceptanceDecision?
Asproductowner,youneedtoberesponsibleformakingthebusinessdecisionaboutwhethertoship
theproductasisortodelaydeliverytoallowmorefunctionalitytobeaddedortoimprovethequality
byfixingknownbugs.Youwilllikelyrelyondatafromotherpartiestomakethisdecisionbutdelegating
thisdecisiontooneofthosepartiesisatyourperilbecausetheyareunlikelytounderstandthebusiness
tradeoffsnearlyaswellasyoudo.
Itisreasonableforyoutoexpectthatthepartiesprovidinginformationaboutthequalityoftherelease
candidatedosointermsthatyouunderstand.Theimpactofeverymissingfeatureorknownbugshould
bedescribedinbusinessterms,nottechnicaljargon.Ideally,thisshouldbeeasilytranslatedinto
monetaryimpact(lostordelayedrevenue)andprobability(likelihoodofoccurrence).
9.3 OperationalRequirements
Mostsoftwareproductshaveoperationalrequirements.Thesearetherequirementsofwhoeverwill
havetoprovisiontheruntimeenvironment,installthesoftware,monitoritsperformance,install
patchesandupgrades,startupandshutdownthesoftwareduringservermaintenancewindows,andso
on.Eventhesimplestshrinkwrapproducthasoperationalrequirements(suchasautomaticupdating)
andcomplexserverproductsoftenhaveveryextensiveoperationalrequirements.Ensurethatyouhave
theappropriatepeopleengagedduringtheproductdefinitionandproductacceptanceprocessto
ensurethattheserequirementsarenotmissed.Makesureyouhavetherightpeopleinvolvedinthe
decisionmakingprocesstoensurethattherequirementsaresatisfied.Operationalrequirementsmay
requirespecializedtestingandtesttools.
SomeexamplesofoperationalrequirementsforSoftwareasaServicearesimilartothoseofanITshop:
Thesystemneedstointegratewiththespecificsystemsmonitoringsoftwareinuseatyour
company.
Thesystemneedstohaveupgrade,startupandshutdownscriptsthatcanbeusedduring
automatedservermaintenancewindows.
Thesystemneedstobebuiltusinganapprovedtechnologystack.Theoperations
departmentmaynothavetheskillstosupportsometechnologies.E.g.a.NETshopmaynot
havetheskillstosupportanapplicationbuiltinJavaorPHP.Sometechnologiesmaybe
incompatiblewithtechnologiesusedatyourcompany.
Theoperationsgroupmayhavespecificwindowsduringwhichtheycannotinstallortest
newversionsofsoftware.Thismayaffectthereleasescheduleyouhavedevisedanditmay
makeontimedeliveryofsoftwarebythesupplierevenmorecriticalassmallscheduleslips
couldresultinlonginservicedelays.

Theoperationalrequirementsofasoftwareproductinclude:

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 135
Thesoftwarewillneedtointegratewithavarietyofsystemsmonitoringsoftware
frameworks.
Thesoftwaremayneedtoworkinavarietyofhardwareplussoftwareconfigurationsfor
bothserverandclient(possiblybrowser)components.Thismayrequireextensive
compatibilitytesting.Itmayalsoconstrainthefeaturestobedeliveredduetocrossplatform
issuesthatrestrictthefunctionalitytoalowestcommondenominator.

9.4 DealingwithLargeProducts
Whenaproductistoolarge,complexorincludestoomanydifferenttechnologies,itcanbedifficultto
commissionasingleteamtodevelopitforyou.Anotherchallengeistheintegrationofseveralexisting
productseachofwhichhasitsownsupplierorganization.Ifyouhavethisissueyouhaveseveralwaysto
dealwithit.Themoreobvioussolutionistobreakdowntheproductrequirementsintoseparate
requirementsforeachsubcomponentandcommissioncomponentteamstodothework.Themain
issuewiththisfromaproductmanagementperspectiveisthatitmakesproductarchitectureandthe
subsequentsubcomponentintegrationyourproblem.SeethesidebarComponentTeamsforamore
detaileddiscussion.
Theotheralternativeistocommissionfeatureteamsthatworkacrosscomponentsanddividethe
productrequirementsintosubsetofproductrequirementsforeachfeatureteam.Thishasthe
advantageofdealingwithanyintegrationissueswithinthefeatureteam.Thesupplierorganizationmay
needtoprovideamechanismtoensureconsistencyofapproachacrossthefeatureteamsuchasa
virtualarchitectureteamorstandardsteam.SeethesidebarFeatureTeamsonMicrosoftOfficefora
moredetaileddiscussion.Regardlessofwhichapproachyouchoose,youllwanttohavesome
mechanismforcoordinatingtheactionsoftheteams.Twocommonapproachesaretheuseofproject
reviewsandtheScrumofScrumsapproach.
Sidebar:ComponentTeams
Alargeclienthadorganizedtheirteamsaroundthecomponentsoftheirarchitecturewhichconsisted
ofmanycomponentsorganizedinseverallayers.ThebottomlayeriscalledThePlatform.Builtatop
ThePlatformaremanyreusablecomponentsandserviceswecansimplyrefertoasA,BandC.These
componentsareusedbythecustomerfacingcomponentsX,YandZ.Whenmarketingreceivesanew
featurerequesttheproductmanagerdecomposestherequestedfunctionalityintorequirementsfor
eachofthecustomerfacingcomponentsX,YandZ.Thearchitectsofthecorrespondingteamsare
eachaskedtodoafeasibilitystudyofthefunctionalityandtoprovideanestimateoftheeffort.This
mayrequirerequestingnewcapabilitiesfromoneormoreofcomponentsA,BorCwhichmayinturn
requirenewcapabilitiesfromThePlatform.Oncethefeasibilitystudywascompletedandanestimate
wasavailablefromeachoftheaffectedteams,marketingwoulddecidewhethertobuildthefeature
andifso,whichreleasetoincludeitin.Intheappropriaterelease,eachoftheteamsinvolvedinthe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 136
designwoulddotheworkrequiredintheirindividualcomponents.Whenallthecomponentswere
finished,bigbangintegrationwouldstart.Thiswouldoftendiscoverseriousmismatchesbetween
expectationsandwhatwasbuilt.
Otherissuesassociatedwiththecomponentteamapproachincludebalancingworkloadamongthe
teamsanddifficultiesimplementingmultiplefeaturessimultaneously.LarmanandVoddestressthat
theseissuesleadtodelaysandincreasedoverheadduetofrequenthandoffs.Analternativeto
componentteamsisorganizingdevelopmentaroundfeatures,discussedinthesidebarFeatureTeams
atMicrosoft.See[VoddeLarmanOnScalingAgile]and[LeffingwellOnFeatureVsComponentTeams]fora
comparisonbetweencomponentandfeatureteams

Sidebar:FeatureTeamsatMicrosoft
Afeatureisanindependentlytestableunitoffunctionalitythatiseitherdirectlyvisibletoacustomer
(customerfacingfeature)orisapieceofinfrastructurethatwillbeconsumedbyanotherfeature.
Featureteamsaresmallinterdisciplinaryteams(512people)thatfocusondeliveringspecificproduct
features.AsamplefeatureteamcouldbetheoneinchargeofRibboninMSWord.Anotheroneisthe
oneinchargeofSpellchecker.AthirdoneisinchargeofAddressLabels.
MicrosoftimplementstheFeatureTeamsstrategyusingamethodologycalledFeatureCrews.It
extendsthegeneralstrategywithspecificguidelinesonhowtomanagethecodebaseissuchawayas
toavoidpartiallyfinishedfeaturesfromaffectingthestabilityofthemainbranch.Theideaisforall
disciplines(development,PM,UxD,testing)toworkcloselytogetheronprivatebuilds(intheirown,
isolatedprivatefeaturebranches)andonlyaddthefeaturetotheproduct(themainbranch)whenitis
FeatureComplete(similarlytoDoneDonedescribedearlier).ThefeatureinFeatureComplete
stateisexpectedtobefullyimplemented,sufficientlytestedandstableenoughforeithera)dog
fooding(usingonesownproductinhouse)andsharingwithcustomersinaCTP(Community
TechnologyPreview)forcustomerfacingfeatures,orb)codingagainstbyanotherfeaturecrewfor
featuresthatdealwithunderlyinginfrastructure.
Thetypicaldurationforfeaturedeliveryisbetweenthreetosixweekstheideaistoensureasteady
flowoffeaturesandregular,frequentfeedbackfromthecustomers.Importantly,eachfeatureteamis
freetodecidewhatdevelopmentapproach/processtouse,aslongastheiroutputmeetsallquality
gates(commonforallfeaturecrews)anddoesntdestabilizetheproduct.(Foracasestudyontheuse
oftheFeatureCrewmethodologyatMicrosoftVisualStudioToolsforOfficeproductunitanddetails
onintegrationoffeaturesintoparentandmainbranches,see[MillerCarterOnLargeScaleAgile])
9.5 BugTrackingandFixing
Usersandtesterswillreportbugsnomatterhowgoodajobyoudobuildingthesoftware.Someof
thesebugswillbelegitimatedefectsintroducedintothesoftwareaccidently.Otherswillberequests
fornewfunctionalityyouconsciouslychosetoleaveoutoftherelease.Exceptfromacontractual(who
pays)perspective,bugsneednotbetreatedanydifferentlyfromfeaturerequests.Eitherway,youneed

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 137
todecidewhattimeframeyouwantthefunctionalityofthesystemchangedandthereforeinwhich
streamofsoftwareyouwanttoincludeit.TheConcernResolutionModelprovidesacommonwayof
thinkingaboutbugs,featurerequestsandotherprojectissues.
9.6 MaintainingMultipleVersionsoftheSoftware
Therewillbetimesthatyoumaychoosetomaintainseveralversionsofthesoftwareatthesametime.
Acommonsituationisafterrelease,duringthewarrantyperiod,ifyouarealsodevelopingasubsequent
releaseatthesametime.Anothersituationiswhenyouchoosetosupportmultipleversionofsoftware
inproductionandneedtoapplyanybugfixestoallsupportedversions.Beforewarnedthatthereisan
extracosttomaintainingversionsinparallelasanybugfixesdoneinthesupport/warrantystreamwill
needtobepropagatedintothedevelopmentstreamatsomepointandretestedthere.
9.7 Summary
TheProductManageristypicallytheProductOwnerofasoftwareintensiveproductandisresponsible
formakingthedecisionabouthowmuchmoneyorresourcestoinvestinbuildingorextendingthe
productfunctionality.
AsummaryoftheresponsibilitiesoftheProductOwnercanbefoundinChapter1TheBasicAcceptance
Process.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 138
Chapter10 TestManagersPerspective
SomeorganizationshaveaseparateTestManagerrolewhileothershavethetestersreportingdirectlyto
theDevelopmentManageroreventheBusinessLeadorProductManager.WhentheTestManagerrole
exists,thetestmanagerisusuallyresponsibleforplanningthebulkofthetestingactivitiesand
managing/coordinatingtheirexecution.Thetestmanagermayormaynotactasthegatekeeperthat
is,maketheacceptancedecisiononbehalfofotherstakeholders.Itisimportantforallpartiesto
understandtheroleoftheTestManagerinthisregard.
SeetheTestManagerpersonainthecompanystereotypecalledAProductCompanyinAppendixX
ReaderPersonasforanexample.

TheTestManagersrolecanbeadifficultonetodowellbecausesomanyprojectfactorsareoutside
yourcontrol.Testplanningisessentialforallbutthemosttrivialprojects;whatthetestplanspecifies
dependheavilyontheroleoftestingasitrelatestotheproductdevelopmentteamandtheproduct
manager.
10.1 TestManagersRoleinAcceptanceDecision
Theroleoftestingseemssimpleenough:usetheproductinvariouswaysandreportanybugsthatyou
find.Butthisisnottheonlyresponsibilityofthetestorganizationinmanyenterprises.Makesureyou
understandwhetheryouandthetestingteamaredoingreadinessassessment,acceptancetestingor
makingtheacceptancedecision.
10.1.1 TestingasAcceptanceDecisionMaker
Insomecircumstances.Thetestorganizationisexpectedtocollectdatatohelptheproductowner
decidewhetherornotthesoftwareisreadytobereleasedtousers.Makesureyouhaveaclear
understandingofwhetheryouarethegatekeeper(acceptancedecisionmaker)oraresupplying
informationtosomeoneelsewhowillbemakingthedecision(preferred.)Recognizethatthe
acceptancedecisionisabusinessdecision.Ifyouaregoingtomakeit,youhadbetterunderstandthe
businessfactorswellenoughtomakeagooddecisionthatyoucanexplaintoeveryone.Otherwise,
youlljustbethebadguyholdinguptheshow.
IfyouaretoactastheAcceptanceDecisionMaker,youmusthaveaccesstoallthebusinessrelevant
informationtomakeagooddecision.Thatis,youmustunderstandthebusinessconsequencesof:
acceptingthesoftwaregiventhenatureofthebugsthatareknowntoexistandthelevelof
confidencethatallseriousbugshavealreadybeenfound,and,
notacceptingthesoftwaretherebydelayingthedelivery.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 139
Eachofthesechoicescanhaveseriousnegativebusinessconsequencesandtheacceptancedecision
cannotbemadewithoutafullunderstandingofthem.TheTestManagershouldnotaccept
responsibilityformakingtheacceptancedecisionlightly.Inmostcasesitisbetterforthisbusiness
decisiontobemadebyabusinessperson,typicallytheproductmanager,businessleadoroperations
managerbasedoninformationprovidedbythetestmanager.
10.1.2 TestingasAcceptanceTesters
Whenyouareprovidinginformationtoanotherpartytomaketheacceptancedecision,makesureyou
havecommonunderstandingofwhatinformationthatparty(orparties)willrequiretomakethe
decision.Ensurethetestersunderstandtheusersforwhomtheyareactingasaproxysotheycandefine
testthatreflectrealistic(notnecessarilyalwaystypical)usageofthesoftware.UserModels(eitheruser
rolesorpersonas)canbeaneffectivewaytogainunderstandingofrealusers.
10.1.3 TestingasReadinessAssessors
IfyouaredoingreadinessassessmenttohelptheDevManagerdecidewhetherthesoftwareisreadyfor
acceptancetestingbythebusinessusers,youllwanttobuildagoodrelationshipwiththedevteamso
thatyoucanhelpthemunderstandthequalityofwhattheyhaveproducedandhowtheycanimprove
it.Embeddingtesterswiththedevelopmentteam(oftenusedinconjunctionwithatechniquecalled
pairtesting)canbeagoodwaytohelpthemlearnhowtobuildqualityinbytestingcontinuouslyrather
thantossinguntestedsoftwareoverthewalltothetestteam.(Kohldescribesbenefitsofpairtestingin
thisexperiencereport[KohlOnPairTesting])
10.2 TestPlanning
Youwillprobablybeexpectedtocoordinatealltestingactivitiesandkeeptrackofthetest
environmentsused,bugslogged,andversioningrelatedtothefixesperformedonthesourcecode,all
thewayfromreadinessassessmentthroughtoacceptancetesting.Thisworktypicallygoesbythename
TestPlanning.Youllneedtocommunicatethisplantoallinterestedparties;especiallythosewhoare
expectedtoexecutepartsoftheplanandthosewhoareinterestedinknowingwhatkindsoftestingwill
bedonebywhom,whereandwhen.ThiscommunicationoftentakestheformofaTestPlandocument
butitcanalsobedoneorcomplementedorallythroughtargetedpresentations,discussionsor
workshops.Manyofthebesttestplanscomeaboutbyinvitingtheinterestedpartiestoparticipatein
thetestplanningprocesseitherinoneononesessionsorviaworkshops.
10.2.1 TestStrategy
Youwilllikelyendupbeingthedefactoowneroftheoverallteststrategy.Theonlyothercontender
wouldbethesupplierorganizationandiftheywanttobeinvolved,probablyonthetestautomation
side,byallmeansencouragetheirinvolvementbecauseitwillmaketestautomationmucheasier.
Theteststrategyincludesthemajordecisionssuchaswhethertestingwillbedoneinafinaltestphase
orincrementally,whethertestingwillbeprimarilyscriptbasedtestingorexploratorytestingora

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 140
combinationofthetwo.Exploratorytestingisaveryeffectivewayoffindingbugsfastbutitisnt
intendedtoberepeatable.Scriptbasedtestingisbetterforensuringaknowndegreeoftestcoverage
andshouldbeautomatedtoallowrapidregressiontestexecution.
Anotherkeydecisionisregardingtheroleanddegreeofuseofautomatedtoolsincludingautomated
executionoffunctionalandparafunctionaltests,theuseofmodelbasedtestcasegeneratorsandthe
useofautomationaspowertools(includingtheuseofautomatedcomparatorsorverifiers)tosupport
manualtestactivities.Someofthesedecisionsrequirelongleadtimestodecidewhethertochoose&
acquireorbuildtoolsortohiretheappropriateresourcesandthereforemustbemaderelativelyearly
intheproject.Thisdoesntimplythatyouwontrefinethemovertimeasmoredetailedinformation
comestolight.
Theteststrategywillbeheavilyinfluencedbythenatureofyourrelationshipwiththeorganization
developingthesoftware.Theidealcaseiswhentheyarepreparedtocollaboratewithyoubydoing
incrementaldeliveryoffunctionalitytosupportincrementalacceptancetesting.Collaborationontest
automationanddesignfortestabilityisalsoakeyfactorinreducingtestexecutiontimeandcost.Ifthis
levelofcollaborationisnotavailableyoumayneedtohiretestautomationengineersandtest
toolsmithswithstrongprogrammingskillstosupportyourtestautomationactivities.
10.2.2 TestAutomation
Anotherareathatgreatlybenefitsfromcollaborationwiththeproductdevelopmentteamistheareaof
testautomation.Akeysuccessfactorisdesignfortestabilityofthesystem.Onlythesupplier
organizationcandothissoyouneedtoworkwiththemtobuilditin.Agoodwaytoincentthemto
collaborateistoprovidethemwithaccesstothe1buttonautomatedexecutionoftheacceptancetests
youdefine.Thisprovidesthesupplierorganizationwithaninstantscorecardthattellsthemhowthey
aredoingatdeliveringqualitysoftware.Everyonehatessurprisesandnothingannoysadevelopment
teammorethanworkinghardtodeliverqualitysoftwareandbeingtoldafterthefactthatitwasnt
goodenough.
Whilethedevelopmentteamneedstomakethesystemamenabletotesting,thetestautomationtools
canbebuiltbyeitherthesupplierorganizationorthetestteam.Thelatterwillrequiretechnicaltesters,
sometimescalledtestautomationengineers.Eitherway,thetestsshouldbedevelopedprimarilybythe
testersorbusinesspeople/analyststoensuretheyreflectrequirementsratherthandesigndecisions.
Testautomationisnotappropriateinallsituations,whereformsofmanualtestingaremoreeffective
oreconomical.SeeSection17.2inChapter17(PlanningforAcceptnace)inPartIIIforadiscussionof
whereautomationisappropriate.Itsthejobofthetestmanagertodeterminethetestautomation
strategy.
10.2.3 AgreeingonExpectations/Requirements
Onepurposeoftestingistoverifythesystemmeetstherequirements.Unfortunately,therequirements
areoftenvagueorambiguousandthismakesverifyingthemadifficultproposition.Onepersonsbug
canbeanotherpersonsfeature.Testersthinkdifferentlyfromanalystsorbusinesspeopleandthisis
bothablessingandacurse.Theyllcomeupwithallmanneroftestscenariosthattherequirement

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 141
specifiersneverimagined.Thisresultsinadefactodivergencebetweentherequirementsandthe
specificationsthatneedtobemanaged.Theproductowner(ProductManager,BusinessLead,etc.)
needstoagreethattheseadditionaltestscenariosareinfactrequirements.
Therequirementsneedtoaddresstheneedsofallstakeholders,notjusttheusers.Operational
requirementsneedtobetested.Ifthetestteamdoesnthavetheskillstoexecutetheparafunctional
teststhenmakesurethattheproductdevelopmentteamexecutesthemandprovidestestplansto
reviewaheadortimeandtestresultsaspartofthehandovertotesting.Theacceptancedecisionmaker
willtoknowthattheparafunctionaltestingwasdoneandmayevenwanttoseetheresults.
Thepreferredsolutionistotreattestscriptsasanextensionoftherequirementsbyusingthemto
illustratehowspecificrequirementsplayoutintheproduct.Thisimpliesthattheproductmanageror
businesslead(ortheirteammembers)needtoagreethatthetestsinterprettherequirementscorrectly.
Ideally,thetestcaseswouldbearticulatedbeforethesoftwareisbuiltsothatthedevelopmentteams
canusethetestcasestounderstandhowthesystemshouldbehave.ThisisknownasAcceptanceTest
DrivenDevelopment.
10.2.4 AgreeingonDone
Arewedoneyet?Thatisaquestionthatneedstobeaskedandansweredcontinuously.Anddone
doesntalwaysmeanthesamething.Onepersonsdefinitionofdonemaybeanotherpersons
definitionofnotevenclosetodone!Itiscriticaltogetagreementonhowdonesoftwareneedstobe
beforeitgoesthrougheachofthequalitygatesofthegatingmodel.Whenissoftwareconsidered
readyforacceptancetesting?Whenisitconsideredacceptable?Theseneedtobeagreedupon
betweenthevariouspartiesinvolved.Thesupplierandtestingneedtoagreeontheminimumquality
requirementofsoftwarebeforeitisreadyfortesting.Thisbecomestheexitcriteriaforthereadiness
acceptancephasethatthesupplierteammustensureismet.AprominentlypostedDoneDone
Checklistisagoodwayforthesupplierteamtokeepthisqualitycriteriafrontandcenter.
10.2.5 EstimatingTestEffort
Estimatingtheamountoftimeitwilltaketogetthroughonecompletecycleoftestingisachallengebut
onethatcanbeovercomewithexperience.Therealchallenge,however,isguessinghowmanytest&fix
cycleswillberequiredbeforethesoftwareisofgoodenoughqualitytoreleasetoacceptancetestingor
tothemarket.Theneedtoguesscanbeavoidedbydeliveringsoftwareincrementallyandtestingeach
incrementassoonasitisavailable.Thishelpsinseveralways:
Itreducestheamountofsoftwarethathasntgonethroughthetest&fixcycleatleastoncewhen
thefinalroundoftestingoccurs.Hereuntestedsoftwarecanbethoughtofasinventoryand
incrementaltestinganddeliveryasinventoryreduction,whereinventoryisconsidereda
formofwaste(see[PoppendiecksOnLean])
Itprovidesdataonhowmanytest&fixcyclesarerequiredforsoftwaredeliveredbythissupplier
organizationearlyenoughtoallowthetestplanstobeadjustedtoensureontimedelivery
ofsoftware.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 142
Itprovidesfeedbacktothesupplierteamonthequalityofthesoftwaretheyaredeliveringearly
enoughtoallowthesupplierteamtimetolearnhowtodeliverbetterqualitysoftwarethat
requiresfewertest&fixcycles.TheDoneDoneChecklistmayneedtobeupdatedbasedon
whatwaslearned.

10.3 ConcernResolution
Testingwillinvariablyresultinanumberofconcernsbeingraised.Someoftheseconcernswillbebased
onfailedtestcasesandothersmaybebasedongeneralimpressionsabouttheproduct.Allconcerns,
however,arebasedonexpectationsanditisimportanttoensurethattheexpectationsarecorrect.Test
casesthatdontmaptoactualrequirementscanresultinbugsthatwillbeclosedasByDesign.Some
failedtestcasesmaypointoutinconsistenciesbetweenhowthesystemoperatesandhowthetester
expectsthesystemtooperate.Thesemaypointtolegitimateusabilityissuesifthetestersmental
modelofthesystemmatchesthatofrealusers.
Partoftheroleofmanytestorganizationsistoactasthesecondopinion,thehouseofsobersecond
thought,aboutthesoftwarebeingbuilt.Itisimportanttobalancethisformofrequirementselaboration
withtheneedtogettheproductoutinatimely,andevenmoreimportantly,predictablefashion.
Discoveryofsuchrequirementsissuesduringafinaltestingphasemakestheproductmanagersjob
harderbecausetheinformationcomestoolatetoaddresswithoutcompromisingtheschedule.
Therefore,thetestmanagershouldstrivetofindsuchissuesasearlyaspossiblesothattheproduct
managercanmakedecisionswithouttheirbacktothewall.
TheConcernResolutionModeldescribesagenericwaytothinkaboutallmannerofconcernsincluding
bugs,requirementsissues,projectissuesandothermismatchesbetweenexpectationsandactualor
perceivedbehaviors.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 143
Chapter11 DevelopmentManagers
Perspective
Thedevelopmentmanagermaygobymanyofficialtitlesbutforourpurposes,thisisthepersoninthe
supplierorganizationwhoisaccountablefordeliveringsoftwaretotheproductowner(PO)whetherthe
POistheinternalbusinesssponsorofanITprojectortheproductmanagerinaproductcompany.Where
aseparatetestorganization(oftencalledQualityAssurance,IndependentVerificationorsomevariation
onthese)existsforthepurposeofdoingacceptancetesting,thedevelopmentmanagerisresponsiblefor
makingthereadinessdecision,thatis,thedecisiontodeliverthesoftwaretowhoeverisresponsiblefor
makingtheacceptancedecision.Forlargerproductsthedevelopmentmanagermaybeassistedbya
solutionarchitect;forsmallerproductsthedevelopmentmanagershouldunderstandthesolution
architectsresponsibilitiesandeithercarrythemoutthemselfordelegatetosomeinthedevelopment
team.
SeetheDevelopmentManagerpersonainthetwocompanystereotypesinAppendixXReader
Personasforanexample.

AstheDevManageryourkeyresponsibilityistomanagethedevelopmentofthesoftwareinsuchaway
thattheproductowner(productmanager,businesslead,etc.)willacceptthesoftware.Youshoulddo
thisinawaythatmaximizesthevalue(orutility)thatthesoftwarewillprovidetotheproductowner(by
meansofvalueprovidedtotheirusersorcustomer)whileminimizingthecost.Thebestwayto
minimizecostistobuildtherightsoftwareandtobuildittherightwaythefirsttimeanddeliveritto
theacceptancetestingorganizationand/orproductownerassoonaspossible.Thisavoidsexpensive
andtimeconsumingtest&fixcyclesinwhichsomeoneelseteststhesoftware,findsbugswhichthey
thenaskyoutofix.Thisdelay(aformofwaste)contributessignificantlytochurnandcost.Whetherthis
istheproductownerorsomeonelookingoutfortheirinterests,thefewerbugstheyfind,theless
reworkyouneedtodo,thelowertheoverallcostandthemorepredictabletheschedulebecomes.
11.1 TheRoleofReadinessAssessment
Mostdevelopmentteamswanttodogoodwork.Theydontwanttodeliversubstandardsoftwareto
theircustomer.Mostcustomerswanttoreceivegoodqualitysoftware.
Deliveringgoodqualitysoftwareiswhatthecustomerexpectsfromthesupplier.Thisisareasonable
expectation,onethataprofessionalsoftwaredevelopmentorganizationshouldbepreparedtomeet.
Deliveringgoodqualityworkingsoftwareisnteasyortrivial;ifitwere,theproductownerprobably
wouldntneedadevelopmentmanager!

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 144
So,whatdoesittaketodelivergoodqualitysoftware,firsttime,everytime?Whatdoesittaketobea
professionalsoftwaredevelopers?Thecompletedetailsofasoundsoftwareengineeringprocessare
beyondthescopeofthisbook.But,therelationshipbetweenthesoftwaredevelopmentorganization
andthepartiesinvolvedinacceptancethesoftwarearenot.Partofthedevelopmentprocessneedsto
beanhonestselfassessmentofthesoftwaretheteamhasproduced.Iftheteamsaysitisready,giveit
tothetestteam.Iftheteamsaysitisntready,askthemwhatstillneedstobedonetomakeitready.A
gooddevelopmentteamwillalwaysbeabletosaysomethingisreadyandshouldbeabletoclearly
articulatetothetestteamwhatisreadyandworthtesting.
RobertC.Martinarguesthatitisirresponsibleforadevelopertoshipasinglelineofcodenotcovered
byatestandtestsmustbekepttothesamelevelofcodequalityastheproductioncode
[MartinOnProfessionalismAndTDD].Althoughthisargumentappliestodifferentdegreesdependingon
whatisbeingbuilt,thedevelopmentmanagerisresponsibleformakingsurethatthecodeisadequately
securedbysolidsuiteoftestsbeforetheproductownerevaluatesit.
11.2 EffectiveReadinessAssessment
11.2.1 WhosJobisQuality?
Howwouldyoufeelaboutbeingthefirstpersontotryabrandnewproductthatnoone,noteventhe
builderhadtriedbefore.Howmanypeopleyouknowwouldbecomfortableinthatsituation?Ifwe
applythesamelogictothesoftwarewebuild,howmanypeopledoyouknowthatwouldwanttobe
thefirstonetotryapieceofsoftwarethatnooneelsehastriedbefore?Yetthisisexactlywhatmany
developmentteamsdowhentheythrowuntestedorpoorlytestedsoftwareoverthewalltothetest
organizationorthecustomer.Manypeoplewouldarguethatthisisjustplainunprofessional.
Allsoftwareshouldbetestedbythedevelopmentorganizationbeforebeinghandedtoanyoneelsefor
testing.
11.2.2 BuildingQualityInStartwiththeEndinMind
Buildingsoftwareshouldnotfeellikeaguessinggamewheredevelopersguesswhatsrequiredand
testersorusersshoutoutwrong!Thedevelopmentorganizationneedstohaveaclearunderstanding
ofwhatsuccesslookslikebeforeitstartsbuildingthesoftware.Partoftheproblemisthatdevelopment
teamsoftendonthaveagoodsenseofhowthesoftwarewillactuallybeused.Developersdonotthink
likeauserbecausemostdevelopersthriveoncomplexitywhilemostusersabhorit.Thismakesit
challengingfordeveloperstodoagoodjobofreadinessassessmentwithoutoutsidehelp.Acceptance
testsprovidedbytheproductownerortestingdonebytesterscanbridgethisgap.Theformeris
proactive,positiveinputthathelpsdevelopmentteamsunderstandwhatdonelookslikebeforethey
buildthesoftware.Thelatterisnegative,reactivefeedbackthattellsthedevelopmentteamyou
haventdoneagoodjob.Whatkindofguidancewouldyouprefer?
Aprojectcharterisonewaytostarttheprocessofdevelopingthiscommonunderstanding.
Requirementsdocuments,usermodels,usecasesanduserstoriesareallwaysthatwetrytodevelopa

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 145
commonunderstandingbetweenthesupplierandthecustomer.Butthesestaticdocumentswrittenin
naturallanguagetypicallycontainmanyambiguousstatementsandinmostcasesdonotprovide
enoughtosufficientlydescribewhatsuccesslookslike.Thedesignsandworkestimatesprovidedbythe
developmentteamthereforelikelycontainassumptions,manyofwhichareeitherwrongorwillresultin
aproductthatismorecomplexandexpensivetobuild.Itiscriticaltoclarifythesepotential
misunderstandingsasquicklyaspossible.
Asthedevelopmentmanageryoushouldworkwiththeproductownerandacceptancetestteamto
ensurethateveryoneontheteamhasaccesstothedefinitionofsuccessintheformofacceptancetests
beforethesoftwareisbuilt.Thisshouldincludeanythingthatisusedasanacceptancecriteriaincluding
bothfunctionalteststhatverifythebehaviorofthesystemfeaturebyfeatureandparafunctional
qualitycriteriasuchassecurity,availability,usability,operationalcriteriaetc.Thisprocessisknownas
AcceptanceTestDrivenDevelopment(akaExampleDrivenDevelopmentorStorytestDriven
Development).Thismayrequirethatproductownerortestersprepareacceptancetestsearlierinthe
projectthantheymighthavetraditionallydone.Itmayalsorequirethemtobemoreinvolvedforthe
durationoftheprojectratherthanjustatthebeginningandtheendsothattheycananswerany
questionsthatcomeupduringtheinterpretationoftherequirementsandalsodoincremental
acceptance.
11.2.3 DefectPreventionbeforeDefectDetection
Traditionalapproachestotestingfocusondefectdetection.Thatis,theemphasisisonfindingbugs
ratherthanpreventingtheiroccurrenceinthefirstplace.Howcantheemphasisbechangedto
prevention?Itisntenoughtodefinethetestsaheadoftime;wemustalsorunthemfrequently.By
doingso,wealwaysknowthescore.Thetestresultstellushowfarfromdoneweare.Theyprovidea
veryclearindicationoftheprogresstowardsdone,onethatdoesntrequirealotofextraworkto
calculateandonethatishardtofudge.Thetestresults,availablethroughouttheproject,providethe
stakeholderswithvisibilityintotheprojectinamuchmoretransparentfashionthantraditionalmetrics
measuringprogressagainstaphasedactivityprojectplan.
Topreventdefectswedefinethetestsbeforebuildingthesoftware,automatethem,andrunthem
frequentlywhilebuildingthesoftware.Weinstitutesimpleteamnormstoavoidregressing:teststhat
havepassedbeforemustcontinuetopassasnewfunctionalityisadded.Noexceptions!Eitherthetest
hastobechanged(changestofunctionalacceptancetestsmayrequirethecustomersagreement)or
thecodehastobechangedtoreturnthetesttopassingstatus.
Whatroledothevariouskindsoftestsplayinthisprocess?Functionaltestsdefinedbytheacceptance
testersdefinewhatdonelookslikeandthesoftwarecannotbedeliveredtothemwhenanyacceptance
testsarefailing.Automatedunitandcomponenttestsdefinethedesignintentofthesoftware;writing
thesefirstandwritingjustenoughcodetomakethempassensureswedontbuildunnecessary
softwareandthatthesoftwarewedowritesatisfiesthedesignintent.Italsoensuresthatthesoftware
youbuildisdesignedfortestability,acriticalsuccessfactorfortestautomation.Anotherbenefitisthat
theyactasalargechangedetectorthatwillinformtheteamofanyunexpectedchangesinbehaviorof
thesoftware.Thishelpstheteamcatchregressionbugsbeforetheycansneakthroughtotheusers.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 146
11.2.4 ReduceUntestedSoftware
InleanthinkingsuchasexemplifiedbytheleanmanufacturingparadigmusedbyToyota
[ShingoOnToyotaProductionSystem],unfinishedwork(inventory)isconsideredaformofwaste.In
softwaredevelopment,softwarethathasbeenwrittenbuthasnotbeenacceptedisunfinishedwork.
Weshouldstrivetofinishthisworkassoonaspossiblebydoingacceptancetestingassoonaspossible
afterthesoftwareiswrittenandreadinessassessmentiscompleted.Thisincrementalacceptance
testingrequirescollaborationbetweendevelopment,testingandtheproductownerasallpartiesmust
bepreparedtoworkfeaturebyfeatureratherthanwaitingforthewholesystemtobeavailablebefore
anyacceptancetestingisstarted.
Theminimumqualityrequirement(MQR)shouldbeagreeduponaheadoftimewiththetesting
organizationortheproductowner.Ideally,anyandallteststhatwillberunaspartoftheacceptance
testingshouldberunbythesupplierorganizationbeforemakingthesoftwareavailableforacceptance
testing.Knowndefectsshouldbeidentifiedaheadoftimetoavoidwastingpeoplestimetestingbroken
functionality.
11.2.5 WhatKindsofTestsAreRequired?
Functionaltestsshouldberunassoonasthecorrespondingfunctionalityisbuilt.Thisshouldinclude
businessworkflowteststhatverifyendtoendbusinessprocesses,usecaseteststhatverifythevarious
scenariosofasingleusecaseorbusinesstransaction,andbusinessruleteststhatensurethatbusiness
rulesareimplementedcorrectly.
Operationalrequirementsalsoneedtobeverifiedbyensuringthatacceptancetestsprovidedbythe
operationsstakeholdersarerunandresultsareanalyzedregularly.
Parafunctionaltestingshouldbedoneonaregularbasisassoonasenoughsoftwareisbuilttoallow
themtoberun.Theearliertheyarerun,themoretimethesupplierhastocorrectanydeficienciesthat
arediscovered.Thisisespeciallyimportantwithparafunctionaltestsbecausechangingthepara
functionalattributesofthesystemmayrequireachangeinarchitecture,apropositionthatmayget
moreexpensivethelaterthechangeismade.
Manyofthesetestscanbedonemuchearlierintheprojectiftheearlyfocusoftheprojectistobuilda
walkingskeletonoftheapplication.Thewalkingskeletonimplementsthefullarchitectureinavery
minimalistway.Forexample,allthelogicmightbehardcodedtherebysupportingonlyasinglehighly
simplifiedbusinesstransactionorworkflow.Butallthemajorarchitecturalcomponentswouldbe
presenttoensuretheruntimecharacteristicsweretrulyrepresentativeofthefinishedproductevenif
thefunctionalbehaviorwasnotimplemented.
11.2.6 SharpeningtheSaw
Anybugsthatarefoundduringacceptancetestingshouldbeasurpriseandshouldpromptthequestion
howdidthatslipthroughourreadinessassessment?Clearly,thesoftwarewasnttrulyready.Ifthe
answerisbecausetheyuseditadifferentwayfromwhatweexpectedthenthesupplierorganization
hastodoabetterjobunderstandingtheusersofthesoftware.Therefore,everybugfoundbecomesa

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 147
learningopportunityfortheteambypromptingthemtolookforwaystoimprovehowtheybuild
software.ForexampleatMircrosoftspatterns&practicesgroup,duringtheWebServiceSoftware
Factory:ModelingEditionproject,theteamdidjustthatonseveraloccasions,modifyingtheirCIsystem
topreventissuesfromreoccurring.
Asmanagerofthedevelopmentorganizationyouneedtocreatetheconditionsinwhichthe
developmentteamcanproducetherightsoftwarebuilttherightway.Thismayrequireovercoming
organizationalandculturalhurdles.Youneedtoworkwithyourcounterpartsinthetestorganizationto
breakdownthetraditionaladversarialrelationship(ifoneexists)andworktogethertocreateamore
collaborativerelationship.AsAdeMiller,patterns&practicesdevelopmentlead,eloquentlyputit:I
thinkoneofthekeythingsaDevManagercandohereistrytosendtheclearmessagethattestingand
thetestorganizationareimportant.Infartoomanycasestestersareseenassomesortoflesser
functionbecausetheylackthetechnicalskillsofdevelopers.Thetestorganizationhastheskillsto
helpyourteambuildbetterqualitysoftware,notjusttotellyouwhenyouhavent.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 148
Chapter12 UserExperienceSpecialists
Perspective
Someorganizations,especiallythosethatbuildconsumerproducts,haveaseparaterolethatis
responsibleforensuringagoodandconsistentuserexperience.Whatistheroleofpeoplewiththiskind
ofskillsetinthemakingoftheacceptancedecision?
SeetheUserExperienceSpecialistpersonainthecompanystereotypecalledAProductCompanyin
AppendixXReaderPersonasforanexample.

Asapersonwiththeresponsibilityofuserexperienceyoumayberesponsiblefordesigningtheuser
experienceassociatedwiththeproduct.Thismayinvolvevariouskindsofactivitiestobetterunderstand
andcharacterizethepotentialusersincludingethnographicresearch,usermodeling,taskmodeling,etc.
Itlikelyalsoincludesverificationactivitiessuchasusabilitytestingatvariouspointsthroughoutthe
designanddevelopmentprocess.
12.1 TheRoleofUserExperienceintheAcceptanceDecision
Itisimportanttohaveaclearunderstandingwiththeproductowner(productmanagerorbusiness
lead)astowhatyourroleshallbeinmakingtheacceptancedecision.Itcouldbeanyof:
1. Beinganinputintotheproductdefinitionprocessbutnodirectinvolvementinthe
acceptancedecision.Thatis,helpingtheproductownerdefinetheproductbutleavingitto
otherpartiestodefinetheacceptancecriteria.
2. Beinganinputintothesoftwaredevelopmentprocesswithnoinvolvementinthe
acceptancedecision.Thisissimilartothefirstpointbutworkingmorecloselywiththe
softwaredevelopmentteamratherthantheproductownership/definitionteam.
3. Asinpoint1butalsoprovidingusabilitytestingdatafortheacceptancedecisionmadeby
theproductowner.
4. Asinpoint1and3butalsohavingasayintheacceptancedecisioneitheraspartofthe
productownersteamorasapartofaseparate,parallelacceptancedecision.

Forproductswhereusabilityisakeydifferentiator,thepreferredrolewouldbeeither3or4.For
productswithcaptiveuserssuchassystemsdevelopedforinternalusersitismorelikelytobe1or2.
Inanyofthesemodelsitisimportanttoworkwiththebusinessleadorproductmanagertodefinethe
usabilitycomponentoftheacceptancecriteriaasearlyaspossibletogivethedevelopmentteamas
muchguidanceaspossible.Earlyintheprojectthismayconsistofdescriptionsofthekindsoftesting

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 149
proceduresthatwillbeusedtoverifyusability;laterintheproject,itmaybespecificfunctionalitytobe
testedsuchasthenoviceuserstop5transactions,specificusabilityguidelinestobecompliedwith,and
soon.
Whereusabilityisimportantandhowtoachieveitisunclear,itiscrucialtoaddresstheusabilityrisksas
earlyaspossible.Someoftheserisksmaybeaddressedthroughtheuseoflowcostusabilitytesting
techniquessuchassketchingandWizardofOztesting,aformoflowfidelityusabilitytestingconducted
withpaperproductsandmockups.Visualcommunicationisverypowerfultopresentandvalidate
ideas.Abilitytogatherfeedbackfromtheuserseffectivelyisasimportantasproductionofthe
prototypes.Figure1presentsasamplelowfielectronicprototypebuiltwithMicrosoftSketchFlow
[SketchFlow]thatauserisabletopreviewinabrowserandproviderapidfeedback.

Figure1.
UseraddingfeedbackonaSketchFlowprototype
Othersituationmayrequiretheconstructionofelectronicprototypesforhigherfidelityusabilitytesting.
Constructionofspecialpurposehighfidelityprototypesmaybeavoidediftheactualsystemcanbebuilt

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 150
anddeliveredincrementallysequencedbyusabilityrisk.Thismayrequireworkingwiththeproduct
ownertohelpthemunderstandtherisksandconsequencesandtousethesetoinfluencetheorderin
whichfunctionalityisbuilt.Usabilityrelatedriskscanbemitigatedbydefininganinternalmilestoneat
whichsufficientfunctionalityisavailabletoallowusabilitytestingoftheactualproductwithrealor
proxyusers.Thisrequirescollaborationbetweentheproductownerandthesupplierorganizationtodo
incrementaldevelopmentanddeliveryofthesoftwarebasedontheusabilitytestschedule.Theareas
withhighestimpactofusabilityissuesshouldbebuiltandtestedfirsttoensureenoughtimeforany
usabilityrelatedissues,whichmaybepervasiveenoughtohaveawiderangingimpact,tobeaddressed
beforethecostofchangehasbecometoohigh.
Wheretheproductownerhasnotspecifiedincrementaldeliveryitmaybepossibletoworkdirectlywith
thedevelopmentmanagertoarrangeforopportunitiestodousabilitytestingusingearlyversionsofthe
actualsoftwaresystem.
Sidebar:MarryingUsabilityPracticeswithIncrementalDevelopmentMethods
AcommonlyheldopinionisthatInteractionDesignandUsabilityTestingneedtobefinishedbefore
softwaredevelopmentcanbestarted.Thisisindirectcontradictionwiththeagileapproachof
incrementaldevelopment.Modernproductdevelopmenttimescalesoftenprecludeaseparatedesign
phasesotheworkofusabilityspecialistsneedstobedonewithinthedevelopmentiterationsasthe
projectunfoldsratherthanbeforetheirstart.
AnumberofcasestudiesdemonstratethatInteractionDesignandagilemethodsdonotneedtobeat
oddswitheachother.In[AstonAndMeszarosOnAgileUsability]JaniceAstonandGerardMeszaros
comparetwophasesofanagilewherelightweightusabilitypracticeswereincorportatedintothe
secondrelease.TheyusedpaperprototypingandWizardofOztestingtovalidatetheuserinterface
designandsawsignificantimprovementsinusabilityanduseracceptance.Thebenefitswereso
significantthatinasubsequentprojectthebusinesspartnersinitiatedtheprototypingandusertesting
activitieswithoutpromptingfromtheITdepartment.
In[MillerOnCustomerInput]LynnMillerconcludes:TheUsabilityEngineeringteamatAliashasbeen
gatheringcustomerinputformanyyears,butneveraseffectivelyaswhenweworkwithanagile
developmentteam.ForSketchBookPro,wewereabletomaximizethequantityandimpactof
customerinputbyhavingtheinteractiondesignersworkinaparallelandhighlyconnectedtrack
alongsideofthedevelopers.Dailyinteractionbetweenthedevelopersandinteractiondesignerswas
essentialtothesuccessofthisprocess.
In[SyOnAgileUCD],DesireeSydescribeshowAliasSoftware(nowAutodesk)adaptedtheirUCD
processtosupportagiledevelopment.Sheconcludes:
*Agileusercentereddesignresultedinbetterdesignedsoftwarethanwaterfallusercentered
design.Agilecommunicationmodesnarrowedthegapbetweengatheringusabilitydataandactingon
it.
*BecauseAgiledevelopmentishighlyfeedbackdriven,productteamsmayrelyonuseropinionin
situationswhereuserobservationismoreappropriate.Usabilitypractitionerscanbethebestsuited

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 151
membersofanAgileteamtomitigatethisbiasbecauseoftheirskillsingatheringandanalyzinguser
experiencedata.
*ItispossibletousethefamiliararsenalofusabilityinvestigationmethodsonAgileprojects,
includingformativeusabilitytesting,userandtaskanalysis,interviews,andevenfieldbasedworklike
contextualinquiry.Thisisachievedbychangingthetimingandgranularityoftheinvestigations,and
howresultsarereported.
*usabilityanddesignactivitiescanbescopedasincrementalminidesigns.Differentvalidationand
elicitationactivitiescanbeblendedwithinsinglesessionsconductedatausabilitylaborinthefield.
DesignactivitiesoccuratleastoneAgilecycleorsprintaheadofthedevelopmentteaminan
InteractionDesignerTrackseparatefromtheDeveloperTrack.Developersreceivevalidateddesigns.
*Prototypedemonstrationsanddailyconversationhavelargelyreplaceddetaileddocumentssuch
asusabilitytestreportsandUIspecificationswhencommunicatingwiththeproductteam.Documents
arenowwrittenforinteractiondesigners,torecordahistoryofdesigndecisions.
In[McInerneyMaurerOnAgileAndUCD],PaulMcInerneyandFrankMaurerdescribethreecasestudies
ofhowuserexperiencespecialistsintegratedwithagileteamsandreportvarioustechniquesfrompair
observationstoinformalusabilitytestingintheteamroomwiththelatestbuildinvolvingboth
developersandcustomerproxies.
Manynotableadvocatesofusabilitypracticesdescribetechniquesforadaptingthemintoagile
methods.JeffPattonhasawebsite[PattonOnAgileProductDesign]andaYahoo!Groupdedicatedto
thetopicofagileusabilityandproductdesign.
In[MeszarosOnAgileProductDesign]GerardMeszaroswritesthatonmostprojectsthereisplentyof
timetounderstandtheusersandtheirgoalsandtodotheoverallproductdesignactivitiescalledfor
byUxDmethodsbeforeiterativedevelopmentstartsbecausethebusinesscaseneedstobemadeand
fundingneedstobesecuredbeforetheprogrammerscanstartdevelopoment.
LarryConstantinehaspointedoutthatevenwhenthisisntthecase,thereisusuallysomebackend
(ortechnicaldevelopment)thatcanbestartedimmediatelythatdoesnotdependontheinteraction
designortheuserinterface[ConstantineOnAgileUsability].)
In[NielsenOnAgileUsability],JakobNielsen(knownasthekingofusability)writes:Therearegood
reasonstobelievethatusabilityandAgiledevelopmentmethodscanworktogetherandimproveuser
experiencequality:
*Agileoffersmanyopportunitiesforovercomingproblemswithtraditionaldevelopmentmethods
thathavelongimpededusability.
*ApproachingAgilenarrowly,asaprogrammingmethodologyratherthanasystemdevelopment
methodology,threatenstodestroythelastdecade'sprogressinintegratingusabilityanddevelopment.
But,asoutlinedabove,therearewaysaroundeachofthesethreats.Solongasteamsrecognizethe
threatsasexplicitissues,theyneednotharmproductquality.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 152
*Finally,weknowfromourresearchthatmanycompanieshavemadethingsworkswimmingly
oncetheyadaptedtheAgilemethodologytosuitqualityfocusedsystemdevelopment.
ForuserexperiencepractitionerswhosupportAgileteams,themainchangeisinmindset.Having
good,generaluserexperienceknowledgewillhelpyouunderstandhowtochangetraditionaldesign
andevaluationmethodstomeetyourAgileteam'sdifferentfocus.Ultimately,however,youmust
bothbelieveinyourselfandembraceAgiledevelopmentconceptsifyouwanttosucceed.
12.2 RecommendationsforSuccess
HerewesummarizealistofrecommendationsbyJeffPatton,anexpertinUserCenteredDesignand
agilemethods[PattonOnAgileProductDesign]andothers:
Helpproductmanagersunderstandandbalancebusinessandusergoals;Useendusertesting
asawaytogivevoicetoenduserneedstobalancethemwithbusinessgoals[SyOnAgileUCD].
Helpproductdevelopmentteamunderstandtheunderlyingproductexperienceanddefine
interactionandvisualdesignpatterns
Dojustenoughuserresearchanddesigntogetstartedbutplanforongoingusercontactfor
researchandvalidation.
Involvethedevelopmentteamintheinteractiondesign;becomeadesignfacilitatorratherthan
purelyabehindcloseddoorsdesigner.
Focusonbigpicture,helptheproductdevelopmentteamsetthecontext
Validateusabilitybeforedevelopmentthroughtheuseoflowfiprototypingandtesting
[AstonMeszarosOnAgileUsability]
Validateusabilityafterdevelopment(throughlightweightusabilitytestingondeveloped
features)[AstonMeszarosOnAgileUsability]
Userparalleltrackdevelopmenttoworkaheadandfollowbehind[MillerOnCustomerInput]and
allowdevelopmenttoproceedonbackendfunctionalityinparallelwithinteractiondesign
[ConstantineOnParallelUx]toavoidseparateinteractiondesignphase.
Increasethefrequencyandtimingofenduserinvolvement:buildareadysupplyofusersand
usersurrogatesinsideandoutsideofyourorganizationtoleverageforcontinuoususertesting.
Considerschedulingregularusertestsessionsanddeferdecidingwhattotestjustintime.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 153
Chapter13 OperationsManagers
Perspective
Someorganizationsbuildorcommissionsoftwarethattheyinstallandruntoprovideaservicetotheir
customersorinternalusers.Intheseorganizationstheacceptabilityofthesoftwaremaybedetermined
bytwoentirelyseparatecommunities,thosethatbenefitfromthefunctionalitythatthesoftware
providesandthoseforwhomoperatingthesoftwareisacost.Intheseorganizationstheacceptance
decisionmustincludeoperationalcriteriaandtheroleoftheoperationsgroupinthemakingofthe
acceptancedecisionmustbeclear.Thiscategorycouldalsobeexpandedtoencompassothergroups
suchassecurity,dataarchitecture,ITarchitectureandinfrastructurethatprovidestewardshipof
corporateintereststocounterbalancepotentialprojectpressureinducedshortcutssuchassecurity,data
architecture,ITarchitectureandinfrastructure.(SeeChapter14EnterpriseArchitectPerspective.)
SeetheOperationsManagerpersonainthecompanystereotypecalledITBuildingSoftwareforthe
BusinessinAppendixXReaderPersonasforanexample.

Asanoperationsmanageryoulikelyhaveacompletelydifferentperspectiveonthedeploymentofa
neworupgradedsoftwaresystem.Otherpartiescareaboutwhetherthenewfunctionalityworks
properlyandwhetheritwillprovidesufficientbusinessbenefitinthenearfuture;youcareaboutwhatit
willdotoyoursupportcostsandwhetheritwillbesustainableinthelongerterm.Therefore,youridea
ofacceptabilitywillbequitedifferentanditshouldinfluencetheacceptancedecision.
13.1 RoleinAcceptanceDecision
Itiscrucialthatyourroleinthemakingoftheacceptancedecisioniscleartoeveryoneconcerned.Do
youprovideinformationtotheacceptancedecisionmakerorareyouoneofseveralacceptancedecision
makerswhooperatebyconsensus,votingorveto?Orperhapsyouparticipateinthereadinessdecision
beforethesoftwareismadeavailableforacceptancetesting?
Aswithotherformsofacceptancecriteriaitishighlydesirableforthesecriteriatobeclearlydefined
beforethesoftwareisbuiltsoastoavoidunnecessarytest&fixcyclesbeforethesoftwarecanbe
deployed.Itisalsodesirabletobeingabletoconductthetestingincrementallythroughouttheproject
ratherthanjustattheend.IncrementalAcceptanceTestingdecreasesprojectrisksbyuncoveringissues
earlyenoughtohavethemfixedbeforethefinalacceptancetestphasewhichisthenmuchmorelikely
topasswithoutuncoveringanyshowstopperproblems.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 154
13.2 OperationalAcceptanceCriteria
Theactualcriteriathatanoperationsgroupwillusetodetermineacceptabilitywillbedependentonthe
natureoftheorganizationandtheservicesitprovidesaswellastheservicelevelagreementforthe
softwareapplicationinquestion.Thebaselevelrequirementstypicallyinclude:
Installeranduninstallertesting
Autoupdateorpatchinstallation
CompatibilitywithyourSystemsmonitoringframework
Performanceunderexpectedloads
Systemsmanagementscriptsforstartup,restartandshutdown.
Additionalacceptancecriteriamayinclude:
Frequentlyaskedquestionsandanswers
Troubleshootingscriptsforthehelpdesk
Failoverandrecoverytestingforhighavailabilityapplications(includingapplicationshutdown,
OSfailure,hardwarefailure,occasionalconnectivityetc.)
Testingofhowsystemhandlesbad(illformated)data
Penetrationtestingforpublicfacingapplications
Stresstestingforapplicationswithpotentiallylargenumbersofusers
Datamigrationfromapreviousversionofthesystemorfromasystemthisonereplaces
Transactionaldatapreservationandarchiving
ITArchitectureapprovalofthetechnologystack.
Simultaneousmaintenance(performingserviceoperationswithconstrainedornoperformance
degradationofthecurrentsystm)Reliability,availability,meetingcertainSLAs(uptimeetc.)
Selfmonitoring,errordetection,systemlogging
Recoverability(mustverifymultiplefailoverandrecovery),disasterrecovery,databackup&
recoveryetc
Supportdocumentation(preandpostrollout),training
Thesystemmustbemigratedtothenewversionoftheplatform
Multipleversionsofthesystemmustrunsidebyside.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 155
13.3 OtherConsiderations
Asanoperationsteam:
Weshouldcontributetotherequirementsdiscussionsasearlyaspossibletoensurethat
realistictargetsarebeingproposedfortheproduct.Weshouldhelptheproductownerandthe
productdevelopmentteamunderstandoperationalcontexts.
Werecognizecriticalityofoursystemundertestandunderstandhowsignificantlyoperational
outagesaffectourbusinessandreputation.
Weunderstandusagepatternsandfluctuationsbasedonthetimeofday,season,special
events.Theprojectedusagewillmostlikelychangeovertime.
Managingtestenvironmentisimportant.Ourtesterswouldlikelyneedtheflexibilitytomake
tuningandconfigurationchanges.Forexample,manyoperabilityscenariosoftenrequiretesters
topurposefullyconfigurethesystemwrongtoobservetheoutcome.
Thechoiceswemakefortestenvironmentwillreflectacostbenefitanalysisbasedonyourrisk
tolerance.
Testlabproceduresarecrucial.Itisimperativetokeepalogofallchangesthataremadetothe
environmentinorderforustobeconfidentthatwhenacceptancetestingisdone,the
configurationandstateoftheproductinthetestenvironmentisalignedwiththeintended
productionconfiguration.
Wecommunicateregularlywiththedeploymentteamandtheproductdevelopmentteam.
OperationalacceptancepracticesarediscussedinVolumeIVofthisGuide.Foraserioustreatmentand
discussionofthepatternsforperformanceandoperability,see[PnPonPerfGuide]and
[FordOnOperabilityPatterns].

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 156
Chapter14 SolutionArchitects
Perspective
Thesolutionarchitectistypicallypartoftheproductdevelopmentteam.Theirroleintheacceptance
processusuallyinvolvesapprovingdesignandtechnologydecisions,designingdatamodelsanddomain
models,andensuringthattheproductasbuiltsatisfiestheneedsoftheendusers.Theroletranscends
differentpartsoftheproductdevelopmentteamsorganizationasthesolutionarchitectisresponsible
forthewholesolutionwhereasindividualcomponentteamsorfeatureteamsareeachonlyresponsible
forapart.
14.1 ResponsibilitiesOverview
Asasolutionarchitect,yourjobistoensuretheentiresolutionsatisfiestheproductownerandthe
potentialusersofthesystem.Youresponsibilitiesinclude:
conductingarchitecturalreviewsandaudits;evaluatingarchitecturalalternatives;
ensuringtheuseofagreeduponarchitectureanddesignpractices;
ensuringthatarchitecturaldocumentsareuseful,uptodate,andsufficientlycomplete;
makingdocumentationtradeoffdecisions;
planningarchitecturalmigration;
andcoordinatingandcommunicatingwitharchitectsandotherstakeholdersregarding
architecturalgovernance,compliance,andpolicymatters.
Yourroleintheacceptanceprocesswillprimarilyfocusonthereadinessassessmentactivitiesbutyou
shouldalsobeinvolvedinhelpingtheproductownerunderstandwhattheywanttheproductowner
teamtobuildandfacilitatethecommunicationofthatinformationtotheproductownerteam.Your
involvementinacceptancetestingmaybeminimalbutyoumaybecalledupontointeractwith
acceptancetesterswhichmayincludeenterprisearchitectsassessingcompliancetoarchitectural
guidelinesandstandards.
14.2 UnderstandingtheSolution
Youneedtoworkwiththeproductownertounderstandtheirvisionoftheproductandtohelpthem
understandwhatoptionsareavailabletothemwithrespecttobuildingthatproduct.Lesstechnical
productownersmayneedyourhelptodefinetheproductbasedonmoregeneralneeds.Youwilllikely
wanttoengageuserexperienceexpertsifgoodusabilityisarequirement.Youwillalsoneedtohelpthe
productownerunderstandtherelativecostsofvariousoptionspresentedtothemsotheycanmakethe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 157
righttradeoffsbasedonreturnoninvestment.Thismayrequireunbundlingofsomefunctionalityinto
smaller,moreatomic,featuresthatcanbeprioritizedseparately.
Beawarethattheproductownermaynotbeawareofalltherequirementsfortheproduct;itmaybe
uptoyoutohelpthemengagetheoperationsmanager,securityarchitects,andotherstakeholdersso
thatalltherequirementsfortheproductbecomeknown.Discoveringsecurityrequirementsor
mandatoryservicelevelagreementslateinaproductdevelopmentcycleissuretoresultinalate
product.
Forlargerproductsyoumaybecalleduptohelpmaketheprojectmanagerorproductownerdecide
whethertodividetheproductownerteamintofeatureteamsorcomponentteamsandtosubdividethe
featurebacklogintofeatureteambacklogsordecomposethefeaturebacklogintocomponentbacklogs.
Whereamixoftechnologiesisinvolved(includingmixedhardware/softwareproducts),youmustbe
preparedtobeinvolvedinnegotiatingtheinterfacesbetweenthevarioustechnologies.
Regardlessofthesizeoftheproductownerteam,itisimportantforeveryonetounderstandtheoverall
solutionandhowtheirpartfitsintoit.Thiscanhelpavoidsituationswhereeachcomponentworks
accordingtospecificationbutwhenintegratedfailtoprovidetheseamlessexperiencetheproduct
ownerwouldexpect.
14.3 EnsuringReadiness
Assolutionarchitect,theonusisonyoutodoeverythingyoucantoensurethattheproductisready
beforethereadinessdecisionneedstobemade.Thingswillgoalotsmootherifeveryoneontheteam
understandswhatdonelookslike.Thisisespeciallytrueforcomponentteamswhoseworkneedstobe
integratedbeforeacceptancetestingcanbedone.Withcomponentteams,youmayneedtodoaform
ofacceptancetestingofthecomponentsfollowedbyintegrationtestingoftheintegratedcomponents
beforeyoucanconductreadinessassessment.Thiswillbeeasiertodoifyouhavebeendoingdesign
reviewsofeachofthecomponentsandhavedefinedclearinterfacesandacceptancecriteria.
Someoftheacceptancecriteriamaybederivedfromenterpriserequirements.Whereenterprise
architectsexist,itisimportanttoengagethemearlyintheproductdevelopmentlifecycletoensureyou
understandwhatstandardstheproductneedstocomplyto,whatreusablecomponentsareavailable
and/orareexpectedtobeusedandwhatenterprisedataimpactstheproduct(andviceversa.)Thiswill
avoidnastysurprisesduringreadinessassessmentandacceptancetesting.
14.4 AssessingReadiness
Assolutionarchitect,youwilllikelybeaskedforyouropinionastowhethertheproductis
readyforacceptancetesting.Youllwanttomakethisrecommendationbasedonharddata
andwillthereforewanttoconductarchitecturereviewsandsecuritycodeinspectionsandto
runstaticcodeanalysisandcoveragetools.Youwillalsowanttounderstandhowwellthe
nonfunctionalcharacteristicsofthesystemcomplywiththerequirements;thatwillrequire
testingofthenonfunctionalcharacteristics.Youmaybeinvolvedintheactualtesting,either

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 158
inpersonorthroughplanninganddelegating,oryoumayreviewresultsprovidedby
someoneelsesuchasthetestorganization.Eitherway,youwillwanttoensurethatthe
systemmeetsanyoperationalrequirements,includingSLAs,inadditiontothefunctional
requirements.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 159
Chapter15 EnterpriseArchitects
Perspective
Theenterprisearchitectisresponsibleforensuringconsistencybetweenproductsemployedinorbuiltby
anenterprise.Theirgoalistoreducetotallifecyclecosts(includingoperatingcosts,technologylicensing
costs,etc.)andtomaximizetheROIofanycommoninfrastructure.Theenterprisearchitectoftenhas
vetopowersoverindividualproductsthatfailtocomplywithenterprisearchitectureprinciples.This
sometimemakesthempartoftheacceptancedecisionwhentheproductisbuiltbyanexternalsupplier
andsometimespartofthereadinessdecisionwhentheproductisbeingbuiltinhousebyanIT
department.

Asanenterprisearchitect,yourjobistoensurethealltheproductsusedbytheenterpriseworkwell
togetherandprovideacceptableperformanceatanacceptablecost.Dependingonthenatureofyour
organizationyoumaybecalledupontoprovidedataforeitherthereadinessdecisionortheacceptance
decision.Insomeorganizationsyoumightevenbeaskedtobeinvolvedinmakingtheacceptance
decision.Theroleyouplaywilldependonwhereintheorganizationyoureportrelativetothesupplier
organization.Inoutsourceddevelopmentsituationsyouaremorelikelytobeinvolvedinacceptance
testingandtheacceptancedecision;forinhousesoftwaredevelopmentandproductcompaniesyou
maybemoreinvolvedinreadinessassessment.Whoyouinteractwithwillalsodependonthenatureof
theorganization.Ifthereisasolutionarchitectforeachproduct,muchofyourinteractionwiththe
supplierteamwillbewiththesolutionarchitect.Inothercasesyoumightbeinteractingprimarilywitha
developmentleadoraprojectmanageroreventheproductowner.
15.1 UnderstandingtheSolution
Whileitisprimarilythesupplierteamsresponsibilitytoworkwiththeproductownertounderstandthe
desiredproduct,youmayneedtobeinvolvedinthesediscussionifthesupplierteamdoesnthavea
solutionarchitect.Otherwise,youwouldlikelyworkwiththesolutionarchitecttoundersandhowthe
requirementsofthespecificproductrelatetotherestoftheenterprisearchitecture.Youwould
determine(alongwithSolutionArchitect)whatformsofintegrationarerequiredandhowthat
integrationshouldbeimplementedfromatechnicalpointofview.Youwouldalsocollaborateto
understandhowthenewproductimpactsanycommoninfrastructuresuchasnetwork,applicationand
databaseservers,etc..Iftheproducthasanyimpactonthelongtermvisionoftheenterprise
architecture,eitherbychangingthevisionorbyimplementinganykeycomponents,thatinformation
needstobecommunicatedtoallpartiesconcernedasthelattermaybecomepartofthereadiness
and/oracceptancecriteria.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 160
15.2 EnsuringReadiness
Insomeorganizationsyouwillbeaskedtoprovidereadinessassessmentinformationtothereadiness
decisionmaker.Youwillneedtoengagethesolutionarchitectordevelopmentleadstoensurethey
understandwhatstandardsorguidelinesareapplicabletotheproductandwhatreusablecomponents
shouldbeused.Youmayalsobeinvolvedindefiningextensionstothecorporatedatamodelsto
supportthenewproduct.Itiscrucialthatyoucommunicateanyexpectationsofhowthesupplierteam
wouldengagetheenterprisearchitectsduringthereadinessassessmentand/oracceptancetesting
(whicheverisapplicable.)
15.3 AssessingReadinessorAcceptability
Thenatureofyourorganizationwillinfluencewhetheryouareformallyinvolvedinreadiness
assessmentoracceptancetestingbutthenatureoftheactivitiesyoudowillprobablybethesame
eitherway.Unfortunately,thelateryouareinvolvedthebiggeranimpactanoncompliancediscovery
willhaveontheoveralldeliveryschedule.Thereforeitispreferabletoengagethesupplierteamsearlier,
beforeevenreadinessassessment,toensurethatthereadinessassessmentoracceptance
testing/reviewsarejustaformalsignoffofinformationthatwaspreviouslydiscoveredratherthanan
unpleasantsurpriseforeveryoneinvolved.
Someoftheactivitiesyoumightwanttostartdoingevenwhiledevelopmentisinprogressinclude
designandarchitecturereviewsthatfocusonstandardscomplianceparafunctionalattributessuchas
security,capacity,scalability,andinfrastructureimpact.Insomecasesyoumayevenbecalleduponto
arrangefortestingofparafunctionalcharacteristicsofthesystemsuchaspenetrationtesting.
Ifyoufindyourselfonanacceptancedecisioncommittee,youwillneedtobepreparedtogatherand
reviewtheresultsoftheparafunctionalreviewsandtestingandcastyourvoteontheacceptabilityof
theproductfromatechnicalperspective.Lettheproductownervotebasedonthefunctionality
providedandtheoperationsmanagerontheirsatisfactionwiththeoperationalrequirements.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 161
Chapter16 Legalperspective
Thedecisiontoacceptsoftwareisoftengovernedbytermsinalegallybindingcontract.Theacceptance
decisionmaythereforehavelegalramificationswhichmayoverrideconsiderationsrelatedtofunctional
andnonfunctionalacceptancecriteria.

Disclaimer:Thischapterisnotmeanttobeusedaslegaladvice.

Sinceacceptanceisamajorlegalmilestoneofcommercialsoftwaresystemdevelopment,itisimportant
tokeepinmindthefollowing:
16.1 AcceptanceMayHaveLegalRamifications
Aspartofthecontractualprocess,acceptancetypicallyresultsinrenderingapaymentaswellasit
affectstheapplicationofanywarrantyprovisionsandpotentiallyanyremediesavailabletothe
customer.Whendefiningthecontract,makesurethattheappropriatefactorsaretakeninto
consideration.Therelationshipbetweentheacceptanceoftheproductintoaproductionenvironment
andthecontractualacceptanceoftheproductascompleteshouldbeclearlyspelledout.
Acceptancecanbeapproachedfromtwopositions:explicitprovisionsspecifiedinthesoftware
developmentcontractandthetermsimpliedintothecontractbylaw(whichalsodependsonthe
geographicjurisdiction).Implicitprovisionsmayalsodifferbasedonwhetherthesoftwareisrendered
asaproductorasaservice.Toseewhatprovisions(explicitandimplicit)doapplytoyourcase,consult
yourlegalcounselor.
16.2 ContractShouldStipulateAcceptanceProcess
Whensoftwaredevelopmentisoutsourced,thecontractshouldclearlystipulatetheacceptanceprocess
andhowtheacceptancecriteriawillbedetermined.Itisimportanttoalsospecifyanyexpectations
abouthowmuchreadinessassessmentwillbeconductedbythesupplierandhowthecriteriafor
readinessassessmentwillbedetermined.Forexample,willthecustomerbeprovidingthesupplierwith
theacceptancetests,arepresentativesampleoftheacceptancetestsorleavingthesupplierto
determinewhatteststorun(theBattleshipapproach.)
Thecontractshouldnotjuststipulatewhenthesuppliermustdeliverthesystembutalsohowlongthe
customercantakeforacceptancetesting.Acceptancetestingisusuallytimeboxed.Thecustomeris
givensomuchtimetoacceptsoftwareandreportdeficiencies,afterwhichthesoftwareisdeemed
completeiftherearenoshowstopperbugs.Thecontractmuststipulatethecriteriausedto
determinewhetherabugisindeedashowstopperorgatingbug.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 162
Thecontractshouldalsostipulatewherethedifferentstepsoftheacceptanceprocesswillbecarried
out.Incaseofcustombuildsoftwaresystem,preliminaryacceptancetestingmaybeperformedatthe
locationofsupplier,whilefinalacceptancetesting(bothfunctionalandoperational)isperformedwhen
thesoftwareisdeployedtothecustomer.Thecontractmaystipulatethat,aspartoftheacceptance
procedure,thesoftwaresystembesuccessfullyrunwithcurrentdatainaliveoperatingenvironmentfor
aperiodoftime,withminimumbugs,beforethesystemiscontractuallyaccepted.
Thecontractshouldnotbeonesided;itshouldencouragebothpartiestogettoamutuallyagreeable
acceptanceasquicklyaspossible.Contractsthatpenalizeonepartyoftenleadtodysfunctionalbehavior
thatresultsinloseloseoutcomes.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 163
PartIIReferences
[AstonMeszarosOnAgileUsability]Aston,JaniceandGerardMeszarosAddingUsabilityTestingtoan
AgileProject,Proc.OfAgile2006.

[CecilOnAgileAndUCD]RichardF.Cecil,"ClashoftheTitans:AgileandUCD"
http://www.uxmatters.com/mt/archives/2006/12/clashofthetitansagileanducd.php,Sep20,2009

[ConstantineOnAgileUsability]Constantine,Larry,LucyLockwood,Processagilityandsoftwareusability:
Towardlightweightusagecentereddesignhttp://www.foruse.com/articles/agiledesign.pdf

[ConstantineOnParallelUx]Constantine,Larry,TutorialonAgileMethodsandUsabilityIntegrating
UsageCenteredDesign(slide19),presentedatSoftwareDevelopmentBestPracticesSeptember2005

[DruckerOnManagementChallenges]Drucker,P.ManagementChallengesforthe21stCentury,Harper
Business,2001,p.33.

[EckfeldtEtAlOnTargetCostContracts]Eckfeldt,B.;Madden,R.;Horowitz,J.SellingAgile:TargetCost
Contracts.InProc.Agile
2005.IEEEComputerSociety:160166,2005.

[FordOnOperabilityPatterns]ChrisFordetal.PatternsforPerformanceandOperability:Buildingand
TestingEnterpriseSoftware,Auerbach:2008.

[KohlOnPairTesting]Kohl.J.PairTesting.BetterSoftwareMagazine.
www.kohl.ca/articles/pairtesting.pdf

[LeffingwellOnFeatureVsComponentTeams]DeanLeffingwell,ScalingSoftwareAgility,TheBlog:Best
PracticesforLargeEnterprises,
http://scalingsoftwareagility.wordpress.com/2009/07/15/organizingagileatscalefeatureteams
versuscomponentteamspart1/

[MartinOnProfessionalismAndTDD]Martin,C.ProfessionalismandTestDrivenDevelopment.IEEE
Software24(3):32362007.

[McInerneyMaurerOnAgileAndUCD]PaulMcInerneyandFrankMaurer."UCDinAgileProjects:Dream
TeamorOddCouple?",ACMInteractions,12(6):1923,2005.

[MeszarosOnAgileProductDesign]Meszaros,GerardConcepttoBackWhatHappensBeforeIteration
Zero,http://concept2backlog.gerardm.com.

[MillerOnCustomerInput]LynnMiller,"CaseStudyofCustomerInputForaSuccessfulProduct,"Proc.
AgileDevelopmentConference(ADC'05):225234,2005.

[MillerCarterOnLargeScaleAgile]Miller,A.andCarter,E.AgileandtheInconceivablyLarge,inProc.
Agile2007,IEEEComputerSociety:304308,2007.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 164

[NielsonOnAgileUsability]Nielson,Jakob,AgileDevelopmentProjectsandUsability,
http://www.useit.com/alertbox/agilemethods.html

[PattonOnAgileProductDesign]JeffPatton,"AgileProductDesign",http://www.agileproductdesign.com,
Oct19,2009

[PerryEtAlOnTargetCostContracts]Perry,J.,Barnes,M.Targetcostcontracts:ananalysisofthe
interplaybetweenfee,target,
shareandprice.InJ.Engineering,ConstructionandArchitecturalManagement,7(2):202208,2000.

[PnPOnPerfGuide]patterns&practices:PerformanceTestingGuidanceforWebApplications,Microsoft
Press:2007.Alsoonline:http://www.codeplex.com/PerfTestingGuide

[PoppendiecksOnLean]Poppendieck,M.,Poppendieck,T.ImplementingLeanSoftware
Development:FromConcepttoCash.AddisonWesley,2007

[ShingoOnToyotaProductionSystem]Shingo,S.AStudyoftheToyotaProductionSystem,Productivity
Press,1989.
[SyOnAgileUCD]Sy,DesireeAdaptingUsabilityInvestigationsforAgileUserCenteredDesignJournalof
UsabilityStudies,Volume2,Issue3,May2007,pp.112132
http://www.upassoc.org/upa_publications/jus/2007may/agileucd.html
[VoddeLarmanOnScalingAgile]Larman,CraigandBasVodde.ScalingLeanandAgileDevelopment.
AddisonWesley,2009.
[WittgensteinOnPhilosophicalInvestigations]Wittgenstein,L.Philosophicalinvestigations,3
rd
Ed.,
Oxford:Blackwell,1967.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 165
PartIIIAcceptingSoftware
PartIintroducedseveralmodelstohelpusthinkabouttheroleofacceptancetestingintheoverall
contextofthesoftwaredevelopmentlifecycle.PartIIdescribedtheacceptanceprocessfromthe
perspectivesofcommonlyoccurringrolesplayedbythepeopleinvolvedinmakingtheacceptanceand
readinessdecisionsontypicalprojects.PartIIIaddressestheprocessofacceptingsoftware,applying
theconceptsintroducedinPartsIandII.InPartIII,weoutlinethemainactivitiesthatleadtothe
acceptancedecisionanddescribestrategiesfortheireffectivemanagement.

Chapter17PlanningforAcceptanceintroducesthepracticesusedbyoneormoreoftheroles
describedinPartItoplantheactivitiesthatwillprovidethedecisionmakerswiththedatathatthey
needtomakeanacceptancedecision.
Chapter18AssessingSoftwareintroducestheacceptancetestingandverificationtechniquesusedin
softwareassessment,withparticularemphasisontestdefinition,executionandmaintenance.It
addressesbothfunctionalandnonfunctionalacceptancetesting.
Chapter19ManagingtheAcceptanceProcessintroducesthepracticesusedtomanageacceptance
testexecutionandacceptancedecisionmakingandaddressesthetradeoffsbetweendifferent
approachestotheseactivities.
Chapter20StreamliningtheAcceptanceProcessdescribestheorganizationoftheacceptanceprocess
tooptimizetheacceptancescheduleandtheresourcesusedintheacceptancedecisionbyfocusingon
threemainstrategies:incrementaldevelopment,wastereductionbyavoidingoverproductionand
concurrentexecutionoftheactivitiesunderlyingsoftwareacceptance.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 166
Chapter17 PlanningforAcceptance
PartIintroducedseveralmodelstohelpusthinkabouttheroleofacceptancetestingintheoverall
contextofthesoftwaredevelopmentlifecycle.Italsointroducedtheabstractrolesofthepeopleinvolved
inmakingtheacceptanceandreadinessdecisions.Thischapterintroducespracticesusedbyoneormore
ofthoserolestoplantheactivitiesthatwillprovidethedataonwhichtheacceptancedecisionmaker(s)
canbasetheirdecision.
Acceptanceisanimportantpartofthelifecycleofaproduct;itisimportantenoughthatitshouldbe
theresultofacarefullythoughtthroughprocess.Thetestplanistheendresultofallthisthinking.Like
mostsuchdocuments,itcanserveanimportantroleincommunicatingtheplansfortesting,butthereal
valueliesinthethinkingthatwentintoproducingit.
Testplanningbuildsontheworkdoneduringprojectchartering,whichdefinestheinitialprojectscope.
Intestplanning,youwedefinethescopeofthetestingthatwillbedone,selecttheteststrategy,and
drilldowntodetailedtestingplansthatdefinewhowilldowhat,when,andwhere.
Mostprojectsprepareatestplanthatlaysoutthefollowing,amongotherthings:
Thescopeoftheacceptanceprocessandthebreakdownintoreadinessassessmentandacceptance
testingphases
Theoverallteststrategyincludingbothmanualandautomatedtesting
Theactivities,testingandotherwise,thatwillbeperformedineachphase(thereadinessphaseand
theacceptancephase)
Theskillsthatarerequiredtoperformtheactivities
Theotherresourcesthatwillbeutilizedtocarryouttheactivities(suchasfacilitiesandequipment)
Thetimeframeandsequence,ifrelevant,inwhicheachactivitywillbeperformed

17.1 DefiningTestObjectives
Weneedtohaveaclearunderstandingofthescopeoftheprojectandthesoftwarebeforewestart
thinkingabouthowwewillacceptthesoftware.Mostorganizationshavesomekindofproject
charteringactivitythatdefinestheproductvisionorscope.Itmayalsoincludeariskassessmentactivity.
Oneformofriskassessmentactivityinvolvesbrainstormingallthepotentiallynegativeeventsthat
couldcausegrieffortheproject.Someofthecommonlyoccurringrisksthatwemayaddressthrough
testingare:
TheProductDevelopmentTeammisinterpretedtheProductOwnersdescriptionresultingina
productthatbehavesdifferentlyfromexpected.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 167
TheProductOwnersdescriptionoftheproductleftoutimportantbehaviorsresultinginmissing
functionality.
TheProductDevelopmentTeamimplementedthebehaviorspoorlyresultinginfrequentcrashesof
theproduct.
Theproductexhibitstherightbehaviorwhenusedbysmallnumbersofusersbutbehaveserratically
orrespondsslowlywhenthenumberofusersortransactionsincreasestoexpectedlevels.
Foreachpossibleeventweclassifyeachofthelikelihoodandtheimpactaslow,medium,orhigh.
Anythingrankedmedium/highorhigh/highneedstobeaddressed.Somerisksmaycauseustochange
thewayweplanourproject,butotherrisksmaycauseustotakeonspecifictestplanningactivities.
Together,thevision/scopeandriskassessmenthelpdrivetheteststrategydefinitionandtestplanning.
17.1.1 TraditionalApproachestoTesting
Traditionally,mostproductsaretestedfromtheusersprespective.Thatis,oncetheproductisfinished
theproductsbehaviorisverifiedviawhateverinterfacestheuserswilltypicallyuse.Often,verylittleif
anytestingisdoneofthepartsofthesystembeforeitisassembledintothefinalproduct.Thisresultsin
adistributionofteststhatlookslikeFigureAInvertedTestPyramid.

FigureA
InvertedTestPyramid

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 168
Thisfigureshowsahighlevelviewofhowvariouskindsoftests(andconsequently,testeffort)are
typicallydistributed(basedontheleveloftestgranularityfromunittocomponent/subsystemto
system/wholeproduct)whentestingisdoneprimarilybytestersusingthesameinterfacesasendusers.
Systemtestingisdoneatthelevelofthewholeproductandcanincludearangeofsystemsizes:
individualsystemsorintegratedworkflowsthroughmultiplesystemsthathavetointeract.Similarly,
componenttestsareusedfortestingpackages,individualexecutablesorsubsystems;whilethe
granularityofunittestingcanbeonlittlethingslikemethods/functionsorbiggerthingslikemodules
andclasses.SeethesidebardTestingTerminologyPitfallsforadiscussionaboutwhatwecallvarious
kindsoftests.
TheproblemwiththeapproachpresentedinFigureA(wheremosteffortisfocusedonsystemtesting
andverylittleonunittesting)isthatitishardtoverifythatthesoftwarebehavesinallpossible
circumstancesbecausemanyofthecircumstancesaretriggeredinsidethesoftware.Evenwhenthisis
notthecase,muchoflogicwithinthesystemiscumbersometoverifyviatheenduserinterface.This
makesverifyingthebehaviorandfindinganybugsveryexpensiveandtimeconsuming.Mostofthe
behaviorcanbeverifiedandmanyofthebugscanbefoundmuchmorecheaplythroughappropriate
unittesting,componenttestingandsubcutaneoussystemtesting.Thesesortsoftestscanandshouldbe
automatedaswillbediscussedinsection17.2.3AutomatedTestingbelow.
Sidebar:TestingTerminologyPitfalls
WhenpreparingFigureA(andFigure4laterinthischapter)westruggledwithwhattocallthetestsin
thetoplayeroftheupsidedownpyramid.Someofthealternativesweconsideredwere:Acceptance
Tests,WholeProductTests,CustomerTests,FunctionalTests,andIntegrationTests.Whilethese
namesareincommonusetodescribetheteststhatoccupythetoplayer,wewerenthappywithany
ofthesenamesbecausetheydidntclearlyconveythedifferencebetweenthetoplayerandthelayers
belowit.Theproblemwitheachofthesehistoricalnamesisthattheyoftenexisttohelpdifferentiate
onekindoftestsfromanotherkind.Forexample,FunctionalTestsasopposedtowhat?Theobvious
answerisNonfunctionalTests,butwefeelnonfunctionaltestsalsofitintothetoplayerofthe
(inverted)pyramid.IfnotAcceptanceTeststhenwhat?ThelogicalalternativeisnotRejectionTests
butRegressionTests!Acceptancetestshelpeusverifynewfunctionaltyworkswhileregressiontests
verifythatexisitingfunctionalitystillworks.Theacceptancetestsforthisreleasearelikelyaddedto
theregressiontestsuiteforthenextrelease.Sothetermsacceptanceandregressionrefertopointsof
differentiationonthetimehorizon.
Examiningthepotentialnameswiththiscriticallenshelpeduseliminatemanypotentialnamesleaving
usonlywiththosenamesthathelpedusfocusonwhetherweweretestingthefullyassembled
productoritsconstituentparts.ThisledustoWholeProductTest,SystemTestandIntegrationTest.
Unfortunately,thetermIntegrationTestcanbeusedatvariouslevelsdependingonwhetherweare
integratingunits,components,subsystemsorsystems(intoasystemofsystems.)Wholeproductis
probablytheclearestnamebutmanyteamsthinkofthemselvesasbuildingsolutiostobusiness
problemsratherthanproductsforsale.ThisledustofocusonSystemTestsandthecontrastwith
ComponentTestsandUnitTests.

Acceptanc
Therear
vsScript
scripted
tomake
testing.
usedthe
17.1.2
Theupsid
unitandc
Theagile
qualityso
developm
BrianMar
understan
Figure1
Purposeo
Thediagra
Whet
Whet
th
ceTestEngin
resimilarissu
tedtesting)a
vs.datadriv
esureweund
Theymaywe
esameword.
Flipp
dedowntestp
componentle
softwaredev
oftwarewitho
mentlifecycle
rick,hasdefin
ndthepurpos
ofTests
aminFigure
therthetests
therthetests
heproductaft
eeringGuide
uesintheterm
ndhowtests
venvs.keywo
derstandwha
ellmeansome
.
pingthePy
pyramidcan
evel.Butwha
velopmentco
outsignificant
e.Thishasled
nedamodel
sebehindthe
1classifiesva
arebusiness
areintended
teritisbuilt
eVolumeI,
minologyaro
sareautomat
orddriven,etc
tsomeonem
ethingentire
yramidRig
beturnedrig
atkindsoftes
mmunityhas
tlyincreasing
dtoarethinki
[MarickOnTe
edifferentkin
arioustypeso
facingortec
dtosupportd
ReleaseCand
undhowtest
ted(Handscr
c.)Themoral
meanswhenth
lydifferentfr
ghtSideU
ghtsideupby
stingcanbep
sshownthat
theeffortby
ingoftherole
estingDimens
ndsoftestsw
oftestingwe
chnologyfacin
development
didate1
tsareprepare
riptedorprog
ofthissideb
heyuseoneo
romwhatwe
Up
ymovingmuc
pushdown?
itispossiblet
yintegratingt
eoftesting(t
ions](seeFig
wecouldexec
candoalong
ng
(byhelpingth
edandexecu
grammedvs.
aristhatwe
oftheseword
wouldhave
chofthetesti
toproduceco
testingthroug
theactivity)a
gure1)thath
ute.
twokeydim
hemgetitrig
ted(Explorat
record/playb
needtobeca
dstodescribe
meanthadw
ngactivityto
onsistentlyhi
ghoutthe
ndoftesttea
helpsus
ensions:
ght)ortocriti
169
tory
back,
areful
e
we
othe
gh
ams.

ique

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 170
17.1.3 TestsThatSupportDevelopment
TestscansupportdevelopmentbyhelpingtheProductDevelopmentTeamunderstandwhatthe
productissupposedtodobeforetheybuildit.Thesearetheteststhatcanbepreparedinadvanceand
runaswebuildtheproduct.Aspartofthereadinessassessment,theProductDevelopmentTeamcan
runtheseteststoselfassesswhethertheproductimplementsthenecessaryfunctionality.
Thetestsinthiscolumnfallintotwocategories:thebusinessfacingteststhatdescribewhattheproduct
shoulddointermsunderstandablebythebusinessorProductOwnerandthetechnologyfacingtests
thatdescribehowthesoftwareshouldworkbeneaththecovers.
BusinessFacingTests
Thebusinessfacingteststhatdrivedevelopmentarethesystemaccepatancetests(alsoknownas
customertests).Thesetestselaborateontherequirementsandtheveryactofwritingthesetestscan
exposemissingorambiguousrequirements.Whenwe(ProductOwner,oftentogetherwithsomeone
fromtheProductDevelopmentTeam)preparethesetestsbeforedevelopmentstarts,wecanbesure
thattheProductDevelopmentTeamunderstandswhattheyneedtobuild.ThisisknownasAcceptance
TestDrivenDevelopment.
Ifwepreparethesystemacceptancetestsafterdevelopmentiscompleteorwepreparetheminparallel
withdevelopmentanddonotsharethem(thisisreferredtoas"independentverification"),thetestsdo
nothelpusbuildtherightproduct;instead,thetestsactasanalternativeinterpretationofthe
requirements.Iftheyfailwhenwefinallyrunthem,weneedtodeterminewhichinterpretationofthe
requirementsismoreaccurate:theoneimplementedbythedevelopmentteaminthecodebaseorthe
oneimplementedinthefunctionaltestsbythetestteam.Ifthelatter,timewillbeconsumedwhilethe
developmentteamreworksthecodetosatisfythisinterpretation,reworkthatcouldhavebeenavoided
ifwehadsharedthetests.Eitherway,wehavesetupanadversarialrelationshipbetweendevelopment
andtesting.Itishighlypreferabletopreparethetestsbeforethesoftwareisbuiltsothattestingcan
helpdevelopmentunderstandwhatneedstobebuiltratherthansimplycriticizewhattheyhavebuilt.
Thesetestsmayberunmanuallyortheymaybeautomated.ThelatterallowstheProductDevelopment
Teamtorunthemthroughoutthedevelopmentcycletoensurethatallspecifiedfunctionalityis
correctlyimplemented.TheProductOwnerwillwanttorunadditionalacceptanceteststomakethe
finalacceptancedecision,butsupplyingasetofteststotheProductDevelopmentTeamearlysothey
candrivedevelopmentgoesalongwaytowardbuildingthecorrectproduct.Thisismuchmorelikelyto
happenwhenthetestsareeasyandcheaptorunandthatrequiresautomatedexecution(formore
information,seetheAutomatedFunctionalTestExecutionthumbnail).Thesetestsmaybeimplemented
asprogrammatictests,buttheyaremoretypicallyimplementedusingKeyworddrivenTestAutomation.
TechnologyFacingTests
Therearemanytestsusedbyproductdevelopmentthatarenotbusinessfacing.Developersmay
prepareunitteststoverifythatthecodetheywrotehassuccessfullyachievedthedesignintent.Thisis
howtheydeterminethattheycorrectlybuiltthecode(asopposedtobuildingthecorrectproduct).
Testdrivendevelopment(TDD)iswhendevelopersimplementautomatedunittestsbeforetheybuild

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 171
thecodethetestsverify.Thisdevelopmentprocesshasbeenshowntosignificantlyimprovethequality
ofthesoftwareinseveralwaysincludingbettersoftwarestructure,reducedsoftwarecomplexity,and
fewerdefectsfoundduringacceptancetesting[JeffriesMelnikOnTDDEffectiveness].Thesetestsareever
morefrequentlyautomatedusingmembersofthexUnittestingframework.Formoreinformation,see
[MeszarosOnUnitTestPatterns]
17.1.4 TestsThatCritiquetheProduct
Assumingthattheproducthasimplementedthecorrectfunctionality,weneedtoknowwhetherthe
productmeetsthenonfunctionalrequirements.Thesetestssupporttheacceptancedecision.Wedo
thisbyassessingthenonfunctionalattributesoftheproductafterithasbeen(atleastpartially)built.
Thesetestscritiquetheproductinsteadofdrivingthedevelopmentprocess.Theytelluswhetheritis
goodenoughfromanonfunctionalperspective.Wecandividethesetestsintothetwocategories:
businessfacingandtechnologyfacing.
TechnologyFacingTests
Technologyfacingteststhatcritiquetheproductmeasurehowwelltheproductmeetstechnically
orientedqualityattributes(scalability,availability,selfconsistency,etc.)
Thesetestsprovidemetricswecanusewhendecidingwhethertheproductisreadytobeshipped.In
mostcases,thesetestswillberunaspartofreadinessassessmentbecauseoftheirtechnicalnature.
However,aProductOwnerchargedwithdecidingwhethertoacceptaproductmaybeinterestedin
seeingtheresultsandcomparingthemwiththeminimumrequirement.Theymayevenhireathird
partytestlabtoconductthetestingontheirbehalf.Formoreinformation,seetheTestOutsourcing
thumbnail.
BusinessFacingTests
Ifsystemacceptancetestsareusedtodrivedevelopmenttobuildtheproductpertherequirements,
howdowemakesurewearebuildingtherightproduct?Thebusinessfacingteststhatcritiquethe
productfulfillthisrole.Thesetestsassesstheproduct(eitherasbuiltorasproposed)forfitnessfor
purpose.Usabilitytestsareexamplesofteststhatcritiquetheproductfromabusinessperspective
Typically,thesetestscannotbeautomatedbecausetheyarehighlysubjectiveandsomeevenrequireus
toobservepeopletryingtousetheproducttoachievetheirgoals.
17.2 WhatTestingWillWeDo?AndWhy?
Nowthatwehavebeenintroducedtoawayofreasoningaboutthekindsoftests,wecandecidethe
typesoftestsweneedtorunandwhichteststoautomate.Thisisanoverallteststrategythathelpsus
determinehowtobestaddressourtestingneedsatthelowestcost.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 172
17.2.1 TestStrategy
Definingtheteststrategymaybeconsideredtobepartofthetestplanningprocessoradistinctactivity.
Eitherway,thepurposeofdefiningateststrategyistomakesomehighleveldecisionsaboutwhat
kindsoftestingneedtobedoneandhowtheywillbeexecuted.Teststrategyistightlylinkedtothetest
objectives.Forexample,youwouldhaveadifferentteststrategyifyourobjectivesaretoverifyand
acceptfunctionalityasopposedtoverifycompliance.
Oneofthekeydecisionsiswhatkindsoftestsshouldbeautomatedandwhichapproachtotesting
shouldbeusedformanualtests.Thegoalofthesedecisionsistotrytominimizeprojectriskwhilealso
minimizingthetimeandeffortspenttestingthesoftware.
Previoussectionsintroducedtheconceptsoffunctionalandnonfunctionalrequirements.Aspartofthe
teststrategyweneedtodecidewheretofocus.Clearly,testingcannotprovethatsoftwareworks
correctly;itcanonlyprovethatitdoesnotworkcorrectly.Therefore,wecouldspendaninfiniteamount
oftimetestingandstillnotprovethesoftwareisperfect.Theteststrategyisaboutmaximizingthe
returnoninvestment(ROI)oftestingbyidentifyingthetestingactivitiesthatwillmitigatetherisksmost
effectively.Thisimpliesthatsomerequirementsmaybetestedlessthoroughly,bychoice.
Wealsoneedtodecidewhetherwewilldoalltheacceptancetestingattheendoftheproject(ina
separatetestingphase,orinatestlastorbigbangtestingstyle)orincrementallyasfunctionality
becomesavailable.Thelatterapproach,incrementalacceptancerequireschangesinhowtheprojectis
plannedandhowthesoftwareisdevelopedtoensureacontinuousstreamoffunctionalityisdelivered
startingfairlyearlyintheproject.Thisisthesytlethatwestronglyadvocatesincebigbangtesting
carriestoomanyrisksandisavoidableinmanycontexts.Thepaybackofincrementalacceptanceisearly
discoveryofmisunderstoodandmissedrequirements,therebyallowingtimeforremediationoffthe
criticalpathoftheproject.
Anotherstrategicdecisionmayrelatetotestoracles;oursourceoftruth.Howwillwedefinewhata
correctoutcomelookslike?Isthereacomparablesystemthatwecanuseasanoracle?(Formore
information,seeComparableSystemTestOracle.)Canwehandcraftexpectedresults?(Formore
information,seeHandCraftedTestOracle.)OrwillweneedtouseaHumanTestOracle?Ifso,whatcan
wedofromadesignfortestabilityperspectivetoreducethedependencyonhumantestoracles?
17.2.2 ManualTestingFreedom
Forfunctionaltesting,thekeystrategydecisionsrelatetohowwewillexecutethetests.Whenwe
manuallyexecutethetests,weneedtodecidehowmuchfreedomtograntthetesters.Figure2
containsthe"freedom"scaleusedbyJonandJamesBachtodescribethechoiceswehave.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 173

Figure2
"Freedom"scalefortesters(JonBach,JamesBachusedwithpermission)
Atoneextremeofthetestfreedomscale,thereisfreestyleexploratorytesting,inwhichthetesterscan
testwhatevertheythinkisimportant.Attheotherendofthescaleisscriptedtesting,inwhichtesters
attempttofollowawelldefinedtestscript.Inbetween,thereischarteredexploratorytesting,which
haschartersofvaryingdegreesoffreedomincludingscenarios,userroles/personas,andcharters.
Scriptedtestinginvolveshavinganexpertpreparedetailedtestscriptstobeexecutedmuchlaterby
someoneelse(oracomputerwhenautomated).Thereisverylittleopportunityfortestdesignduring
testexecution.
Exploratorytestingisaneffectiveandefficientapproachtotestingthatleveragestheintelligenceofthe
testertomaximizethebugsfoundinafixedamountoftime.Unlikewithscriptedtesting,testersare
encouragedtoconceivenewthingstotrywhiletheyareexecutingtests.Somepeopledescribeitas
concurrenttestcasedesignandexecutionwithanemphasisonlearning.Basedonasurveyof
companiesusingexploratorytesting,ItkonenandRautiainen[ItkonenRautiainenOnExploratoryTesting,
[ItkonenEtAlOnDefectDetection]reportseveralreasonsforapplyingthisapproach:
whentestingaspectsthatrequiretheusersperspective;
whenwritingsufficientlycomprehensiveautomatedtestsareimpractical;
forleveragingthetestersexperienceandcreativitytodiscoverunusualbehavior;
whenfastfeedbackonproductsbehaviorisrequiredintheabsenceofscriptedtests;
whenrequirementsandfeatureschangerapidly;and
tolearnabouttheproductsbehaviorwhentherequirementsarevague,incomplete,orpossibly
misunderstood.
Formoreinformationonexploratorytesting,see[KanerOnExploratoryTesting]and
[BachOnExploratoryTesting].

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 174
17.2.3 AutomatedTesting
Automatedtestingcoversawiderangeoftopics:fromautomatedexecutiontoautomatedtestcase
generation.Somekindsofnonfunctionaltestsrequireautomatedexecutionbecauseofthenatureof
thetestingbeingperformed.Acommonlyoverlookedareaforautomationistheuseof"powertools"
whileperformingmanualtesting.Toolscanalsobeusedtogeneratetestdataortododatainspection.
Thevarioususesoftestautomationneedtobedeterminedonaprojectbyprojectbasis.Formore
informationaboutthisprocess,seethePlanningTestAutomationthumbnailofthisguideand
[FewsterGrahamOnSoftwareTestAutomation].
MaximizingAutomationROI
AneffectivetestautomationstrategystrivestomaximizetheROIoftheinvestmentinautomation.
Therefore,thetestsweautomateshouldcostless,atleastinthelongrun,thanwewouldhavespent
manuallyexecutingthecomparabletests.Sometestsaresoexpensivetoautomatethatwewillnever
recouptheinvestment.Thesetestsshouldberunmanually.
AutomatingtheRightTests
ToensurethatwegetthebestpossibleROIforourtestautomationinvestment,weneedtofocusour
energiesonthefollowing:
Teststhathavetobeautomatedbytheirverynature
Teststhatareinherentlyeasiertoexecuteusingacomputerthanahuman
Teststhatneedtoberunmanytimes
Tasks(nottests)thatcanmakemanual(orautomated)testingfasterandmoreeffective

17.2.4 AutomatedExecutionofFunctionalTests
Automatedfunctionaltestexecutionisapowerfulwaytogetrapidfeedbackonthequalityofthe
softwareweproduce.Whenusedcorrectly,itcanactuallypreventdefectsfrombeingbuiltintothe
product;whenusedincorrectly,itcanrapidlyturnintoablackholeintowhichtimeandeffortare
sucked.Whenautomatedregressiontestsarerunfrequently,suchasbeforeeverycodecheckin,they
canpreventnewdefectsfrombeinginsertedintotheproductduringenhancementormaintenance
activities.EnsuringtheProductDevelopmentTeamunderstandstheacceptancetestsaheadoftimecan
ensuretheProductDevelopmentTeambuildsthecorrectproductthefirsttimeinsteadofasaresultof
testandfixcycles.Forinformationabouthowthisworks,seetheAcceptanceTestDrivenDevelopment
thumbnailinVolumeII.
Acommonstrategyonprojectsthathaveanextensivesuiteofautomatedtestsistorunthesetestsfirst
asaformofregressiontestasthefirstactivityinatestcycle;thisisaformofextendedsmoketest.This
ensuresthatthesoftwarefunctionsproperly(totheextentoftheautomatedtestcoverage)beforea
humantesterspendsanytimedoingmanualtesting.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 175
Thekeytoeffectiveautomatedfunctionaltestingistouseanappropriatetoolforeachtypeoftest
onesizedoesnotfitall.Thetwomostcommonapproachestoautomatedtestpreparationaretest
recording(seetheRecordedTestthumbnail)andtestscripting.Recordedtestsareeasytoproduce,but
theyareoftenhardtomaintain.Scriptedtestscaneitherbeprogrammatictestautomation,which
involvestechnicalpeoplewritingcodetotestthecode,orkeyworddriventestautomation,whichnon
technicalpeoplecanusetowritetestsusingamuchmoreconstrainedtestingvocabulary.Because
keyworddriventestsaretypicallywrittenintheubiquitouslanguagedefinedfortheproduct,theyare
alsomucheasiertounderstandthanmostprogrammatictests.Whateverapproachweuse,weshould
thinkbeyondtheinitialtestauthoringandalsoconsiderthelifecyclecostsofthetests.Recordedtest
toolsdohavesomevaluableuses.Theycanbeusedtoquicklyrecordthrowawaytestsuitestosupport
thedevelopmentteamwhiletheyrefactortestabilityintotheproductundertest.Theycanalsobeused
inarecordandrefactorstyleasawayofquicklybuildingupacollectionofkeywordsortestutility
methodstobeusedinkeyworddriventestsorprogrammatictests,respectively.
Keyworddriventestinginvolvesspecifyingtestscriptsinanonprogrammaticstyle.Thestepsofthetest
aredatainterpretedbyakeywordlanguageinterpreter.Anotherstyleofdatadriventestautomationis
thereuseofatestscriptwithmultipledatasets.Thisisparticularlyeffectivewhenwecangeneratetest
data,includinginputsandexpectedoutputs,usingacomparablesystemtestoracle.Thenwerunthe
datadriventestonetimeforeachsetofinputs/outputs.Commercialrecordedtesttoolstypically
providesupportforthisstyleoftestingandoftenincludeminimalsupportforrefactoringofthe
recordedtestscriptsintoparameterizedscriptsbyreplacingtheconstantvaluesfromtherecordedtest
withvariablesorplaceholderstobereplacedbyvaluesfromthedatafile.
TestAutomationPyramid
Thetestautomationpyramidisagoodwaytovisualizetheimpactofdifferentapproachestotest
automation.Whentestautomationisanafterthought,thebestwecanusuallydoistousegraphical
userinterface(GUI)basedtestautomationtoolstodrivetheproductundertest.Thisresultsina
distributionoftestsasshowninFigureAInvertedTestPyramid.
Frequently,thesetestsareverydifficulttoautomateandverysensitivetoanychangesinthe
application.BecausetheyrunthroughtheGUI,theyalsotendtotakealongtimetoexecute.Soiftest
automationisanafterthought,weendupwithalargenumberofslow,fragiletests.
Animportantprinciplewhenautomatingtestsistousethesimplestpossibleinterfacetoaccessthe
logicwewanttoverify.Projectsthatusetestdrivendevelopmenttechniquesattackthisproblemat
multiplelevels.Theydodetailedunittestingofindividualmethodsandclasses.Theydoautomated
testingoflargergrainedcomponentstoverifythattheindividualunitswereintegratedproperly.They
augmentthiswithsystemtests(whichincludeusecaseorworkflowtests)atthewholeproductlevel.At
eachhigherlevel,theytrytofocusontestingthosethingsthatcouldnotbetestedatthelowerlevels.
Thisleavesthemwithmuchfewersystemteststoautomate.ThisisillustratedinFigure4ProperTest
Pyramid

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 176

Figure4
ProperTestPyramid
Onewaytoreducetheeffortinvolvedistominimizetheoverlapbetweentheunittestsandcomponent
testswiththesystemtests.Aspecificexampleofthisistheuseofbusinessunitteststotestbusiness
logicwithouthavingtogothroughtheuserinterface.Anothertechniqueistheuseofsubcutaneous
(literally,undertheskinmeaningbehindtheGUI)workflowteststotestbusinessworkflowswithout
beingforcedtoaccessthefunctionalitythroughtheuserinterface.Bothoftheseapproachesrequire
theproducttobedesignedfortestability.
17.2.5 AutomatedTestingofNonfunctionalRequirements
Manytypesofnonfunctionaltestsrequiretheuseofautomatedtesttools.Manyofthesetoolsare
speciallycraftedforthespecificpurposeofassessingtheproductwithrespecttoaparticularkindof
nonfunctionalrequirement.Commonexamplesincludeperformancetestingtoolsthatgenerateloadto
seehowtheproductcopeswithhightransactionratesorusabilitytestingtoolsthatrecordtestsessions
andanalyzedataonusersperformance(e.g.successfultaskcompletion,timeontask,navigationpaths,
errorratesetc.)
17.2.6 AutomationasPowerToolsforManualTesters
Automatedtestsprovideahighdegreeofrepeatability.Thisworksveryeffectivelyasachangedetector,
butitprobablywillnotfindbugsthathavealwaysbeenthere.Forthatweneedhumantesterswhoare

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 177
continuallylookingforwaystobreakthesoftware.Forhumantesterstobeeffective,theymustbeable
tofocusonthecreativetaskofdreamingupandexecutingnewtestscenarios,notthemundanetasksof
settinguptestenvironments,comparingoutputfiles,orgeneratingorcleansinglargeamountsoftest
data.Alotofthesetaskscanbemadefairlypainlessthroughappropriateuseofautomation.
Wecanuseautomatedscriptsandpowertoolsforthefollowing:
Tosetuptestenvironments
Togeneratetestdata
Tocompareactualoutputfilesordatabaseswithtestoracles
Togenerateandanalyzetestreports
Toteardowntestenvironments

Ifweneedalargedataset,wecanwriteaprogramtogenerateonewithknowncharacteristics.Ifwe
needtotesthowparticulartransactionsbehavewhentheproductisstressed,wecanwriteaprogram
thatusesupallthememoryordiskspaceorCPUoncommand.Theseareallexamplesofpowertools
thatmakehumantestersmoreeffective.ForspecificexamplesseeaseriesofVisualStudioTeamTest
walkthroughs[SterlingOnTestTools].

17.2.7 AutomatedTestGeneration
Oneofthegrandobjectivesofsoftwaretestingisautomatedtestgeneration.Theindustryisstillsome
distancefrombeingabletopushonebuttonandhaveatoolgenerateandrunallthetestswewillever
need,buttherearesomeselectedsituationswhereautomatedtestgenerationispractical.Oneexample
iscombinatorialtestoptimization.Forexample,ifwehaveamodulewearetestingthattakesfive
differentparameters,eachofwhichcouldbeanyoneoffourvalues,eachofthevaluescausesthe
moduletobehavesomewhatdifferently,butindifferentway.Totestthiseffectively,wewouldhaveto
test1024(4*4*4*4*4)differentcombinations,whichisnotverypractical.Wecanuseatoolthat
analysesthefivedimensionsandgeneratesaminimalsetoffivevaluetuplesthatwillverifyeach
interactionofaparticularpairofvaluesatleastonce(seeallpairstestgenerationandstressfulinput
testgenerationin[BachOnTestTools]includesall).
AnotherformoftestgenerationisModelBasedTesting(MBT).InMBT,webuildamodelofthesalient
aspectsoftheproductundertestanduseittoderivetests.Thetestscanbederivedmanuallyorwecan
buildatestgenerationtoolthatanalysesthemodelandgeneratesthetestsneedtoachievefulltest
coverage.Thetoolcansimplygeneratetestscriptsthatwillbeexecutedbyothertoolsorhuman
testers,oritcanexecuteeachtestasitisgenerated.
17.2.8 ReadinessvsAcceptance
AsdescribedChapter1TheAcceptanceProcessandChapter2DecisionMakingModel,the
acceptanceofsoftwarecanbedivided,atleastlogically,intotwoseparatedecisions.Thereadiness

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 178
decisionismadebytheProductDevelopmentTeambeforegivingthesoftwaretotheProductOwner
whomakestheacceptancedecision.Akeydecisionisdeterminingwhichtestsarerunaspartof
readinessassessmentandwhicharerunaspartofacceptancetesting.Inmostcases,thereadiness
assessmentismuchmoreextensivethantheacceptancetesting.Whenfunctionaltestsareautomated,
itislikelytheyruninreadinessassessment,andthesoftwareisnotreleasedtoacceptancetestinguntil
allthetestspass.ThisresultsinabetterqualityproductbeingpresentedtotheProductOwnerfor
acceptancetesting.
17.2.9 ManagingTestDevelopment&Automation
Itisusefultohaveacommonunderstandingofhowthedevelopmentoftestsrelatestothe
developmentoftheproduct.Insequentialprocesses,thedevelopmentofthetestsstartswellafter
productdevelopmentanddoesntinfluenceit.Weprefertothinkoftestdevelopmentasoccurringin
parallelwithproductdevelopmentspecificallysothatitcaninfluencewhatisbuilt.Itisalsousefulto
thinkabouthowthetestsaredefinedandhowthatrelatestotestautomation.Ifautomationisfocused
onentiretestcases,theautomationcannotbeginuntilatleastsomeofthetestcasesarefullydefined.
Analternativeistothinkoftestautomationattheteststeplevel.Thenautomationdevelopmentcan
beginassoonassomeofthestepsrequiredbytestcasesaredefined.Thisalsoallowsthetest
automationdevelopmenttobecarriedoutbypeoplewithdifferentskills(testautomation)thanthose
doingthetestdesign(testing).Theautomationworkcanbecarriedoutinparallelwiththetest
definitionandtheproductdevelopment.Thismakesitpossibletohaveautomatedtestsavailableas
soonastheproductsoftwareisavailableratherthanatsomelatertime.
TheteststepsoractionsshouldbedefinedusingtheUbiquitousLanguageadoptedbyeveryoneinthe
project.ThetestscriptsdefinedusingthesetermscanbeexecutedbyhumantestersoriftheROIis
sufficient,automated.Ifallthestepsneededbyatestscripthaveautomatedimplementations,the
entiretestscriptcanbeexecutedautomatically.Ifonlysomeofthestepsareautomated,ahuman
testermaystillbeneededbutthetimerequiredtoexecutethetestmaybereducedsignificantly.
17.3 WhoWillAccepttheProduct?
Ultimately,theacceptancedecisionbelongstotheProductOwner.Insomecases,theProductOwner
maynotbeasingleperson.Inthesecases,wemayhaveaProductOwnerCommitteethatmakesthe
acceptancedecisionusingsomesortofdemocraticorconsensusbasedprocess.Or,wemayhaveaset
ofacceptancedecisionmakersthateachcanvetoacceptance(seeChapter1TheAcceptanceProcess
inPartI)Inothercases,theProductOwnermaybeunavailable.Inthesecases,wemayneedaProduct
Ownerproxytoactasthe"goaldonor"whobothprovidesrequirementsandmakestheacceptance
decision.TheproxymaybeeitheradelegateselectedbytheProductOwner,amediatorbetweena
groupofcustomers,orasurrogatewhoactsonbehalfofalargegroupofanonymouscustomers.The
latterroleisoftenreferredtoastheproductmanager.Formoreinformationaboutthisprocess,seethe
CustomerProxySelectionthumbnailinVolumeII.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 179
Arelatedquestioniswhowilldotheacceptancetesting?Andbyextension,whowilldothereadiness
assessment.Thisverymuchdependsonthebusinessmodelandthecapabilitiesandskillsetofthe
partiesinvolved.Thesidebar"DecisionMakingModelStereotypes"enumeratesseveralcommon
scenarios.WheneithertheProductDevelopmentTeamortheProductOwnerfeelstheyneedassistance
conductingthereadinessassessmentoracceptancetesting,theymayresorttoaTestOutsourcing
model.Thethirdpartytestlabwoulddotheassessment,butthereadinessdecisionandacceptance
decisionstillbelongtotheProductDevelopmentTeamandProductOwnerrespectively.
17.4 WhenWillWeDotheTesting?
Thetestplanneedstoaddresswhenthetestingwillbedone.Sometestingactivitieswillbedonebythe
ProductDevelopmentTeamaspartofreadinessassessment,whileothersaretheresponsibilityofthe
ProductOwnerwhowillbedecidingwhethertoaccepttheproduct.Thetestplanneedstoincludethis
infurtherdetailtothepointwherewehaveanunderstandingofhowmuchtimeweneedforreadiness
assessmentandacceptancetestingandwhatwewillusethattimefor.Therearetwomainstrategiesfor
whentestingisdoneonaproject.Thesequentialapproachtoproductdevelopmentlifecycleplaces
mosttestingactivitiesintoaseparatetestphaseafteralldevelopmentiscompleted.Theagileapproach
istotesteachincrementoffunctionalityasitiscompleted.
17.4.1 TestPhase
Onewaytoplantestingistodefineatestingphaseoftheprojectafterallnewfunctionality
developmentiscomplete.ThisisillustratedinFigure5TestingPhaseinaSequentialProduct
DevelopmentLifecycle.

Figure5
TestingPhaseinaSequentialProductDevelopmentLifecycle
Thistestingphasetypicallyconsistsofseveraltimeboxedtestcycles,eachofwhichcontainsboth
readinessassessmentandacceptancetestingactivitiesasillustratedinFigure6.

Acceptanc
Figure6
MultipleT
Eachtest
cyclethe
Teamint
question,
ProductO
resultinc
addressed
forthene
decisionm
Withinea
asaPertc
sameplan
goodauto
Becauseo
timesand
activitiest
Thetestr
Evenwhe
involveme
10weekp
ceTestEngin
TestCyclesin
cyclewouldb
ProductDeve
heacceptanc
"Isitgooden
Ownerfeedba
concernsthat
dbytheProd
exttestcycle.
maker(s).
achtestcycle,
chartoraGa
nforeachoft
omatedregre
ofthis,arisk
dfastertimet
tolatertestc
esourcesona
entheyareinv
entwhilethe
projectisillus
eeringGuide
AcceptPhase
befocusedon
elopmentTea
cetesting.Int
noughtoboth
ackonthesof
tneedtobein
uctDevelopm
Werepeatth
,itislikelywe
nttcharttore
thetestcycle
essiontesting
basedapproa
tomarket.W
cyclesortodo
asequentialp
volvedinfles
softwareisb
stratedinFigu
eVolumeI,
eofProject
nonerelease
ammakesare
theearlytest
herhavingth
ftwareeveni
nvestigated.A
mentTeam,a
hisprocessu
ewillhaveap
eflecttiming
es,orwecoul
inplace,we
achtoplannin
emayalsode
otheminear
projectarety
shingoutthe
beingbuiltten
ure7Seque
ReleaseCand
ecandidate(R
eadinessdeci
tcycles,thisr
eProductOw
fweknowth
Anyconcerns
andanewrele
ntilthereleas
predefinedse
andinterdep
ddefineaun
shouldfindfe
ngthesubseq
eliberatelych
rliertestcycle
ypicallyinvolv
requirement
ndstobemin
entialTestSpe
didate1
RC)versionof
isionbeforei
readinessdec
wnerTeamte
herearesome
sthatneedso
easecandida
secandidate
etoftestinga
pendencies.W
niqueplanfor
ewerandfew
quentcyclesc
hoosetodefe
es.
vedmostlyat
satthebegin
nimal.Thesta
ecialistStaffin
fthesoftware
nvolvingthe
cisionismade
estit?"Itisva
edefects.The
oftwarechang
teisbuilt.Th
isacceptedb
activities;thes
Wemaydecid
reachcycle.I
werdefectsea
couldresulti
ersomekinds
thebackend
nningofthep
affingprofile
ngProfile.
e.Withinthe
ProductOwn
ebyanswerin
aluabletoget
etestcyclese
gesarethen
issetsthesta
bytheaccepta
semaybelai
detoreuseth
npractice,w
achtestcycle
nshortercyc
oftesting
doftheproje
project,their
ofahypothet
180

test
ner
ngthe

each
age
ance
dout
he
with
e.
le
ect.
tical

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 181

Figure7
SequentialTestSpecialistStaffingProfile
Thisprojectinvolvestenfeaturesandeachfeaturerequiresonepersonweekofrequirements,one
personweekoftestdesignandonepersonweekoftestexecutionfollowedbyapersonweekofbug
fixingbydevelopmentandonepersonweekofregressiontesting.Notehowthebulkoftheactivity
occursatthebackendoftheprojectandrequirestentestspecialistsforweeks8and10(week9
involvesprimarilydevelopmentresources.)Thisburstynatureoftestingisusuallyaccommodatedby
testerssplittingtheirtimeacrossseveralprojects.Thismakesithardforthemtokeepuptodateon
whatisbeingbuiltandhowtherequirementsareevolvingbecausemultiplexing(taskswitching)isa
formofwaste.(SeethesidebaronMultiplexingvs.UndividedAttention).Also,thisrequires
developmenttobefinishedinweek7toleavetimefortwo1weektestcycleswithaweekofbugfixing
inbetween.
Sidebar:Mutliplexingvs.UndividedAttention
Havingthebulkoftestingactivitydoneattheendoftheproductdevelopmentlifecyclegivesthe
managerstheillusionthattestersexperienceslackintheirworkloadsatdifferenttimesandtherefore
canbeassignedtomultipleprojectsthatwouldenablethemtousethetimemoreefficientlyand
effectively.Notonlydoesthisapproachplaceconstraintsonwhendevelopersreceivefeedbackon
theproduct,butalsoonhowmuchtimetheyhavetoactonthatfeedback.
Multiplexingrequirescontextswitiching.Thecostofcontextswitchingincludestheeffortfor
understandingeachprojectsstatusandreimmersingonselfintheotherprojectcontextaswellas
findingpropertestenvironments,resettingthem,relearningcertainrequirementsoftheproduct,re
establishingcontactwiththerightpeople,andoftenregainingfocusandrethinkingissuesthathave
alreadybeensettled.AllofthesearereferredtobyTomDeMarcoasimmersion[DeMarcoOnSlack].
Multiplexingmayalsorequirecatchingupontheworkdoneinonesabsence.
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10
BugFix
Regression
TestExecution
TestDesign
Requirements

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 182
GeraldWeinbergestimatesthatthecontextswitchingcostofassigninganadditionalprojecttoone
personisamassive20%[WeinbergOnContextSwitching].Forthreeprojects,itsa40%dropin
productivity!
JoelSpolskystestimonyindicatesthatwithsoftwareengineersthecostofcontextswitchingeven
greater75%[SpolskyOnContextSwitching].Spolskyarguesthatsoftwaredevelopmentisthekindof
intellectualactivitywherewehavetokeepalotofthingsinourheadatonce.Themore[information]
you[can]keepinmind,themoreproductiveyouare.Suchinformationincludesvariablenames,APIs,
datastructures,requirements,helperfunctions,testcases,detailsofstacktraces,debugging
information,therepositorystructure,configurationparametersandwheretheyareset.
Researchersincognitivepsychologyandorganizationalbehaviourwhospecificallystudiedsoftware
engineersalsocametothesameconclusion:multiplexingisharmful.Itsoneofthecontributing
factorstotimefamine,whichisdestructivetoindividualslivesandnotinthebestinterestoftheir
employerseither[PerlowOnTimeFamine]and[OcasioOnWorkFragmentation].

Ooopsthepieceofcodeaboveclearlydoesntbelonghere.Itcrawledinfromadifferentproject.
Whennoticedit,Irealizedwhatatravestythiswas!Iwasworkingonademoofthefluent
configurationinterfaceforfortheEnterpriseLibrary5.0projectwhilealsoworkingontheAcceptance
TestEngineeringGuide.
Thiswassuchawonderfulillustrationofanotherpointhowthebugscaneasilycrawlinwhile
multiplexingandnotgivingthefullattentiontooneprojectthatIvedecidedtoleavethisinhereas
acaseinpoint!
Thebottomlineisthis:fulltimeinvolvementinasingleprojectimprovesindividualperformance.

17.4.2 IncrementalTesting
Thealternativetoleavingallthetestingtotheendofaprojectistodoincrementaltestingassoftwareis
developed.ThisrequiresthattheProductDevelopmentTeamorganizesitsworksuchthatthereisa
steadystreamoftestablesoftwareavailablestartingearlyintheproject.ThisisillustratedinFigure8
IncrementalProductDevelopmentLifecycle.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 183

Figure8
TestinginanIncrementalProductDevelopmentLifecycle
Thisincrementalproductdevelopmentlifecycleallowsacceptancetestingtobespreadoutovermostof
theprojectdurationinsteadofbeingjammedintoamuchshortertestingphaseonthecriticalpathnear
theendoftheproject.(Thisisratherdifferentthanthesequentialapproachusedinmanyorganizations;
thesidebarWhatitTakestodoIncrementalAcceptancedescribessomeofthechallengesand
solutions.)Aseachfeatureoriterationisfinished,itisassessedforreadinessimmediatelybythe
ProductDevelopmentTeamandturnedovertotheProductOwnerforacceptancetesting.Ideally,any
bugsfoundarefixedimmediatelyasdescribedinthesidebarIncrementalAcceptanceandBugTracking
onAgileProjectsinChapter8ratherthanbeingstockpiledfortheendoftheproject.
Ifthesametenfeaturesdescribedinthesequentialexamplearebuiltbyanagileteamusing
IncrementalAcceptanceTesting,thetestingstaffprofilelooksmorelikeFigure9AgileTestSpecialist
StaffingProfile.

Figure9
AgileTestSpecialistStaffingProfile.
Onthisprojectwestartoutbyspecifyingtwofeaturesinthefirstweek,designthetestsforthosetwo
featuresinthesecondweekaswellasspecifyingathirdfeature.Inthethirdweekwetestthefirst
featureanddotestdesignforthethirdfeature.Notehowthestaffingrampsuptofourpersonweeksof
0
1
2
3
4
5
1 2 3 4 5 6 7 8 9 10
Regression
TestExecution
TestDesign
Requirements

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 184
workandstaystherefortherestoftheduration.Towardstheendasnewfeaturetestingrampsdown
theslackistakenupbyregressiontestingoftheearlyfeatures.Becausethetestersarepartofthe
developmentteamtheyareabletokeepabreastofhowtherequirementsandproductdesignare
evolving.Sincethesamepeoplewillbedesigningandexecutingtheteststheneedtowritedetailedtest
scriptsarereducedandmoreeffortcanbespentonexploratorytestingandtestautomation.Sincethe
testingisdonesoonafterthecodeisdevelopedthedeveloperscanfixthebugsimmediatelysothereis
noseparatetestingandbugfixphaseattheendthereforedevelopmentcancontinueclosertothe
deliverymilestone.

17.4.3 SessionBasedTestManagement
Analternativetousingaplandrivenapproachwithinthetestcycleistouseamoreiterativestyle
knownasSessionBasedTestManagement.Wecreateaprioritizedbacklogoftestingactivitiesthatwe
addressinaseriesoftestsessions.Asnewconcernsareidentifiedintestsessions,wemayadd
additionaltestactivitiestothetestbacklog.Thekeyistokeepthebacklogprioritizedbythevalueofthe
testing.Thisvalueistypicallybasedontheexpecteddegreeofriskreduction.Thedepthofthebacklog
givesusanideaofhowmuchtestingworkwehaveleft(forexample,bycreatingatestingburndown
chart)andifwearemakingheadwaybyaddressingconcernsorifyouarelosingground(ifthisisthe
case,thebacklogisincreasingindepth).SessionBasedTestManagementiscommonlyusedwith
exploratorytesting.

17.5 WhereWillWeDotheTesting?
Thetestplanneedstoidentifywherethetestingwillbeperformed.Whenallthetestingwillbe
performedinhouse,theprimaryconsiderationiswhichphysical(orvirtual)environmentswillbeused.
Thisisparticularlyimportantwhennewenvironmentsneedtobecreatedorsharedenvironmentsneed
tobebooked.Ifwelackphysicalresourcesortheskillstodothetesting,wemaychoosetodotest
outsourcingtoathirdpartytestlab.
Wealsoneedtodefinethecriteriaformovingthesoftwarebetweentheenvironments.Thetransition
fromthereadinessassessmentenvironmenttotheacceptanceenvironmentisgovernedbythe
readinessassessmentcriteria.Whendevelopershavetheirownindividualdevelopmentenvironments,
wealsoneedcriteriaforwhensoftwarecanbesubmittedintotheteam'sintegrationenvironment
wherereadinessassessmentwilloccur.ThesecriteriaareoftenreferredtoastheDoneDoneChecklist
becausethedefinitionof"done"ismorestringentthanwhatadevelopertypicallyreferstoasdone.
17.6 HowLongWilltheTestingTake?
Becausetestingisusuallyonthecriticalpathtodeliveryofsoftwareintensiveproducts,project
sponsorsandprojectmanagersusuallywanttoknowhowlongtestingwilltake.Themostcommon

Acceptanc
answeris
enoughd
flippanta
canonlyd
diminishin
isbarelys
levelofte
Sometest
othertest
dependen
Wealson
ProductD
software
FigureY
Alternatin
Thedead
Teamisp
processo
candidate
Ownerfo
Thedurat
theProdu
fixingdefe
FigureX
ceTestEngin
"Howmucht
atatomakea
sitsoundsbe
disproveitby
ngreturnswi
sufficienttog
estestimation
tsmayhaved
tenvironmen
nciesondate
needtoknow
Development
asillustrated
ngTest&Fix
periodsoccu
rovidedthel
fcharacterizi
ewhichmust
rthenextrou
tionoftheac
uctDevelopm
ectscontinuo
Continuous
eeringGuide
timedoweh
ahighconfide
ecauseofthe
yfindingbugs
llkickinand
getenoughco
ntoestablish
dependencies
ntssuchaswh
.
howlongwe
Team.Willth
inFigureY
uriftheaccep
istofmustf
ngandfixing
thenundergo
undofaccept
ceptancepha
mentTeamisn
ouslyanddeli
Test&Fix.
eVolumeI,
have?"Freque
encereadines
enatureofte
.Evenifweh
wewillnotfi
onfidenceabo
thelowerbo
stherestrict
hendoinginte
ellneedtow
herebedead
AlternatingT
ptancedecisio
ixbugsafter
thebugs.Th
oreadinessa
tancetesting.
ase(consistin
notifiedofbu
veringanew
ReleaseCand
ently,thetim
ssoraccepta
sting.Wecan
hadaninfinite
ndmanymor
outthequalit
oundoftheti
whentheyca
egrationtest
aitforanyde
dtimebetw
Test&Fix?
onisthefirst
rwhichtheP
ismaytaked
ssessmentbe
.
gofseveralt
ugsassoonas
wreleasecand
didate1
eavailableis
ncedecision.
nnotproveso
eamountoft
redefects.So
tylevel.Thisr
meandeffor
anberun.The
ingwithothe
efectswenee
eentestcycle
pointatwhic
roductDevel
daysorweeks
eforebeingp
estcycles)ca
stheyarefou
didateonare
notlongeno
Thisanswer
oftwareworks
timetotest,a
o,wehaveto
requiresatle
rtweneedto
esecouldbe
ersystemsor
edaddressed
eswhilewew
chtheProduc
opmentTeam
stodevelopa
resentedtot
anbereduced
und.Thisallow
egularschedu
oughtogathe
isnotquitea
scorrectlyw
afterapoint,
odeterminew
astaminima
expend.
dependencie
theymaybe
tobefixedb
waitforfixed
ctDevelopme
mstartsthe
anewrelease
theProduct
dsignificantly
wsthemtobe
uleasshowni
185
er
as
we
what
l
eson
ythe

ent
e
yif
e
in

Acceptanc
FigureX
Continuou
Thisdepe
ProductD
thesoftw

Ifwearep
cases,we
preparatio
moreinfo
17.7
Alearning
understan
moreeffe
performa
They
w
Te
They
Ef
17.8
Thepurpo
concerns
ceTestEngin
usTest&Fix
endsonthePr
Development
are)assoon
planningtoa
wantsepara
onofthetest
ormation,see
HowWill
gorganization
ndinghowwe
ective.Measu
nce:
measurehow
weknowenou
estStatusRe
measurethe
ffectivenesst
HowWill
oseoftesting
willrequirec
eeringGuide
roductOwne
Teamismad
aspracticable
utomatetest
teestimates
tsbecauseof
theTestAut
lWeDete
nisonethati
ellwearedoi
uringeffective
wfaralongar
ughtomaket
portingthum
effectiveness
thumbnail.
lWeMana
gistoidentify
changestothe
eVolumeI,
r(orproxy)d
eawareofal
e.
ts,weshould
fortheconst
fthedifferent
omationPlan
ermineOu
isconstantly
ngtodayand
enessrequire
eweinexecu
thereadiness
mbnail.
sofourtestin
ageConce
yanyconcern
esoftwareei
ReleaseCand
doingbugtria
llgatingbugs
haveaneffo
tructionofthe
tskillsandkn
nningthumbn
urTestEf
strivingtoim
dtryingnewa
esTestMetric
utingourtest
oracceptanc
ng.Formore

erns?
nswiththeso
therbecause
didate1
geonallnew
(bugsthatm
rtestimatefo
eautomation
nowledgenee
nail.
ffectivene
mprovehowit
approachesto
cs.Thesemet
tplan.Thatis
cedecision?F
information,
oftwareoftho
esomethingw
wlyfoundcon
mustbefixedb
ortheautom
ninfrastructu
ededtodothe
ess?
tworks.Thisi
oseewhethe
tricsmeasure
,howmuchw
Formoreinfo
seetheAsse
osewhomatt
wasincorrectl
cernssothat
beforeaccept
ation.Inmos
reandthe
etwojobs.Fo
involves
ertheymake
twokeyarea
workisleftbe
ormation,see
ssingTest
ter.Manyoft
lyimplement
186

tthe
ting
st
or
us
asof
efore
ethe
these
ted(a

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 187
bug)orbecausetheProductOwnerrealizedthatwhattheyhadrequestedwillnotsatisfythebusiness
need(anenhancementorchangerequest).Thetestplanneedstolayouthowtheseconcernswillbe
managedandtracked;italsoneedstoincludetheprocessfordecidingwhatneedstobechangedand
whatisacceptableasis.
Asweconductthevariousreadinessassessmentandacceptanceactivities,wenoteanyconcernsthat
comeup.Investigatingtheseconcernsmorecloselyrevealsthat,eachconcernfallsintooneofthe
followingcategories:
Bugordefect
Requirementschange
Projectissue
Nonconcern

Figure3illustratesthelifecycleforeachofthemajortypesofconcerns.

Figure3
ConcernResolutionModel
Partoftheadvantageofcategorizingtheseconcernsistohelpidentifyhowtheywillbeaddressed.Each
categoryofconcernshasitsownresolutionprocessandeachmustbeaddresseddifferently.
17.8.1 BugsorDefects
Bugsaredefectsfoundinthesoftwarethatrequireasoftwarechange.Thebugsneedtobeunderstood
wellenoughtomakedecisionsaboutwhattodoaboutthem.Acommonprocessfordoingthisisknown
asBugTriage,whichdividesthebugsintothreecategorieswithrespecttothenextmilestoneorrelease:
MustFix,WouldLiketoFix(ifwehavetime),andWillNotFix.Ofcourse,thesoftwaremustberetested

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 188
afterthefixesaremade,whichiswhytherearetypicallymultipletestcycles.Fixesmayalsoneedtobe
propagatedintootherbranchesofthesoftware.Ifthebugisfoundtoexistinpreviousversionsthatare
currentlyinuse,apatchmayneedtobepreparedifthebugisseriousenough.Securityrelatedbugsare
anexampleofbugsthataretypicallypatchedbackintoallpreviousversionsstillinuse.Likewise,abug
fixedintheacceptancetestbranchmayneedtobepropagatedintothecurrentdevelopmentbranchif
newdevelopmenthasalreadystartedinaseparatedevelopmentbranch.
17.8.2 RequirementsChanges
TheProductOwnermayhaverealizedthateventhoughtheProductDevelopmentTeamdeliveredwhat
theProductOwneraskedfor,itwillnotprovidetheexpectedvalue.TheProductOwnershouldhavethe
rightandresponsibilityformakingthebusinessdecisionaboutwhethertodelaythereleasetomakethe
changeorcontinuewiththelessusefulfunctionality.Oncewevedecidedtoincludeachangeinthis
releasewecantreatitmoreorlessthesameasabugfromatrackingandretestperspective.
17.8.3 OtherIssues
Someconcernsthatareexposeddonotrequirechangestothesoftware.Theymaybeprojectissues
thatneedtobetrackedtoresolution,additionalthingsthatshouldbetested,andsoon.Thesetypically
donotgettrackedinthebugmanagementsystembecausemostprojectshaveothermeansfortracking
them.Otherconcernsmaybenotedbutdeemedtonotbeconcernsatall.
17.9 Summary
Thischapterintroducedtheactivitiesandpracticesinvolvedinplanningatestingeffort.Akeyactivityis
thedefinitionofateststrategybecausethisiswhatguidesusaswestrivetomaximizetheROIofour
efforts.Thereisaplaceforbothautomatedtestingandmanualtestingonmostprojectsbecausethe
twoapproachesarecomplementary.Automatedfunctionaltestsarehighlyeffectivechangedetectors
thatgoalongwaytowardpreventingnewbugsfrombeingintroducedduringsoftwaremaintenance
activities.Theyarealsorepeatable.Carehastobetakentousetheappropriatefunctionaltest
automationtoolstoavoidtheslow,fragiletestsquagmire.Exploratorytestingmanualtestingcanbe
effectiveandefficientwaysoffindingbugs,especiallyunusualones,thosethatarehardtoautomate,
andthosethatstemfromincomplete,misunderstood,orvaguerequirements.Theuseofpowertoolsby
humantesterscansignificantlyincreasetheeffectivenessofmanualtesting.
17.10 WhatsNext?
Inthischapterwehavedescribedthepracticesinvolvedinplanningfortheacceptanceoftheproduct.
Inthenextchapterwewillexaminethepracticesweusewhileexecutingtheacceptanceprocess.
17.11 References
[MarickOnTestingDimensions]Marick,Brianhttp://www.exampler.com/oldblog/2003/08/21/

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 189
[CrispinGregoryOnAgileTesting]Crispin,LisaandJanetGregory.AgileTesting,AddisonWesley,2008.
[MeszarosOnUnitTestPatterns]Meszaros,Gerard.xUnitTestPatterns:RefactoringTestCode.Addison
WesleyProfessional.2007.
[KanerOnExploratoryTesting]Kaner,Cem,JackFalk,andHungQ.Nguyen.TestingComputerSoftware,
2
nd
Edition.Wiley.1999.
[BachOnExploratoryTesting]Bach,James.WhatisExploratoryTesting?
http://www.satisfice.com/articles/etarticle.pdf
[FewsterGrahamOnTestAutomation]Fewster,M.,Graham,D.SoftwareTestAutomation.Addison
Wesley,1999.
[ItkonenRautiainenOnExploratoryTesting]JuhaItkonenandKristianRautiainen,ExploratoryTesting:A
MultipleCaseStudy,InternationalSymposiumonEmpiricalSoftwareEngineering,ISESE2005,1718
Nov.2005,pp.8494.
[ItkonenEtAlOnDefectDetection]Itkonen,J.;Mantyla,M.V.;Lassenius,C.DefectDetectionEfficiency:
TestCaseBasedvs.ExploratoryTesting,inProc.ofFirstInternationalSymposiumontheEmpirical
SoftwareEngineeringandMeasurement,ESEM2007.2021Sept.2007,pp6170.
[JeffriesMelnikOnTDDEffectiveness]Jeffries,R.andMelnik,G.,TDD:TheArtofFearlessProgramming,
GuestEditorsIntroduction,IEEESoftwareSpecialIssueonTestDrivenDevelopment,MayJune2007,
pp.2430.
[SterlingOnTestTools]CharlesSterling,VisualStudioTeamSystem2010TestFeatureswalkthrough
withscreenshots.http://blogs.msdn.com/charles_sterling/archive/2008/11/05/visualstudioteam
system2010testfeatureswalkthroughwithscreenshots.aspx,October20,2009
[BachOnTestTools]JamesBach.RepositoryofTestTools.http://www.satisfice.com/tools.shtml,October
20,2009
[WeinbergOnContextSwitching]GeraldWeinberg,QualitySofwtareManagement,Vol.1:Systems
Thinking,NewYork,NY:DorsetHousePublishing,1992.
[SpolskyOnContextSwitching]JoelSpolsky.HumanTaskSwitchesConsideredHarmful
http://www.joelonsoftware.com/articles/fog0000000022.html,2001,visitedOctober20,2009
[DeMarcoOnSlack]TomDeMarco,Slack,Broadway,2001.
[OcasioOnWorkFragmentation].WilliamOcasio.Towardsanattentionbasedviewofthefirm.
StrategicManagementJournal,18:187206,1997.
[PerlowOnTimeFamine]LesliePerlow.Thetimefamine:Towardasociologyofworktime,
AdministrativeScienceQuaterly,44(1):5781,1999.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 190
Chapter18 AssessingSoftware
Inthepreviouschapterswehaveintroducedmanyoftheconceptsaroundhowweplantheassessment
oftheproductagainsttheMinimumMarketableFunctionality(MMF)andminimumqualityrequirement
set(MQR)towhichwehaveagreed.Inthischapterwewillintroducethevarioustechniquesthatweuse
aswedotheassessmentincludingtestconception,testdesignandtestexecution.
Assessmentisagenerictermwecanusetodescribetheactivities,testingorotherwise,thatweuseto
evaluatethesystemundertest.Someoftheseactivitiesarefocussedonpreparingandexecutingtests
whileothersmaybereviewactivities.Someoftheactivitiesaredoneaspartofreadinessassessmentby
theproductdevelopmentteamwhileothersmaybedonebyorfortheproductownerunderthebanner
ofacceptancetesting.Wherethesamepracticeisusedinbothformsoftesting,themechanicsofthe
practicestypicallydontchangeverymuchalthoughtheemphasisofhowmucheachpracticeisdone
andtheobjectivesofapplyingthatpracticemightvary.
Westartthediscussionwithanoverviewofthelifecycleofanindividualtests,somethingthatboth
functionalandnonfunctionaltestsdoshareandthenmoveintoadiscussionofthepracticesusedin
eachstateoftheindividualtestlifecyclewhereweintroducethetechniquesthatareuniquetospecific
ofkindsofrequirements.
18.1 TheLifecycleofanIndividualTest
Everysingletest,howeversimpleorcomplex,whethermanualorautomated,goesthroughanumberof
stagesduringitslifetime.ThislifecycleisillustratedinFigure1.

Figure1
IndividualTestLifecycle

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 191
Thestatesofthelifecycleare:
Conception Anacceptancetestisconceivedtoaddressaparticularriskand/orachievea
specificgoal.
Authoring/Design Thetestiswritteneitherindetailedstepformorsomekindofoutlineofwhat
needstobedone,bywhom,when,where,andhow.
Scheduling Theexecutionofthetestisplannedorscheduledforaspecifictimeframeand
resources(people,testenvironment(s),etc.)
Execution Thetestisexecutedagainstthesystemundertest.
ResultAssessment Theresultsofthetestareassessedagainsttheexpectations.(Thismayoccuras
partofexecutionorseparately.)
Reporting Theassessedtestresultsareaggregatedandreported.
Actioning Thetestresultsmayresultineitherfurthertestingbeingidentifiedand/orbug
reportsbeingcreatedandtriaged.
Maintenance Eachtestisanassetandmostmustbemaintainedsothatitcanprovidevaluein
thefuture.Sometests,however,e.g.whendoingexploratorytesting,,arenot
meanttobemaintainedorrepeated.
EndofLife Thetesthasoutliveditsusefulness.Itisabandonedandnolongerrunor
maintained.
Notethattestconceptionandtestauthoringareoftenlumpedtogetherunderthelabeloftestdesign.
Someformsoftesting,oftencalledstatictesting,involveinspectingartefactsthatdescribethesystem
undertestratherthanrunningtheactualcode.Theterminologyusedfortheseformsoftestingis
somewhatdifferent(forexample.theyareoftencalledreviewsratherthantests)butforthepurposeof
discussingtestlifecycle,weshallusethetestterminology.
Letsexamineeachofthesestatesinabitmoredetail.
18.1.1 TestConception
Atsomepoint,someonedecidesthatweneedtoverifyoneormoreaspectsofthesystemundertests
behavior;wecallthesethingstotesttestconditions.Atthistimethetestisjustafigmentof
someonesimagination.Itstartsitstransitionfromanimplicitrequirementtoonethatismuchmore
explicitwhenitgetswrittendownorcapturedinadocument.Itmightappearinalistoftestconditions
associatedwithafeature,requirementoruserstory.Typically,itwilljustbeaphraseornamewithno
associateddetail.Nowthatthetestexistsinconcept,wecanstartmovingitthroughitslifecycle.
18.1.2 TestAuthoring/TestDesign
Testauthoringortestdesignisthetransformationofthetestortestconditionfrombeingjustanamed
itemonalistintoconcreteactions.Itmayalsoinvolvemakingdecisionsaroundhowtoorganizetest

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 192
conditionsintotestcaseswhicharethesequencesofstepsweexecutetoverifythem.Notethattest
authoring/designmayhappenlongbeforetestexecutionorconcurrentlywithtestexecution.
18.1.3 TestScheduling
Onceatestcasehasbeenidentifiedandauthoredoratestcharterdefined,wemustdecidewhenitwill
beexecuted.Theschedulemayindicatetheonetimethetestisrunorthefrequencywithwhichitisrun
andthetriggeringmechanism.Itmayalsoidentifywhoorwhatisexecutingthetestandinwhichtest
environment(s).
18.1.4 TestExecution
Onceauthoredandscheduled,weneedtoactuallyexecutethetests.Fordynamicteststhisinvolves
runningthesystemundertest;statictestsinvolveinspectingvariousartefactsthatdescribethe
systemundertestbutdonotinvolvedexecutingthecode.Dependingonthekindoftestinquestion
thetestmaybeexecutedmanuallybyaperson,byanautomatedtestingtool,orbyapersonusingtools
thatimprovetesterproductivitythroughautomation.
18.1.5 ResultAssessment
Dependingonthetoolsinvolved,thepass/failstatusofthetestsmaybedeterminedasthetestsare
executedortheremaybeaseparatesteptodeterminethetestresultsafterthetestexecutionhasbeen
completed.Wedeterminewhetheratestpassedorfailedbyinspectingtheactualresultsobservedand
determiningwhethertheyareacceptable.
18.1.6 TestReporting
Onceasuiteoftestshasbeenexecutedandassessed,wecanreportonthetestresults.Agoodtest
reporthelpsalltheprojectstakeholdersunderstandwheretheprojectstandsrelativetotherelease
gate.Chapter1TheAcceptanceProcessprovidesdetailsonwhatinformationmightaffectthis
decision.Testreportingincludesbothteststatusreportingtoindicatehowmuchtesteffortremainsand
testeffectivenessreportingwhichdescribesourlevelofconfidenceinourtests.
18.1.7 TestActioning
Thepurposeofexecutingtestsistolearnaboutthequalityofourproductsothatwecanmake
intelligentdecisionsaboutwhetheritisreadyforuseorrequiresfurtherdevelopmentortesting.The
AcceptanceProcessdescribestheprocessfordecidingwhetherornottoacceptthesoftwarebutbefore
wecanmakethatdecisionwemayneedtofixsomeofthedefectswehavefound.TheBugTriaging
processisusedtomaketheIsitgoodenough?decisionbydeterminingwhichbugsneedtobefixed
beforewecanrelease.(SeetheDonenessModelformoredetails.)
18.1.8 TestMaintenance
Sometestsareonlyrunoncewhileothersmayneedtoberunmanytimesoverlongperiodsoftime.
Teststhatwillberunmorethanoncemayrequiremaintenancebetweenrunsasaresultofchangesto

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 193
thepartsofthesystemundertestthattheyinteractwith(forexample,thedatabasestate).These
kindsoftestsmaywarrantmoreofanupfrontinvestmenttoensurethattheyarerepeatableand
robust.Testsintendedformanualexecutionwillneedtobeupdatedwheneverthepartsofthesystem
beingtestedundergosignificantchangesinfunctionality.Whereashumantesterscanusuallywork
aroundminorchanges,fullyautomatedtestswilltypicallybeimpactedbyeventhesmallestchanges
(withteststhatinteractwiththesystemundertestthroughtheGUIbeingthemostfragile)and
thereforemayrequiresignificantlymorefrequentmaintenance.
18.1.9 EndofLife
Soonerorlateratestmaynolongerbeworthexecuting.Perhapsthefunctionalityitverifieshasbeen
removedfromthesystemundertestormaybewehavedeterminedthatthefunctionalityiscovered
sufficientlywellbyothertestssothatwenolongergetmuchadditionalvaluefromrunningthistest.At
thispointthetesthasreacheditsendoflifeandnolongerwarrantseitherexecutionormaintenance.
18.2 VariationsinTestLifecycleTraversal
Sometestsspendalotoftimeineachstateofthetestlifecyclewhileothersmaypassthroughthe
statesveryquickly.Forexample,inautomatedfunctionaltestingwemightspendsweekspreparinga
complextest,waitseveralweeksbeforewecanfirstexecuteit,andthenrunitseveraltimesadayfor
manyyears.Incontrast,duringasingleonehourexploratorymanualtestingsession,thetestermay
conceiveofseveraltestconditions,designatesttoexplorethem,learnsomethingaboutthesystem
undertest,conceiveseveralmoretestconditionsanddesignteststoexplorethemalso,allinthe
spaceofafewminutes.Theautomatedtestswillspendmostoftheirlifetimeinthemaintenancestate
whileexploratorytestsareveryethereal;thereisntaconcreterepresentationthatneedstobe
maintained.
18.2.1 HighlyCompressedTestLifecycleExploratoryTesting
ExploratorytestingissummarizedbyCemKanerasSimultaneoustestdesignandexecutionwithan
emphasisonlearning.Fromthisdescriptionitshouldbeclearthatthereisnoclearseparationofthe
variousstagesofthetestlifecyclewhendoingexploratorytesting.Thetesterlearnsaboutthesystem
undertestbyusingitandforminghypothesisabouthowitshouldbehave.Basedonthesehypotheses
thetesterconceivesofoneormoretestconditionstowhichtheymightsubjectthesystemundertest.
Theyrapidlydesign,intheirmindseye,atestcasetheycouldusetoachievethis.Theyexercisethetest
caseandobservetheresulttherebyformingmorehypotheseswhichinturnleadtomoretest
conditions.Thereisnotanattemptmadetohavethetestpersistbeyondthetestsessionunlessit
revealedabug.Thisremovestheneedtodocumentandmaintainthetests(thoughagoodexploratory
testerkeepsasetofnotes/journalofkeypointsanddiscoveriesduringhisorhertesting);theobvious
consequenceisthatexploratorytestingisnotintendedtobeveryrepeatable.
Thisprocessisverylightweightwithverylittleoverheadgettinginthewayofthetesterinteractingwith
thesoftware.Thismakesitpossibleforexploratorytesterstoformulatealotofhypotheses,testthem

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 194
andfindalotofbugsinaveryshortperiodoftime.Thetestertakesnotesastheygofocussedprimarily
onthefollowingoutputs:
1. Whatfunctionalitytheyhavetestedandtheirgeneralimpressionsofwhattheyhaveseen.
2. Whatbugstheyfoundandwhattheyhaddonetocausethem.
3. Whattestconditionstheyhadconceivedthattheywerenotabletogetto.Thesemaybe
usedasthecharterforasubsequentexploratorytestsession.
4. Howmuchtimewasspentactuallytestingversushowmuchwasspentgettingready.This
informationisusefulwhendecidingwhatkindofpowertoolswouldmaketheexploratory
testermoreefficientinthefuture.

Despiteitssomewhatchaoticappearanceexploratorytestingcanbequitedisciplinedandmethodical
eventhoughitisnotveryrepeatable.Exploratorytestingcanrangefromcompletelyunstructuredto
highlydisciplined.Themoredisciplinedformsofexploratorytestinguseasequenceoftimeboxedtest
sessionstostructurethetestingactivities.
Planningofexploratorytestingconsistsofdefininganinitiallistofhighleveltestconditions(asinkinds
ofthingsweshouldtest)foruseastestsessionchartersanddecidinghowmanytestsessionstobudget
forexecutingthecharters.Examplesofchartersmightincludethefollowing:
Pretendthatyouareahackerandtrybreakingintothesystem(apersonabasedcharter)
Tryoutvariationsoftheinvoicingworkflowfocussingonrejecteditems(ascenariobased
charter)
Tryusingtheuserinterfaceusingonlythekeyboard(adevicebasedorpersonabasedcharter).
Tryscenarioswhereseveraluserstryaccessingthesameaccountatthesametime.(ascenario
basedcharteroftencalledtugofwar.)
Thetestchartersareprioritizedbeforebeingscheduledviaassignmenttoatesterexecutingaspecific
testsession.Uponcompletionofthetestsession,thetestermayrecommendadditionalchartersbe
addedtothebacklogofchartersbasedonwhattheyhadlearnedaboutthesystemundertest,business
domain,usersneedsetc.Exploratorytestingisoftendoneinaniterativestylewithmanyofthetest
chartersforlateriterationsbeingdiscoveredduringexecutionofthetestsessionsinearlieriterations.
Thisallowsexploratorytestingtofocusontheareasofthesoftwarethathavebeenfoundtobethe
mostsuspiciousratherthanprovidingthesameamountofeffortforallareasregardlessofthequality
levelactuallyobserved.This,combinedwiththelowoverheadnatureofthesimultaneoustestdesign
andexecution,iswhatallowsexploratorytestingtobesuchaneffectivewayoffindingthebugswe
knowmustlurkinthesoftware.
18.2.2 ASpreadOutTestLifecycleScriptedTesting
Inscriptedtesting,theaveragetestlifecycleismuchlongerthatinexploratorytesting.Thetestsmaybe
conceivedaspartofthetestplanningexerciseorinmoredetailedtestdesignactivities.Theactualtests

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 195
arethendocumentedorprogrammed,andpotentiallyreviewed,oftenbeforethesoftwareisavailable
fortesting.Thetestsmayrequiremaintenanceevenbeforetheyarefirstexecutedagainstsystem
undertestifthedesignofthesoftwarehasevolvedsincethetestsweredesigned.Eventually,we
determinethescheduleforexecutingthetestsandthetestsareexecutedattheappropriatetime
(whichmaybeweeksorevenmonthslater.)Anybugswefindareloggedandthetestresultsare
reportedtothestakeholders.Thebugsareactioned,oftenweeksorevenmonthsaftertheywere
found.Ifthetestsaretoberepeatedatalatertime,usuallyagainstasubsequentversionofthesystem
undertest,thetestsmayrequiremaintenancetotrackchangesinthesystemtheytest.Eventually,
someonedecidesthatthisparticulartestisnolongeraddinganyvalueandthetestisabandoned.
Thiscyclecouldtakeanywhereforseveraldaysorweektomanyyears.ThetestteamforMicrosoft
Officepreparesextensiveautomatedtestscriptstoverifythebehavioroffeaturesinapplicationslike
MSWord.Thetestsforaspecificgenerationoftheproduct(e.g.Word2003)havealifetimeofover10
yearsbecauseofMicrosoftscommitmentof5yearsofmainstreamsupportandafurther5yearsof
limitedextendedsupportforeachbusinessanddevelopmentsoftwareproduct,followedbyaminimum
of1yearofselfhelponlinesupportviatheknowledgebase[Ref
http://support.microsoft.com/lifecycle/].Newbuildsaretypicallycreatedeveryweekthroughthis
productlifetimeandthetestsarerunagainsteachnewbuild.Inthisexamplethemaintenancephaseof
thetestdominatestheindividualtestlifecycle.Microsoftpatternsandpracticesteamusecontinuous
integrationthatpotentiallyproducesseveralbuildsadaywithcontinuousautomatedtestexecution.
18.2.3 IntermediateTestLifecyclesHybridTestApproaches
Thetwoprevioussectionsdescribedthetwoextremesoftestlifecycleduration.Inpractice,thetest
lifecyclescanfitanywherebetweenthesetwoextremes.Therecouldalsobeamixoftestlifecycle
durationseveninthesametestingsession.Forexample,amanualtestercouldbefollowingadetailed
testscriptthatwaswrittenmonthsago.Theynoticesomethingoddthatisntspecificallyrelatedtothe
scriptanddecidetogooffscripttoexploretheoddity.Inthisoffscriptexcursiontheyaredoing
simultaneoustestdesignandexecution(inotherwords,exploratorytesting).Atsomepointtheymay
returntotheoriginalscriptaftereitherconfirmingthatthesystemisworkingproperlyorloggingthe
bugsthattheyhavefound.
18.3 PracticesforAssessingSoftware
InChapter16PlanningforAcceptanceweintroducedmanypracticesinthecontextofplanningthe
readinessassessmentandacceptancetestingactivities.Nowitistimetolookatthepracticesthatwe
usewhiledesigning,executingandactioningtheindividualtests.Atthispointwefocusonthepractices
andnotonwhodoesthem;itreallydoesntmatterwhethertheyaredoneaspartofreadiness
assessmentoracceptancetestingasthepracticesthemselvesarenotchangedbywhenandwhodoes
them.VolumeIIcontainsacollectionofthumbnailsandjobaidesforeachpracticedescribedherewith
VolumeIIIpresentingsampleartefactsofapplyingthosepracticesinaproject.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 196
18.3.1 TestConceptionPractices
Therearequiteafewpracticesforconceivingtestsortestconditions.Somearemorestructuredor
formalthanothers.Theyallsharethegoalofcreatinganextensivetodolistforoursubsequenttest
designandexecutionefforts.Moststartwitheitherrequirements,whetherfunctionalornonfunctional,
orrisks(concernsaboutsomethingthatmightgowrong.)Someofthemorecommontechniques
includethefollowing:
Riskbasedtestidentification
Threatmodeling
Heuristicsorchecklists
UsecasebasedtestingDefinetestsbasedonspecificusecasesofthesystemundertest(see
theFunctionalTestingthumbnail.)
BusinessruletestingDefinetestsforvariouscombinationsofvaluesusedasinputstobusiness
rulesorbusinessalgorithms.
InterfacebasedtestingDefinetestsbasedonthecharacteristicsoftheuserinterface(human
computerinterface)orcomputercomputerinterfaceprotocol.
Scenariobasedtestingusingrealworldusagescenariostoinspirethedesignoftestcases.
SoapoperatestingUsingexaggeratedrealworldusagescenariostoinspirethedesignoftest
cases.
ModelbasedtestgenerationBuildingoneormoremodelsofkeycharacteristicsofthe
systemundertestandgeneratingtestsfromthemodel(s).
GroupBrainstorming.
Paired/collaborativetestingWorkingtogethertodesignbettertestcases.
RiskBasedTestIdentification
Inriskbasedtestidentificationwedoriskmodelingtoidentifyareasrelatingtofunctionalornon
functionalaspectsthatweareconcernedmightnotbeimplementedcorrectlyormighthavebeen
adverselyaffectedbychangestothefunctionality.Weusethisinformationtoidentifytestconditions
wewanttoensureareverifiedbytests.Agoodexampleofakindoftestthatmightbeidentified
throughriskanalysisisthefaultinsertiontest.Forexample,theriskweidentifiedwasThenetwork
connectionfails.WeaskourselvesWhywouldthenetworkconnectionfail?andcomeupwith3
differentpossiblecauses:Unpluggedcable,networkcardfailure,networkcardoutofservicedueto
maintenanceactivityinprogress.Thesearethreetestconditionswewouldwanttoexerciseagainstour
systemundertest.Otherformsofriskmightrelatetopotentialmistakesduringsoftwaredevelopment.
E.g.Thisshippingchargealgorithmisverycomplex.Thismightcauseustodefinealargenumberof
testconditionstoverifyvariousaspectsofthealgorithmbasedonthekindsofmistakesadeveloper
wouldbelikelytomake.Forexampleapplyingthevarioussurchargesinthewrongorder.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 197
ThreatModeling
Anotherformofriskmodelingisthreatmodelingforsecurity.Thepotentialthreatsidentifiedbythe
threatmodelcanleadustochoosefromasetofsecurityassurancepractices.Wemightdefinespecific
penetrationtestscenariostoensurethesoftwarerepelsattemptsatpenetration.Wemightconduct
securityreviewsofthecodebasetoensuresafecodingpracticeshavebeenfollowed.Wecandecideto
doFuzzTestingtoverifythatthesoftwarecannotbecompromisedbyinjectingspeciallychosendata
valuesviauserinputfields.
UseCaseBasedTestIdentification
Whenwehavebusinessrequirementsdefinedintheformofusecaseswecanidentifytestconditions
byenumeratingallthepossiblepathsthroughtheusecaseanddeterminingforeachpathwhatinput
value(s)wouldcausethatpathtobeexecuted.Eachpathconstitutesatleastonetestcondition
dependingonhowmanydistinctcombinationsofinputsshouldcausethatpathtobeexecuted.
InterfaceBasedTestIdentification
Anothersourceoftestconditionsisthedesignoftheinterfacethoughwhichtheusecaseisexercised
whetheritbeauserinterfaceusedbyahumanoranapplicationprogramminginterface(API)or
messagingprotocol(suchasawebservice)usedbyacomputer.Theinterfacemayhaveverydetailed
designintricaciesinadditiontotheelementsrequiredtoexercisetheusecase.Theseintricaciesarea
richsourceoftestconditions.Forexample,auserinterfacemayhavepulldownlistsforsomeinput
fields.Testconditionsforthesepulldownlistswouldincludecaseswheretherearenovalidentries
(emptylist),asinglevalidentry(listof1item)andmanyvalidentries(longlistsofitems.)Eachofthese
testconditionswarrantsverification.
BusinessRulesandAlgorithms
Businessrulesandbusinessalgorithmsareanotherrichsourcefortestconditions.Forarulethat
validatesausersinputsweshouldidentifyatestconditionforeachkindofinputthatshouldbe
rejected.Rulesthatdescribehowthesystemmakesdecisionsaboutwhattodoshouldresultinatleast
onetestconditionforeachpossibleoutcome.Rulesthatdescribehowcalculationsshouldbedone
shouldresultinatleastonetestconditionforeachformofcalculation.Forexample,whencalculatinga
graduatedshippingchargewiththreedifferentresultsbasedonthevalueoftheshipment,wewould
identifyatleastonetestconditionforeachofthegraduatedvalues.
ScenarioBasedTestIdentification
Scenariobasedtestingistheuseofreallifescenariostoderivetests.Therearevariouskindsof
scenarios.Thescenariobasedtestingthumbnaildescribesawiderangeofscenariotypesthis
introductiontouchesononlyafewofthem.Acommontypeofscenariobasedtestisthebusiness
workflowtest.Thesetestsexercisethesystemundertestbyidentifyingcommonandnotsocommon
endtoendsequencesofactionsbythevarioususersofthesystemasaparticularworkitemispassed
frompersontopersonasitprogressestowardssuccessfulcompletionorrejection.Wecanexaminethe

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 198
workflowdefinitionslookingforpointsintheworkflowwheredecisionsaremade,anddefinesufficient
workflowscenariostoensurethateachpathoutofeachdecisioniscovered.
Aparticularlyinterestingformofscenariotestisthesoapoperatestinwhichthetesterdreamsupa
particularlytorturousscenariothattakesthesystemundertestfromextremesituationtoextreme
situation.Thenamecomesfromitssimilaritytoasoapoperatelevisionprogramwhichcondensesmany
days,monthsandpotentiallyyearsofextraordinaryeventsinpeopleslivesintoshortmelodramatic
episodes.Thisformoftestidentificationisgoodforthinkingoutsidethebox.
Scenariosabouthowsoftwareisinstalledbyapurchasercouldleadtoidentificationofpotential
compatibilityissuesandtheneedforcompatibilitytesting.
ModelBasedTestGeneration
Inmodelbasedtestgenerationwebuildadomainorenvironmentalmodelofthesystemundertests
desiredbehaviour(expressedinmathematicaltermsorinsomeotherabstractnotation)anduseitto
generatealltherelevanttestcases.Forexample,whentestingafunctionthattakes4parameterseach
with3possiblevalues,wecouldgenerate81testconditions(3*3*3*3)byiteratingthrougheachvalue
foreachparameter.Forlargeandcomplexsystems,thenumberofsuchcaseswillbehuge.Various
guidingorselectiontechniquesareusedtoreducethetestcasespace.Tofullydefinetheexpected
result,wewouldneedtohaveanindependentwaytodeterminetheexpectedreturnvalue,perhapsa
ComparableSystemTestOracleoraHeuristicTestOracle.Generationofthetestsfromthemodelmay
befullyautomatedormanual.
IdentifyingNonfunctionalTests
Thetestcasesusedtoassesscompliancetothenonfunctionalrequirementscanbeidentifiedinmuch
thesamewaysasthoseforfunctionalityrequirementswiththemaindifferencebeinghowthe
requirementsarediscoveredandenumerated.
OtherTestIdentificationPractices
Allofthesepracticescanbeappliedbyasinglepersonworkingaloneattheirdesk.Butasingleperson
canbebiasedorblindtocertainkindsoftestconditions;therefore,itisbeneficialtoinvolveseveral
peopleinanytestidentificationactivities.Onewaytodothisistoreviewthetestconditionswithat
leastoneotherperson,aformoftestreview.Anotherpracticeispairedtestinginwhichtwopeople
worktogethertoidentifythetestconditions(ortowriteorexecutetests.)Wecanalsousegroup
techniqueslikelisting,brainstormingorcardstormingtoinvolvelargergroupsofpeople
[TabakaOnCollaboration].
Allthesepracticescanbenefitfromthejudicioususeofchecklistsandheuristics.Thesecanbeusedto
triggerthoughtprocessesthatcanidentifyadditionaltestsorentirecategoriesoftests.Theyarealso
usefulwhendoingexploratorytesting.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 199
18.3.2 TestAuthoring/TestDesignPractices
Nowthatwehaveoflistofthingswewanttotest,ourtodolist,wecangetdowntodesigningthetest
cases.Akeytestdesigndecisioniswhetherwewillprepareaseparatetestcaseforeachtestcondition
oraddressmanytestconditionsinasingletestcase.Thereisnosinglebestwayasitdependsverymuch
onhowthetestswillbeexecuted.Whenhumantesterswillbeexecutingthetestcasesitmakesalotof
sensetoavoidexcessivetestenvironmentsetupoverheadbytestingmanytestconditionsinasingle
testcase.Thehumantesterhastheintelligencetoanalysetheimpactofafailedteststepanddecide
whethertocontinueexecutingthetestscript,abandontestexecutionortoworkaroundthefailedstep.
Automatedtestsarerarelythisintelligentthereforeitisadvisabletotestfewertestconditionspertest
case.Inthemostextremecase,typicalwhenautomatingunittests,eachtestcaseincludesasingletest
condition.
TestsasAssets
Testsareassets(notliabilities)thatneedtobeprotectedfromlossorcorruption.Theyshouldbe
managedwiththesamelevelofcareanddisciplinethatisusedformanagingtheproductcodebase.
Thismeanstheyshouldbestoredinaversioncontrolledrepositorysuchasasourcecodemanagement
(SCM)system.Thelineupofteststhatcorrespondtoaparticularlineupofproductcodeneedstobe
labelledortaggedinthesamewaytheproductcodeislabelledsothattheteststhecorrespondtoa
specificproductcodebuildcanberetrievedeasily.
TestsasDocumentation
Regardlessofwhetheratestcasewillbeexecutedbyapersonoracomputer,thetestcaseshouldbe
writteninsuchawaythatapersoncanunderstanditeasily.Thisbecomescriticalforautomatedtests
whenthetestsneedtobemaintainedeitherbecausetheyhavefailedorbecausewearechangingthe
expectedbehaviorofthesystemundertestandweneedtomodifyallthetestsforthechanged
functionality.
Manyofthepracticesusedforidentifyingtestconditionscarrythroughtotestcasedesignbutthe
emphasischangestoenumeratingthestepsofthetestcaseanddeterminingwhattheexpected
outcomeshouldbeforeachtest.Forusecasetestswemustenumeratetheuseractionsthatcausethe
particularpathtobeexercised.Wealsoneedtoincludestepstoverifyanyobservableoutcomesaswe
executethestepsandawayofassessingwhethertheoutcomematchesourexpectations.(Seethe
section17.3.5onResultAssessmentPractices.)
PickinganAppropriateLevelofDetailforTestScripts
Aswedefinethestepsofourtestsitisveryimportanttoensurethatthelevelofdetailisappropriate
forthekindoftestwearewriting.Forexample,inabusinessworkflowtest,eachstepofthetestcase
shouldcorrespondtoanentireusecase.Ifweweretousethesametestvocabularyandlevelofdetail
asinausecasetest,ourworkflowtestswouldbecomeexceedinglylongandreadersofthetestwould
haveahardtimeunderstandingthetest.Thisisaclassicexampleofnotbeingabletoseetheforestfor
thetrees.Soapoperatestsarewrittenmuchlikeaworkflowtestexceptthatthestepsand
circumstancearemoreextreme.Again,eachstepinthetestshouldcorrespondtoanentireusecase.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 200
BusinessRuleTesting
Businessrulestestscanbedesignedmuchlikeusecasetestsbyinteractingwiththesystemundertest
viatheuserinterface.Iftherearealotoftestconditions,itmaymaketestingproceedmuchmore
quicklyifweuseadatadriventestapproachwhereinweenumerateeachtestconditionasarowina
tablewhereeachcolumnrepresentsoneoftheinputsoroutputs.Thenwecansimplywritea
parameterizedtestcasethatreadstherowsfromthetableoneatatimeandexercisesthesystem
undertestwiththosevalues.Anevenmoreeffectiveapproachrequiresmorecooperationwiththe
productdevelopmentteambecauseitinvolvesinterfacingthetestcasethatreadstherowsofthetable
directlytotheinternalcomponentthatimplementsthebusinessrule(wecallthemsubcutaneous
tests).Thisapproachresultsinautomatedteststhatexecutemuchfasterthanteststhatexercisethe
softwarethroughtheuserinterface.Thetestscanusuallyberunmuchearlierinthedesignphaseofthe
projectbecausetheydontevenrequiretheuserinterfacetoexist.Thesetestsaremuchmorerobust
becausechangestotheuserinterfacedontaffectthem.Thesetestsareparticularlywellsuitedtotest
drivendevelopment.

ModelBasedTesting
Amoresophisticatedwayofusingmodelsistogenerateexecutabletestcasesthatincludeinputvalues,
sequencingofcallsandoracleinformationtochecktheresults.Inordertodothat,themodelmust
describetheexpectedbehaviourofthesystemundertest.Modelbuildingiscomplexbutisthekey.
Oncethemodelisbuilt,atool(basedonsomemethodornotation)istypicallyusedtogenerateabstract
testcases,whichlatergettransformedintoexecutabletestscripts.Manysuchtoolsallowthetesterto
guidethetestgenerationprocesstocontrolthenumberoftestcasesproducedortofocusonspecific
areasofthemodel.

UsabilityTesting
Thedesignofusabilitytestingrequiresanunderstandingofthegoalsofuserswhowillbeusingthe
systemundertestaswellasthegoalsoftheusabilitytestingitselfandthepracticestobeused.The
goalsoftestingwillchangefromtestsessiontotestsessionandthepracticeswillevolveastheproject
progresses.Earlyroundsofusabilitytestingmaybefocusedongettingtheoveralldesignrightandwill
involvepaperprototypes,storyboards,andWizardofOztesting.Laterroundsoftestingaremore
likelytoinvolvetestingrealsoftwarewiththepurposebeingtofinetunethedetailsoftheuser
interfaceanduserinteraction.Allofthesetests,however,shouldbebasedontheusagemodelsdefined
aspartofUserModelingandProductDesignpractices.
OperationalTesting
Functionalrequirementstendtofocusontheneedsoftheendusersbutthereareotherstakeholders
whohaverequirementsforthesystem.Theneedsoftheoperationsdepartment,thepeoplewho

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 201
supportthesoftwareasitisbeingused,needtobeverifiedaspartoftheacceptanceprocess.The
specificneedsmayvaryfromcasetocasebutcommonformsofoperationalacceptancetestinginclude:
Testingofinstallers,uninstallersandsoftwareupdates.
Testingofbatchjobsforinitialdataloading
Testingofdatareformattingforupdatedsoftware.
Testingofdatarepairfunctionality.
Testingofstartupscriptsandshutdownscripts.
Testingofintegrationwithsystemmonitoringframeworks.
Testingofadministrativefunctionalitysuchasusermanagement.
Testingdocumentationcontent.
ReducingtheNumberofTestsNeeded
Ifwehavetoomanytestconditionstobeabletesteachone,wecanusevariousreductiontechniques
toreducethenumberoftestconditionswemustverify.Whenwehavealargesetofpossibleinputsto
verify,wecanreducethenumberoftestcasesweneedtoexecutebygroupingtheinputvaluesinto
equivalenceclassesbasedontheexpectedbehaviorofthesystemundertest.Thatis,anequivalence
classincludesalltheinputsforwhichthesystemundertestshouldexhibitthesameorequivalent
behavior(includingbothendstateandoutputs.)Wethenselectafewrepresentativeinputvaluesfrom
eachequivalenceclassforuseinourtests.Whenthevaluesarenumericandordered(e.g.,integersor
reals)wepickthevaluesrightattheboundariesbetweenthedifferentbehaviors,atechniqueknownas
boundaryvaluesanalysis(BVA).Whentheyarenominal,i.e.,theyrepresentclassificationofbehaviours
withnonaturalordering(e.g.,afinitesetofartbitrarystringsorenumerationtypeswithnomeaningful
orderinginthecontext),wecandesignasingledatadriventestforeachequivalenceclassandrunthe
testforeachinputvalueintheclass.
Ifwehaveseveralinputsthatcaneachvaryandwesuspectthatthebehaviorofthesystembasedon
theindividualinputsisnotindependent,weavoidtestingallcombinationsofinputvaluesbyusing
combinatorialtestoptimizationtoreducethenumberofdistinctcombinationswetest.Examples
include:
Manyindependentvariationsorexceptionpathsinausecase.
Manydifferentpathsthroughastatemodel.
Algorithmsthattakemanyindependentinputvaluesthateachaffecttheexpectedoutcome.
Manysystemconfigurationsthatshouldallbehavethesameway.
ThemostcommonvariationofcombinatorialtestoptimizationisknownasPairwiseorAllPairstesting;
itinvolvespickingthesmallestsetofvaluesthatensurethateveryinputvalueispairedwitheveryother
valueatleastonce.Thistechniqueisusedsofrequentlythatthereisa
website[PairWiseOnAllPairsTesting]dedicatedtolistingthemanyopensourceandcommercial

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 202
programsthatexisttohelpuspickthevaluestouse.Ithasbeenshownthatpairwisetestingprovides
bettercoverageandresultsinfewerteststhanrandomtesting.

18.3.3 TestExecutionPractices
Thedetailsofhowthetestsareexecutedvarygreatlyacrossthedifferentkindsoftestsbutseveral
thingsarecommon.Akeyoutcomeoftestexecutionisthedatathatwillbeusedasinputtothe
subsequentreadinessandacceptancedecisions.Thoughvariousprojectcontextswillrequiremuch
moreextensiverecordkeepingthanothers,itisreasonabletoexpectaminimumlevelofrecord
keeping.Thisrecordkeepingconsistsofthefollowingkeypiecesofinformation:
Whattestshavebeenrun,bywhom,whenandwhere?
Whatresultswereobserved?
Howdidtheycomparetowhatwasexpected?
Whatbugshavebeenfoundandlogged?
ThecomparisonwithexpectationsisdescribedinthesectionResultAssessmentPractices.
FunctionalTestExecutionPractices
Thenatureofatestdetermineshowweexecuteit.Dynamictestsinvolverunningthesystemundertest
whilestatictestsinvolveinspectingvariousartefactsthatdescribethesystem.Dynamicteststypically
fallintooneofthefollowingcategories:
AutomatedFunctionalTestExecutionThisinvolvesusingcomputerprogramstorunthetests
withoutanyhumanintervention.Thetestautomationtoolsetsupthetestenvironment,runs
thetests,assessestheresultsandreportsthem.Itmayevenincludeloggingofanybugs
detected.Thetestsmaybestartedbyahumanorbyanautomatedtestschedulerorstarted
automaticallywhencertainconditionsaresatisfiedsuchaschangestothecodebase.
ManualtestexecutionThisinvolvesapersonexecutingthetest.Thepersonmaydoallsteps
manuallyormayusesomeautomationaspowertoolstomakethetestinggofaster.Thehuman
mayadjustthenatureofthetestastheyexecuteitbasedonobservedbehaviorortheymay
executethestepsofatestcaseexactlyasdescribed.
ExploratorytestexecutionThisisaformofmanualtestexecutionthatgivesthetestermuch
morediscretionregardingexactlywhatstepstocarryoutwhiletesting.Eachtestsessionis
usuallyscopedusingatestcharter.
NonfunctionalTestExecutionPractices
Nonfunctionaltestsaresomewhatdifferentfromfunctionaltestsinseveralways.Asintimatedbytheir
name,parafuncationaltestsspanspecificfunctionsofthesystem.Staticnonfunctionaltestsmay
involverunningtoolsthatanalysethesoftwareinquestionortheymanyinvolvereviewerswholookat

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 203
thecodeorotherartefactstofindpotentialdesignorcodingdefects.Dynamicnonfunctionaltestsmay
involverunningtoolsthatinteractwiththesystemundertesttodetermineitsbehaviorinvarious
circumstances.
StaticNonfunctionalTestPractices
Staticanalysisisdonebyexaminingthecodeorhigherlevelmodelsofthesystemtounderstandcertain
characteristics.Specificformsofstaticanalysisincludethefollowing:
ADesign/ArchitectureReviewisusedtoexaminehigherlevelmodelsofthesystemtounderstand
certaincharacteristics.Themostcommoncharacteristicsofinterestincludecapacity/scalability,
responsetimeandcompliancewithstandards(internalandexternal.)
ASecurityReviewisaspecializedformofDesignorArchiecturereviewthatinvolvesexaminingthe
architectureandthecodelookingforwaysamalicioususerorprogrammightbeabletobreakintothe
system.
StaticCodeAnalysisinvolvesexaminingthesourcecodeeithermanuallyorusingtoolstoascertain
certaincharacteristicsincluding:
Reachabilityofcodesegments
Correctusageofkeylanguageconstructssuchastypesafety
DynamicNonfunctionalTestPractices
Performanceandstresstestsaregoodexamplesofnonfunctionalteststhatrequirespecializedtools.
Sometimeswehavecomplexorlongrunningtestproceduresthatexercisethesystemundertestjustto
seewhatwillhappen;thereneednotbeenumeratedexpectationsperse.
18.3.4 ResultAssessmentPractices
Thevalueinexecutingatestcaseistodeterminewhetherornotsomerequirementhasbeensatisfied.
Whileasingletestcannotprovetherequirementhasbeenmet,asinglefailingtestcancertainlyprove
thatithasnotbeenmetcompletely.Therefore,mosttestcasesincludesomeformofassessmentor
checkingofactualresultsagainstwhatweexpect.Thisassessmentcanhappeninrealtimeasthetestis
executedoritmaybedoneafterthefact.Thischoiceisapurelypragmaticdecisionbasedonthe
relativeeaseofoneapproachversustheother.Therearethefollowingthreebasicapproachesto
verifyingwhethertheactualresultisacceptable.
Comparetheactualresultwithapredeterminedexpectedresultusingacomparatorwhichmay
eitherbeapersonoracomputerizedalgorithm.
Examinetheactualresultforcertainpropertiesthatavalidresultmustexhibit.Thisisdone
usingaverifier.
Justrunthetestsandnotchecktheresultsatall.Thismaybeappropriatewhenthetestingis
beingconductedexpresslytogatherdata.Forexample,thepurposeofusabilitytestingisto
findoutwhatkindsofissuespotentialusershaveusingtheproduct.Wewouldntreportan

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 204
individualusabilitytestsessionashavingfailedorsucceeded.Rather,weaggregatethefindings
ofalltheusabilitytestsessionsforaspecificpieceoffunctionalitytodeterminewhetherthe
designofthesystemundertestneedstobechangedtoimproveusability.
UsingComparatorstoDetermineTestResults
Themostconvenientformofassessmentiswhenwecanpredictwhattheactualresultsshouldbeina
highlydeterministicfashion.Themechanismthatgeneratestheexpectedresultissometimescalleda
trueoracleandthereareseveralwaysthetestresultscanbegenerated.Teststhathaveatrueoracle
areveryusefulwhendoingAcceptanceTestDrivenDevelopmentbecausethereisacleardefinitionof
whatdonelookslike.
Whenthereisanexistingsystemwithsimilarfunctionalityandweexpectthenewsystemtoproduce
exactlythesameresults,itmaybeconvenienttousetheexistingsystemasaComparableSystemTest
Oracle.
Whenwehaveanewsystemforwhichnocomparablesystemexists,weoftenhavetodefinethe
expectedresultsmanually.ThisisknownasaHandcraftedTestOracle.Oncethesystemisupand
runningwemayalsohavetheoptionofcomparingsubsequentreleasesofthesoftwarewithprevious
versions,anapproachwecallPreviousResultTestOracle.Thisistheapproachthatmostrecordand
playbackorcapture/replaytesttoolsuse.InsomecasesitmaybeappropriatetoforgotheHand
CraftedTestOracleandwaitforthesystemtogenerateresultswhicharetheninspectedbyaperson,a
HumanTestOracle,beforebeingusedasaPreviousResultTestOracle.Thisapproachforgoesthe
benefitsofAcceptanceTestDrivenDevelopment.
UsingVerifierstoDetermineTestResults
Whenwecannotpredetermineexactlywhattheexpectedresultshouldbe,wecaninsteadexaminethe
actualresultforcertaincharacteristics.Ratherthanhaveanoracledescribewhattheresultshouldbe,
weasktheoracletomakeajudgementastowhethertheresultisreasonablegiventheinput(s).This
approachhastheadvantageofnotrequiringthatwepredeterminetheexpectedresultforeach
potentialinput.Themaindisadvantageisthatwemayacceptsomeresultsthatsatisfytheinvariantbut
whicharenotactuallycorrect.
Forexample,aHumanTestOraclecouldexamineageneratedgraphictodeterminewhethertheycan
recognizeitasanacceptablerenderingofanunderlyingmodel.Oracomputeralgorithmcouldverify
thatwhentheactualresultisfedintoanotheralgorithmtheoriginalinputvalueisrecovered.The
programthatimplementsthisalgorithmissometimescalledaHeuristicTestOracle.HeuristicTest
Oraclesmaybeabletoverifysomeresultsareexactlycorrectwhileforotherresultsitmayonlybeable
toverifytheyareapproximatelycorrect.
Forexample,wecouldwriteatestscriptthatstepsthroughallintegerstoverifythatthesquareroot
functionreturnstherightvalue.Ratherthaninspecttheactualvaluesreturnedbyeachfunctioncalland
comparethemtoahandcraftedtestoracleoracomparablesystemoracle,wecouldinsteadmultiply
theresultbyitselfandverifythatwegetbacktheoriginalnumberwithinaspecifiedtolerance,say,+/
.001.Inthisexample,weshouldalsotestagainstanotherinvarianttoensurethattheactualresultisnot

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 205
negativewhichwouldclearlybeatestfailure.Acomputerizedheuristicoracleissometimescalleda
verifier.
LoggingBugs
Oneofthekeyreasonsfortestingandreviewsistofinddifferencesbetweenexpectationsandreality.
Whenwedofindsuchadifferenceitisimportantthatitbeloggedsothatitcanbefurtherinvestigated,
prioritizedandtheappropriateactiondetermined.Theinvestigationcouldrevealittobeabug,a
misunderstandingbytestersabouthowthesystemwasintendedtobeused,missingdocumentation,or
anyofamyriadofotherreasons.Toavoidpresumingtheoutcome,weprefertocalltheseconcerns.To
allowtheinvestigationtobeconductedefficiently,itisimportanttologthekeycharacteristicsofeach
concern.Ataminimum,weneedtologthefollowing:
1. Theexactstepsrequiredtoreproducetheproblem.Thismayrequirererunningthetesta
numberoftimesuntilwecanconfidentlydescribeexactlywhatittakestocausetheproblem
tooccur.
2. Whatactuallyoccurred.
3. Whatweexpectedtohappen.Weshouldprovideasmuchdetailaswouldbenecessaryfor
thereadertounderstand.Weshouldnotjustrefertoarequirementbutratherdescribe
exactlywhatweexpected,whathappened,andhowwhatactuallyoccurredwasdifferent
fromwhatweexpected.

Everypotentialbugreportshouldbegivenacleartitlethatdescribesspecificallywhatwastested;this
avoidsconfusionbetweenbugswithverysimilartitlesyetcompletelydifferentexpectedandactual
behaviours.
Referto[Kaneretal,Chapter.Reporting and Analyzing Bugs.] on bug reporting guidance and
http://blogs.msdn.com/productfeedback/archive/2004/09/27/235003.aspxforanadditionalexample.

18.3.5 TestMaintenancePractices
Sometestsareonlyintendedtoberunoncewhilesomeareintendedtoberunmanytimesoveralong
periodoftime.Somekindsoftestsholdtheirvaluelongerthanothers;somekindsoftestsdeteriorate
veryquicklybecausetheyaresotightlycoupledtotheSUTthatevensmallchangestotheSUTmake
themobsolete.Teststhatareexpectedtobeusedmorethanoncemaywarrantanupfrontinvestment
toensurethattheyarerepeatableandrobust.
Usefultechniquesformakingtestsmorerobustincludethefollowing:
Buildmaintainabilityintothetests.Writethetestsatappropriatelevelofabstraction.Dont
couplethetesttoanypartofthesystemitisnttesting.Dontprovideanyunnecessaryor
irrelevantdetailinanyofthestepsofthetest.Strivetodescribetheteststepsinbusiness
ratherthantechnicalterms.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 206
Designthesystemundertestfortestability.Designingfortestabilityiscommonpracticein
computerhardwarebutistoooftenneglectedinsoftwaredesign.Designthesystemtomakeit
easytoputitintoaspecificstate.Makeiteasyfortestprogramstointerfacewiththesystem
throughapplicationprogramminginterfaces(API)ratherthanforcingprogramstouse
interfacesintendedforhumans.Writingthetestsbeforethesystemisdesignedisagoodway
toinfluencethedesigntosupporttestability.
Refactortheteststoimprovemaintainability.Testsshouldbeassets,notliabilities.Teststhat
arehardtounderstandarehardtomaintainwhenthesystemundertestismodifiedtomeet
changingrequirements.Automatedtestsinparticularshouldberefactoredtoavoid
unnecessaryduplicationandirrelevantinformation.SeetheTestEvolution,Refactoringand
Maintenancethumbnail.
18.4 Summary
Thischapterintroducedthepracticesweusewhiledefining,executingandmaintainingindividualtests.
Whilesomeofthesepracticesarespecifictofunctionaltestingandothersarespecifictononfunctional
testing,theoveralllifecycleofatestisconsistentforbothcategoriesoftests.Thepracticesusedfor
readinessassessmentaremoreorlessthesameasthoseusedforacceptancetestingalthoughthe
specifictestsproducedforeachwilldependontheoveralltestplanasdescribedinChapter16
PlanningforAcceptance.
18.5 WhatsNext?
Thenextchapterdescribespracticesrelatedtomanagingtheacceptanceprocess,especiallyhowit
relatestomonitoringandreportingontestprogressandtheresultsandmanagingtheprocessof
decidingwhichbugstofix.
18.6 Resources
[PairWiseOnAllPairsTesting]Pairwise.OrgAwebsitededicatedtocatalogingallthetoolsavailablefor
allpairstesting.http://www.pairwise.org/tools.asp
[TabakaOnCollaboration]Tabaka,JeanCollaborationExplainedAddisonWesley.NJAddisonWesley.
NJ[Kaneretal]Kaner,C.etal.TestingComputerSoftware,2/e.Wiley,1999.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 207
Chapter19 ManagingtheAcceptance
Process
Thepreviouschapterintroducedtheindividualtestlifecycleandthepracticestheassessorsusefor
identifyingtestconditions,designingthetests,executingthetestsandassessingtheoutcomes,and
maintainingthetests.Thischapterintroducesthemanagementpracticesweusewhileexecutingthe
testsandwhilemakingtheacceptancedecision.
Theprocessofacceptingsoftwareinvolvesmanyactivitiesthatgeneratethedataweuseasinputinto
theacceptancedecision.Eachoftheseactivitiestakestimeandconsumesvaluablehumanandother
resources.Awellmanagedacceptanceprocesswillusetheseresourceswiselywhileapoormanaged
processhasthepotentialtowastetheseresources,delaytheacceptancedecisionandevencompromise
thequalityofthedecisionmade.
Whileweareexecutingourtestplans,wewanttoknowtheanswerstotwoimportantandinterrelated
questions:
First,howistestingprogressing?Whenwillwehaveahighconfidenceassessmentofthe
productquality?Willitbeintimetomakeourreadinessoracceptancedecision?
Second,whatisourcurrentassessmentoftheproductquality?Isitgoodenoughtorelease
whethertocustomersortotheacceptancetesters?Whatactionsdoweneedtotaketomakeit
goodenough?
19.1 TestSchedulingPractices
Theoverallschedulethatdefineswhichtestswillberunandwhenshouldhavealreadybeendefinedas
partofthetestplan.Thatincludesthestrategicdecisionsaboutwhetheralltestingisdoneduringafinal
testphaseorincrementallythroughouttheproject.Inthissectionwefocusonthetechniquesweuse
forschedulingoftestexecution.Asareminder,undertestexecutionweincludebothdynamictesting
whentestcasesarerunthatinvolveexecutingsoftwareinthesystemundertestandstatictestingthat
isperformedintheformofreviewsorinspectionswithoutactuallyrunningcodeinthesystemunder
test.
19.1.1 PlanDrivenTestScheduling
Thetraditionalapproachtotestschedulinginvolvesdefiningaperiodoftime,sometimescalledatest
cycle,inwhichonecompleteroundoftestingcanbecompleted.Thetestplandefineshowmanytest
cyclesareplanned.Thedetailsofwhathappensineachtestcycleoftenemergeastheprojectunfolds.
Oneapproachistodefineadetailedprojectplanforthefirsttestcycle,onewhichdefinesthesetof
testingactivitiesthatwillbedoneonadaybydaybasis.ProjectmanagementtoolssuchasPertorGantt
chartsmaybeusedtodocumentthisdetailedtestexecutionplan.(SeePlanDrivenTestManagement.)

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 208
19.1.2 SessionBasedTestScheduling
Thealternativetodefiningadetailedprojectplanoftestactivitiesistoscheduleaseriesoftestsessions
andcreateabacklog(prioritizedlist)oftestsessionchartersthatareexecutedinthesetestsessions.
Eachsessionwouldbeofastandardduration,say90minutes,andmostchartersshouldbeexecutable
withinonesession.ThisSessionBasedTestManagementismosttypicallyusedformanaging
exploratorytestingbutcouldalsobeusedforexecutionofscriptedtesting.Wheneachtestsessionis
completedthetestermarksthetestcharterasoneofcompletedorneedsoneormoreadditional
sessions,orsuggestsadditionaltestchartersforfuturetestsession.Thisprovidesagoodindicationof
howtestingisprogressingasillustratedinfigureK(TestCharterBurndown)whichshowsthenumberof
testchartersdroppingovertimebutwiththeoccasionalupwardsspikeasnewtestchartersare
identified.

FigureK
TestCharterBurndown
Inthischart,thebarsrepresentingtheoriginalchartersshowthechartersthatweredefinedbeforetest
executionbegan.ThebarslabelledAddedChartersstackedontopofthemrepresentnewchartersthat
wereidentifiedwhiletestexecutionwasoccurring.TheTotalLeftlineindicateshowmanytestcharters
remaintobefullfiledwhiletheChartersCompletedindicatesthenumberofchartersalreadyfulfilled.
Wecangetagoodideaatanypointduringtestexecutionofwhentestingwillbecompletedby
projectingtheTotalLeftlineouttotherightuntilitintersectszerocharters.Thatistheearliestprobably
completiondate.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 209
19.1.3 SelfOrganizedTestScheduling
Athirdapproachisoftenusedbyagileorselforganizingteams.Allthetestingtasksarepostedonawall
chart,variouslycalledabigvisiblechart[JeffriesOnBigVisibleCharts]oraninformationradiator
[CokburnOnAgileDevelopment],andteammemberssignuptodospecifictestingtasks.Astheyfinish
onetask,theymarkitdoneandpickanothertasktoperform.ThisiscalledSelfOrganizedTest
Scheduling.
19.1.4 EventTriggeredTestScheduling
Someformsofautomatedtestingcanbesetuptorunautomaticallyatcertaintimes(suchasevery
nightat2am)orwhencertaineventsoccur(suchaswhensomeonechecksinachangetopartofthe
codebase.)Theresultscanbepostedonawebsite,automaticallyemailed/IMdtoallteammembers,
orcommunicatedbyamulticoloredlightdisplayorlavalampsintheteamworkspaceorbyaniconin
everyonescomputerssystemtrayorsidegadget(foronesuchtoolseeTeamBuildMonitor
[LambOnTeamBuildMonitor]).ThispracticeisoftencalledContinuousIntegration
[MartinFowlerOnContinuousIntegration]orCIforshort.Teamsthatusecontinuousintegrationtypically
adoptastopthelinementalitytofailingtests.Thegoalistopromotecodehealthearly.Whenevera
testfailureisreportedbytheautomatedtestharness,fixingthefailedtestbecomesthetoppriorityfor
everyoneontheteam.Thisisthesoftwareequivalentofthestopthelinepracticeusedinlean
manufacturing.Oncethefailureisfixed,everyonecangobacktoworkingontheirrespectivetasks.This
focusonkeepingtheproductcodehealthyimprovesqualityandandreducesthedurationofthe
acceptancetestcycle.
Anotherapproachistopreventchangesthatbreakthebuildfrombeingcheckedin.TheGatedCheckin
featureoftheTeamFoundationServerdefersadeveloperscheckinuntilitcanbemergedand
validatedbyanautomatedbuild.Passingautomatedtestsorvalidatingresultsofthestaticanalysis
couldareexamplesofthevalidationpolicies.FormoreinformationonTeamFoundationBuild,see
[MSDNOnTeamFoundationBuild].
19.2 TestProgressReportingPractices
Whenwefirststartexecutingthetestswedontknowwhetherthequalityishighorlowbutwedo
knowthatwedonthaveahighdegreeofconfidenceyet.Astestingprogresses,weshouldbegettinga
betterideaofwhatthequalitylevelisandhowmuchlongeritwilltakeustogettotherequiredlevelof
confidence.Thisisverysimilartotheconeofuncertaintyconceptthatpredictsthecompletiondate
and/orcostofdevelopingthesoftware.FigureXillustratestheconeofuncertaintyforqualityfora
typicalproject.Theinitialguestimatewasanywherebetween10and100personmonths.Asthe
requirementswerebetterunderstood,therangereducedsomewhatbutasuddendiscoveryof
additionalscoperaisedtheestimatesonceagain.Aneffortwasmadetodescopetheprojecttorecover
theoriginaltimelines.Workcreepslowlyraisedboththeupperandlowerlimitandtheestimatesfinally
convergewhentheproductisacceptedasis.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 210

FigureX
ConeofUncertaintyforQuality
Wecan,ofcourse,assumethatwewillhaveareasonablyaccurateassessmentofqualitywhenwehave
finishedallourplannedtesting.Thisassumptionmayormaynotbecorrectbecauseitdependsonhow
effectivelyourplannedtestactivitieswillfindalltheimportantbugs.Thisalsoimpliesaclear
understandingofwhatisimportanttotheproductowner.Thisisverydifficulttoassessaheadoftime.
Wecertainlyneedtomonitorhowmuchtestingworkremainstobeexecutedinthecurrenttestcycle
andwhenweexpecttohaveitallcompleted.Thisappliestoeachtestcycleweexecute.
Sessionbasedtestingintroducesafeedbackmechanismthatexplicitlyallowsustoadjustthetestplans
aswelearnnewinformationbothabouttheproductandabouttheproductownerandtheproduct
ownersneeds.Aseachtestsessioniscompletedthetestersaddanynewlyidentifiedareasofconcern
tothebacklogoftestchartersyettobeexecuted.Developersmayalsosuggestadditionaltestcharters
toaddresstheriskassociatedwiththemodificationstheymaketothesoftwareasaresultofchange
requestsandbugfixesidentifiedbytests.Wemonitorthesizeofthebacklogoftestcharterstogeta
senseofwhetherwearemakingheadway.Astheconfidenceofthetestersanddevelopersimproves
theywillsuggestfewerandfewernewtestchartersandthereforethesizeofthecharterbacklogwould
dropfaster.
Predictingwhenthesoftwarewillbeofgoodenoughqualitytodeliverisdifficultbecausethatinvolves
predictinghowmanytestcycleswewillrequireandhowmuchtimetheproductdevelopmentteamwill
requirebetweenthetestcyclestofixthebugs.(SeeChapter16PlanningforAcceptanceforamore
detaileddiscussiononthistopic.)Thisrequiresmonitoringthebugbacklogandthearrivalrateofnew
bugstoassessandadjustthepredicteddeliverydate.
0
20
40
60
80
100
120
140
Latest(90%)
MostLikely(50%)
Earliest(10%)
Scope
Increases
Initial
Guess
Detailed
Estimate
Descoped
Release1
NeededExtra
Test&FixCycle
Acceptance!
Work
Creep

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 211
19.3 AssessingTestEffectiveness
Oneofthechallengesoftestingisassessinghoweffectiveourtestsreallyaresothatwecanknowhow
confidentweshouldbeinourassessmentofthequalitywehave.Thereareseveralwaystocalculate
theeffectivenessofthetestsincludingtheuseofcoveragemetricsandusingthefindrateof
intentionallyseededbugs.
19.3.1 CoverageMetrics
Wecanusemetricsliketestcoverageandcodecoveragetocalculatethetheoreticaleffectivenessof
ourtests.Thesemetricscantelluswhatpercentageoftherequirementshasbeentestedandwhat
percentageofthecodehasbeenexecutedbythetestsbutneitheroftheseisadirectmeasureofwhat
percentageofthebugswehavefound.Ofcourse,theprimaryissueisthatwehavenoideaofhow
manybugsreallyexistsoitisprettydifficulttosaywithanycertaintywhatpercentagewehavefound.
Foracautiousapproachtousingcodecoverage,seeBrianMaricksessay
[MarickOnCodeCoverageMisuse].
19.3.2 DefectSeeding
Onetechniquethecanprovideamoredirectmeasureisdefectseeding.Itinvolvesdeliberatelyplacinga
knownsetofdefectsinthesoftware.Asbugsarefoundduringtestingwecanestimatethepercentage
ofdefectsfoundbydividingthenumberofseededdefectsfoundbythetotalnumberofseededdefects
asshowninFigureY.
PercentDefectsFound
SeededDefectsFound
TotalSeededDefects

FigureY
PercentageofBugsFound
WecanprojectthenumberofdefectsyettobediscoveredusingtheformulainFigureZ.
FoundDefects
TotalDefects

SeededDefectsFound
TotalSeededDefects

Or,
TotalDefects FoundDefects
TotalSeededDefects
SeededDefectsFound

FigureZ
TotalBugsCalculation
ThesecalculationsaredescribedinmoredetailintheTestStatusReportingthumbnail.
19.4 BugManagementandConcernResolution
Akeyoutputoftestingandreviewsisalistofknowngapsbetweenwhattheproductownerexpectsof
theproductandhowitactuallybehaves.Whiletheprogressoftestingisusuallyreportedintermsof

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 212
whichtestshavebeenrunandwhichhavent,thegapbetweenexpectationsandrealityisusually
expressedasalistofknownbugsorissuesthatmayormaynothavetobeaddressedbeforethe
productwillbeaccepted.InChapter16PlanningforAcceptance,weintroducedtheConcern
ResolutionModelwhichdescribeshowconcerns,includingsoftwarebugs,changerequests,issuesand
documentationbugs,canbemanaged.Concernsthatfallintoanyofthesecategoriescouldbe
consideredgating(alsoknownasblockingorblockers),thatis,wouldpreventthesystemfrombeing
accepted.Oncewehavefinishedourfirstfullpassoftesting,presumablyattheendofourfirsttest
cycle,wecanthinkofthesetofgatingbugsasbeingtheoutstandingworklistfortheproduct
developmentteam.Ourgoalistodrivethelistofgatingbugsdowntozerosothatwecanconsider
acceptingandreleasingtheproduct.(Notethatjustbecauseitreacheszerodoesnotimplythatthere
arenogatingbugs,justnonewecurrentlyknowabout.)Theproductownercaninfluencethegating
countintwoways:theproductownercanclassifynewlyfoundbugsasgatingortheycanreclassify
existinggatingbugsasnongatingiftheydecidethatthebugcanbetoleratedbecausethereisalow
enoughlikelihood(reducedprobabilityasdescribedinChapter5RiskModel)itwillbeencounteredor
thereareacceptableworkaroundsintheeventthatitisencountered(reducedimpactpertherisk
model.)
19.4.1 BugTriage
Itiscommonpracticetoclassifytheseverityofbugsbasedontheirbusinessimpact.UsuallyaSeverity1
(orSev1asitiscommonlyabbreviated)bugisacompleteshowstopperwhileaSev5bugismerely
cosmeticandwontimpacttheabilityofuserstousetheproducteffectively.Manyproductowners
insistthatallSev1&2bugsbefixedbeforetheywillaccepttheproduct.Someproductownersconsider
Sev3bugscriticalenoughtoinsisttheyareresolvedbeforetheproductcanbeaccepted.Notethatthe
interpretationoftheseverityscaleismerelyavocabularyforcommunicationoftheimportanceofbugs
betweentheproductownerandtheproductdevelopmentteam;itisentirelyuptotheproductowner
tomakethedecisionwhetherthebugneedstobefixed.Theproductdevelopmentteammayinfluence
thatdecisionbypointingoutsimilaritiesordifferencewithotherbugsthathadthesameseverityrating
orbypointingoutthepotentialimpactaspartofanargumenttoincreaseordecreasetheseverity.They
mightalsopointoutpotentialworkaroundsorpartialfixesthattheyfeelmightjustifyreducingthe
severity.Butultimately,itistheproductownerdecisionsastowhethertheproductisacceptablewith
thebugstillpresent.
Theproductownerneedstobereasonableaboutwhatbugsshouldbeclassifiedasgating.Ifthereare
1,000bugsandtheteamcanfix20bugsperweek,itwilltaketheteam50weekstofixalltheexisting
bugsassumingthatnonewbugsarefoundandnoregressionbugsareintroducedbythebugfixes.Both
theseassumptionsarehighlyoptimistic.Therefore,thecustomerneedstoaskWhichofthesebugs
trulyneedtobefixedbeforeIcanaccepttheproduct?Thisispurelyabusinessdecisionbecauseevery
bughasabusinessimpact.Somemayhaveaninfinitesimallysmallbusinessimpactwhileothersmay
havealargebusinessimpact.Theproductownerneedstobeopinionatedaboutthisandtobeprepared
tolivewiththeconsequencesoftheirdecisionwhetherthatistodelayacceptance,deploymentand
usageoftheproductuntilaparticularbugisfixedortoputthesoftwareintousewiththeknownbug
present.Thereisnopointininsistingthatallbugsmustbefixedbeforeacceptancesimplytobeableto

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 213
saythattherearenoknownbugs.Doingsowilllikelydelayaccruinganybenefitsofthesystem
unnecessarily.Theprocessofdecidingtheseverityofeachbugandwhetherornotitshouldeverbe
fixedissometimescalledBugTriage.
19.4.2 BugBacklogAnalysis
Thislistofknownbugsisacollectionofusefulknowledgeaboutthestateoftheproduct.Mostbug
managementtoolsincludeasetofstandardreportsthatcanbeusedtobetterunderstandthebug
backlog.Theseinclude:
BugFixRateReportDescribestherateatwhichbugsarebeingfixed.
BugArrivalRateReportDescribestherateatwhichnewbugsarebeingreported.A
decreasingarrivalratemaybeanindicationthatthereturnoninvestmentoftestinghas
reachedthepointofdiminishingreturns.Oritcouldjustmeanthatlesstestingisbeingdoneor
thatthetestingisrepeatingitself.Longlivedproductsthatreleaseonaregularcycletypically
findthattheshapeofthecurveisfairlyconstantfromreleasetoreleaseandthiscanbeusedto
predictthereadytodeliverdatequiteaccurately.SeeFigureDBugArrivalRateandFigureE
CumulativeBugsFound.
BugDebtorBugBurnDownReportDescribestherateatwhichthenumberofbugsisbeing
reducedorisincreasing.Theburndownreportaggregatesthebugarrivalrateandthebug
fixingratetoallowustopredictwhenthenumberofgatingbugswillbelowenoughtoallow
acceptingtheproductandreleasingittousers.
BugAgingReportClassifiesbugsbyhowlongithastakentofixthemandhowlongithasbeen
sinceunresolvedbugswerefirstreported.Thelatterwillgiveanindicationofpotentiallevelof
customerdissatisfactioniftheaverageageofbugsislarge.
BugCorrelationReportClassifiesbugsbasedontheirrelationshipwithattributesofthe
systemundertest.Theproductowneristypicallymostinterestedinwhichfeatureshavethe
mostbugsbecausethishelpsthemunderstandthebusinessimpactofacceptingthesystem
withoutresolvingthem.Theproductdevelopmentteam,ontheotherhand,istypically
interestedinwhichcomponents,subsystems,ordevelopmentteamshavethemostbugs
associatedwiththembecausethishelpsthemunderstandwheretheirowninternalprocesses
needtobeimprovedmost.FigureZisanexampleofaBugsbyAreaorBugsbyFeatureTeam
report.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 214

FigureD
BugArrivalRate

FigureE
CumulativeBugsFound.
0
10
20
30
40
50
60
70
80
NumberofBugsFoundbyDate
Num
berof
Bug
0
200
400
600
800
1000
1200
1400
01/03/2008 01/04/2008 01/05/2008 01/06/2008
CumulativeBugsFoundbyDate
Cumulative
BugsFoundby
Date

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 215

FigureZ
BugCorrelationReport

FigureW
BugDebt/BurnDownReport
0
50
100
150
200
250
Usability
Installation
Documentation
Localization
Globalization
Logging/Tracing
Performance
Functionality
Security

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 216
19.5 TestAssetManagementPractices
Theartefactsproducedwhileplanningandexecutingtheassessmentprocessareassets;thesetest
assetsneedtobemanaged.Ifrepeatabilityoftestingisconsideredimportant,suchaswhenthesame
testsneedtobeusedforregressiontestingsubsequentreleases,testscriptsandthecorrespondingtest
datasetsneedtobestoredinaversioncontrolledtestassetrepository.Thismaybeadocument
repositorysuchasSharePointoracoderepositorysuchastheTeamFoundationServerrepository,
SubversionorCVS.ThisisdescribedinmoredetailintheTestAssetManagementthumbnail.
Whentestassetsareexpectedtobelonglivedsuchaswhenthesametestswillbeusedtoregression
testseveralreleasesofaproduct,itisimportanttohaveastrategyfortheevolvingthetestassetsto
ensuremaintainability.TheTestEvolution,RefactoringandMaintenancethumbnaildescribeshowwe
cankeepourtestassetsfromdegradingovertimeastheproducttheyverifyevolves.
19.6 AcceptanceProcessImprovement
Theacceptanceprocessconsumesasignificantportionofaprojectsresources.Therefore,itisagood
placetolookwhentryingtoreducecostsandimprovetheeffectivenessofonesprocesses.Twogood
candidatesforprocessimprovementareimprovingtheeffectivenessofthetestpracticesand
streamliningtheacceptanceprocesstoreducetheelapsedtime.Thelatteristhesubjectofthenext
chapter.
19.6.1 ImprovingTestEffectiveness
Multireleaseprojectsandlonglivedproductswillgothroughtheacceptanceprocessmanytimesin
theirlifetime.Eachreleasecanbenefitfromlessonslearnedinthepreviousrelease,ifwecaretoapply
thelessons.Itisworthconductingoneormoreretrospectivesaftereachreleasetobetterunderstand
whichreadinessassessmentandacceptancetestingactivitieshadthemostimpactonproductquality
andwhichhadtheleast.Thehighlyeffectiveactivitiesshouldbecontinuedinfuturereleasesandthe
ineffectiveonesshouldbereplacedwithsomethingelse.Someteamsmakeitapointtotryatleastone
newpracticeeachreleasetoseeifitwilldetectbugsthatpreviouslyslippedthroughthereadiness
assessmentandacceptancetestingsafetynets.Itisalsoworthanalysingthelistofdefectfoundinthe
usagephasetoidentifyanyshortcomingsinthesafetynet.Bugcorrelationreportscancomeinhandyin
thisexercise.SeeFigureQBugsbyHowFoundReport.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 217

FigureQ
BugsbyHowFoundReport.
19.7 Summary
Likeanyprocess,executionoftheacceptanceprocessneedstobemanaged.Thisincludesmonitoring
theexecutionoftheplannedtestingandensuringitresultsindatathatallowsforahighconfidence
acceptancedecision.Akeyaspectofmanagingacceptanceisthemanagingthebugbacklogtogetthe
bestpossiblequalityproductattheearliestpossibletime.Thesetwogoalsareatoddswitheachother
anddecidingwhichtakesprecedenceshouldbeabusinessdecision.Whenbuildingproductisan
ongoinggoaloftheproductownerorganization,continuousimprovementoftheacceptanceprocess
shouldalsobemanagedtofurtherreducetimetousageandimproveproductquality.
19.8 WhatsNext
Theacceptanceprocesstypicallytakesupasignificantportionofaprojectsresourcesandelapsedtime.
Thenextchapterlooksatwaysbothelapsedtimeandresourceusagecanbereduced.
19.9 Resources
[MartinFowlerOnContinuousIntegration]Fowler,M.ContinuousIntegration.
http://martinfowler.com/articles/continuousIntegration.html
[MSDNOnTeamFoundationBuild]http://msdn.microsoft.com/enus/library/bb668958.aspx
[CockburnOnAgileDevelopment]Cockburn,AlistairAgileSoftwareDevelopment:TheCooperative
GameAddisonWesleyProfessional
0
50
100
150
200
250
300
350
NumberofBugsbyHowFound
NumberofBugs

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 218
[JeffiresOnBigVisibleCharts]RonJeffries,BigVisibleCharts,
http://xprogramming.com/xpmag/bigvisiblecharts/
[LambOnTeamBuildMonitor]http://blogs.msdn.com/jimlamb/attachment/3467297.ashx
[MarickOnCodeCoverageMisuse]BrianMarick,HowtoMisuseCodeCoverage
http://www.exampler.com/testingcom/writings/coverage.pdf

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 219
Chapter20 FineTuningtheAcceptance
Process
ThepreviouschaptersofPartIIIintroducedthepracticesinvolvedinplanningandexecutingthe
acceptanceprocess.Byreadingthosechaptersthereadershouldbeabletogetabasic
understandingofwhatisinvolvedinacceptingsoftware.Thischapterdescribeshowwecanuse
theinformationweobtainwhileexecutingtheacceptanceprocessascluesforhowtoimprove
theproductdevelopmentprocessesofourorganizationtoreducedefectlevelsandtominimize
theelapsedtimeandresourcesconsumedwhilemakingtheacceptancedecision.
Theacceptanceprocessisanecessarybutpotentiallywastefulexercise.Weshouldstrivetokeepitas
simpleandvalueaddingaspossible.Thelongerittakes,themoreitcostsinwastedeffortanddelayed
deliveryofbusinessvalue.Wecantakestepstostreamlinetheacceptanceprocessbyreorganizinghow
wedoreadinessassessmentandacceptancetesting.Buttheacceptanceprocessisjustonepartofthe
productdevelopmentprocess.Theissueswefindwhileexecutingtheacceptanceprocessare
consequencesofthechoiceswehavemadeinhowwestructureourproducts,markets,andworkflows.
Wecanuserthesesymptomstomotivateabetterunderstandingoftheunderlyingproblemsinhowwe
work.
20.1 DebuggingtheAcceptanceProcess
Issuesencounteredduringtheacceptanceprocesscanbevaluablecluesaboutproblemsinhowour
organizationandworkflowsarestructured.Thesecluesprovidehintsabouthowweshouldrestructure
ourorganizationanditsprocessestoworkmoreefficiently.Thefollowingarealistofcommon
symptomsandpossiblesolutions.
20.1.1 OverlyLongDurationofAcceptanceProcess
Anacceptanceprocessthattakesalongperiodoftimehintsatissueswithhowtheorganizationis
structured.Itcanbecausedby:
Toomanygroupseachneedingtotheirownspecializedworkandtoomanyhandoffsbetween
them.Considerusingvaluestreammappingtoidentifywhichstepsaddrealvalueandwhich
couldbeeliminatedentirely(preferable)ordoneinparallel(secondchoice.)Therootcauseis
likelythewaytheorganizationisstructured;changingthestructuremaybeanecessarythough
highrisk/effortendeaverthoughonewithapotentiallyhugepayback.
Toomanytest&fixcyclesbeingneeded.Thisisdescribedbelow.
20.1.2 TooManyDefectsfoundduringAcceptanceProcess
Iftheacceptanceprocessfindsmanydefects,thisislikelyasignthatTheProductDevelopmentTeams():

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 220
Dontknowhowthefinishedproductshouldbehave.
ArenotheldaccountablefordeliveringafinishedproducttotheProductOwneracceptancetesting.
AlackofunderstandingofwhatdonelookslikeisasignthattheProductOwnerhasnotdonean
effectivejobofcommunicatingthenatureoftheproducttotheProductDevelopmentTeam.Thiscould
bebecausetheproductownerdoesntknoweither,oritcouldbeineffectivecommunicationsuchas
occurswhentheprimarymeansofcommunicationiswrittenrequirementspecifications.TheProduct
OwnercanlearnmorequicklywhattheyreallywantbydoingIncrementalAcceptanceTesting.This
givesthePDTmoretimetoaddressanychangesthaniftheacceptancetestingweretobedoneinafinal
AcceptanceTestPhase.Ineithercase,thejointunderstandingoftheacceptancecriteriacanbe
improvedbyincludingacceptancetestsaspartoftherequirementsprocess.Thesetestscanbe
providedbytheProductOwnerordevelopedjointlythroughcollaborationbetweentheProductOwner
andtheProductDevelopmentTeam.
Lackofaccountabilityoccursforseveralreasons.Oneiswhentheproductisdecomposedinto
componentsthatarefarremovedfromtheenduserfunctionality.TheProductDevelopmentTeam
oftendoesntunderstandhowtheircomponentsupportsthebusinessgoalsandthereforebuilds
functionalitythatmayactualbeatcrosspurposestoit.Thisoftenresultsinbugsfoundonlyafterthe
componentsareintegrated.ThisisoneofthedownfallsoftheComponentTeamapproachtoorganizing
thework.
Anotherreasonforlackofaccountabilityiswhenmanagementvaluesscheduleoverquality.Ofcourse
wesayqualityisimportantbutitsmanagementsactionsthatreallycount.Whenmanagement
pressuresdeveloperstomeetunrealistictimelineswearesayingscheduletrumpsquality.Aclear
definitionoftheMinimumQualityRequirement(MQR)iscrucial.WeneedtoencouragetheProduct
DevelopmentTeamtoimproveitsprocessestodeliverbetterqualitybymistakeproofingtheprocesses
asmuchaspossible.TestDrivenDevelomentandautomatedtestsarebothexamplesofhowtodothis.
Oncetheycanbuildsoftwarewithfewerdefects,theywillbeabletodeliverfasteraswell;theinverseis
nottrue.
20.1.3 TooManyTest&FixCyclesNeeded
WhentheproductrequirementshavebeenclearlycommunicatedandtheProductDevelopmentTeam
hasdoneaneffectivereadinessassessment,theacceptancetestingdoesnotfindmanydefects.Those
defectsthatarefoundshouldbefairlyminorandeasilyfixed.Butwhatiftherearealotofdefects?
Thentheproductwillneedtogobackthroughtheconstruction,readinessassessmentandacceptance
testingprocessanothertime.Whenthiscyclehastoberepeatedseveraltimesbeforetheproductis
goodenoughtoconsiderreleasing,itmaybeduetooneofseveralrootcauses:
Newbugsarebeingintroducedbymanyofthefixesfortheexistingbugs.Thiscouldbebecausethe
softwarehasbecomebrittleduetoageandexcessiveinternalcoupling(ormaybeitwasntdesigned
verywellevenwhenitwasnew.)Oritcouldbeduetorushingtheworkandalackofasafetynetto
catchthedefectsbeingintroduced.Thelattercanbeaddressedbyincorporatingautomatedunit
testingandpossiblytestdrivendevelopmenttoavoidinsertingthedefects.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 221
TheproductDevelopmentTeamisnotdoingeffectivereadinessassessmentofeachreleasecandidate.
Thismaybebecausetheyarebeingrushedorbecausetheyarerelyingprimarilyonmanual
regressiontestingandcannothopetoretestalltheaffectedsoftware.Considerintroduceing
automatedregressiontestingatvariouslevels(unit,component,system)toreducethelengthof
timeittakestodoaneffectiveregressiontestcycle.
20.1.4 LotsofDebateBetweenBugsvs.ChangeRequests
Whenisabugadefectandwhenisitachangerequest?Ifthisdiscussionisoccurringalotduringthe
acceptanceprocessitmaybeanindicationthattheProductOwnerandtheproductDevelopmentTeam
arenotstrivingtosolvethesameproblem.Somepossiblerootcausesare:
FixedpricecontractsencouragetheproductDevelopmentTeamtoclassifyeverythingasaCRevenifit
islegimatelyabug.Solutionistoavoidcreatingdysfunctionalrelationshipsthatcomeoutoffixed
pricecontracts.ConsiderusingTargetPricecontracts[PoppendiecksOnLean]instead;these
motivatebothcustomerandsuppliertominimizethechangeswhilemaximizingvalue.Thereisno
distinctionbetweenabugandaCR;theonlydistinctionisbetweenchangesthatareworthmaking
andthosethatthecustomercanlivewithout.
VaugerequirementscausedbyalackofunderstandingoftheProductOwnerastowhattheywanted.
See
VaguerequirementsleadingtolackofclarityaboutwhatPOreallyaskedfor.Thegapbetweenwhatthe
POthoughttheyaskedforandwhattheproductDevelopmentTeaminterpretedtherequestcaused
thebugandthesubsequentdebate.ThesolutionistoimprovethecommunicationbetweenthePO
andtheproductDevelopmentTeam.Thisisbestachievedthroughcollaborationratherthansimply
writingmorecopiousrequirementdocumentation.Thecommunicationcanbesupportedby
detailedexampleswhichcanbeusedassampleacceptancetests.Considerincludingreadiness
assessorsandacceptancetestersintheproductdesigndiscussionsasthiswillusuallyunearth
interestingscenariousthattheProductOwnermayneedtoconsider.

DebuggingtheAcceptanceProcess
Issuesencounteredduringtheacceptanceprocesscanbevaluablecluesaboutproblemsinhowour
organizationandworkflowsarestructured.Thesecluesprovidehintsabouthowweshouldrestructure
ourorganizationanditsprocessestoworkmoreefficiently.Thefollowingflowchartsuggestpossible
solutionsbasedoncommonsymptoms:
Psuedocodeforaflowchart:
IfAcceptanceTakingTooLong
o IftoomanyTest&fixcycles
Iftoomanyregressionbugsintroduced

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 222
See0UseAutomatedTestExecutiontoReduceRegressionBugs
else
o Elseiftoolongpercycle
ConsiderstreamliningtheprocessusingValueStreamMapping
Else(nottakingtoolong)
o Iftoomannewbugsfound
Iffoundbyendusers
IfUsersfindsystemhardtouse
o See0UseIncrementalUsabilityTestingtoDiscoverDesign
DefectsEarlier
Ifbugsfoundinafewspecificareas
o See0IncreaseBreadthofAcceptanceTesting
Elseifbugsfoundeverywhere
o See0IncreaseDepthofAcceptanceTesting
Iffoundbyacceptancetesters
IfPDTisnotdoingthroughreadinessassessment
o IfPDTisorganizedaroundcomponents
See0UseFeatureTeamstoImproveAccountabilityof
ProductDevelopmentTeamforEndUserFunctionality
o ElseIfPDThasaccesstoacceptancetests
Runthemmorefrequently
See0UseAutomatedTestExecutiontoReduce
RegressionBugs
o Else
See0UseAcceptanceTestDrivenDevelpmenttoHelp
ProductDevelopmentTeamUnderstandRequirements
Better
isbeingrushedbymanagementorPOordeadlines
o See0FocusonQualitytoGetSpeedofDelivery
Elseifa

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 223
IffoundbyReadinessAssessors
Iffoundbyindependenttesters
o UseATTDtoimprovecommunicationofdone
SeealsoIffoundbyacceptancetesters
o ElseiftoomanyaccumulatedbugstofixtherebyrequiringBugTriage
Ifduetohighbugarrivalrates
seeIftoomanynewbugsfound
ElseifBugsarentbeingfixedduetolackoftime
EducateProductownerondecidingbetweenBugsvsNewFeatures
Ifduetofearofintroducingnewbugsduetofragilityofcodebase
Findwaystogetcodebaseunderautomatedtestandrefactor
o ElseifdisagreementsbetweenBugsandChangeRequests
IfFixedPriceContracts
See0UseTargetPriceContractstoAlignInterestsofProductOwner
and
Else
See0UseAcceptanceTestDrivenDevelpmenttoHelpProduct
DevelopmentTeamUnderstandRequirementsBetter

o Else
Isthereaproblem?
Thesolutionsreferencedinthisflowchartareelaboratedbelow.
PossibleRemedies
Basedontheresultsoftheanalysisofacceptanceprocessissuesencountered,oneormoreofthe
followingremediesmaybeuseful:
UseTargetPriceContractstoAlignInterestsofProductOwnerandProductDevelopment
Team
UseIncrementalUsabilityTestingtoDiscoverDesignDefectsEarlier
Incorporateusabilitytestingofearlyversionorprototype(includingpaperprototypes)

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 224
UseIncrementalAcceptanceTestingtoDiscoverDefectsEarlier
UseIncrementalAcceptanceTestingtoHelpProductOwnerDiscoverRequirementsEarlier
InvolveTestersDuringRequirementsDefiintiontoEnsureCompletenessofRequirements
UseAcceptanceTestDrivenDevelpmenttoHelpProductDevelopmentTeamUnderstand
RequirementsBetter
ProductOwnershouldprovideacceptanceteststoProductDevelopmentTeam
o CollaboratewithProductDevelopmenttodevelopthem

UseAutomatedTestExecutiontoReduceRegressionBugs
UseFeatureTeamstoImproveAccountabilityofProductDevelopmentTeamforEndUser
Functionality
IncreaseBreadthofAcceptanceTesting
Includemorevariedfunctionality/scopewithinthescopeoftesting.
IncreaseDepthofAcceptanceTesting
Usemore/differenttestdesign(e.g.Scenariobasedtesting)andexecutiontechniques(e.g.
ExploratoryTesting)togetbettertestcoverage.
FocusonQualitytoGetSpeedofDelivery
FocusonQuality;speedwillfollowduetolesstimespentfindingandfixingbugs.
StreamliningtheAcceptanceProcess
Theacceptanceprocessisanecessarybutpotentiallywastefulexercise.Weshouldstrivetokeepit
assimpleandvalueaddingaspossible.Thelongerittakes,themoreitcostsinwastedeffort
anddelayeddeliveryofbusinessvalue.itItcanbecomeaseriousimpedimenttobeing
responsivetoourusers(theproductownerscustomers.)Beforewecantakestepsto
streamlineitwemustfirstunderstandit.

20.1.5 UseValueStreamMappingtoUnderstandtheAcceptanceProcess
Wecanimproveourunderstandingofanexistingorproposedacceptanceprocessthroughan
exercisecalledValueStreamMapping.Thisisaformofbusinessprocessmodelingthatfocuses
ourattentionontheratioofelapsedtimeandtheamountofvalueadded.Figure1isavalue
streammapofahypotheticalacceptanceprocess.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 225

Figure1
AsIsAcceptanceProcess
Thisprocesstakesanaverageof211daystoexecuteandprovidesonly60daysofactualvalueresulting
inacycleefficiencyofonly28%.Someofthefactorsthatmakethisprocesstakesolongtoexecuteare:
Sixtydaysofsoftwaredevelopmentoutputissentthroughtheprocessinasinglebatch.Therefore,
thereisalargeinventoryofuntestedsoftwarewhichresultsinmanybugsbeingfoundduring
thereadinessassessmentandacceptancetestingactivities.Thefixingofthesebugsisdone
entirelyonthecriticalpathoftheproject.
Thereareseveralcyclesinprocesseachofwhicharetypicallyexecutedseveraltimes.Forexample,
ReadinessAssessmentsendsthecodeback,onaverage4timesforanadditional5dayselapsed
timewhichaddszerodaysofvaluebecauseitispurerework.Eachcycletakes7.5days(2.5days
fixingand5daysreadinessassessment)thereforethisfeedbackloopadds30daysofelapsed
time.
Sometaskstakelongerthannecessarybecausetheresourcesarenotdedicated.Forexample,the
readinessassessorshaveotherresponsibilitiesandittakesthem5daystodo2.5daysoftesting.
Customeracceptanceisdoneintwoseparatephasesandisfollowedbythreeotherformsof
acceptancedecisionmaking,twoofwhichcansendthesoftwarebackallthewaytobugfixing.
Thesoftwareneedstoberetestedeachtimethesoftwareissentbacktofixbugs.
Thesecurityteamisoverworkedresultinginanaveragewaittimeof10daysforthesecurityreview.
Thepreparationofthesecuritydocumentstakesanaverageof10daysasdevelopersdiscover
whatisrequiredbutonly1dayofthateffortaddsrealvalue.
Readiness
Decision
Readiness
Assessment
Prepare
Security
Document
Feature
Acceptance
Decision
Feature
Acceptance
Testing
Bugs found Avg4 x
Insufficient information 30%
Insufficient Quality Data
25%
1 day
10 days
Security
Acceptance
Decision
Queue
Legend:
Value in days
Time in days
Avg10
days 0.05 day
0.25 day
Construct
Software
40* days
40 days
Severity 1 &2 bugs found Avg 3 x
5 days
10 days
2.5 day
5 days
0.5 day
0.5 day
0.5 day
0.5 day
Operations
Acceptance
Decision
Operations
Assessment
Prepare
CMB
Documents
Insufficient Quality Data 10%
Insufficient information 30%
1 day
10 days
CMB
Acceptance
Decision
Queue
Avg10
days 0.05 day
0.25 day
Inadequate design 25%
2.5
days
5 days
0.5 day
0.5 day
Integration
Acceptance
Decision
Integration
Acceptance
Testing
Insufficient Quality Data
25%
5 days
10 days 0.5 day
0.5 day
FixBugs
0 days
2 days
Severity 1 &2 bugs found Avg 0.5 x
MonthlyReview
(every20days)
MonthlyReview
(every20days)
Testers50% Testers50% Testers50%
Testers50%
Valueadding
activities=
60days
total=211days
28%efficiency

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 226
TheChangeManagementBoard(CMB)meetsmonthlyresultinginanaveragewaitof10business
days.
Note:Thecalculationsassumethatnovalueisprovidedbythebugfixingpassesthroughthesame
activitiesasthisisunplannedrework.Ofcourse,thiswholecalculationisbasedontheassumptionthat
wecanaddvaluebyreviewingwork;anargumentcanbemadethatthewholeacceptanceprocessisa
formofwastecausedbythecustomersinabilitytoaccuratelydescribewhattheywantbuiltandthe
suppliersinabilitytofollowthespecificationadequately.
20.1.6 StreamlinetheAcceptanceProcess
Thisacceptanceprocesscanbestreamlinedbychanginghowtheworkpassesthroughthevarious
readinessandacceptanceactivities.Figure2illustratestheresultofhavingappliedanumber
transformationsontheprocess.ThesetransformationsareinspiredbyLeanThinkingwhichfocuseson
eliminatingwaste.SeethesidebarWasteinSoftwareDevelopmentforadescriptionoftheseven
commonformsofwaste.Whichformsofwastewetrytoeliminatefirstdependsonwhatisimportant
tous.Iftimetomarketisthekeyconsideration,wemaywanttotacklethequeuingdelaysfirstand
thenlookforwaystoreducetheamountofprocessing.Ifcost/resourceconstraintsareholdingusback,
wemaywanttolookforwaystoreducetheamountofprocessingfirst.

Figure2
StreamlinedAcceptanceProcess
Thisstreamlinedversionoftheprocessistheresultofthefollowingchanges:
PerIterationActivities
Readiness
Decision
Readiness
Assessment
1on1
Security
Review
Feature
Acceptance
Decision
Feature
Acceptance
Testing
4 Iterations +
1 bug fix
Insufficient Quality Data
25%
0.5 day
0.5 day
Security
Readiness
Decision
Legend:
Value in days
Time in days
0.05 day
0.05 day
Construct
Software/
Fixbugs
10 days
10 days
Severity 1 &2 bugs found Avg1 x
1 day
1 day
2.5 days
2.5 days
0.5 day
0.5 day
0.1 day
0.1 day
Operations
Acceptance
Decision
Operations
Assessment
PrepareCMB
Documents
Insufficient Quality Data
10%
Insufficient information 30%
1 day
10 days
CMB
Acceptance
Decision
Queue
Avg2.5
days
0.05 day
0.25 day
Inadequate design
25%
1 day
1 day
0.1 day
0.1 day
Integration
Acceptance
Decision
Integration
Acceptance
Testing
Insufficient Quality Data
25%
1 day
1 day
0.1 day
0.1 day
Note:AllTesters
100%Dedicated
WeeklyReview
(every5days)
Security
Consultation
0.5 day
0.5 day
1 day
2 days
Prepare
Security
Document
Insufficient information
<10%
Consultations &
Interviewsprebooked
Repeatedonceper
iterationplusonefinal
bugfixcycle

Valueadding
activities=
58days
total=110days
53%efficiency

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 227
Wehavebrokentheprojectinto4twoweek(10businessday)iterations.Eachiterationissentthrough
readinessassessment,featureacceptance,integrationacceptanceandoperationalacceptance.Bugs
foundinthefirstthreeroundsoftestingarefixedinthesubsequentiteration.Afterthefourth
iteration,thesoftwareisretestedandtheresultingbugsarefixedinasinglebugfixingiteration.
Contrastthiswiththe4roundsofbugfixasaresultofreadinessassessmentandthreeroundsdue
tofeatureacceptancetesting.Eachroundoftestingtakesroughly30%ofthetimeittookintheAs
Isprocessbecausethebacklogofuntestedsoftwareismuchsmaller.Thisisanexampleofreducing
wasteassociatedwithinventory.
Theacceptancetestsareprovidedtothedevelopmentteamalongwiththerequirements.Thisallows
thedevelopersandthereadinessassessmenttesterstoverifythefunctionalityisworkingcorrectly
beforehandingitofftotheacceptancetesterstherebyreducingthenumberofbugsfoundin
acceptancetesting.Thereadinessassessmentshownasoccurringaftertheiterationactually
happenscontinuously(assoonasthedevelopersaysthefeatureisready)thereforeanybugs
foundinreadinessassessmentarefixedbeforethedeveloperhasshiftedtheirfocusonthenext
fixture.Thisreducesthecostoffixingthefewbugsthatarefoundinreadinessassessmentbya
factorof4.Thisisanexampleofreducingthewasteassociatedwithdefects.
Theacceptancereviewforsecurityhasbeenchangedfromanacceptanceactivitytoareadinessactivity.
Thisallowsittobedoneinparallelwithdevelopment.Ithasalsobeenchangedfrombeingapush
modelwheretheteampreparesadocumenttobereviewedtoapullmodel.Thisstartswitha
consultationwiththesecurityspecialistwhohelpsthedevelopmentteamunderstandthe
appropriatesecurityrequirementsanddesignandwhatneedstobeinthesecuritydocument.This
providesinputintothesoftwareconstructionprocessresultinginasecuritycompliantdesignaswell
asmoreappropriatecontentinthesecuritydocument.Thisreducestheturnbackratefrom30%to
10%andthetotaleffortfrom10daysto3dayswithoutanyreductioninvalue.Theeffortto
producethedocumentisreducedandtheefforttoreadthedocumentisalsoreduced;definitelya
winwinsolution.Afinalsecurityreviewisheldpriortothefinaliterationtoreviewthefinished
documentaswellasanylatebreakingsecurityrelateddesignconsiderations.Thisisanexampleof
eliminatingwastebyreducingdelaysandbyreducingextraprocessing.
Thefeatureacceptancetesting,integrationacceptancetestingandoperationsacceptancetestingare
doneinparallelwithanunderstandingthatanyshowstopperbugsfoundinanyoftheparallel
testingactivitiescanresultinthesoftwarebeingrejected.Thisreducestheelapsedtimetothe
longestofthethreeinsteadofthesumofthethreeactivities.Thiseliminateswasteintheformof
delaysincurredwhileonegroupwaitsforanothergrouptofinishtheirwork.
Becauseonly25%ofthefunctionalityisbeingtestedanewineachiteration,theacceptancetestingcan
befinishedin1day.Thisisashortenoughperiodthatthetesterscanfocusontheoneprojectand
finishit1dayelapsedtime.Thisisntaformofwastereductionasmuchasitisanexampleof
improvingflowbymakingtheoutputoftheprocesslessburstyviasmallerbatchsizes.Achieving
flowisanotherofthekeyprinciplesofleandevelopment.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 228
TheChangeManagementBoarddocumentationispreparedinparallelwiththefinalbugfixingiteration
andcanbesubmittedtotheCMBassoonasthethreeparallelacceptancedecisionsarepositive.To
furtherreducethedelay,theCMBnowmeetsweeklyfor1hourratherthanmonthlyforahalfday.
Thisreducestheaveragewaitfrom10businessdaysto2.5.Thisisanexampleofreducingwaste
associatedwithwaiting.
Thecollectedimpactofthesechangestotheacceptanceprocesshasreducetheaverageelapsedtime
to94daystoexecute110daysofvalueaddedprocessingthatdelivers58daysofvalueforacycle
efficiencyof53%.
Note:Thecalculationsassumethatvalueisprovidedbytheplanneddevelopmentiterationsbutno
valueisprovidedbythefinalbugfixingiteration.Thecycleefficiencywasbasedonthecumulative
durationincludingactivitiesdoneinparalleleventhoughtheydidnotaddanyelapsedtime.

Summary
Asignificantportionoftheelapsedtimebetweenthefinishofdevelopmentofsoftwareandwhenthe
softwarecanstartprovidingvaluetothecustomerisconsumedbytheacceptanceprocess.Theelapsed
timecanbereducedsignificantlybybuildingsoftwareincrementallyanddoingincrementalreadiness
assessmentandacceptancetestingaseachincrementofsoftwareisfinished.Differentkindsof
inspectionandtestingactivitiescanbedoneinparallelwithdevelopmentofthelaterincrements
reducingtheamountofworkthatneedstobedoneonthecriticalpathbetweencompletionof
developmentandfinalacceptance.Thetypesoftheissuesdiscoveredduringacceptanceprocesscanbe
usedtodiagnoseorganizational,culturalandprocessissuesintheorganizationsinvolved.Changingthe
processestoavoidtheissuesisusuallymorebeneficialthanimprovingtheefficiencyofaddressingthe
issuesoncefound.
WhatsNext?
VolumeIhasintroducedanumberoftoolsyoucanuseforreasoningabouthowyouacceptsoftware.It
hasdescribedwhentousealargenumberofacceptancerelatedplanning,requirements,inspection,
reviewandtestingpractices.Youmaywanttoresearchsomeofthesepracticesinmoredetail.Volume
2inthisseriesdescribesmanyofthesepracticesinmoredetailwhileVolume3providesexamplesof
theartefactsthatmighthavebeenproducedonafictionalproject.

Sidebar:FormsofWasteinSoftwareDevelopment
ThesevencommonwastesofmanufacturingcanberememberedusingtheacronymTIM
WOOD.Insoftware,thereisanequivalentofeachformwasteasexemplifiedbythelist
providedbyMary&TomPoppendieckintheirbookImplementingLeanSoftware

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 229
Development[PoppendiecksOnImplementingLean].Unfortunately,thesoftwarespecific
namesdontformaniceacronymsoweveincludedbothnameshere.Thesoftwarespecific
nameshavea*nexttothem.
T=ThewasteofTransport(Handoffs*)
Transportationiswastebecauseitdoesntaddanyvaluetotheendproductbutitincreases
thecostandtheelapsedtime.
Insoftware,transportcorrespondstohandoffsbetweenpartiesusuallyviadocuments.The
requirementsdocumenthandedbyspecifierstosoftwaredevelopersisoneexample,the
designdocumenthandedbyarchitectstodevelopersisanother.Thepreparationofthese
documentstakeslargeamountsofeffortoftenmuchmorethancommunicatingthesame
informationverbally.Handoffsusuallyresultinlossofinformationandthisistypicallyworse
whenthehandoffisasynchronous(e.g.documents)ratherthanfacetoface.Seethesidebar
UsingWholeTeamstoBuildProductsinChapter1forwaystoreducethenumberofhandoffs.
I=Inventory(PartiallyDoneWork*)
Inventoryisbadbecauseitcostsmoneytoproducetheinventoryandoftencostsmoneyto
storeormanagetheinventory.Inventoryalsomasksissuesbydelayingwhendefectiveparts
arediscovered.Justintimemanufacturingisallaboutreducinginventorytothelowestlevels
possible.
Insoftware,inventoryisanyartifactthathastakenefforttoproducebutwhichisnotyet
providingthecustomerwiththevalueexpectedtobeprovidedbyhavingthesoftwareinuse.
Somecommonformsofinventoryinclude:
UntestedsoftwareSoftwarethathasbeenwrittenbutnotyettested
Softwarewithalonglistofbugsuntriagedortriagedbutnotfixed
RequirementsdocumentsDocumentsthatprovidedetaileddescriptionsoffunctionality
thatwontbebuiltrightaway.
M=Movement(TaskSwitching*)
Movementisbadbecausewhileaworkerismovingtheycantbeproducing.Thisaddsno
valuetotheproductbutreducesproductivityoftheworkertherebyincreasingcost.
Insoftwaredevelopment,theequivalentofmovementistaskswitching.Thisiscausedby
askingpeopletoworkonseveralthingsatthesametime.Everytimetheyswitchbetweenone
taskandanothertask,timeiswastedwhiletheyreestablishtheirworkingcontext.
W=Waiting(Delays*orQueuing)
Wheneverworkstopswhilewaitingforsomethingtohappen,itisaformofwaste.
Commoncausesofwaitinginsoftwareincludewaitingforapprovals,waitingforclarificationof
requirementsandwaitingforslowcomputersortoolstofinishtheirprocessing.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 230
O=Overprocessing(LostKnowledge*orProcessInefficiency)
Overprocessingiswastecausedbydoingunnecessarystepsordoingasteplongerthan
necessary.Thisaddsnoadditionalvaluebutitdoesaddcost.
Insoftware,overprocessingisanystepsintheproductionprocessthatarerequiredtoproduce
highqualitysoftware.Themostcommonexampleistheproductionofanydocumentation
thatnoonewilleverread.Anothercommonformishavingtorediscoverinformationthatwas
lostsomewherebeforeitcouldbeused.
O=Overproduction(ExtraFeatures*)
Overproductionisproducingtoomuch.Itislikeinventoryexceptthatitisfinishedproduct
whileinventoryisworkinprogress.
Insoftware,overproductionisthedevelopmentofunnecessaryfeatures.Ithasbeenreported
[REF]thatonaverage80%offeaturesdevelopedarerarelyorneverused.Thisisclearly
overproduction!
D=Defects
Defects,bugs,problemreports,usabilityissuesallrequireextraworktoanalyse,understand
andaddress.Thisextraworkispurewaste.Itisexactlythesameinsoftware.
Resources
[PoppendiecksOnLean]Poppendieck,M.LeanSoftwareDevelopmentAnAgileToolkit,AddisonWesley
Professional#ISBN10:0321150783ISBN13:9780321150783,
[PoppendiecksOnImplementingLean]Poppendieck,M.,Poppendieck,T.ImplementingLeanSoftware
DevelopmentFromConcepttoCash,AddisonWesley,.(Seerefinanotherchapter.)

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 231
AppendixAOrganizationStereotypes
andReaderPersonas
Thisappendixdescribestwostereotypicalorganizationsandthepeopleinvolvedinmakingorproviding
dataforthereadinessandacceptancedecisions.Thepeoplearedescribedaspersonas.
Notetoproduction
Reviewers:Thesetwostereotypesareincludedtoactasabusinesscontextforthepersonas.The
personasareincludedfortworeasons:
1. Toremindtheauthorsandreviewersofourprimarytargetaudience.
2. Togivereadersafiguretoempathizeorassociatethemselveswith.Thisshouldhelpthem
pickthepersonathatmostcloselyresemblestheirownjobresponsibilities.

Contents Primarypersonas Secondarypersonas

VolumeI
Thinkingabout
Acceptance
LisaAndrews
EricAndersen
JosephKrawczak
YvetteKirwan
ChattyCathy
FredSpook
VolumeII
Managing
Acceptance
JosephKrawczak
LisaAndrews
EricAndersen
YvetteKirwan
ChattyCathy
FredSpook
VolumeIII
Functional
Acceptance
Testing
JosephKrawczak
YvetteKirwan
ChattyCathy
FredSpook
LisaAndrews
EricAndersen
VolumeIV
Operational
Acceptance
Tetsing
ChattyCathy
FredSpook
YvetteKirwan
LisaAndrews
EricAndersen

OrganizationStereotypes
ITOrganizationofaBusiness
XYZCorpisalargetransportationcompany.Computersoftwarehasbeenintegraltothecompanys
businessforseveraldecades.Thecompanyhasseveralthousandapplications.Atthehighendare

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 232
systemsbuiltbytheITdepartmentrangingfromlargemainframesystemswritteninCobolandPL1to
WebbasedapplicationswritteninASP.NET.Atthelowendthereareenduserapplicationsbuiltin
VisualBasicandAccessandspreadsheetapplicationsbuiltbyendusersscatteredthroughoutthe
businessareas.
TheITdepartmentwillbuildthesoftwarebasedonthespecificationsprovidedbythebusinessand,
afterdoingitsownsystemtesting,expectsthebusinesstoconductuseracceptancetesting.Thereisno
separateQAdepartmentbuteachdevelopmentteamhasatleastonetestingspecialistwhohelpsthe
teamtesttheirownsoftware.TheITdepartmentsqualitygateprocessalsorequiressignoffsfromtheIT
ArchitecturegroupandtheOperationalsupportgroupbeforesoftwarecanbedeployedintothe
Acceptanceenvironment.Asaresultofrecentlyintroducedcorporategovernanceinitiatives(suchas
SOX),companypolicyrequiresapprovalbytheCorporateSecuritydepartmentbeforeanynew
applicationcanbedeployedintoproduction.
TheITArchitecturedepartmentincludesateamofdataarchitectswhoseprimaryresponsibilityisto
ownthecorporatedatamodel.Thecompanyviewsthedataintheapplicationsasacorporateassetand
thedataarchitectshavebeenworkinghardtomanagetheduplicationofdataacrossthem.TheIT
departmentalsomanagesthelifecyclesofthevariousITtechnologiesusedinthecompany.New
technologieshavetobeapprovedbytheITArchitecturegroupbeforetheycanbeusedandold
technologiesmaybemarkedforreplacement.TheCorporateSecuritydepartmenttakesanactive
interestinallnewapplicationsandmustapprovethesecuritymeasuresbeforeanynewapplicationcan
bedeployed.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 233

AProductCompany
AdventureWorksisaproductcompanythatsellsbusinessproductivityimprovementproductsthat
includebothhardwareandsoftware.Someoftheproductsaresoldashardwarewithembedded
softwareandsomeassoftwarepackagesthatleverageexistinghardwareproducts.Thecompanyhasa
separateUxDandQAdepartments.TheUxDdepartmentisinvolvedinpartoftherequirements
definitionactivitiesassociatedwithdesigningtheproductaswellasensuringthattheproductshavea
consistentlook&fell.TheQAdepartmentteststhesoftwarebuiltbytheProductDevelopmentgroup
andadvisestheProductManageronwhetherthesoftwarecanbereleasedtothemarket.

ReaderPersonas
PersonasOftenFoundintheITOrganizationofaBusiness
ProductManagerorProductOwner
EricAndersenisaproductmanagerwhohasbeenaskedtoworkwithanagiledevelopmentteamasthe
ProductOwnerforoneofthesoftwareonlyproducts(anewsoftwarereleasethatworkswithexisting
hardwaresuppliedbythecompanyandthirdparties.)Becausethecompanysproductsareconsidered
hightechnology,Ericisfairlytechsavvy.Hehasagoodunderstandingofthecustomersrequirements
butdoesgetabitlostwhenthediscussiondescendstothelevelofnetworkingprotocols.
KeyquestionsEricneedstofindanswersto:
WhatkindsofinformationdoIneedfromtheQAdepartmenttodecidewhetherornotto
acceptthesoftware?Whatqualitycriteriamustbemet?HowdoIknowtheyhavebeen
met?

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 234
Whoisresponsibleforverifyingthesecriteriaaremet?Whoisresponsibleforensuring
thesecriteriaaremet?
HowdoestheUxDteamparticipateinacceptancetestingofthesoftware?
Howcanthereleasestrategy(multiplereleases,alphas,betas,etc.)helpme?
E.g.Improveusability,fitnessforpurpose
Getrevenueearlier
Viralmarketing
Whatistheprocessforaddressingdeficienciespriortorelease?

BusinessPersonintheMarketingDivision
LisaAndrewsisabusinesspersoninthemarketingdivisionofXYZCorp.Lisaisinherlatetwenties
(relativelyyoungforherpositioninthesomewhatconservativecompany)soshehasgoodcomputer
usageskillsbutisnotbyanystretchoftheimaginationconsideredtechnical.ThisisLisasfirstIT
project.Lisahasbeenaskedtobethebusinessleadoftheprojecttoreplaceseveralexistingsystems
usedbythestaffinthemarketing,salesandcontractsdepartmentswithasingleintegratedsystem
usingthelatesttechnologies.Inthisroleshewillhavethefinalsayonwhetherthesystemisacceptable
tobedeployed.Shewillalsoberesponsibleforensuringthatthebusinessbenefitsusedtojustifythe
projectareactuallyrealized.Shewillbeassistedonthisprojectbyseveralusersoftheexisting
applicationsaswellasabusinessanalystandatestprofessional.Shewillneedtooverseethisteamasit
definesthedetailedrequirementsforthenewsystem.
KeyquestionsLisaneedstoanswer:
Whatkindsoftestingwillmyteamneedtodotohelpmedecidewhetherornottoacceptthe
software?
Howcanthereleasestrategy(multiplereleases,alphas,betas,communitytechnologypreviews,
etc.)helpme?
E.g.Improveusability,fitnessforpurpose,userbuyin
Getcostsavingsearlier
WhenshouldIplanondoingAT?HowmuchtimeshouldIallowforAT?Howmucheffortwillbe
involved?HowmuchofstaffdoIneedtodoit?
Testlastorincremental?
AmIinvolvedinATordoIdelegateittosomebodyelse?Frommyorg?athirdpartylab?
WhoelsecanhelpmewithATifIdonthavetheskill/confidence
WhatkindsoftestingdoIexpecttheITdepartmenttodobeforeaskingmyteamtodoacceptance
testing?

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 235
WhatinformationdoIneedITtoprovidetomeaboutthekindsoftestingtheyhavedoneandhow
willIusethatinformationindecidingwhetherornottoacceptthesoftware(orevenstart
acceptancetesting?)
WhatinformationdoIneedtoprovidetoITpriortomedoingAT?

OperationsDepartmentManager
ChattyCathyisthemanageroftheoperationsdepartment.Sheisresponsiblefordeployingallserver
basedapplicationsandkeepingthemrunningaccordingtoeachapplicationsservicelevelagreement
(SLA).Shesbeenwiththecompany30yearshavingworkedherwayupfrombeinganadministrative
assistantinoneofthebusinessunits.Herformaltrainingwassecretarialschoolandsheslearned
whateversheneededtoknowonthejob.WhenshewasfirstmovedintoI.T.itwasasatesterfor
desktopapplicationsbutshequicklymovedtoupmanagethedesktopsupportteam.Whenthat
functionwasoutsourced,shewasaskedtotakeovertheoperationsgroup.Thatwastwoyearsagoand
shehasonlyrecentlybecomecomfortablewithherunderstandingofhowthegroupfunctions.Shehas
yettomakeanychangestothegroupsprocesses.

ITSecuritySpecialist
FredSpookistheITSecuritySpecialistintheCorporateSecuritydepartment.Hesaformerspy(IfItold
youwhichagencyIworkedforIwouldhavetokillyou,hesayswithawinkwheneverhesasked.)who
movedintotheprivatesectorwhenthefirstmajorsecuritybreecheswerepublicized.Hisbackground
hasmadehimratherparanoidandhedelightsinskeweringthedevelopmentteamswithhisarsenalof
nastysecurityscenarios.Asaresult,mostapplicationsfailtheirfirstsecurityreviewandrequire
significantarchitecturalrefactoringbeforepassing.Thebetterdevelopmentteamshavelearnedto
approachhimforanearlyofftherecordsecurityauditassoonastheirdesignhasstabilized.

PersonasOftenFoundinaProductCompany
TestManager
JosephKrawczakisthemanageroftheproductverificationgroupatAdventureWorks.Histeamof
professionaltestersverifiesthattheproductsbuiltbytheproductdevelopmentgroupworkasspecified
bytheproductmanagementteam.Hisbackgroundisintestingtelecomsystemswherehemanageda
30persontestingdepartment.Whenthetelecomindustrydownsizedhewasgivenasizablepackage
andwashiredbyABCCorptoheadupitsnewlycreatedIndependentTestdepartment.Hismandate
wastoimprovethequalityoftheproductsproducedbyAdventureWorks.Histeamofprofessional
testersisexpectedtoprovidehimwiththedatarequiredtodeterminewhethertheproductis
acceptable.Heprovidesthisdata,alongwithaaccept/rejectrecommendationtotheProductManager
whomakesthefinaldecision.Heisalwayspreparedtobackuphisrecommendationwithaclear

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 236
descriptionofthebusinessimpactofthebugsthathefeelsmustbefixedbeforehewouldbe
comfortableadvisingtheproductbeaccepted.

UserExperienceSpecialist
YvetteKirwanisaFineArtmajorwhogotajobasagraphicartistinamediacompany.When
conventionaladvertisingworktookadownturn,shewasmovedintotheproductdesigngroupwhere
shedidgraphicdesignforwebbasedsoftwareapplications.Shequicklyrealizedthatthegraphicswere
mainlyeyecandyandthattherealvaluewasprovidedbytheinteractiondesignsoshewentbackto
schoolparttimetogettrainedasaninteractiondesigner.Withthistrainingunderherbeltshequickly
becameakeyplayerinthedesignofsoftwareintensiveproductsatthecompany.Shewashiredby
AdventureWorks.specificallytoworkonthenewproduct.Herroleistodefinetheinteractiondesign
basedonmarketrequirements,usermodelsandfeedbackfromtheproductmanagerandthe
development.Thedesignsneedtobeimplementableinacostefficientmannerusingthetechnology
stackusedbytheproductdevelopmentteam.WhilesheistechnicallypartoftheUxDTeam,shesitsin
thedevelopmentareawiththedevelopmentteamasshehasfoundthistobeamoreeffectivewayto
transitiondesignknowledgetothedevelopersandtheirmanager.Shehasalsofoundittobeusefulto
enlistfriendsandpeopleoffthestreettodoinformalusabilitytestingofpaperprototypesaswellasthe
actualsoftware.Thisgivesherrealdatashecanusetohelpinfluencethetestdepartments
understandingtheusers.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 237
AppendixBKeyPoints
ThechaptersinPartIIdescribedtheprocessleadinguptotheacceptancedecisionfromthe
perspectivesoftheBusinessLead,ProductManager,TestManager,DevelopmentManager,User
ExperienceSpecialist,SolutionsArchitect,EnterpriseArchitect,OperationsManager,andLegal.For
convenience,wesummarizedthekeypointshere.ThereareKeyPointscommontoallrolesandothers
specifictoeachindividualrole.
ThefollowingtablelistsKeyPointsthatarerolespecificandwhatrolestheyarespecificto.
<Thetablewillbeupdatedandavailableontestingguidance.codeplex.com>

Figure1
KeyPointsspecifictoindividualroles

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 238
KeyPointsApplicabletoAllRoles
Treatacceptanceasaprocessnotanevent:Appreciatethefactthatacceptancetestingwillnot
beasinglediscreteevent,butaprocesswithadistinctstartandend.
Definethescopeoftesting:Haveaclearunderstandingofwhatwillbetestedbythesupplier
andwhatyouareexpectedtotest.Typically,supplierisexpectedtodothorough(e.g.)testing
ofallcapabilitiesandqualitycharacteristics(suchasperformance,reliability,usabilityetc.)
beforehandingoffsystemforacceptancetesting.Doesthesupplierhavetheenvironmentsfor
integrationtestingwithoutinhousesystems?
Delegatedecisions:Ifyoudonthaveenoughtimeorunderstandingtotakepartintheproject,
assignaproxytorepresentyouandhaveaclearunderstandingofthekindsofdecisionsthey
canmakewith/withoutconsultingwithyou.
Delegatework/testing:Haveaclearunderstandingofyour(organizations)abilitiesw.r.t.
testingskills.Contractotherpartiestohelpyoutestifnecessary.Butdontunderestimatethe
effortinvolvedinmanagingthetestingoutsourcerastheywillneedtolearnyourbusiness
domainandrequirementsbeforetheycanbeeffective.
Understandthetestenvironments:Understandwhereyouwilltest.Beclearonhowyoullget
thenewreleasecandidatesdelivered
Viewtestsasrequirements:Acceptancecriteria/testsshouldbearticulatedandprepared
beforethesoftwareisbuiltandsharedwithvendorasawaytofleshoutthedetailsofthe
requirements.
Estimateacceptanceeffort&durationwhilerealizingitisdifficulttoestimate:Estimatingthe
effort&durationofacceptancetestingishard;timeboxingthefinalacceptancetestingcycleis
commonbutrequirespastexperiencetocomeupwithreasonableduration/effort.Ifindoubt,
guesshighandallowforseveralcyclesofacceptancetestingtogivethevendortimetofixbugs.
Useincrementalacceptancetestingtoreduceuncertaintyofestimating:Canavoidsomeof
theuncertaintyandmostoftheriskbydoingincrementalacceptancetesting.Bugsfound
earlieridentifyambiguityinthespecwhilethereisstilltimetoaddressitandcanbefixedlong
beforethefinalacceptancetestphase
Understandthedecisionmakingprocess:Makesureyouhaveaclearunderstandingofthe
decisionmakingprocess.OtherpotentialdecisionmakersincludeTestManagers,Security
Manager,UserExperience/UsabilityManager,Architect,LegalDepartment,etc.Ensureyou
haveagreementonwhethertheymakeadecisionorprovidedataorarecommendationtoyou.
Communicateinformationflowclearly:Ensureanyonewhoyouexpecttoprovideyouwith
dataonwhichtobaseyourdecisionunderstandswhatdatatheyareprovidingyouandwhen.
E.g.TheTestManagerprovidestestresultstoyouandyoumaketheacceptancedecision.
Understandtheoperationalrequirements:Makesureyouunderstandtheoperational
requirementsforyoursystem.Whetheryouaresellingasoftwarepackagetocustomersor

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 239
sellingSoftwareasaService(SaaS),beawarethatwhoeveroperatesthesoftwarewillhave
operationalrequirementsoverandabovethefunctionalrequirementsoftheendusers.
Clarifydecisionmakingvsdecisionsupporting:Makesureyouhaveaclearunderstandingof
whetheryouarethegatekeeper(acceptancedecisionmaker)orasupplierofinformationto
theProductManagerforhertomakethedecision(preferred)
Maximizelearning:Workwiththeproductownertoensurethateveryoneontheteamhas
accesstothedefinitionofsuccessintheformofacceptancetestsbeforethesoftwareisbuilt
KeyPointsforBusinesslead
3. Treatacceptanceasaprocessnotanevent:Appreciatethefactthatacceptancetestingwill
notbeasinglediscreteevent,butaprocesswithadistinctstartandend.
4. Recognizetheremaybemanydecisionmakers:Theprocessmayrequiredecisionsbymany
partieseachsupportedbydatacollectionactivities.Eachpartylikelyhasverydifferent
requirementsandinterests;youneedtobeawareofthemandcommunicatethemtothe
supplier(s)toensuretheyaddressthem.
5. Definethescopeoftesting:Haveaclearunderstandingofwhatwillbetestedbythe
supplierandwhatyouareexpectedtotest.Typically,supplierisexpectedtodothorough
(e.g.)testingofallcapabilitiesandqualitycharacteristics(suchasperformance,reliability,
usabilityetc.)beforehandingoffsystemforAT.Doesthesupplierhavetheenvironmentsfor
integrationtestingwithoutinhousesystems?
6. Delegatedecisions:Ifyoudonthaveenoughtimeorunderstandingtotakepartinthe
project,assignaproxytorepresentyouandhaveaclearunderstandingofthekindsof
decisionstheycanmakewith/withoutconsultingwithyou.
7. Delegatework/testing:Haveaclearunderstandingofyour(organizations)abilitiesw.r.t.
testingskills.Contractotherpartiestohelpyoutestifnecessary.Butdontunderestimate
theeffortinvolvedinmanagingthetestingoutsourcerastheywillneedtolearnyour
businessdomainandrequirementsbeforetheycanbeeffective.
8. Understandthetestenvironments:Understandwhereyouwilltest.Beclearonhowyoull
getthenewreleasecandidatesdelivered
9. Viewtestsasrequirements:Acceptancecriteria/testsshouldbearticulated/prepared
beforethesoftwareisbuiltandsharedwithvendorasawaytofleshoutthedetailsofthe
requirements.
10. Recognizethechoiceofprocessimpactseverything:TwokeystylesofAT:FinalTestPhase
andIncrementalAT.Theformerconcentratesmostacceptanceactivitiesattheendofthe
project.Incrementalacceptancetestingspreadsouttheworkandreducesriskbutrequires
allpartiestoworkindifferentways.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 240
11. Estimateacceptanceeffort&durationwhilerealizingitisdifficulttoestimate:Estimating
theeffort&durationofacceptancetestingishard;timeboxingthefinalacceptancetesting
cycleiscommonbutrequirespastexperiencetocomeupwithreasonableduration/effort.If
indoubt,guesshighandallowforseveralcyclesofacceptancetestingtogivethevendor
timetofixbugs.
12. Useincrementalacceptancetestingtoreduceuncertaintyofestimating:Canavoidsomeof
theuncertaintyandmostoftheriskbydoingincrementalacceptancetesting.Bugsfound
earlieridentifyambiguityinthespecwhilethereisstilltimetoaddressitandcanbefixed
longbeforethefinalacceptancetestphase.
KeyPointsforProductManager
AllBusinessLeadsKeyPointsplus:
Understandthedecisionmakingprocess:Makesureyouhaveaclearunderstandingofthe
decisionmakingprocess.OtherpotentialdecisionmakersincludeTestManagers,Security
Manager,UserExperience/UsabilityManager,Architect,LegalDepartment,etc.Ensureyou
haveagreementonwhethertheymakeadecisionorprovidedataorarecommendationto
you.
Recognizethatcomplexproductsandorganizationshavecomplexdecisionmaking
processes:Thereareseveraldifferentwaystoorganizetheteamstodealwithverylarge
productsorproductsthataretheamalgamationofseveralproducts/productlines(e.g
SharePoint=Server+Office+IE).Thechoiceshouldconsidertheresultingdecisionmaking
model(e.g.componentteams,featureteams.)
Communicateinformationflowclearly:Ensureanyonewhoyouexpecttoprovideyouwith
dataonwhichtobaseyourdecisionunderstandswhatdatatheyareprovidingyouand
when.E.g.TheTestManagerprovidestestresultstoyouandyoumaketheacceptance
decision.
Ensureallsuppliersunderstandusersandusagemodel:Makesureyouunderstandthe
usersandcommunicatethistobothsupplierandtestorganizations.
Understandtheoperationalrequirements:Makesureyouunderstandtheoperational
requirementsforyoursystem.Whetheryouaresellingasoftwarepackagetocustomersor
sellingSoftwareasaService(SaaS),beawarethatwhoeveroperatesthesoftwarewill
haveoperationalrequirementsoverandabovethefunctionalrequirementsoftheend
users.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 241
KeyPointsforTestManager
Determineyourtestmandate:Makesureyouunderstandwhetheryouaredoingreadiness
assessmentoracceptancetestingorboth.Foracceptancetestingbesurethetestershave
realistictestsfortheuserstheyproxy.Considerusingpersonas.Forreadinessassessmentit
isimportanttohaveagoodrelationshipwiththedevelopmentteamtohelpthem
understandthequalityoftheproductandwhatwillimproveit.
Clarifydecisionmakingvsdecisionsupporting:Makesureyouhaveaclearunderstanding
ofwhetheryouarethegatekeeper(acceptancedecisionmaker)orasupplierofinformation
totheProductManagerforhertomakethedecision(preferred)
Getagreementfromtheproductownerthatthetestcasesreflecttherequirements:Aim
towritetheacceptancetestcasesfirstandprovidethemtothedevelopmentteamsbefore
thesoftwareisbuilttobeusedasaidestoassistinunderstandinghowthesystemshould
behave.ThisisknownasAcceptanceTestDrivenDevelopment.
Planamultiprongedteststrategy:Planforamultiprongedteststrategythatincludes
incrementaltestingandbothscriptedandexploratorytests.
Collaboratewiththedevelopmentteam:Aimforanincrementaldeliveryoffunctionalityto
supportincrementalacceptancetesting.Workwiththedevelopmentteamtodesignfor
testabilitytofacilitateautomatedtestingandenablingthemtobeabletoruntestson
demand.
Dotestautomationwhenfeasibleandinthewayitsupports(notdictates!)meetingyour
testobjectives.Considerhavingadedicatedtestautomationengineer.
Agreeonadefinitionofwhatdoneis:Getthevariouspartiesinvolvedtoagreeona
definitionofwhatdoneisforsoftwarebeforeitgoesthrougheachofthequalitygatesofthe
gatingmodelandincludeachecklistofrequirements.
KeyPointsforDevelopmentManager
Performreadinessassessments:Allsoftwareshouldbeadequatelytestedbythe
developmentteambeforepresentingittothecustomer(s).
Agreeontheminimumcrediblerequirement(MCR)aheadoftime:Theproductowner,test
organizationandthesuppliershouldallagreeontheMCRaheadoftime,thatis,beforethe
softwareisbuilt.
Useacceptanceteststodefinesuccess:UseacceptancetestsbasedontheMQRtodefine
successbeforethesoftwareisbuiltandensurethateveryoneonthedevelopmentteamhas
accesstothemandrunsthetestsastheybuildthesoftware.
Runthetestsaspartofreadinessacceptance:Runalltheteststhattheproductownerwill
runandsomeyouvedefined.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 242
Minimizeuntestedcode:Testasyougo,ideallyincrementaltesting.
Usetestingtopreventbugs:Testscanhelppreventbugsbeingintroducedduringthe
developmentcycle.Includefunctionaltesting,component/unittests,andparafunction
testingasearlyaspossible.
KeyPointsforUserExperienceSpecialist
Understandyourrole:Haveaclearunderstandingwiththeproductowner,product
managerorbusinessleadwhatyourroleshallbeinmakingtheacceptancedecision.
Maximizelearning:Workwiththeproductownertoensurethateveryoneontheteamhas
accesstothedefinitionofsuccessintheformofacceptancetestsbeforethesoftwareis
built.
Definetheusabilitycomponentoftheacceptancecriteriaasearlyaspossible:Workwith
thebusinessleadorproductmanagertodefinetheusabilitycomponentoftheacceptance
criteriaasearlyaspossible.
Addresstheusabilityrisksasearlyaspossible:Theareaswithhighestusabilityimpact
shouldbebuiltandtestedfirstsothatanyusabilityrelatedissuescanbeaddressedasearly
aspossibleinthedevelopmentcycle.
KeyPointsforOperationsManager
Understandroleinacceptancedecision:Youandeveryoneconcernedshouldbeclearon
yourroleinthemakingoftheacceptancedecision.Areyouadataprovidertothe
acceptancedecisionmakeroranacceptancedecisionmaker?
Identifywaystoreducethenumberoftest&fixcycles:Workwiththedevelopment
managertoidentifywaystoreducethenumberoftest&fixcyclesthroughearlydeliveryof
testablesoftwareandincrementalacceptancetesting.
Defineacceptancefromtheoperationalperspectiveasearlyaspossible:Ideallyyouhave
clearlydefinedacceptancecriteriabeforethesoftwareisbuilt.
KeyPointsforSolutionArchitect
Describethewholesolutiontoeveryone:Communicatethesolutioninitsentiretytothe
wholesupplierteamtoavoidtheMypartworksjustfine(evenifthesolutiondoesnt!)
mentality.
Ensureavailabilityofacceptancetests:Workwiththerequirementsanalystsandtestersto
ensurethattheacceptancetestsareavailabletothesupplierteambeforethesoftwareis
built.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 243
Educateabouttradeoffs:Workwiththeproductownertohelpthemunderstandwhatkinds
offunctionalityiseasytobuildandwhatishard.Helpthemgetthemostbangfortheir
money.
Unbundlefunctionality:Workwiththeproductownertohelpthemidentifythelowervalue
functionalitythatcouldpotentiallybedeferredincasethereisaneedtotradeoff
functionalityforqualityordeliverydate.
DefineSLAmetrics
Engageenterprisearchitects:Engagetheenterprisewidearchitects(responsiblefor
infrastructure,standards,technologydecisions,longtermvision,security,dataarchitecture,
etc.)earlysotheycanhelpyouunderstandwhatrequirestheirapprovalbeforeitbecomes
partofthecriticalpath.
KeyPointsforEnterpriseArchitect
Publisharchitecturalacceptancecriteria:Publishgeneralacceptancecriteriasothatall
supplierteamsunderstandtherulesofthegameandwhatkindsofapprovalstheyneedto
getfromyou.
ArticulatecrosssystemintegrationrequirementsandSLAs:Identifyapplicableintegration
requirementsandservicelevelagreements.Makesurethesupplierteamisawareofthem
andthattheyareacceptancecriteria.
Ensurestandardscompliance:Verifythatproductscomplywithapplicablestandards.
Encouragereuse:Verifythatproductteamsareawareofandcapitalizeonopportunitiesto
reusecomponentsanddata.
Engagesupplierteamsearly:Engagesupplierteamsearlytoensurethatinspectionsprevent
defectsratherthanfindthem.
Delegatenoncriticaldecisions:Ifitisnotimportanttobeinvolvedinadecisionthenbow
outandlettheprojectteammakethedecision.
Keepapprovalprocesslightweight:Maketheprocessforseekingguidanceandapprovalas
lightweightaspossible.Emphasizeeffectivenessofcommunicationovercomprehensive
documentation.Avoidoffloadingwork(e.g.moredocumentation)totheprojectteamsto
saveyoueffortifthatresultsinincreasingthetotalwork.
Bepragmatic:Bepragmaticastowhatprogressindividualprojectorproductteamsare
expectedtomaketowardslongtermarchitecturalobjectives.Theyareresponsibletothe
productowner,nottoyou.

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 244
KeyPointsforLegal
Includeclearstipulationsofanycontractualimplications:Acceptancetypicallyisa
conditionforrenderingapaymentandaffectstheapplicationofanywarrantyprovisionsand
potentiallyanyremediesavailabletothecustomer.
Discourageonesidedcontracts:Thecontractshouldencouragebothpartiestogettoa
mutuallyagreeableacceptanceasquicklyaspossible.Contractsthatpenalizeoneparty
oftenleadtodysfunctionalbehaviorthatresultsinloseloseoutcomes.
Clearlydefinehowmuchtimeisallowedforacceptancetests:Acceptancetestingisalways
timeboxed.Thereisalimitedamountoftimeforacceptanceandreportingdeficiencies.
Includeacleardefinitionofacceptancecriteria:Besidestestingperformedatthelocationof
supplier,andfunctionalandoperationaltestsperformedwhenthesoftwareisdeployedto
thecustomer,softwaresystemoftenarerequiredtosuccessfullyrunwithcurrentdataina
liveoperatingenvironmentforaperiodoftime,withminimumbugs,beforethesystemis
formallyaccepted.
Knowtheexplicitandimplicitcontractualconditions:Acceptancecanbeviewedintermsof
explicitprovisionsspecifiedinthesoftwaredevelopmentcontractandthetermsimpliedinto
thecontractbylaw(whichalsodependsonthegeographicjurisdiction).

AcceptanceTestEngineeringGuideVolumeI,ReleaseCandidate1 245