Acceptance Test Driven Development

Naresh Jain naresh@agilefaqs.com

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 1

Tutorial Schedule
• Part discussion, part exercise • Starts light and fluffy • You’ll get your hands dirty by the end • Questions welcomed all the time
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 2

Welcome

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 3

Welcome
• Continuum

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 3

Welcome
• Continuum • Hopes and Concerns

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 3

Getting Started Exercise

Warmup Scenarios

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 4

Getting Started Exercise

Warmup Scenarios
• Going out for Movie (THX sound and Digital
projection)

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 4

Getting Started Exercise

Warmup Scenarios
• Going out for Movie (THX sound and Digital
projection)

• Going out for meal (one veg.)

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 4

Getting Started Exercise

Warmup Scenarios
• Going out for Movie (THX sound and Digital
projection)

• Going out for meal (one veg.) • Going shopping ($50)

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 4

Getting Started Exercise

Warmup Scenarios
• Going out for Movie (THX sound and Digital
projection)

• Going out for meal (one veg.) • Going shopping ($50) • You are a party of 5.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 4

What are Acceptance Tests?

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 5

What are Acceptance Tests?

Criteria for Completion
• Written by the “Customer” • Given to developers

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 6

What are Acceptance Tests?

Communication
• Forces customers, testers, and developers to
work together

• creation • implementation • execution • maintenance
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 7

What are Acceptance Tests?

Feedback
• A source of data • A tool for project management

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 8

Data From Acceptance Tests
Total ATs Failing ATs Passing ATs

90 72 54 36 18 0 1 2 3 4 5 6 7 8 9 10
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 9

Acceptance Tests: A Critical Piece of Agile

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 10

Acceptance Tests: A Critical Piece of Agile

Traditional Approach
1 May Analysis Design Implementation DFD ERD DD ST 1 Jul 1 Sep 1 Nov

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 11

Acceptance Tests: A Critical Piece of Agile

The Agile Aproach
May 1 Jul 1 Sep 1 Nov 1

Analysis Test Implementation Design

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 12

Acceptance Tests: A Critical Piece of Agile

The Agile Approach

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13

Acceptance Tests: A Critical Piece of Agile

The Agile Approach
Acceptance Criteria

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13

Acceptance Tests: A Critical Piece of Agile

The Agile Approach
Acceptance Criteria

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13

Acceptance Tests: A Critical Piece of Agile

The Agile Approach
Iteration

Acceptance Criteria

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13

Acceptance Tests: A Critical Piece of Agile

The Agile Approach
Iteration

Acceptance Criteria

Automated Acceptance Tests

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13

Acceptance Tests: A Critical Piece of Agile

The Agile Approach
Iteration

Acceptance Criteria

Automated Acceptance Tests

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13

Acceptance Tests: A Critical Piece of Agile

The Agile Approach
Iteration

Acceptance Criteria

Automated Acceptance Tests

Automated Acceptance Tests
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13

Acceptance Tests: A Critical Piece of Agile

The Agile Approach
Iteration

Acceptance Criteria

Automated Acceptance Tests

Automated Acceptance Tests
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 13

Acceptance Tests: A Critical Piece of Agile

The Agile Approach
Iteration

Acceptance Criteria

Automated Acceptance Tests

Automated Acceptance Tests
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved.

Acceptance Criteria

13

Acceptance Tests: A Critical Piece of Agile

Criteria for DONE
• Every story must have at least one
Acceptance Test

• A story is not DONE until it passes it’s
Acceptance Tests

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 14

Acceptance Tests: A Critical Piece of Agile

Manual Acceptance Tests

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 15

Acceptance Tests: A Critical Piece of Agile

Manual Acceptance Tests
Manual Acceptance Tests

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 15

Acceptance Tests: A Critical Piece of Agile

Manual Acceptance Tests
Manual Acceptance Tests

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 15

Acceptance Tests Are Automated

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 16

Acceptance Tests: A Critical Piece of Agile

The Button
• How often would
you press it? press it? it?

• When would you • Who would press

Testers, Developers, Managers, Customers, Spectators, etc.
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 17

Acceptance Tests: A Critical Piece of Agile

Automated Tests Require:
• High level, yet, precise language • Unambiguous detail • “Examples” • Interaction with the system
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 18

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 19

Criteria for DONE

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 19

Criteria for DONE +

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 19

Criteria for DONE + Automated

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 19

Criteria for DONE + Automated

Executable Specification
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 19

Acceptance Tests: A Critical Piece of Agile

Executable Specification
• A new paradigm for testing • Puts quality first • Removes ambiguity from requirements
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 20

Who Writes Acceptance Tests?

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 21

Who Writes Acceptance Tests?

The Customer
• Yeah right! Then who? • The Customer Role • Stake holder • Business Analyst • Quality Assurance • Product Owner • Developer

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 22

Who Writes Acceptance Tests?

Tests Get Technical
• The “Customer” may need technical help to
write tests

• Developers and QAs are technical • Pair test authoring
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 23

Who Writes Acceptance Tests?

Business Rules Get Fuzzy
• Sometimes developers need help
understanding tests

• Customers know business rules • Pair test implementation
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 24

Who Writes Acceptance Tests?

Business Decisions are Hard To Make
challenged to make decisions

• Sometimes, the “customer” will be • The Man ($$$) can help • Do you see a pattern of communication?
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 25

Exercise #1

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 26

Exercise #1

The Login Test
• Write a test plan, in plain text, for
the business rules of logging in.

• Web application • User credentials are stored in
relational database “Welcome” page

• Successful login redirects to
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 27

Writing Good Acceptance Tests

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 28

Writing Good Acceptance Tests

Login Test Possibilities
1. Direct browser to URL for login page

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 29

Writing Good Acceptance Tests

Login Test Possibilities
1. Direct browser to URL for login page

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 29

Writing Good Acceptance Tests

Login Test Possibilities
1. Direct browser to URL for login page 1. Enter the username ‘wallace’

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 29

Writing Good Acceptance Tests

Login Test Possibilities
1. Direct browser to URL for login page 1. Enter the username ‘wallace’

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 29

Writing Good Acceptance Tests

Login Test Possibilities
1. Direct browser to URL for login page 1. Enter the username ‘wallace’

Build a Testable Environment First
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 29

Writing Good Acceptance Tests

BOC

Build Operate Check
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 30

Writing Good Acceptance Tests

Login Test Possibilities
1. Add some users to the system

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 31

Writing Good Acceptance Tests

Login Test Possibilities
1. Add some users to the system

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 31

Writing Good Acceptance Tests

Login Test Possibilities
1. Add some users to the system 3. Enter a value into the username field

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 31

Writing Good Acceptance Tests

Login Test Possibilities
1. Add some users to the system 3. Enter a value into the username field

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 31

Writing Good Acceptance Tests

Login Test Possibilities
1. Add some users to the system 3. Enter a value into the username field

Be Specific
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 31

Writing Good Acceptance Tests

Tests are Examples
• Use concrete examples • Specify concrete behavior • No ambiguity allowed
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 32

Writing Good Acceptance Tests

Login Test Possibilities
1. Insert into User table values (’wallace’, ‘ilikecheeze’)

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 33

Writing Good Acceptance Tests

Login Test Possibilities
1. Insert into User table values (’wallace’, ‘ilikecheeze’)

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 33

Writing Good Acceptance Tests

Login Test Possibilities
1. Insert into User table values (’wallace’, ‘ilikecheeze’) 2. Open a browser to the URL http:// localhost/myapp

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 33

Writing Good Acceptance Tests

Login Test Possibilities
1. Insert into User table values (’wallace’, ‘ilikecheeze’) 2. Open a browser to the URL http:// localhost/myapp

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 33

Writing Good Acceptance Tests

Login Test Possibilities
1. Insert into User table values (’wallace’, ‘ilikecheeze’) 2. Open a browser to the URL http:// localhost/myapp

Avoid Implementation Details
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 33

Writing Good Acceptance Tests

Avoid Implementation Details
Tests UI

System
Database
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 34

Writing Good Acceptance Tests

Login Test: Possible Solution
• Add user to system: (’wallace’, ‘ilikecheeze’) • Process login with username ‘wallace’ and
password ‘blah’

• Check login failed • Process login with username ‘wallace’ and
password ‘ilikecheeze’

• Check login succeeded
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 35

Tools

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 36

Tools

Commercial Tools
• WinRunner • Silk • RFT
TestPartner QTP Squish EggPlant TestComplete WindowTester

Are not suitable for Acceptance Testing in an Agile environment
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 37

Tools

Open Source Options
• FIT • FitNesse • Selenium
Sahi Watir Abbot Frankenstein SharpRobo WET

Among the few tools that support Test Driven Development
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 38

Tools

FIT
• Framework for Integrated Tests • Created by Ward Cunningham • Open Source • The most accepted solution for agile
acceptance testing
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 39

Tools

FitNesse
• Environment build around FIT • Makes everything easier • Created by Object Mentor, Inc. • Open Source
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 40

FIT
- Tests written in HTML - Tests are executed on the command line - Tables are executed - Non-table markup is ignored - Tables map to Fixtures - Fixtures are code that is aware of the system - Supplies foundational Fixtures - Implementations ported to many languages

FitNesse
- Stand alone web server - Is a wiki - Tests written in wiki text - Tests are executed from within the wiki - Translates tests into HTML - Uses FIT to execute tests - Supports test suites - Supports variables in tests - Supports test refactoring - Written in Java - Supports FIT implementations in any language

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 41

Thinking in Tables

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 42

Thinking in Tables

Only Tables Execute
Ignored

Executed

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 43

Thinking in Tables

Foundational Table Structure
Name of Fixture Interaction with Application

• Table structure depends on type of Fixture
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 44

Thinking in Tables

3 Foundation Fixtures
• Column Fixture • Row Fixture • Action Fixture
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 45

Thinking in Tables

Column Fixture
import fit.ColumnFixture;

package eg; // Copyright (c) 2002 Cunningham & Cunningham // Released under the terms of the GNU Genera Public …

public class Division extends ColumnFixture { public float numerator; public float denominator; public float quotient() { return numerator / denominator; } }

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 46

Thinking in Tables

Row Fixture
Analogous to comparing against rows in a database table
package fitnesse.fixtures; import fit.RowFixture; public class EmployeePayRecordsRowFixture extends RowFixture { public Object[] query() throws Exception { EmployeePayRecord[] records = new EmployeePayRecord[2]; records[0] = new EmployeePayRecord(1, 1000); records[1] = new EmployeePayRecord(2,2000); return records; } public Class getTargetClass() { return EmployeePayRecord.class; } }
public class EmployeePayRecord { public int id; private double salary; public EmployeePayRecord(int id, double salary){ this.id = id; this.salary = salary; } public double pay() { return salary; } }

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 47

Thinking in Tables

Action Fixture

Think GUI window
Counter Window
Counter: Counter: 6 Count

public class CountFixture extends Fixture { private int counter = 0; public void count() { counter++; } public int counter() { return counter; } }

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 48

Exercise #2

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 49

Exercise #2

Login With Tables
• Write the Login test using tables • Keep in mind what we learned from the
previous exercise

• BOC • Examples • Avoid Implementation Details
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 50

Exercise #2

Possible Solution

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 51

FitLibrary
• Extension to FIT • Written by Rick Mugridge • Adds some handy Fixtures
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 52

FitLibrary

FitLibrary Fixtures
• ArrayFixture for ordered lists • SetFixture for unordered lists • SetUpFixture • Supports • Graphics • Tree structures • Nested Tables
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 53

FitLibrary

DoFixture
• Broken tables • Highly readable • Flexibility

Text

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 54

Wiki

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 55

Wiki

What is it?
• A collaborative web site • Editable by any • Created by Ward Cunningham • Every project should have one • http://c2.com/wiki • http://en.wikipedia.com
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 56

Wiki

Creating Tests
• Use Wiki syntax to create a page with test tables • Label the page as a Test Page • Use a page name of the form Test… • Turn on the Test property • Make sure your Fixtures are in the classpath • Use !path widget • Mechanics • !path values are concatenated • Java command to start FitServer is executed • Testable HTML is passed to FitServer • FitServer runs the tests • Results are passed back to FitNesse
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 57

Wiki

Creating Suites
There are 2 ways to make Suites

• •

• Create a page with the Suite property • Created test pages inside this page • When the suite is executed, all child test pages will • !see <name of test page> • All “included” tests pages will be included in the
suite execution

Set the Suite property

Use the !see widget

be included in the suite execution

Run a Suite by clicking the Suite button
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 58

Hands-on Session
• Conference Proposal Submission Portal • Some sample Stories
• • • • • • • •
Should be able to submit new proposal Should be able to list all submitted proposal Submitting proposal with same title should display appropriate error message Should be able to delete submitted proposal based on the title Should be able to delete submitted proposal based on the title Should be able to search proposals by title Should be able to search proposals by ID Should be able to find all proposal by an author's name

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 59

Break

http://www.fitnesse.org

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 60

Patterns

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 61

Patterns

Organizing Tests
• Allowing customers to add new tests
without breaking the build

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 62

Patterns

Version Control
• Keeping the acceptance test in version
control with the code.

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 63

Patterns

Cross-Functional Pairing
• Using FitNesse based acceptance tests for
collaboration between cross-functional team members.

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 64

Patterns

ATDD

• Acceptance Test Driven Development

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 65

Patterns

CSTT

• Cleanup, Setup, Test, Teardown

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 66

Patterns

Independent Tests
• Tests shouldn’t depend on each other. • Tests leave the system in the same state it
started in.

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 67

Patterns

Dynamic Stubbing

• Avoiding complications of external systems.

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 68

Patterns

Non-Production Setup/Teardown

• Using non-production light weigh code for
setup and teardown.

• Helps test only what you want to test.

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 69

Patterns

Suite Levels
• Creating different levels of suites depending
on the depth/level of feedback desired.

• Smoke, Current Iteration/Sprint, Regression

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 70

Patterns

DRY

• Using !include to avoid repeating yourself.

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 71

Patterns

Make it Real
• Write ATs as close as possible to the real
environment.

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 72

Patterns

Fixture Evolution
• Allow Fixture implementation to evolve over
time.

• Treat fixtures as first class citizens.

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 73

Patterns

At Least One Test/Story
• Every story should have at least one
acceptance test

• Avoid long/multipurpose tests.

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 74

Anti-Patterns

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 75

Anti-Patterns

Developer ATs
• Developers writing acceptance tests by
themselves, for themselves.

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 76

Anti-Patterns

Unit Testing
• Don’t write ATs at the unit testing level • Unit tests are implementation specific • ATs are NOT implementation specific
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 77

Anti-Patterns

QA Testing Tool
• Hard to write tests up front. • Perhaps only on large projects.

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 78

Anti-Patterns

Silver Bullet
• Trying to use FitNesse for all types of
Acceptance Tests

• UI testing • XML testing
copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 79

Anti-Patterns

Test After
• Writing tests after the code is already
written.

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 80

Anti-Patterns

Hidden Test Data

• Hiding test data in the fixtures.

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 81

Anti-Patterns

Implementation Dependant ATs

• Making test pages (tables) dependent on

implementation details and data structures.

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 82

Anti-Patterns

Logging in Your Fixtures
• Putting log statements or print statements in
the fixture code.

• Fixtures are probably too complicated.

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 83

The End

http://www.fitnesse.org

copyright (c) 2007 by Micah Martin & Naresh Jain. All rights reserved. 84

Sign up to vote on this title
UsefulNot useful