You are on page 1of 84
David Noble
David Noble

dnoble@exist.com Sr. Solutions Architect, Exist Global

Testing in Java

San Gabriel Valley Java Users Group http://www.sgvjug.org/ June 25, 2007

Goal

Motivate excellence in testing

with concepts, practices, and tools

2
2

Scope

requirements

continuous

integration

debugging

Testing concepts, practices, tools
Testing
concepts,
practices,
tools

profiling

quality

assurance

static code analysis

3
3

Agenda

Test-Driven DevelopmentAgenda Terminology Tools Application Discussion 4

TerminologyAgenda Test-Driven Development Tools Application Discussion 4

ToolsAgenda Test-Driven Development Terminology Application Discussion 4

ApplicationAgenda Test-Driven Development Terminology Tools Discussion 4

DiscussionAgenda Test-Driven Development Terminology Tools Application 4

4
4

TTeessttiinngg

SSyysstteemm CCoonncceepptt

SSyysstteemm RReeqquuiirreemmeennttss SSyysstteemm DDeessiiggnn

DDeettaaiilleedd DDeessiiggnn

IImmpplleemmeennttaattiioonn

http://www.flickr.com/photos/h2oalchemist/342496829/

5
5
Demo A Taste of TDD

Demo

A Taste of TDD

Test-Driven Development

Add test

Fail

Test-Driven Development Add test Fail Make small change Pass Commit changes Pass See also:

Make small change

Pass

Test-Driven Development Add test Fail Make small change Pass Commit changes Pass See also:

Commit changes

Development Add test Fail Make small change Pass Commit changes Pass See also: http://www.agiledata.org/essays/tdd.html 7
Development Add test Fail Make small change Pass Commit changes Pass See also: http://www.agiledata.org/essays/tdd.html 7
Development Add test Fail Make small change Pass Commit changes Pass See also: http://www.agiledata.org/essays/tdd.html 7

Pass

See also: http://www.agiledata.org/essays/tdd.html

7
7

Test-Driven Development

Add test

Add test

Fail

Add test Fail
Test-Driven Development Add test Fail Fix bug Pass Commit changes 8
Fix bug

Fix bug

Pass

Fix bug Pass
Test-Driven Development Add test Fail Fix bug Pass Commit changes 8

Commit changes

8
8

Test-Driven Development

Refactor

Code

Test-Driven Development Refactor Code Commit changes Add test Fail Make small change Pass Commit changes Pass

Commit changes

Commit changes
Commit changes
Commit changes
Test-Driven Development Refactor Code Commit changes Add test Fail Make small change Pass Commit changes Pass

Add test

Fail

Test-Driven Development Refactor Code Commit changes Add test Fail Make small change Pass Commit changes Pass

Make small change

Pass

Test-Driven Development Refactor Code Commit changes Add test Fail Make small change Pass Commit changes Pass

Commit changes

Test-Driven Development Refactor Code Commit changes Add test Fail Make small change Pass Commit changes Pass
Test-Driven Development Refactor Code Commit changes Add test Fail Make small change Pass Commit changes Pass

Pass

Test-Driven Development Refactor Code Commit changes Add test Fail Make small change Pass Commit changes Pass
Test-Driven Development Refactor Code Commit changes Add test Fail Make small change Pass Commit changes Pass
Test-Driven Development Refactor Code Commit changes Add test Fail Make small change Pass Commit changes Pass

Pass

Test-Driven Development Refactor Code Commit changes Add test Fail Make small change Pass Commit changes Pass
9
9

Test-Driven Development

Small stepsTest-Driven Development Frequent check-ins 10

Frequent check-insTest-Driven Development Small steps 10

10
10

Test-Driven Development

Deeper than “test first”Test-Driven Development Design for testability 11

Design for testabilityTest-Driven Development Deeper than “test first” 11

11
11

Test-Driven Development

Unit tests as example codeTest-Driven Development Interface before implementation Function drives form 12

Interface before implementationTest-Driven Development Unit tests as example code Function drives form 12

Test-Driven Development Unit tests as example code Interface before implementation Function drives form 12

Function drives form

12
12

Test-Driven Development

Loose couplingTest-Driven Development Services with clear responsibilities Mock objects 13

Services with clear responsibilitiesTest-Driven Development Loose coupling Mock objects 13

Mock objectsTest-Driven Development Loose coupling Services with clear responsibilities 13

13
13

Ta ste th e soup!

14
14

http://flickr.com/photos/basak/227480302/

Intent

Reality

Requirements, Implementation, Architecture, Deployment Design 15
Requirements,
Implementation,
Architecture,
Deployment
Design
15

Intent

Reality

Requirements, Implementation, Architecture, Deployment Design 16
Requirements,
Implementation,
Architecture,
Deployment
Design
16

Intent

Accountability Reality

Requirements, Testing, QA Implementation, Architecture, Deployment Design 17
Requirements,
Testing, QA
Implementation,
Architecture,
Deployment
Design
17

Intent

Accountability Reality

Requirements, Testing, QA Implementation, Architecture, Deployment Design 18 Reality Check!
Requirements,
Testing, QA
Implementation,
Architecture,
Deployment
Design
18
Reality
Check!

Terminology

Terminology QA – quality assurance Do the right things Do things right Ensure quality standards 19

QA – quality assurance

Do the right thingsTerminology QA – quality assurance Do things right Ensure quality standards 19

Do things rightTerminology QA – quality assurance Do the right things Ensure quality standards 19

Ensure quality standardsTerminology QA – quality assurance Do the right things Do things right 19

19
19

Terminology

Terminology Test – success criteria Is it right? Is it done? 20

Test – success criteria

Is it right?Terminology Test – success criteria Is it done? 20

Is it done?Terminology Test – success criteria Is it right? 20

20
20

Terminology

Test planTerminology Test case Test suite Test report Test coverage 21

Test caseTerminology Test plan Test suite Test report Test coverage 21

Test suiteTerminology Test plan Test case Test report Test coverage 21

Test reportTerminology Test plan Test case Test suite Test coverage 21

Test coverageTerminology Test plan Test case Test suite Test report 21

21
21

Terminology

Unit testTerminology Functional test Integration test System test Acceptance test Regression test 22

Functional testTerminology Unit test Integration test System test Acceptance test Regression test 22

Integration testTerminology Unit test Functional test System test Acceptance test Regression test 22

System testTerminology Unit test Functional test Integration test Acceptance test Regression test 22

Acceptance testTerminology Unit test Functional test Integration test System test Regression test 22

Regression testTerminology Unit test Functional test Integration test System test Acceptance test 22

22
22

Terminology

Non-functional testingTerminology Performance test Load test Soak test 23

Performance testTerminology Non-functional testing Load test Soak test 23

Load testTerminology Non-functional testing Performance test Soak test 23

Soak testTerminology Non-functional testing Performance test Load test 23

23
23

Terminology

Terminology Continuous integration 24

Continuous integration

24
24

Tools

Mock Objects Unit Tests Build

IDE

jMock, EasyMock

JUnit, TestNG

Ant, Maven

Eclipse, NetBeans

25
25

Tools

Web Apps JEE Servers

Databases

Selenium, HttpUnit

Cactus

DBUnit

26
26

Tools

Acceptance

Performance

Coverage

Selenium, FitNesse, Stiq

JMeter, Eclipse TPTP

Cobertura, Emma

27
27

Tools

Continuous

Integration

Cruise Control, Continuum, Hudson

28
28

Tools

Continuous Integration Acceptance Performance

Unit testing
Unit testing

Coverage Web Apps JEE Servers Databases Mock Objects

Unit Tests

Build

IDE

Cruise Control, Continuum, Hudson

Selenium, FitNesse, Stiq

JMeter, Eclipse TPTP

Cobertura, Emma

Selenium, HttpUnit

Cactus

DBUnit

jMock, EasyMock

JUnit, TestNG

Ant, Maven

Eclipse, NetBeans

29
29
Tools
Tools

Demo

Resources: SGV JUG

Web Site

Mailing List (Hosted by SGV Linux Users Group)

java-sig@sgvlug.net

31
31

Resources: Unit Testing

JUnit Framework

JUnit FAQ

http://junit.sourceforge.net/doc/faq/faq.htm

JUnit with Ant

JUnit with Maven

TestNG Framework

Jester – Mutation Testing

34
34

Resources: Mock Objects

Mock Objects Blog

EasyMock Library

jMock Library

35
35

Resources: Unit & Functional

XmlUnit Library

DBUnit Library

HttpUnit Library

Cactus for JEE Server Code

Cargo for JEE Server Code

36
36

Resources: Test Coverage

Cobertura

http://cobertura.sourceforge.net/

http://www.ibm.com/developerworks/java/library/j-cobertura/

Cobertura with Ant

Cobertura with Maven

Emma

37
37

Resources: Perf. & Load

JUnitPerf

JMeter

Eclipse TPTP (Test & Performance Tools Platform)

Netbeans Profiler

38
38

Resources: Acceptance Tests

Selenium for Web Apps

http://www.openqa.org/selenium/

Automating Selenium (“Remote Control”)

Selenium & Maven

FitNesse

Stiq

39
39

Resources: Cont. Integration

Cruise Control

Continuum

Hudson

40
40

Resources: App used in Demo

Open Source Medical Record System

Prerequisites: SVN, MySQL 5, Tomcat 5

SVN Repository

Installation

Edit build.properties and properties.xml to set tomcat.user/password, or add test/test to $CATALINA_HOME/conf/tomcat-users.xml file. Edit build.properties and/or properties.xml to set tomcat.home file. Comment out creation of "test" MySQL user in schema if it already exists. Default username/password is admin/test.

41
41
Thank you To all who participated!

Thank you

To all who participated!

David Noble dnoble@exist.com Sr. Solutions Architect, Exist Global Testing in Java San Gabriel Valley Java
David Noble
dnoble@exist.com
Sr. Solutions Architect, Exist Global
Testing in Java
San Gabriel Valley Java Users Group
http://www.sgvjug.org/
June 25, 2007

My background:

Mostly C & Perl from 1990 – 1997 Mostly Java since then. Telemetry processing, mission ops, EIS @ JPL. BPM, workflow, assorted biz apps.

Goal

Motivate excellence in testing

with concepts, practices, and tools

2
2

Goal falls within the larger context of pursuing excellence in software development.

Scope

requirements

continuous

integration

debugging

Testing concepts, practices, tools profiling
Testing
concepts,
practices,
tools
profiling

quality

assurance

static code analysis

3
3

Agenda

Test-Driven DevelopmentAgenda Terminology Tools Application Discussion 4 4

TerminologyAgenda Test-Driven Development Tools Application Discussion 4 4

ToolsAgenda Test-Driven Development Terminology Application Discussion 4 4

ApplicationAgenda Test-Driven Development Terminology Tools Discussion 4 4

DiscussionAgenda Test-Driven Development Terminology Tools Application 4 4

4
4
SySyststeemm CCononcepceptt SySyststeemm RReeqquuiremiremenenttss SySyststeemm DesigDesignn DetDetaailileded
SySyststeemm CCononcepceptt SySyststeemm RReeqquuiremiremenenttss SySyststeemm DesigDesignn DetDetaailileded
SySyststeemm CCononcepceptt
SySyststeemm RReeqquuiremiremenenttss
SySyststeemm DesigDesignn
DetDetaailileded DeDesisiggnn
IImmppllemenementtaattionion
http://www.flickr.com/photos/h2oalchemist/342496829/
TTeessttiinngg
5
5

In traditional software development, testing is usually at the end of the schedule. If development runs late, the testing often suffers.

Demo A Taste of TDD
Demo
A Taste of TDD

Test-Driven Development

Add test

Test-Driven Development Add test Fail Make small change Pass Commit changes Pass See also:

Fail

Make small change

Test-Driven Development Add test Fail Make small change Pass Commit changes Pass See also:

Pass

Commit changes

Development Add test Fail Make small change Pass Commit changes Pass See also: http://www.agiledata.org/essays/tdd.html 7

Pass

See also: http://www.agiledata.org/essays/tdd.html

Test-Driven Development

Add test

Add test

Fail

Add test Fail
Test-Driven Development Add test Fail Fix bug Pass Commit changes 8 8
Fix bug

Fix bug

Pass

Fix bug Pass
Test-Driven Development Add test Fail Fix bug Pass Commit changes 8 8

Commit changes

8
8

Test-Driven Development

Refactor

Code

Test-Driven Development Refactor Code Commit changes Add test Fail Make small change Pass Commit changes
Commit changes

Commit changes

Commit changes
Commit changes
Commit changes Add test Fail
Add test

Add test

Fail

Add test Fail
Commit changes Add test Fail
Make small change

Make small change

Pass

Make small change Pass
Development Refactor Code Commit changes Add test Fail Make small change Pass Commit changes Pass Pass

Commit changes

Pass

Development Refactor Code Commit changes Add test Fail Make small change Pass Commit changes Pass Pass
Development Refactor Code Commit changes Add test Fail Make small change Pass Commit changes Pass Pass

Pass

Development Refactor Code Commit changes Add test Fail Make small change Pass Commit changes Pass Pass
9
9

Test-Driven Development

Small stepsTest-Driven Development Frequent check-ins 10 10

Frequent check-insTest-Driven Development Small steps 10 10

10
10

Test-Driven Development

Deeper than “test first”Test-Driven Development Design for testability 11 11

Design for testabilityTest-Driven Development Deeper than “test first” 11 11

11
11

Test-Driven Development

Unit tests as example codeTest-Driven Development Interface before implementation Function drives form 12 12

Interface before implementationTest-Driven Development Unit tests as example code Function drives form 12 12

Test-Driven Development Unit tests as example code Interface before implementation Function drives form 12 12

Function drives form

12
12

Test-Driven Development

Loose couplingTest-Driven Development Services with clear responsibilities Mock objects 13 13

Services with clear responsibilitiesTest-Driven Development Loose coupling Mock objects 13 13

Mock objectsTest-Driven Development Loose coupling Services with clear responsibilities 13 13

13
13
T a ste t h e sou p! 14 http://flickr.com/photos/basak/227480302/
T a ste
t h e
sou p!
14
http://flickr.com/photos/basak/227480302/

An old man calls the waiter over to his table. "What would you like, sir?" asks the waiter. "Taste the soup," he says. "Is the soup too cold?" asks the waiter. "Taste the soup," says the old man. "Is it too salty?" asks the waiter. "Taste the soup!" says the old man. "Is there a fly in it?" asks the waiter. "Taste the soup!" the old man insists. The waiter looks down: "There's no spoon!" "Aha!" exclaims the old man.

Intent Reality Requirements, Implementation, Architecture, Deployment Design 15
Intent
Reality
Requirements,
Implementation,
Architecture,
Deployment
Design
15
Intent Reality Requirements, Implementation, Architecture, Deployment Design 16 Gap!
Intent
Reality
Requirements,
Implementation,
Architecture,
Deployment
Design
16
Gap!

Intent

Accountability Reality

Requirements, Testing, QA Implementation, Architecture, Deployment Design 17
Requirements,
Testing, QA
Implementation,
Architecture,
Deployment
Design
17

Intent

Accountability Reality

Requirements, Testing, QA Implementation, Architecture, Deployment Design 18 Reality Check!
Requirements,
Testing, QA
Implementation,
Architecture,
Deployment
Design
18
Reality
Check!

Terminology

Terminology QA – quality assurance Do the right things Do things right Ensure quality standards Miriam-Webster:

QA – quality assurance

Do the right thingsTerminology QA – quality assurance Do things right Ensure quality standards Miriam-Webster: 19 “a program for

Do things rightTerminology QA – quality assurance Do the right things Ensure quality standards Miriam-Webster: 19 “a program

Ensure quality standardsQA – quality assurance Do the right things Do things right Miriam-Webster: 19 “a program for

Miriam-Webster:

19
19

“a program for the systematic monitoring and evaluation of the various aspects of a project, service, or facility to ensure that standards of quality are being met”

Terminology

Terminology Test – success criteria Is it right? Is it done? 20 20

Test – success criteria

Is it right?Terminology Test – success criteria Is it done? 20 20

Is it done?Terminology Test – success criteria Is it right? 20 20

20
20

Terminology

Test planTerminology Test case Test suite Test report Test coverage 21 Test plan: document Test case: implementation

Test caseTerminology Test plan Test suite Test report Test coverage 21 Test plan: document Test case: implementation

Test suiteTerminology Test plan Test case Test report Test coverage 21 Test plan: document Test case: implementation

Test reportTerminology Test plan Test case Test suite Test coverage 21 Test plan: document Test case: implementation

Test coverageTerminology Test plan Test case Test suite Test report 21 Test plan: document Test case: implementation

21
21

Test plan: document Test case: implementation of specific test. Test suite: collection of test cases. Test report: results of running the tests. Test coverage: description of how much code (and which code) was exercised by tests.

Terminology

Unit testTerminology Functional test Integration test System test Acceptance test Regression test 22 Unit test: typically at

Functional testTerminology Unit test Integration test System test Acceptance test Regression test 22 Unit test: typically at

Integration testTerminology Unit test Functional test System test Acceptance test Regression test 22 Unit test: typically at

System testTerminology Unit test Functional test Integration test Acceptance test Regression test 22 Unit test: typically at

Acceptance testUnit test Functional test Integration test System test Regression test 22 Unit test: typically at the

Regression testFunctional test Integration test System test Acceptance test 22 Unit test: typically at the granularity of

22
22

Unit test: typically at the granularity of a class. Functional test: a feature or use case. Integration test: multiple modules. System test: all the modules. Acceptance test: defined by user/customer.

Terminology

Non-functional testingTerminology Performance test Load test Soak test 23 Non-functional testing: the “-ilities”. Performance test: how

Performance testTerminology Non-functional testing Load test Soak test 23 Non-functional testing: the “-ilities”. Performance test:

Load testTerminology Non-functional testing Performance test Soak test 23 Non-functional testing: the “-ilities”. Performance

Soak testTerminology Non-functional testing Performance test Load test 23 Non-functional testing: the “-ilities”. Performance

23
23

Non-functional testing: the “-ilities”. Performance test: how fast. Load test: how many. Soak test: how long.

Terminology

Terminology Continuous integration 24 This is an alternative to “big bang integration”, otherwise known as

Continuous integration

24
24

This is an alternative to “big bang integration”, otherwise known as “integration hell”.

Tools

Mock Objects Unit Tests Build

IDE

jMock, EasyMock

JUnit, TestNG

Ant, Maven

Eclipse, NetBeans

25
25

Tools

Web Apps

Selenium, HttpUnit

JEE Servers

Cactus

Databases

DBUnit

26
26

HttpUnit is better suited to situations where you want to examine the DOM at a low level. If you're looking for convenience, and don't need to get deep into the details, look at Selenium instead.

Tools

Acceptance

Performance

Coverage

Selenium, FitNesse, Stiq

JMeter, Eclipse TPTP

Cobertura, Emma

27
27

Selenium IDE, a Firefox plugin, lets users or product managers create their own scenarios. Selenium is also useful during development, to quickly fast forward through actions to get the web application into a desired state.

Tools

Continuous

Integration

Cruise Control, Continuum, Hudson

28
28

Tools

Continuous Integration Acceptance Performance Coverage Web Apps JEE Servers Databases Mock Objects Unit Tests
Continuous Integration
Acceptance
Performance
Coverage
Web Apps
JEE Servers
Databases
Mock Objects
Unit Tests
Build
IDE
Unit testing

Cruise Control, Continuum, Hudson

Selenium, FitNesse, Stiq

JMeter, Eclipse TPTP

Cobertura, Emma

Selenium, HttpUnit

Cactus

DBUnit

jMock, EasyMock

JUnit, TestNG

Ant, Maven

Eclipse, NetBeans

29
29

The items in green aren't really a stack, just a collection of technology-specific libraries that can be useful for creating test cases.

Demo Tools
Demo
Tools

Be sure to design Selenium tests so they can be repeatable. Either delete data when they are done, or always start from a fixed known state.

Resources: SGV JUG

Web Site

http://sgvlug.net/mailman/listinfo/java-sig

Mailing List (Hosted by SGV Linux Users Group)

http://sgvlug.net/mailman/listinfo/java-sig

java-sig@sgvlug.net

31
31

Resources: TDD

Articles

http://www.agiledata.org/essays/tdd.html http://en.wikipedia.org/wiki/Test_driven_development http://www.testdriven.com/modules/xoopsfaq/ http://www.artima.com/intv/testdriven.html http://www.google.com/search?hl=en&q=test+driven+development (really!)

32
32

Resources: IDE & Build Tools

Eclipse IDE

http://www.eclipse.org/

NetBeans IDE

http://www.netbeans.org/

Ant

http://ant.apache.org/

Maven 2

http://maven.apache.org/

http://www.devzuz.com/web/guest/products/resources#BBWM

Maven & Eclipse

http://m2eclipse.codehaus.org/

Maven & Netbean

http://maven.apache.org/guides/mini/guide-ide-netbeans/guide-ide-netbeans.html

33
33

Resources: Unit Testing

JUnit Framework

http://junit.org/

JUnit FAQ

http://junit.sourceforge.net/doc/faq/faq.htm

JUnit with Ant

http://ant.apache.org/manual/OptionalTasks/junit.html

http://ant.apache.org/manual/OptionalTasks/junitreport.html

JUnit with Maven

http://maven.apache.org/plugins/maven-surefire-plugin/

TestNG Framework

http://testng.org/doc/

Jester – Mutation Testing

http://jester.sourceforge.net/

http://www.ibm.com/developerworks/java/library/j-jester/

34
34

Resources: Mock Objects

Mock Objects Blog

http://mockobjects.com/

EasyMock Library

http://easymock.org/

jMock Library

http://jmock.org/

35
35

Resources: Unit & Functional

XmlUnit Library

http://xmlunit.sourceforge.net/

DBUnit Library

http://dbunit.sourceforge.net/

HttpUnit Library

http://httpunit.sourceforge.net/

Cactus for JEE Server Code

http://jakarta.apache.org/cactus/

Cargo for JEE Server Code

http://cargo.codehaus.org/

36
36

Resources: Test Coverage

Cobertura

http://cobertura.sourceforge.net/

http://www.ibm.com/developerworks/java/library/j-cobertura/

Cobertura with Ant

http://cobertura.sourceforge.net/anttaskreference.html

Cobertura with Maven

http://mojo.codehaus.org/cobertura-maven-plugin/usage.html

Emma

http://emma.sourceforge.net/

37
37

Resources: Perf. & Load

JUnitPerf

http://clarkware.com/software/JUnitPerf.html

JMeter

http://jakarta.apache.org/jmeter/

Eclipse TPTP (Test & Performance Tools Platform)

http://www.eclipse.org/tptp/

Netbeans Profiler

http://profiler.netbeans.org/

38
38

Resources: Acceptance Tests

Selenium for Web Apps

http://www.openqa.org/selenium/

http://www.openqa.org/selenium-ide/download.action

Automating Selenium (“Remote Control”)

http://www.openqa.org/selenium-rc/

http://wiki.openqa.org/display/SRC/Selenium-RC+and+Continuous+Integration

Selenium & Maven

http://raibledesigns.com/rd/entry/selenium_plugin_for_maven

FitNesse

http://fitnesse.org/

Stiq

http://storytestiq.sourceforge.net/

39
39

Resources: Cont. Integration

Cruise Control

http://cruisecontrol.sourceforge.net/

Continuum

http://maven.apache.org/continuum/

Hudson

https://hudson.dev.java.net/

40
40

Resources: App used in Demo

Open Source Medical Record System

http://www.openmrs.org/

Prerequisites: SVN, MySQL 5, Tomcat 5

http://subclipse.tigris.org/update_1.2.x

http://mysql.org/

http://tomcat.apache.org/

SVN Repository

http://svn.openmrs.org/openmrs/branches/alpha

Installation

Edit build.properties and properties.xml to set tomcat.user/password, or add test/test to $CATALINA_HOME/conf/tomcat-users.xml file. Edit build.properties and/or properties.xml to set tomcat.home file. Comment out creation of "test" MySQL user in schema if it already exists. Default username/password is admin/test.

41
41
Thank you To all who participated!
Thank you
To all who participated!