TestNG

… and improving use of JUnit

 Agenda
• TestNG testing framework

• Improving use of JUnit

• Testing integrated classes vs. 1 class

2
 What is TestNG?
• Automated testing framework
• NG = Next Generation
• Similar to JUnit (especially JUnit 4)
• Not a JUnit extension (but inspired by JUnit)
• Designed to be better than JUnit, especially
when testing integrated classes
• Created by Dr. Cédric Beust (of Google)
• Open source (http://testng.org)

3
 History
• sUnit for Smalltalk (~1998, Kent Beck)
• JUnit (~2000, Kent Beck & Erich Gamma)
– Latest 3.x: 3.8.2 (March 2006)

• TestNG (~2004)
– Latest: 5.8 (March 2008)

• JUnit 4.x (2006)

– Latest: 4.5 (August 2008)
4
 What is TestNG?
• TestNG is a testing framework inspired
from JUnit, but introducing some new
functionalities that make it more powerful
and easier to use.

5
 TestNG Features
• Supports annotations.
• TestNG uses more Java and OO features.
• Supports testing integrated classes
• Separates compile-time test code from run-
time configuration/data info.
• Flexible runtime configuration.
• Introduces ‘test groups’.
• Support for multi-threaded testing.
6
 TestNG philosophy
• Use more Java and OO features
• Feature-rich (JUnit: simplicity is valued)
• Support testing integrated classes (e.g., by
default, don’t create a new test class
instance for every test method).
• Separate compile-time test code from run-
time configuration/data info.

7
 Basic Testing Terms

Test class Class(es) under test

(“Production” class)
“CUT” (class under test)
“SUT” (system under test)

8
Test class/method (JUnit 4 vs. TestNG)
import org.junit.Test; // JUnit 4
... or
import org.testng.annotations.Test; // TestNG

public class MyTestClass {

@Test
public void aTestMethod() throws ... { ... }
}

9
 TestNG Assertions
import static org.testng.Assert.*;
import org.testng.annotations.Test;

public class MyTest {

@Test
public void myTestMethod() {
// ... Possibly some setup
// ... Call production method(s)
assertTrue(boolExpression);
// ... more assertions
}
}
10
 Expected Exceptions
JUnit 4 (can simplify test code)
@Test(expected = AThrowable.class)

TestNG (can have more than one exception)

@Test(expectedExceptions = AThrowable.class)
Or ...
@Test(expectedExceptions = { T1.class, ... })

11
 TestNG “Groups”
• Each test method is tagged with any # of groups.
– @Test // no groups
– @Test (groups = “group1”)
– @Test (groups = { “g1”, “g2”, ... })
• A group therefore contains any # of test methods.
• Can also annotate test class this way; test method groups are union
of any explicit test method groups, and any of its test class.
• Groups can span classes.
• Groups can also be externally defined (TestNG xml configuration
file).
• A group is identified by a unique string (don’t use white space).
– There are no pre-defined group names.
– E.g., “slow”, “db”, “ui”, “unit”, “integration”, “broken.unknownReason”,
“check-in”, “week-end”, “functional.billing”

12
 TestNG Groups (continued)
• TestNG community suggests hierarchical
names from more general to less. E.g.:
– db.table.CUSTOMER
– state.broken.fix-in-progress
• Design group names so that you can
select them with prefix patterns.
• Groups complement other features (as
we’ll see).

13
 Setup/Teardown (JUnit 3/4)
JUnit 3
• protected void setUp() throws Exception { ... }
• protected void tearDown() throws Exception { ... }

JUnit 4 (also, @After*; annotation package: org.junit)

• @BeforeClass public static void before1() ...
• @Before public void before2() ...
• Multiple per class (order not defined)
• These 4 annotations are “inherited” (order is [kind of] defined).

14
 Setup/Teardown (TestNG)
@BeforeMethod
@BeforeClass (no need to be static)
@BeforeGroups ({“group1”, …})
@BeforeTest
@BeforeSuite

• And @After*

15
 TestNG run-time concepts
• A org.testng.TestNG instance runs tests.
– Invoked by IDE, ant, or on command line.
– Run params specified as setters, or in xml.

• Hierarchy of suite/test/class/method.

• A TestNG run executes 1 or more suites.

– Can be transparent, e.g., run 1 test method via Eclipse.

16
 testng.xml
• Controls test runs
• Optional
• File can have any name (.xml suffix advised)
– TestNG creates “testng-failed.xml”, for easy re-run.
• Root: <suite> (then <test>/<classes>/<method>)
• Suite can also point to a list of additional xml suite files.
• Some of the available specifications are described next.

17
 Disable Tests
• TestNG
– @Test(enabled = false)
– Add to a group which is excluded
– Exclude in other ways in testng.xml

• JUnit 4: @Ignore, @Ignore(“reason”)

– Can also annotate test class.
– Runners report # ignored
18
 Test Timeouts
TestNG
• @Test(timeout = 1000)
• testng.xml <suite|test> time-out attribute

JUnit 4
• @Test(timeout = 1000)

19
 The Costs of Testing
• Time, money, schedule, production scope.
• Doubled (very roughly) code base.
– Maintenance (production and test).
• Sloppily written tests??
• Is test code quality as good as production?
• Slow tests?
• True buy in from business sponsors/owners,
project managers, user reps?

20
 “High View” of Test Benefits
• “High View”: using tests to best advantage
• “Do the thing right”
• “Do the right thing” (acceptance tests)
• Safety net for modifications/refactoring
• Defect localization (1 fail is more info than 100)
• Documentation/training
• Minimize costs of testing
• This view leads to better tests
21
 Integration vs. Unit Tests
• Unit test: tests 1 invocation of 1 method of 1
isolated class with 1 thread
• Integration test: tests more than 1 method
invocation of an isolated class, often much more,
or more than 1 thread
• Distinction: unit test vs. a test written in JUnit
• Examples
– Test hits db
– Life-cycle of an object (e.g., a customer order)
– Testing units by testing integrated classes

22
 Integration vs. Unit (continued)
• Should both be tested?
• Unit
– How big do you want the holes in your
refactoring safety net?
– How many permutations can integration tests
handle?
• Integration
– If the units work independently, will they work
together?

23
If Unit Tests are Really Integration Tests

• Slow tests.
• False sense of security for refactoring safety net.
• Permutations of business rules likely not
covered.
• Poor defect isolation: 100 tests failing is less
information than 1 test failing.
• High maintenance for test/production
modification.
• Counter to the goals of TDD and Agility.
24
 Test Method Dependencies
• JUnit 3: Antithetical concept; each test method
gets its own test class instance; high isolation;
unit oriented by intentional design
• TestNG:
– @Test(dependsOnMethods = {“m1”, “m2”, ...})
– @Test(dependsOnGroups = {“g1”, “g2”, ...})
– E.g.: configuration dependencies; production lifecycle
• JUnit 4: tests ignored if “assumptions” (similar to
assertions) are violated: assumeTrue(bool),
assumeNoException(Throwable), assumeNotNull(refs
…), assumeThat(…) high-readability method

25
 Parallel (multiple threads)
@Test(threadPoolSize = #,
invocationCount = #,
timeout = #)
testng.xml:
– <suite|test
parallel =“tests|methods|none”
thread-count=“#” (5); 1 if not parallel
time-out =“#” ms (default: 0=none)
...>
• Testing production code with multiple threads (unless
not supported); stress testing; speeding up tests!
26
 Test Method Parameters I
@Test
@Parameters({“name1”, “name2”})
public void aTest(String name1, int name2) {
...
}

• Values come from testng.xml (<test> is

checked first, then <suite>).
• Only basic types (String, [Bb]ool, #’s, …).
27
 Test Method Parameters II
@Test(dataProvider = “myDataProvider”)
public void aTestMethod(/* params! */) {
...
}

@DataProvider(name = “myDataProvider”)
public Object[][] getSomeTestData() {
...
}
• Runs (possibly multiple: first []) parameterized tests.
• Multiple parameters: second []
• Return type can also be Iterator<Object[]>
• Data provider itself can take 0, 1, or 2 args: Method, and test context

28
 Reference
• [TestNG] Next Generation JavaTM Testing;
TestNG and Advanced Concepts; Cédric Beust and Hani
Suleiman; Addison-Wesley; ©2008

29