Software Testing

Introduction

Beat Fluri

software evolution & architecture lab

 Organization
Course Info
VU Software Testing
AP (ECTS): 3
Language: English
For MSc & PPO’01 after 5th semester
http://seal.ifi.uzh.ch/software_testing

Exercises
Teams of two

Grade
Wirten or oral exam depending on the number of course participants

Dr. Beat Fluri © 2011 2

 Course
No course: 17.03., 24.03., 28.04., 02.06.

Preparation for course

Reading
Solving the exercises

During a course
Discussion of topics (summary of reading)
Presentation of a possible solution for the exercises
Information for upcoming exercise

Information on the website

Dr. Beat Fluri © 2011 3

 Resources

John F. Smart’s blog

http://easytesting.org

http://java-source.net/open-source/testing-tools
Dr. Beat Fluri © 2011 4
 Content (tentative)
Test-Driven Development (TDD)
JUnit and Eclipse

Unit Testing
Testing smallest units (in isolation if possible)

Continuous Integration
One environment to build and test the application
Multi-stage builds for rapid feedback

Integration Testing
Testing interaction between components (compounds of units)

Dr. Beat Fluri © 2011 5

 Content (tentative)
Web Testing with Selenium
End-to-end (acceptance) testing of web applications

Software Quality Assurance with Continuous Integration

Tools for Continuous Integration environments

System Testing (guest speaker)

Regression Testing
How to choose system tests for regression

Load and Performance Testing

How does your system behave under heavy load?

Dr. Beat Fluri © 2011 6

 What is software testing?

Design and implementation of a special

kind of software system; one that
exercises another software system with
the intent to finding bugs.
Rober V. Binder, TOOSMPT

Dr. Beat Fluri © 2011 7

 Test design
Identify, model, and analyze the responsibilities of the system
under test
On different level of abstraction: methods, classes, packages, subsystems, system

Design test cases based on this external perspective

Functional testing

Add test cases based on code analysis, suspicions, and heuristics

Structural testing

Develop expected results for each test case or choose an approach

to evaluate the pass or no pass status of each test case

Dr. Beat Fluri © 2011 8

 Test execution
Implementation under test is minimally operational
Bottom-up to reduce test driver and test stub overhead

Test suite execution: result is evaluated as pass or no pass.

Coverage tool to instrument the implementation under test.

Define goal: 95% statements, 80% branches, 70% conditions, etc.:

If necessary, develop additional tests to exercise uncovered code.

Stop: coverage goal is met and all tests pass.

Dr. Beat Fluri © 2011 9

 Why is software testing so hard?
Input/state space
public Line(int x0, int y0, int x1, int y1)
Input per int: 232 different values: 232 x 232 x 232 x 232 = 2128
1000 lines per second: 1028 years

Execution sequences

for (int i = 0; i < n; i++) {

if (a.get(i) == b.get(i)) {
x[i] += 100;
2n + 1 paths with n iterations
} else {
x[i] /= 2;
}
}

Dr. Beat Fluri © 2011 10

 Why is software testing so hard?
Coincidental correctness

public class Account {

protected long lastTxDate;
public long daysSinceLastTx() {
long today = Calendar.getInstance().getTimeInMillis();
return 1 + (today - lastTxDate) / (24 * 3600 * 1000);
}
}

public class DepositAccount extends Account {

public long daysSinceLastTx() {
long today = Calendar.getInstance().getTimeInMillis();
return (today - lastTxDate) / (24 * 3600 * 1000);
}
}
Dr. Beat Fluri © 2011 11
 Why is software testing so hard?
Undecidability problem (halting problem)
There isn’t any program that can decide whether the execution of another program
will halt or not
For simple program it is decidable but not in general

Dr. Beat Fluri © 2011 12

 Why software testing?
Find bugs early
Airbag does not work: find out before or after releasing the car?

Part of quality assurance

Would you trust the airbag software if it wasn’t tested?

Supports changing a software system

Verify component integrity after a change

Dr. Beat Fluri © 2011 13

 Why is testing neglected?
Expensive (software testing is hard)
Test case design and selection
Building test environment
Integration environment
Few methodologies for test automation

Not part of development process

Hard to include for specific projects (customer won’t pay extra effort)
Not part of the software engineering culture
Missing testing environment

Dr. Beat Fluri © 2011 14

 Short glossary
Test case
Set of inputs, execution conditions, and expected result (pass and no pass criterion)

Test suite
Set of test cases, usually related to a testing goal

Test driver
Class or utility program that applies test cases

Stub
Partial, temporary implementation of a component

Test harness
Substitution for parts of the deployment environment, i.e., a system of test drivers
and other tools to support test execution
Dr. Beat Fluri © 2011 15
 Short glossary
Oracle
A mechanism to evaluate the actual result of a test case as pass or no pass

Equivalence class
Set of input values: if one value is processed correctly/incorrectly, all other values will
also be processed correctly/incorrectly

Coverage
Percentage of elements required by a test strategy that have been exercised by a
given test suite

Dr. Beat Fluri © 2011 16

 Short glossary
Fault model
Identifies relationships and components that are most likely to have faults

Test model
Testable representation of the relationships among elements of an implementation
based on a fault model

Test strategy
Algorithm or heuristic to create test cases from an implementation or test model

Test design
Produces a test suite using a test strategy: identification of interesting test inputs,
placing these test inputs into a sequence, and defining the expected results

Dr. Beat Fluri © 2011 17

 Short glossary
Fault-directed testing
Testing that seeks to reveal implementation faults

Conformance-directed testing
Testing that seeks to establish conformance to requirements or specifications

Dr. Beat Fluri © 2011 18

 V-Model (historically)
Specification Implementation
Acceptance Test
User
Delivery
needs

System Test

System Spec System

Integration Test

Subsystem
Subsystem
Design/Spec

Unit Test

Unit
Unit
Design/Spec

Dr. Beat Fluri © 2011 19

 V-Model (historically)
Specification Implementation

User
needs Validation Delivery

System Spec System

Subsystem
Design/Spec Verification Subsystem

Unit
Unit
Design/Spec

Dr. Beat Fluri © 2011 20

 Test-Driven Development (TDD)
Software development process with very short cycles

Write a test Test

case succeeds

Check if the Test Write app Test(s)

test case fails fails code fail

Test(s)
Run all tests Refactor
succeed

Repeat

Dr. Beat Fluri © 2011 21

 Test-Driven Development (TDD)
Three rules of TDD of Robert Martin1
1. You may not write production code until you have written a failing unit test.
2. You may not write more of a unit test than is sufficient to fail, ant not compiling is
failing.
3. You may not write more production code than is sufficient to pass the currently
failing test.

1Robert Martin. Clean Code, Prentice Hall, 2009.

Dr. Beat Fluri © 2011 22

 Exercise 0

Line segment Triangle (three line segments)

(x1,y1)

(x0,y0)
scalene isosceles equilateral

Valid triangle (s-rule)

No side may have a length of zero and each side must be
shorter than the sum of all sides divided by 2:
s > a, s > b, s > c with s = (a + b + c) / 2
Dr. Beat Fluri © 2011 23
 Exercise 0
Install Eclipse 3.6.x

Download assignment_0.jar
LineSegment and Triangle classes

Download assignment_0-javadoc.jar
Javadoc for LineSegment and Triangle classes

Write test cases for LineSegment and Triangle

Dr. Beat Fluri © 2011 24