Software

Testing

B. Lund. (2013). Lunch. Available: http://www.lunchstriper.no, http://www.dagbladet.no/tegneserie/lunch/

Hans-Petter Halvorsen, M.Sc.

 Planning
Deployment
Maintenance

Testing
The Software
Development
Requirements
Lifecycle Analysis

Implementation

Design
“If your code works, but you don’t know why
– Then it does not work, you just don’t know
it yet”
 Main purpose of Testing: Find Bugs
• Requirements errors: 13%
• Design errors: 24%
• Code errors: 38%
• Documentation errors: 13%
• Bad-fix errors: 12%
http://proquest.safaribooksonline.com/book/software-engineering-and-development/9781449691998/chapter-3-engineering-of-software/42?uicode=telemark
 Quality vs. Features

Testing must be an unavoidable aspect of development, and the marriage of development

and testing is where quality is achieved.
 Who does the Testing?
• Programmers
– Programmers usually create test cases and run them as they write the
code to convince themselves that the program works. This programmer
activity related to testing is usually considered to be unit testing.
• Testers
– A tester is a technical person whose role for the particular item being tested is
just to write test cases and ensure their execution. Although programming
knowledge is extremely useful for testers, testing is a different activity with
different intellectual requirements. Not all good programmers will be good
testers.

• Users/Customers
– It is a good idea to involve users in testing, in order to detect usability
problems and to expose the software to a broad range of inputs in
real-world scenarios.
 “How Google Tests Software”
• The first piece of advice I give people when they
ask for the keys to our success: Don’t hire too
many testers.
• How does Google get by with such small ranks of
test folks?
• If I had to put it simply, I would say that at Google,
the burden of quality is on the shoulders of those
writing the code.
• Quality is never “some tester’s” problem. Everyone
who writes code at Google is a tester, and quality is
literally the problem of this collective
 Types of Testing User Testing

Stress Testing Stress Testing Deployment

Testing

...
Usability Regression
Testing Testing
Performance
... Testing
...
Different Systems Needs Different Testing

Why?
 Client
Different Platforms:

Client

Client
Different WinForms Android, iOS, Windows
Browsers 8/Windows Phone,
Presentation Tier Mobile
etc.
Desktop App App Presentation Tier
Web App
Presentation Tiers Internet
Firewall
API Web Server Local
Presentation Tier
Web Service Network
ASP.NET Web Forms

API
Business Tier
Logic Tier
Clients
API Data Access Tier Testing is
API e.g., ADO, ADO.NET
Complex!!
Database -> Systematic
Database
Server Approach
Stored Procedures
Views Data Tier
needed!
Tables
 11
Read the whole article here: http://www.idg.no/computerworld/article283659.ece
 Who are going to use the
User software?
Software Testing How are they going to
use it?

Application Desktop, Web, Mobile?

Windows, OS X, Linux,
Operating System Android, iOS, etc.

PC, Mac, Smartphone,

Tablet, SmartTV, etc.
Hardware

Infrastructure, Network,
Internet, Servers, etc.
 Why Test your Software?
• Finding Bugs in the Software before it is released to the
Customer
• Finding unwanted system behaviours
• Verify/Validate that the Software works as expected (according
to the Specifications)
• Find bugs as soon as possible!
• Software systems not working = Money lost for the Customers!

13
 Why Test your Software?
It is commonly believed that the earlier a defect is found, the cheaper it is to fix it:

14
Why Find Bugs early?

Cost per defect/Bug

Software Development Life Cycle (SDLC)

 Testing
Code
Requirements Development & Coding Final
Freeze
& Design Alpha Delivery

RTM
Beta
... ... ...

Finished
RC
Start

Continuous Testing in the whole SDLC!

Testing Testing Testing Testing

Increased Increased Increased Increased

Focus Focus Focus Focus

Requirements & Functionality Software without

Requirements
Functionality Critical Bugs
You can never find all Bugs!
Released Software do have Bugs!
Agile/Scrum: Periodically Iterations/Sprint every 14-30 days
 Software Development
Software Finished

“100%”
Details, small adjustments, etc.
“90%” The last 10% takes a lot of time!!!

Sooner or later you have to say enough is

enough and release version 1.0.

One must define within the development

company, development team or in dialogue
with the customer what is defined as "good
enough". Software will never be 100%
complete or error-free!

Time
 Software Testing

When should you stop Testing?

(depends on Time, Budget, etc.)

In the beginning it it easy to

find bugs with few resources

Critical Point Time

 Customer Perspective

Remember – It is your Customers that are going to use your Software (and pay for
it)!

è The Customer needs to be involved in the Requirements, User

Experience and Testing of the Software
“If your code works, but you don’t know why
– Then it does not work, you just don’t know
it yet”
 Software Testing
“If you don’t know how your
code works, it does not work
– you just don’t know it yet”

“50% of the software

development is about
testing your software”
Motivation
– Is Testing needed?
“It works on my Computer”

Make sure to test your software

on other Computers and
Environments!

• Everything works on the

Developer Computer
• The Customers Database is not
the same as yours
• The Customer may nor use the
same OS
• The Customer may nor use the
same Web Browser
• Etc.

=> Test Environment is needed!

 Developers Developers & Testers Customers

Development Testing Production

Typically the Developers Personal A Clean PC/Server (or a network The Customers
Computer with Database, Web with PCs and Servers) where you environment where you
Server and Programming install and test your Software.
unstall the final
Software Today we typically set-up a Virtual
Test Environment software (Servers and
Clients)
Development
Production
Environment Test Environment Environment

Programming environments such as

Visual Studio, etc. should not be
installed in this environment. You
need to create .exe files etc. in
order to make your software run.
 Virtualization Software
A lot of Virtualization Software exists. Here are some
examples:
• VMware Workstation Player
• VMware Workstation
• VMware vSphere
• VMware Fusion (Mac)
• Parallels Desktop (Mac)
• Microsoft Hyper-V
• VirtualBox
• etc.
Software Testing

What is a bug???
 Bugs
• A software bug is an error, flaw,
failure, defect, or fault in a
computer program or system
that produces an incorrect or
unexpected result, or causes it
to behave in unintended ways
• They found a bug (actually a
moth) inside a computer in
1947 that made the program
not behaving as expected. This
was the “first” real bug.

27
 Debugging
• Debugging is about different techniques for finding and
fixing bugs (errors that make your code not work as
expected) in your code.
• It is difficult to write code without errors (bugs), but e.g.,
Visual Studio and other tools have powerful Debugging
functionality (break-points, etc.)
• The Compiler will also find syntax errors, etc.
• For more “advanced” bugs other methods are required
(Unit Testing, Integration Testing, Regression Testing, Acceptance Testing, etc.) . The main focus

here will be on these methods, while Debugging is

something you learned in Programming courses.

28
The Developers Test Perspective
 Bugs vs. Features
“It's not a bug - it's an undocumented feature”

“For as long as I've been a software developer and used bug tracking systems, we have struggled
with the same fundamental problem in every single project we've worked on: how do you tell bugs
from feature requests?”
 Software Releases

Alpha Release

Beta Releases

RC (Release Candidate)

RTM (Release To Manufacturing)

 Software Releases
Before the software is released
• Alpha Release(s)
• Beta Release(s)
• RC - Release Candidate(s)
• RTM – Release To Manufacturing
Maintenance (after the software is released)
• Patches (small fixes)
• SP - Service Packs
(lots of small fixes and pathes bundle together)
…
Start Planning next release

32
 Example - Windows 8
• Start planning and development of Windows 8, 2008/2009 (the
planning started before Windows 7 was released)
– Internal Builds xxxx…xxxx
– Internal Alpha versions, Alpha 1, 2, 3
– Internal Builds xxxx…xxxx
– Internal Milestone1 Release (build 7850), 2010.09.22
– Internal Milestone2 (build 7955), Milestone3 (build 7989)
• Developer Preview (build 8102), 2011.09.13
– Internal Builds xxxx…xxxx
• Consumer Preview (build 8250), 2012.02.29
– Internal Builds xxxx…xxxx
• Release Preview (build 8400), 2012.05.28
– Internal Builds xxxx…xxxx
major.minor.maintenance.build
• RTM Release (build 9200), 2012.08.01
It is normal to build the software automatically every night, ready for internal tester the day after
 Test Documentation
Software Test Plan (STP)

Test Logs

Document
Planning Tests Perform Tests
Test Results

Software Test
Documentation
(STD)

Software Design Document (SDD) - Functional & Non-Functional Requirements

Software Requirements Specifications (SRS) - User & System Requirements

These documents will be the foundation for all Testing

Functional & Non-Functional Requirements
I. Sommerville, Software Engineering: Pearson, 2010.

Functional Requirements
Statements of services the system should provide, how the
system should react to particular inputs and how the system
should behave in particular situations.
May state what the system should not do.

Non-Functional Requirements
Constraints on the services or functions offered by the system
such as timing constraints, constraints on the development
process, standards, etc.
Often apply to the system as a whole rather than individual
features or services.

-> Functional and Non-Functional Testing

 User & System Requirements
I. Sommerville, Software Engineering: Pearson, 2010.

User Requirements:
Statements in natural language plus diagrams of the
services the system provides and its operational
constraints. Written for customers.

System Requirements:
A structured document setting out detailed
descriptions of the system’s functions, services and
operational constraints. Defines what should be
implemented so may be part of a contract between
client and contractor.
Testing Approaches &
Documentation

Hans-Petter Halvorsen, M.Sc.

 Test Methods
• Unit Testing
• Functional/Non-Functional Testing
• Integration Testing
• Regression Testing
• Whitebox/Blackbox Testing
• Code Review
• Acceptance Testing (FAT/SAT )
 Levels of Testing

Is the responsibility of the customer – in general.

The goal is to gain confidence in the system;
Acceptance Testing especially in its non-functional characteristics

The behavior of the whole product(system) as

System Testing defined by the scope of the project

Interface between components; interactions

Integration Testing with other systems (OS, HW, etc)

Any module, program, object separately

Unit Testing testable
 Levels of Testing
Unit Testing: Test each parts
independently and isolated

Regression Testing: Test

that it still works after a
Integration Testing: Make sure change in the code
that different pieces work
together. Test the Interfaces
between the different pieces.
Interaction with other systems
(Hardware, OS, etc.)

System Testing: Test the whole system

 Levels of Testing
Start Requirements & Design

Unit Testing

Regression Testing

Integration Testing

System Testing

Finish Acceptance Testing

 Levels of Testing
Requirements & Design
Start
Start Development

Unit Tests are written by the Developers as part of the

Unit Testing Programming. Each part is developed and Unit tested
separately (Every Class and Method in the code)

Regression testing is testing the system to check that

Regression Testing changes have not “broken” previously working code.
Both Manually & Automatically (Re-run Unit Tests)

Integration testing means the system is put

Integration Testing together and tested to make sure everything works
together.
System testing is typically Black-box Tests that validate
System/Validation the entire system against its requirements, i.e Checking
Testing
that a software system meets the specifications

The Customer needs to test and approve the software

Finish Acceptance Testing before he can take it into use. FAT/SAT.
 Test Methods
Black-box vs. White-box Testing
Black-box Testing: You need no
knowledge of how the system is created.

White-box Testing: You need to have

knowledge of how (Design and
Implementation) the system is built
 Test Methods
White-box Testing Black-box Testing

Black-box testing is a method of software testing that examines the functionality of an application
(what the software does) without going inside the internal structure (White-box Testing). We also have
something called Gray-box testing which is something in between.

Test Levels

Unit Testing Unit Testing

Integration Integration
Testing Testing

System System
Testing Testing
Acceptance
Testing
 White-box Testing
• You need to have knowledge of how (Design
and Implementation) the system is built
• Also called “Glass-box testing”

46
 Black-box Testing
• Black-box testing is a method of software testing
that examines the functionality of an application
(what the software does) without going inside
the internal structure (White-box Testing).
• You need no knowledge of how the system is
created.
• Black-box testing can be done by a person who
only know what the software is supposed to do
• Compare to driving a Car – you don’t need to
know how it is built in order to test it.
 Unit Testing
• Unit Testing (or component testing) refers to
tests that verify the functionality of a specific
section of code, usually at the function level.
In an object-oriented environment, this is
usually at the class and methods level.
• Unit Tests are written by the developers as
part of the programming
• Automatically executed (e.g., Visual Studio and Team
Foundation Server have built-in functionality for Unit Testing)
 Regression Testing
• Regression testing focuses on finding defects after a
major code change has occurred. Specifically, it seeks to
uncover software regressions, or old bugs that have
come back.
• Regression testing is testing the system to check that
changes have not “broken” previously working code.
• In a manual testing process, regression testing is
expensive but, with automated testing, it is simple and
straightforward. All tests are rerun every time a change
is made to the program.
• Tests must run “successfully” before the change is
committed.
 Integration Testing
• Individual software modules are combined
and tested as a group.
• Each part is developed and Unit tested
separately. The Whole is then assembled (put
together) and tested that everything works
together.
• It occurs after Unit Testing and before
Validation Testing.
 System Testing/Validation Testing
• System Testing follows Integration Testing.
• It consists of Black-box Tests that validate the entire
system against its requirements
• Checking that a software system meets
specifications and that it fulfills its intended purpose
• Often executed by an independent group (QA
group, QA – Quality Assurance)
• Since system tests make sure the requirements are
fulfilled, they must mystematically validate each
requirement in the SRS (Software Requirements
Spesification).
 Acceptance Testing (FAT/SAT)
• Used for tailormaid software, not for generic products
• The Customer needs to test the software before he can take it
into use.
• Customers test a system to decide whether or not it is ready to
be accepted from the system developers and deployed in the
customer environment.

• FAT – Factory Acceptance Testing. Usually performed in the Test

Environment at the software company.
• SAT – Site Acceptance Testing. Performed at the Customer in
the actual Production Environment.

• If the test is accepted, the software is officially handed over to

the customer.
 Test Process

Document Test
Planning Tests Perform Tests
Results
 Test Documentation
Software Test Plan (STP)

Test Logs

Document
Planning Tests Perform Tests
Test Results

Software Test
Documentation
(STD)

Software Design Document (SDD)

Software Requirements Specifications (SRS)
 Testing & Documents

• SRS – Software Requirements Specifications

– A document stating what at application must accomplish

• SDD – Software Design Document

– A document describing the design of a software application

• STP - Software Test Plan

– Documentation stating what parts of an application will be tested, and the
schedule of when the testing is to be performed

• STD - Software Test Documentation

– Introduction, Test Plan, Test Design, Test Cases, Test procedures, Test Log, …,
Summary
 Project Documentation
Project Start High-Level WHAT (SRS)
Requirements and
Design Documents HOW (SDD)
1. Planning
Project Management (Gantt Chart, etc.)

(stakeholders, the Detailed ER Diagram

software team; Requirements and UML Diagrams
architects, UX Design Documents CAD Drawings
designers,
developers) How to Test/ (STP)
Test Plans
What to Test
2. Testing
Time

Proof that you have tested

(QA people) Test Documentation and that the software works
as expected (STD)
System Technical Stuff
Documentation (Super User/ IT dep.)
3. End-user
Installation Guides How to install it
Documentation
(The people that How to use it
shall actually use User Manuals (End User)
the software)

4. Handover Final Report

Project Finish (Business people)
 Software Test Plan (STP)
A Document that answers the following:
• Testing should be based on Requirements & Design Documents
• What shall we test?
• How shall we test?
• Hardware/Software Requirements
• Where shall we test?
• Who shall test?
• How often shall we test (Test Schedule)?
• How shall tests be documented?
§ It is not enough simply to run tests; the results of the tests must be
systematically recorded. It must be possible to audit the testing process to
check that it has been carried out correctly
§ System tests: This section, which may be completely separate from
the test plan, defines the test cases that should be applied to the
system. These tests are derived from the system requirements
specification.
http://www.softwareengineering-9.com/Web/Testing/Planning.html
E. J. Braude and M. E.Bernstein, Software Engineering: Modern Approaches, 2 ed.: Wiley, 2011. (Ch. 25) 58
 Test Planning
• Test planning involves scheduling and estimating the system testing
process, establishing process standards and describing the tests that
should be carried out.
• As well as helping managers allocate resources and estimate testing
schedules, test plans are intended for software engineers involved in
designing and carrying out system tests.
• They help technical staff get an overall picture of the system tests and
place their own work in this context.
• As well as setting out the testing schedule and procedures, the test plan
defines the hardware and software resources that are required.
• Test plans are not a static documents but evolve during the
development process. Test plans change because of delays at other
stages in the development process.
• Test planning is particularly important in large software system
development.
• For small and medium-sized systems, a less formal test plan may be
used, but there is still a need for a formal document to support the
planning of the testing process.
http://www.softwareengineering-9.com/Web/Testing/Planning.html
 Testing in TFS with Agile/Scrum
TFS – Team Foundation Server/Visual Studio Online

Software Test Plan (STP)

“Bugs” Work Items

Test Logs
“Test Case” Work Items Manual Testing by
Developers and
Tester Document
Planning Tests Perform Tests
Test Results

Execution of Unit Tests Software Test

Documentation
“Product/Sprint Backlog” Work Items (STD)

Software Design Document (SDD) - Functional & Non-Functional Requirements

Software Requirements Specifications (SRS) - User & System Requirements
 Test Planning with VSTS

http://msdn.microsoft.com/en-us/library/vstudio/dd380763(v=vs.120).aspx 61
Test Planning with VSTS

62
Software Testing
Part II

Hans-Petter Halvorsen, M.Sc.

 HA - High Availibility
O. Widder. (2013). geek&poke. Available: http://geek-and-poke.com
 Developer vs. Tester
• In small software companies the Developer
and Tester is the same person.
• In bigger software companies you have own
software testers – but still the developer have
an important part role in testing the software.
• The Developer should always test their code
before they check the code into the source
code control system
• Unit Tests are written by the Developer
 Team Teams and Roles
Customer/Stakeholders
• Customer
• Project Manager
• Software Architect
Collaboration!
• UX Designer
• Developer Software Architect Software Tester

• Tester
Project Manager
• etc. UX Designer

Programmer

Everybody (on different levels) needs to be involved in Testing – not just the Tester!
Software Development Process - Testing

Document Test
Planning Tests Perform Tests
Results
Software Development Process - Testing
Always test your application!
• Test it yourself (take responsibility for your own code)
• Test it on other computers and environments
• Make sure others test your application
• Eat Your Own Dogfood
 Development Environment vs. Test
Environment vs. Production Environment
Development Environment:
This is where the Developers create their
code, typically their own PC, or a
Development Server

Test Environment:
The Software is perodically deployed to a
test environment so other resources besides
the developers can test the software

Production Environment:
When the software is finished, it is deployed
to the customer. The software is set into
production and used by the end-user.
Use Realistic Test Data!
 Code Coverage
Code coverage is a measure used in software
testing. It describes the degree to which the
source code of a program has been tested.
int foo (int x, int y)
{
int z = 0;
if ((x>0) && (y>0))
{
z = x;
}
return z;
}

è Depending on the input arguments, different parts of the code will be

executed. Unit Tests should be written to cover all parts of the code.
 Logfiles
Logfiles are a method used to
store errors happening during
execution
 24-7
• Most of the software today
needs to be able to run “24-7”
• This demand makes the
testing very important

73
 “Eat your own Dogfood”
“Eating your own dog food”, also called “dogfooding”, is a slang term used to
reference a scenario in which a company (usually, a computer software
company) uses its own product to demonstrate the quality and capabilities of
the product.
Example: Microsoft uses Windows PCs and Visual Studio to create their
software.
Code/Feature Freeze

The Developer cannot add new

features to the software, only fix
bugs

When it is very close to release,

thay cannot fix bugs either
Code Review
Why Do Reviews?

Cost per defects

SDLC
 Code Review
• We all are human beings. You may do some mistakes irrespective of your
experience in a particular technology or module. If you just review your code
by a second eye, those mistakes might caught at that time only. This way you
can reduce the no. of bugs reported by the testers or end users.
• If you are working in a geographically distributed team, your coding
conventions may differ and if you have some strict coding guidelines, this
code review process will make it possible to recheck the standards in the code
that you have written.
• There are some possibilities of repetitive code block which can be caught
during a code review process. Refactoring can be done based on that.
• Unused code blocks, performance metrics etc. are some additional check
points of doing a review.
• If you are new to development, this code review process will help you to find
out your mistakes and help you to improve them. This is a perfect knowledge
sharing mechanism.
• Find out the defects and correct them at the beginning before it commits to
the source control system.
Better code always starts with review process!
Code Review

79
 Test-Driven Development (TDD)
• Coding and Testing are done in parallel
• The Tests are normally written before the
Code
• Introduced as part of eXreme Programming
(XP)
O. Widder. (2013). geek&poke. Available: http://geek-and-poke.com
Test Driven Development - TDD
 Development-Driven Testing (DDT)
• DDT is all about giving more responsibility to developers specifically, and
the development process in general. It works especially well when using
test cases as requirements, and having the developers write these test
cases. But it’s really not DDT unless those tests are written near the end
of the process, when the code is checked in, and the developers figure
they’re done.
• The advantages of Development-Driven Testing are many. Instead of tests
driving the development, it’s developers driving the tests, so you get just a
few tests, and they almost always all pass. The project team can deliver on
time for a change, with zero bugs found in every iteration. This makes
management happy, and isn’t that really the ultimate barometer of
success? Also, velocity is increased dramatically when using this process.
• Development-driven testing makes all the sense in the world for those
who practice agile
TDD vs. DDT
 Summary
• Testing is an important
part of software
development
• Make sure your code
works everywhere!
 Testing - Tips & Tricks
• Looking for failures requires:
– Curiosity
– Professional pessimism
– Attention to details
– Good communication skills
– Experience on error guessing
• Communicate failures in a constructive way: fact-
focused; give factual reports and review findings
• Be clear and objective
• Confirm that:
– You have understood the requirements
– The person that has to fix the bug has understood the
problem
 Support

Less focus on Testing - more Support calls from Customers!

87
 Stress Testing

Looking for Performance, Robustness & Stability issues. It involves testing beyond normal
operational capacity, often to a breaking point, in order to observe the results. Example:
How many users can be connected to a Database or a Web Site before it crashes.

http://en.wikipedia.org/wiki/Stress_testing
 References
• I. Sommerville, Software Engineering: Pearson, 2010.
• E. J. Braude and M. E.Bernstein, Software Engineering: Modern Approaches, 2 ed.:
Wiley, 2011.
• Wikipedia. (2013). Software Development Process. Available:
http://en.wikipedia.org/wiki/Software_process
• NTNU. (2013). TDT4140 Systemutvikling. Available:
http://www.ntnu.no/studier/emner/TDT4140
• UiO. (2013). INF1050 - Systemutvikling. Available:
http://www.uio.no/studier/emner/matnat/ifi/INF1050/
• O. Widder. (2013). geek&poke. Available: http://geek-and-poke.com
• B. Lund. (2013). Lunch. Available: http://www.lunchstriper.no,
http://www.dagbladet.no/tegneserie/lunch/
• S. Adams. Dilbert. Available: http://dilbert.com
• Course at UiO: Systemutvikling:
http://www.uio.no/studier/emner/matnat/ifi/INF1050/
• Course at UiO: Foundations of Software
Testing:http://www.uio.no/studier/emner/matnat/ifi/INF5530
• Software Testing Tutorial:
http://www.tutorialspoint.com/software_testing/software_testing.pdf
Hans-Petter Halvorsen, M.Sc.

University College of Southeast Norway

www.usn.no

E-mail: hans.p.halvorsen@hit.no
Blog: http://home.hit.no/~hansha/