Continental Learning Lab: August 2014

Continental Learning Lab
August 2014
Interior Infotaiment and Connectivity R&D Romania

Agenda
1Day 1 - The need of testing, Principles of testing, SDLC

2Day 2 - Test Levels and Functional Testing
3 Day 3 - Non-functional testing and Defect LifeCycle
4Day 4 - Test Types

5Day 5 - Test techniques
6Day 6 - Discussion over homeworks

2 11-03-26 Autor, © Continental Automotive Romania SRL
Table of contents
1The need of testing

2Testing and Quality
3Principles of testing
4Software development lifecycle – SDLC
5Testing lifecycle

The need for testing
Why is necessary
Why do we test?
18 August 2014 , © Continental Automotive Romania SRL

Software development involves ambiguity, assumptions and flawed human

communication.
Each change made to a piece of software, each new piece of functionality, each attempt
to fix a defect, introduces the possibility of a new defect.
With each defect, the risk that the software will not fulfill it`s intended purpose increases.
Testing reduces that risk.

Airplane Forgot to Airplane Pilot error

Pilot on deploy the crashed on
landing landing gear landing
Airplane still
Pilot verifies the led crashes on landing
Airplane Pilot on
landing
Deploy the landing
gear
for landing gear (landing gear was System failure
deployment not totally
deployed)
Federal crash investigators suspect that a partially closed landing-gear door may have prevented part of the main wheels of a
United Airlines jet from extending last weekend at New Jersey's Newark Liberty International Airport, resulting in a harrowing
but safe landing for 48 passengers and five crew members.
6/
Autor
/Interior Infotaiment and Connectivity R&D Romania
Datu
m ©
Conti
7/
Autor
Datu
m ©
Conti
To discover defects
To ensure that product works as expected - verification what the system

should do
To ensure that product works how is expected - validation
To avoid detecting problems by the final user (on later stages of the product)
To ensure the reliability of the software
To provide confidence of reliability of the product, correctness and detection

of particular faults
To stay in business

Table of contents

5Testing lifecycle

Testing and Quality
Why testing and quality?
Quality checks that the

testing process is respected
Testing verifies if the

products is according to the
client requirements

Testing and Quality
With the help of testing:
It is possible to measure the quality of software in terms of defects found
Testing can give confidence in the quality of the software if it has few or no
defects
If the software has defects, the quality of the software system will increase
when the defects are fixed
Lessons should be learned from previous projects
Testing should be integrated as one of the quality assurance activities

Table of contents

5Testing lifecycle

Principles of testing
What are the principles of testing?
A number of testing principles have been suggested over the past 40 years and
offer general guidelines common for all testing.
Software testing is an extremely creative and intellectually challenging task.
When testing follows the defined principles, the creative element of test design
and execution rivals any of the preceding software development steps.

Principle 1
Testing show presence of defects

Testing can show that defects are present, but cannot prove that there are no
defects. Testing reduces the probability of undiscovered defects remaining in the
software but, even if no defects are found, it is not a proof of correctness
Samsung Galaxy S4 with magnetic flip cover

Samsung says that they have tested it, but on the last Software update I got this
defect on closing the cover:

Windows 98 launch conference

Blue screen

Principle 2
Exhaustive testing is impossible

Testing everything (all combinations of inputs and preconditions) is not feasible
except for trivial cases. Instead of exhaustive testing, risk analysis and priorities
should be used to focus testing efforts
For making picture more clear, let us take a simple example of Microsoft
Windows Calculator

Principles of testing – Microsoft Windows Calculator
Principle 2 - Exhaustive testing is impossible

Start with addition operation: You enter 1+0 = and get output 1.
Next you enter 1+1= and get output 2.
How far will you check?
Calculator ( Microsoft windows 7) accepts 16 digits. You have to check for all possibilities up
to 1+ 9999999999999999 =
After that, you can move to 2+0 =, 2+1= and so on.
Like this, you will get to 9999999999999999+9999999999999999 =
Next, you should try for decimal values: 1.0+0.1= , 1.0+0.2 = and so on.
It is not enough yet, we still have to check by illegal inputs to ensure that, it will be handled
properly.
Like this, there are so many inputs that we can’t achieve.
Even you achieve above inputs, still multiplication, subscription and division etc. are left.
The aim of this example is only to give idea that it is impossible to test completely a
software, even software as simple as a calculator.

Principle 3
Early testing
To find defects early, testing activities shall be started as early as possible in the
software or system development life cycle, and shall be focused on defined
objectives.
A study conducted by NIST in 2002 reports that software bugs cost the U.S. economy
$59.5 billion annually
It is commonly believed that the earlier a defect is found, the cheaper it is to fix it.
Time detected
Cost to fix a defect
Requirements Architecture Construction System test Post-release
Requirements 1× 3× 5–10× 10× 10–100×

Time
introduced Architecture – 1× 10× 15× 25–100×
Construction – – 1× 10× 10–25×

Principle 4
Defect clustering
Testing effort shall be focused proportionally to the expected and later observed
defect density of modules. A small number of modules usually contains most of
the defects discovered during pre-release testing, or is responsible for most of
the operational failures.
This is particularly true for large systems where the complexity, size, change and
developer mistakes can impact the quality of the system and affect particular
modules.
This can give a good indication that when a defect is found in one area of the
application, chances are there are more defects in that particular area, so it is
worth investing more time to test that particular area of the application to find as
many defects as possible

Principle 4 – Defect clustering

Smartphone application – analyze test results
Gallery
Contacts
Phone
Column2
MediaPlayer
Wifi
0 5 10 15 20 25

Principle 5
Pesticide paradox
If the same tests are repeated over and over again, eventually the same set of
test cases will no longer find any new defects. To overcome this "pesticide
paradox", test cases need to be regularly reviewed and revised, and new and
different tests need to be written to exercise different parts of the software or
system to find potentially more defects.

Principle 5 - Pesticide paradox
This is similar to the Paradox of the pesticides in the case of pests.
Pests will get immune to the pesticides over a period of time.
The Paradox of the Pesticides implies the need for specialized pesticides that are
customized to the target pest.
Insects that survive the use of pesticide are those that are more immune to the
poison than others.
If you keep on applying the same pesticide/insecticide, the insects build up resistance
and the pesticide no longer works.
Solution: Tests are regularly revisited to ensure that all functionalities of test are
covered under the requirement.
Test cases should be improved to optimize long term results.

Principle 6
Testing is context dependent

Testing is done differently in different contexts. For example, safety-critical
software is tested differently from an e-commerce site.
Consider two projects:

Developing the control software for an airplane
Developing a Word Processor (Microsoft Word)

Principle 6 - Testing is context dependent

Control Software for an airplane
Federal Aviation Administration regulations must be followed
Anything you do or don`t do – would be evidence in a lawsuit 20 years from now
This means caution, precision and double-checking everyone`s work
Word processor
No government mandatory regulations
Time to market matters
Defects found after the release to the market, are not critical.
Conclusion
Testing practices appropriate to the first project will fail in the second
Practices appropriate to the second project would be criminally negligent in the

first

Principle 7
Absence-of-errors fallacy
Finding and fixing defects does not help if the system built is unusable and does
not fulfill the user needs and expectations.

Table of contents

5Testing lifecycle

Software development lifecycle – SDLC
V-Cycle model
Requirements Acceptance tests
Design System tests
Architecture Integration tests
Implementation Unit Tests

Example for an online store

Requirements phase
“The store needs to sustain a maximum of 10.000 products”
“The selection of the products will be held in a shopping cart”
“The system will have to remember the user and do the login
automatically”
“Each category will need to support 3 levels of subcategories”
“The application will need to run on Fedora Linux 7.2”


Design phase – This is where the analyze of the requirements starts
Can we sustain using our own framework 10.000 products? No
If not, how many can we sustain? We can sustain only 7.500
Our application will be compatible with Fedora 7.2? Yes
Do we support a login/register system? Yes


Architecture phase – high-level design
Consists of the list of modules, brief functionality of each module, their interface
relationships, dependencies, database tables, architecture diagrams, technology
details etc.


Implementation phase – the coding takes place
Global variables
Connect to the DB
Data validation

Unit tests – test modules or individual units of source code, in our case we can test:
Login
Register
Shopping cart
Search
Category

Integration tests – the interaction between different modules
Verifying correct reading/writing of products from/to the DB
Verifying the whole data flow of receiving an order, creating an invoice, saving it to
the DB and mailing it to the client.
Verifying correct integration of the Application with the mail server
Change the price for a product in DB and check if the GUI is updated on refresh

System tests
Add a product from a Category to the Shopping cart and place a order
After registration, the user is automatically logged in
Search for a product and add it to the Shopping cart
Use the contact form to submit a request

Acceptance tests
Usability – inspection and demonstration to the customer

Is the GUI configured to their liking?
The Customer Branding is in all the right places?
Do we have all the fields/screens they asked for?

Software development lifecycle – Static testing
Static testing
From the black box testing point of view, this involves reviewing requirements
and specifications
Test Plan
A test plan is a systematic approach to testing a system i.e. software.
The plan typically contains detailed understanding of what the eventual testing
workflow will be

Software development lifecycle – Static testing
Test case against requirements

This is a specific procedure of testing a particular requirement

Software development lifecycle – Dynamic testing
Dynamic testing
Dynamic testing adopts the opposite approach and is executed while a
program is in operation
Dynamic testing involves testing the software for the input values and
output values and then are analyzed
Dynamic testing, on the other hand, is capable of exposing a subtle flaw or

vulnerability
A dynamic test, however, will only find defects in the part of the code that is
actually executed

When to start testing in SDLC -

Requirements
Design
Architecture
Implementation
Unit Tests
Integration Tests
System Tests
User Acceptance Tests
Testing starts from Requirement Phase

Table of contents

5Testing lifecycle

Testing lifecycle

Testing lifecycle – A sample testing cycle
Requirements analysis
Testing should begin in the requirements phase of the software development life
cycle.
During the design phase, testers work with developers in determining what
aspects of a design are testable and with what parameters those tests work
Test planning
Test strategy and test plan creation.
Since many activities will be carred out during testing, a plan is needed.

Test development
Test procedures, test scenarios, test cases, test scripts to use in testing software
Test execution
Testers execute the software based on the plans and test documents then report
any errors found to the development team

Test reporting
Once testing is completed, testers make the final reports on their test effort and
whether or no the software tested is ready for release
Test result analysis

Is done by the development team, usually along with the client, in order to decide
what defects should be assigned, fixed, rejected or deferred to be dealt with later

Defect retesting
Once a defect has been dealt with by the development team, it is retested by the
testing team
Regression testing
It is common to have a small test program built of a subset of tests, each
integration of new, modified or fixed software, in order to ensure that the latest
delivery has not ruined anything and that the software product as a whole is still
working correctly

Test closure
Once the test meets the exit criteria, the activities such as capturing the key
outputs, lessons learned, results, logs, documents related to the project are
archived and used as a reference for future projects

Software development lifecycle – Requirements
Quiz
This requirement is not well written. Why?
The alarm shall remain active for 5 seconds.

Quiz
The alarm shall remain active for 5 seconds.
It is an incomplete statement. Is it audio or visual alarm or both?

Quiz
When primary is lost, secondary unit takes over.

Quiz
When primary is lost, secondary unit takes over.
Is this a “must” or is this “optional”? Use words like “must” (or “shall”)
consistently

Quiz
The system shall purge state control records and files that are older than the
retention period

Quiz
The system shall purge state control records and files that are older than the
retention period
The requirement is incomplete and vague without specifying the retention

period or providing reference as to where the information can be obtained. The
requirement cannot be implemented or tested as stated.

Quiz
When performing calculations the software shall produce correct results.

Quiz
When performing calculations the software shall produce correct results.
This requirement is not necessary. This type of requirement is inherent to

correctness should not need to be specified, unless it refers to a specific
(complex) calculation and what is considered as correct..

Quiz
The output of the program shall usually be given within 10 seconds

Quiz
The output of the program shall usually be given within 10 seconds
The requirement contains a general word “usually”. Is there any condition in

which this requirement should not be the case? The requirement is also
incomplete: the output of the program is not specified. Also, when do you start
counting the 10 seconds? the Requirement cannot be implemented or tested
as stated.

Quiz
Which requirement has the best form?
The system could be used to manage one or more libraries

The system shall manage the lending of books
The system is initially either off or on
The system could have several users
The system goal should be to let users to borrow books

Quiz
Which requirement has the best form?
The system could be used to manage one or more libraries

The system shall manage the lending of books
The system is initially either off or on
The system could have several users
The system goal should be to let users to borrow books

Quiz
Is this specification written well?
When password and user name are entered, they are validated

Quiz
When password and user name are entered, they are validated
Who checks for correctness—the user or application?
Who enters the password and user name?
The system validates the user name and password that the user provided

Quiz
Notification is sent

Quiz
Notification is sent
Who sends the notification—user, system?

The system sends notification to the client.

Quiz
The system usually should respond fast

Quiz
The system usually should respond fast
What is the definition of fast—2 seconds or 10 seconds?

What does usually mean—50% of the time, or 90% of the time?
The system must respond to user requests in two seconds or under for
90% of requests.

Quiz
Because the user almost always requests a hard copy report, the
system should provide this capability on demand

Quiz
Because the user almost always requests a hard copy report, the
system should provide this capability on demand
What does almost always mean?
The system must provide a hard copy report on demand.

Homework – part 1
1.What do we need in order to start testing?
2. When testing starts?
3. When testing stops?
4. What is the role of a tester?
5. Describe one of the most relevant principle of testing that you will apply for testing
your personal mobile phone. Argument your selected principle.
6. Define by your own at minimum 5 specifications (requirements) for your personal TV
7.Define by your own at minimum 5 specifications for a USB device?
67 /
Autor
Datu
m ©
Conti

Continental Learning Lab: August 2014

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Continental Learning Lab: August 2014

Uploaded by

Copyright:

Available Formats

Continental Learning Lab

Interior Infotaiment and Connectivity R&D Romania

1Day 1 - The need of testing, Principles of testing, SDLC

4Day 4 - Test Types

Interior Infotaiment and Connectivity R&D Romania

1The need of testing

Interior Infotaiment and Connectivity R&D Romania

Interior Infotaiment and Connectivity R&D Romania

18 August 2014 , © Continental Automotive Romania SRL

Software development involves ambiguity, assumptions and flawed human

Testing reduces that risk.

Interior Infotaiment and Connectivity R&D Romania

18 August 2014 , © Continental Automotive Romania SRL

Airplane Forgot to Airplane Pilot error

To ensure that product works as expected - verification what the system

To ensure that product works how is expected - validation

To ensure the reliability of the software

To provide confidence of reliability of the product, correctness and detection

Interior Infotaiment and Connectivity R&D Romania

18 August 2014 , © Continental Automotive Romania SRL

1The need of testing

Interior Infotaiment and Connectivity R&D Romania

Why testing and quality?

Quality checks that the

Testing verifies if the

Interior Infotaiment and Connectivity R&D Romania

18 August 2014 , © Continental Automotive Romania SRL

With the help of testing:

It is possible to measure the quality of software in terms of defects found

Lessons should be learned from previous projects

Testing should be integrated as one of the quality assurance activities

Interior Infotaiment and Connectivity R&D Romania

18 August 2014 , © Continental Automotive Romania SRL

1The need of testing

Interior Infotaiment and Connectivity R&D Romania

What are the principles of testing?

Software testing is an extremely creative and intellectually challenging task.

Interior Infotaiment and Connectivity R&D Romania

18 August 2014 , © Continental Automotive Romania SRL

Testing show presence of defects

Samsung Galaxy S4 with magnetic flip cover

Interior Infotaiment and Connectivity R&D Romania

18 August 2014 , © Continental Automotive Romania SRL

Windows 98 launch conference

Interior Infotaiment and Connectivity R&D Romania

18 August 2014 , © Continental Automotive Romania SRL

Exhaustive testing is impossible

Interior Infotaiment and Connectivity R&D Romania

18 August 2014 , © Continental Automotive Romania SRL

Principle 2 - Exhaustive testing is impossible

Next you enter 1+1= and get output 2.

How far will you check?

After that, you can move to 2+0 =, 2+1= and so on.

Like this, you will get to 9999999999999999+9999999999999999 =

Like this, there are so many inputs that we can’t achieve.

Interior Infotaiment and Connectivity R&D Romania

18 August 2014 , © Continental Automotive Romania SRL

Requirements 1× 3× 5–10× 10× 10–100×

Interior Infotaiment and Connectivity R&D Romania

18 August 2014 , © Continental Automotive Romania SRL

Interior Infotaiment and Connectivity R&D Romania

18 August 2014 , © Continental Automotive Romania SRL

Principle 4 – Defect clustering