UNIT 5: QUALITY AND

TESTING
 Basics of Software Testing

Objectives:
• Understand the concept of Software Testing
• Understand the importance of Quality Software

1.1 Software Quality, Definition of Software Testing, Role

of
Testing
1.2 Failure, Error, Fault, Defect, Bug Terminology
1.3 Objectives of Testing
1.4 Test Case
1.5 When to Start and Stop Testing of Software
(Entry and Exit Criteria)
1.6 Skills for Software Tester
1.7 Quality Assurance, Quality Control, Verification and
Validation, V Model
 Software Quality, Definition of Software Testing
Role of Testing
1 Software Quality,
Definition(s) of Quality:
 Predictable degree of uniformity, dependability at low cost and suited to
market.
 Degree to which a set of inherent characteristics of the product / service fulfills
the requirement.
 Ability to a product or service that bears upon its ability to satisfy implied or
expressed need
Quality is:
 Fitness for use, i.e. Usability of software.
 Conformance to specification
 Judgment of the customer /user about the attributes of a product.
 Meet customer as per national/ international standards
 Fulfill customers need as max as possible.
Software quality is: the degree to which a system, component, or process
meets specified requirements.
 Fundamental principles of testing are:
1. Find defects before customers find them out.
2. Exhaustive testing not possible; Program testing can only
show the presence of defects, never their absence.
3. Testing applied all through the software life cycle
4. Understand the reason behind the test.
5. Test the tests first.
6. Tests develop immunity and have to be revised constantly.
7. Defects occur in convoy or clusters, and testing, should focus
on those clusters.
8. Testing encompasses defect prevention.
9. Testing is a fine balance of defect prevention and defect
detection.
10.Intelligent and well-planned automation - Benefits of
testing.
11.Requires talented, committed people who work in teams.
 Goals of Software Testing
Immediate Goals: The major objectives of Software testing
 Bug Discovery are as follows:
 Bug Prevention  Finding defects which may get created by
Software
Long-term Goals: the programmer while developing the
Testing:
 Reliability software.
 Testing
 Gaining confidence in and providing
produces  Quality
information about the level of quality.
reliability  Customer
satisfaction  To prevent defects.
and quality
 Risk  To make sure that the end result meets the
 Quality
Management business and user requirements.
leads to
 To ensure that it satisfies the BRS that is
customer Post-Implementation
Business Requirement Specification and
satisfactio Goals:
 Reduced SRS that is System Requirement
n
maintenance Specifications.
cost  To gain the confidence of the customers
 Improved testing by providing them a quality product.
process
 Roles And Responsibilities of A Test Leader
• Involved in the planning, monitoring, and control
• Devise the test objectives, organizational test policies, test strategies

and test plans.

• Estimate the testing, and management to acquire the necessary

resources.
• Recognize when test automation is appropriate, select the tools, and

ensure training of the team.

• Testers lead, guide and monitor the analysis, design, implementation

and execution of the test cases, test procedures and test suites.
• Ensure proper configuration management of the test-ware produces

traceability of the tests.

• Schedule the tests for execution and monitor, measure, control and

report on the test progress, the product quality status and the test
results.
• Write summary reports on test status.

• Test leaders also known as Test Manager or Test Coordinator

 Bug Terminology
Bug: defined in simple term as any error or mistake that leads to the
failure of the product or software either due to the specification
problem or due to communication problem, regarding what is
developed and what had to be developed.

Bug Terminology: Failure, Error, Fault

The various terms related to software failure with respect to the area
of application are listed as Defect, Variance, Fault, Failure,
Problem, Inconsistency, Error, Feature, Incident, Bug, and
Anomaly.
i. Problem, error, and bug are probably the most generic terms used.
ii. Anomaly, incident, and variance don’t sound quite so negative
and infer more unintended operation than an all-out failure.
 Causes Of Software Defects
Some of them are:
1. Faulty requirements definition
2. Client-developer communication failures
3. Deliberate deviations from software requirements
4. Logical design errors
5. Coding errors
6. Non-compliance with documentation and coding
instructions
7. Shortcomings of the testing process
8. User interface and procedure errors
9. Documentation errors
 Test Scenario and its design aspect
 Test cases are written using test scenario. It describes each
transaction and expected result of each transaction included in test
scenario.
 Test cases are executed through test data.
 Actors (Person, system, hardware, or software) are taking part in
transaction. (Simple, Complex, Average Actors),
 Transactions represent the interactions of the actors with system
under testing.
 Test case characteristics: Accurate, Economical, Real, and
reusable, Traceable to requirements, Appropriate, Self standing,
Self-cleaning… etc.)
 How to write a good test case?
Test Case:
Test case is an important document from validation of system
perspective. These must be written at each stage of validation from unit
test till acceptance testing.
Basic principles of writing a good test case are:
 Must be testable.
 What is to be done when to wait for system to do it?
 Inform tester each transaction displayed/ replied by the system on
screen at each step. And wait for user response.
 Use simple conversational language for writing test case, which
improves clarity and avoid communication losses.
 Use consistent names of fields must be used in place of generic
names. Any change in field name must be incorporated in test cases.
 Tester should aware windows basics.
 Order of the test cases must follow business scenario. Avoid time
wastage,
 Basic principles of writing a good test case

 Test case must be testable.

 Tester should know what is to be done when to wait for system to
do it.
 Inform tester each transaction displayed/ replied by the system on
screen at each step. And wait for user response.
 Use simple conversational language for writing test case, which
improves clarity and avoid communication losses.
 Use consistent names of fields must be used in place of generic
names. Any change in field name must be incorporated in test
cases.
 Tester should aware windows basics.
 Order of the test cases must follow business scenario. Avoid time
wastage,
 Common Mistakes in writing test cases
 Making test cases too long and combining two or more test cases
in single test should be avoided
 Incomplete, incorrect, and incoherent test cases can create
confusion and frustrate testers.
 Steps should be made very clear in test case steps.
 Test case changes must be updated in software user interface.
 Define pass/fail criteria correctly, i.e. test are successful or not,
there is a defect or not?
Essential activities in Testing:
 Maintain test strategy, test plan, test scenario and test cases in a
location so that they are available to concerned people when
required for test artifacts.
 Follow configuration management standards, and reasons for
updates, Test cases used for testing must be reviewed before using
them.
 Entry and Exit Criteria related to Testing
Sample Criteria For Component Testing
Entry Criteria Exit Criteria
Periodic unit test progress No extreme and critical outstanding
report showing 70% defects in features
completion rate
Stable build with basic All 100% components test executed
features working with at least 98% pass ratio.
Component test cases can Component test progress report and
ready for execution defect trends sorted based on features
and analyzed
--- Component level performance and load
testing report and analysis of the same
 Entry/Exit Criteria OR
Entry Task Verification eXit Model
 Exit Criteria for Validation Testing
When to Start and Stop Testing of Software (Entry and Exit Criteria)
Process model - way to represent any given phase of software
development that prevent and minimize the delay between defect injection
and defect detection/correction.
 Entry criteria –specifies when that phase can be started also included
are the inputs for the phase.
 Tasks or steps that need to be carried out in that phase, along with
measurements that characterize the tasks.
 Verification, which specifies methods of checking that tasks have been
carried out correctly.
 Exit criteria, which stipulate the conditions under which one can
consider the phases as done and included are the outputs for only the
phase.
This is known as the Entry Task Verification exit or EVTX model which
offers several advantages for effective verification and validation.
 Clear entry criteria make sure that a given phase does not start
prematurely.
 Exit Criteria for Validation Testing
 All test scripts have been executed.
 All Software Planning Resolutions have been satisfactorily
resolved. (Resolution could include bugs being fixed, deferred to
a later release, determined not to be bugs, etc.) All parties must
agree to the resolution. This criterion could be further defined to
state that all high-priority bugs must be fixed while lower-priority
bugs can be handled on a case-by-case basis.
 All changes made as a result of SPRs have been tested.
 All documentation associated with the software (such as SRS,
SDD, test documents) has been updated to reflect changes made
during validation testing.
 The test report has been reviewed and approved.
 Software quality assurance is:
• Software engineering is systematic, disciplined and quantitative approach at its
core, makes the software engineering environment a good infrastructure for
achieving SQA objectives.

• The methodologies and tools applied, to a considerable extent, the level of

quality to be expected from the software process and the maintenance services.

Software quality assurance is:

• A systematic, planned set of actions necessary to provide adequate
confidence that the software development process or the maintenance process
of a software system product conforms to established functional technical
requirements as well as with the managerial requirements of keeping the
schedule and operating within the budgetary confines.
1. A planned and systematic pattern of all actions necessary to provide
adequate confidence that an item or product conforms to established technical
requirements.
2. A set of activities designed to evaluate the process by which the
products are developed or manufactured. Contrast with: quality control
 Quality Assurance
• Concentrates on the process
of producing the product.
• Defect-prevention oriented.
• Usually done throughout the
life cycle.
• This is usually a staff
function
• i.e. Reviews and Audits
Quality Control
• Concentrates on specific
product
• Defect-detection and
correction-oriented
• Usually done after the
product is build
• This is usually a line function
• i.e. Software testing at
various levels.
V-Model
 Advantages / Disadvantages & usage of V-Model
Advantages of V-model: Disadvantages of V-model:
 Simple and easy to use.  Very rigid and least flexible.
 Testing activities like planning, test  Software is developed during
designing happens well before coding. the implementation phase, so
 Saves a lot of time. no early prototypes of the
 Higher chance of success over the software are produced.
waterfall model.  If any changes happen in
 Proactive defect tracking – that is defects midway, then the test
are found at early stage. documents along with
 Avoids the downward flow of the defects. requirement documents has
 Works well for small projects where to be updated.
requirements are easily understood.
When to use the V-model:
 The V-shaped model should be used for small to medium sized projects

where requirements are clearly defined and fixed.

 The V-Shaped model should be chosen when ample technical resources are

available with needed technical expertise.

 Attribute of test case suite
1. Project Name
2. Test suite ID and name
3. Version date, number, and author of test case
4. Approval and distribution date
5. Revision history with reasons for updates
6. Environment pre-requisites

Attribute for each test case

• Test pre-condition • Priority
• Test sequence • Test data
• Traceability to test scenario • Steps to reproduce
• Test case name and number • Expected results
• Types of testing • Actual results
• Objectives • Test result
• Valid/Invalid conditions • Remarks
What Parts Make Up a Software Product?
 What is a computer bug?
• In 1947 Harvard University was
operating a room-sized computer
called the Mark II.
– mechanical relays
– glowing vacuum tubes
– technicians program the computer by reconfiguring it
– Technicians had to change the occasional vacuum tube.
• A moth flew into the computer and
was zapped by the high voltage
when it landed on a relay.
• Hence, the first computer bug!
– I am not making this up :-)
 Bugs a.k.a. …
• Defect • Failure
• Fault • Inconsistency
• Problem • Product
• Error Anomaly
• Incident • Product
• Anomaly Incidence
• Variance • Feature :-)
 Defective Software
• We develop programs that contain
defects
– How many? What kind?
• Hard to predict the future, however…
it is highly likely, that the software we
(including you!) will develop in the
future will not be significantly better.
 Sources of Problems
• Requirements Definition: Erroneous, incomplete,
inconsistent requirements.
• Design: Fundamental design flaws in the software.
• Implementation: Mistakes in chip fabrication, wiring,
programming faults, malicious code.
• Support Systems: Poor programming languages,
faulty compilers and debuggers, misleading
development tools.
Sources of Problems (Cont’d)
• Inadequate Testing of Software:
Incomplete testing, poor verification,
mistakes in debugging.
• Evolution: Sloppy redevelopment or
maintenance, introduction of new flaws
in attempts to fix old flaws, incremental
escalation to inordinate complexity.
 Adverse Effects of
Faulty Software
• Communications: Loss or corruption of
communication media, non delivery of
data.
• Space Applications: Lost lives, launch
delays.
• Defense and Warfare: Misidentification
of friend or foe.
 Adverse Effects of Faulty
Software (Cont’d)
• Transportation: Deaths, delays,
sudden acceleration, inability to brake.
• Safety-critical Applications: Death,
injuries.
• Electric Power: Death, injuries, power
outages, long-term health hazards
(radiation).
 Adverse Effects of Faulty
Software (Cont’d)
• Money Management: Fraud, violation of
privacy, shutdown of stock exchanges and
banks, negative interest rates.
• Control of Elections: Wrong results
(intentional or non-intentional).
• Control of Jails: Technology-aided escape
attempts and successes, accidental release
of inmates, failures in software controlled
locks.
• Law Enforcement: False arrests and
imprisonments.
 Bug in Space Code
• Project Mercury’s FORTRAN code had the
following fault:
DO I=1.10 instead of ... DO I=1,10
• The fault was discovered in an analysis of
why the software did not seem to generate
results that were sufficiently accurate.
• The erroneous 1.10 would cause the loop to
be executed exactly once!
 Military Aviation Problems
• An F-18 crashed because of a missing
exception condition:
if ... then ... without the else clause
that was thought could not possibly
arise.
• In simulation, an F-16 program bug
caused the virtual plane to flip over
whenever it crossed the equator, as a
result of a missing minus sign to
indicate south latitude.
 Year Ambiguities
• In 1992, Mary Bandar received an
invitation to attend a kindergarten in
Winona, Minnesota, along with others
born in '88.
• Mary was 104 years old at the time.
 Year Ambiguities (Cont’d)
• Mr. Blodgett’s auto insurance rate
tripled when he turned 101.
• He was the computer program’s first
driver over 100, and his age was
interpreted as 1.
• This is a double blunder because the
program’s definition of a teenager is
someone under 20!
 Dates, Times, and Integers
• The number 32,768 = 215 has caused
all sorts of grief from the overflowing of
16-bit words.
• A Washington D.C. hospital computer
system collapsed on September 19,
15
1989, 2 days after January 1, 1900,
forcing a lengthy period of manual
operation.
 Dates, Times, and Integers
(Cont’d)
• COBOL uses a two-character date field
…
• The Linux term program, which allows
simultaneous multiple sessions over a
single modem dialup connection, died
word wide on October 26, 1993.
• The cause was the overflow of an int
variable that should have been defined
as an unsigned int.
 Shaky Math
• In the US, five nuclear power plants
were shut down in 1979 because of a
program fault in a simulation program
used to design nuclear reactor to
withstand earthquakes.
• This program fault was, unfortunately,
discovered after the power plants were
built!
 Shaky Math (Cont’d)
• Apparently, the arithmetic sum of a set
of numbers was taken, instead of the
sum of the absolute values.
• The five reactors would probably not
have survived an earthquake that was
as strong as the strongest earthquake
ever recorded in the area.
 Therac-25 Radiation
“Therapy”
• In Texas, 1986, a man received between
16,500-25,000 rads in less than 1 sec, over
an area of about 1 cm.
• He lost his left arm, and died of complications
5 months later.
• In Texas, 1986, a man received at least 4,000
rads in the right temporal lobe of his brain.
• The patient eventually died as a result of the
overdose.
 Therac-25 Radiation
“Therapy” (Cont’d)
• In Washington, 1987, a patient received
8,000-10,000 rads instead of the
prescribed 86 rads.
• The patient died of complications of the
radiation overdose.
 AT&T Bug: Hello? ... Hello?
• In mid-December 1989, AT&T installed
new software in 114 electronic switching
systems.
• On January 15, 1990, 5 million calls
were blocked during a 9 hour period
nationwide.
 AT&T Bug (Cont’d)
• The bug was traced to a C program that
contained a break statement within an
switch clause nested within a loop.
• The switch clause was part of a loop.
Initially, the loop contained only if
clauses with break statements to exit
the loop.
• When the control logic became
complicated, a switch clause was
added to improve the readability of the
code ...
 Bank Generosity
• A Norwegian bank ATM consistently
dispersed 10 times the amount
required.
• Many people joyously joined the queues
as the word spread.
 Bank Generosity (Cont’d)
• A software flaw caused a UK bank to
duplicate every transfer payment
request for half an hour. The bank lost
2 billion British pounds!
• The bank eventually recovered the
funds but lost half a million pounds in
potential interest.
 Making Rupee!
• An Australian man purchased $104,500
worth of Sri Lankan Rupees.
• The next day he sold the Rupees to
another bank for $440,258.
• The first bank’s software had displayed
a bogus exchange rate in the Rupee
position!
• A judge ruled that the man had acted
without intended fraud and could keep
the extra $335,758!
 Bug in BoNY Software
• The Bank of New York (BoNY) had a
$32 billion overdraft as the result of a
16-bit integer counter that went
unchecked.
• BoNY was unable to process the
incoming credits from security transfers,
while the NY Federal Reserve
automatically debited BoNY’s cash
account.
 Bug in BoNY Software
(Cont’d)
• BoNY had to borrow $24 billion to cover
itself for 1 day until the software was
fixed.
• The bug cost BoNY $5 million in interest
payments.
 Discussion …
• Have you heard of other software bugs?
– In the media?
– From personal experience?
• Does this embarrass you as a future
software engineer?
 Specification
“if you can’t say it, you can’t do it”
• You have to know what your product is before
you can say if it has a bug.
• A specification defines the product being
created and includes:
– Functional requirements that describes the
features the product will support. E.g., on a word
processor
• Save, print, check spelling, change font, …
– Non-functional requirements are constraints on
the product. E.g,
• Security, reliability, user friendliness, platform, …
A software bug occurs when at
least one of these rules is true
• The software does not do something that the
specification says it should do.
• The software does something that the
specification says it should not do.
• The software does something that the
specification does not mention.
• The software does not do something that the
product specification does not mention but
should.
• The software is difficult to understand, hard to
use, slow …
Most bugs are not because of
mistakes in the code …
• Specification (~= 55%)
• Design (~= 25%)
• Code (~= 15%)
• Other (~= 5%)
 Relative cost of bugs
“bugs found later cost more to fix”
• Cost to fix a bug increases exponentially (10x)
– i.e., it increases tenfold as time increases
• E.g., a bug found during specification costs $1 to
fix.
• … if found in design cost is $10
• … if found in code cost is $100
• … if found in released software cost is $1000
 Bug Free Software
• Software is in the news for the wrong reason
– Security breach, Mars Lander lost, hackers getting
credit card information, etc.
• Why can’t software engineers develop
software that just works?
– As software gets more features and supports more
platforms it becomes increasingly difficult to make
it create bug-free.
 Discussion …
• Do you think bug free software is
unattainable?
– Are their technical barriers that make this
impossible?
– Is it just a question of time before we can
do this?
– Are we missing technology or processes?
 Goal of a software tester
• … to find bugs
• … as early in the software development
processes as possible
• … and make sure they get fixed.

• Advice: Be careful not to get get caught

in the dangerous spiral of unattainable
perfection.
What to look for when interviewing someone
for the position of software tester
• Are they explorers?
• Are they troubleshooters?
• Are they relentless?
• Are they creative?
• Are they perfectionists (within reason)?
• Do they exercise good judgment?
• Are they tactful and diplomatic?
• Are they persuasive?
 You now know …
• … what is a bug
• … the relationship between specification and
bugs
• … the cost of a bug relative to when it is
found
• … the unattainable goal of perfect software
• … the goal of the software tester
• … valuable attributes of a software tester
 The Software
Development Process

[Reading assignment: Chapter 2, pp. 23-36]

 Software is …
• requirements specification documents
• design documents
• source code
• test suites and test plans
• interfaces to hardware and software
operating environment
• internal and external documentation
• executable programs and their persistent
data
 Software effort
• Specification
• Product reviews
• Design
• Scheduling
• Feedback
• Competitive information acquisition
• Test planning
• Customer surveying
• Usability data gathering
• Look and feel specification
• Software architecture
• Programming
• …
 Discussion …
• What is software engineering?
• Where does testing occur in the
software development process?
 Customer requirements
• The software development team must
determine what the customer wants.
• How can you do this?
– Guess?
– Collect detailed information from surveys?
– Get feedback from a previous version of the
software?
– Read reviews in magazines?
– Get information about the competition?
– Other ways?
• The collected data is used to guide the
specification effort.
 Specification
“If you don't know where you're going any road will take you there”

• The specification takes the data from the

customer requirements and other sources
and defines:
– The features of the software (functional
requirements).
– The constraints on these features (non-functional
requirements).
• Specifications can be:
– formal (e.g., aerospace industry), rigid
– informal (e.g., a .com start up), on a cocktail
napkin or a whiteboard.
 Schedules
• The goals of scheduling are to know:
– What work needs to be completed?
– How much work is left to do?
– When will the work be finished?
– Who will finish each task?
– Other measurable queries.
• A Gantt chart is a popular type of bar chart
that illustrates a project schedule.
Gantt Chart
 Design
• Before coding begins on non-trivial software
projects, a set of design documents are
created to serve as blueprints.
– Software Architecture
– Data flow diagram
– State transition diagram
– Flowchart
– Commented code
 Source code … of course
• The ultimate specification of the
software!
• ‘Code is king’ philosophy is
still prevalent.
• Many programming
languages and tools to
choose from.
 Test documents
• Test plan
– Quality objectives, resource needs, schedules,
assignments, methods, etc.
• Test cases
– Inputs and expected outputs.
• Bug reports
– E.g., the Bugzilla web-based bug tracker.
• Test tools and automation
• Metrics, statistics, and summaries
– Number of unresolved bugs, mean time to repair a
bug, etc.
 Software Project Staff
• Project managers
– Write product specification, manage the schedule, critical
decision tradeoffs
• Software architects, system engineers
– Design the software, work closely with programmers
• Programmers, developers, coders
– Write code, fix bugs
• Testers, quality assurance staff
– Find bugs, document bugs, track progress of open bugs
• Technical writers
– Write manuals, on line documentation
• Configuration managers, builders
– Packaging and code, documents, and specifications
Software Development Lifecycles
• Code and Fix
• Waterfall
• Spiral
 You now know …
• … what is software
• … what is software engineering
• … what is the composition of a software
development organization
• … what are the major phases of a software
development project
• … how major phased are organized