BITP 3253:

VERIFICATION &
VALIDATION
Week 4: Testing Categorization
and Testing Targets

Fakulti Teknologi Maklumat dan Komunikasi

 Overview
• Testing Categorization
• Static Test Vs Dynamic Test
• White Box Vs Black Box
• Testing Targets
• Maintenance Testing

Fakulti Teknologi Maklumat dan Komunikasi

Different Testing Categorization

Testing

Static
Dynamic

Static
Review Analysis

White Black
Box Box

Fakulti Teknologi Maklumat dan Komunikasi

Static Testing Vs Dynamic Testing

Static Testing: Testing of component or system at

specification or implementation level without
execution of that software.
(e.g Reviews or static code analysis)
Dynamic Testing: Testing that involves the
execution of the software of a component or
system

Fakulti Teknologi Maklumat dan Komunikasi

Black Box Testing Vs White Box
Testing
Black Box Testing: Testing, either functional or non-
functional without reference to the internal
structure of the component or system. And,
focuses solely on the output generated in
response to selected inputs and execution
conditions
White Box Testing: Testing based on an analysis of
the internal structure of the component or
system

Fakulti Teknologi Maklumat dan Komunikasi

 Testing Targets

Each testing types targeted on a particular

objective:-
• Testing of function (Functional Testing)
• Testing of non-functional (Non-functional
Testing)
• Testing of software structure /architecture
(Structural Testing)
• Testing related to changes (Confirmation
Testing and Regression Testing)
Fakulti Teknologi Maklumat dan Komunikasi
 Functional Testing
Functional ?? – “What system does?” – commonly
described by work products such as
requirements specification, use cases, functional
specification and maybe undocumented.
** If non documented, tester must understand
“reasonable behavioral” of the system
• Functional testing is based on functional and
features and their interoperability with specific
systems.
• It maybe performed at all test levels

Fakulti Teknologi Maklumat dan Komunikasi

 What Functional Testing Do?

• Reasonable or required action not provided,

inaccessible, or seriously impaired
• No add function on a calculator
• Add function implemented, “+” key doesn’t work
• Can only add integers, not real numbers
• Right action, wrong result
• Add function: 2+2=5
• Right action, right result, wrong side effect
• Divide function: 2/2=I (Roman numeral format)

Fakulti Teknologi Maklumat dan Komunikasi

 Functional Testing

• Specification-based techniques maybe used to

derive test conditions and test cases from the
functionality of the software or system
• Functional testing includes:-
• Security testing
• Interoperability testing

Fakulti Teknologi Maklumat dan Komunikasi

 Security Testing
Security Testing : investigates the functions (e.g
firewall) relating to detection of threats, such as
viruses, from malicious outsiders.
• Security threats includes:-
• Virus, Cracking into servers, Denial of services
• Things to focus in security:-
• Authentication
• Authorization
• Confidentiality
• Integrity
• Its is a specialized field of test expertise
Fakulti Teknologi Maklumat dan Komunikasi
 Interoperability Testing

Interoperability Testing : evaluates the capability of

the software product to interact with one or more
specified components or systems
• Interoperability with programs, operating
systems, databases.
• Includes..
• Interfaces
• Data interchange

Fakulti Teknologi Maklumat dan Komunikasi

 Non-Functional Testing
Non-functional: Describes the test required to measure characteristics
of systems and software that can be quantified on varying scale.
Maybe performed at all levels of testing

ISO 9126 – Software Engineering- Software Product Quality.

Functional: Suitability, accuracy, interoperability, security, compliance
Reliability: Robustness, fault-tolerance, recoverability, compliance
Usability: Understandability, learnability, operability, attractiveness,
compliance
Efficiency: Time behavior, resource utilization, compliance
Maintainability: Analyzability, changeability, stability, testability,
compliance
Portability: Adaptability, installability, co-existence, replaceability,
compliance
Fakulti Teknologi Maklumat dan Komunikasi
 Performance and Reliability

• Performance
• measure the time behavioral with respect to the
specific resource (e.g RAM, CPU, harddrive,
network) usage
• Part of Efficiency characteristic
• Reliability
• System fails to complete normal functions
• System functions normally, but randomly crashes
or hangs

Fakulti Teknologi Maklumat dan Komunikasi

 Stress, Capacity and Volume

• Stress: extreme conditions cause failure

 Combinations of error, capacity, and volume tests
• Capacity: functionality, performance, or reliability
problems due to resource depletion
• Fill hard drive or memory to 80+%
• Volume: functionality, performance, or reliability
problems due to rate of data flows
• Run 80+% of rated transactions per minute, number
of simultaneous users, etc.

Fakulti Teknologi Maklumat dan Komunikasi

Maintenance And Maintainability
Maintenance Maintainability
• Update and patch • Software itself (source

install and de-install code) not maintainable

process don’t work • Databases not

• Configurations can’t be upgradeable

changed appropriately • Databases grow
(e.g. plug-and-play, hot monotonically
plugging, adding disk • Software not efficiently
space, etc.) testable during
maintenance; e.g
excessive regression

Fakulti Teknologi Maklumat dan Komunikasi

 Usability and User Interface
Usability : How comfortable user with the system.
• The capability of the software to be understood,
learned, used and attractive to the user when used
under specified condition
Example:-
• A system can function properly but unusable by the
intended customer
• Cumbersome interfaces that do not follow workflows
• Inaccessible functionality
• Inappropriately difficult for user to learn
• Instructional, help, and error messages that are
misleading
Fakulti Teknologi Maklumat dan Komunikasi
 Configuration and Portability

• A single platform may be configured in many

different ways in software
• A family of platforms may support various
hardware configurations
• Are configuration changes handled?
• Add disk space or other storage
• Add memory
• Upgrade or add CPU
• Portability to various environments

Fakulti Teknologi Maklumat dan Komunikasi

 Structural Testing
• Test based on how the system is built:-
• Code, data, design

• Structural technique are best used after

specification based techniques, to help
measure the thoroughness of testing
through assessment of coverage of a type
of structure.
• Coverage: the extent that a structure has
been exercised by a test suite, expressed
as a percentage of the items being covered
Fakulti Teknologi Maklumat dan Komunikasi
 Regression and Confirmation
Testing
• Regression testing is the repeated testing of an already
tested program, after modification, to discover any
defects introduced or uncovered as a result of the
changes.
• Confirmation testing confirms that..
• Changes made to the system are present
• Bug fixes introduced in the system solve the
observed symptoms
• Repeatability of tests helps with regression and
confirmation testing
• Regression testing is a strong candidate for automation
Fakulti Teknologi Maklumat dan Komunikasi
 Maintenance Testing

• Maintenance testing is done on an existing

operational system triggered by:-
• Modification: enhancements, bugs fixes, operational
environment changes, patches
• Migration: moving to a newly supported environment
• Retirement: end-of-life of a subsystems or entire
systems triggers replacement
• Maintenance testing addresses the change itself
and what wasn’t changed and shouldn’t change

Fakulti Teknologi Maklumat dan Komunikasi

 Role of Regression Testing and
Impact Analysis
• Maintenance testing includes extensive
regression testing.

• Thus, do impact analysis to determine how the

existing system is affected by changes and how
much regression testing to do.

• Impact analysis should consider the risk of

change, the size of the existing system and the
size of the changes.
Fakulti Teknologi Maklumat dan Komunikasi
 Testing Maintenance Release
• Regression testing is the big risk for
maintenance release
• Some organizations try to put a major release
worth of features into a short maintenance
release
• Time to develop new test is scarce
• Large project test estimation rules-of thumb

Fakulti Teknologi Maklumat dan Komunikasi

 Regression Testing

• Regression types:-
• Local (fix creates new bugs)
• Exposed (fix reveals existing bugs)
• Remote (fix in one area breaks something in
another area)
• Regression can effect new and existing features

Fakulti Teknologi Maklumat dan Komunikasi

Regression Strategy 1: Repeat All
Tests
• If our tests address the important quality risks,
then repeating all tests should find most
important regressions.
• Automation is the only practical means for large
complex system

Fakulti Teknologi Maklumat dan Komunikasi

 Regression Strategy 2: Repeat
Some Tests
• Often, full automation is impossible
• Select some tests to repeat based on:-
• Traceability
• Change/impact analysis
• Risk analysis
• Use cross functional tests for “accidental
regression testing”
• Can use code coverage to access level of risk

Fakulti Teknologi Maklumat dan Komunikasi

 3 Other Regression Strategies

• Release more slowly

• Release every six months rather than every month
• Partial or even full repetition can increase coverage
on bigger release
• Combine emergency patches with slower release
process to allow for flexibility while keeping
regression risk low
• User customer or user testing
• Beta for mass market software
• Pilot, staged or phase, parallel release for IT
Fakulti Teknologi Maklumat dan Komunikasi
Thank You

Fakulti Teknologi Maklumat dan Komunikasi