Software Testing

Technology
Chapter 2 Concepts

李鹏飞（ Pengfei
Li ）
pengfei.li@hbut.edu.cn
 Chapter 1 Review

① What is Software Testing

② Two sides
 Forward-Verify that the
software works normal
 Backward-Assume the
software is defected
 Verification & Validation
③ Testing VS Developing
④ Test Driven Development
 Chapter 2
Concepts of Software Testing
2.1 Software Defects
2.2 Classifications of ST
2.3 Static Testing & Dynamic Testing
2.4 Active Testing & Passive Testing
2.5 Black Box Testing vs. White Box Testing
2.6 Software Testing Levels
2.7 Software Testing Plans and Use Cases
2.8 Responsibilities and Requirements of Professional
Testers
 Defects are the antithesis
of quality
To understand what a defect is, you must be clear
about the concept of "Quality", because defects exist
relative to quality. Defects come into being when we
violate quality, violate the wishes of customers, or fail to
meet customers’ requirements.

Software Testing
Requirments Contradictions Find

Defects
Quality

Customer
Expectations
 2.1 Software Defects

2.1.1 The connotation of software quality

2.1.2 Definition of Software Defects
2.1.3 How software defects come into being
2.1.4 Composition of software defects
2.1.5 The cost of fixing software defects
What is quality ？
What is quality ？
 Quality = Brands =
customer satisfaction
 The connotation of
software quality
IEEE ： Quality is the degree to which the system,
component or process meets
① clear needs
② Customer/user needs or expectations
 Software quality ： The software product has the
relevant characteristics and the sum of the
characteristics (ISO 8492) that meet the specified
or implicit requirements
 Software quality ： the degree to which the
software product meets the requirements for use
 High quality software
standard system
Product quality
 It is the attributes and behaviors of people's practices that can
be understood and described scientifically. And some
methods and human activities can be used to improve the
quality
 Quality models: McCall, Boehm, ISO 9126

Process quality: 流程 产品
 CMM ( Capability Maturity Model).
 ISO 9000 商业环境
 SPICE ( Software Process Improvement and Capability
dEtermination)

Quality content in the business process:

Training, publicity, release date, customers, risk, cost, etc
 Standards for product
quality

- Functionality
- Usability
- Reliability
- Performance
- Capacity non-functional
characteristics
- Scalability
- Service manageability
- Compatibility
- Extensibility
 software quality
characteristics (ISO 9126)
 Functionality ： A set of properties related to a set of
functions and their specified properties, where functions are
those functions that satisfy an explicit or implicit
requirement
 Reliability ： A set of attributes related to the ability of
software to maintain its level of performance over a specified
period of time and conditions
 Usability ： A set of attributes related to the effort and
evaluation required by a set of prescribed or potential users
to use the software
 software quality
characteristics (ISO 9126)
 Efficiency ： A set of properties related to the
relationship between the performance level of software and the
amount of resources used under specified conditions
 Maintainability ： A set of properties related to the
effort required to make a specified modification
 Portability ： A set of properties related to the ability of
software to transfer from one environment to another. Each of
these quality features corresponds to several sub-features.
 ISO/IEC 9126
Software Quality Model
Boehm Model
 Two standard systems of
ISO/IEC 9126-1991

16
 Latest quality standards:
ISO25000 series

ISO/IEC25000 SE
Systems and software Quality Requirements and Evaluation (SQuaRE)
 Internal qualityexternal
quality  quality in use

18
Internal quality and
external quality
Internal quality and
external quality
 Quality in use

21
 Quality in Use for Web Portal

22
2.1.2 definition of software bugs

Any problem/disfigurement/limitation in product design & development

Feature or function can’t work

Any problems in the program; the
Unreasonable design
What is a bug?
Partly realization in function
inconsistency of the
Data error
Run error
specification; not meeting the
Limitation in features
needs of users
Difference between actual results and expected results
Unfriendly UI, Low performance
Others
First Bug

http://en.wikipedia.org/wiki/Grace_Hopper
 缺陷 – Defect, Bug

缺点（ defect ） 偏差 （ variance ）

谬误（ fault ） 失败 （ failure ）
问题（ problem ） 矛盾
（ inconsistency ）
错误（ error ） 毛病 （ incident ）
异常（ anomy ）
 Software defects

ISO 29119
(1) a flaw in a component or system that can cause it to fail to perform
its required function.
(2) any condition that deviates from expectation based on
requirements specifications, design documents,

NOTE Defects may be found during, but not limited to, reviewing, testing,
analysis, compilation, or use of software products or applicable documentation
 Phenomenon of software
defects
 Functions, features are not
implemented or partially implemented
 Poorly designed
 Actual results do not match expected
results
 Operation errors, including
interruptions, system crashes,
interface confusion
 Incorrect data results and
insufficient precision
 Other problems that users cannot
accept, such as long access time and
unsightly interface
 How software defects
come into being
① Technical issues
Algorithm error, syntax error, calculation and precision
problem, interface parameter passing mismatch

② Teamwork issues
Inadequate communication, misunderstanding

③ Software issues
 Document errors 、 user scenario
 Problems caused by time inconsistency
 Self-recovery of the system or off-site backup of data,
catastrophic recovery, etc.
Software Defects
Composition

6%

15%

54%
25%

Specificaiton design
coding others
 Distribution of software
defects at different stages
Defects
prevention
Early discovering of defects
70-90% before testing
Defects

Requireme Design coding Unit test release

nts Time test

Before the actual program testing, more defects can be found through the review and
review meeting. Defects in specifications will be gradually discovered in the process of
requirements analysis and review, design, coding, testing, etc., but cannot be found in
one stage of requirements analysis.
Cost of defects
 2.3 classifications of
software testing
phases
Acceptance Test

System test

Integration test
Unit test

methods
functional test
Robustness Test White- Black-
performance test box test box test

Accessibility testing
Security test
reliability test

objectives
 Different classifications

 Based on testing objects : unit testing, document testing, system

testing
 Based on test purpose : functional testing, regression testing,
performance testing, reliability testing, security testing and
compatibility testing
 According to whether the software under test is executed during
the test : static testing and dynamic testing
 According to whether the test is completed for the internal
structure of the system and the specific implementation
algorithm, it can be divided into white box testing and black box
testing
 2.3 static testing and
dynamic testing

2.3.1 product review

2.3.2 Static Analysis
2.3.3 Verification and
Validation
 Static and dynamic

chairman internal auditor

programmers

others

technical
professionals

recorder
user

formal
informal
execution
Pass- walk- Peer Inspecti
round throu review on
gh
 static testing and dynamic
testing

 Incorporating the review of requirements and

design into the category of testing. can be
regarded as generalized testing
 Static testing includes review of software product
requirements and design specifications, review of
program code, etc.
 static testing and dynamic
testing

 The error checking and analysis functions of static analysis

cannot be replaced by other methods. Manual detection and
computer-aided static analysis methods can be used for
detection, but more and more tools are used for automatic
analysis.
 Dynamic testing is to find errors by actually running the
program, obtain system information by observing the running
process of the code, and verify the system behavior
 2.3.1 product review

 Through software reviews, problems in all aspects of

requirements engineering and software design can be
identified earlier, greatly reducing the amount of post-
production rework and transforming quality costs from
expensive post-production rework to upfront defect
detection.
 A review is a means of evaluating the status of a
software element or project to determine whether it is
consistent with planned results and for improvement.
Verify that the work product correctly meets the
specifications established in the previous work product.
 Format/method of the review

 互为评审 (Peer review)

 轮查 (Pass-round)
 走查 (walk-through)
 会议评审 (Inspection)

formal
informal

Pass- walk- Peer review Inspection

round through
 Classifications of Review

 management review
 technical review
 document review
 Process review
 Requirements and Design
Review

Testers participate in product

requirements analysis and system
design, carefully read relevant
documents, truly understand customer
needs and technical design, check the
accuracy and consistency of product
descriptions in the requirements
specification, and check the rationality
and testability of system design Wait
Requirements and Design
Review
 2.3.2 static analysis

 Manual inspection: Manual inspection focuses on the

inspection of coding style and quality, analyzes design
and code, and effectively finds logical design and coding
errors.
 Computer-aided static analysis: Use static analysis tools
to analyze the characteristics of the program under test,
extract some information from the program, so as to check
various defects of program logic and suspicious program
structure.
 2.3.3 Verification and
Validation （ V & V ）

Verification ： Are we building the product right ？

 Is the software structured correctly? That is, doing things
right, verifying that the development process adheres to
what has been defined. Verify that the product meets the
design specification and conformance to the specification

Validation ： Are we building the right product?

 Has the software been constructed exactly what the user
needs? i.e. is doing the right thing. Verify that the
functions implemented by the product meet the needs of
users
Verification and
Validation （ V &
V）
 2.4 Active and passive
testing
 Active Testing ： Testers actively send requests to
the object under test, or drive the behavior of the
object under test with the help of data and events,
so as to verify the object's response or output
results
 Passive Testing ： The tester does not interfere with
the operation of the product, but passively monitors
the operation of the product in the actual
environment, and obtains the data of the system
operation through a certain passive mechanism,
including input and output data
 2.5 Black box testing and white
box testing
Customer requirements
Structure test
Logic-driven test
output
input

Event-driven

Functional test
Data-driven test
conclusion
2.6 software testing levels

49
 Testing purposes

Functionality and
User Interface
Security Efficiency
Acceptance Testing User Acceptability

System function
System testing Security
Robustness
Efficiency
Integration Testing

interface between components

Unit testing
Component functionality

coding Robustness
Efficiency
debug
 Unit testing

 Unit testing is aimed at testing the smallest unit in a program system

—a module or component, generally synchronizing with coding. The
white-box testing method is mainly used to design test cases from
the internal structure of the program to check whether the
implemented functions of the program modules or components are
consistent with the defined functions, and whether there are errors
in the coding. Usually write driver modules and stub modules
 Unit testing is generally completed by programmers and testers,
and mainly by developers
 Unit testing includes code review, which can find 50% to 70% of the
defects in the code.
 Examples

http://www.appperfect.com/products/java-unit-test-features.html
 Integration testing

Integration testing, also known as assembly

testing and joint testing, is based on unit
testing. Modules are assembled according to
design requirements and tested at the same
time. The main goal is to find problems
between modules related to interfaces. Two
integration methods: one-time integration and
incremental integration
Continuous integration,
continuous testing
 System Functionality Testing

Functional testing is generally performed after integration testing is

completed, and it is tested against the application system. The
functional test is based on the product function specification, which is
to verify the functions of the known product from the user's point of
view to confirm whether each function can be used normally.
 System Non-Functional
Testing

System non-functional testing is to put the software in the entire

computer environment, including software and hardware platforms, some
supporting software, data and personnel, etc., and conduct a series of
tests in the actual operating environment, including:
 recovery test
 Security test
 Strength test
 Performance Testing
 ……
 Acceptance Testing &
Installation Testing
 The purpose of acceptance testing is to demonstrate
to future users that the system works as intended,
and to verify that the software functions and
performs as users reasonably expect

 Installation test refers to a step-by-step installation

operability test in accordance with the software
product installation manual or corresponding
documents in an environment that is exactly the
same as the user's use of the product or equivalent
to the user's use environment
 α 、 βtesting &testing
in production
 Alpha testing is simulated or actual operational testing
by potential users/customers or an independent test
team at the developers‘ site ； Is a form of internal
acceptance testing
 Beta testing comes after α testing. Versions of the
software, known as beta versions, are released to a
limited users outside of the programming team.
 Testing in production

http://blogs.msdn.com/b/seliot/
http://www.thetestingplanet.com/2011/11/the-future-of-software-testing-part-one-testing-in-production /
 2.7 Software Test Plans and
Use Cases
 Organization and management of software testing work:
formulate test strategies and test plans, confirm the test
methods and specifications adopted, control the test
progress, and manage test resources.
 Implementation of testing work: Compile standard-
compliant test documents, build test environments,
develop test scripts, and collaborate with development
organizations to implement testing activities at various
stages
Testing Workflow
 Test plan

 Objectives and Scope

 Project Estimation
 Risk Plan
 schedule
 Resource configuration
 Tracking and Control

Mechanisms
 Test cases

 Test cases are an important reference for testers in

the testing process.

 Test cases will help save testing time and improve

testing efficiency.
 Good test cases are continuously reused, making the

testing process more efficient.

 Test cases are a process of accumulation of

knowledge
2.8 Responsibilities and Requirements
of Professional Testers
 QA/test manager: personnel management, resource allocation, test
method improvement, etc.;
 lab manager: set up, configure and maintain the test environment
in the lab
 Internal auditor: review process, establish test templates, track the
quality of defect test reports, etc.;
 test team leader: responsible for project management, test plan,
test cases, task scheduling, etc.
 Test designer/senior test engineer, review of product specification,
design of test cases, solution of technical problems, training and
guidance, execution of actual test tasks; general (junior) test
engineer, execution of test cases and related test tasks
 A day in the life of a
Microsoft test engineer

a) The product compilation must be completed before this time.

b) The test compilation starts automatically at 3:00 am every day.
c) If the test compilation is successful, the BVT test starts automatically.
d) The test engineer comes to work every morning and checks the
email
e) If there is a BVT error, analyze the cause in the first time, isolate the
error code and report the Pri 0 Bug (level 0 defect)
f) The development team should complete the revision of the Pri 0 bug
within the same day
g) The test engineer then uses Product Studio Check the bug situation
and verify that the bug assigned to you has been modified and
qualified
 A day in the life of a
Microsoft test engineer
h) Close the bug and add a Regression Test for this bug
i) Verify the latest Lab Run results
j) If there are new errors, isolate and report new bugs
k) Develop new test Spec and new test code
l) Use personal Private Run To verify the newly developed test program
m) Use personal Private Run to verify that the product program newly
developed by the developer has no major errors
n) Improve and improve the function of the automated test system
o) Participate in the Spec, Test Spec Review meeting, and do the test
partner test code Review, UE help file Review, answer questions from
inside and outside Newsgroup
 Requirements for testers

Technology, programming ability

Sense of responsibility, endurance
Communication ability, understanding ability
Analysis ability (critical thinking)
Project management ability
Organization ability

…
 Qualities of an Excellent
Test Engineer

High sense of responsibility

Very good communication skills, sense of humor

Technical skills, self-confidence, patience
Doubtful spirit, diligent spirit Insight, moderate curiosity

Reverse thinking and divergent thinking ability, memory

 Self-learning ability, innovation ability, etc.
Q&A
Thank you