Software Testing

Technology
Chapter 1 Introduction

李鹏飞（ Pengfei
Li ）
pengfei.li@hbut.edu.cn
 Chapter 1 Introduction

1.1 Necessity for Software Testing

1.2 Why Software Testing
1.3 What is Software Testing
1.4 Software Testing & Developing
1.5 Testing & Quality Assurance
1.6 The idea of ​Test-Driven Development(TDD)
 Question

Why do we need
software testing
 1.1 Necessity

1.1.1 Disney doesn't always bring laughs

1.1.2 One flaw caused hundreds of millions of
dollars of damage
1.1.3 Crash of Mars exploration spacecraft
1.1.4 More tragedies
 Disney doesn't always bring
laughs
On Christmas Eve in 1994, Disney released the first
multimedia CD-ROM game for children, "The Lion
King".
The first day after Christmas, the Disney customer
support phone started ringing non-stop, and people
kept asking and complaining about why they could not
play the game.
The installation always failed, and the game can't be
used normally. It turned out that it can only run
normally in a few systems.
 One flaw caused hundreds of
millions of dollars of damage

（ 4195835 ／ 3145727 ） ×3145727- 4195835 = ?

In the end, Intel finally paid a lot of money to recycle the

product, causing a loss of US$400 million
 Mars spacecraft

 Mechanical shock in most cases also

triggers the switch. When the spacecraft
starts to land, the computer will possibly
turn off the thrusters, and the spacecraft
fell 1800 meters without the help of
reverse thrusters and crashed into
pieces.
 There's nothing wrong for the two groups.
However integration test was not carried
out , and there is the tragedy.

https://www.bilibili.com/video/BV1YK4y1A7Jc?
from=search&seid=16393811987439043206&spm_id_from=333.337.0.0
 More tragedies

 A software flaw in the computer led to several patients

exposed to severe overdose of radiation therapy, 4 of
them therefore died
 When the Patriot Missile Defense System‘s clock
continuously runs over 14 hours, the system’s tracking
system becomes inaccurate. This lead to several failing
Scud missile attacks, one of which exploded in Doha,
Saudi Arabia, 28 US soldiers killed.
 Bug to prison
 On April 21, 2006,, Xu Ting came to
ATM to withdraw money. After taking
out 1,000 yuan, the balance was only
deducted by 1 yuan. After that, Xu
Ting withdrew money successively
171 times, with a total number of
175,000 Yuan. Xu Ting was caught
and sentenced to life imprisonment
for theft

http://news.163.com/07/1217/01/3VSLHQ4E0
0011229.html
 08 Olympic Games ticketing

On October 30, 2007, the ticket sales of the

second stage Beijing Olympics Games were
forced to suspend because there were too
many ticket buyers. Too many and forced to
suspend. The underestimating of crowd
ticketing enthusiasm lead to bottlenecks in the
ticketing system.
 Wenzhou
train collision
 Wenzhou
train collision

After the collision, four

cars of the rear train fell
off the Ou River bridge,
striking the ground more
than 20 m (66 ft) below

The accident occurred due

to severe defects in the
design of control center
equipment, lax equipment
inspection and failure to
adequately respond to
equipment malfunction
caused by lightning
 Why do we need software
testing
 Software bugs are inevitable. Only through testing can software
defects be found. And a software defect can only be removed from a
software product or when the defect is found.
 The defects in the software bring us huge losses, which also shows
that the necessity and importance of software testing
 Testing is a fundamental unit of all engineering disciplines, including
software development
 The better testers are, the sooner the software problems will be
found, and the software will be, the easier it is to correct, the more
stable the product is after it is released, the more the company earns
money. Microsoft is a typical example
 1.3 what is software
testing ？

 1.3.1 Formation of the

discipline
 1.3.2 Arguments

 1.3.3 Definitions

 1.3.4 Other Perspectives

 What is testing

• Check ？
• Detect error ？
• Verification ？
• Validation ？
• Correction proof ？
• Quality evaluation ？
• Quality Assurance ？
 Formation of the discipline

 1957 ～ 1978, Functional verification oriented, testing is to prove

that the software is correct （ Forward thinking ）
 1978 ～ 1983, destructive testing oriented, testing is to find bugs in
software （ Backward Thinking ）
 1983 ～ 1987, Quality assessment oriented, testing is to provide
the product evaluation and quality measurement
 Since 1988, Defect prevention-oriented, testing is to demonstrate
software compliance with design requirements, to discover and
prevent defects 。
 Formation of the
discipline

 Primary Stage (1957 ~ ~ 1971 testing is generally considered to be

post-test, don’t have effective testing methods.
 Development stage (1972 ～ ~1982), the first formal meeting of the
about software testing in 1972, promoted the development of
software testing
 Mature stage (since 1983 to now, international standards Std 829-
1983, became an independent discipline and major, and become an
important part of the software engineering.
 Forward thinking of
software testing
Dr. Bill Hetzel
 Software testing is to establish confidences for a program to
perform as expected
 Software testing is a series of activities to evaluate the
characteristics of a program or system or capability and to
determine whether expected results are achieved
 Testing is to verify whether the software meets user requirements,
that is, to verify that the software product can work properly
 Backward thinking of
software testing
Glenford J. Myers
 Testing is to prove that the program is wrong, not to
prove that the program is bug-free
 A good test case is one that finds undiscovered errors
 A successful test is a test that finds undetected errors
 Two Sides of Definition

Forward －
Verify that the software works normal

Evaluate a program or department of a system's

characteristics or abilities, and determine whether the
expected results are met Determine if expectations are
met

Run all software functions in the environment specified in

the design, until all have passed.
 Two Sides of Definition

Backward －
Assume the software is defected

Testing is for finding faults in the execution of a sequence

or system

Look for error-prone places and weaknesses of systems, try

to destruct the system, until no problem can be found
 Definition
IEEE ：
 Evaluate certain aspects of the system by operating
the system or component under specified conditions,
and observing or recording the results.
 Analyze a software item to find differences between
the existing items and the required conditions (i.e.
errors) and evaluate the characteristics of the
software item.
 ISO29119

 An activity in which a system or component is executed

under specified conditions, the results are observed or
recorded, and an evaluation is made of some aspect of
the system or component.
[ISO/IEC 24765, Systems & Software Engineering Vocabulary]

 Testing is comparing what the test item does with what it

is expected to do
 More complete definition

Software testing as a whole is consisted by Verification

and Validity activities

 “Verification” is to verify that the software has correctly

implemented the system functions and features
defined by the requirements of the product
specification
 “Validation” is an activity to confirm whether the
developed software satisfies the user's real needs
 Others

 Software testing is considered to be the activity of

conducting assessment of the potential risk of various
quality risks in a software system.
 The Economic Perspective of the Test It is to obtain the highest

software product quality at the lowest cost quantity 。

 Values

• Comprehensively assess product quality and

obtain comprehensive, objective information
about product quality
• Identify problems, supervise problem solving,
and improve product quality
• Continuous quality feedback and timely
disclosure of quality risks can help control
project risks and improve the quality of builds
• Defect analysis to obtain defect patterns that
contribute to defect prevention
1.4 test vs development
 1.5 Testing vs SQA
Software Quality Assurance (Software Quality
Assurance,, SQASQA) is to validate software through
planned reviews and audits of software products
Compliance with system engineering, through
coordination, review and tracking to obtain useful
information, resulting in analytical results to guide the
software process
 SQA

 Application of technical
methods
 Implementation of a formal
technical review
 Software test
 Standard implementation
 Modified Control
 Measurement
 Record
SQA vs testing
 Test vs. SQA

 SQA Instruct and supervise the planning and execution

of software testing, and supervise the testing work The
results are objective, accurate and effective and valid
and assist in the improvement of the testing process 。
 Software test one of the important means of SQA,
which provides the required data, and serves as an
objective basis for quality evaluation 。

 SQA is a management
 Testing is a technique.
1.6 Test Driven Development
 classifications
phases
Acceptance Test

System test

Integration test
Unit test

methods
functional test
Robustness Test White- Black-
performance test box test box test

Accessibility testing
Security test
reliability test

objectives
 History

Software testing didn’t evolve in a single day; it took time and

sweat to get it where it is today. Testing gurus like Hetzel and
Dave Gelprin divide testing into five significant eras:

Debugging-oriented era
1950’s
Demonstration-oriented era
1957-1978
Destruction-oriented era
1979-1982
Evaluation-oriented era
1983-1987
Prevention-oriented era
1988-2000
 History

Debugging-oriented era: This phase was during the

early 1950s, when there was no distinction between testing and
debugging. The focus was on fixing bugs. Developers used to
write code, and when faced with an error would analyze and
debug the issues. There was no concept of testing or testers.
(However, in 1957, Charles L Baker distinguished program testing
from debugging in his review of the book Digital Computer
Programming by Dan McCracken.)
 first actual case of bug
being found
The term "bug" was used in an account by computer pioneer Grace Hopper, who
publicized the cause of a malfunction in an early electromechanical computer. A typical
version of the story is:

In 1946, when Hopper was released from active duty, she joined the Harvard Faculty at
the Computation Laboratory where she continued her work on the Mark II and Mark III.
Operators traced an error in the Mark II to a moth trapped in a relay, coining the term
bug. This bug was carefully removed and taped to the log book. Stemming from the first
bug, today we call errors or glitches in a program a bug
 History

Demonstration-oriented era: From 1957 to 1978, the

distinction between debugging and testing was made and testing
was carried out as a separate activity. During this era, the major
goal of software testing was to make sure that software
requirements were satisfied. As an example, the requirement
might have been ‘We need a web application that displays a list of
10 products only’. Testers used to make sure that only 10 products
were displayed. This failed because of the probability that a
software’s function decreases as testing increases, i.e. the more
you test, the more likely you'll find a bug. The concept of negative
testing (or breaking the application) was not practiced in this era.
 History

Destruction-oriented era: From 1979 to 1982, the focus

was on breaking the code and finding the errors in it. It was
Glenford J. Myers who initially introduced the separation of
debugging from testing in 1979 although his attention was on
breakage testing. (‘A successful test case is one that detects an
as-yet-undiscovered error.’) It illustrated the software engineering
community’s desire to separate fundamental development
activities, such as debugging, from verification. As an example, a
tester would test software in such a way that it would break (e.g.
entering letters in a field that should only accept numbers). There
was no defect prevention approach during this phase. However,
the destruction-oriented approach also failed because software
would never get released because you could find one bug after
another. Also, fixing a bug could also lead to another bug.
 History

Evaluation-oriented era: From 1983 to 1987, the focus

was on evaluating and measuring the quality of software. Testing
improved the confidence index on how the software was working.
Testers tested until they reached an acceptable point, where the
number of bugs detected was reduced. This was mainly applicable
to large software.
 History

Prevention-oriented era: 1988 to 2000 saw a new

approach, with tests focusing on demonstrating that software met
its specification, detecting faults and preventing defects. Code was
divided into testable and non-testable. Testable code had fewer
bugs than code that was hard to test. In this era, identifying the
testing techniques was the key. The last decade of the 20th
Century also saw exploratory testing, where a tester explored and
deeply understood the software in an attempt to find more bugs.
 History

The early 2000s saw the rise of new concepts of testing like test-
driven development (TDD) and behavioural-driven development
(BDD). We’ll be highlighting these in upcoming articles.
 History

The year 2004 saw a major revolution in testing, with the advent of
automation testing tools like Selenium. Likewise, API testing using
tools like SOAP UI marked another turning point in the history of
testing. These will be examined in detail in upcoming blogs.
 History

Finally, the current era is moving towards testing using artificial

intelligence (AI) tools, and cross-browser testing using tools like
SauceLabs, Browserstack, etc.
 Career

The role of a software test engineer is to coordinate the process for analyzing
software programs. This process will involve creating and implementing testing
methods, recording the test results, and providing recommendations to improve
software programs based on the results. As a software test engineer, you can work
for a variety of companies. If you work for a government agency or contractor, you
may need to obtain a security clearance level.

Your job duties as a software test engineer may include collaborating with
organizational management to streamline operations, utilizing programming
languages to create testing scripts, and documenting software testing procedures.
You could be tasked with evaluating your organization's technical infrastructure for
security issues. You could also be tasked with working with team members to
develop software programs that meet business needs. As a software test engineer,
you may also be responsible for interfacing with end users to ascertain areas of
improvement, such as cost reduction solutions and automation solutions.
 Microsoft Software Test Engineer

Responsibilities
Design/leverage existing automation harness to automate
performance/large-scale test scenarios which involves integrating different
tools.
Develop tools/scripts to facilitate test execution, data collection and
processing from our 4G/5G cores as well as tools.
Process data and generate automated reports and develop alert system.
Run/automate Performance/system test scenarios using different tools to
test the performance/reliability of our 4G/5G wireless cores.
 Microsoft Software Test Engineer

Qualifications
Ten or more years of experience in Software development & Test or Test tool
development with proven track record of validating high quality products.
Excellent experience developing software testing harness/tools for Linux
environments.
Development experience in Python as well as any other scripting language
(iTCL, R, Expect)
Strong knowledge with testing embedded systems with focus on wireless cores
4G/5G is preferred (SGW, PGW, SMF, UPF, NRF)
Experience building dashboard visualizations, automated reporting and alerting.
Hands-on experience with cloud native architecture, Kubernetes, Docker,
Grafana, Prometheus is highly preferred.
Experience in using public cloud such as AZURE or AWS is preferred.
Experience in mining data from different sources.
Passion for problem solving and data exploration.
Strong interpersonal and communication skills.
Bachelor’s degree in electrical engineering / computer science or equivalent.
 Career
System test

junior Interface test

Function test
Software Test

Automated test
Engineer

intermediate Performance test

Test development

Test frame design

Test tools R&D

senior
Integration and optimization

Test platform
 Career
System test

junior Interface test

Software Test Engineer

Function test

Automated test

intermediate Performance test

Test development

Test frame design

Test tools R&D

senior
Integration and optimization

Test platform
 Reference
TextBook:
朱少民 . 软件测试方法和技术（第三版） . 北京：清华大学出版社 , 2014.
Reference:
Bill Hetzel.The Complete Guide to Software Testing,1993.
Ron Patton.TheSoftware Testing (2nd Edition). SAMS Publishing, 2006.
James A Whittaker, Jason Arbon, Jeff Carollo. How Google Tests Software.
2012
Rex Black. Managing the Testing Process (2nd Edition). Jihn Wiley & Sons.
Inc., 2002
Q&A
Thank you