You are on page 1of 27

Real World Software Testing

www.sqtl.com

Part 1 Agenda
Software Testing
Importance

Challenges and Opportunity

How Important is Software Testing?


In March 1992, a man received a bill for his as yet
unused credit card stating that he owed $0.00. He
ignored it and threw it away. In April, he received
another and threw that one away too.
The following month, the credit card company sent
him a very nasty note stating they were going to
cancel his card if he didn't send them $0.00 by return
of post. He called them and talked to them; they said
it was a computer error and told him they'd take care
of it.
.
3

The Intel Pentium Bug


Enter the following in your PC calculator (4195835 / 3145727) * 3145727 4195835
If the answer is not zero, you have an old Intel Pentium CPU with the
floating-point division bug.

Discovered in 1994
Intels test engineers had found the problem before release still
management decided to release without fixing this
Once found (after release), Intel tried to diminish its severity by
press releases & statements
When pressured, Intel offered to replace the faulty chips, but only if
a user could prove that he was affected
Ultimately there was a public outcry. Intel had to spend $ 400
million to replace the faulty chips.

The eBay Disaster


In 1999 21 Hour
Outage
$ 5 million net loss
11% drop in share price
1.2 million visitors lost
bids daily
Happened as eBay was
preparing for an open
house for Wall Street
analysts.

Business Impact

WEB SITE

DAILY INTERNET
COMMERCE REVENUES

LOST REVENUE PER


HOUR OF DOWNTIME

www.techdata.com

$1,000,000

$8,280

www.amazon.com

$2,700,000

$22,500

www.dell.com

$10,000,000

$91,320

www.cisco.com

$20,000,000

$182,640

www.intel.com

$33,000,000

$274,980

Source: Forrester Research Inc.

Competition is Only a Click Away


9%

1%

13%

53%
24%
53% did not leave
24% returned, only after going to competitors site
13% did not return after completing session
9% left site, never returned
1% gave no answer

Source: Jupiter/NFO Consumer Survey

How Important is Software Testing?


Cost of Inadequate
Software Testing

Potential Cost
Reduction from
Feasible Improvements

Financial Services

$3.3 billion

$1.5 billion

Total U.S. Economy

$59.5 billion

$22.2 billion

* NIST Report: The Economic Impact of Inadequate Infrastructure for Software Testing, 2002.

Causes of Unplanned Application


Downtime
Operator
Errors

Hardware
Failures

20%
40%

40%

Application
Failures

Source:

Trends & Directions

To
ol
s

Complexity

Wireless

Client/Server

Web

e
c
i
v
r
e
S
s

Mainframe

Visibility

Source:

10

e-Complexity

11

Repetitive Process

Requirements

Build 1

Build 2

Build 3

Build n

Test
Cycle 1

Test
Cycle 2

Test
Cycle 3

Test
Cycle n

12

Significance in Terms of Effort?


Depending on the risk and complexity of the
application under test,
Testing comprises about 30-80% of total SDLC time and
sometimes even more
Testing is about 50% of the total application development
cost
Testing can account for 70% of costs during the application
life

(All numbers from Mercury Interactive)

13

Strategic Planning Assumptions


IT organizations will outsource 85% of application
testing by 2002
There will continue to be a shortage of professional
testers through 2005

Source:

14

Why software usually has faults?


Programming errors - programmers, like anyone else,
can make mistakes
Software is written by human beings
who know something, but not everything
who have skills, but arent perfect
who do make mistakes (errors)

15

Why software usually has faults?


Miscommunication or no communication - as to
specifics of what an application should or shouldn't
do (the application's requirements)

16

Why software usually has faults?

17

Why software usually has faults?

Faster Time to
Market

e
B
u
s
i
n
e
s
s

Higher
Quality

18

Why software usually has faults?


Under increasing pressure to deliver to strict
deadlines
no time to check but assumptions may be wrong
systems may be incomplete
scheduling of software projects is difficult at best, often
requiring a lot of guesswork. When deadlines loom and the
crunch comes, mistakes will be made

19

Why software usually has faults?


Software complexity
the complexity of current software applications can be
difficult to comprehend for anyone without experience in
modern-day software development
Windows-type interfaces, client-server and distributed
applications, data communications, enormous relational
databases, and sheer size of applications have all
contributed to the exponential growth in software/system
complexity

20

Why software usually has faults?

Complexity of today's IT systems

21

Why software usually has faults?


Changing requirements (whether documented or
undocumented)
The customer may not understand the effects of changes, or
may understand and request them anyway
Redesign, rescheduling of engineers, effects on other
projects, work already completed that may have to be redone
or thrown out, hardware requirements that may be affected,
etc
If there are many minor changes or any major changes,
known and unknown dependencies among parts of the
project are likely to interact and cause problems, and the
complexity of coordinating changes may result in errors

22

Why software usually has faults?


Changing requirements (whether documented or
undocumented)
Enthusiasm of engineering staff may be affected. In some
fast-changing business environments, continuously modified
requirements may be a fact of life
In this case, management must understand the resulting
risks, and QA and test engineers must adapt and plan for
continuous extensive testing to keep the inevitable bugs
from running out of control

23

Why software usually has faults?


Poorly documented code
It's tough to maintain and modify code that is badly written or
poorly documented; the result is bugs
In many organizations management provides no incentive for
programmers to document their code or write clear,
understandable, maintainable code. In fact, it's usually the
opposite: they get points mostly for quickly turning out code

24

Why software usually has faults?


Software development tools
visual tools, class libraries, compilers, scripting tools, etc.
often introduce their own bugs or are poorly documented,
resulting in added bugs

25

Software Testing.
Robots
Web

Vendors

Methodology

Performance
IP Spoofing
Architecture
White Box Testing
Load
Internet Time
Regression
Record & PlaybackPlanning
Virtual Users

Beta

Thin Clients
Black Box Testing
User Acceptance
Security
Source:

Browser

26

Questions?

27