Chapter 1 Principles of testing

Chapter 1
Principles of Testing
1.1 OVERVIEW
In this module you will get an overview of the most fundamental principles of testing.
Firstly, we point out that although you will come across different terminology throughout
your career, professional testers are all pretty much agreed on these basic ideas that we
describe in this module.
Secondly, we tae a loo at the need for proper testing loo at the cost of getting it wrong and
we show why e!haustive testing is neither possible not practical. "hat are errors and how do
they get into the software#
$hirdly, we describe a fundamental test process, base on industry standards, and underline
importance of planning% tests and determining e!pected results in advance of test e!ecution.
"e conclude with a loo at re&testing and regression testing' why it is so important to go that
e!tra mile and run a suite of tests again, (ust when you thought it was safe to hit the release
date.
1.2 OBJECTIVES
)fter completing this module you will*
+ ,nderstand basic testing terminology.
+ ,nderstand why testing is necessary.
+ -e able to define error, fault and failure.
+ )ppreciate why errors occur and how costly they can be.
+ ,nderstand that you cannot test everything and that testing is therefore a ris management
process.
+ ,nderstand the fundamental test process.
+ ,nderstand that developers and testers have different mindsets.
+ .earn how to communicate effectively with both developers and testers.
+ Find out why you cannot test your own wor.
+ ,nderstand the need for regression testing.
+ ,nderstand the importance of specifying your e!pected results in advance.
+ ,nderstand how and why tests should be prioriti/ed.
1
Chapter 1 Principles of testing
1.3 TESTI! TER"IO#O!$
$here is no generally accepted set of testing definitions used by the worldwide testing
community. $he -ritish Standard for Software Component $esting published in )ugust 1001
provides a new source of testing definitions. "hen involved in a testing pro(ect it is
important to ensure everyone understands terminology adopted' you may find different
terminology is used in your organi/ation.
Exercise
$he -ritish Standard for Software Component $esting is nown as -S___________
) useful glossary of terms used in software testing is called -S22222222222
)lthough it is a -ritish Standard published by the -SI 3-ritish Standards Institute%, the
Specialist Interest 4roup in Software $esting 3SI4IS$% developed it over a period of several
years.
ANSWERS: 7925-PART2, 7925-PART1
1.% W&$ IS TESTI! ECESS'R$(
$his section e!plains why testing is necessary and closely at the cost and conse5uences of
errors in computer software.
1.%.1 )efinitions Errors* +a,lts an- +ail,res
)n error is a human action that produces an incorrect result. ) fault is a manifestation of an
error in software. Faults are also nown collo5uially as defaults or bugs. ) fault, if
encountered, may cause a fault, which is a deviation of the software from its e!isting delivery
or service.
"e can illustrate these points with the true story 6ercury spacecraft. $he computer program
aboa spacecraft contained the following statement wri1 the F78$8)9 programming
language.
:7 1;; i < 1.1;
$he programmer=s intention was to e!ecute a succeeding statements up to line 1;; ten times
then creating a loop where the integer variable I was using the loop counter, starting 1 and
ending at 1;.
,nfortunately, what this code actually does is writing variable i do to decimal value 1.1 and it
does that once only. $herefore remaining code is e!ecuted once and not 1; times within the
loop. )s a result spacecraft went off course and mission was abort considerable cost>
$he correct synta! for what the programmer intended is
:7 1;; i <1,1;
?
Chapter 1 Principles of testing
Exercise
"hat do you thin was the error, fault and failure in this e!ample#
$he error is 2222222222
$he fault is 22222222222
$he failure is 2222222222
1.%.2 Relia.ilit/
8eliability is the probability that software will not cause the failure of a system for a
specified time under specified conditions. 6easures of reliability include 6$-F 3mean time
between failure%, 6$$F 3mean time to failure% as well as service level agreements and other
mechanisms.
1.%.3 Errors an- ho0 the/ occ,r
"hy do we mae errors that cause faults in computer software leading to potential failure of
our systems# "ell, firstly we are all prone to maing simple human errors. $his is an
unavoidable fact of life. @owever, this is compounded by the fact that we all operate under
real world pressures such as tight deadlines, budget restrictions, conflicting priorities and so
on.
1.%.% Cost of errors
$he cost of an error can vary from nothing at all to large amounts of money and even loss of
life. $he aborted 6ercury mission was obviously very costly but surely this is (ust an isolated
e!ample. 7r is it# $here are hundreds of stories about failures of computer systems that have
been attributed to errors in the software. ) few e!amples are shown below*
) nuclear reactor 0as sh,t -o0n because a single line of code was coded as A < B instead
of A<)-S 3B% i.e. the absolute value of B irrespective of whether B was positive or negative.
-lue Cross of "isconsin installed a new C?;;m claims processing system & it sent out CD;
million in unwarranted and -,plicate pa/1ents. )lso, when a data entry cler typed =none=
in the town field the system sent hundreds of checs to the non&e!istent town of =979E= in
"isconsin.
In 6ay 100?, Pepsi fan a promotion in the Philippines. It told customers they could win a
million pesos 3appro!. CF;,;;;% if they bought a bottle of Pepsi and found number GF0
stamped on the underside of the bottle cap. ,nfortunately, due to a software error, 1;;,;;;
bottle caps were produced with number GF0 instead of one, which was an e5uivalent of CF?
billion in pri/e money. It cost the company dearly as some people pursued their claims
through the courts and Pepsi pai- o,t 1illions of -ollars in co1pensation.
)nother story was printed in the 9ew Bor $imes on 11th February 100F. Chemical -an
G
Chapter 1 Principles of testing
managed to allow C1H million to be withdrawn incorrectly from 1;;,;;; accounts & a single
line error in the program caused every )$6 on their networ to process the transaction
twice.
1.%.2 0hat happene- on Octo.er 23th 4 25
th
1662(
$he #on-on '1.,lance Ser7ice 8#'S9 covers an area of (ust over D;; s5uare miles and is
the largest ambulance service in the world. It covers a resident population of some D.1
million, but its daytime population is larger, especially in central .ondon. Since 10IF South
"est $hames 8egional @ealth )uthority has managed it.
$he .)S carries over H;;; patients every day and receives between ?;;; and ?H;; calls
daily 3including between 1G;; and 1D;; emergency calls i.e. 000 calls%. In terms of resources
the .)S has some ?I;; staff, IH; ambulances and a small number of various other vehicles
including 1 helicopter. .)S mae almost ? million patient (ourneys each year. In 100?J100G
its budgeted income was KD0.I million.
7n the ?Dth and ?7th 7ctober 100? the new system failed, ambulances failed to turn up and
people lost their lives. )lthough no Coroners Court ever apportioned blame for any deaths
directly to the computer systems failure, it was by any standards a ma(or disaster and made
the main evening news bulletins on several occasions.
1.%.3 #on-on '1.,lance Ser7ice
In summary the problems were*
Computer )ided :ispatch & 1
$he system relied on near perfect information to propose optimum resource to be allocated to
an emergency. @owever, there were imperfections in the information and changes in
operational procedures made it difficult for staff to correct the system.
$his was not a problem when it went live at I am on ?Dth 7ctober 100? as the system load
was light' however as the number of emergency calls increased throughout the&day it became
increasingly difficult for staff to correct errors' this led to*
Poor, duplicated and delayed allocations.
-uild&up of e!ception messages and awaiting attention list.
Slow&down of system as messages and lists built up.
Increased number of callbacs and hence delays in telephone answering.
$he cost of these errors were ultimately that ambulances didn=t turn up and people lost their
lives although the official en5uiry report did not attribute any fatalities to the system
problems. $he costs in terms of loss of confidence in the computer system, industrial
relations and so on were probably also high.
1.%.5 E:ha,sti7e testing 0h/ not test e7er/thing(
F
Chapter 1 Principles of testing
It is now widely accepted that you cannot test everything. E!hausted testers you will find, but
e!haustive testing you will not. Complete testing is neither theoretically, nor practically
possible. Consider a 1;&character string that has ?1; possible input streams and
corresponding outputs. If you e!ecuted one test per microsecond it would tae appro!. F
times the age of the ,niverse to test this completely. For a survey of the methodology and
limitations of formal proofs of program correctness see L6anna I1M
1.%.; Testing an- ris<
@ow much testing would you be willing to perform if the ris of failure were negligible#
)lternatively, how much testing would you be willing to perform if a single defect could cost
you your life=s savings, or, even more significantly, your life# L@et/el 11M.
$he amount of testing performed depends on the riss involved. 8is must be used as the
basis for allocating the test time that is available and for selecting what to test and where to
place emphasis. ) priority must be assigned to each test.
$est 6anagers and Pro(ect 6anagers come up with different prioriti/ation schemes but ht
basic principle is that you must focus the testing effort on those areas of the system that are
liely to have the most defects. )nother ey principle is that you must e!ecute the most
important test first. 7therwise, if you run out of time, which is liely, you will not have
e!ercised the tests that give you the best paybac in terms of faults found.
1.%.6 Testing an- =,alit/
$esting identifies faults whose removal increases the software 5uality by increasing the
software=s potential reliability. $esting is the measurement of software 5uality. "e measure
how closely we have achieved 5uality by testing the relevant factors such as correctness,
reliability, usability, maintainability, reusability, testability etc.
1.%.1> Testing an- legal* contract,al* reg,lator/ or 1an-ator/ re=,ire1ents
7ther factors that may determine the testing performed may be legal, contractual
re5uirements, normally defined in industry specific standards or based on agreed best
practice 3or more realistically non&negligent practice%.
1.%.11 &o0 1,ch testing is eno,gh(
It is difficult to determine how much testing is enough. $esting is always a matter of (udging
riss against cost of e!tra testing effort. Planning test effort thoroughly before you begin, and
setting completion criteria will go some way towards ensuring the right amount of testing is
attempted. )ssigning priorities to tests will ensure that the most important tests have been
done should you run out of time.
H
Chapter 1 Principles of testing
1.2 +?)'"ET'# TEST PROCESS
1.2.1 Intro-,ction
$esting must be planned. $his is one of -ill @et/el=s D testing principles L@et/el 11 p?HM and
he says we are all agreed on this one. @owever, he points out that the problem is that most of
us do not discipline ourselves to act upon it. 4ood testing re5uires thining out an overall
approach, designing tests and establishing e!pected results= for each of the test cases we
choose.
Bou have seen already that we cannot test everything, we must mae a selection, and the
planning and care we e!pend on that selection accounts for much of the difference between
good and poor testers.
$he 5uality and effectiveness of software testing are primarily determined by the 5uality of
the test processes used LNit 0HM. $his is one of Ed Nit=s D essentials of software testing. $est
groups that operate within organi/ations that have an immature development process will feel
more pain than those that do not. @owever, the test group should strive to improve its own
internal testing processes. $his section of the course shows a fundamental test process, based
on the -SI0?H&? standard for software component testing.
$he fundamental test process comprises planning, specification, e!ecution, recording and
checing for completion. Bou will find organi/ations that have slightly different names for
each stage of the process and you may find some processes that have (ust F stages, where F O
H are combined, for e!ample. @owever, you will find that all good test processes adhere to
this fundamental structure.
1.2.2 Test process stages
See -SI0?H&? for diagram of test process. $est planning involves producing a document that
describes an overall approach and test ob(ectives noting any assumptions you have made and
stating any e!ceptions to your overall test strategy for your pro(ect. $est planning can be
applied at all levels. Completion or e!it criteria must be specified so that you now when
testing 3at any stage% is complete. 7ften a coverage target is set and used as test completion
criteria.
Test specification 3sometimes referred to as test design% involves designing test conditions
and test cases using recogni/ed test techni=,es identified at the planning stage. @ere it is
usual to produce a separate document or documents that fully describe the tests that you will
carry out. It is important to determine the e:pecte- res,lts prior to test e!ecution.
Test e:ec,tion involves actually running the specified test on a computer system either
manually or by using an automated test tool.
Test recor-ing involves eeping good records of the test activities that you have carried out.
Persions of the software you have tested and the test specifications are software you have
tested and the test specifications are recorded along with the actual outcomes of each test.
Chec<ing for test co1pletion involves looing at the previously specified test completion
D
Chapter 1 Principles of testing
criteria to see if they have been met. If not, some test may need to be rerun and in some
instances it may be appropriate to design some new test cases to meet a particular coverage
target.
9ote that -SI0?H&? does not specify the format of any test documentation. @owever, $he
IEEE standard, nown as 1?0, specifies in detail a standard for software test documentation.
-SI0?H&? shows a diagram of a suggested hierarchy of test documentation.
&O"E WOR@
Exercise
Putting aside management problems. 8ead through test documentation e!amples in -SI0?H&
? and answer following 5uestions*
What test techni=,es -oes co1ponent test strateg/ stip,late(
What percentage of -ecision co7erage is re=,ire-(
What sho,l- .e -one if errors are fo,n-(
The proAect co1ponent test plan is ,sef,l .eca,se the approach o,tline- allo0sB
a9 Strict a-herence to the co1ponent test strateg/
.9 "ore fa,lts to .e i-entifie- in the #O! co1ponents
c9 ' .asic 0or<ing s/ste1s to .e esta.lishe- as earl/ as possi.le
-9 Isolation of the co1ponents 0ithin the test strateg/
The co1ponent test plan 1,st consist of a single -oc,1ent( TR?EC+'#SE
The co1ponent test plan 1,st specif/ test co1pletion criteria( TR?EC+'#SE
Wh/ -oes co1ponent test plan specif/ 1>>D )C 0hereas strateg/ re=,ire- 6>D(
Which test case -eals 0ith nonEn,1eric inp,t(
#ist the e:pecte- o,tco1e an- the test con-ition
Wh/ -oes the CTP ha7e a--itionalCaltere- test cases(
What action has .een ta<en as a res,lt of the test report(
I
Chapter 1 Principles of testing
1.2.3 S,ccessf,l tests -etect fa,lts
)s the ob(ective of a test should be to detect faults, a successful test is one that does detect a
fault. $his is counter&intuitive, because faults delay progress' a successful test is one that
may cause delay. $he successful test reveals a fault which, if found later, may be many more
times costly to correct so in the long run, is a good thing.
1.2.% "eaning of co1pletion or e:it criteria
Completion or e!it criteria are used to determine when testing 3at any stage% is complete.
$hese criteria may be defined in terms of cost, time, faults found or coverage criteria.
1.2.2 Co7erage criteria
Coverage criteria are defined in terms of items that are e!ercised by test suites, such as
branches, user re5uirements, and most fre5uently used transactions etc.
1.3 T&E PS$C&O#O!$ O+ TESTI!
1.3.1 P,rpose
$he purpose of this section is to e!plore differences in perspective between tester and
developer 3buyer O builder% and e!plain some of the difficulties management and staff face
when woring together developing and testing computer software.
1.3.2 )ifferent 1in-sets
"e have already discussed that none of the primary purposes of testing is to find faults in
software i.e., it can be perceived as a destructive process. $he development process on the
other hand is a naturally creative one and e!perience shows that staff that wor in
development have different mindsets to that of testers.
"e would never argue that one group is intellectually superior to another, merely that they
view systems development from another perspective. ) developer is looing to build new and
e!citing software based on user=s re5uirements and really wants it to wor 3first time if
possible%. @e or she will wor long hours and is usually highly motivated and very
determined to do a good (ob.
) tester, however, is concerned that user really does get a system that does what they want, is
reliable and doesn=t do thing it shouldn=t. @e or she will also wor long hours looing for
faults in software but will often find the (ob frustrating as their destructive talents tae their
tool on the poor developers. )t this point, there is often much friction between developer and
tester. :eveloper wants to finish system but tester wants all faults in software fi!ed before
their wor is done.
1
Chapter 1 Principles of testing
In summary:
:evelopers*
)re perceived as very creative & they write code without which there would be no
system> .
)re often highly valued within an organi/ation.
)re sent on relevant industry training courses to gain recogni/ed 5ualifications.
)re rarely good communicators 3sorry guys%>
Can often speciali/e in (ust one or two sills 3e.g. P-, C++, Q)P), SR.%.
$esters*
)re perceived as destructive & only happy when they are finding faults>
)re often not valued within the organi/ation.
,sually do not have any industry recogni/ed 5ualifications, until now
,sually re5uire good communication sills, tac O diplomacy.
9ormally need to be multi&talented 3technical, testing, team sills%.
1.3.3 Co11,nication .C0 -e7eloper an- tester
It is vitally important that tester can e!plain and report fault to developer in professional
manner to ensure fault gets fi!ed. $ester must not antagoni/e developer. $act and diplomacy
are essential, even if you=ve been up all night trying to test the wretched software.
1.3.% &o0 not to approach
$ester* S@ey Fred. @ere=s a fault report )81?G. .oo at this code. "ho wrote this# "as it
you# "hy, you couldn=t program your way out of a paper bag. "e really want this fi!ed by H
o=cloc or else.S
"e were unable to print Fred=s reply because of the language> 9eedless to say Fred did not
fi! the fault as re5uested.
0
Chapter 1 Principles of testing
Exercise
Bour trainer will split you into small test teams. 7ne of you will be the test team leader. Bou
have found several faults in a program and the team leader must report these to the developer
3your trainer%. $he bacground is that your team has tested this program twice before and
their are still 5uite a lot of serious faults in the code. $here are also several spelling mistaes
and wrong colors on the screen layout. $he test team is getting a bit fed up. @owever, you
have to be as nice as possible to the developer.
1.3.3 Wh/ canFt 0e test o,r o0n 0or<(
$his seems to be a human problem in general not specifically related to software
development. "e find it difficult to spot errors in our own wor products. Some of the
reasons for this are*
"e mae assumptions
"e are emotionally attached to the product 3it=s our baby and there=s nothing
wrong with it%.
"e are so familiar with the product we cannot easily see the obvious faults.
"e=re humans.
"e see e!actly what we want to see.
"e have a vested interest in passing the product as o and not finding faults.
4enerally it is thought that ob(ective independent testing is more effective. $here are several
levels of independence as follows*
$est cases are designed by the person3s% writing the software.
$est cases are designed by another person3s%.
$est cases are designed by a person3s% from a different section.
$est cases are designed by a person3s% from a different organi/ation.
$est cases are not chosen by a person.
$he discussion of independence test groups and outsourcing is left to another section.
1;
Chapter 1 Principles of testing
1. 5 REETESTI! ') RE!RESSIO TESTI!
"e find and report a fault in .74 G, which is duly fi!ed by the developer and included in the
latest release which we now have available for testing. "hat should we do now#
E!amples of regression tests not carried out include*
$he day the phones stopped. .
.)S failure on Fth 9ovember 3perhaps%
)riane H failure.
"henever a fault is detected and fi!ed then the software should be re&tested to ensure that the
original fault has bee successfully removed. Bou should also consider testing for similar and
related faults. $his is made easier if your tests are designed to be repeatable, whether they are
manual or automated.
8egression testing attempts to verify that modifications have not caused unintended adverse
side effects in the unchanged software 3regression faults% and that the modified system still
meets re5uirements. It is performed whenever the software, or its environment, is changed.
6ost companies will build up a regression test suite or regression test pac over time and
will add new tests, delete unwanted test and maintain tests as the system evolves. "hen a
ma(or software modification is made then the entire regression pac is liely to be run 3albeit
with some modification%. For minor planned changes or emergency fi!es then during the test
planning phase the test manager must be selective and identify how many of the regression
tests should be attempted. In order to react 5uicly to an emergency fi! the test manager may
create a subset of regression test pac for immediate e!ecution in such situations.
8egression tests are often good candidates for automation provided you have designed and
developed automated scripts properly 3see automation section%.
In order to have an effective regression test suite then good configuration management of
your test assets is desirable if not essential. Bou must have version control of your test
Documentation (test plans, scripts etc.% as well as your test data and baseline databases. )n
inventory of your test environment 3hardware configuration, operating system version etc.% is
also necessary.
11
Chapter 1 Principles of testing
1.; EGPECTE) RES?#TS
$he specification of e!pected results in advance of test e!ecution is perhaps one of the most
fundamental principles of testing computer software. If this step is omitted then human
subconscious desire for tests to pass will be overwhelming and tester may perhaps interpret a
plausible, yet erroneous result, as correct outcome. .
)s you will see when designing test using blac bo! and white bo! techni5ues there is ample
room within the test specification to write down you e!pected results and therefore no real
e!cuse for not doing it. If you are unable to determine e!pected results for a particular test
that you had in mind then it its not a good test as you will not be able to 3a% determine
whether it has passed or not and 3b% you will never be able to repeat it.
Even with a 5uic and dirty ad&hoc test it is advisable to write down beforehand what you
e!pect to happen. $his may all sound pretty obvious but many test efforts have floundered by
ignoring this basic principle.
S $he ma(or difference between a thing that might go wrong and a thing that cannot possibly
go wrong is that when a thing that cannot possibly go wrong does go wrong it usually turns
out to be impossible to get at or repair.S
--Douglas Adams
1?
Chapter 1 Principles of testing
SO"E TESTI! TER"IO#O!$
+a,lts & a mistae in the code that causes the software to not behave as e!pected (causes)
+ail,res & the act of a product not behaving as e!pected & the manifestation of a fault
(symptoms)
Vali-ation & establishing the correspondence between the software and its specification &
"are e !uilding t"e rig"t product#"
Test care & the collection of inputs, predicted results and e!ecution conditions for a single
test
'-Eli.Ca-Ehoc test care & a test e!ecuted without prior planning & especially if the e!pected
behaviors is not nown prior to running the test.
PassCfail criteria & decision rules used to determine whether a product passes or fails a given
test
Coinci-ental correctness & when behavior appears to be what is e!pected, but it is (ust a
coincidence
Test s,ite & a collection of test cases necessary to Sade5uatelyS test a product
Test plan & a document describing the scope, approach, resources and schedule of intended
testing activity & identifies features to be tested, the testing tass, who will do each tas, and
any riss re5uiring contingency planning.
Oracle & a procedure, process or magical phenomenon that can determine if the actual
behavior matches the e!pected behavior
Inci-ent & when a test produces an une!pected outcome, & further effort is necessary to
classify the incident as a software error, a design error, a specification error, a testing error,
etc.
B,g report & a method of transmitting the occurrence of a discrepancy between actual and
e!pected output to someone who cares for Sfollow&upS & also nown as discrepancy report,
defect report, problem report, etc.
Wor<Earo,n- & a procedure by which an error in the product can be Sby&passedS and the
desired function achieved.
1G
Chapter 1 Principles of testing
$"y $rite %rograms#
Concept* If you want your computer to do e!actly what you want it to do, you must write a
program.
) computer does nothing on its own. In fact, a computer is a dumb machine with no
intelligence whatsoever. :espite what you might read in science fiction stories, a computer
does nothing more than blindly follow instructions supplied by a programmer. Computers
cannot thin.
:efinition* ) program is a set of instructions that tells the computer e!actly what to do.
"hen someone buys a computer today, the computer sits on the des doing nothing until he
loads a program into the computer=s internal memory and starts running the program. Qust as
a PC8 does not record shows on its own without being programmed to do so, a computer
re5uires detailed instructions found only in programs.
Suppose that you own rental properties and want to use your computer to trac your tenant
records. Bour computer will not help you out in any way until you load and run a rental
property program. "here do you find such a program# $here are two ways to obtain
programs for computers. Bou can
31% -uy one and hope that the program does e!actly what you want it to do.
3?% "rite your own program.
It=s much easier and faster to buy a program that you need. $housands of programs are on the
maret today. In fact, there are so many programs out there that you might not see the need
for writing your own programs.
If you can find a program that does e!actly what you want, you are ahead of the game. If you
find a program that meets your e!act re5uirements, you should buy that program because
purchasing a program is often less e!pensive and much 5uicer than writing the same
program yourself or hiring programmers to write it for you.
$hin about this for a moment, though* If there are so many programs sold today that do
virtually everything, why are programming languages such as Pisual -asic continuing to
brea previous sales records each year# $he answer is simple* People buy a computer so that
the computer will do (obs that they need done. Firms cannot adapt their business to a
computer program. $hey must find programs, or write their own programs, so that the
computer processes information according to the business procedures already in place. $he
only way to ensure that a program e!actly fits the needs of a firm is for the firm to develop its
own programs.
1F
Chapter 1 Principles of testing
-usiness people are not the only ones who need custom&designed programs. 9o two people
manage their finances e!actly the same way' no two scientists need computers for e!actly the
same inds of computations' and no two graphic artists need the same inds of computer
drawing tools. )lthough people buy spreadsheets and word processors for their general&
purpose computing needs, many people re5uire speciali/ed programs for specific (obs.
$he art of programming computers is rewarding not only from a re5uirements standpoint, but
also on a more personal level. Programming computers is fun> Qust as a sculptor loos on a
finished wor of clay, programmers are often proud of the programs that they write. -y the
time you finish this boo, you will have written programs that were not available before you
wrote them. "hen you want your computer to do something specific and you cannot find a
program that does the (ob e!actly the way you want, you will be able to design and write the
program yourself.
Some Programs are Changeable* $here is a third method for getting e!actly the program that
you need if you want to computeri/e your company=s accounting records. )ccounting
software firms often sell not only accounting programs but also the source code for those
programs. $he source code is a listing of the program=s instructions. -y having access to the
source code, you can tae what the software company wrote and modify the behavior of the
program to suit your own re5uirements.
-y starting with a functional program instead of starting from scratch, you save programming
time and money. Sadly, most non&accounting software firms do not supply the source code.
6ost programs sold today have been compiled. )fter compiling, the source code is translated
into a loced&in e!ecutable program. $he bottom line is that you cannot easily change the
behavior of compiled programs. For most programs, therefore, you have the choice of buying
them or writing them yourself from scratch.
)efinition* Code is another name for program.
&e'ie( 9o single program pleases everyone. "hen a company sells a program, it must be
general enough to please most purchasers. Some people need programs to behave in a
specific manner in order to fulfill a specific need. $hey must resort to writing their own
programs. .ucily, Pisual -asic taes a lot of the pain out of writing programs.
1H
Chapter 1 Principles of testing
' Brief &istor/ of Te:t,al Progra11ing
Concept( Computers cannot understand (ust any language. Bou must learn a language that
your computer nows before you can write programs for your computer.
)efinitionB )n application is yet another name for program.
6any people use computers all day long for word processing, database storage, and
spreadsheet analysis, without reali/ing what is going on behind the scenes. Bou must always
eep in mind that computers cannot thin. Bour computer does not now how to be a word
processor. If you want your computer to do word processing, you must supply detailed
instructions in the form of a program. 7nly by following the detailed instructions of a word
processor program that you load can your computer perform word processing.
It would be nice if writing a program is as easy as telling the computer what you want done.
6any people can handle ambiguous instructions, but computers are not smart enough to
understand vague re5uirements. Computers can only follow orders given to them, and you
must supply those orders in the form of a program. $herefore, you must supply the programs
that you write. "riting programs, especially comple! programs, taes time and several
procedural steps. Pisual -asic speeds the process of creating programs, but even with Pisual
-asic some programs tae time to write and perfect.
)efinition* ) bug is a program error.
$hese are the typical steps that most programmers go through when writing programs. First,
you have an idea for a program. 9e!t, you use a program&development system, such as
Pisual -asic, to write the program. Errors, or bugs, often appear in programs because of the
details needed for even the simplest of programs. $herefore, you must test the program
thoroughly and fi! the errors. Fi!ing errors is called debugging. 7nce all the errors are out of
the program, you have a finished application.
PRO!R'""I!
) computer is an information&processing machine. LE!ecutes given commands.
,ses memory.
Information < data. $here are various types of data e.g. numerical, te!t, graphics or
pictures, sound signals etc.
) computer program is a se5uence or set of instructions in a programming language.
)ll data and instructions for a program are stored in the computer=s memory in coded
form. $his coded form is similar to the mores code 3&;&&&;&;;&%. $he coding or
translation is now done automatically. I.e. by commercial or computer programming.
Programs will be written in the programming languages of which there are many.
.ie Pascal, Cobalt, 3used for records%, Pisual -asic, etc.
1D
Chapter 1 Principles of testing
For the purpose of this module, Pisual -asic 3P-% programming language will be
used. @ence the P- Integrated :evelopment Environment 3I:E% will do the
translation into coded form, which is a software program.
.744I94 I9$7 PIS,). -)SIC
)*+ 3adds topics%, enter 7pen, enter
"hen in the form use & ,a!el 1. Caption < "-ello Class of .///"
$o delete an ob(ect from the form Select Edit menu.
E&&0&+ 12 %&03&A44123
$here are three common errors
1 Syntax errors( are due to structure or grammar of the language 3rules% applied, e.g.
more or less spacing, full stops, commas etc.
2 Routine errors( are due to non&e!isting situations lie 1 divided by ;, is impossibility.
$hese are errors where the program has instructed the computer to perform an impossible
operation e.g. as shown above.
3 Loi!a" errors( are those in the meaning of the program.
$o save, always S'VE +OR" 'S, followed by S'VE PROJECT 'S.
1I
Chapter 1 Principles of testing
The Cost of B,gs
Programming, and testing, to its use by the public, there=s the potential for bugs to be found.
$he Figure below shows how the cost of fi!ing these grows over time.
Cost
Specification :esign Code
$ime "hen -ug Is Found
5"e cost to fix !ugs increased dramatically o'er time6
$he cost are logarithmic & that is, they increase tenfold as time increases. ) bug found and
fi!ed during the early stages when the specification is being written might cost ne!t to
nothing or 1; pence in our e!ample. $he same bug, if not found until the software is coded
and tested, might cost K1 to K1;. If a customer finds it, the cost could easily top K1;;.
W&$ )OES SO+TW'RE &'VE B?!S(
6iscommunication or no communication & as to specifics of what an application
should or shouldn=t do 3the application=s re5uirements%.
Software comple!ity & the comple!ity of current software applications can be difficult
to comprehend for anyone without e!perience in modern&day software development.
"indows&type interfaces, client&server and distributed applications, data
communications, enormous relational databases, and sheer si/e of applications have
all contributed to the e!ponential growth in softwareJsystem comple!ity. )nd the use
of ob(ect&oriented techni5ues can complicate instead of simplify a pro(ect unless it is
well engineered.
11
Chapter 1 Principles of testing
Programming errors & programmers, lie anyone else, can mae mistaes.
Changing re5uirements & the customer may not understand the effects of changes, or
may understand and re5uest them anyway & redesign, rescheduling of engineers,
effects on other pro(ects, wor already completed that may have to be redone or
thrown out, hardware re5uirements that may be affected, etc. If there are many minor
changes or any ma(or changes, nown and unnown dependencies among parts of the
pro(ect are liely to interact and cause problems, and the comple!ity of eeping trac
of changes may result in errors. Enthusiasm of engineering staff may be affected. In
some fast&changing business environments, continuously modified re5uirements may
be a fact of life. In this case, management must understand the resulting riss, and R)
and test engineers must adapt and plan for continuous e!tensive testing to eep the
inevitable bugs from running out of control
$ime pressures & scheduling of software pro(ects is difficult at best, often re5uiring a
lot of guesswor. "hen deadlines loom and the crunch comes, mistaes will be made.
Egos & people prefer to say things lie* =no problem=, =piece of cae=, =I can whip that
out in a few hours= =it should be easy to update that old code=
Instead of* =that adds a lot of comple!ity and we could end up maing a lot of
mistaes= or Twe have no idea if we can do that' we=ll wing it=, =I can=t estimate how
long it will tae, until I tae a close loo at it=, =we can=t figure out what that old
spaghetti code did in the first place=
If there are too many unrealistic =no problems=, the result is bugs.
Poorly documented code & it=s tough to maintain and modify code that is badly written
or poorly documented' the result is bugs. In many organi/ations management
provides no incentive for programmers to document their code or write clear,
understandable code. In fact, it=s usually the opposite* they get points mostly for
5uicly turning out code, and there=s (ob security if nobody else can understand it 3=if
it was hard to write, it should be hard to read=%.
Software development tools & visual tools, class libraries, compilers, scripting tools,
etc. often introduce their own bugs or are poorly documented, resulting in added
bugs.
10
Chapter 1 Principles of testing
E:a1ple E "icrosoft Wor-
$ES$ 97 I9P,$S EAPEC$E: 8ES,.$S
I Clic on File hold down and 7pen dialog appears
:rag to 7pen
? 8ight clic on paragraph 6enu options appear & font, paragraph etc.
Cut Copy Paste are grayed out
G 8ight clic within a table 1? menu options appear related to table
)ctions e.g. insert row, delete row
F Spelling Spelling checing starts and displays first
Error but chec grammar off
3:epends on option%
H File Print :isplays printer dialog
D Clic on print icon "ill attempt to print to default printer
?;
Chapter 1 Principles of testing
)ri7ing #icense E:ercise
Consider the following simple program*
IF ageU 1D and age V<DH $@E9
Issue drivers .icense
IF age V?D $@E9
Set Insurance Premium to @I4@
E.SE
Set Insurance Premium to S$)9:)8: E9: IF
E.SE
Error message :.77l &S Sorry, unable to issue drivers licenseS
E9: IF
TEST O IP?TS EGPECTE) RES?#TS
1 1;
? 1D
G 1I
F ?H
H ?D
D DH
I I;
?1
Chapter 1 Principles of testing
'T" E:ercise
Some simplified rules for an ) $6*
a% Card must be valid for this ban
b% Correct PI9 must be entered
c% "ithdrawal must not tae account overdrawn unless there is an overdraft agreed
d% 9otes available are K 1 ; and K?; only
TEST O IP?TS EGPECTE) RES?#TS
1 Palid Card
Incorrect PI9
KH; re5uested =S
? Palid Card
Correct PI9
K?;; re5uested
-alance KH;
9o overdraft 7i-
G Palid Card
Correct PI9
KH; re5uested
-alance KH;
9o overdraft J
F Palid Card
Correct PI9
KH; re5uested
-alance KF;;
"ithdrawn K1D; earlier 1&
??
Chapter 1 Principles of testing
Telephones E:ercise
TEST O IP?TS EGPECTE) RES?#TS
1 Pic ,: receiver
? :ial 1;;
G :ia11F? or 10?
F :ial 000
H :ial -usy 9umber
D Press H )fter @earing
Engaged tone
I :ia11FI1
1 :ia11HI1
?G
Chapter 1 Principles of testing
1. 6 PrioritiHation of Tests
Since we have discussed already that there is never enough time to do all of the testing that
you would lie an obvious approach is to prioriti/e the tests $his will then mean that
whenever you stop testing you will have done the best testing possible in the time available.
$he test manager should identify the criteria to hi used when prioriti/ing tests. $hese will
include but are not limited to*
Severity.
Probability.
Pisibility of failure.
-usiness priority assigned to the re5uirement.
Customer wants.
@ow much change the function has undergone.
@ow error prone is this module.
@ow critical is it to the business.
@ow technically comple! is it.
@ow difficult is it to test.
@ow costly is it to test.
Neep the test priority with the test at all times. $his2 will prevent you from developing and
designing lo2 priority tests that never get run. ,se a scheme of high, medium, low or a
numeric scheme or 1,?, G, F but try not to have more than about F categories.
O TEST C'SE )ESCRIPTIO PRI
1 ) manual bloc on a ban is introduced 3using the @7.: function%
Causing payments to be moved between priority classes by the
Scheduler
? Failure of the primary scheduler process
G Internal authentication failures
F 9ormal close of day
H 9ormal system start up
D 7ne of each type of re5uest to the scheduler
I Payments are released correctly from the ,SE8 class when limits
E!ceeded but funds received from another ban causing position to
Improve
1 Payments can be routed to the scheduler 5ueue from several sources
with the correct class and user defined priority 3including payments
that have been referred or repaired%
?F
Chapter 1 Principles of testing
0 Payments cancelled manually be central control
1; 8egression testing of e!isting telecommunications lins
11 8egression testing of recompiled interface with mainframe system
1? 8emote ban changes their limit on us causing payments to be
8eleased from the ,SE8 class 3if held due to e!ceeding limit%
1G 8emote ban issues temporary raise
1F 8oute all payments to the scheduler with parameters set for
immediate release
1H Special processing when internal cut&off time reached
1D SuspendJactivate limits
1I SuspendJactivate scheduler
?H
Chapter 1 Principles of testing
1.1> S,11ar/
Bou should now be familiar with some of the fundamental principles of testing
and you will recogni/e much of the terminology. $he horror stories that you will
have heard are not (ust sensationalist to motivate the course' they are true stories
and there are 3unfortunately% many more stories about the costs of not testing
systems properly.
In particular you can now*
8ecogni/e and use basic testing terminology.
,nderstand why testing is necessary.
:efine error, fault and failure.
E!plain why errors occur.
4ive e!amples of the cost of errors.
,nderstand why e!haustive testing is unachievable.
)ppreciate why testing is a ris management process.
,nderstand the fundamental test process.
,nderstand the difference between the mindsets of developers and testers.
,nderstand why you cannot test your own wor.
,nderstand the need for regression testing.
,nderstand the importance of specifying your e!pected result in advance.
.ist some criteria for prioriti/ing your tests.
?D