You are on page 1of 32

Introduction to Software Testing

Contents
Why is Testing Necessary ............................................................................................. 2
Software Systems Context ......................................................................................... 2
Causes of Software Defects ....................................................................................... 3
When Do Defects Arise? ........................................................................................... 3
Role of Testing in Software Development, Maintenance and Operations ................ 5
Testing and Quality ................................................................................................... 5
How Much Testing is Enough? ................................................................................. 6
What is Testing? ............................................................................................................ 8
Testing as a Process ................................................................................................... 8
Different Testing Objectives...................................................................................... 8
Dynamic and Static Testing ....................................................................................... 9
Testing and Debugging .............................................................................................. 9
Seven Testing Principles.............................................................................................. 11
The Psychology of Testing .......................................................................................... 14
Mindsets of Developers and Testers ........................................................................ 14
Balance of Self-Testing and Independence of Testing ............................................ 14
Clear Objectives ...................................................................................................... 15
Communication Aspects of Testing......................................................................... 15
Software Development Models ................................................................................... 17
Waterfall Model ....................................................................................................... 17
V-model (Sequential Development Model)............................................................. 18
Iterative-Incremental Development Models ............................................................ 20
Testing within a Life Cycle Model .............................................................................. 23
Testing in Sequential Lifecycle Models .................................................................. 23
Testing in Iterative-Incremental Lifecycle Models ................................................. 23
Alignment in V-Model ............................................................................................ 24
Characteristic of Good Testing Regardless of Lifecycle Model.............................. 25
Metrics & Measurement .............................................................................................. 26
Code of Ethics ............................................................................................................. 28
Questions ..................................................................................................................... 29

Why is Testing Necessary
Learning Objectives:
Describe, with examples, the way in which a defect in software can cause harm
to a person, to the environment or to a company
Distinguish between the root cause of a defect and its effects
Give reasons why testing is necessary by giving examples
Describe why testing is part of quality assurance and give examples of how
testing contributes to higher quality
Explain and compare the terms error, defect, fault, failure, and the corresponding
terms mistake and bug, using example
Software Systems Context
Software systems are an integral part of life, from business applications (e.g., banking)
to consumer products (e.g., cars). However, most people have had an experience with
software that did not work as expected: an error on a bill, a delay when waiting for a
credit card to process and a website that did not load correctly are common examples
of problems that may happen because of software problems.
Some of the problems we encounter when using software are quite trivial, but others
can be costly and damaging. Software that does not work correctly can lead to many
problems, including loss of money, time or business reputation, and could even cause
injury or death. Incorrect software can harm:
 people (e.g. by causing an aircraft crash in which people die, or by causing a
hospital life support system to fail)
 companies (e.g. by causing incorrect billing, which results in the company losing
money)
 the environment (e.g. by releasing chemicals or radiation into the atmosphere)
The same software problem can have different effects in different systems, depending
on the context.
Some well-known examples of software failures:
 The first launch of the European Space Agency Ariane 5 rocket in June 1996
failed after 37½ seconds: a software error caused the rocket to deviate from its
vertical ascent, and the self-destruct capabilities were enacted before the then
unpredictable flight path resulted in a bigger problem
 In November 2005, information on the UK’s top 10 wanted criminals was
displayed on a website. The publication of this information was described in
newspapers and on morning radio and television and, as a result, the site was hit
more than 350,000 times. The performance of the website proved inadequate
under this load and the website had to be taken offline. The publicity created
performance peaks beyond the capacity of the website
 A software bug in the alarm system at a control room of the FirstEnergy
Corporation, located in Ohio, caused a widespread power outage that occurred
throughout parts of the Northeastern and Midwestern United States and the
2

Canadian province of Ontario on Thursday, August 14, 2003 (the so-called
Northeast blackout of 2003)
Causes of Software Defects
People make mistakes because they are fallible, but there are also many pressures that
make mistakes more likely. Our fallibility is compounded when we lack experience,
don’t have the right information, misunderstand, or if we are careless, tired or under
time pressure. Pressures such as deadlines, complexity of systems and organizations,
changing technologies, and/or many system interactions all bear down on designers of
systems and increase the likelihood of errors in specifications, in designs and in software
code. This is because our brains can only deal with a reasonable amount of complexity
or change—when asked to deal with more our brains may not process the information
we have correctly.
An error (mistake) during design of software can produce a defect (fault, bug) in the
program code, or in a document. If a defect in code is executed, the system may fail to
do what it should do (or do something it shouldn’t), causing a failure.

Defects in software specification, program code, systems or documents may result in
failures, but not all defects do so; some defects stay dormant in the code, and we may
never notice them. While failure is not always guaranteed, it is likely that errors in
specifications will lead to faulty components and faulty components will cause system
failure.
There are other reasons why systems fail. Failures can be caused by environmental
conditions as well, such as the presence of radiation, magnetism, electronic fields, and
pollution. These factors can affect the operation of hardware and firmware and lead to
system failure.
Failures may also arise because of human error in interacting with the software, perhaps
a wrong input value being entered or an output being misinterpreted. Also, failures may
be caused by someone deliberately trying to cause a failure in a system—malicious
damage.
When Do Defects Arise?
Consider the following figure. We can see when defects may arise in different cases.
Requirement 1 is implemented correctly. We understood the customer’s requirement,
designed correctly to meet that requirement, built correctly to meet the design, and
3

they will be hard to fix because design changes will be required. Unless we check against the requirements definition. If an error is made and the consequent defect is detected in the requirements at the specification 4 . when we make some mistakes and introduce defects. errors have been made at different stages. The cost of finding and fixing defects rises considerably across the life cycle. If we test the product meets its requirements and design. it does what it is supposed to do. Defects introduced during requirements and design make up close to half of the total number of defects. We built exactly what we were told to but unfortunately the designer made some mistakes so there are defects in the design.delivered the requirement with the right attributes. Functionally. The defects in Requirement 4 were introduced during the definition of the requirements. Probably. because we can see the product does not meet its design specification. these are easily spotted and corrected during testing. Requirement 2 is fine until the software is coded. so it is fast enough. and it also has the right non-functional attributes. Defects reported by the customer can be very costly. When we do notice them. The defects introduced in Requirement 3 are harder to deal with. With the other requirements. Requirements and design defects are not rare (cases 3 and 4). easy to understand and so on. the product has been designed and built to meet that flawed requirements definition. it will pass its tests but may be rejected by the user or customer. we will not spot those defects during testing.

the system was swamped by requests for access (non-functional failure). Testing can contribute to both avoidance and rectification. For example. usability. or the percentage of the software code that must be exercised. Testing can give confidence in the quality of the software if it finds few or no defects. Similarly. maintainability and portability. efficiency. Testing ensures that key requirements are examined before the system enters service and any defects are reported to the development team for rectification. if the defects found are corrected before the system is released for operational use. we need to begin testing as soon as we begin making errors—right at the beginning of the development process—and we need to continue testing until we are confident that there will be no serious system failures—right at the end of the development process. If. The same applies for construction. Role of Testing in Software Development. a defect is introduced in the requirement specification and it is not detected until the customer notices it. The specification can be corrected and re-issued. if an error is made. The higher the potential failure cost associated with the industry using the software. One of the definitions is that if a system meets its users’ requirements. then it will be much more expensive to fix. motor. Testing helps to measure the quality of software in terms of defects found. Software testing is neither complex nor difficult to implement. to be discussed in the following lectures). we must either avoid errors and faults or find them and rectify them. a poor test may uncover few defects and leave us with a false sense of 5 . the tests run. then it is relatively cheap to find and fix. because rework will be needed in the specification and design before changes can be made in construction. medical and pharmaceutical industries all have standards covering the testing of software. in the top 10 criminals case mentioned above. then the design can be corrected and re-issued with relatively little expense. Rigorous testing of systems and documentation can help to reduce the risk of problems occurring during operation and contribute to the quality of the software system. for both functional and non-functional software requirements and characteristics (such as reliability. These standards may specify what type of techniques we must use. yet it is a discipline that is seldom applied with anything approaching the necessary rigor to provide confidence in delivered software.stage. or industry-specific standards. Software testing may also be required to meet contractual or legal requirements. or even once the system has been implemented. the more likely it is that a standard for testing will exist. however. To influence errors with testing. The avionics. then it is of high quality. Of course. Maintenance and Operations To avoid failure. and therefore was not able to deliver its services to its users. and the system covered by the tests. Testing and Quality Quality is hard to define. and the consequent defect detected in the design at the design stage.

For instance. Error (mistake): A human action that produces an incorrect result. nor can it directly enhance quality. The next most important aspect is setting criteria. Not all software systems carry the same level of risk and not all problems have the same impact when they occur. A well-designed test will uncover defects if they are present and so. How Much Testing is Enough? A risk is something that has not happened yet and it may never happen. These uncertainties become more significant as the system complexity and the implications of failure increase. Testing cannot directly remove defects. e. By reporting defects it makes their removal possible and so contributes to the enhanced quality of the system. These two factors can be used to decide how much testing to do. E. may cause a failure of the component or system. usually known as completion criteria. so that time and all the other pressures do not confuse the outcome. the system may not work or the project may not be completed on time. it is a potential problem.. Risk is inherent in all software development.security. Do the most important tests (those that test the most important functional and non-functional aspects of the system as defined by the users) first so that at any time you can be certain that the tests that have been done are more important than the ones still to be done. 6 .g. Glossary: Defect (bug. we will rightly be more confident in the software and be able to assert that the overall level of risk of using the system has been reduced. an incorrect statement or data definition. Deciding how much testing is enough should take account of the level of risk (including technical. if such a test passes. that give an objective estimate of whether it is safe to stop testing. Testing is one component in the overall quality assurance activity that seeks to ensure that systems enter service without defects that can lead to serious failures. Testing should be integrated alongside development standards.g. and there is a level of quality that is acceptable for a given system. Every system is subject to risk of one kind or another. for the next development step or handover to customers. fault): A flaw in a component or system that can cause the component or system to fail to perform its required function. because the risk (and hence the probability of failure) is greater in the earlier case. The most important aspect of achieving an acceptable result from a finite and limited amount of testing is prioritization. A defect. and business risks). safety. Testing should provide sufficient information to stakeholders to make informed decisions about the release of the software or system being tested. training and defect analysis as one of the quality assurance activities. if encountered during execution. and project constraints such as time and budget. we would expect to test an automatic flight control system more than we would test a video game system.

service or result. Software: Computer programs. and possibly associated documentation and data pertaining to the operation of a computer system. Quality: The degree to which a component. usually expressed as impact and likelihood.Failure: Deviation of the component or system from its expected delivery. Risk: Factor that could result in future negative consequences. procedures. system or process meets specified requirements and/or user/customer needs and expectations. 7 .

reporting on the testing process and system under test.. Test activities exist before and after test execution. to gain confidence that it has met the requirements  In some cases the main objective of testing may be to assess the quality of the software (with no intention of fixing defects).g.What is Testing? Learning Objectives: Recall the common objectives of testing Provide examples for the objective of testing in different phases of the software life cycle Differentiate testing from debugging Testing as a Process A common perception of testing is that it only consists of running tests. Identifying defects has another benefit: by analyzing their causes. the main objective may be to confirm that the system works as expected. integration and system testing). we can improve the development processes and make fewer mistakes in future work  Gaining confidence about the level of quality  Providing information for decision-making  Preventing defects Different viewpoints in testing take different objectives into account:  In development testing (e. testing activities include planning and control. i. After test execution there is some work needed to record the results and check whether the tests are complete. Testing also includes reviewing documents (including source code) and conducting static analysis. designing and executing test cases. and fixing the defects improves the quality of the products.e. and finalizing or completing closure activities after a test phase has been completed. evaluating exit criteria. Before test execution there is some preparatory work to do to design the tests and set them up. Even more important is deciding what we are trying to achieve with the testing and setting clear objectives for each test. but not all of the testing activities. It helps us understand the risks associated with putting the software into operational use. choosing test conditions.. In general. checking results. the main objective may be to cause as many failures as possible so that defects in the software are identified and can be fixed  In acceptance testing. to give information to stakeholders of the risk of releasing the system at a given time  Maintenance testing often includes testing that no new defects have been introduced during development of the changes 8 . This is part of testing. component. executing the software. Different Testing Objectives Common testing objectives include:  Finding defects.

Glossary: Acceptance testing: Formal testing with respect to user needs. Debugging: The process of finding. Testing. so we speak of test execution in this context. the main objective may be to assess system characteristics such as reliability or availability Dynamic and Static Testing Static testing is the term used for testing where the code is not exercised. Failures often begin with a human mistake in a software specification. Testing and Debugging Debugging and testing are different kinds of activity. which can be effective in preventing defects. Code: Computer instructions and data definitions expressed in a programming language or in a form output by an assembler. is a systematic exploration of a component or system with the main aim of finding and reporting defects. Debugging does not give confidence that the component or system meets its requirements completely. usually in the development environment by developers Dynamic testing: Testing that involves the execution of the software of a component or system. 9 . Both dynamic testing and static testing can be used as a means for achieving similar objectives. During operational testing. Dynamic testing is the kind that exercises the program under test with some test data. analyze and remove the cause of bugs or defects in code. compiler or other translator.g. Testing makes a rigorous examination of the behavior of a component or system and reports all defects found for the development team to correct. on the other hand. Testing such documents is very important because errors are much cheaper to fix than defects or failures. customers or other authorized entity to determine whether or not to accept the system. by removing ambiguities and errors from specification documents. and will provide information that can be used to improve both the system being tested and the development and testing processes. and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user. Static testing involves techniques such as reviews. Subsequent re-testing by a tester ensures that any changes and corrections in the code are checked for their effect on other parts of the component or system. analyzing and removing the causes of failures in software. requirements. Debugging is the process that developers go through to identify. Maintenance testing: Testing the changes to an operational system or the impact of a changed environment to an operational system. e. The thought process and activities involved in designing tests early on can help to prevent defects from being introduced into code. Development testing: Formal or informal testing conducted during the implementation of a component or system.

Requirement: A condition or capability needed by a user to solve a problem or achieve an objective that must be met or possessed by a system or system component to satisfy a contract.e. e. specification. Test objective: A reason or purpose for designing and executing a test. Examples include management review. without execution of these artifacts. inspection.. standard. technical review.g. Review: An evaluation of a product or project status to ascertain discrepancies from planned results and to recommend improvements.g. reviews or static analysis. or other formally imposed document. requirements.. and walkthrough. Test case: A set of input values. preparation and evaluation of software products and related work products to determine that they satisfy specified requirements. both static and dynamic. such as to exercise a particular program path or to verify compliance with a specific requirement. informal review.. 10 . developed for a particular objective or test condition. hardware and software products installed at users’ or customers’ sites where the component or system under test will be used. design or code. Static testing: Testing of a software development artifact. to demonstrate that they are fit for purpose and to detect defects. concerned with planning.Operational testing: Testing conducted to evaluate a component or system in its operational environment. i. e. Testing: The process consisting of all lifecycle activities. expected results and execution postconditions. execution preconditions.

and can be difficult to do well.Seven Testing Principles Learning Objectives: Explain the seven principles in testing Testing is a very complex activity. How many tests would you need to do to completely test a one-digit numeric field? There are 10 possible valid numeric values. Instead of exhaustive testing. time pressure can increase dramatically. Although there may be other objectives. However. tests should be designed to find as many defects as possible. In the same way. and this is bad news if the only testing we are doing is after all the development has been completed. testing activities shall be started as early as possible in the software or system development life cycle. The earlier the 11 . Principle 1 – Testing shows the presence of defects Testing can show that defects are present. risk analysis and priorities should be used to focus testing efforts Let’s look at how much testing we’d need to do to be able to test exhaustively. usually the main purpose of testing is to find defects. systems have more than one input field with the fields being of varying sizes. Testing reduces the probability of undiscovered defects remaining in the software but. we cannot say “All swans are white”. we have shown “This code is not bug-free”. If we take an example where one screen has 15 input fields. Principle 2 – Exhaustive testing is impossible Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases. even if no defects are found. each having 5 possible values. and shall be focused on defined objectives As a proposed deployment date approaches. it is not a proof of correctness Regardless of how many white swans we see. 26 lower case. we have not shown “There are no bugs”. In practice. but cannot prove that there are no defects. regardless of how many tests we execute without finding a bug. Therefore. There is a real danger that testing will be squeezed. which is impossible to carry out in a project timescale. at least 6 special and punctuation characters as well as a blank value. As soon as we find a bug. So there would be at least 68 tests for this example of a one-digit field. and there are invalid values—26 uppercase alpha characters. as soon as we see one black swan we can say “Not all swans are white”. Principle 3 – Early testing To find defects early. then to test all of the valid input value combinations you would need 30 517 578 125 (515) tests. A number of testing principles have been suggested over the past 40 years and offer general guidelines common for all testing.

test cases 12 . we can test them. Testers do not have to wait until software is available to test. To overcome this “pesticide paradox”. E. In a large application. However. documents etc. and will focus on known “hot spots”. Principle 4 – Defect clustering Testing effort shall be focused proportionally to the expected and later observed defect density of modules. so the creation of acceptance tests can begin as soon as requirement documents are available. it is often a small number of modules that exhibit the majority of the problems.. some of which are:  System complexity  Volatile code  The effects of change upon change  Development staff experience  Development staff inexperience Testers will often use this information when making their risk assessment for planning the tests. eventually the same set of test cases will no longer find any new defects. Principle 5 – Pesticide paradox If the same tests are repeated over and over again. A small number of modules usually contains most of the defects discovered during prerelease testing. requirement documents are the basis for acceptance testing. Studies have shown what is known as the cost escalation model presented below in a simplified way. As soon as work products (requirements. Carrying out testing as early as possible leads to finding and fixing defects more cheaply and preventing defects from appearing at later stages of the project. This can be for a variety of reasons. but testers still need to search diligently for them. There may be fewer defects in the remaining code. or is responsible for most of the operational failures One phenomenon that many testers have observed is that defects tend to cluster. code.testing activity is started.) are ready.g. the longer the elapsed time available. it must be remembered that testing should not concentrate exclusively on these parts.

the more we need to invest in testing the software before it is implemented. and new and different tests need to be written to exercise different parts of the software or system to find potentially more defects Running the same set of tests continually will not continue to find new defects. for example. except when they are directly affected by the instability of the software. Using other techniques will find different defects. Principle 7 – Absence-of-errors fallacy Finding and fixing defects does not help if the system built is unusable and does not fulfill the users’ needs and expectations The fact that no defects are outstanding is not a good reason to ship the software. 13 . so continuing to use the same test set will result in decreasing effectiveness of the tests. where goods can be bought using credit/debit cards. Developers will soon know that the test team always tests the boundaries of conditions. The higher the possibility of losses. safety-critical software is tested differently from an e-commerce site Different testing is necessary in different circumstances. For example. Principle 6 – Testing is context dependent Testing is done differently in different contexts.need to be regularly reviewed and revised. so they will test these conditions before the software is delivered. A website where information can merely be viewed will be tested in a different way to an e-commerce site. Risk can be a large factor in determining the type of testing that is needed. Glossary: Exhaustive testing: A test approach in which the test suite comprises all combinations of input values and preconditions. The customers for software—the people and organizations who buy and use it to aid in their day-to-day tasks—are not interested in defects or numbers of defects. We need to test an air traffic control system with more rigor than an application for calculating the length of a mortgage. The people using software are more interested in the software supporting them in completing tasks efficiently and effectively. This does not make defects elsewhere in the code less likely.

However. and developers can efficiently find many defects in their own code.g. This approach is called independence of testing. Looking for failures in a system requires curiosity. professional pessimism. By this.. but the testing done by them cannot be assumed to be complete. we mean that. a critical eye. Several levels of independence can be defined as shown here from low to high:  Tests designed by the person(s) who wrote the software under test (low level of independence)  Tests designed by another person(s) (e. when we test or review a product. usability or performance test specialists)  Tests designed by a person(s) from a different organization or company (i. Separation of this responsibility to a tester is typically done to help focus effort and provide additional benefits.g. 14 . a replacement for familiarity.. attention to detail. good communication with development peers. and experience on which to base error guessing.The Psychology of Testing Learning Objectives: Recall the psychological factors that influence the success of testing Contrast the mindset of a tester and of a developer Mindsets of Developers and Testers The mindset to be used while testing and reviewing is different from that used while developing software. if we are building something we are working positively to solve problems in the design and to realize a product that meets some need. however. from the development team)  Tests designed by a person(s) from a different organizational group (e. With the right mindset developers are able to test their own code. an independent test team) or test specialists (e.e. Independence is not. such as an independent view by trained and professional testing resources. we are looking for defects in the product and thus are critical of it... The reason is that the creator of anything has a special relationship with the created object: flaws in the created object are rendered invisible to the creator.g. outsourcing or certification by an external body) A certain degree of independence (avoiding the author bias) often makes the tester more effective at finding defects and failures. Balance of Self-Testing and Independence of Testing Testing can be more effective if it is not undertaken by the individual who wrote the code.

Communication problems may occur. such as the customers. Because people and projects are driven by objectives. what those objectives are. As a result. there may be a clash of views about whether the testing has met its objectives. upset or depressed when someone points them out. Defects found and fixed during testing will save time and money later. Many of us find it challenging to actually enjoy criticism of our work.g. This applies to defects found during reviews as well as in testing. re-testing and regression testing. designers and developers can be avoided. depending on the objective.Clear Objectives Each organization and each project will have its own goals and objectives. If there are not clearly stated objectives and exit criteria for testing which all the stakeholders have agreed. The tester and test leader need good interpersonal skills to communicate factual information about defects. But if one stakeholder is less influential during the project but more influential at delivery. the development team and the managers of the organization. during the testing or after release. will have different viewpoints about quality and have their own objectives. there are several ways to improve communication and relationships between testers and others:  Start with collaboration rather than battles. We usually believe that we have done our best to produce work which is correct and complete. Explain that by knowing about the found defect now. E. arguments might arise. we can work round it or fix it so the delivered system is better for the customer 15 . and reduce risks. defect information can help them improve their skills. the stakeholder with the strongest views or the greatest influence over a group will define. bad feelings between the testers and the analysts. defects or failures are communicated in a constructive way. Testers need to use tact and diplomacy when raising defect reports. consciously or subconsciously. One manager may want the confirmation that the software works and that it is “good enough” if this is seen as a way of delivering as fast as possible. Different stakeholders. If errors. Another manager may want the testing to find as many defects as possible before the software is released. testing is often seen as a destructive activity. about whether “enough” testing has been done. particularly if testers are seen only as messengers of unwanted news about defects. Keep the focus on delivering a quality product. However. a tester might focus either on finding defects or on confirming that software works.. even though it is very constructive in the management of product risks. People tend to align their plans with these objectives. Communication Aspects of Testing Identifying failures during testing may be perceived as criticism against the product and against the author. For the author of the software or document. The aim is to work together rather than be confrontational. Defect reports need to be raised against the software. progress and risks in a constructive way. which will take longer to do and will require time for fixing. We all make mistakes and we sometimes get annoyed. not against the individual who made the mistake.

fact-focused way without criticizing the person who created it  Try to understand how the other person feels and why they react as they do  At the end of discussions. and to design tests specifically to expose them. which encourages the accomplishment of objective testing. Independence of testing: Separation of responsibilities. 16 . Communicate findings on the product in a neutral. non-personal. confirm that the other person has understood what you have said and vice versa Glossary: Error guessing: A test design technique where the experience of the tester is used to anticipate what defects might be present in the component or system under test as a result of errors made.

to creating the system. It has its place within a software development life cycle model and therefore the life cycle applied will largely determine how testing is organized.Software Development Models Learning Objectives: Explain the relationship between development. from capturing a customer requirement. The software life cycle models specify the various stages of the process and the order in which they are carried out. 17 . work-products such as code and associated documentation are generally created in a series of defined stages. It has a natural timeline where tasks are executed in a sequential fashion. Testing processes are related to others such as:  Requirements engineering & management  Project management  Configuration and change management  Software development  Software maintenance  Technical support  Production of technical documentation The development process adopted for a project will depend on the project aims and goals. There are numerous development life cycles that have been developed in order to achieve different required objectives. to delivering the system. Waterfall Model A development life cycle for a software product involves capturing the initial requirements from the customer. test activities and work products in the development life cycle. writing the code and testing the product. ready for release. We start at the top of the waterfall with a feasibility study and flow down through the various project tasks finishing with implementation into the live environment. Testing is not a stand-alone activity. by giving examples using project and product types Describe how testing is a part of any software development and maintenance activity Recognize the fact that software development models must be adapted to the context of project and product characteristics In software development. A simple development model known traditionally as the waterfall model is presented below. These stages are usually shown as steps within a software life cycle (software development life cycle). expanding on these to provide the detail required for code production.

a decision can be made on whether the product can be released into the live environment. The V-model provides guidance that testing needs to begin as early as possible in the life cycle. Verification helps to ensure that we are building the product in the right way  Validation changes the focus of work-product evaluation to evaluation against user needs. and testers need to work with developers and 18 . What is needed is a process that assures quality throughout the development life cycle. V-model (Sequential Development Model) The V-model was developed to address some of the problems experienced using the traditional waterfall approach. At every stage. There are a variety of activities that need to be performed before the end of the coding phase. it is unlikely that we can simply reject the parts of the system found to be defective. each activity is completed before moving on to the next one. In software development. The checks throughout the life cycle include verification and validation:  Verification checks that the work-product meets the requirements set out for it. Testing is carried out once the code has been fully developed. The product can be accepted or rejected at this point. however. Once this is completed. the testing at the end serves only as a quality check. In the waterfall model. and release the rest.This type of model is often referred to as a linear or sequential model. These activities should be carried out in parallel with development activities. This means ensuring that the behavior of the work-product matches the customer needs as defined for the project. Within this model. Validation helps to ensure that we are building the right product as far as the users are concerned Two types of development model facilitate early work-product evaluation. We will discuss them next in turn. a check should be made that the work-product for that stage meets its objectives.

Searching for defects and verifying the functioning of software components (e. For each work-product. The right-hand side focuses on the testing activities. acceptance testing would be planned for right at the start of the development. Although variants of the V-model exist. modules. Testing interfaces between components. programs. these are:  User requirements: capturing of user needs  System requirements: definition of functions required to meet user needs  Global design: technical design of functions identified in the system requirements  Detailed design: design of each module or unit to be built to meet required functionality The middle of the V-model shows that planning for testing should start with each workproduct.) that are separately testable  Testing against the technical specification takes place at the integration testing stage. providing successively more technical detail as the development progresses. interactions to different parts of a system such as an operating system. classes etc.business analysts so they can perform these activities and tasks and produce a set of test deliverables. corresponding to the four development levels. For instance. The left-hand side of the model focuses on elaborating the initial requirements. a common type of V-model uses four test levels. objects. using the requirement specification as an example. file system and hardware or interfaces between systems 19 .g. a testing activity is identified:  Testing against the program specification takes place at the unit (component) testing stage. In the model shown.

and business processes conducted to determine whether or not to accept the system This allows testing to be concentrated on the detail provided in each work-product. Iterative-Incremental Development Models Not all life cycles are sequential. As with the V-model. so that defects can be identified as early as possible in the life cycle. Testing against the functional specification takes place at the system testing stage. there are many variants of iterative life cycles. depending on the project and the software product. this approach to software development pushes validation of the system by the user representatives right to the end of the life cycle. or if they change. there may be component integration testing after component testing. in a series of increments. or builds. fewer or different levels of development and testing. If the customer needs were not captured accurately in the requirement specification. Instead. This type of development is often referred to as cyclical. code and test. design. a working version of the product is built. The main focus is verification against specified requirements  Testing against the requirement specification takes place at the acceptance testing stage. when the workproduct has been created. a V-model may have more. Each increment encompasses requirements definition. Other test levels can also be defined. we cycle through a number of smaller self-contained life cycle phases for the same project. with each increment adding new functionality. This is the main drawback of this model. Within these models. requirements. Validation testing with respect to user needs. then these issues may not be uncovered until the user testing is carried out. For example. the requirements do not need to be fully defined before coding can start. such as:  Hardware-software integration testing  Feature interaction testing  Customer Product integration testing Remembering that each stage must be completed before the next one can be started. In iterative and incremental models. 20 . In practice. and system integration testing after system testing.

This life cycle can give early market presence with critical functionality.The initial increment will contain the infrastructure required to support the initial build functionality. The requirements are prioritized and delivered in priority order in the appropriate increment. the working environment may be such that developers make any changes required. 21 . It is performed when the software or its environment is changed. rapid application development (RAD). and can reduce initial investment although it may cost more in the long run. without formally recording them. as a result of the changes made. Several drawbacks of this model can be pointed out. Subsequent increments will need testing for the new functionality. and integration testing of both new and existing parts. To mitigate this. Forms of iterative development include prototyping. The lack of formal documentation makes it difficult to test. They are empowered to request changes to the software in order to meet their needs. This approach could mean that changes cannot be traced back to the requirements or to the parts of the software that have changed. An iteration is a complete development loop resulting in a release (internal or external) of an executable product. traceability as the project progresses is reduced. and agile development models. Incremental development model: A development lifecycle where a project is broken into a series of increments. Iterative development model: A development lifecycle where a project is broken into a usually large number of iterations. Integration testing: Testing performed to expose defects in the interfaces and in the interactions between integrated components or systems. a robust process must be put in place at the start of the project to manage these changes. a subset of the final product under development. each subproject follows a ‘mini V-model’ with its own design. each of which delivers a portion of the functionality in the overall project requirements. regression testing of the existing functionality. coding and testing phases. The increment produced by an iteration may be tested at several levels as part of its development. can be simpler to manage because the workload is divided into smaller pieces. A proprietary methodology is called the rational unified process (RUP). thereby giving early feedback on the business value and fitness-for-use of the product. Regression testing is increasingly important on all iterations after the first one. Also early market presence will mean validation testing is carried out at each increment. In some (but not all) versions of this lifecycle model. A key feature of this type of development is the involvement of user representatives in the testing. Thus. Glossary: Component (unit) testing: The testing of individual software components. which grows from iteration to iteration to become the final product. Regression testing: Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software. In addition.

Validation: Confirmation by examination and through provision of objective evidence that the requirements for a specific intended use or application have been fulfilled. Note these phases may overlap or be performed iteratively. V-model: A framework to describe the software development lifecycle activities from requirements specification to maintenance. design phase. System testing: The process of testing an integrated system to verify that it meets specified requirements. requirements phase. test phase. implementation phase.Software lifecycle: The period of time that begins when a software product is conceived and ends when the software is no longer available for use. retirement phase. 22 . installation and checkout phase. The software lifecycle typically includes a concept phase. and sometimes. operation and maintenance phase. Verification: Confirmation by examination and through provision of objective evidence that specified requirements have been fulfilled. The V-model illustrates how testing activities can be integrated into each phase of the software development lifecycle.

making testing a preventive activity. This planning. but they require careful test management. effectively. This problem causes significant portions of the test schedule to be consumed by what is. Failure detection would start much later in the lifecycle. analysis. or incremental model. Instead. with no defect prevention. Testing in Sequential Lifecycle Models In a sequential lifecycle model. In such a situation. testing is an integral part of the various software development models such as sequential. retroactive unit testing  A third issue is the common failure to include all the testing activities described in the model. if the team follows a formal requirements process. Testing typically devolves to an ad hoc or at best reactive strategy. once system test execution began. Testing in Iterative-Incremental Lifecycle Models In the incremental lifecycle model. a key assumption is that the project team will define the requirements early in the project and then manage the changes to those requirements during the rest of the project. iterative. the test team would start planning and designing tests early in the project. That is. following an analysis of the requirements specification to identify test conditions. and limited value These sequential lifecycle issues are surmountable. and design work might identify defects in the requirements. A no-win situation occurs where the test manager is subjected to immense pressure to approve the release followed by howls of condemnation when the release proves bug-ridden in the field  The second issue is the common problem of development groups. the test team will receive requirements at the 23 . The use of sequential lifecycle models creates certain issues for testing that the test manager must manage:  The first and most infamous issue is that of schedule compression during testing at the end of the project. Very little preparation time is allowed. Testing must be appropriately integrated into the software lifecycle to succeed. the test team won’t receive a complete set of requirements early in the project. Using such a strategy in a sequential model. delivering unstable and often untestable systems to the test team. likewise pressured to achieve dates. proper alignment between the testing process and other processes in the lifecycle is critical for success. an independent test team in charge of the system test level can follow an analytical requirements-based test strategy. no clear coverage.Testing within a Life Cycle Model Learning Objectives: Recall characteristics of good testing that are applicable to any life cycle model As we already know.

in each increment after the first one. In both models discussed above. but the test manager must manage them carefully. at the end of the first iteration. implementation. you allow the activities associated with increments to overlap rather than requiring that each increment complete entirely before the next one starts. given the frequent and large changes to the code base—every increment being likely to introduce as much new and changed code as the previous increment—the risk of regression is high. once the test team starts to locate bugs. this can seem efficient at first. design. with varying degrees of success  The second issue is the common failure to plan for bugs and how to handle them. A lack of proper change management results in an inability for the test team to keep up with what the system is and what it should do. designers. This failure manifests itself when business analysts. to be able to regression test all the functions and capabilities provided in the previous increments. is the lack of rigor in and respect for testing These are all surmountable issues.beginning of each iteration. In such a case. an overbooked situation occurs for the business analysts. The availability of testable systems earlier in the lifecycle would seem to be a benefit to the test manager. testing activities in the fundamental testing process overlap and are concurrent with each other as well as with major activities in the software lifecycle. good change management and configuration management are critical for testing. In other words. designers. and it can be. it is very important that these functions and capabilities not be broken. in conjunction with the project management team. they can follow an analytical risk-based test strategy. However. which is particularly common in the agile world. the iterative lifecycle models create certain test issues for the test manager:  The first issue is the need. and developers who must address them  The final common issue. Rather than analyzing requirements at the outset of the project. and developers are assigned to work full-time on subsequent increments while testers are testing the current increment. and test results reporting are carried out according to the plan. Specific test designs and implementation will occur immediately before test execution. and test control continues until system test execution and closure are complete. This risk tends to lead to attempts to automate regression tests. We’ll further assume that we are talking about the system test level:  Test planning occurs concurrently with project planning. evaluation of exit criteria. However. and continues in repetitive. Defect detection starts very early in the project. Deviations from the plan are managed 24 . the best the test team can do is to identify and prioritize key quality risk areas. potentially reducing the preventive role of testing.e. execution.. Alignment in V-Model Let us use the V-model as an example to illustrate the concept of alignment between the testing process and other processes in the lifecycle. I. At the same time. Analysis. short cycles throughout the project. Because the most important functions and capabilities are typically provided in the earlier increments.

Characteristic of Good Testing Regardless of Lifecycle Model In any life cycle model. system and architectural (high-level) design specification. and component (lowlevel) design specification  Test implementation. Test analysis and design occurs concurrent with requirements specification. For each test level. there are several characteristics of good testing:  For every development activity there is a corresponding testing activity  Each test level has test objectives specific to that level  The analysis and design of tests for a given test level should begin during the corresponding development activity  Testers should be involved in reviewing documents as soon as drafts are available in the development life cycle 25 . and completes just before test execution begins  Test execution begins when the test entry criteria are all met. the test manager must perform this alignment during the test planning and/or project planning. including test environment implementation starts during system design. and for any selected combination of software lifecycle and test process. test execution starts when most entry criteria are met and any outstanding entry criteria are waived. generally with greater frequency and urgency as project deadlines approach  Test closure activities occur after test exit criteria are met and test execution is declared complete Such alignment of activities with each other and with the rest of the system lifecycle will not happen simply by accident. Test analysis starts immediately after or even concurrently with test planning. Test execution continues until system test exit criteria are met  Evaluation of test exit criteria and reporting of test results occurs throughout test execution. More realistically.

Well-established metrics and measures. So. Milestones and scope of testing. their evolution and their impact in terms of schedule. which I mentioned as the baselines previously. Planned schedule. a baseline must be defined.) should be applied throughout the software development life cycle (e. As you can see in the figure below. aligned with project goals and objectives. and their evolution over time 7. It also results in a lack of clearly perceived and communicated value. As part of test reporting. we can measure actual outcomes and trends against these expectations and adjust our approach as indicated. this project is in a lot of trouble at this point. etc. successful testing is usually impossible. and the weekly defect discovery rate remains very high. effectiveness. we establish expectations. The only good news. achievable targets. we can consistently explain to management various important aspects of the process. there is a significant backlog of defects. apparent in this graph. and efficiency for testing. and project. resources and tasks 3. product. average closure periods During test planning. Requirements. 26 . Defects (total found. and then progress tracked with relation to this baseline. planning. A lack of metrics and measurements leads to purely subjective assessments of quality and testing. etc. current backlog. As part of test control. To evaluate results.Metrics & Measurement A variety of metrics (numbers) and measures (trends. Possible aspects that can be subjected to a metric and tracked through measurement: 1. Without defined baselines. and their evolution over time 5. using objective. There are a very large number of total defects. graphs.). workload. Risks and mitigation actions. Let us consider a simple example of using the metrics and measurements. coverage. Planned and actual costs 6. coverage. and to disputes over the meaning of test results toward the end of the lifecycle. is that the weekly defect fix rate is about the same as the discovery rate. and their evolution over time 2. and their evolution over time 4. agreed-upon metrics with realistic. while the discovery rate isn’t declining.g. total fixed). enable us to report and track test and quality results to management in a consistent and coherent way. Workload and resource usage. at least the backlog is not growing.

Once these metrics have been defined. good testing Good reports based on metrics should be easily understood. A graph of a trend over time might be a useful way to present other information. Variations of data over time for a specific metric may reflect other information than the interpretation agreed upon in the metric definition phase  Reporting of metrics: the objective is to provide an immediate understanding of the information for management purpose.When working with testing metrics and measurement program. Not all types of graphical displays of metrics are equally useful. for components or systems. good testing reports based on metrics and measures will help management guide the project to success. not toward trivialities. not overly complex and certainly not ambiguous. their interpretation must be agreed upon by all stakeholders. pertinent. They should also draw the viewer’s attention toward what matters most. 27 . in order to avoid future discussions when metric values evolve. Metrics should be defined according to objectives for a process or task. such as the total number of defects reported and the total number of defects resolved since the start of testing Glossary: Measurement: The process of assigning a number or category to an entity to describe an attribute of that entity. three main areas are to be taken into account:  Definition of metrics: a useful. for individuals or teams  Tracking of metrics: reporting and merging metrics should be as automated as possible to reduce the time spent in producing the raw metrics values. Metric: A measurement scale and the method used for measurement. and concise set of quality and test metrics should be defined for a project. In that way. A table with a snapshot of data at a moment in time might be the right way to present such information as the coverage planned and achieved against certain critical quality risk areas. not confuse or misdirect them. Reporting should enlighten management and other stakeholders.

among other reasons to ensure that the information is not put to inappropriate use. Testers can have access to confidential and/or privileged information. employers and the wider public interest. Recognizing the ACM and IEEE code of ethics for engineers. A code of ethics is necessary. and act responsibly to the owner(s) of this information. consistent with the public interest  PRODUCT — Certified software testers shall ensure that the deliverables they provide (on the products and systems they test) meet the highest professional standards possible  JUDGMENT — Certified software testers shall maintain integrity and independence in their professional judgment  MANAGEMENT — Certified software test managers and leaders shall subscribe to and promote an ethical approach to the management of software testing  PROFESSION — Certified software testers shall advance the integrity and reputation of the profession consistent with the public interest  COLLEAGUES — Certified software testers shall be fair to and supportive of their colleagues.Code of Ethics Testers must adhere to a code of ethics: they are required to act in a professional manner. and they are to treat any information with care and attention. and promote cooperation with software developers  SELF — Certified software testers shall participate in lifelong learning regarding the practice of their profession and shall promote an ethical approach to the practice of the profession 28 . the ISTQB states the following code of ethics:  PUBLIC — Certified software testers shall act consistently with the public interest  CLIENT AND EMPLOYER — Certified software testers shall act in a manner that is in the best interests of their client and employer.

c) enable those responsible for software failures to be identified. concentrating upon individuals. b) give an indication of the software quality. d) show there are no problems remaining. I and I hinder. V. repairing the code and checking the fix is correct. 6. III and IV aid. c) When there are no remaining high priority defects outstanding. III and IV aid. IV and V aid. II. III. 2. 5. II and III aid. When is testing complete? a) When time and budget are exhausted. Try to understand how the other person feels. d) When every data combination has been exercised successfully. 4. II and V hinder. Each discussion is a battle to be won. c) I. Which of the following is correct? Debugging is: a) Testing/checking whether the software performs correctly. Which of the following are aids to good communication. 29 . Confirm the other person has understood what you have said and vice versa. IV. starting with the most independent first? a) Tests designed by the author. which may lead to an error.Questions 1. Emphasize the common goal of better quality. a) I. IV and V hinder. b) Checking that a previously reported defect has been corrected. The effect of testing is to: a) increase software quality. tests designed by another member of the development team. d) Checking that no unintended consequences have occurred as a result of a fix. 3. d) the result of a failure. Communicate personal feelings. b) When there is enough information for sponsors to make an informed decision about release. c) the result of an error or mistake. Which list of levels of tester independence is in the correct order. I and V hinder. A bug or defect is: a) a mistake made by a person. b) III. b) a run-time problem experienced by a user. tests designed by someone from a different company. d) II. and which hinder it? I. c) Identifying the cause of a defect.

d) It is not suitable for developing websites. Profession: Certified software testers shall advance the integrity and reputation of their industry consistent with the public interest. tests designed by someone from a different company. III. a) I and II. c) Changes to the system do not need to be formally recorded. tests designed by the author. tests designed by someone from a different department within the company. II. Which of the following is true about the V-model? a) It has the same steps as the waterfall model for software development. tests designed by the author. Profession: Certified software testers shall consider the wider public interest in their actions. The analysis and design of tests for a given test level should begin after the corresponding development activity. Each test level has the same test objectives. 30 . b) It is referred to as a cyclical model for software development. c) Tests designed by someone from a different company. Which of the following is true of iterative development? a) It uses fully defined specifications from the start. 8. d) It enables test planning to start as early as possible. 7. d) Public: Certified software testers shall consider the wider public interest in their actions. c) Public: Certified software testers shall consider the wider public interest in their actions. c) It enables the production of a working version of the system as early as possible. IV. b) It involves the users in the testing throughout. Testers should be involved in reviewing documents as soon as drafts are available in the development life cycle. b) Public: Certified software testers shall advance the integrity and reputation of the profession consistent with the public interest. Profession: Certified software testers shall participate in lifelong learning regarding the practice of their profession and shall promote an ethical approach to the practice of their profession. d) Tests designed by someone from a different department within the company. 10. Which statement correctly describes the public and profession aspects of the code of ethics? a) Public: Certified software testers shall act in the best interests of their client and employer (being consistent with the wider public interest). Which of the following statements are true? I.b) Tests designed by someone from a different department within the company. 9. tests designed by another member of the development team. For every development activity there is a corresponding testing activity. tests designed by someone from a different company. Profession: Certified software testers shall advance the integrity and reputation of their industry consistent with the public interest.

Is more expensive if found in requirements than functional design. a) b) c) d) 17. When what is visible to end-users is a deviation from the specific or expected behavior. b) A fault. Which one of the following test objectives might conflict with the proper tester mindset? a) Show that the system works before we ship it. test high risk areas. c) A failure. c) Reduce the overall level of product risk. Test objectives vary between projects and so must be stated in the test plan. 16. In prioritizing what to test. a) b) c) d) An exhaustive test would include: All combinations of input values and preconditions. d) I and IV. 12. All states and state transitions. e) A mistake. 15. this is called: a) An error.b) III and IV. 11. test whatever is easiest to test. Negative consequences that will occur. Increases as we move the product towards live use. d) Prevent defects through early involvement. Can never be determined. 13. a) b) c) d) e) The cost of fixing a fault: Is not important. All combinations of input values and output values. the most important objective is to: find as many faults as possible. Negative consequences that could occur. b) Find as many defects as possible. 14. obtain good test coverage. Negative consequences for the test object. d) A defect. Which of the following should not normally be an objective for a test? 31 . a) b) c) d) A risk relates to which of the following? Negative feedback to the tester. Decreases as we move the product towards live use. c) II and III. All pairs of input value and preconditions.

C. Which one of the following describes the major benefit of verification early in the life cycle? a) It allows the identification of changes in user requirements. B and D are false. To demonstrate that the software doesn’t work. b) Exhaustive testing is. B and C are false. d) C and D are true. 19. Which of the following statements are true? A. a) B and C are true. c) It is normally impossible to test all input / output combinations for a software system. D. Software testing is mainly needed to improve the quality of the developer’s work. 18. 32 . d) The purpose of testing is to demonstrate the absence of defects. c) A and C are true. 20. Which of the following statements best describes one of the seven key principles of software testing? a) Automated tests are better than manual tests for avoiding the Exhaustive Testing. To assess whether the software is ready for release. A and D are false. To prove that the software is correct. Software testing may be required to meet legal or contractual requirements. b) It facilitates timely set up of the test environment.a) b) c) d) To find faults in the software. c) It reduces defect multiplication. feasible for all software. Rigorous testing and fixing of defects found can help reduce the risk of problems occurring in an operational environment. Rigorous testing is sometimes used to prove that all failures have been found. B. with sufficient effort and tool support. A and B are false. b) A and D are true. d) It allows testers to become involved early in the project.