Subraya - Software Quality & Testing

MSIT 32 Software Quality and Testing
SOFTWARE QUALITY & TESTING

(MSIT - 32)
: Contributing Author :
Dr. B.N. Subraya

Infosys Technologies Ltd., Mysore
PDF created with pdfFactory Pro trial version www.pdffactory.com
Contents
Chapter 1 INTRODUCTION TO SOFTWARE TESTING 1.1 1.2 1.3 1.4 1.5 1.6 1.7 Learning Objectives.......................................................................... Introduction...................................................................................... What is Testing?............................................................................... Approaches to Testing....................................................................... Importance of Testing....................................................................... Hurdles in Testing............................................................................. Testing Fundamentals........................................................................ 1 1 1 3 5 6 6 7
Chapter 2 SOFTWARE QUALITY ASSURANCE 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 Learning Objectives.......................................................................... Introduction...................................................................................... Quality Concepts............................................................................... Quality of design............................................................................... Quality of Conformance.................................................................... Quality Control (QC)......................................................................... Quality Assurance (QA).................................................................... Software Quality ASSURANCE (SQA)............................................. Formal Technical Reviews (FTR)....................................................... Statistical Quality Assurance.............................................................. Software Reliability........................................................................... The SQA Plan.................................................................................. 10 10 10 11 12 12 13 13 14 21 27 30 31
4
Chapter 3 PROGRAM INSPECTIONS, WALKTHROUGHS AND REVIEWS QUALITY ASSURANCE 3.1 3.2 3.3 3.4 3.5 3.6 Learning Objectives.......................................................................... Introduction...................................................................................... Inspections and Walkthroughs............................................................ Code Inspections.............................................................................. An Error Check list for Inspections.................................................... Walkthroughs....................................................................................
Contents
36 36 36 37 38 39 42
Chapter 4 TEST CASE DESIGN 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 Learning Objectives.......................................................................... Introduction...................................................................................... White Box Testing............................................................................ Basis Path Testing............................................................................ Control Structure testing.................................................................... Black Box Testing............................................................................ . Static Program Analysis.................................................................... Automated Testing Tools................................................................... 43 43 43 44 45 49 53 57 58
Chapter 5 TESTING FOR SPECIALIZED ENVIRONMENTS 5.1 5.2 5.3 5.4 5.5 Learning Objectives.......................................................................... Introduction...................................................................................... Testing GUIs.................................................................................... Testing of Client/Server Architectures................................................ Testing documentation and Help facilities............................................ 60 60 60 60 63 63
Chapter 6 SOFTWARE TESTING STRATEGIES 6.1. 6.2. 6.3 6.4 6.5 6.6 6.7 6.8 Learning Objectives.......................................................................... Introduction...................................................................................... A Strategic Approach To Software Testing......................................... Verification and Validation.................................................................. Organizing for software testing.......................................................... A Software Testing Strategy.............................................................. Strategic issues................................................................................. Unit Testing...................................................................................... 65 65 65 69 70 71 72 75 75
5
80 85 86 89
6.9 6.10 6.11 6.12
Integration Testing............................................................................ Validation Testing.............................................................................. System Testing.................................................................................. Summary..........................................................................................
Chapter 7 TESTING OF WEB BASED APPLICATIONS 7.1 7.2 7.3 7.4 7.5 Introduction...................................................................................... Testing of Web Based Applications: Technical Peculiarities.................. Testing of Static Web- based applications........................................... Testing of Dynamic Web based applications........................................ Future Challenges............................................................................. 91 91 91 92 94 96
Chapter 8 TEST PROCESS MODEL 8.0 8.1 Need for Test Process Model............................................................ Test Process Cluster.......................................................................... 97 97 98
Chapter 9 TEST METRICS 9.0 9.1 9.2 9.3 9.4 9.5 Introduction...................................................................................... Overview of the Role and Use of Metrics........................................... Primitive Metric and Computed Metrics.............................................. Metrics typically used within the Testing Process................................ Defect Detection Effectiveness percentage (DDE)............................. Setting up and administering a Metrics Program.................................. 103 103 104 104 105 106 106
Chapter 1
Introduction to Software Testing
1.1 LEARNING OBJECTIVES
Y
l l l l l
ou will learn about:
What is Software Testing? Need for software Testing, Various approaches to Software Testing, What is the defect distribution, Software Testing Fundamentals.
1.2 INTRODUCTION
Software testing is a critical element of software quality assurance and represents the ultimate process to ensure the correctness of the product. The quality product always enhances the customer confidence in using the product thereby increases the business economics. In other words, a good quality product means zero defects, which is derived from a better quality process in testing. The definition of testing is not well understood. People use a totally incorrect definition of the word testing, and that this is the primary cause for poor program testing. Examples of these definitions are such statements as Testing is the process of demonstrating that errors are not present, The purpose of MSIT 32 Software Quality and Testing
Chapter 1 - Introduction to Software Testing
testing is to show that a program performs its intended functions correctly, and Testing is the process of establishing confidence that a program does what it is supposed to do. Testing the product means adding value to it, which means raising the quality or reliability of the program. Raising the reliability of the product means finding and removing errors. Hence one should not test a product to show that it works; rather, one should start with the assumption that the program contains errors and then test the program to find as many errors as possible. Thus a more appropriate definition is: Testing is the process of executing a program with the intent of finding errors.
Purpose of Testing
To show the software works: It is known as demonstration-oriented To show the software doesnt work: It is known as destruction-oriented To minimize the risk of not working up to an acceptable level: it is known as evaluation-oriented
Need for Testing

Defects can exist in the software, as it is developed by human beings who can make mistakes during the development of software. However, it is the primary duty of a software vendor to ensure that software delivered does not have defects and the customers day-to-day operations do not get affected. This can be achieved by rigorously testing the software. The most common origin of software bugs is due to:
l l l l
Poor understanding and incomplete requirements Unrealistic schedule Fast changes in requirements Too many assumptions and complacency
Some of major computer system failures listed below gives ample evidence that the testing is an important activity of the software quality process.
l
In April of 1999, a software bug caused the failure of a $1.2 billion military satellite launch, the costliest unmanned accident in the history of Cape Canaveral launches. The failure was the latest in a string of launch failures, triggering a complete military and industry review of U.S. space launch programs, including software integration and testing processes. Congressional oversight hearings were requested. On June 4, 1996, the first flight of the European Space Agencys new Ariane 5 rocket failed shortly after launching, resulting in an estimated uninsured loss of a half billion dollars. It was reportedly due to the lack of exception handling of a floating-point error in a conversion from a 64-bit integer to a 16-bit signed integer.
In January of 2001 newspapers reported that a major European railroad was hit by the aftereffects of the Y2K bug. The company found that many of their newer trains would not run due to their inability to recognize the date 31/12/2000'; the trains were started by altering the control systems date settings. In April of 1998 a major U.S. data communications network failed for 24 hours, crippling a large part of some U.S. credit card transaction authorization systems as well as other large U.S. bank, retail, and government data systems. The cause was eventually traced to a software bug. The computer system of a major online U.S. stock trading service failed during trading hours several times over a period of days in February of 1999 according to nationwide news reports. The problem was reportedly due to bugs in a software upgrade intended to speed online trade confirmations. In November of 1997 the stock of a major health industry company dropped 60% due to reports of failures in computer billing systems, problems with a large database conversion, and inadequate software testing. It was reported that more than $100,000,000 in receivables had to be written off and that multi-million dollar fines were levied on the company by government agencies. Software bugs caused the bank accounts of 823 customers of a major U.S. bank to be credited with $924,844,208.32 each in May of 1996, according to newspaper reports. The American Bankers Association claimed it was the largest such error in banking history. A bank spokesman said the programming errors were corrected and all funds were recovered.
All the above incidents only reiterate the importance of thorough testing of software applications and products before they are put on production. It clearly demonstrates that cost of rectifying defect during development is much less than rectifying a defect in production.
1.3 WHAT IS TESTING?

l
Testing is an activity in which a system or component is executed under specified conditions; the results are observed and recorded and an evaluation is made of some aspect of the system or component - IEEE Executing a system or component is known as dynamic testing. Review, inspection and verification of documents (Requirements, design documents Test Plans etc.), code and other work products of software is known as static testing. Static testing is found to be the most effective and efficient way of testing.
l l
10
l l
Successful testing of software demands both dynamic and static testing. Measurements show that a defect discovered during design that costs $1 to rectify at that stage will cost $1,000 to repair in production. This clearly points out the advantage of early testing. Testing should start with small measurable units of code, gradually progress towards testing integrated components of the applications and finally be completed with testing at the application level. Testing verifies the system against its stated and implied requirements, i.e., is it doing what it is supposed to do? It should also check if the system is not doing what it is not supposed to do, if it takes care of boundary conditions, how the system performs in production-like environment and how fast and consistently the system responds when the data volumes are high.
Reasons for Software Bugs

Following are the reasons for Software Bugs:
l
Miscommunication or no communication - as to specifics of what an application should or shouldnt do (the applications requirements). Software complexity - the complexity of current software applications can be difficult to comprehend for anyone without experience in modern-day software development. Windowstype interfaces, client-server and distributed applications, data communications, enormous relational databases, and sheer size of applications have all contributed to the exponential growth in software/system complexity. And the use of object-oriented techniques can complicate instead of simplify a project unless it is well-engineered. Programming errors - programmers, like anyone else, can make mistakes. Changing requirements - the customer may not understand the effects of changes, or may understand and request them anyway - redesign, rescheduling of engineers, effects on other projects, work already completed that may have to be redone or thrown out, hardware requirements that may be affected, etc. If there are many minor changes or any major changes, known and unknown dependencies among parts of the project are likely to interact and cause problems, and the complexity of keeping track of changes may result in errors. Enthusiasm of engineering staff may be affected. In some fast-changing business environments, continuously modified requirements may be a fact of life. In this case, management must understand the resulting risks, and QA and test engineers must adapt and plan for continuous extensive testing to keep the inevitable bugs from running out of control. time pressures - scheduling of software projects is difficult at best, often requiring a lot of guesswork. When deadlines loom and the crunch comes, mistakes will be made.
l l
11
Poorly documented code - its tough to maintain and modify code that is badly written or poorly documented; the result is bugs. In many organizations management provides no incentive for programmers to document their code or write clear, understandable code. In fact, its usually the opposite: they get points mostly for quickly turning out code, and theres job security if nobody else can understand it (if it was hard to write, it should be hard to read). Software development tools - visual tools, class libraries, compilers, scripting tools, etc. often introduce their own bugs or are poorly documented, resulting in added bugs.
1.4 APPROACHES TO TESTING

Many approaches have been defined in literature. The importance of any approache depends on the type of the system in which you are testing. Some of the approaches are given below:
Debugging-oriented:
This approach identifies the errors during debugging the program. There is no difference between testing and debugging.
Demonstration-oriented:
The purpose of testing is to show that the software works. Here most of the time, the software is demonstrated in a normal sequence/flow. All the branches may not be tested. This approach is mainly to satisfy the customer and no value added to the program.
Destruction-oriented:
The purpose of testing is to show the software doesnt work. It is a sadistic process, which explains why most people find it difficult. It is difficult to design test cases to test the program.
Evaluation-oriented:
The purpose of testing is to reduce the perceived risk of not working up to an acceptable value.
Prevention-oriented:
It can be viewed as testing is a mental discipline that results in low risk software. It is always better to forecast the possible errors and rectify it earlier. In general, program testing is more properly viewed as the destructive process of trying to find the errors (whose presence is assumed) in a program. A successful test case is one that furthers progress in this direction by causing the program to fail. However, one wants to use program testing to establish some
12
degree of confidence that a program does what it is supposed to do and does not do what it is not supposed to do, but this purpose is best achieved by a diligent exploration for errors.
1.5 IMPORTANCE OF TESTING

Testing activity cannot be eliminated in the life cycle as the end product must be bug free and reliable one. Testing is important because:
l l l
Testing is a critical element of software Quality Assurance Post-release removal of defects is the most expensive Significant portion of life cycle effort expended on testing
In a typical service oriented project, about 20-40% of project effort is spent on testing. It is much more in the case of human-rated software. For example, at Microsoft, tester to developer ratio is 1:1 whereas at NASA shuttle development center (SEI Level 5), the ratio is 7:1. This shows that how testing is an integral part of Quality assurance.
1.6 HURDLES IN TESTING

As in many other development projects, testing is not free from hurdles. Some of the hurdles normally encountered are:
l l l l l l l
Usually late activity in the project life cycle No concrete output and therefore difficult to measure the value addition Lack of historical data Recognition of importance is relatively less Politically damaging as you are challenging the developer Delivery commitments Too much optimism that the software always works correctly
Defect Distribution
In a typical project life cycle, testing is the late activity. When the product is tested, the defects may be due to many reasons. It may be either programming error or may be defects in design or defects at any stages in the life cycle. The overall defect distribution is shown in fig 1.1 .
13
Design 27%
Rqmts. Design Code Other
Rqmts. 56%
Code
7%
Other
10%
Fig 1.1: Software Defect Distribution
1.7 TESTING FUNDAMENTALS

Before understanding the process of testing software, it is necessary to learn the basic principles of testing.
1.7.1 Testing Objectives

l l l
Testing is a process of executing a program with the intent of finding an error. A good test is one that has a high probability of finding an as yet undiscovered error. A successful test is one that uncovers an as yet undiscovered error.
The objective is to design tests that systematically uncover different classes of errors and do so with a minimum amount of time and effort. Secondary benefits include:
l l l
Demonstrate that software functions appear to be working according to specification. Those performance requirements appear to have been met. Data collected during testing provides a good indication of software reliability and some indication of software quality.
14
Testing cannot show the absence of defects, it can only show that software defects are present.
1.7.2 Test Information Flow

A typical test information flow is shown in Fig 1.2.
Fig 1.2: Test information flow in a typical software test life cycle
In the above figure:

l
Software configuration includes a Software Requirements Specification, a Design Specification, and source code. A test configuration includes a test plan and procedures, test cases, and testing tools. It is difficult to predict the time to debug the code, hence it is difficult to schedule.
l l
1.7.3 Test Case Design

Some of the points to be noted during the test case design are:
l l l
Can be as difficult as the initial design. Can test if a component conforms to specification - Black Box Testing. Can test if a component conforms to design - White box testing.
15
Testing cannot prove correctness as not all execution paths can be tested.
Consider the following example shown in fig 1.3,
Fig: 1.3
A program with a structure as illustrated above (with less than 100 lines of Pascal code) has about 100,000,000,000,000 possible paths. If attempted to test these at rate of 1000 tests per second, would take 3170 years to test all paths. This shows that exhaustive testing of software is not possible. QUESTIONS
1. What is software testing? Explain the purpose of testing? 2. Explain the origin of the defect distribution in a typical software development life cycle? _________
16
Chapter 2
Chapter 2 - Software Quality Assurance
Software Quality Assurance
Y
l l l
Basic principles about the Software Quality, Software Quality Assurance and SQA activities Software Reliability
2.2 INTRODUCTION
The quality is defined as a characteristic or attribute of something. As an attribute of an item, quality refers to measurable characteristics-things we are able to compare to known standards such as length, color, electrical properties, malleability, and so on. However, software, largely an intellectual entity, is more challenging to characterize than physical objects. Quality design refers to the characteristic s that designers specify for an item. The grade of materials, tolerance, and performance specifications all contribute to the quality of design. Quality of conformance is the degree to which the design specification s are followed during manufacturing. Again, the greater the degree of conformance, the higher the level of quality of conformance.
16
17
Software Quality Assurance encompasses

q q q q q q q
A quality management approach Effective software engineering technology Formal technical reviews A multi tiered testing strategy Control of software documentation and changes made to it A procedure to assure compliance with software development standards Measurement and reporting mechanisms
Software quality is achieved as shown in figure 2.1:

Formal Technical Review
Software Engineering Methods
Measurement
Quality
Standards And And
SCM & & SQA Figure 2.1: Achieving Software Quality
Testing
2.3 QUALITY CONCEPTS

What are quality concepts?
l l l l
Quality Quality control Quality assurance Cost of quality
18
The American heritage dictionary defines quality as a characteristic or attribute of something. As an attribute of an item quality refers to measurable characteristic-things, we are able to compare to known standards such as length, color, electrical properties, and malleability, and so on. However, software, largely an intellectual entity, is more challenging to characterize than physical object. Nevertheless, measures of a programs characteristics do exist. These properties include 1. Cyclomatic complexity 2. Cohesion 3. Number of function points 4. Lines of code When we examine an item based on its measurable characteristics, two kinds of quality may be encountered:
l l
Quality of design Quality of conformance
2.4 QUALITY OF DESIGN

Quality of design refers to the characteristics that designers specify for an item. The grade of materials, tolerance, and performance specifications all contribute to quality of design. As higher graded materials are used and tighter, tolerance and greater levels of performance are specified the design quality of a product increases if the product is manufactured according to specifications.
2.5 QUALITY OF CONFORMANCE

Quality of conformance is the degree to which the design specifications are followed during manufacturing. Again, the greater the degree of conformance, the higher the level of quality of conformance. In software development, quality of design encompasses requirements, specifications and design of the system. Quality of conformance is an issue focused primarily on implementation. If the implementation follows the design and the resulting system meets its requirements and performance goals, conformance quality is high.
19
2.6 QUALITY CONTROL (QC)

QC is the series of inspections, reviews, and tests used throughout the development cycle to ensure that each work product meets the requirements placed upon it. QC includes a feedback loop to the process that created the work product. The combination of measurement and feedback allows us to tune the process when the work products created fail to meet their specification. These approach views QC as part of the manufacturing process QC activities may be fully automated, manual or a combination of automated tools and human interaction. An essential concept of QC is that all work products have defined and measurable specification to which we may compare the outputs of each process the feedback loop is essential to minimize the defect produced.
2.7 QUALITY ASSURANCE (QA)

QA consists of the editing and reporting functions of management. The goal of quality assurance is to provide management with the data necessary to be informed about product quality, there by gaining insight and confidence that product quality is meeting its goals. Of course, if the data provided through QA identify problems, it is managements responsibility to address the problems and apply the necessary resources to resolve quality issues.
2.7.1 Cost of Quality

Cost of quality includes all costs incurred in the pursuit of quality or in performing quality related activities. Cost of quality studies are conducted to provide a base line for the current cost of quality, to identify opportunities for reducing the cost of quality, and to provide a normalized basis of comparison. The basis of normalization is usually money. Once we have normalized quality costs on a money basis, we have the necessary data to evaluate where the opportunities lie to improve our process further more we can evaluate the effect of changes in money based terms. QC may be divided into cost associated with
l l l
Prevention Appraisal Failure
Prevention costs include

q q
Quality Planning Formal Technical Review
20
q q
Test Equipment Training
Appraisal costs include activity to gain insight into product condition the First time through each process. Examples for appraisal costs include:
l l l
In process and inter process inspection Equipment calibration and maintenance Testing
Failure Costs are costs that would disappear if no defects appeared before shipping a product to customer. Failure costs may be subdivided into internal and external failure costs. Internal failure costs are costs incurred when we detect an error in our product prior to shipment. Internal failure costs includes
l l l
Rework Repair Failure Mode Analyses
External failure costs are the cost associated with defects found after the product has been shipped to the customer. Examples of external failure costs are 1. Complaint Resolution 2. Product return and replacement 3. Helpline support 4. Warranty work
2.8 SOFTWARE QUALITY ASSURANCE (SQA)

Quality Is defined as conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software.
21
The above definition emphasizes three important points. 1. Software requirements are the foundation from which quality is measured. Lack of conformance to requirements is lack of quality. 2. Specified standards define a set of development criteria that guide the manner in which software is engineered. If the criteria are not followed, lack of quality will almost surely result. 3. There is a set of implicit requirements that often goes unmentioned. (e.g. the desire of good maintainability). If software conforms to its explicit requirements but fails to meet implicit requirements, software quality is questionable.
2.8.1 Background Issues

QA is an essential activity for any business that produces products to be used by others. The SQA group serves as the customer in-house representative. That is the people who perform SQA must look at the software from customers point of views. The SQA group attempts to answer the questions asked below and hence ensure the quality of software. The questions are 1. Has software development been conducted according to pre-established standards? 2. Have technical disciplines properly performed their role as part of the SQA activity?
SQA Activities
SQA Plan is interpreted as shown in Fig 2.2 SQA is comprised of a variety of tasks associated with two different constituencies 1. The software engineers who do technical work like
l l l
Performing Quality assurance by applying technical methods Conduct Formal Technical Reviews Perform well-planed software testing.
2. SQA group that has responsibility for

l l l
Quality assurance planning oversight Record keeping Analysis and reporting.
22
QA activities performed by SE team and SQA are governed by the following plan.
l l l l l l
Evaluation to be performed. Audits and reviews to be performed. Standards that is applicable to the project. Procedures for error reporting and tracking Documents to be produced by the SQA group Amount of feedback provided to software project team.
Software Engineers Activities
SQA Planning Team Activities
SQA Plan
Figure 2.2: Software Quality Assurance Plan

l l l l l
What are the activities performed by SQA and SE team? Prepare SQA Plan for a project Participate in the development of the projects software description Review software-engineering activities to verify compliance with defined software process. Audits designated software work products to verify compliance with those defined as part of the software process. Ensures that deviations in software work and work products are documented and handled according to a documented procedure. Records any noncompliance and reports to senior management.
23
2.8.2 Software Reviews

Software reviews are a filter for the software engineering process. That is, reviews are applied at various points during software development and serve to uncover errors that can then be removed. Software reviews serve to purify the software work products that occur as a result of analysis, design, and coding. Any review is a way of using the diversity of a group of people to: 1. Point out needed improvements in the product of a single person or a team; 2. Confirm that parts of a product in which improvement is either not desired, or not needed. 3. Achieve technical work of more uniform, or at least more predictable quality that can be achieved without reviews, in order to make technical work more manageable. There are many different types of reviews that can be conducted as part of software- engineering like 1. An informal meeting if technical problems are discussed. 2. Formal presentation of software design to an audience of customers, management, and technical staff is a form of review. 3. Formal technical review is the most effective filter from a quality assurance standpoint. Conducted by software engineers for software engineers, the FTR is an effective means of improving software quality.
2.8.3 Cost impact of Software Defects

To illustrate the cost impact of early error detection, we consider a series of relative costs that is based on actual cost data collected for large software projects. Assume that an error uncovered during design will cost 1.0 monetary unit to correct. Relative to this cost, the same error uncovered just before testing commences will cost 6.5 units; during testing 15 units; and after release, between 60 and 100 units.
2.8.4 Defect Amplification and Removal

A defect amplification model can be used to illustrate the generation and detection of errors during
24
preliminary design, detail design, and coding steps of the software engineering process. The model is illustrated schematically in Figure 2.3. A box represents a software development step. During the step, errors may be inadvertently generated. Review may fail to uncover newly generated errors from previous steps, resulting in some number of errors that are passed through. In some cases, errors passed through from previous steps, resulting in some number of errors that are passed through. In some cases errors passed through from previous steps are amplified (amplification factor, x) by current work. The box subdivisions represent each of these characteristics and the percent efficiency for detecting errors, a function of the thoroughness of review. DEVELOPMENT STEP DEVELOPMENT STEP Errors from previous Step DEFECTS Errors fromprevious Step DEFECTS DETECTION DETECTION
Errors passed through Amplified errors 1:x Newly generated errors
Percent detection
efficiency
for
error Errors passed to next step
FigureFigure 2.3: Defect Amplification Model. 2.3: Defect Amplification Model.

Figure 2.4 illustrates hypothetical example of defect amplification for a software development process in which no reviews are conducted. As shown in the figure each test step is assumed to uncover and correct fifty percent of all incoming errors without introducing new errors (an optimistic assumption). Ten preliminary design errors are amplified to 94 errors before testing commences. Twelve latent defects are released to the field. Figure 2.5 considers the same conditions except that design and code reviews are conducted as part of each development step. In this case, ten initial preliminary design errors are amplified to 24 errors before testing commences. Only three latent defects exist. By recalling the relative cost associated with the discovery and correction of errors, overall costs (with and without review for our hypothetical example) can be established. To conduct reviews a developer must expend time and effort and the development organization must spend money. However, the results of the preceding or previous, example leave little doubt that we have encountered a Pay now or pay much more lately syndrome. Formal technical reviews (for design and other technical activities) provide a demonstrable cost benefit and they should be conducted.
25
Preliminary design 0 070% 10 3, 2 Detail Design 2 50% 1-1.5 1 25 15 5 Code/Unit Test 5 10 60% -3 10 25 To integration 12 070% 10 Validation test 2 50% 1-1.5 25 6 System Test 3 060% 0 Figure2.4: Defect Amplification -No Reviews Latent errors
24
24
Integration Test
26
Preliminary design 0 00% 10 10, 6 Detail Design 6 4 x 0% 1.5 4 x = 1.5 25 37 Code/Unit Test 10 20% 27x3 x=3 25 To integration 94 Integration Test 47 050% 10 Validation test 2 50% 1-1.5 25 24 System Test 12 060% 0 Latent errors Figure 2.5: Defect Amplification - Reviews Conducted 94
27
2.9 FORMAL TECHNICAL REVIEWS (FTR)

FTR is a SQA activity that is performed by software engineers. Objectives of the FTR are
l l l l l
To uncover errors in function, logic, are implementations for any representation of the software. To verify that software under review meets its requirements. To ensure that software has been represented according to predefined standards To achieve software that is developed in an uniform manner To make projects more manageable
In addition, the FTR serves as a training ground, enabling junior engineers to observe different approaches to software analysis, design, and implementation. The FTR also serves to promote backup and continuity because numbers of people become familiar with parts of the software that they may not have other wise seen. The FTR is actually a class of reviews that include walkthrough inspection and round robin reviews, and other small group technical assessments of software. Each FTR is conducted as meeting and will be successful only if it is properly planned, controlled and attended.
Types of Formal Technical Review

While the focus of this research is on the individual evaluation aspects of reviews, for context several other FTR techniques are discussed as well. Among the most common forms of FTR are the following: 1. Desk Checking, or reading over a program by hand while sitting at ones desk, is the oldest software review technique [Adrion et al. 1982]. Strictly speaking, desk checking is not a form of FTR since it does not involve a formal process or a group. Moreover, desk checking is generally perceived as ineffective and unproductive due to (a) its lack of discipline and (b) the general ineffectiveness of people in detecting their own errors. To correct for the second problem, programmers often swap programs and check each others work. Since desk checking is an individual process not involving group dynamics, research in this area would be relevant but none applicable to the current research was found. It should be noted that Humphrey [1995] has developed a review method, called Personal Review (PR), which is similar to desk checking. In PR, each programmer examines his own products to find as many defects as possible utilizing a disciplined process in conjunction with Humphreys Personal Software Process (PSP) to improve his own work. The review strategy includes the use of checklists to guide the review process, review metrics to improve the process, and defect causal analysis to prevent the same defects from recurring in the future. The approach
28
taken in developing the Personal Review process is an engineering one; no reference is made in Humphrey [1995] to cognitive theory. 2. Peer Rating is a technique in which anonymous programs are evaluated in terms of their overall quality, maintainability, extensibility, usability and clarity by selected programmers who have similar backgrounds [Myers 1979]. Shneiderman [1980] suggests that peer ratings of programs are productive, enjoyable, and non-threatening experiences. The technique is often referred to as Peer Reviews [Shneiderman 1980], but some authors use the term peer reviews for generic review methods involving peers [Paulk et al 1993; Humphrey 1989]. 3. Walkthroughs are presentation reviews in which a review participant, usually the software author, narrates a description of the software and the other members of the review group provide feedback throughout the presentation [Freedman and Weinberg 1990; Gilb and Graham 1993]. It should be noted that the term walkthrough has been used in the literature variously. Some authors unite it with structured and treat it as a disciplined, formal review process [Myers 1979; Yourdon 1989; Adrion et al. 1982]. However, the literature generally describes walkthrough as an undisciplined process without advance preparation on the part of reviewers and with the meeting focus on education of participants [Fagan 1976]. 4. Round-robin Review is a evaluation process in which a copy of the review materials is made available and routed to each participant; the reviewers then write their comments/questions concerning the materials and pass the materials with comments to another reviewer and to the moderator or author eventually [Hart 1982]. 5. Inspection was developed by Fagan [1976, 1986] as a well-planned and well-defined group review process to detect software defects defect repair occurs outside the scope of the process. The original Fagan Inspection (FI) is the most cited review method in the literature and is the source for a variety of similar inspection techniques [Tjahjono 1996]. Among the FIderived techniques are Active Design Review [Parnas and Weiss 1987], Phased Inspection [Knight and Myers 1993], N-Fold Inspection [Schneider et al. 1992], and FTArm [Tjahjono 1996]. Unlike the review techniques previously discussed, inspection is often used to control the quality and productivity of the development process. A Fagan Inspection consists of six well-defined phases: i. Planning. Participants are selected and the materials to be reviewed are prepared and checked for review suitability. ii. Overview. The author educates the participants about the review materials through a presentation. iii. Preparation. The participants learn the materials individually. iv. Meeting. The reader (a participant other than the author) narrates or paraphrases the review materials statement by statement, and the other participants raise issues and questions. Questions continue on a point only until an error is recognized or the item is deemed correct.
29
v. Rework. The author fixes the defects identified in the meeting. vi. Follow-up. The corrected products are reinspected. Practitioner Evaluation is primarily associated with the Preparation phase. In addition to classification by technique-type, FTR may also be classified on other dimensions, including the following: A. Small vs. Large Team Reviews. Siy [1996] classifies reviews into those conducted by small (1-4 reviewers) [Bisant and Lyle 1996] and large (more than 4 reviewers) [Fagan 1976, 1986] teams. If each reviewer depends on different expertise and experiences, a large team should allow a wider variety of defects to be detected and thus better coverage. However, a large team requires more effort due to more individuals inspecting the artifact, generally involves greater scheduling problems [Ballman and Votta 1994], and may make it more difficult for all participants to participate fully. B. No vs. Single vs. Multiple Session Reviews. The traditional Fagan Inspection provided for one session to inspect the software artifact, with the possibility of a follow-up session to inspect corrections. However, variants have been suggested. Humphrey [1989] comments that three-quarters of the errors found in well-run inspections are found during preparation. Based on an economic analysis of a series of inspections at AT&T, Votta [1993] argues that inspection meetings are generally not economic and should be replaced with depositions, where the author and (optionally) the moderator meet separately with inspectors to collect their results. On the other hand, some authors [Knight and Myers 1993; Schneider et al. 1992] have argued for multiple sessions, conducted either in series or parallel. Gilb and Graham [1993] do not use multiple inspection sessions but add a root cause analysis session immediately after the inspection meeting. C. Nonsystematic vs. Systematic Defect-Detection Technique Reviews. The most frequently used detection methods (ad hoc and checklist) rely on nonsystematic techniques, and reviewer responsibilities are general and not differentiated for single session reviews [Siy 1996]. However, some methods employ more prescriptive techniques, such as questionnaires [Parnas and Weiss 1987] and correctness proofs [Britcher 1988]. D. Single Site vs. Multiple Site Reviews. The traditional FTR techniques have assumed that the group-meeting component would occur face-to-face at a single site. However, with improved telecommunications, and especially with computer support (see item F below), it has become increasingly feasible to conduct even the group meeting from multiple sites. E. Synchronous vs. Asynchronous Reviews. The traditional FTR techniques have also assumed
30
that the group meeting component would occur in real-time; i.e., synchronously. However, some newer techniques that eliminate the group meeting or are based on computer support utilize asynchronous reviews. F. Manual vs. Computer-supported Reviews. In recent years, several computer supported review systems have been developed [Brothers et al. 1990; Johnson and Tjahjono 1993; Gintell et al. 1993; Mashayekhi et al 1994]. The type of support varies from simple augmentation of the manual practices [Brothers et al. 1990; Gintell et al. 1993] to totally new review methods [Johnson and Tjahjono 1993].
2.2.2 Economic Analyses of Formal Technical Review

Wheeler et al. [1996], after reviewing a number of studies that support the economic benefit of FTR, conclude that inspections reduce the number of defects throughout development, cause defects to be found earlier in the development process where they are less expensive to correct, and uncover defects that would be difficult or impossible to discover by testing. They also note these benefits are not without their costs, however. Inspections require an investment of approximately 15 percent of the total development cost early in the process [p. 11]. In discussing overall economic effects, Wheeler et al. cite Fagan [1986] to the effect that investment in inspections has been reported to yield a 25-to-35 percent overall increase in productivity. They also reproduce a graphical analysis from Boehm [1987] that indicates inspections reduce total development cost by approximately 30%. The Wheeler et al. [1996] analysis does not specify the relative value of Practitioner Evaluation to FTR, but two recent economic analyses provide indications.
l
Votta [1993]. After analyzing data collected from 13 traditional inspections conducted at AT&T, Votta reports that the approximately 4% increase in faults found at collection meetings (synergy) does not economically justify the development delays caused by the need to schedule meetings and the additional developer time associated with the actual meetings. He also argues that it is not cost-effective to use the collection meeting to reduce the number of items incorrectly identified as defective prior to the meeting (false positives). Based on these findings, he concludes that almost all inspection meetings requiring all reviewers to be present should be replaced with Depositions, which are three person meetings with only the author, moderator, and one reviewer present. Siy [1996]. In his analysis of the factors driving inspection costs and benefits, Siy reports that changes in FTR structural elements, such as group size, number of sessions, and coordination of multiple sessions, were largely ineffective in improving the effectiveness of inspections. Instead, inputs into the process (reviewers and code units) accounted for more outcome variation than
31
structural factors. He concludes by stating better techniques by which reviewers detect defects, not better process structures, are the key to improving inspection effectiveness [Abstract, p. 2]. (emphasis added) Vottas analysis effectively attributes most of the economic benefit of FTR to PE, and Siys explicitly states that better PE techniques are the key to improving inspection effectiveness. These findings, if supported by additional research, would further support the contention that a better understanding of Practitioner Evaluation is necessary.
2.2.3 Psychological Aspects of FTR

Work on the psychological aspects of FTR can be categorized into four groups. 1. Egoless Programming. Gerald Weinberg [1971] began the examination of psychological issues associated with software review in his work on egoless programming. According to Weinberg, programmers are often reluctant to allow their programs to be read by other programmers because the programs are often considered to be an extension of the self and errors discovered in the programs to be a challenge to ones self-image. Two implications of this theory are as follows: i. The ability of a programmer to find errors in his own work tends to be impaired since he tends to justify his own actions, and it is therefore more effective to have other people check his work. ii. Each programmer should detach himself from his own work. The work should be considered a public property where other people can freely criticize, and thus, improve its quality; otherwise, one tends to become defensive, and reluctant to expose ones own failures. These two concepts have led to the justification of FTR groups, as well as the establishment of independent quality assurance groups that specialize in finding software defects in many software organizations [Humphrey 1989]. 2. Role of Management. Another psychological aspect of FTR that has been examined is the recording of data and its dissemination to management. According to Dobbins [1987], this must be done in such a way that individual programmers will not feel intimidated or threatened. 3. Positive Psychological Impacts. Hart [1982] observes that reviews can make one more careful in writing programs (e.g., double checking code) in anticipation of having to present or share the programs with other participants. Thus, errors are often eliminated even before the actual review sessions. 4. Group Process. Most FTR methods are implemented using small groups. Therefore, several
32
key issues from small group theory apply to FTR, such as group think (tendency to suppress dissent in the interests of group harmony), group deviants (influence by minority), and domination of the group by a single member. Other key issues include social facilitation (presence of others boosts ones performance) and social loafing (one member free rides on the groups effort) [Myers 1990]. The issue of moderator domination in inspections is also documented in the literature [Tjahjono 1996]. Perhaps the most interesting research from the perspective of the current study is that of Sauer et al. [2000]. This research is unusual in that it has an explicit theoretical basis and outlines a behaviorally motivated program of research into the effectiveness of software development technical reviews. The finding that most of the variation in effectiveness of software development technical reviews is the result of variations in expertise among the participants provides additional motivation for developing a solid understanding of Formal Technical Review at the individual level. It should be noted that all of this work, while based on psychological theory, does not address the issue of how practitioners actually evaluate software artifacts.
2.9.1 The Review Meeting

The Focus of the FTR is on a work product - a component of the software. At the end of review all attendees of the FTR must decide 1. Whether to accept the work product without further modification. 2. Reject the work product due to serve errors (Once corrected another review must be performed) 3. Accept the work product provisionally (minor errors have been encountered and must be corrected but no additional review will be required). Once the decision made, all FTR attendees complete a sign-off indicating their participation in the review and their concurrence with the review team findings.
2.9.2 Review reporting and record keeping

The review summary report is typically is a single page form. It becomes part of the project historical record and may be distributed to the project leader and other interested parties. The review issue lists serves two purposes. 1. To identify problem areas within the product
33
2. To serve as an action item. Checklist that guides the producer as corrections are made. An issues list is normally attached to the summary report. It is important to establish a follow up procedure to ensure that item on the issues list have been properly corrected. Unless this is done, it is possible that issues raised can fall between the cracks. One approach is to assign responsibility for follow up for the review leader. A more formal approach as signs responsibility independent to SQA group.
2.9.3 Review Guidelines

The following represents a minimum set of guidelines for formal technical reviews
l l l l l l l l l l
Review the product, not the producer Set an agenda and maintain it Limit debate and rebuttal Enunciate problem areas but dont attempt to solve every problem noted Take return notes Limit the number of participants and insist upon advance preparation Develop a check list each work product that is likely to be reviewed Allocate resources and time schedule for FTRs. Conduct meaningful training for all reviewers Review your earlier reviews
2.10 STATISTICAL QUALITY ASSURANCE

Statistical quality assurance reflects a growing trend throughout industry to become more quantitative about quality. For software, statistical quality assurance implies the following steps
l l l
Information about software defects is collected and categorized An attempt is made to trace each defect to its underlying cause Using Pareto principle (80% of the defects can be traced to 20% of all possible causes), isolate the 20% (the vital few)
34
l
Once the vital few causes have been identified, move to correct the problems that have caused the defects.
This relatively simple concept represents an important step toward the creation of an adaptive software engineering process in which changes are made to improve those elements of the process that introduce errors. To illustrate the process, assume that a software development organization collects information on defects for a period of one year. Some errors are uncovered as software is being developed. Other defects are encountered after the software has been released to its end user. Although hundreds of errors are uncovered all can be tracked to one of the following causes.
q q q q q q q q q q q q
Incomplete or Erroneous Specification (IES) Misinterpretation of Customer Communication (MCC) Intentional Deviation from Specification (IDS) Violation of Programming Standards ( VPS ) Error in Data Representation (EDR) Inconsistent Module Interface (IMI) Error in Design Logic (EDL) Incomplete or Erroneous Testing (IET) Inaccurate or Incomplete Documentation (IID) Error in Programming Language Translation of design (PLT) Ambiguous or inconsistent Human-Computer Interface (HCI) Miscellaneous (MIS)
To apply statistical SQA table 2.1 is built. Once the vital few causes are determined, the software development organization can begin corrective action. After analysis, design, coding, testing, and release, the following data are gathered.
Ei = The total number of errors uncovered during the ith step in the software Engineering process Si Mi Ti PS = The number of serious errors = The number of moderate errors = The number of minor errors = Size of the product (LOC, design statements, pages of documentation at the ith step
35
Ws, Wm, Wt = weighting factors for serious, moderate and trivial errors where recommended values are Ws = 10, Wm = 3, Wt = 1. The weighting factors for each phase should become larger as development progresses. This rewards an organization that finds errors early. At each step in the software engineering process, a phase index, PIi, is computed PIi = Ws (Si/Ei)+Wm (Mi/Ei)+Wt (Ti/Ei) The error index EI ids computed by calculating the cumulative effect or each PIi, weighting errors encountered later in the software engineering process more heavily than those encountered earlier. EI =S (i x PIi)/PS = (PIi+2PI2 +3PI3 +iPIi)/PS The error index can be used in conjunction with information collected in table to develop an overall indication of improvement in software quality.
DATA COLLECTION FOR STATISTICAL SQA Total Error IES MCC IDS VPS EDR IMI EDL IET IID PLT HCI MIS No. 205 156 48 25 130 58 45 95 36 60 28 56 % 22 17 5 3 14 6 5 10 4 6 3 6 100 No. 34 12 1 0 26 9 14 12 2 15 3 0 128 % 27 9 1 0 20 7 11 9 2 12 2 0 100 Serious No 68 68 24 15 68 18 12 35 20 19 17 15 379 Moderate % 18 18 6 4 18 5 3 9 5 5 4 4 100 No 103 76 23 10 36 31 19 48 14 26 8 41 435 % 24 17 5 2 8 7 4 11 3 6 2 9 100 Minor
TOTALS 942
36 2.11 SOFTWARE RELIABILITY

Software reliability, unlike many other quality factors, can be measured, directed and estimated using historical and developmental data. Software reliability is defined in statistical terms as Probability of failure free operation of a computer program in a specified environment for a specified time to illustrate, program x is estimated to have reliability of 0.96 over 8 elapsed processing hours. In other words, if program x were to be executed 100 times and required 8 hours of elapsed processing time, it is likely to operate correctly to operate 96/100 times.
2.11.1 Measures of Reliability and Availability

In a computer-based system, a simple measure of reliability is Mean Time Between Failure (MTBF), where MTBF = MTTF+MTTR The acronym MTTF and MTTR are Mean Time To Failure and Mean Time To Repair, respectively. In addition to reliability measure, we must develop a measure of availability. Software availability is the probability that a program is operating according to requirements at a given point in time and is defined as: Availability = MTTF / (MTTF+MTTR) x100% The MTBF reliability measure is equally sensitive to MTTF and MTTR. The availability measure is somewhat more sensitive to MTTR an indirect measure of the maintainability of the software.
2.11.2 Software Safety and Hazard Analysis

Software safety and hazard analysis are SQA activities that focus on the identification and assessment of potential hazards that may impact software negatively and cause entire system to fail. If hazards can be identified early in the software engineering process software design features can be specified that will either eliminate or control potential hazards. A modeling and analysis process is conducted as part of safety. Initially hazards are identified and categorized by criticality and risk. Once hazards are identified and analyzed, safety related requirements could be specified for the software i.e., the specification can contain a list of undesirable events and desired system responses to these events. The roll of software in managing undesirable events is then indicated. Although software reliability and software safety are closely related to one another, it is important to
37
understand the subtle difference between them. Software reliability uses statistical analysis to determine the likelihood that a software failure will occur however, the occurrence of a failure does not necessarily result in a hazard or mishap. Software safety examines the ways in which failure result in condition that can be lead to mishap. That is, failures are not considered in a vacuum. But are evaluated in the context of an entire computer based system.
2.12 THE SQA PLAN

The SQA plan provides a road map for instituting software quality assurance. Developed by the SQA group and the project team, The plan serves as a template for SQA activities that are instituted for each software project. ANSI/IEEE Standards 730-1984 and 983-1986 SQA plans is defined as shown below. I. Purpose of Plan II. References III Management 1. Organization 2. Tasks 3. Responsibilities IV. Documentation 1. Purpose 2. Required software engineering documents 3. Other Documents V. Standards, Practices and conventions 1. Purpose 2. Conventions VI. Reviews and Audits 1. Purpose 2. Review requirements
38
a. Software requirements b. Designed reviews c. Software V & V reviews d. Functional Audits e. Physical Audit f. In-process Audits g. Management reviews VII. Test VIII. Problem reporting and corrective action IX. Tools, techniques and methodologies X. Code Control XI. Media Control XII. Supplier Control XIII. Record Collection, Maintenance, and retention XIV. Training XV. Risk Management.
2.12.1 The ISO Approach to Quality Assurance System

ISO 9000 describes the elements of a quality assurance in general terms. These elements include the organizational structure, procedures, processes, and resources needed to implement quality planning, quality control, quality assurance, and quality improvement. However, ISO 9000 does not describe how an organization should implement these quality system elements. Consequently, the challenge lies in designing and implementing a quality assurance system that meets the standard and fits the companys products, services, and culture.
2.12.2 The ISO 9001 standard

ISO 9001 is the quality assurance standard that applies to software engineering. The standard contains
39
20 requirements that must be present for an effective quality assurance system. Because the ISO 9001 standard is applicable in all engineering disciplines, a special set of ISO guidelines have been developed to help interpret the standard for use in the software process. The 20 requirements delineated by ISO9001 address the following topic. 1. Management responsibility 2. Quality system 3. Contract review 4. Design control 5. Document and data control 6. Purchasing 7. Control of customer supplied product 8. Product identification and tractability 9. Process control 10.Inspection and testing 11. Control of inspection, measuring, and test equipment 12.Inspection and test status 13.Control of non confirming product 14.Corrective and preventive action 15.Handling, storage, packing, preservation, and delivery 16.Control of quality records 17.Internal quality audits 18.Training 19.Servicing 20.Statistical techniques In order for a software organization to become registered to ISO 9001, it must establish policies and procedure to address each of the requirements noted above and then be able to demonstrate that these policies and procedures are being followed.
40 2.12.3 Capability Maturity Model (CMM)

The Capability Maturity Model for Software (also known as the CMM and SW-CMM) has been a model used by many organizations to identify best practices useful in helping them increase the maturity of their processes. It was developed by the software development community along with Software Engineering Institute and Carnegie Melon University under direction of the US department of defense. It is applicable to any size software company. Its five levels as shown in Figure: 2.6 provide a simple means to assess a companys software development maturity and determine the key practices they could adopt to move up to the next level of maturity.
Optimizing Managed Defined Repeatable Initial
5. Continuous proce ss impro ve ment through

ne w approaches. 4. Controlled Process
quantita tive feedback and
3. Organizatio nal level thinking 2. Project leve l thin king 1 A dhoc

and chaotic pr ocess.
Fig 2.6: The software capability maturity model is used to assess a software companys maturity at software development
Level 1: Initial. The software development processes at this level are ad hoc and often chaotic. There are no general practices for planning, monitoring or controlling the process. The test process is just as ad hoc as the rest of the process. Level 2: Repeatable. This maturity level is best described as project level thinking. Basic project management processes are in place to track the cost, schedule, functionality and quality of the project. Basic disciplines like software testing practices like test plans and test cases are used. Level3: Defined: Organizational, not just project specific, thinking comes into play at this level. Common management and engineering activities are standardized and documented. These standards are adapted and approved for use in different projects. Test documents and plans are reviewed and approved before testing begins. Level4: Managed. At this maturity level, the organizations process is under statistical control. Product quality is specified quantitatively beforehand and the software isnt release until that goal is met. Level5:Optimizing.This level is called optimizing which is a continuously improving from level 4. New technologies and processes are attempted, the results are measured, and both incremental and revolutionary changes are instituted to achieve even better quality levels. Perspective on CMM ratings: During 1997-2001, 1018 organizations were assessed. Of those,
41
27% were rated at Level 1, 39% at 2, 23% at 3, 6% at 4, and 5% at 5. (For ratings during the period 1992-96, 62% were at Level 1, 23% at 2, 13% at 3, 2% at 4, and (0.4% at 5.) The median size of organizations was 100 software engineering/maintenance personnel; 32% of organizations were U.S. federal contractors or agencies. For those rated at Level 1, the most problematical key process area was in Software Quality Assurance. QUESTIONS
1. Quality and reliability are related concepts, but are fundamentally different in a number of ways. Discuss them. 2. Can a program be correct and still not be reliable? Explain. 3. Can a program be correct and still not exhibit good quality? Explain. 4. Explain in more detail, the review technique adopted in Quality Assurance.
42
Chapter 3
Program Inspections, Walkthroughs and Reviews
Y
l l l l l
ou will learn about
What is static testing and its importance in Software Testing. Guidelines to be followed during static testing Process involved in inspection and walkthroughs Various check lists to be followed while handling errors in Software Testing Review techniques
3.2 INTRODUCTION
Majority of the programming community worked under the assumptions that programs are written solely for machine execution and are not intended to be read by people. The only way to test a program is by executing it on a machine. Weinberg built a convincing strategy that why programs should be read by people, and indicated this could be an effective error detection process. Experience has shown that human testing techniques are quite effective in finding errors, so much so that one or more of these should be employed in every programming project. The method discussed in this Chapter are intended to be applied between the time that the program is coded and the time that computer based testing begins. We discuss this based on two ways:
42
Chapter 3 - Program Inspections, Walkthroughs and Reviews
43
It is generally recognized that the earlier errors are found, the lower are the costs or correcting the errors and the higher is the probability of correcting the errors correctly. Programmers seem to experience a psychological change when computer-based testing commences.
3.3 INSPECTIONS AND WALKTHROUGHS

Code inspections and walkthroughs are the two primary human testing methods. It involve the reading or visual inspection of a program by a team of people. Both methods involve some preparatory work by the participants. Normally it is done through meeting and it is typically known as meeting of the minds, a conference held by the participants. The objective of the meeting is to find errors, but not to find solutions to the errors (i.e. to test but not to debug).
What is the process involved in inspection and walkthroughs?

The process is performed by a group of people (three or four), only one of whom is the author of the program. Hence the program is essentially being tested by people other than the author, which is in consonance with the testing principle stating that an individual is usually ineffective in testing his or her own program. Inspection and walkthroughs are far more effective compared to desk checking (the process of a programmer reading his/her own program before testing it) because people other than the programs author are involved in the process. These processes also appear to result in lower debugging (error correction) costs, since, when they find an error, the precise nature of the error is usually located. Also, they expose a batch or errors, thus allowing the errors to be corrected later enmasse. Computer based testing, on the other hand, normally exposes only a symptom of the error and errors are usually detected and corrected one by one.
Some Observations:
l
Experience with these methods has found them to be effective e in finding from 30% to 70% of the logic design and coding errors in typical programs. They are not, however, effective in detecting high-level design errors, such as errors made in the requirements analysis process. Human processes find only the easy errors (those that would be trivial to find with computerbased testing) and the difficult, obscure, or tricky errors can only be found by computer-based testing. Inspections/walkthroughs and computer-based testing are complementary; error-detection efficiency will suffer if one or the other is not present. These processes are invaluable for testing modifications to programs. Because modifying an existing program is a more error-prone process(in terms of errors per statement written) than writing a new program.
44 3.4 CODE INSPECTIONS

An inspection team usually consists of four people. One of the four people plays the role of a moderator. The moderator is expected to be a competent programmer, but he/she is not the author of the program and need not be acquainted with the details of the program. The duties of the moderator include:
l l l l
Distributing materials for scheduling inspections Leading the session, Recording all errors found, and Ensuring that the errors are subsequently corrected.
Hence the moderator may be called as quality-control engineer. The remaining members usually consist of the programs designer and a test specialist. The general procedure is that the moderator distributes the programs listing and design specification to the other participants well in advance of the inspection session. The participants are expected to familiarize themselves with the material prior to the session. During inspection session, two main activities occur: 1. The programmer is requested to narrate, statement by statement, the logic of the program. During the discourse, questions are raised and pursued to determine if errors exist. Experience has shown that many of the errors discovered are actually found by the programmer, rather than the other team members, during the narration. In other words, the simple act of reading aloud ones program to an audience seems to be a remarkably effective error-detection technique. 2. The program is analyzed with respect to a checklist of historically common programming errors (such a checklist is discussed in the next section). It is moderators responsibility to ensure the smooth conduction of the proceedings and that the participants focus their attention on finding errors, not correcting them. After the session, the programmer is given a list of the errors found. The list of errors is also analyzed, categorized, ad used to refine the error checklist to improve the effectiveness of future inspections. The main benefits of this method are;
l l
Identifying early errors, The programmers usually receive feedback concerning his or her programming style and choice of algorithms and programming techniques. Other participants are also gain in similar way by being exposed to another programmers errors and programming style.
45
The inspection process is a way of identifying early the most error-prone sections of the program, thus allowing one to focus more attention on these sections during the computer based testing processes.
3.5 AN ERROR CHECK LIST FOR INSPECTIONS

An important part of the inspection process is the use of a checklist to examine the program for common errors. The checklist is largely language independent as most of the errors can occur with any programming language.
Data-Reference Errors
1. Is a variable referenced whose value is unset or uninitialized? This is probably the most frequent programming error; it occurs in a wide variety of circumstances. 2. For all array references, is each subscript value within the defined bounds of the corresponding dimension? 3. For all array references, does each subscript have an integer value? This is not necessarily an error in all languages, but it is a dangerous practice. 4. For all references through pointer or reference variables, is the referenced storage currently allocated? This is known as the dangling reference problem. It occurs in situations where the lifetime of a pointer is greater than the lifetime of the referenced storage. 5. Are there any explicit or implicit addressing problems if, on the machine being used, the units of storage allocation are smaller than the units of storage addressability? 6. If a data structure is referenced in multiple procedures or subroutines, is the structure defined identically in each procedure? 7. When indexing into a string, are the limits of the string exceeded?
Data-Declaration Error
1. Have all variables been explicitly declared? A failure to do so is not necessarily an error, but it is a common source of trouble. 2. If all attributes of a variable are not explicitly stated in the declaration, are the defaults well understood? 3. Where a variable is initialized in a declarative statement, is it properly initialized? 4. Is each variable assigned the correct length, type, and storage class?
46
5. Is the initialization of a variable consistent with its storage type?
Computation Errors
1. Are there any computations using variables having inconsistent (e.g. Nonarithmetic) data types? 2. Are there any mixed mode computations? 3. Are there any computations using variables having the same data type but different lengths? 4. Is the target variable of an assignment smaller than the right-hand expression? 5. Is an overflow or underflow exception possible during the computation of an expression? That is, the end result may appear to have a valid value, but an intermediate result might be too big or too small for the machines data representations. 6. Is it possible for the divisor in a division operation to be zero? 7. Where applicable, can the value of a variable go outside its meaningful range? 8. Are there any invalid uses of integer arithmetic, particularly division? For example, if I is an integer variable, whether the expression 2*I/2 is equal to I depends on whether I has an odd or an even value and whether the multiplication or division is performed first.
Comparison Errors
1. Are there any comparisons between variables having inconsistent data types (e.g. comparing a character string to an address)? 2. Are there any mixed-mode comparisons or comparisons between variables of different lengths? If so, ensure that the conversion rules are well understood. 3. Does each Boolean expression state what it is supposed to state? Programmers often make mistakes when writing logical expressions involving and, or, and not. 4. Are the operands of a Boolean operator Boolean? Have comparison and Boolean operators been erroneously mixed together?
Control-Flow Errors
1. If the program contains a multi way branch (e.g. a computed GO TO in Fortran), can the index variable ever exceed the number of branch possibilities? For example, in the Fortran statement, GOTO(200,300,400), I Will I always have the value 1,2, or 3?
47
2. Will every loop eventually terminate? Devise an informal proof or argument showing that each loop will terminate 3. Will the program, module, or subroutine eventually terminate? 4. Is it possible that, because of the conditions upon entry, a loop will never execute? If so, does this represent an oversight? For instance, for loops headed by the following statements: DO WHILE (NOTFOUND) DO I=X TO Z What happens if NOTFOUND is initially false or if X is greater than Z? 5. Are there any non-exhaustive decisions? For instance, if an input parameters expected values are 1, 2, or 3; does the logic assume that it must be 3 if it is not 1 or 2? If so, is the assumption valid?
Interface Errors
1. Does the number of parameters received by this module equal the number of arguments sent by each of the calling modules? Also, is the order correct? 2. Do the attributes (e.g. type and size) of each parameter match the attributes of each corresponding argument? 3. Does the number of arguments transmitted by this module to another module equal the number of parameters expected by that module? 4. Do the attributes of each argument transmitted to another module match the attributes of the corresponding parameter in that module? 5. If built-in functions are invoked, are the number, attributes, and order of the arguments correct? 6. Does the subroutine alter a parameter that is intended to be only an input value?
Input/Output Errors
1. If files are explicitly declared, are their attributes correct? 2. Are the attributes on the OPEN statement correct? 3. Is the size of the I/O area in storage equal to the record size? 4. Have all files been opened before use? 5. Are end-of-file conditions detected and handled correctly?
48
6. Are there spelling or grammatical errors in any text that is printed or displayed by the program?
3.6
WALKTHROUGHS
The code walkthrough, like the inspection, is a set of procedures and error-detection techniques for group code reading. It shares much in common with the inspection process, but the procedures are slightly different, and a different error-detection technique is employed. The walkthrough is an uninterrupted meeting of one to two hours in duration. The walkthrough team consists of three to five people to play the role of moderator, secretary (a person who records all errors found), tester and programmer. It is suggested to have other participants like:
l l l l l l
A highly experienced programmer, A programming-language expert, A new programmer (to give a fresh, unbiased outlook) The person who will eventually maintain the program, Someone from different project and Someone from the same programming team as the programmer.
The initial procedure is identical to that of the inspection process: the participants are given the materials several days in advance to allow them to study the program. However, the procedure in the meeting is different. Rather than simply reading the program or using error checklists, the participants play computer. The person designated as the tester comes to the meeting armed with a small set of paper test casesrepresentative sets of inputs (and expected outputs) for the program or module. During the meeting, each test case is mentally executed. That is, the test data are walked through the logic of the program. The state of the program (i.e. the values of the variables) is monitored on paper or a blackboard. The test case must be simple and few in number, because people execute programs at a rate that is very slow compared to machines. In most walkthroughs, more errors are found during the process of questioning the programmer than are found directly by the test cases themselves. QUESTIONS
1. Is code reviews are relevant to the software testing? Explain the process involved in a typical code review. 2. Explain the need for inspection and list the different types of code reviews. 3. Consider a program and perform a detailed review and list the review findings in detail.
49
CHAPTER 4
Test Case Design
Y
l l l l l l
Dynamic testing of Software Applications White box and black box testing Various techniques used in White box testing Various techniques used in black box testing Static program analysis Automation of testing process
4.2 INTRODUCTION
Software can be tested either by running the programs and verifying each step of its execution against expected results or by statically examining the code or the document against its stated requirement or objective. In general, software testing can be divided into two categories, viz. Static and dynamic testing. Static testing is a non-execution-based testing and carried through by mostly human effort. In static testing, we test, design, code or any document through inspection, walkthroughs and reviews as discussed in Chapter 2. Many studies show that the single most cost-effective defect reduction process is the classic structural test; the code inspection or walk-through. Code inspection is like proof reading and
49
50
developers will be benefited in identifying the typographical errors, logic errors and deviations in styles and standards normally followed. Dynamic testing is an execution based testing technique. Program must be executed to find the possible errors. Here, the program, module or the entire system is executed(run) and the output is verified against the expected result. Dynamic execution of tests is based on specifications of the program, code and methodology.
4.3 WHITE BOX TESTING

This testing technique takes into account the internal structure of the system or component. The entire source code of the system must be available. This technique is known as white box testing because the complete internal structure and working of the code is available. White box testing helps to derive test cases to ensure: 1. All independent paths are exercised at least once. 2. All logical decisions are exercised for both true and false paths. 3. All loops are executed at their boundaries and within operational bounds. 4. All internal data structures are exercised to ensure validity. White box testing helps to:
l l l l l l l
Traverse complicated loop structures Cover common data areas, Cover control structures and sub-routines, Evaluate different execution paths Test the module and integration of many modules Discover logical errors, if any. Helps to understand the code
Why the white box testing is used to test conformance to requirements?

l
Logic errors and incorrect assumptions most likely to be made when coding for special cases. Need to ensure these execution paths are tested.
51
May find assumptions about execution paths incorrect, and so make design errors. White box testing can find these errors. Typographical errors are random. Just as likely to be on an obscure logical path as on a mainstream path.
Bugs lurk in corners and congregate at boundaries
4.4 BASIS PATH TESTING

It is a testing mechanism proposed by McCabe. The aim is to derive a logical complexity measure of a procedural design and use this as a guide for defining a basic set of execution paths. Test cases, which exercise basic set, will execute every statement at least once.
4.4.1 Flow Graph Notation

Flow graph notation helps to represent various control structures of any programming language. Various notations for representing control flow are: (fig 4)
Fig 4.1: Notations used for control structures
On a flow graph:
l l l
Arrows called edges represent flow of control Circles called nodes represent one or more actions. Areas bounded by edges and nodes called regions.
52
l
A predicate node is a node containing a condition
Any procedural design/ program can be translated into a flow graph. Later the flow graph can be analyzed for various paths within it. Note that compound Boolean expressions at tests generate at least two predicate nodes and additional arcs. Example:
Fig 4.2: Control flow of a program and the corresponding flow diagram
4.4.2 Cyclomatic Complexity

The cyclomatic complexity gives a quantitative measure of the logical complexity. This value gives the number of independent paths in the basis set, and an upper bound for the number of tests to ensure that each statement is executed at least once. An independent path is any path through a program that introduces at least one new set of processing statements or a new condition (i.e., a new edge)
53
Fig 4.3 : Sample program and corresponding flow diagram
In Fig 3.3, the statements are numbered and the corresponding nodes also numbered with the same number. The sample program contains one DO and three nested IF statements. From the example we can observe that:
l
Cyclomatic Complexity of 4 can be calculated as: 1. Number of regions of flow graph, which is 4. 2. #Edges - #Nodes + 2, which is 11-9+2=4. 3. #Predicate Nodes + 1, which is 3+1=4.
The above complexity provides the upper bound on the number of tests cases to be generated or independent execution paths in the program. The independent paths(4 paths) for the program shown in fig 4.3 is given below:
l
Independent Paths: 1. 1, 8
54
2. 1, 2, 3, 7b, 1, 8 3. 1, 2, 4, 5, 7a, 7b, 1, 8 4. 1, 2, 4, 6, 7a, 7b, 1, 8 Cyclomatic complexity provides upper bound for number of tests required to guarantee the coverage of all program statements.
4.4.3. Deriving Test Cases

Test cases are designed in many ways. The steps involved for test case design are: 1. Using the design or code, draw the corresponding flow graph. 2. Determine the cyclomatic complexity of the flow graph. 3. Determine a basis set of independent paths. 4. Prepare test cases that will force execution of each path in the basis set. Note: some paths may only be able to be executed as part of another test.
4.4.4. Graph Matrices

Graph matrices can automate derivation of flow graph and determination of a set of basis paths. Software tools to do this can use a graph matrix. A sample graph matrix is shown is Fig 3.4. The graph matrix:
l l l
Is a square matrix with number of sides equal to number of nodes. Rows and columns of the matrix correspond to the number of nodes in the flow graph. Entries correspond to the edges.
The matrix can associate a number with each entry of the edge. Use a value of 1 to calculate the cyclomatic complexity. The cyclomatic complexity is calculated as follows:
l l
For each row, sum column values and subtract 1. Sum these totals and add 1.
55
Which is 4. Some other interesting link weight can be measured by the graph as:
l l l l
Probability that a link (edge) will be executed Processing time for traversal of a link Memory required during traversal of a link Resources required during traversal of a link
Fig 4.4: Example of a graph matrix
4.5 CONTROL STRUCTURE TESTING

In programs, conditions are very important and testing such conditions is more complex than other statements like assignment and declarative statements. Basic path testing is one example of control structure testing. There are many ways in which control structure can be tested.
56 4.5.1 Conditions Testing

Condition testing aims to exercise all logical conditions in a program module. Logical conditions may be complex or simple. Logical conditions may be nested with many relational operations. Can define:
l
Relational expression: (E1 op E2), where E1 and E2 are arithmetic expressions. For example, (x+y) (s/t), where x, y, s and t are variables. Simple condition: Boolean variable or relational expression, possibly proceeded by a NOT operator. Compound condition: composed of two or more simple conditions, Boolean operators and parentheses along with relational operators. Boolean expression: Condition without relational expressions.
l l
Normally errors in expressions can be due to one or all or the following:

l l l l l l
Boolean operator error Boolean variable error Boolean parenthesis error Relational operator error Arithmetic expression error Mismatch of types
Condition testing methods focus on testing each condition in the program of any type of conditions. There are many strategies to identify errors. Some of the strategies proposed include:
l l
Branch testing: Every branch is executed at least once. Domain Testing: Uses three or four tests for every relational operator depending on the complexity of the statement. Branch and relational operator testing: Uses condition constraints. Based on the complexity of the relational operators, many branches will be executed.
Example 1: C1 = B1 & B2
l
where B1, B2 are boolean conditions..
57
l l
Condition constraint of form (D1,D2) where D1 and D2 can be true (t) or false(f). The branch and relational operator test requires the constraint set {(t,t),(f,t),(t,f)} to be covered by the execution of C1.
Coverage of the constraint set guarantees detection of relational operator errors.
4.5.2. Data Flow Testing

First, a proper data flow diagram like control flow(see basis path flow) is drawn. Then selects test paths according to the location of definitions and use of variables. Any variables that have been defined in any program behaves in the following way: D: define the variable, normally defined in declarative section, U: use the variables which is defined earlier, in the program. K: kill the variable, which is another state of the variable at any time of the execution of the program. Any variable that is part of the program will undergo any of the above states. However, the sequence of states is important. We can avoid following anomalies during the program execution:
l l l l l l l l
DU: Normal, UK, UU: Normal, DD: Suspicious DK: Probable bug KD: Normal KK: Probable bug KU: bug UD: Normal
For example, DU: Normal means a variable is defined first and then used in the program which is normal behavior of the data flow in the program. DK: Probable bug means a variable is defined and then killed before using in the program. This may be bug as why the variable is defined and killed with out using in the program.
58 4.5.3. Loop Testing

Loops are fundamental to many algorithms. Loops can be categorized as, define loops as simple, concatenated, nested, and unstructured. Loops can be defined in many ways. Examples:
Fig 4.5 : Different types of Loops
To test the loops, following guidelines may be followed:

l
Simple Loops of size n:

q q q q q
Skip loop entirely Only one pass through the loop Two passes through the loop m passes through loop where m<n. (n-1), n, and (n+1) passes through the loop. This helps in testing the boundary of the loops.
Nested Loops
q
Start with inner loop. Set all other loops to minimum values.
59
q q q l
Conduct simple loop testing on inner loop. Work outwards and take the next nested loop. Continue until all loops are tested.
Concatenated Loops
q q
If independent loops, use simple loop testing. If dependent, treat as nested loops.
Unstructured loops
q
Dont test - redesign. This is known as poor design.
4.6 BLACK BOX TESTING

Functional tests examine the observable behavior of software as evidenced by its outputs, without any reference to internal functions. This kind of tests is from the user point of view, which means as if the user is testing as in the normal business functions.
l
Black box tests normally determine the quality of the software. It is an advantage to create the quality criteria from this point of view from the beginning. In black box testing, software is subjected to a full range of inputs and the outputs are verified for their correctness. Here, the structure of the program is immaterial. Black box testing technique can be applied once unit and integration testing is completed. It focuses on functional requirements. It is compliment to the white box testing.
l l l
The main objective of the black box testing is to find: 1. Incorrect Or Missing Functions 2. Interface Errors 3. Errors In Data Structures Or External Database Access 4. Performance Errors 5. Initialization and Termination Errors.
60
Some of the techniques used for black box testing are discussed below:
4.6.1 Equivalence Partitioning

The main objective of this method is to partitioning the input so that an optimal input data is selected. Steps to be followed are: 1. Divide the input domain into classes of data for which test cases can be generated. 2. Attempting to uncover classes of errors, if any. 3. Identify both right and wrong input data while partitioning the data. 4. Test the program for all types of data. Based on equivalence classes for input conditions. An equivalence class represents a set of valid or invalid states An input condition is either a specific numeric value, range of values, a set of related values, or a boolean condition. Equivalence classes can be defined by:
l
If an input condition specifies a range or a specific value, one valid and two invalid equivalence classes defined. If an input condition specifies a boolean or a member of a set, one valid and one invalid equivalence classes defined.
Test cases for each input domain data item developed and executed. This method uses less number of input data compared to exhaustive testing. However, the data for boundary values are not considered. This method though reduces significantly the number of input data to be tested, it does not test the combinations of the input data.
4.6.2 Boundary Value Analysis

It is observed that boundary points for all inputs are not tested properly. This leads to many errors. Large number of errors tends to occur at boundaries of the input domain. Boundary Value Analysis (BVA) leads to selection of test cases that exercise boundary values.
61
BVA complements equivalence partitioning i.e. select any element in an equivalence class, select those at the edge of the class. Examples: 1. For a range of values bounded by a and b, test (a-1), a, (a+1), (b-1), b, (b+1). 2. If input conditions specify a number of values n, test with (n-1), n and (n+1) input values. 3. Apply 1 and 2 to output conditions (e.g., generate table of minimum and maximum size). 4. If internal program data structures have boundaries (e.g., buffer size, table limits), use input data to exercise structures on boundaries. BVA and Equivalence partitioning both helps in testing the programs and covers most of the conditions. This method does not test the combinations of input conditions.
4.6.3 Cause Effect Graphing Techniques

Translation of natural language descriptions of procedures to software based algorithms is error prone. Example: From US Army Corps of Engineers: Executive Order 10358 provides in the case of an employee whose work week varies from the normal Monday through Friday work week, that Labor Day and Thanksgiving Day each were to be observed on the next succeeding workday when the holiday fell on a day outside the employees regular basic work week. Now, when Labor Day, Thanksgiving Day or any of the new Monday holidays are outside an employees basic workbook, the immediately preceding workday will be his holiday when the non-workday on which the holiday falls is the second non-workday or the non-workday designated as the employees day off in lieu of Saturday. When the non-workday on which the holiday falls is the first non-workday or the non-workday designated as the employees day off in lieu of Sunday, the holiday observance is moved to the next succeeding workday. How do you test code, which attempts to implement this? Cause-effect graphing attempts to provide a concise representation of logical combinations and corresponding actions. 1. Causes (input conditions) and effects (actions) are listed for a module and an identifier is assigned to each. 2. A cause-effect graph developed. 3. Graph converted to a decision table. 4. Decision table rules are converted to test cases.
62
Simplified symbology:
Fig 4.6: Representation of cause-effect nodes
4.6.4 Comparison Testing

l l l l l l
In some applications the reliability is critical. Redundant hardware and software may be used. For redundant software, use separate teams to develop independent versions of the software. Test each version with same test data to ensure all provisional identical output. Run all versions in parallel with a real-time comparison of results. Even if it will only run one version in final system, for some critical applications can develop independent versions and use comparison testing or back-to-back testing.
63
l l
When outputs of versions differ, each is investigated to determine if there is a defect. Method does not catch errors in the specification.
4.7 STATIC PROGRAM ANALYSIS

This strategy helps in identifying errors without executing the program. Peer reviewers and programmers will use this strategy to uncover probable static errors.
4.7.1 Program Inspections

Have covered in the previous Chapter.
4.7.2 Mathematical Program Verification

If the programming language semantics are formally defined, one can consider program to be a set of mathematical statements. We can attempt to develop a mathematical proof that the program is correct with respect to the specification. If the proof can be established, the program is verified and testing to check verification is not required. There are a number of approaches to proving program correctness. We will only consider axiomatic approach. Suppose that at points P(1), .. , P(n) assertions concerning the program variables and their relationships can be made. The assertions are a (1), ..., a(n). The assertion a(1) is about inputs to the program, and a(n) about outputs. We can now attempt, for k between 1 and (n-1), to prove that the statements between P(k) and P(k+1) transform the assertion a(k) to a(k+1). Given that a(1) and a(n) are true, this sequence of proofs shows partial program correctness. If it can be shown that the program will terminate, the proof is complete. Note: Students are requested to note this as an introductory section.
64 4.7.3 Static Program Analysers

Static analysis tools scan the source code to try to detect errors. The code does not need to be executed. Most useful for languages which do not have strong typing. It can check: 1. Syntax. 2. Unreachable code 3. Unconditional branches into loops 4. Undeclared variables 5. Uninitialised variables. 6. Parameter type mismatches 7. Uncalled functions and procedures. 8. Variables used before initialization. 9. Non-usage of function results. 10.Possible array bound errors. 11. Misuse of pointers.
4.8 AUTOMATED TESTING TOOLS

Automation of testing is the state of the art technique where in number of tools will help in testing program automatically. Programmers can use any tool to test his/her program and ensure the quality. There are number of tools are available in the market. Some of the tools which helps the programmer are: 1. Static analyser 2. Code Auditors 3. Assertion processors 4. Test file generators 5. Test Data Generators
65
6. Test Verifiers 7. Output comparators. Programmer can select any tool depending on the complexity of the program. QUESTION
1. What is black box testing? Explain 2. What are the different techniques are available to conduct black box testing? 3. Explain different methods available in white box testing with examples.
66
Chapter 5
Testing f or Specialized Environments
Y
l l l l
Concept of Graphic User Interface (GUI) Testing, GUI checklist for Windows, Data entry and related activities Testing Client/Server Architecture Testing documentation and help facilities
5.2 INTRODUCTION
The need for specialized testing approaches is becoming mandatory as computer software has become more complex. The White-box and black box testing methods are applicable across all environments, architectures and applications, but unique guidelines and approaches to testing are sometime important. We address the testing guidelines for specialized environments, architectures, and applications that are commonly encountered by software engineers.
5.3 TESTING GUIS

The growth of Graphical User Interfaces (GUIs) in various applications has become a challenge for
66
Chapter 5 - Testing for Specialized Environments
67
test engineers. Because of reusable components provided as part of GUI development environments, the creation of the user interface has become less time consuming and more precise. GUI is becoming mandatory for any application as users are used to it. Sometime, the user interface may be treated as a different layer and easily separated from the traditional functional or business layer. The design and development of user interface layer requires separate design and development methodology. Here the main problem is to understand the user psychology during the development time. Due to complexity of GUIs, testing and generating test cases has become more complex and tedious. Because of modern GUIs standards (same look and feel), common tests can be derived.
What are the guidelines to be followed which helps for creating a series of generic tests for GUIs?
Guidelines can be categorized into many operations. Some of them are discussed below: For windows:
l l l l l l l
Will the window open properly based on related typed or menu-based commands? Can the window be resized, moved, scrolled? Does the window properly regenerate when it is overwritten and then recalled? Are all functions that relate to the window available when needed? Are all functions that relate to the window available when needed? Are all functions that relate to the window operational? Are all relevant pull-down menus, tool bars, scroll bars, dialog boxes, and buttons, icons, and other controls available and properly represented? Is the active window properly highlighted? If multiple or incorrect mouse picks within the window cause unexpected side effects? Are audio and/or color prompts within the window or as a consequence of window operations presented according to specification? Does the window properly close?
l l l
For pull-down menus and mouse operations:

l l
Is the appropriate menu bar displayed in the appropriate context? Does the application menu bar display system related features (e.g. a clock display)
68
l l l l l l l l
Do pull-down operations work properly? Do breakaway; menus, palettes, and tool bars work properly? Are all menu functions and pull-down sub functions properly listed? Are all menu functions and pull-down sub functions properly listed? Are all menu functions properly addressable by the mouse? Are text typeface, size, and format correct? Is it possible to invoke each menu function using its alternative text-based command? Are menu functions highlighted (or grayed-out) based on the context of current operations within a window? Does each menu function perform as advertised? Are the names of menu functions self-explanatory? Is help available for each menu item, and is it context sensitive? Are mouse operations properly recognized throughout the interactive context? If multiple clicks are required, are they properly recognized in context? If the mouse has multiple buttons, are they properly recognized in context? Do the cursor, processing indicator (e.g. an hour glass or clock), and pointer properly change as different operations are invoked?
l l l l l l l
Data entry:
l l l l l l
Is alphanumeric data entry properly echoed and input to the system? Do graphical modes of data entry (e.g., a slide bar) work properly? Is invalid data properly recognized? Are data input messages intelligible? Are basic standard validation on each data is considered during the data entry itself? Once the data is entered completely and if a correction is to be done for a specific data, does the system requires entering the entire data again? Does the mouse clicks are properly used?
69
Does the help buttons are available during data entry?
In addition to the above guidelines, finite state modeling graphs may be used to derive a series of tests that address specific data and program objects that are relevant to the GUI.
5.4 TESTING OF CLIENT/SERVER ARCHITECTURES

Client/server architectures represent a significant challenge for software testers. The distributed nature of client/server environments, the performance issues associated with transaction processing, the potential presence of a number of different hardware platforms, the complexities of network communication, the need to service multiple clients from a centralized (or in some cases, distributed) database, and the coordination requirements imposed on the server all combine to make testing of C/S architectures and the software that reside within them considerably more difficult than testing standalone applications. In fact, recent industry studies indicate a significant increase in testing time and cost when C/S environments are developed.
5.5 TESTING DOCUMENTATION AND HELP FACILITIES

Errors in documentation can be as devastating to the acceptance of the program as errors in data or source e code. You must have seen the difference between following the user guide and getting results or behaviors that do not coincide with those predicted by the document. For this reason, documentation testing should be a meaningful part of every software test plan. Documentation testing can be approached in two phases. The first phase, formal technical review, examines the document for editorial clarity. The second phase, live test, users the documentation in conjunction with the use of the actual program. Some of the guidelines are discussed here:
l l l l l l l
Does the documentation accurately describe how to accomplish each mode of use? Is the description of each interaction sequence accurate? Are examples accurate and context based? Are terminology, menu descriptions, and system responses consistent with the actual program? Is it relatively easy to locate guidance within the documentation? Can troubleshooting be accomplished easily with the documentation? Are the document table of contents and index accurate and complete?
70
l
Is the design of the document (layout, typefaces, indentation, graphics) conducive to understanding and quick assimilation of information? Are all error messages displayed for the user described in more detail in the document? If hypertext links are used, are they accurate and complete?
l l
The only viable way to answer these questions is to have an independent third party to test the documentation in the context of program usage. All discrepancies are noted, and areas of document ambiguity or weakness are defined for potential rewrite. QUESTIONS
1. Explain the need for GUI testing and its complexity? 2. List the guidelines required for a typical tester during GUI testing? 3. Select your own GUI based software system and test the GUI related functions by using the listed guidelines in this Chapter.
71
Chapter 6
Sof tware Testing Strategies
6.1. LEARNING OBJECTIVES
Y
l l l l l l l l
Various testing Strategies in Software Testing Basic concept of Verification and Validation Criteria for Completion of Testing Unit Testing Integration Testing Validation Testing System Testing Debugging Process
6.2. INTRODUCTION
A strategy for software testing integrates software test case design methods into a well-planned series of steps that result in the successful construction of software. As important, a software testing strategy provides a road map for the software developer, the quality assurance organization, and the customer- a road map that describes the steps to be conducted as part of testing, when these steps are
71
72
planned and then undertaken, and how much effort, time, and resources will be required. Therefore any testing strategy must incorporate test planning, test case design, test execution, and resultant data collection and evaluation. A software testing strategy should be flexible enough to promote the creativity and customization that are necessary to adequately test all large software-based systems. At the same time, the strategy must be rigid enough to promote reasonable planning and management tracking as the project progresses. Shooman suggests these issues: In many ways, testing is an individualistic process, and the number of different types of tests varies as much as the different development approaches. For many years, our only defense against programming errors was careful design and the native intelligence of the programmer. We are now in an era in which modern design techniques are helping us to reduce the number of initial errors that are inherent in the code. Similarly, different test methods are beginning to cluster themselves into several distinct approaches and philosophies. These approaches and philosophies are what we shall call strategy. Different test methods begin to cluster into several distinct approaches and philosophies, which is called strategy. A testing strategy incorporates :
l l l l l
Test planning Test case design Test execution And Resultant data collection and evaluation A software testing strategy should be flexible enough to promote the creativity and customization that are necessary to adequately test all large software based systems. At the same time, the strategy must be rigid enough to promote reasonable planning and Management tracking as the project progresses.
Types of Testing
The level of test is the primary focus of a system and derives from the way a software system is designed and built up. Conventionally this is known as the V model, which maps the types of test to each stage of development.
73
Component Testing
Starting from the bottom the first test level is Component Testing, sometimes called Unit Testing. It involves checking that each feature specified in the Component Design has been implemented in the component. In theory an independent tester should do this, but in practise the developer usually does it, as they are the only people who understand how a component works. The problem with a component is that it performs only a small part of the functionality of a system, and it relies on co-operating with other parts of the system, which may not have been built yet. To overcome this, the developer either builds, or uses special software to trick the component into believing it is working in a fully functional system.
Interface Testing
As the components are constructed and tested they are then linked together to check if they work with each other. It is a fact that two components that have passed all their tests, when connected to each other produce one new component full of faults. These tests can be done by specialists, or by the developers. Interface Testing is not focussed on what the components are doing but on how they communicate with each other, as specified in the System Design. The System Design defines relationships between components, and this involves stating:
l
What a component can expect from another component in terms of services.
74
l l l
How these services will be asked for. How they will be given. How to handle non-standard conditions, i.e. errors.
Tests are constructed to deal with each of these. The tests are organised to check all the interfaces, until all the components have been built and interfaced to each other producing the whole system.
System Testing
Once the entire system has been built then it has to be tested against the System Specification to check if it delivers the features required. It is still developer focussed, although specialist developers known as systems testers are normally employed to do it. In essence System Testing is not about checking the individual parts of the design, but about checking the system as a whole. In effect it is one giant component. System testing can involve a number of specialist types of test to see if all the functional and nonfunctional requirements have been met. In addition to functional requirements these may include the following types of testing for the non-functional requirements:
l l l l l
Performance - Are the performance criteria met? Volume - Can large volumes of information be handled? Stress - Can peak volumes of information be handled? Documentation - Is the documentation usable for the system? Robustness - Does the system remain stable under adverse circumstances?
There are many others, the needs for which are dictated by how the system is supposed to perform.
Acceptance Testing
Acceptance Testing checks the system against the Requirements. It is similar to systems testing in that the whole system is checked but the important difference is the change in focus:
l l
Systems Testing checks that the system that was specified has been delivered. Acceptance Testing checks that the system delivers what was requested.
The customer, and not the developer should always do acceptance testing. The customer knows what is required from the system to achieve value in the business and is the only person qualified to make that judgement.
75
The forms of the tests may follow those in system testing, but at all times they are informed by the business needs.
Release Testing
Even if a system meets all its requirements, there is still a case to be answered that it will benefit the business. The linking of Business Case to Release Testing is looser than the others, but is still important. Release Testing is about seeing if the new or changed system will work in the existing business environment. Mainly this means the technical environment, and checks concerns such as:
l l l
Does it affect any other systems running on the hardware? Is it compatible with other systems? Does it have acceptable performance under load?
These tests are usually run the by the computer operations team in a business. The answers to their questions could have significant a financial impact if new computer hardware should be required, and adversely affect the Business Case. It would appear obvious that the operations team should be involved right from the start of a project to give their opinion of the impact a new system may have. They could then make sure the Business Case is relatively sound, at least from the capital expenditure, and ongoing running costs aspects. However in practise many operations teams only find out about a project just weeks before it is supposed to go live, which can result in major problems.
6.3 A STRATEGIC APPROACH TO SOFTWARE TESTING

Testing activity can be planned and conducted systematically; hence, to be very specific test case design methods are defined called as templates. A number of software testing has been proposed in the literature. All provide the software developer with a template for testing and all have the following generic characteristics.
l
Testing begins at module level or class or object level in object-oriented systems and works Outward toward the integration of the entire computer based system. Different techniques are appropriate at different points in time Testing is conducted by the developer of the software and, an independent test group for large projects Testing and debugging are different activities, but debugging must be accommodated in any testing strategy
l l
76
How should a strategy be?
l
A strategy for software testing must accommodate low-level tests that are necessary to verify that a small source code segment has been correctly implemented as well as high-level tests that validate major customer requirements. A strategy must provide guidance for the practitioner and a set of milestones for the manager. Because the steps of the test strategy occur at a time when deadline pressure begins to rise, progress must be measurable and problems must surface.
6.4 VERIFICATION AND VALIDATION

Software testing is one element of a broader topic that is often referred to as verification and validation(V&V). Verification refers to the set of activities that ensure that software correctly implements a specific function. Validation refers to a different set of activities that ensure that the software that has been built is traceable to customer requirements. Software testing is one element of a broader topic that is often referred to as verification and validation (V&V).
l l
Verification refers to the set of activities that ensure that correctly implements a specific function. Validation refers to a different set of activities that ensure that the software that has been built is traceable to customer requirements.
Boehm states like this. Verification: Are we building the product right Validation: Are we building the right product?
Software Engineering Methods Formal Technical Review Measurement
Quality
Standards And Procedures
SCM & SQA Fig 6.1: Achieving Software Quality
Testing
Fig 6.1. Achieving Software Quality
77
Fig 5.1 shows by application of methods and tools, effective formal technical reviews, and solid management and measurement all lead to quality that is confirmed during testing. Testing provides the last bastion from which quality can be assessed and, more pragmatically, errors can be uncovered. However, testing should not be viewed as a safety net. Quality cannot be tested it wont be when you begin testing and when finished testing Quality is incorporated throughout software process.
4 Note:
It is important to note that V&V encompass a wide array of SQA activities that include formal technical reviews, quality and configuration audits, performance monitoring, Simulation, feasibility study, documentation review, database review, algorithm analysis, development testing, qualification testing and installation testing. Although testing plays an extremely important role in V&V, many other activities are also necessary.
6.5 ORGANIZING FOR SOFTWARE TESTING

The software developer is always responsible for testing the individual units (modules) of the program ensuring that each performs the function for which it was designed. In many cases, the developer also conducts integration testing - A testing step that leads to construction of the complete program structure. Only after the software architecture is complete, does an independent test group (ITG) become involved? The role of an ITG is to remove the inherent problems associated with letting the builder test the thing that has been built. Independent testing removes the conflict of interest that may otherwise present. After all, personnel in the ITG team are paid to find errors. How ever, the software developer does not turn the program over to ITG and walk away. The developer and the ITG work closely throughout a software project to ensure that thorough tests will be conducted. While testing is conducted, the developer must be available to correct errors that are uncovered. The ITG is part of the software development project team in the sense that it becomes involved during the specification process and stays involved (planning and specifying test procedures) throughout a large project. However, in many cases the ITG reports to the SQA organization, there by achieving a degree of independence that might not be possible if it were a part of the software development organization.
78 6.6 A SOFTWARE TESTING STRATEGY

The software engineering process may be viewed as a spiral, illustrated in figure 6.2, Initially system engineering defines the roll of software and leads to software requirements and analysis, where the information domain, function, behavior, performance, constraints, and validation criteria for software are established. Moving inward along the spiral, we come to design and finally to coding. To develop computer software, We spiral in along streamlines that decrease the level of Abstraction on each turn. S R
D
System Engineering Requirements
Design Code U
C Unit Test Integration test Validation Test System test I V S T

Figure 5.2: Testing Strategy
The strategy for software testing may also be viewed in the context of the spiral. Unit testing begins at the vortex of the spiral and concentrates on each unit of the software as implemented in source code. Testing progresses by moving outward along the spiral to integration testing, where the focus is on design and the construction of the software architecture. Talking another turn outward on the spiral, we encounter Validation testing where requirements established as part of software requirements analysis are validated against the software that has been constructed. Finally, We arrive at system testing where the software and other system elements are tested as a whole. To test computer software, we spiral out along streamlines that broaden the scope of testing with each turn.
79
Considering the process from a procedural point of view testing within the context of software engineering is a series of four steps that are implemented sequentially. The steps are shown In Figure 5.3 initially tests focus on each module individually, assuring that it functions as a unit hence the name unit testing. Unit testing makes heavy use of white-box testing techniques, exercising specific paths in a modules control structure to ensure complete coverage and maximum error detection. Next, modules must be assembled or integrated to form the complete software package. Integration testing addresses the issues associated with the dual problems of verification and program construction. Black-box test case design techniques are most prevalent during integration, although a limited amount of white -box testing may be used to ensure coverage of major control paths. After the software has been integrated (constructed), sets of high-order test are conducted. Validation criteria (established during requirements analysis) must be tested. Validation testing provides final assurance that software needs all functional, behavioral and performance requirements. Black-box testing techniques are used exclusively during validation. The last high-order testing step falls outside the boundary of software engineering and into the broader context of computer system engineering. Software once validated must be combined with other system elements (e.g., hardware, people, and databases). System testing verifies the tall elements mesh properly and that overall system function/performance is achieved.
High order tests Requirements Integration test Design Coding
Unit test
Code
Testing Direction
Figure 5.3: Software Testing Steps
80
Criteria for completion of testing
Using statistical modeling and software reliability theory, models of software failures (uncovered during testing) as a function of execution time can be developed. Version of the failure model called a logarithmic Poisson execution-time model takes the form: f(t) = (1/p)ln(lo pt +1) Where f(t) lo p = cumulative number of failures that are expected to occur once the software has n has been tested for a certain amount of execution time, t = the initial software failure intensity (failures per unit time) at the beginning of testing = the exponential reduction in failure intensity as errors are uncovered and repairs are made. l(t) = lo(lo pt +1) (2) (1)
The instantaneous failure intensity, l(t) can be derived by taking the derivative of f(t), Using the relationship noted in equation 2, testers can predict the drop off of errors as testing progresses. The actual error intensity can be plotted against the predict curve figure 5.4. If the actual data gathered during testing and logarithmic Poisson execution-time model are responsibly close to one another over a number of data points, the model can be used to predict total testing time required to achieve an acceptable low failure intensity.
Data Collected during testing per test hour
Predicted failure intensity, l(t) lo
Failures
Execution time,
Figure5. 4: Failure intensity as a function of execution time
81
6.7 STRATEGIC ISSUES

Following issues must be addressed if a successful software strategy is to be implemented
l
Specify product requirements in a quantifiable manner long before testing commences. Although the overriding objective of testing is to find errors good testing strategy also assesses other quality characteristics such as portability, maintainability , usability .These should be specified in a way that is measurable so that testing results are unambiguous. State testing objectives explicitly. The specific objectives of testing should be stated in measurable terms for example, test effectiveness, test coverage, meantime to failure, the cost to find and fix defects, remaining defect density or frequency of occurrence, and test work hours per regression test should all be stated within the test plan. Understand the users of the software and develop a profile for each user category.use cases ,which describe interaction scenario for each class of user can reduce overall testing effort by focussing testing on actual use of the product. Develop a testing plan that emphasizes rapid cycle testing. The feedback generated from the rapid cycle tests can be used to control quality levels and corresponding test strategies. Build robust software that is designed to test itself. Software should be designed in a manner that uses antibugging techniques. that is software should be capable of diagnosing certain classes of errors. In addition, the design should accommodate automated testing regression testing. Use effective formal technical reviews as a filter prior to testing. formal technical reviews can be as effective as testing in uncovering errors. For this reason, reviews can reduce the amount of testing effort that is required to produce high-quality software. Conduct formal technical reviews to assess the test strategy and test cases themselves. Formal technical reviews can uncover inconsistencies, omissions, and outright errors in the testing approach. This saves time and improves product quality. Develop a continuous improvement approach for the testing process. The test strategy should be measured. The metrics collected during testing should be used as part of a statistical process control approach for software testing.
6.8 UNIT TESTING

Unit testing focuses verification efforts on the smallest unit of software design the module. Using the procedural design description as guide, important control paths are tested to uncover errors within the
82
boundary of the module . the relative complexity of tests and uncovered errors are limited by the constraint scope established for unit testing. The unit test is normally white-box oriented, and the step can be conducted in parallel for multiple modules.
6.8.1 Unit test consideration

The tests that occur as part of unit testing are illustrated schematically in figure 6.5. The module interface is tested to ensure that information properly flows into and out of the program unit under test. The local data structure is examined to ensure the data stored temporarily maintains its integrity during all steps in an algorithms execution.
Boundary conditions are tested to ensure that the module operates properly at boundaries established to limit or restrict processing. All independent paths through the control structure are exercised to ensure that all statements in a module have been executed at least once. And finally, all error-handling paths are tested. Tests of data flow across a module interface are required before any other test is initiat Tests of data flow across a module interface are required before any other test is initiated. If data do If data do not enter and exit properly, all other tests are doubtful. not enter and exit properly, all other tests are doubtful.
Module ----------------------------------
Interface Local data structures Boundary conditions Independent paths Error handling paths
Test Cases
Figure 6.5 : Unit Test
Checklist for interface tests
83
6.8.2 Checklist for interface tests

1. Number of input parameters equals to number of arguments. 2. Parameter and argument attributes match. 3. Parameter and argument systems match. 4. Number of arguments transmitted to called modules equal to number of parameters. 5. Attributes of arguments transmitted to called modules equal to attributes of parameters. 6. Unit system of arguments transmitted to call modules equal to unit system of parameters. 7. Number attributes and order of arguments to built-in functions correct. 8. Any references to parameters not associated with current point of entry. 9. Input-only arguments altered. 10.Global variable definitions consistent across modules. 11. Constraints passed as arguments.
When a module performs external I/O, following additional interface test must be conducted.
1. File attributes correct? 2. Open/Close statements correct? 3. Format specification matches I/O statements? 4. Buffer size matches record size? 5. Files opened before use? 6. End-of-File conditions handled? 7. I/O errors handled 8. Any textual errors in output information? The local data structure for a module is a common source of errors .Test cases should be designed to uncover errors in the following categories 1. Improper or inconsistent typing 2. erroneous initialization are default values
84
3. incorrect variable names 4. inconsistent data types 5. underflow, overflow, and addressing exception
Chapter 6 - Software Testing Strategies
In addition to local data structures, the impact of global data on a module should be ascertained during unit testing. Selective testing of execution paths is an essential task during the unit test. Test cases should be designed to uncover errors to erroneous computations; incorrect comparisons are improper control flow. Basis path and loop testing are effective techniques for uncovering a broad array of path errors. Among the more common errors in computation are : 1. misunderstood or incorrect arithmetic precedence 2. mixed mode operation 3. incorrect initialization 4. precision Inaccuracy 5. incorrect symbolic representation of an expression. Comparison and control flows are closely coupled to one another. Test cases should uncover errors like: 1. Comparison of different data types 2. Incorrect logical operators are precedence 3. Expectation of equality when precision error makes equality unlikely 4. Incorrect comparison or variables 5. Improper or non-existent loop termination. 6. Failure to exit when divergent iteration is encountered 7. Improperly modified loop variables. Good design dictates that error conditions be anticipated and error handling paths set up to reroute or cleanly terminate processing when an error does occur. Among the potential errors that should be tested when error handling is evaluated are: 1. Error description is unintelligible
MSIT 32 Software Quality and MSIT32 Software Quality and TestingTesting
85
2. Error noted does not correspond to error encountered 3. Error condition causes system intervention prior to error handling 4. Exception-condition processing is incorrect 5. Error description does not provide enough information to assist in the location of the cause of the error. Boundary testing is the last task of the unit tests step. software often files at its boundaries. That is, errors often occur when the nth element of an n-dimensional array is processed; when the ith repetition of a loop with i passes is invoke; or when the maximum or minimum allowable value is encountered. Test cases that exercise data structure, control flow and data values just below, at just above maxima and minima are Very likely to uncover errors.
6.8.3 Unit test procedures

Unit testing is normally considered as an adjunct to the coding step. After source-level code has been developed, reviewed, and verified for correct syntax, unit test case design begins. A review of design information provides guidance for establishing test cases that are likely to uncover errors in each of the categories discussed above. Each test case should be coupled with a set of expected results. Because a module is not a standalone program, driver and or stub software must be developed for each unit test. The unit test environment is illustrated in figure 5.6.In most applications a driver is nothing more than a Main program that accepts test case data, passes such data to the test module and prints relevant results. Stubs serve to replace modules that are subordinate to the module that is to be tested. A stub or dummy sub program uses the subordinate modules interface may do minimal data manipulation prints verification of entry, and returns. Drivers and stubs represent overhead. That is, both are software that must be developed but that is not delivered with the final software product. If drivers and stubs are kept simple, actual overhead is relatively low. Unfortunately, many modules cannot be adequately unit tested with simple overhead software. In such cases, Complete testing can be postponed until the integration test step (Where drivers or stubs are also used). Unit test is simplified when a module with high cohesion is designed. When a module addresses only one function, the number of test cases is reduced and errors can be more easily predicted and uncovered.
86
Chapter 6 - Software Testing Strategies
Driver
Module to be tested
Interface Local data structures Boundary conditions Independent paths Error handling paths
Stub
Stub
RESULTS
Figure 5.6 : Unit Test Environment
Test Cases
6.9 INTEGRATION TESTING

Integration testing is a systematic technique for constructing the program structure while conducting tests to uncover errors associated with interfacing. The objective is to take unit tested modules and build a program structure that has been dictated by design.
6.9.1 Different Integration Strategies

Integration testing is a systematic technique for constructing the program structure while conducting tests to uncover errors associated with interfacing. The objective is to take unit tested modules and build a program structure that has been dictated by design. There are often a tendency to attempt non-incremental integration; that is, to contrtruct the program using a big bang approach. All modules are combined in advance. The entire program is tested as a
MSIT 32 Software Quality and MSIT32 Software Quality and TestingTesting
87
whole. And chaos usually results! A set of errors is encountered. Correction is difficult because isolation of causes is complicated by the vast expanse of the entire program. Once these errors are corrected, new ones appear and the process continues in a seemingly endless loop. Incremental integration is the antithesis of the big bang approach. The program is constructed and tested in small segments, where errors are easier to isolate and correct; interfaces are more likely to be tested completely; and a systematic test approach may be applied. We discuss some of incremental methods here:
6.9.2 Top down integration

Top-down integration is an incremental approach to construction of program structure. Modules are integrated by moving downward through the control hierarchy, beginning with the main control module. The integration process is performed in a series of five steps: 1. The main control module is used as a test driver, and stubs are substituted for all modules directly subordinate to the main control module. 2. Depending on the integration approach selected (i.e., depth-or breadth first), subordinate stubs are replaced one at a time with actual modules. 3. Tests are conducted as each modules are integrated 4. On completion of each set of tests, another stub is replaced with real module 5. Regression testing may be conducted to ensure that new errors have not been introduced The process continues from step2 until the entire program structure is built. Top-down strategy sounds relatively uncomplicated, but in practice, logistical problems arise. The most common of these problems occurs when processing at low levels in the hierarchy is required to adequately test upper levels. Stubs replace low-level modules at the beginning of top-down testing; therefore, no significant data can flow upward in the program structure. The tester is left with three choices 1. Delay many tests until stubs are replaced with actual modules. 2. Develop stubs that perform limited functions that simulate the actual module 3. Integrate the software from the bottom of the hierarchy upward The first approach causes us to lose some control over correspondence between specific tests and incorporation of specific modules. this can lead to difficulty in determining the cause of errors tends to
88
violate the highly constrained nature of the top down approach. The second approach is workable but can lead to significant overhead, as stubs become increasingly complex. The third approach is discussed in next section.
6.9.3 Bottom -Up Integration

Modules are integrated from the bottom to top, in this approach processing required for modules subordinate to a given level is always available and the needs for subs is eliminated. A bottom-up integration strategy may be implemented with the following steps: 1. Low-level modules are combined into clusters that perform a specific software sub function. 2. A driver is written to coordinate test case input and output. 3. The cluster is tested. 4. Drivers are removed and clusters are combined moving upward in the program structure. As integration moves upward, the need for separate test drivers lessens. In fact, if the top two levels of program structure are integrated top-down, the number of drivers can be reduced substantially and integration of clusters is greatly simplified.
6.9.4 Regression Testing

Each time a new model is added as a part of integration testing, the software changes. New data flow paths are established, new I/O may occur, and new control logic is invoked. These changes may cause problems with functions that previously worked flawlessly. In the context of an integration test, strategy regression testing is the re-execution of subset of tests that have already been conducted to ensure that changes have not propagated unintended side effects. Regression testing is the activity that helps to ensure that changes do not introduce unintended behavior or additional errors.
How is regression test conducted?

Regression testing may be conducted manually, by re-executing a subset of all test cases or using automated capture playback tools. Capture-playback tools enable the software engineer to capture test cases and results for subsequent playback and comparison.
89
The regression test suite contains three different classes of test cases. 1. A representative sample of tests that will exercise all software functions. 2. Additional tests that focus on software functions that are likely to be affected by the change. 3. Tests that focus on software components that have been changed.
4 Note:
It is impractical and inefficient to re-execute every test for every program function once a change has occurred. Selection of an integration strategy depends upon software characteristics and some time project schedule. In general, a combined approach that uses a top-down strategy for upper levels of the program structure, coupled with bottom-up strategy for subordinate levels may be best compromise. Regression tests should follow on critical module function.
What is critical module?

A critical module has one or more of the following characteristics.
l l l l
Addresses several software requirements Has a high level of control Is a complex or error-prone Has a definite performance requirement.
6.9.5 Integration Test Documentation

An overall plan for integration of the software and a description of specific tests are documented in a test specification. The specification is deliverable in the software engineering process and becomes part of the software configuration.
Test Specification Outline

I. II. Scope of testing Test Plan 1. Test phases and builds 2. Schedule
90
3. Overhead software 4. Environment and resources III. Test Procedures 1. Order of integration
q q
Purpose Modules to be tested
2. Unit test for modules in build

q q q
Description of test for module n Overhead software description Expected results
3. Test environment
q q
Special tools or techniques Overhead software description
4. Test case data 5. Expected results for build IV. V. VI. Actual Test Results References Appendices
The Following criteria and corresponding tests are applied for all test phases. Interfaces integrity. Internal and external interfaces are tested as each module is incorporated into the structure. Functional Validity. Tests designed to uncover functional error are conducted. Information content. Tests designed to uncover errors associated with local or global data structures are conducted. Performance Test designed to verify performance bounds established during software design are conducted.
91
A schedule for integration, overhead software, and related topics are also discussed as part of the test Plan section. Start and end dates for each phase are established and availability windows for unit tested modules are defined. A brief description of overhead software(stubs and drivers) concentrates on characteristics that might require special effort. Finally, test environments and resources are described.
6.10 VALIDATION TESTING

At the culmination of integration testing, software is completely assembled as a package. Interfacing errors have been uncovered and corrected, and a final series of software tests - validation testing- may begin. Validation can be defined in many ways, but a simple definition is that validation succeeds when software functions in a manner that can be reasonable expected by customer.
What are reasonable expectations?

Reasonable expectations are defined in the software requirement specification - a document that describes all user-visible attributes of the software. The specification contains a section titled Validation Criteria. Information contained in that section forms the basis for a validation testing approach.
6.10.1 Validation Test Criteria

A test plans outlines the classes of tests to be conducted and a test procedure defines specific test cases that will be used in an attempt to uncover errors in conformity with requirements. Both the plan and procedure are designed to ensure that:
l l l l
All functional requirements are satisfied All performance requirements are achieved Documentation is correct and human-engineered and Other requirements like portability, error recovery, and maintainability are met.
6.10.2 Configuration Review

An important element of the validation process is a configuration review. The intent of the review is to ensure that all elements of the software configuration have been properly developed, are catalogued, and have the necessary detail to support the maintenance phase of the software life cycle.
92 6.10.3 Alpha and Beta Testing

If software is developed as a product to be used by many customers, it is impractical to perform formal acceptance tests with each one. Most software product builders use a process called alpha beta testing to uncover errors that only the end user seems able to find. The alpha test conducted at the developers site by a customer software is used in a natural setting with the developer Looking over the shoulder of the user and recording errors and usage problems. Alpha tests are conducted in a controlled environment. The beta test is conducted at one or more customer sites by the end user(S) of the software . Unlike alpha testing the developer is generally not present; therefore the beta test is live. Application of the software in an environment that cannot be controlled by the developer. The customer records all problems (real/imagined) that are encountered during beta testing and reports these to the developer at regular intervals. Because of problems reported during beta test, the software developer makes modification and then prepares for release of the software product to the entire customer base.
6.11 SYSTEM TESTING

System testing is actually a series of different tests whose primary purpose is to fully exercise the computer-based system. Although each test has a different purpose, all work to verify that all system elements have been properly integrated and perform allocated functions. A classic system testing problem is finger pointing. This occurs when an error is uncovered, and each system element developer blames the other for the problem. Rather than indulging in such nonsense, the software engineer should anticipate potential interfacing problems and 1) design error-handling paths that test all information coming from other elements of the system; 2) conduct a series of tests that simulate bad data or other potential errors at the software interface;3) record the results of tests to use as evidence if finger pointing does occur; and 4) participate in planning and design of system tests to ensure that software is adequately tested. In the section that follows, we discuss the types of system tests that are worthwhile for software based system.
6.11.1 Recovery Testing

Many computer-based systems must recover from faults and resume processing within a pre-specified
93
time. In some cases, a system must be fault tolerant; that is, processing faults must not cause overall system function to cease. In other cases, a system failure must be corrected within a specified period of time or severe economic damage will occur. Recovery testing is a system test that forces the software to fail in a variety of ways and verifies that recovery is properly performed. If recovery is automatic (performed by the system itself), re-initialization, check pointing, mechanism, data recovery, and restart are each evaluated for correctness. If recovery requires human intervention, the mean time to repair is evaluated to determine whether it is within acceptable limits.
6.11.2 Debugging
Software testing is a process that can be systematically planned and specified. Test case design can be conducted, a strategy can be defined, and results can be evaluated against prescribed expectations. Debugging occurs as a consequence of successful testing. That is, when a test case uncovers an error, debugging is the process that results in the removal of the error. Debugging is not testing, but it always occurs as consequence of testing as shown in figure 6.1
6.11.3 The Debugging Process

The debugging process begins with the execution of a test case. As shown in fig 5.7, the debugging process begins with the execution of a test case. Results are assessed and a lack of correspondence between expected and actual is encountered. In many cases, the non-corresponding data is a symptom of an underlying cause as yet hidden. The debugging process attempts to match symptom with cause, thereby leading to error correction. The debugging process attempts to match symptom with cause, there by leading to error correction. The debugging process will always have two outcomes: 1. The cause will be found, corrected, and removed 2. The cause will not be found. In the latter case, the person performing debugging may suspect a cause, design a test case to help validate his/her suspicion, and work toward error correction in iterative fashion.
94
Execution of Cases
Results Test Cases Additional Tests Suspected causes
Regression Tests Corrections

Identified Causes Figure 6.1: Debugging Process
Debugging
Why debugging is so difficult?

Some characteristics of bugs provide some clues: 1. The symptom and the cause may be geographically remote. That is, the symptom may appear in one part of a program, while the cause may actually be located at a site that is far removed. Highly coupled program structures exacerbate this situation. 2. The symptom may disappear(temporarily) when another error is corrected. 3. The symptom may actually be caused by no errors (e.g. round-off inaccuracies). 4. The symptom may be caused by human error that is not easily traced. 5. The symptom may be a result of timing problems, rather than processing problems. 6. It may be difficult to accurately reproduce input conditions(e.g. a real-time application in which input ordering is indeterminate).
95
7. The symptom may be due to causes that are distributed across a number of tasks running on different processors.
6.11.4 Debugging approach

In general, three categories for debugging approaches may be proposed.
l l l
Brute force Back tracking Cause elimination
The brute force category of debugging is probably the most common and efficient method for isolating the cause of a software error. Brute force debugging methods are applied when all methods of debugging fail. Using a philosophy, memory dumps are taken, run time traces are invoked and the program is loaded with WRITE statement. When this is done, one finds a clue by the information produced which leads to cause of an error. Backtracking is a common debugging approach that can be used successfully in small programs. Beginning at the site where a symptom has been uncovered, the source code is traced backward (manually) until the site of the cause is found. This process has a limitation when the source lines are more. Cause Elimination is manifested by induction or deduction and introduces the concept of binary partitioning. Data related to the error occurrence are organized to isolate potential causes. Alternatively, a list of all possible causes is developed and tests are conducted to eliminate each. If initial tests indicate that a particular cause hypothesis shows promise the data are refined in an attempt to isolate the bug.
6.12 SUMMARY
q
Software testing accounts for the largest percentage of technical effort in the software process. Yet, we are only beginning to understand the subtleties of systematic test planning, execution and control. The objective of software testing is to uncover errors. To fulfill this objective, a series of test step-unit, integration, validation, and system tests-are planned and executed.
q q
96
q
Unit and integration tests concentrate on functional verification of a module and incorporation of modules into a program structure. Validation testing demonstrates tractability to software requirements, and System testing validates software once it has been incorporated into a larger system. Each test step is accomplished through a series of systematic test techniques that assist in the design of test cases. With each testing step, the level of abstraction with which software is considered is broadened. Unlike testing, debugging must be viewed as an art. Beginning with a symptomatic indication of a problem, the debugging activity tracks down the cause of an error. Of the many resources available during debugging, the most valuable is the counsel of other software engineers. The requirement for higher-quality software demands a more systematic approach to testing.
q q q
QUESTIONS
1. What is the difference between Verification and Validation? Explain in your own words. 2. Explain unit test method with the help of your own example. 3. Develop an integration testing strategy for any the system that you have implemented already. List the problems encountered during such process. 4. What is validation test? Explain.
REFERENCES
1. Software Engineering, A practitioners Approach, Fourth Edition, by Roger S. Pressman, McGraw Hill. 2. Effective Methods of Testing, by William Perry, Wiley. 3. The Art of Software Testing, by Glenford J. Myers, John Wiley & Sons.
97
Chapter 7
Testing of Web Based Applications
7.1 INTRODUCTION
urrently the web is the most popular and fastest growing information system deployed on the internet, more than 80% of its traffic.
As of date, we can say that web based application deserve a high level of all software quality characters tics defined in the ISO standards namely: Functionality: Verified content of web must be ensured as well as fitness for intended purpose. Reliability: Security and availability are of utmost importance especially for applications that required trusted transactions or that must exclude the possibility that information is tampered. Efficiency: Response times are one of the success criteria for on-line services Usability: High user satisfaction is the basis for success Portability: Platform independence must be ensured at client level. Maintainability: High evolution speed of services requires that applications can be evolved very quickly.
7.2 TESTING OF WEB BASED APPLICATIONS: TECHNICAL PECULIARITIES

The following are the peculiarities of the web based applications :
MSIT 32
Software Quality and Testing
97
98
l
Web based applications consists of a large degree of components written by somebody else and integrated together with application software; User interface is often more complex than many GUI based client-server application Performance behavior is largely unpredictable and depends on many factors which are not under the control of the developers Security threats can come from anywhere We do not have only HTML but also Perl, Java, VRML etc. Browser compatibility is mandatory but is made difficult by layers and multi platforms Reference platforms are brand new and are being changed constantly Interoperability issues are magnified and thorough testing requires substantial investments in software and hardware.
l l
l l l l l
7.3 TESTING OF STATIC WEB- BASED APPLICATIONS

Test levels of static web sites are briefly summarized as follows :
Basic correctness/adherence to standards and guidelines

This refers to syntax, stylistic and lexical testing that has the goal to check the basic correctness of web sites. The common types of problems found in static web applications are
q q q
Syntax Problems Stylistic problems Lexical problems.
User interaction
User interaction covers
l l l l
Links, Fast loading, Compatibility Usability testing
99
The following guidelines provide the necessary steps to test links: Ensure that all hyper-links are valid and keep doing so, by means of continuous checks when the site is operational; Consistency check of internal and external links, as well as anchors; Internal Links shall be relative, to minimize the overhead and faults when the web site is moved to production environment. External link can change without control: thus automated regression testing shall be promoted. Avoid link that require parameter passing Check that content can be accessed by means of: search engine .site map, navigation structure.
Fast Loading Testing

Following aspects needs to be looked into :
l
Fast loading are concerning with aspects like the web pages the presence of a fast loading abstract/index, the presence of width and height attributes for IMG tag. Fast loading testing is very important if we consider that 85% of web users indicate slow loading times as the reason for avoiding further visits to web sites. Two approaches to have to be followed Reduce the size of the transferred data Optimise rendering and HTTP management The following rules related to the page weight should be established as support to fast loading testing Home page weight should be less than a specified size.(e.g. 45K) Every page weight should be less than a specified size (e.g. 50k) Graphical sugar pictures should be less than 3K (e.g. bullet header) Keep table test size to a minimum To reach greater sizes, use multiple tables separated one from the other Avoid nested tables Minimize pictures within the tables and always specify width and height
l l l l
l l
l l l l
100
l
Every page should contain text or other information before the first <TABLE> tag.
Compatibility testing
Compatibility testing concerns cross-Browser compatibility that checks for site behaviour across industry standard browsers and their recent versions. It checks that pages conform to W3C standards for HTML and other languages. It checks that site behaviour for java applets and Active X control. Cross platform java compatibility checks for the sites behaviour across industry standard desktop hardware and OS.
Usability Testing
Usability testing refers to coherence of look and feel, navigational aids, and user interactions and printing. These aspects must be tests with respect to normal behaviour, destructive behaviour and inexperienced users.
Structural Aspects
This includes both portability and integrity topics. All filenames in must be in lowercase which are in server side. Links to URLs outside in the web site be in canonical form and links to URLs into the web site must be in relative form. Moreover it must be checked that every directory must have an index page, every anchor must point to an existing page, and that are no limbo pages.
7.4 TESTING OF DYNAMIC WEB BASED APPLICATIONS

Integration /Security Testing
This includes web specific issues components and proxies and caching. A peculiar aspect of the testing of dynamic WWW applications is testing of cookies. The web is memory less system, with no concept of session. To overcome these issues you can use cookies: a small piece if information sent by a web server to store on a web browser so it can later be read back from browser. The problems related with testing are that cookies expire and that users can disable them in browser. Besides the basic security checking performed during the previous test level, specific security testing has to be performed when web applications make usage of sensitive data. Aspects to be covered by security testing will include: password security and authentication; Encryption of business transaction on WWW (SSL); Encryption of e-mails (PGP-Pretty Good Privacy); firewalls, Routers and Proxy servers.
101
Stress testing
Stress test is designed to confront programs with abnormal situations. In essence, the tester who performs stress testing asks: How high can we crank this up before it fails? Stress testing executes a system in a manner that demands resources in abnormal quantity, frequency, or volume. For example : 1. Special test may be designed that generate 10 interrupts per second when 1 or 2 is the average rate. 2. Input data rates may be increased by an order of magnitude to determine how input function will respond. 3. Test cases that require maximum memory or other resources may be executed 4. Test cases that may cause trashing in a virtual operating system may be designed. 5. Test cases that may cause excessive hunting for disk resident data may be created. Essentially the tester attempts to break the program. A variation of stress testing is a technique called sensitivity testing. In some situation, a very small range of data contained within the bounds of valid data for a program may cause extreme and even erroneous processing or profound performance degradation. This situation is analogous to a singularity in a mathematical function. Sensitivity testing attempts to uncover data combinations within valid input classes that may cause instability or improper processing.
Performance Testing
Performance testing is designed to test run-time performance testing occurs throughout all steps in the testing process. Even at the unit level, the performance of an individual module may be assessed as whitebox tests are conducted. However, it is not until all system elements are fully integrated that the true performance of a system can be ascertained. Performance tests are often coupled with stress testing and often require both hardware and software instrumentation. That is, it is often necessary to measure resource utilization in an exacting fashion. External instrumentation can monitor execution intervals, log events as they occur, and sample machine states on a regular basis. By incrementing a system, the tester can uncover situation that lead to degradation and possible system failure.
102 7.5 FUTURE CHALLENGES

Future challenges are related to the evolution of the WWW namely:
l l l l l l l l l l
HTTP-NG ( New generation HTTP protocol) Integration between TV and Web Emergence of XML New references for user interfaces like Evolution of HTML on MathML (for publishing Math) On SMIL for multimedia presentation SVG for publishing diagrams and vector based graphics Privacy issues Digital signatures Micro payments
These and other emerging technologies and services will require Internet testing approaches to be continually fine-tuned, to guarantee the reliability and quantity of service. QUESTIONS
1. What are the Technical Peculiarities in Web Site Testing? 2. What are the different levels of static web site testing? 3. What is Stress Testing 4. What is fast load testing? 5. What is integrity and security checking?
103
Chapter 8
Test Process Model
8.0 NEED FOR TEST PROCESS MODEL
he standard model for process assessment and improvement includes the main processes for the test processes. As a starting Point for all process improvement activities a more detailed process model is needed.
When looking at the software engineering Process a lot of models are available .They are documented and distributed and their use and tailoring needs are widely known and discussed .The Test Process is often characterized by trial and error implementation from people who are mainly experienced in SE and not in the testing area. In addition the models are not well documented and available to the QA responsible ones in the companies Based on the experiences of the problems known from the BOOTSTRAP assessments in many companies and the experiences more than 15 years in organizing and performing test in a wide area of companies a standard test process model was defined by SQS and integrated in the standard assessment method of the BOOTSTRAP.
MSIT 32
103
104 8.1 TEST PROCESS CLUSTER

Organizational Process Test Management Process Support Process Operative Test Processes
Fig 8.1: Test Process Cluster
Test process cluster is explained in the following sections:
8.1.1 Organisational Process

Test Process definition:
The purpose of the test process definition process is to establish the proceedings for a stable and repeatable test process and to document these in away apt for use.
Human Resource Management

The purpose of the human resource management process is to provide the organization and projects with individuals for testing who possess the skill and knowledge to perform their roles effectively.
Infrastructure management
The purpose of the infrastructure management process is to provide a stable and up-to-date environment with apt methods and tools for the software test and provide the testing staff with an environment for work. The processes of this cluster are dealing with framework which allows performing an efficient test.
105
8.1.2 Management Process

Test management process mainly consists of managing test, quality and risks.
Test management
The purpose of the test management process is to define the necessary processes for co-coordinating and managing a test project and the appropriate resources for testing a software product.
Quality and test strategy

The purpose of the quality and test strategy process is to identify the appropriate strategy to ensure the quality of the products and services of a project to satisfy the customer needs.
Risk Management
The purpose of the risk management process is to continuously identify and mitigate the project risks throughout the life cycle of the project. The process involves establishing a focus on management of risks at both the project and the organizational level. The processes of these clusters are dealing with the management of the test project with a focus on the appropriate organization of the process and a permanent watch on potential risks.
8.1.3 Support Processes

Test documentation
The purpose of the test documentation process is to establish a system for the documentation to assure the documentation of the test preparation and test protocols.
Configuration Management
The purpose of the configuration management process is to provide a mechanism for identifying, controlling and tracking the versions of all work products of test project or process.
Error and Change Management

The purpose of the error and change management process is to ensure that all deviations from the requirements are removed and all changes of the requirement are analyzed and managed and trends are identified.
Joint Reviews
The purpose of the joint reviews process is to maintain a common understanding with the customer of
106
the progress against reaching the customers goals and what should be done to help to ensure the development of a product that satisfies the customer. The processes in this cluster are dealing with the supporting process that provides techniques and infrastructure for a successful test project.
8.1.4 Operative test processes

To implement a well organized test process it is necessary to distinct between test activities in accordance to the test focus of each activity. As shown in figure 3 you can distinguish between the products of each engineering phase and therefore the tests can also be structure in accordance to these products. As a result the standard test process model contains 12 processes in these categories.
Test Documentation
The purpose of test of documentation process is to ensure that the documented work products of the project activities (e.g. requirement documents) comply with their fined requirements regarding form content;
Module Test
The purpose of the module test process is to ensure that modules of the software comply with their defined formal coding requirements and with the requirement of the software design.
Module integration Test

The purpose of the module integration test process is to ensure that the integrated software modules compiled with its defined requirements.
OO class test
The purpose of the OO class test is to ensure that the OO classes comply with their defined formal coding requirements and with the requirements of the software design.
OO class integration test

The purpose of the OO class integration test process is to ensure that integrated OO classes comply with their defined requirements.
Functional Test
The purpose of the functional test process is to ensure that the functions of the application fulfill their functional requirements.
107
Business Workflow test

The purpose of the business workflow test process is to ensure that the business workflow of the functions of the applications fulfill its requirements.
Unit interface test

The purpose of user interface test process is to ensure that the user interface of the application fulfill its requirement.
Performance Test
The purpose of the performance test process is to ensure that the performance of the application complies with its requirements.
Application interface test

The purpose of the application interface test process is to ensure the interfaces of the application with other systems comply with their requirements.
Installation Test
The purpose of the installation test process is to ensure that the deliverable application can be installed in the defined target environments.
Compatibility test
The compatibility test process is to ensure that the application is compatible with other specified application in the target environment.
How to improve test process?

The assessment results consists of two main results 1. The capability of each process 2. The Key Finding: Strength and Improvement areas As a starting point to derive an improvement strategy the evaluated goals and business strategies are analyzed .To support the defining the goals and their priorities as well as to measure the actual quality of the processes metrics are selected and adopted to the needs of the company. Based on these metrics and the analysis of the process capabilities, the strength and weaknesses of the test processes the improvement steps are defined. The selection of improvements from the improvement suggestion will be done by the assessed team, a management representative and will be supported by the
108
experienced assessor in an improvement workshop. The workshop ensures that the improvement areas are defined based on the needs and the experiences of the assessed organization and the knowledge of the experts. The next step is the concrete definitions of activities .The activities for the test improvement are planned and controlled like a normal software engineering project .The predefined metrics are used to define measures that can be used to measure the success of the improvement. Typical improvements for projects and organization that are starting with the test process improvement are
l l l
Guarantee independent QA responsibilities Implement a structure of testing phases with specified goals and criterias for the next phase Evaluation and implementation of tool support for testing activities
Fig 8.2 shows the basic steps on the way to test process improvement
SQA Test /Guidance
SQA Test / Analyze

SQA Test Metrics
SQA Test /Advice
Fig 8.2: Improvement Steps
QUESTIONS
1. Explain different test process and its benefits in software application development.
109
Chapter 10
Test Metrics
10.0 INTRODUCTION
he idea of understanding test metrics is to investigate the benefits of adopting some specific approach to testing.
If no formal, quantitative measurements are made, it is possible only to make qualitative statements about the effectiveness of the testing process, which may in the short term assure senior management, but which in the long term will not help to improve the testing process. Typical examples of where metrics can be used in the testing process include:
l l l
For objective assessment of testing quality against agreed standards For estimating the testing effort required to complete a given testing project/task. For highlighting complex elements of the system under test that may be error-prone and require additional testing effort. To measure and track the cost of testing. To assess and track the progress of the testing task. To predict when it is appropriate to stop testing a particular AUT.
l l l
Formally metrics are objective numerical measures obtained by inspection and analysis of the products and processes of a software project. Although can be collected at all stages of the SDLC, this chapter focuses on those of relevance to the testing process.
MSIT 32
109
110
This chapter discusses the role and use of metrics in process improvement, reviews the metrics employed in such programs , discusses the issues involved in setting up and adopting a metrics program, and makes a number of proposals for a simple and effective metrics set that can be used to improve the testing process.
10.1 OVERVIEW OF THE ROLE AND USE OF METRICS

Although the Statistical Techniques clause of ISO9001 provides guidance on the adoption of a metricsbased approach, there is a present no single universal standard for software metrics in the information technology or software development arena. However, a great deal of work has been performed in the IT industry on software metrics, and many metrics sets have been proposed and adopted by different authors and organizations.
10.2 PRIMITIVE METRIC AND COMPUTED METRICS

A primitive metric is one that can be measured directly Examples include 1. The Number of defects 2. The complexity of the AUT (Application under testing) 3. The cost of testing Primitive metrics typically form the raw data for a metrics program and will represent the observed data collected during the project. Often plotting the progress of primitive matrix over a time is a powerful means of observing the trends within the testing project. For example, plotting the numbers of defects detected during the testing projects against time can provide the test team leader with one means of determining when to stop testing by observing when the rate of detection of defects declines. A computed metric is one that must be calculated from other data or metrics Examples include 1. The number of non comment lines of code written per day 2. The defect density 3. The number of defects detected and or reported per unit time or per development phase Computed metrics typically form the basis of forming conclusion regarding the progress of a process improvement program. For example, observing the defect detection effectiveness percentage achieved by a testing team across a number of testing projects provides a valuable indication of the change in efficiency of the testing process over time.
111
10.3 METRICS TYPICALLY USED WITHIN THE TESTING PROCESS

This section reviews the commonly collected and calculated metrics that can be used in improving the testing process. Size of the AUT: This metric is typically measured as NCSS (Non Comment Source Statements).In an object Oriented or GUI- based development, information on the number of objects (and their methods) or windows could be collected. This Metric must be collected pragmatically it does not make good sense manually to count every line of code of a huge application (however, an automated approaches using a line-counting tool might be appropriate). Complexity of the AUT: This metrics attempt to take account of the complexity of the AUT in terms of iteration, recursion, conditional branching points, function points, and so on. Caution must be exercised in the use of this metric, since it will be difficult to produce generally applicable conclusions for organization involved in a number of software developments and testing projects, each of which utilizes different implementation technologies. This is a difficult metric to collect; However, complexity-estimating tools are commercially available that will simplify the collection and use of complexity information. Cost effort: This metric is typically measured as a payroll month and includes time taken by staff during testing, as well as time spent by managers engaged on testing tasks. It may also be of benefit to measure of the cost /effort involved in administrating and collecting the metrics since comparison of this value with the total effort expended in the testing process will provide information about the efficiency of the metrics program Total Defects Found in Testing :In this context, a defect can be defined as any aspect of the behavior of the software that would not exists if the software were fit for purpose .It may be of benefit to consider recording the severity of the observed defects to help in comparing their relative impact. A simple three-point scale of critical, serious, and minor is often sufficient for this purpose. Communications: This Metrics is typically measured as the number of interfaces that a given project team has come up with the purpose of characterizing the constraints on the project team due to dependencies with entities organizationally and physically distant, such as users, managers, and or the supplier. This metric is most useful for large and or complex organization where development and testing take place across geographically distinct sites. In particular metric can be used to identify possible improvements to the organization and administration of complex testing projects and provides a means for assessing the effectiveness of such improvements. For small testing projects involving few stuff located in the same office or site, there is unlikely to be a great deal of benefit from the use of this metric.
112 10.4 DEFECT DETECTION EFFECTIVENESS PERCENTAGE (DDE)

This computed metric will provide an indication of how effective the testing process is overtime (DDE should increase with an increase in the effectiveness of the testing).DDE is calculated as follows DDE = (TDFT/ (TDFC+TDFT)) X100, where TDFT g Total Defect Found by Testing team TDFC g Total Defect Found by Client(Measured up to some standard point After release-say after 6 months)
Defect Removal Effectiveness Percentage (DRE)

This computed metric will provide an indication of how effective is the testing task is at removal of defects. DRE is calculated as follows. DRE = (TDCT/TDFT) X100, where TDCT g Total Defects Closed during Testing TDFT g Total Defect Found by Testing team
Test Case design efficiency percentage

This compound metric will provide information about the effectiveness of the test case design process .TDE is calculated as follows. TDE = TDFC/NTC X100 where TDFT g Total Defect Found by Testing team NTC g Number of Test Cases Run.
10.5 SETTING UP AND ADMINISTERING A METRICS PROGRAM

The following requirements need to be considered in setting up a typical metrics program, and should be reviewed to determine those appropriate within the context of the scheme planned as part of any specific testing process.
l
The need to define organizational objectives for the process-improvement program to establish:
q
the methods to be used
113
q q q l
the cost that will be deemed acceptable in running the program the urgency of the program the required support from the management for the program
The need to establish the following roles and responsibilities

q q q
what part of the organization will have responsibility for the program? who will manage implement, and administer the program? who will submit/collect the metrics
The need to research which metrics to record and what analysis is required .during process the following issues must be considered.
q
Whatever metrics are selected initially will almost certainly change. They will be modified by experience during the metrics program itself. Ensure that results of analysis are immediately useful to project management. If you continue to collect metrics with no visible benefit, management will become discouraged with the process. Be aware of what is realistic (For example, although of potential interest , the number of keystrokes made by an engineer is not a realistic metric to record) Err on the side of generosity when selecting metrics to collect. It may be difficult to go back and obtain metrics for a parameter that was initially not thought to be of interest, but which subsequently turns out to be useful .You can always drop a metric later if it turns out to be of no use. Be aware that the more metrics you decide to record, the greater the effort to collect , the greater the project resistance to the process , the larger the requirements for storage and processing of the metrics
The need to set up the infrastructure for recording the metrics .The following steps are likely to be implemented:
q q
Set up a system for holding the metrics Generate simple procedures for defining what metrics are to be collected and how this is to be achieved. And formally document them in a form that can be distributed and easily understood.
QUESTIONS
1. 2. What is test process model? Explain. Explain the defect prevent mechanism.

Subraya - Software Quality &amp; Testing

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Subraya - Software Quality &amp; Testing

Uploaded by

Copyright:

Available Formats

MSIT 32 Software Quality and Testing

SOFTWARE QUALITY & TESTING

Dr. B.N. Subraya

PDF created with pdfFactory Pro trial version www.pdffactory.com

PDF created with pdfFactory Pro trial version www.pdffactory.com

MSIT 32 Software Quality and Testing

PDF created with pdfFactory Pro trial version www.pdffactory.com

PDF created with pdfFactory Pro trial version www.pdffactory.com

MSIT 32 Software Quality and Testing

6.9 6.10 6.11 6.12

PDF created with pdfFactory Pro trial version www.pdffactory.com

PDF created with pdfFactory Pro trial version www.pdffactory.com

MSIT 32 Software Quality and Testing

Introduction to Software Testing

1.1 LEARNING OBJECTIVES

ou will learn about:

PDF created with pdfFactory Pro trial version www.pdffactory.com

Chapter 1 - Introduction to Software Testing

Need for Testing

PDF created with pdfFactory Pro trial version www.pdffactory.com

MSIT 32 Software Quality and Testing

1.3 WHAT IS TESTING?

PDF created with pdfFactory Pro trial version www.pdffactory.com

Chapter 1 - Introduction to Software Testing

Reasons for Software Bugs

PDF created with pdfFactory Pro trial version www.pdffactory.com

MSIT 32 Software Quality and Testing

1.4 APPROACHES TO TESTING

PDF created with pdfFactory Pro trial version www.pdffactory.com

Chapter 1 - Introduction to Software Testing

1.5 IMPORTANCE OF TESTING

1.6 HURDLES IN TESTING

PDF created with pdfFactory Pro trial version www.pdffactory.com

MSIT 32 Software Quality and Testing

1.7 TESTING FUNDAMENTALS

1.7.1 Testing Objectives

PDF created with pdfFactory Pro trial version www.pdffactory.com

Chapter 1 - Introduction to Software Testing

1.7.2 Test Information Flow

In the above figure:

1.7.3 Test Case Design

PDF created with pdfFactory Pro trial version www.pdffactory.com

MSIT 32 Software Quality and Testing

Consider the following example shown in fig 1.3,

PDF created with pdfFactory Pro trial version www.pdffactory.com

Chapter 2 - Software Quality Assurance

Software Quality Assurance

2.1 LEARNING OBJECTIVES

ou will learn about:

Chapter 2 - Software Quality Assurance

PDF created with pdfFactory Pro trial version www.pdffactory.com

MSIT 32 Software Quality and Testing

Software Quality Assurance encompasses

Software quality is achieved as shown in figure 2.1:

Software Engineering Methods

Standards And And

SCM & & SQA Figure 2.1: Achieving Software Quality

2.3 QUALITY CONCEPTS

Quality Quality control Quality assurance Cost of quality

PDF created with pdfFactory Pro trial version www.pdffactory.com

Chapter 2 - Software Quality Assurance

Quality of design Quality of conformance

2.4 QUALITY OF DESIGN

2.5 QUALITY OF CONFORMANCE

Subraya - Software Quality & Testing

Subraya - Software Quality & Testing