ISYS6338– Testing and System Implementation

Session 8: Review for Testing

Project
 Learning Outcomes

• LO 3: Design the testing implementation plan for a software

 References

• Burnstein, Ilene. (2003). Practical Software Testing. Springer.

New York. ISBN: 0-387-95131-8

• Black, Rex & Mitchell, Jamie. (2011). Advanced Software Testing.

3. Rocky Nook. Santa Barbara, USA. ISBN: 978-1-933952-39-0
 Sub Topics

• The Principles of Reviews

• Type of Reviews
• Success Factor for Reviews
• Developing a Review Program
• The Need of Review Policies
• Components of Review Plans
• Review Checklist
• Reporting Review Results
The Principles of Reviews
 Review Principles

• A review is a type of static test. The object being reviewed is

not executed or run during the review. Like any test activity,
reviews can have various objectives.

• Reviews usually precede dynamic tests. They should

complement dynamic tests. Because the cost of a defect
increases the longer that defect remains in the system,
reviews should happen as soon as possible.

• Because reviews are so effective when done properly,

organizations should review all important documents. That
includes test documents: test plans, test cases, quality risk
analyses, bug reports, test status report, you name it.
 What Can Happen
after a Review?
• There are three possible outcomes.
– The ideal case is that the document is okay as is or with
minor changes.
– Another possibility is that the document requires some
non-trivial changes but not a rereview.
– The most costly outcome—in terms of both effort and
schedule time—is that the document requires extensive
changes and a re-review.

• Now, when that happens, keep in mind that while this is a

costly outcome, it’s less costly than simply ignoring the serious
problems and then dealing with them during component,
integration, system, or—worse yet—acceptance testing.
 Essential Roles and
Responsibilities
• During a formal review, there are some essential roles and
responsibilities:
• The manager. The manager allocates resources, schedules
reviews, and the like. However, the manager might not be
allowed to attend based on the review type.
• The moderator or leader: This is the chair of the review
meeting.
• The author: This is the person who wrote the item under
review. A review meeting, done properly, should not be a sad
or humiliating experience for the author.
• The reviewers: These are the people who examine the item
under review, possibly finding defects in it. Reviewers can play
specialized roles based on their expertise or based on some
type of defect they should target.
• The scribe or secretary or recorder: This is the person who
Type of Reviews
 Types of Review

• At the lowest level of formality (and, usually, defect removal

effectiveness), we find the informal review. This can be as
simple as two people, the author and a colleague, discussing a
design document over the phone.

• Technical reviews are more formalized, but still not highly

formal.

• Walk-throughs are reviews where the author is the moderator

and the item under review itself is the agenda. That is, the
author leads the review, and in the review, the reviewers go
section by section through the item under review.

• Inspections are the most formalized reviews. The roles are

well defined. Managers may not attend. The author may be
 Six Phases for Formal Review

1. Planning, including for 2. Kickoff, again for

each work product to be
reviewed and for all reviews each work product
to occur on a project and for all reviews

3. Individual preparation
for each work product;
reading the document
and noting problems

4. The review 5. Any rework

necessary based on
meeting itself, for the changes required
each work product by the review results

6. Finally, follow-up both for

Success Factor for Reviews
 Preparing the Review

• You don’t have to wait until a document is done before you

start reviewing it. You can and should review early drafts or
partial documents when you’re dealing with something critical.
This can help to identify and prevent patterns of defects
before they are built into the whole document.

• That said, make sure you have some rules about what it
means for something to be ready for review.

• Checklists are helpful for reviews. It’s too easy to forget

important areas without them. Have some checklists.
 Marick’s Code Review
Checklist
• Technical test analysts are likely to be invited to code reviews.
So, let’s go through Brian Marick’s code review checklist, which
he calls a “question catalog.” This catalog has several
categories of questions that developers should ask
themselves when going through their code. These questions
are useful for many procedural and object-oriented
programming languages, though in some cases certain
questions might not apply.

• However, customization based on your own experience, and

your organization’s needs, is encouraged.
 Marick’s Code Review
Checklist (Cont.)
• For variable declarations, Marick says we should ask the
following questions:
– Are the literal values correct? How do we know?
– Has every single variable been set to a known value before first use?
When the code changes, it is easy to miss changing these.
– Have we picked the right data type for the need? Can the value ever go
negative?

• For each data item and operations on data items, Marick says
we should ask the following questions:
– Are all strings NULL terminated? If we have shortened or lengthened a
string, or processed it in any way, did the final byte get changed?
– Did we check every assignment to a buffer for length?
– When using bitfields, are our manipulations (shifts, rotates, etc.) going to
be portable to other architectures and endian schemes?
– Does every sizeof() function call actually go to the object we meant it to?
 Marick’s Code Review
Checklist (Cont.)
• For every allocation, deallocation, and reallocation of memory,
Marick says we should ask the following questions:
– Is the amount of memory sufficient to the purpose without being
wasteful?
– How will the memory be initialized?
– Are all fields being initialized correctly if it is a complex data structure?
– Is the memory freed correctly after use?
– Do we ever have side effects from static storage in functions or methods?
– After reallocating memory, do we still have any pointers to the old
memory
– location?
– Is there any chance that the memory might be freed multiple times?
– After deallocation, are there still pointers to the memory?
– Are we mistakenly freeing data we don’t mean to?
– Is it possible that the pointer we are using to free the memory is already
NULL?
 Marick’s Code Review
Checklist (Cont.)
• For all operations on files, Marick says we should ask the
following questions:
– Do we have a way of ensuring that each temp file we create is unique?
– Is it possible to reuse a file pointer while it is pointing to an open file?
– Do we recover each file handle when we are done with it?
– Do we close each file explicitly when we are done with it?

• For every computation, Marick says we should ask the

following questions:
– Are parentheses correct? Do they mean what we want them to mean?
– When using synchronization, are we updating variables in the critical
sections together?
– Do we allow division by zero to occur?
– Are floating point numbers compared for exact equality?
 Marick’s Code Review
Checklist (Cont.)
• For every operation that involves a pointer, Marick says we
should ask the following questions:
– Is there any place in the code where we might try to dereference a NULL
pointer?
– When dealing with objects, do we want to copy pointers (shallow copy) or
content (deep copy)?

• For all assignments, Marick says we should ask the following

question:
– Are we assigning dissimilar data types where we can lose precision?

• For every function call, Marick says we should ask the

following questions:
– Was the correct function with the correct arguments called?
– Are the preconditions of the function actually met?

• Finally, Marick provides a couple of miscellaneous questions:

– Have we removed all of the debug code and bogus error messages?
– Does the program have a specific return value when exiting?
 OpenLaszlo Code Review
Checklist
• In the previous section, we discussed Marick’s questions that
should be asked about the code itself. In this section, we will
discuss questions that should be asked about the changes to
the system. These are essentially meta-questions about the
changes that occurred during maintenance. These come from
the OpenLaszlo website.
 OpenLaszlo Code Review
Checklist (Cont.)
• For all changes in code, here are the main questions we
should ask:
– Do we understand all of the code changes that were made and the
reasons for them?
– Are there test cases for all changes? Have they been run?
– Were the changes formally documented as per our guidelines?
– Were any unauthorized changes slipped in?

• In terms of coding standards, here are some additional

questions to ask (assumingyou are not enforcing coding
standards via static analysis):
– Do all of the code changes meet our standards and guidelines? If not, why
not?
– Are all data values to be passed parameterized correctly? In terms of
design changes, here are the questions to ask:
– Do you understand the design changes and the reasons they were made?
– Does the actual implementation match the designs?
 OpenLaszlo Code Review
Checklist (Cont.)
• Here are the maintainability questions to ask:
– Are there enough comments? Are they correct and sufficient?
– Are all variables documented with enough information to understand
why they were chosen?

• Finally, here are the documentation questions included in the

OpenLaszlo checklist:
– Are all command-line arguments documented?
– Are all environmental variables needed to run the system defined and
documented?
– Has all user-facing functionality been documented in the user manual and
help file?
– Does the implementation match the documentation?
Developing a Review Program
 Developing a Review
Program
• Reviews are an effective tool used along with execution-based
testing to support defect detection, increased software
quality, and customer satisfaction. Reviews should be used for
evaluating newly developing products as well as new releases
or versions of existing software. If reviews are conducted on
the software artifacts as they are developed throughout the
software life cycle, they serve as filters for each artifact.
 Developing a Review
Program (Cont.)
• A multiple set of filters allows an organization to separate out
and eliminate defects, inconsistencies, ambiguities, and low-
quality components early in the software life cycle. If we
compare the process of defect detection during reviews and
execution-based testing/debugging we can see that the review
process may be more efficient for detecting, locating, and
repairing these defects, especially in the code, for the
following reasons:
– 1. When testing software, unexpected behavior is observed because of a
defect(s) in the code. The symptomatic information is what the developer
works with to find and repair (fix) the defect.
– 2. Reviews also have the advantage of a two-pass approach for defect
detection. Pass 1 has individuals first reading the reviewed item and pass
2 has the item read by the group as a whole. If one individual reviewer did
not identify a defect or a problem, others in the group are likely to find it.
– 3. Inspectors have the advantage of the checklist which calls their
attention to specific areas that are defect prone. These are important
clues. Testers/developers may not have such information available.
The Need of Review Policies
 Review-related Policies

(i) testing policies with an emphasis on defect detection and quality, and measurements for controlling and monitoring;

(ii) a test organization with staff devoted to defect detection and quality issues;

(iii) policies and standards that define requirements, design, test plan, and other documents;

(iv) organizational culture with a focus on quality products and quality processes.
 Review-related Policies
(Cont.)
• Review policies should specify when reviews should take
place, what is to be reviewed, types of reviews that will take
place, who is responsible, what training is required and what
the review deliverables are.
• Review procedures should define the steps and phases for
each type of review.
• Policies should ensure that each project has an associated
project plan, test plan, configuration management plan, and a
review plan, and/or a software quality assurance plan.
• Project plans and the review plans should ensure that
adequate time and resources are available for reviews and
that cycle time is set aside for reviews.
• Managers need to follow-up and enforce the stated policies.
Only strong managerial commitment will lead to a successful
review program.
Components of Review Plans
 Review Plan Components

• Reviews are development and maintenance activities that

require time and resources. They should be planned so that
there is a place for them in the project schedule. An
organization should develop a review plan template that can
be applied to all software projects. The template should
specify the following items for inclusion in the review plan.
– review goals;
– items being reviewed;
– preconditions for the review;
– roles, team size, participants;
– training requirements;
– review steps;
– checklists and other related documents to be disturbed to participants;
– time requirements;
– the nature of the review log and summary report;
– rework and follow-up.
 Review Goals

• As in the test plan or any other type of plan, the review

planner should specify the goals to be accomplished by the
review and include:
i. identification of problem components or components in the software
artifact that need improvement,
ii. identification of specific errors or defects in the software artifact,
iii. ensuring that the artifact conforms to organizational standards, and
iv. communication to the staff about the nature of the product being
developed.

• Additional goals might be to establish traceability with other

project documents, and familiarization with the item being
reviewed. Goals for inspections and walkthroughs are usually
different; those of walkthroughs are more limited in scope
and are usually confined to identification of defects.
 Preconditions and Items
to Be Reviewed
• Given the principal goals of a technical review—early defect
detection, identification of problem areas, and familiarization
with software artifacts—many software items are candidates
for review. In many organizations the items selected for
review include:
– requirements documents;
– design documents;
– code;
– test plans (for the multiple levels);
– user manuals;
– training manuals;
– standards documents.
 Preconditions and Items
to Be Reviewed (Cont.)
• The preconditions need to be described in the review policy
statement and specified in the review plan for an item.
General preconditions for a review are:
– the review of an item(s) is a required activity in the project plan.
(Unplanned reviews are also possible at the request of management, SQA
or software engineers. Review policy statements should include the
conditions for holding an unplanned review.)
– a statement of objectives for the review has been developed;
– the individuals responsible for developing the reviewed item indicate
readiness for the review;
– the review leader believes that the item to be reviewed is sufficiently
complete for the review to be useful.
 Preconditions and Items
to Be Reviewed (Cont.)
• The review planner must also keep in mind that a given item
to be reviewed may be too large and complex for a single
review meeting. The smart planner partitions the review item
into components that are of a size and complexity that allows
them to be reviewed in 1–2 hours. This is the time range in
which most reviewers have maximum effectiveness. For
example, the design document for a procedure-oriented
system may be reviewed in parts that encompass:
i. the overall architectural design;
ii. data items and module interface design;
iii. component design.

• If the architectural design is complex and/or the number of

components is large, then multiple design review sessions
should be scheduled for each. The project plan should have
time allocated for this.
 Roles, Participants, Team
Size, and Time Requirements
• Two major roles that need filling for a successful review are (i)
a leader or moderator, and (ii) a recorder.
• Some of the responsibilities of the moderator have been
described. These include planning the reviews, managing the
review meeting, and issuing the review report. Because of
these responsibilities the moderator plays an important role;
the success of the review depends on the experience and
expertise of the moderator.
• Reviewing a software item is a tedious process and requires
great attention to details. The moderator needs to be sure
that all are prepared for the review and that the review
meeting stays on track. Reviewers often tire and become less
effective at detecting errors if the review time period is too
long and the item is too complex for a single review meeting.
The moderator/planner must ensure that a time period is
selected that is appropriate for the size and complexity of the
 Roles, Participants, Team Size,
and Time Requirements (Cont.)
• There is no set value for a review time period, but a rule of
thumb advises that a review session should not be longer
than 2 hours. Review sessions can be scheduled over 2-hour
time periods separated by breaks. The time allocated for a
review should be adequate enough to ensure that the
material under review can be adequately covered.

• The review recorder has the responsibility for documenting

defects, and recording review findings and recommendations,
Other roles may include a reader who reads or presents the
item under review. Readers are usually the authors or
preparers of the item under review. The author(s) is
responsible for performing any rework on the reviewed item.
In a walkthrough type of review, the author may serve as the
moderator, but this is not true for an inspection. All reviewers
should be trained in the review process.
 Roles, Participants, Team Size,
and Time Requirements (Cont.)
• The size of the review team will vary depending type, size, and
complexity of the item under review. Again, as with time, there
is no fixed size for a review team. In most cases a size
between 3 and 7 is a rule of thumb, but that depends on the
items under review and the experience level of the review
team. Of special importance is the experience of the review
moderator who is responsible for ensuring the material is
covered, the review meeting stays on track, and review
outputs are produced. The minimal team size of 3 ensures
that the review will be public.
 Review Roles

Source: Burnstein (2003, pg. 318)

 Review Procedures

• For each type of review that an organization wishes to

implement, there should be a set of standardized steps that
define the given review procedure. These are initiation,
preparation, inspection meeting, reporting results, and rework
and follow-up. For each step in the procedure the activities
and tasks for all the reviewer participants should be defined.
The review plan should refer to the standardized procedures
where applicable.
 Review Training

• Review participants need training to be effective.

Responsibility for reviewer training classes usually belongs to
the internal technical training staff. Alternatively, an
organization may decide to send its review trainees to
external training courses run by commercial institutions.
Review participants, and especially those who will be review
leaders, need the training. Test specialists should also receive
review training.
 Review Training Topics

Source: Burnstein (2003, pg. 321)

Review Checklist
 Review Checklists
• Checklists are very important for inspectors. They provide
structure and an agenda for the review meeting. They guide
the review activities, identify focus areas for discussion and
evaluation, ensure all relevant items are covered, and help to
frame review record keeping and measurement. Reviews are
really a two-step process: (i) reviews by individuals, and (ii)
reviews by the group. The checklist plays its important role in
both steps. The first step involves the individual reviewer and
the review material. Prior to the review meeting each
individual must be provided with the materials to review and
the checklist of items. It is his responsibility to do his
homework and individually inspect that document using the
checklist as a guide, and to document any problems he
encounters. When they attend the group meeting which is the
second review step, each reviewer should bring his or her
individual list of defect/problems, and as each item on the
checklist is discussed they should comment. Finally, the
reviewers need to come to a consensus on what needs to be
 Example Components
for an Inspection Checklist

Source: Burnstein (2003, pg. 326)

• The first column lists all the defect types or potential problem
areas that may occur in the item under review. Sources for
these defect types are usually data from past projects.
Abbreviations for detect/ problem types can be developed to
simplify the checklist forms. Status refers to coverage during
the review meeting—has the item been discussed? If so, a
check mark is placed in the column.
 Example Components for an
Inspection Checklist (Cont.)
• Major or minor are the two severity or impact levels shown
here. Each organization needs to decide on the severity levels
that work for them. Using this simple severity scale, a defect
or problem that is classified as major has a large impact on
product quality; it can cause failure or deviation from
specification. A minor problem has a small impact on these; in
general, it would affect a nonfunctional aspect of the software.
The letters M, I, and S indicate whether a checklist item is
missing (M), incorrect (I), or superfluous (S).
 General Review Checklist

Source: Burnstein (2003, pg. 327)

 Extended Checklist

• In addition to covering the items on the general document

checklist as shown in previous table, the following items
should be included in the checklist for a requirements review.
– completeness (have all functional and quality requirements described in
the problem statement been included?);
– correctness (do the requirements reflect the user’s needs? are they stated
without error?);
– consistency (do any requirements contradict each other?);
– clarity (it is very important to identify and clarify any ambiguous
requirements);
– relevance (is the requirement pertinent to the problem area?
requirements should not be superfluous);
– redundancy (a requirement may be repeated; if it is a duplicate it should
be combined with an equivalent one);
– testability (can each requirement be covered successfully with one or
more test cases? can tests determine if the requirement has been
satisfied?);
– feasibility (are requirements implementable given the conditions under
 Design Review

• Some specific items that should be checked for at a design

review are:
– a description of the design technique used;
– an explanation of the design notation used;
– evaluation of design alternatives;
– quality of the high-level architectural model;
– description of module interfaces;
– quality of the user interface;
– quality of the user help facilities;
– identification of execution criteria and operational sequences;
– clear description of interfaces between this system and other software
and hardware systems;
– coverage of all functional requirements by design elements;
– coverage of all quality requirements, for example, ease of use, portability,
maintainability, security, readability, adaptability, performance
requirements (storage, response time) by design elements;
– reusability of design components;
– Testability.
 Detailed Design Review

• For reviewing detailed design the following focus areas should

also be revisited:
– encapsulation, information hiding and inheritance;
– module cohesion and coupling;
– quality of module interface description;
– module reuse.
General Code Checklist

Source: Burnstein (2003, pg. 331)

General Code Checklist
(Cont.)

Source: Burnstein (2003, pg. 331)

 Test Plan Checklist

Source: Burnstein
(2003, pg. 333)
Reporting Review Results
 Review Reports

• The review reports should contain the following information.

– For inspections—the group checklist with all items covered
and comments relating to each item.
– For inspections—a status, or summary, report (described
below) signed by all participants.
– A list of defects found, and classified by type and
frequency. Each defect should be cross-referenced to the
line, pages, or figure in the reviewed document where it
occurs.
– Review metric data.
 IEEE Standards Inspection
Report
• IEEE standards suggest that the inspection report contain vital
data such as [8]:
– number of participants in the review;
– the duration of the meeting;
– size of the item being reviewed (usually LOC or number of
pages);
– total preparation time for the inspection team;
– status of the reviewed item;
– estimate of rework effort and the estimated date for
completion of the rework.
 Purpose of Review

• A final important item to note: The purpose of a review is to

evaluate a software artifact, not the developer or author of the
artifact. Reviews should not be used to evaluate the
performance of a software analyst, developer, designer, or
tester. This important point should be well established in the
review policy. It is essential to adhere to this policy for the
review process to work. If authors of software artifacts believe
they are being evaluated as individuals, the objective and
impartial nature of the review will change, and its
effectiveness in revealing problems will be minimized
Thank