Software Quality Assurance: Techniques and Tools

Matt Heinzelman
Department of Software Engineering
University of Wisconsin – Platteville
Platteville, Wisconsin 53818
heinzelm@uwplatt.edu
Abstract
Software Quality Assurance is essential to every software development process. SQA
improves development to deployment time, reduces the number of incomplete or missed
deadlines, and reduces time spent on making sure that requirements, design, code, and
documentation look the same by ensuring consistency without doing it all manually. Using
SQA audits to evaluate that guidelines have been met before advancing in the development
process is essential. Also, audits can be used to show stakeholders that continuous progress
is being made on a project. There are many tools that can be used to achieve Quality
Assurance on a project including Borland StarTeam and Lucent Technologies Sablime for
configuration, change, and problem management, and Programming Research QA C++ and
Parasoft Jtest for code testing/metrics. Therefore, Software Quality Assurance is a very
essential part of the development process.
What is Software Quality Assurance?
Software Quality Assurance involves the entire software development process – monitoring
and improving the process, making sure that any agreed-upon standards and procedures are
followed, and ensuring that problems are found and dealt with [1]. It is oriented to
‘prevention’ [1]. There are many standards that are followed in this area, involving the most
known and recognized IEEE IS0 9000 software quality and management guidelines.
Software Quality Assurance can also be defined more in depth as a planned and systematic
approach to the evaluation of the quality of and adherence to software product standards,
processes, and procedures [2]. SQA includes the process of assuring that standards and
procedures are established and are followed throughout the software acquisition life cycle.
Compliance with agreed-upon standards and procedures is evaluated through process
monitoring, product evaluation, and audits. Software development and control processes
should include quality assurance approval points, where an SQA evaluation of the product
may be done in relation to applicable standards [2].
Software Quality Assurance is found in almost every computer product that is developed
today. Each software development company develops their own system of standards and
procedures to follow using the IEEE IS0 9000 guidelines. They use this system to help
improve the development of their software system as a whole, as reviews are made during
and at the end of each step in the software development process.
Standards and Procedures
Every software development company must establish standards and procedures to aid in the
development of software, because these standards are the framework for which software
evolves [2]. Standards are the established criteria to which software products are compared.
Procedures are the established criteria to which the development and control procedures are
compared [2]. The whole role of Software Quality Assurance is to make sure that these
standards and procedures are developed and followed exactly in the development of
software. Proper documentation of standards and procedures is necessary since SQA
activities of process monitoring, product evaluation, and auditing rely on unequivocal
definitions to measure project compliance [2].
There are many different standards and procedures that can be implemented into a software
development system; however, this paper will only define and refer to four major areas.
These areas include: Requirements, Design, Code, and Documentation standards.
Requirement standards specify the form and content of how requirements in the system will
be defined. The normal standard in industry today is to establish a numbering system for
each major requirement and sub-requirements. Requirement standards will also establish a
system on how to write each requirement. Almost every system uses the method of writing a
short phrase to summarize a requirement and then elaborates it into more detail. Some
software companies like Rational have requirement development software to aid in the
organization and writing of requirements.
Design standards specify the form and content of how design documents will be developed.
They provide rules and methods for translating the software requirements into the software
design and for representing in the design documentation [2]. Some software companies like
Rational and Microsoft have design development software to aid in the organization and
development of diagrams and charts.
Code standards specify the language in which the code is to be written and define any
restrictions on use of language features. They define legal language structures, style
conventions, rules for data structures and interfaces, and internal code documentation [2].
Coding standards can be a good thing but too many standards will force productivity and
creativity to suffer. Using methods as ‘peer reviews’, ‘buddy checks’, and code analysis
tools can help to enforce standards. Some examples of what good standards would be: the
reduction or elimination of global variables, function and method sizes should be minimized;
each line of code should be seventy characters maximum, one code statement per line, etc
[2].
Documentation standards specify form and content for planning, control, and product
documentation and provide consistency throughout a project [2]. Documentation is critical
in a system. It could be written in any form, such as electronic (e.g. comments in a program),
or in paper form (e.g. a manual). Each practice should be documented clearly so it can be
repeated or changed later in the development process if needed. Specifications, designs,
business rules, inspection reports, configurations, code changes, test plans, test cases, bug
reports, user manuals, etc. should all be documented in some form [1]. There should ideally
be a system for easily finding and obtaining information and determining what
documentation will have a particular piece of information. Change management for
documentation should be used if possible [1].
Software Quality Assurance procedures are explicit steps to be followed in carrying out a
process. All processes should have documented procedures. Examples of processes for
which procedures are needed are configuration management, nonconformance reporting and
corrective action, testing, and formal inspections.
Techniques
The major technique that is used in Software Quality Assurance is the audit. They are used
to perform product evaluation and process monitoring. Audits are performed routinely
throughout the software development process. Their job is to look at a process and/or a
product in depth, comparing them to established procedures and standards. Audits are used
to review management, technical, and assurance processes to provide an indication of the
quality and status of the software product [2].
The purpose of using an audit is to assure that proper control procedures are being followed,
that required documentation is maintained, and that the developer’s status reports accurately
reflect the status of the activity. The SQA product is an audit report to management
consisting of findings and recommendations to bring the development into conformance with
standards and/or procedures [2].
Tools
There are many different tools that have been developed for problem, change, and
configuration management along with many different testing tools. Each tool works
differently than each other; however, they accomplish the same goal: help improve the
development process of a computer system. There are so many different tools out on the
market today to accomplish and, this paper will look at two different tools from configuration
and problem management, and two different tools for testing software.
Borland StarTeam
Borland’s process based configuration management called StarTeam is a comprehensive
system that can be tailored to any software development team. They offer a range of
solutions that can meet any development team according to size, distribution, and work style.
StarTeam can coordinate and manage the whole development process by promoting team
communication and collaboration through a centralized control of all project assets [3].
StarTeam offers integrated change management, defect tracking, file versioning,
requirements management, threaded discussion, and project and task management. Every
project module is put into one repository so each user is able to access it and make changes
as needed. It can be customized to fit your needs, thanks to an open API, highly customable
forms, and workflow [3]. StarTeam uses a Windows server interface, but is multiplatform
for each of the clients.
StarTeam is offered in three different versions varying on the development teams’ size and
type of project. Borland’s biggest version, StarTeam Enterprise Advantage, is used to
support the configuration and management needs of large, widely distributed teams work on
enterprise-level projects. This version supports the whole development cycle by delivering a
flexible, customizable solution with integrated requirements management, change
management, defect tracking, file versioning, threaded discussions, and project and task
management [4]. Enterprise offers project trend analysis and reporting to enhance visibility
for business stakeholders to optimize the delivery of software [5].
Enterprise Advantage has four unique features that are designed to meet the needs of large,
geographically distributed teams. Advantage has search and query capabilities that allow
team members to search for features across multiple objects and repositories for reuse and
sharing. Second, a MPX server that provides multicast communication to ensure that teams
are always up-to-date by broadcasting all events and keeping data for the project current.
Third, a cache agent that supports distributed development with multisite repositories to
ensure maximum scalability and high availability across global terms. Fourth, a Web edition
that lets users work wherever and whenever they would like.
Borland’s middle of the road version is called StarTeam Enterprise. Enterprise is an ideal
solution for medium-to-large development teams [4]. Blending ease of use with the power of
its integrated file, change request, task, and discussion components – in a single interface –
StarTeam Enterprise is an excellent next step for organizations that have outgrown their
current SCM technology [4]. Enterprise uses an unified repository to manage shared and
reusable components to increase team productivity by promoting parallel development.
Enterprise enhances control of the development process by ensuring that all assets are
versioned and changes automatically tracked, making it easier to monitor project status and
keep all team members up to date [5]. Enterprise includes Web client support, a completely
browser-based Web client that offers users access to work wherever and whenever they like.
Enterprise Web Edition allows users to access StarTeam without installing a client
application, increasing the client choices available to users.
Borland’s smallest, or standard version is referred to as StarTeam Standard. Standard is an
easy-to-use, entry-level SCM tool with integrated components for file versioning, defect
tracking, and threaded discussions [4]. StarTeam Standard helps small project workgroups
and distributed teams to effectively collaborate and efficiently manage change across the
application development lifecycle [4]. Some key development tasks with Standard are
enhanced file check-in, checkout, and labeling. Also the ability to have change request
functionality and able to roll back to previous file versions is a feature of StarTeam Standard.
Standard promotes better team productivity, enabling rapid response to change and reducing
the overall burden of development task management [5].
Borland also offers two other, smaller, software tools to aid in process and configuration
management. StarTeam Server provides a unified collaborative process repository for all
application development assets [4]. StarTeam Datamart is a decision support system,
provides in-depth query and analysis capabilities for SCM and change management
environments [4]. Datamart offers data into resource allocation, change requests, potential
defects, and task management. Using this information, it is easier to monitor progress and
perform accurate impact analysis for effective project management. Datamart includes three
different features that can aid in the analysis of process and configuration management.
Datamart Extractor takes the selected StarTeam project data and prepares it for analysis.
Second, Datamart Synchronizer opens the Business Objects Universe and refreshes the data
to reflect any field changes, thereby ensuring that the most up-to-date information is always
used for analysis [5]. Third, Datamart Viewer provides easy access to any reports stored in
StarTeam, and the ability to launch them in a reporting tool such as Business Objects and
Crystal Reports [5].
Lucent Technologies Sablime
The Sablime Configuration Management System is a powerful and efficient tool that
provides integrated version control and change management of your software artifacts such
as source files and documentation [6]. Sablime is easy to learn and maintain, and can be
used on any size project with any number of people. Sablime incorporates an effective
development process that balances the needs of managers, developers, testers, and
integrators, and provides workflow management so the stakeholders are kept informed [6].
Sablime provides version, configuration, and change management to help improve product
quality and shorten release cycles. Sablime also supports concurrent development, so files
can be worked on by two or more developers at the same time. This improves productivity
of the development team. Sablime will automatically merge the files reliably and efficiently.
Sablime was developed at Bell Labs and is a multiplatform software environment. It can be
incorporated into many different development environments including Microsoft Visual
Studio and Eclipse. Also, Sablime’s analysis tools can be incorporated into many different
spreadsheet programs, notably Microsoft Excel and WordPerfect Quattro.
Sablime has a distinctive approach in the configuration management system by building its
development process around the Modification Request. Any changes that are proposed by a
team member or customer must be done by creating a MR. When MRs are reviewed they
can be deferred, killed, assigned for study or accepted for implementation in one or more
generics (codelines) [6]. Sablime tracks changes using MRs, and makes sure that all changes
are delivered together at integration time. Program versions are assembled based on these
MRs states. Each MR is assigned to one or more project members with appropriate priority
and due date. If an MR requires extensive effort or spans several areas of responsibility, it
can be divided into smaller requests and independently assigned, tracked and managed in
different generics [6]. As an MR passes through its life cycle, appropriate project members
are notified of events via email. This communication promotes clear responsibility and
prompt, timely actions. Sublime keeps track of who, when, and why each action was taken
[6].
Sablime has many different benefits in using it to help in the software configuration
management process. Software changes constantly and is often difficult to manage.
Different artifacts (such as source code, documentation, etc.) exist in different versions at
different times [6]. Refer to the list below for benefits of using Sablime [6].
• Helps improve product quality and shortens release cycles
• Supports an out-of-the-box process model
• Tracks, coordinates and integrates product changes and change requests
• Helps prevent fixed bugs from getting reintroduced
• Enables you to reconstruct versions sent to customers
• Makes project status and source files accessible to all team members, even when
geographically distributed
• Enables managers to control and characterize contents of each release, and track
release status
• Enables testers to see what features or fixes are ready for testing, and to review the
requirements, notes, and implementation associated with each change request
• Enables integrators to create consistent product versions automatically, based on
readiness for integration
Sablime provides comprehensive configuration management and version control [6]. Refer
to the table below for key features of Sablime [6].
• Coordinates change requests and actual changes
• Supports multiple active codelines (releases) per product
• Supports concurrent development, with less need for merging
• Detects dependencies automatically
• Integrates with the Eclipse development platform
• Integrates with Visual Studio and other IDE’s
• Integrates with Excel, enabling status reporting and management
• Guides teams to consistent results using defined roles and workflow with email
notification
• Scales easily from small to large objects
• Supports local and web-based users
• Allows scripting and customization
• Easy to learn and use
• Simple to install and maintain
• Does not require dedicated hardware
• Available on UNIX, Linux, and Windows
Borland StarTeam and Lucent Technologies Sablime are just two tools that can be used for
configuration and problem management, and both of these can be used in the software
quality assurance process. They help improve the quality of software by able to organize
everything about a development process (requirement, design, code, documentation) into one
area. However, to complete software quality assurance on a system, some kind of testing
needs to be implemented. There are many different testing/metrics tools out on the Web
today, and I will discuss two of them: Programming Research QA C++ and Parasoft Jtest.
Programming Research QA C++
QA C++ quickly ensures code quality while enhancing productivity in the development
process. QA C++ is fast, non-disruptive, easy-to-use and can be quickly integrated almost
everywhere [7]. QA C++ provides a fully automated environment to introduce and enforce
custom coding standards, those maintained by the software development company, and
required by customers. QA C++ documents and proves compliance – a growing customer
requirement especially for contract developers and offshore development firms [7]. QA C++
helps to ensure that coding standards that were set have been followed; a key in Software
Quality Assurance.
QA C++ can parse any size project quickly and easily. QA C++ detects many different
problems and defects like language implementation errors, inconsistencies, obsolescent
features, and coding standard violations [7]. Detecting these early can prevent delays later in
the development cycle where problems are costly to fix. QA C++ reports many industry-
standard code metrics into graphs, diagrams, and HTML output. These metrics can be
exported into an open format so they can be used in applications such as Microsoft Office
and StarOffice to analyze, share, and present information.
QA C++ includes a huge knowledge base built around it. Any problems that are discovered
are explained in a message browser with a drill-down environment. QA C++ explains why
problems it discovers need to be corrected and then provides detailed examples of how to fix
them [7]. QA C++ helps identify software defects and non-compliance issues early in the
development cycle and prevents them from entering production code – thwarting potentially
huge problems in your deployed products. QA C++ provides the ability to limit complexity
so you can develop code that is truly testable and easier to maintain [7].
QA C++ is a market leader in analyzing source code and enforcing coding standards [7].
Refer to the table below for product highlights of QA C++ [7].
• Identifies coding problems early in the development cycle
• Accelerates the code review process – improves teamwork
• Ensures quality code and coding standard compliance
• Educates and raises programmer awareness
• Reduces the risk of program failure
• Enhances reliability, portability, and maintainability
• Lowers software development costs – increases productivity
• Improves time-to-market while reducing costs
• Allows instant and repeatable code audits and reviews
• Delivers unmatched technology & strong ROI
Parasoft Jtest
Jtest is an automated Java unit testing and coding standard analysis product that improves
Java code reliability, functionality, security, performance, and maintainability. Jtest checks
whether code compiles with over 500 built in development rules and corrects many reported
violations automatically. User defined code guidelines (rules) can be produced without using
code; they can be created graphically or automatically [8]. To expose reliability problems,
Jtest examines each class, then generates and executes JUnit test cases designed to achieve
high coverage and expose uncaught runtime exceptions. To expose functionality problems,
Jtest provides fast and easy ways to add and execute realistic user-defined test cases –
including a Test Case Sniffer that monitors a running application and generates JUnit test
cases that capture application behavior [8]. To ensure continued functionality, Jtest’s
automated regression testing identifies problems introduced by code modifications. Jtest
improves Java code quality throughout the software lifecycle and improves productivity.
Quality Assurance team members can use Jtest to identify critical problems before an
imminent release/deployment deadline [8].
Jtest analyzes code in two different ways. First, Jtest verifies whether code complies with
development rules for preventing functional errors, security vulnerabilities, performance
problems, and pitfalls [8]. Many of these violations can be corrected automatically. Second,
Jtest automatically generates and executes JUnit test cases designed to achieve high
coverage, expose uncaught runtime exceptions and memory leaks, and describe the code’s
current behavior [8]. Test findings are reported as a prioritized task list for quick review and
response. Test cases can be added automatically (using Test Case Sniffer), graphically
(using a graphical object editor and graphical test case editor), or programmatically (by
modifying the generated JUnit test cases) [8].
Jtest’s Test Case Sniffer is the first technology that makes realistic functional unit test
practical for development and Quality Assurance organizations [8]. Test Case Sniffer
monitors a running application and automatically generates unit test cases that capture
application behavior. Simply exercise the application functionality you want to test, and Test
Case Sniffer automatically generates JUnit test cases with real data that represents the paths
taken through the application [8]. The result is a library of test cases against which new code
can be tested to ensure that it meets specifications and does not “break” existing functionality
[8]. Jtest is multiplatform software that can be used on Windows 2000 and XP, Solaris, and
Linux. Refer to the table below for benefits of Jtest [8].
• Improve code reliability, functionality, security, and performance quickly and
painlessly
• Obtain instant expert feedback on code quality and potential defects
• Prevent code modifications from breaking previously-verified functionality
• Perform extensive testing/debugging and more time on creative tasks
• Perform extensive testing with minimal user intervention
• Reduce the risks that cause late, over-budget, incomplete releases
• Identify errors lurking in existing applications
• Optimize code review time
• Ensure that best practices are applied consistently and uniformly across the team
• Monitor overall project quality, specific project segments, and progress toward
quality goals
Benefit of Software Quality Assurance in Projects
Software Quality Assurance is essential to every software development process. Without
SQA, many development groups would not reach their release goals/deadlines, or having
incomplete releases because they would be spending too much time revising requirements,
design, code, and documentation of each portion of a project to match. Establishing certain
standards (rules) lowers the time spent working on making every part of the project look the
same. Also, it decreases the time from development to deployment. Reviewing these
standards against what has been developed can help catch any errors before they become
very costly to fix. These standards can be reused across many projects, which will improve
the development time and lower costs of each project involved.
References
[1] Hower, Rick (2006). Software QA and Testing Resource Center. Updated April
2006. Web site: http://www.softwareqatest.com/
[2] Software Quality Assurance, NASA.
Web site: http://satc.gsfc.nasa.gov/assure/agbsec3.txt
[3] Buchanan, Ian (2005). Borland StarTeam. Updated January 2005.
Web site: http://www.cmcrossroads.com/cgi-bin/cmwiki/bin/view/CM/StarTeam
[4] CM Crossroads. Borland StarTeam Product Review.
Web site: http://www.cmcrossroads.com/toolspot/starteam.php
[5] Borland Software Corporation. Borland StarTeam.
Web site: http://www.borland.com/us/products/starteam/index.html
[6] Lucent Technologies (2006). Sablime.
Web site: http://www.bell-labs.com/project/sablime/
[7] Programming Research, Inc (2003). QA C++ Data Sheet.
Web site: http://www.programmingresearch.com/pdfs/QAC++ -
%20DATASHEET%20FEB05%20HQ.pdf
[8] Parasoft (2006). Jtest Data Sheet.
Web site: http://www.parasoft.com/jsp/printables/-
ParasoftJtestDataSheet.587.pdf?path=/jsp/products/-
quick_facts.jsp&product=Jtest