Measuring Software Product Quality: The ISO 25000 Series and CMMI European SEPG June 14, 2004 Dave Zubrow

Pittsburgh, PA 15213-3890
Measuring Software Product Quality:

the ISO 25000 Series and CMMI
European SEPG
June 14, 2004
Dave Zubrow
Sponsored by the U.S. Department of Defense

© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute
Objectives
Provide status on a new Software Product Quality
Measurement standard and its connection to CMMI
Provide ideas on how to get started with Software Product

Quality Measurement today
page 2
Carnegie Mellon
Outline
Background and Overview
Concepts and Models
Software Product Quality Measurement
Summary
page 3
Carnegie Mellon
Achieving Quality Software

Requires planning and intentional design
More than achieving the desired functionality
Must explicitly attend to both functional and non-functional

requirements
Need to verify all requirements are being met throughout

the life cycle
page 4
Carnegie Mellon
CMMI Definition for Quality Requirements

The phrase “quality and process-performance
objectives” covers objectives and requirements for
product quality, service quality, and process
performance. Process performance objectives include
product quality.
page 5
Carnegie Mellon
Requirements Development
This process area describes three types of requirements:
• customer requirements (quality in use)
• product requirements (external quality attributes)
• product-component requirements (internal quality
attributes)
Taken together, these requirements address the needs of

relevant stakeholders, including those pertinent to various
product life-cycle phases (e.g., acceptance testing criteria)
and product attributes (e.g., safety, reliability,
maintainability).
Requirements also address constraints caused by the

selection of design solutions (e.g., integration of
commercial off-the-shelf products).
page 6
Carnegie Mellon
Requirements Development Goals

SG 1 Develop Customer Requirements
Stakeholder needs, expectations, constraints,
and interfaces are collected and translated into
customer requirements.
SG 2 Develop Product Requirements
Customer requirements are refined and
elaborated to develop product and product-
component requirements.
SG 3 Analyze and Validate Requirements
The requirements are analyzed and validated,
and a definition of required functionality is
developed.
page 7
Carnegie Mellon
Process Management and Performance

The organization’s process needs and objectives
cover aspects that include the following:
• characteristics of the processes
• process performance objectives, such as time to
market and product quality
• process effectiveness
A quantitatively managed process is institutionalized

by doing the following:
• controlling the process using statistical and other
quantitative techniques such that product quality,
service quality, and process performance attributes
are measurable and controlled throughout the
project (internal and external quality measures and
criteria)
page 8
Carnegie Mellon
Key Points in Relationship of CMMI

and ISO 9126/25000 - 1
CMMI takes a total life cycle view and is inclusive in its
approach to requirements development.
Requirements development explicitly seeks to have

the developer consider quality requirements.
Project and Process Management processes explicitly

consider product quality as process performance
objectives.
Neither the standard nor CMMI endorses a uni-

dimensional view of quality.
page 9
Carnegie Mellon
Key Points in Relationship of CMMI

and ISO 9126/25000 - 2
Product Quality Requirements are transformed into
designs and implemented via the Technical Solution
and Product Integration process areas.
The implementation of Product Quality Requirements

are monitored and confirmed via the Project
Management, Verification, and Validation process
areas.
CMMI acknowledges the need for interaction and

perhaps iteration among the related process areas to
satisfactorily identify, specify, and address Product
Quality Requirements.
page 10
Carnegie Mellon
Relating Requirements, Evaluation,

and Measurement
Customer Acquirer
User Evaluator
Development Organization Developer
provide guidance guidance perform
determines
Product Quality Product Quality
Requirements Evaluation
supports supports
Product Quality
Measurement
page 11
Carnegie Mellon
Outline
Concepts and Models
Summary
page 12
Carnegie Mellon
SQuaRE: Architecture
ISO/IEC
ISO/IEC2501n
2501n
Quality
QualityModel
Model
Division
Division
ISO/IEC
ISO/IEC2500n
2500n
Product
ProductQuality
Quality
ISO/IEC
ISO/IEC2503n
2503n General
GeneralDivision
Division ISO/IEC
ISO/IEC2504n
2504n
Quality
Quality General
Quality
Quality
General Overview
Overview and
and
Requirement
Requirement Guide
Guide to
to the
the SQuaRE
SQuaRE Evaluation
Evaluation
Division
Division Division
Division
Planning
Planning and
and
Management
Management
ISO/IEC
ISO/IEC2502n
2502n
Quality
QualityMetrics
Metrics
Division
Division page 13
Carnegie Mellon
Needs and Requirements

Stakeholders’ Collected and Selected and Functional Functional
Needs in their Identified Specified Requirements Design &
Minds Stakeholders’ Needs & QIU Internal
(Business) Requirements Quality
Stated, Needs Requirements
Implied or
Select & Specify

Unaware
Solicit & Identify
Non
Needs External Functional
Quality Design &
Requirements Internal
Quality
Requirements
Internal and External Quality Requirements may be stated in coding

standards, project quality goal statements, process descriptions (e.g., exit
criteria), test case descriptions, etc. They need not be explicitly identified as
requirements.
page 14
Carnegie Mellon
The Product Quality Measurement

Reference Model
page 15
Carnegie Mellon
Quality In Use Model (ISO/IEC 9126)
Effectiveness
Effectiveness
Quality
QualityIn
InUse
Use
Productivity
Productivity
Safety
Safety
Satisfaction
Satisfaction
page 16
Carnegie Mellon
Internal and External Software
Quality Model (ISO/IEC 9126)

Quality Characteristics Subcharacteristics
•Functionality
Suitability Accuracy Interoperability Security Compliance
•Reliability
Maturity Fault tolerance Recoverability Compliance
•Usability
Understandability Learnability Operability Comp Attractiveness
•Efficiency
Time behavior Resource utilization Compliance
•Maintainability
Analyzability Changeability Stability Testability Compliance
•Portability
Adaptability Installability Co-existence Replaceability Comp
page 17
Carnegie Mellon
Outline
Concepts and Models
Summary
page 18
Carnegie Mellon
Quality Model Elements and

Measurement Model Elements
Conceptual Model Operationalization
represents
Characteristic Quality Measure
comprise One or more

produce
represents
Subcharacteristic Quality Measure
comprise One or more

produce
represents Measurement
Attribute
Primitive
page 19
Carnegie Mellon
Relating the Quality and Measurement

Models
Software Product Quality Evaluation
Quality Report
Assessment
Analysis Rating
Quality
Quality
Characteristics Quality Measures
Characteristics
Function
Quality
Quality Sub (Formula)
Characteristics
-characteristics
Measurement
Primitives
Quality
Quality
Attribute
Attributes Measurement
Method
Quality Attribute page 20

Carnegie Mellon
Relating the Quality Measurement Model to the
ISO Software Measurement Process (15939)
Quality Evaluation Information
Report Product
Assessment Interpretation
Analysis Rating Analysis
Indicator 25030
Quality Measures 25040
Analysis Model
Function
(Formula) Derived Measures Definition
&
Measurement Measurement Function Collection
Primitives 2502n
Base Measures
Measurement
Method Measurement Method
Quality Attribute Attribute page 21

Carnegie Mellon
CMMI Measurement & Analysis

Process Area Goals
Align Measurement and Analysis Activities
Provide Measurement Results
Institutionalize a Managed Process
page 22
Carnegie Mellon
Activities for Goal 1

Align Measurement and Analysis Activities
• Establish Measurement Objectives
• Specify Measures
• Specify Data Collection and Storage Procedures
• Specify Analysis Procedures
Note: The first two practices directly address the need to

translate from the conceptual to the operational.
page 23
Carnegie Mellon
Activities for Goal 2

Provide Measurement Results
• Collect Measurement Data
• Analyze Measurement Data
• Store Data and Results
• Communicate Results
page 24
Carnegie Mellon
Mapping of
Software Engineering Institute INDICATOR TEMPLATE
Establish
M&A INDICATOR TEMPLATE
Measurement
INDICATOR GoalTEMPLATE
#_____:
Measurement
Objectives
Practices Measurement
Objective Goal #_____:
Measurement
Questions
Objective Goal #_____:
to Indicator Visual Display

Questions
Objective
Visual Display
Questions 100
Template Visual Display
100
80
100 60
80
40
Specify
Measures
60
80 20
40
60
20
40
Input(s) 20
Input(s) Data Elements Specify
Store Input(s) Data Elements
Responsibility Data
Data & for Reporting Collection
DataResponsibility
Elements
Results Collect
Form(s)
for Reporting
Responsibility Procedures
Algorithm
Data
Form(s)
for Reporting
Algorithm Form(s)
Assumptions
Algorithm
Assumptions
Interpretation
Assumptions
Interpretation
X-reference Specify
Communicate InterpretationProbing Questions
X-reference Analysis
Results Evolution
Probing
X-reference Questions Procedures
Evolution
Probing Questions Analyze
Evolution Data
page 25
Carnegie Mellon
Measuring External Quality to Manage

Software Development
Quality Characteristic/Subcharacteric: Efficiency/Time Behavior
Operational Measure: Response Time
Objective: Track satisfaction of user requirement for system

response time.
Questions: What is the system response time with respect to

common transaction? What is the variability in response time?
3.50
3.00
Response Time
2.50
2.00
1.50
1.00
0.50
0.00
1
11
13
15
17
19
21
23
25
27
29
st
st
st
st
st
st
st
st
st
st
st
st
st
st
st
Te
Te
Te
Te
Te
Te
Te
Te
Te
Te
Te
Te
Te
Te
Te
Obs Avg Target
page 26
Carnegie Mellon
Outline
Concepts and Models
Summary
page 27
Carnegie Mellon
Summary
Measurement links the specification of requirements to
acceptance criteria
Quality is conceptual; measurement is operational.
GQ(I)M provides a means for moving from the conceptual

to the operational.
The ISO 25000 series and the GQ(I)M Indicator Template

together can help with your implementation of CMMI
Requirements Development, Verification, and Validation.
page 28
Carnegie Mellon
Contact Information
Dave Zubrow
3118 SEI
4500 Fifth Ave
Pittsburgh, Pa 15213
USA
+1-412-268-5243 (v)
+1-412-268-5758 (f)
dz@sei.cmu.edu
page 29

Measuring Software Product Quality: The ISO 25000 Series and CMMI European SEPG June 14, 2004 Dave Zubrow

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Measuring Software Product Quality: The ISO 25000 Series and CMMI European SEPG June 14, 2004 Dave Zubrow

Uploaded by

Copyright:

Available Formats

Pittsburgh, PA 15213-3890

Measuring Software Product Quality:

Sponsored by the U.S. Department of Defense

Provide ideas on how to get started with Software Product

Concepts and Models

Software Product Quality Measurement

Achieving Quality Software

More than achieving the desired functionality

Must explicitly attend to both functional and non-functional

Need to verify all requirements are being met throughout

CMMI Definition for Quality Requirements

Taken together, these requirements address the needs of

Requirements also address constraints caused by the

Requirements Development Goals

Process Management and Performance

A quantitatively managed process is institutionalized

Key Points in Relationship of CMMI

Requirements development explicitly seeks to have

Project and Process Management processes explicitly

Neither the standard nor CMMI endorses a uni-

Key Points in Relationship of CMMI

The implementation of Product Quality Requirements

CMMI acknowledges the need for interaction and

Relating Requirements, Evaluation,

provide guidance guidance perform

Concepts and Models

Software Product Quality Measurement

Needs and Requirements

Select & Specify

Internal and External Quality Requirements may be stated in coding

The Product Quality Measurement

Quality In Use Model (ISO/IEC 9126)

Quality Model (ISO/IEC 9126)

Concepts and Models

Software Product Quality Measurement

Quality Model Elements and

comprise One or more

comprise One or more

Relating the Quality and Measurement

Quality Attribute page 20

Quality Attribute Attribute page 21

CMMI Measurement & Analysis

Align Measurement and Analysis Activities

Provide Measurement Results

Institutionalize a Managed Process

Activities for Goal 1

Note: The first two practices directly address the need to

Activities for Goal 2

to Indicator Visual Display

Template Visual Display

Measuring External Quality to Manage

Objective: Track satisfaction of user requirement for system

Questions: What is the system response time with respect to

Concepts and Models

Software Product Quality Measurement

Quality is conceptual; measurement is operational.

GQ(I)M provides a means for moving from the conceptual

The ISO 25000 series and the GQ(I)M Indicator Template

You might also like