Pittsburgh, PA 15213-3890

Software Quality Requirements

and Evaluation, the ISO 25000
Series

PSM Technical Working Group

February 2004

Dave Zubrow
Sponsored by the U.S. Department of Defense
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

Background
This presentation reviews the latest developments with the
ISO 25000 SQuaRE series of standards. This series on
Software Quality Requirements and Evaluation (SQuaRE)
is an effort to harmonize ISO 9126 and ISO 14598. The
information presented here is based on the current state of
the standards/

Note: Many of the slides in this presentation are taken from WG

6 documents and materials, especially slides from Professor
Azuma, WG convener.

page 2
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

Outline
Background and Overview

Concepts and Models

Software Product Quality Measurement

Software Product Quality Requirements and Evaluation

page 3
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

Relationship between ISO/IEC 9126 and

ISO/IEC 14598

R e s o u rc e s E v a lu a tio n E ffe c t o f th e
S o ftw a re
a n d p ro c e s s s o ftw a re
p ro d u c t
e n v ir o n m e n t p ro d u c t

E v a lu a tio n E v a lu a tio n In te rn a l E x te rn a l Q u a lity in

s u p p o rt p ro c e s s m e tr ic s m e t r ic s u s e m e tr ic s

1 4 5 9 8 -1

1 4 5 9 8 -2 1 4 5 9 8 -3 9 1 2 6 -1
1 4 5 9 8 -4
1 4 5 9 8 -6 9 1 2 6 -3 9 1 2 6 -2 9 1 2 6 -4
1 4 5 9 8 -5

page 4
© 2004 by Carnegie Mellon University
 Carnegie Mellon
Software Engineering Institute
Relationship and transition process between ISO/IEC 9126,
ISO/IEC 14598 and SQuaRE series of standards
C U R R E N T
9 1 2 6 : P r o d u c t q u a lity
-1 : Q u a lity m o d e l
-2 : E x te rn a l m e trics
- 3 : In t e r n a l m e t r i c s
-4 : Q u a lity in u s e m e trics
N ew P rop osal
G u id e s to u s e 9 1 2 6 & 1 4 5 9 8
B a s e m e tric s
Q u a lity r e q u ir e m e n ts
1 4 5 9 8 : P r o d u c t e v a lu a tio n
-1 : G e n e ra l o v e rv ie w
-2 : P la n n in g a n d m a n a g e m e n t
-3 : P ro c fo r d e v e lo p e rs
-4 : P ro c fo r a c q u ire rs
-5 : P ro c fo r e v a lu a to rs
-6 : D o c o f e v a lu a tio n m o d u le s
© 2004 by Carnegie Mellon University
S Q u a R E
2 5 0 0 0 : Q u a lit y M a n a g e m e n t D iv is io n
2 5 0 0 0 : G u id e to S Q u a R E (N P )
2 5 0 0 1 : P la n n in g a n d m a n a g e m e n t
2 5 0 1 0 : Q u a lit y M o d e l D iv is io n
2 5 0 1 0 : Q u a lity m o d e l a n d g u id e (R e v )
2 5 0 2 0 : Q u a lity M e a s u r e m e n t D iv is io n
2 5 0 2 0 : M e a s u re m e n t re fe re n c e m o d e l a n d g u id e (N P )
2 5 0 2 1 : M e a s u re m e n t p rim itiv e s (N P )
2 5 0 2 2 : M e a s u re m e n t o f in te rn a l q u a lity
2 5 0 2 3 : M e a s u re m e n t o f e x te rn a l q u a lity
2 5 0 2 4 : M e a s u re m e n t o f q u a lity in u s e
2 5 0 3 0 : Q u a lit y R e q u ir e m e n t s D iv is io n
2 5 0 3 0 : Q u a lity re q u ire m e n ts a n d g u id e (N P )
2 5 0 4 0 : Q u a lit y E v a lu a t io n D iv is io n
2 5 0 4 0 : Q u a lity e v a lu a tio n o v e rv ie w a n d g u id e
2 5 0 4 1 : E v a lu a tio n m o d u le s
2 5 0 4 2 : P ro c e s s fo r d e v e lo p e rs
2 5 0 4 3 : P ro c e s s fo r a c q u ire rs
2 5 0 4 4 : P ro c e s s fo r e v a lu a to rs
page 5
Carnegie Mellon
Software Engineering Institute

State of the Standards

25000, Guide to the SQuaRE series – Ballot out for FCD
25020, Software Product Quality Measurement Reference
Model – Ballot out for 2nd CD
25021, Measurement Primitives – Ballot out for 1st CD
25030, Quality Requirements – Ballot out for 2nd CD

All Ballots closing Mid-April to Early May

Editors assigned but no drafts out

25010, Quality Model
25023, External Quality Measures

page 6
© 2004 by Carnegie Mellon University
 Carnegie Mellon
Software Engineering Institute

SQuaRE: Architecture
ISO/IEC
ISO/IEC2501n
2501n
Quality
QualityModel
Model
Division
Division
ISO/IEC
ISO/IEC2500n
2500n
Product
ProductQuality
Quality
ISO/IEC
ISO/IEC2503n
2503n General
GeneralDivision
Division ISO/IEC
ISO/IEC2504n
2504n
Quality
Quality General
Quality
Quality
General Overview
Overview and
and
Requirement
Requirement Guide
Guide to
to the
the SQuaRE
SQuaRE Evaluation
Evaluation
Division
Division Division
Division
Planning
Planning and
and
Management
Management

ISO/IEC
ISO/IEC2502n
2502n
Quality
QualityMetrics
Metrics
Division
Division page 7
© 2004 by Carnegie Mellon University
 Carnegie Mellon
Software Engineering Institute

SQuaRE general reference model

Software User
Target of process
Product Task
(Effect)

Requirements
Process Specification Requirements
Specification
Planning
Planning
Measurement
Measurement
Evaluatioin
Evaluation

25021( MP) 25041(EM) 25021( MP) 25041(EM)

Execution 25030 (QR) 25001 (PM)
25022 (MfIQ)
25030 (QR) 25001 (PM)
25022 (MfIQ)
25042 (PD) 25042 (PD)
25023 (MfEQ) 25024 (MfQU)
25043 (PA) 25025(DEM) 25043 (PA)
25044 (PE) 25044 (PE)

25030 (Quality Requirements and Guide)

25020 (Measurement Reference Model and Guide) 25040 (Evaluation Process Overview and Guide)
Particular Guidance
Legend :
applied to
25010 (Quality Model and Guide)
used by

MP – Measurement Primitives
General Guidance
MfIQ – Metrics for Internal Quality
MfEQ – Metrics for External Quality 25000 (Guide to SQuaRE)
MfQU – Metrics for Quality in Use
EM – Evaluation Modules
PM – Planning and Management
PA – Process for Acquirers
PE – Process for Evaluators
PD – Process for Developers
QR – Quality Requirements

page 8
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

Outline
Background and Overview

Concepts and Models

Software Product Quality Measurement

Software Product Quality Requirements and Evaluation

page 9
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

The Product Quality Measurement

Reference Model

page 10
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

Software Product Quality Life-Cycle

and Quality Measures
Requirements Products
Needs
Quality
Users
UsersQuality
Quality Quality
Quality
In Use
Needs
Needs In
InUse
Use
Measures
Validation

External External
Quality ExternalQuality
Quality External
External
Requirement
Requirement Quality
Quality
Measures
Verification
And
Internal Validation Internal
Quality Internal Quality
Internal Quality Internal
Measures Requirement
Requirement Quality
Quality
Verification
page 11
© 2004 by Carnegie Mellon University Implementation
Carnegie Mellon
Software Engineering Institute

Quality Model

Quality model

Characteristic 1 Characteristic 2 Characteristic n

Subcharacteristic 1 Subcharacteristic 2 Subcharacteristic m

Quality Attributes

page 12
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

ISO/IEC 9126-1 Quality In Use

Effectiveness
Effectiveness
Quality
QualityIn
InUse
Use

Productivity
Productivity

Safety
Safety

Satisfaction
Satisfaction

page 13
© 2004 by Carnegie Mellon University
 Carnegie Mellon
Software Engineering Institute

ISO/IEC 9126-1 - Quality Model

Quality Characteristics Subcharacteristics
•Functionality
Suitability Accuracy Interoperability Security Compliance
•Reliability
Maturity Fault tolerance Recoverability Compliance
•Usability
Understandability Learnability Operability Comp Attractiveness
•Efficiency
Time behavior Resource utilization Compliance
•Maintainability
Analyzability Changeability Stability Testability Compliance
•Portability
Adaptability Installability Co-existence Replaceability Comp
page 14
© 2004 by Carnegie Mellon University
 Carnegie Mellon
Software Engineering Institute

Process and Product Measurement

External
External Quality
Quality Measure
Measure

Internal
Internal Quality
Quality Measure
Measure Quality
Quality in
in Use
Use Measure
Measure

Software

Computer System
Information System

Business System
Computer Centered System

page 15
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute
Software product quality measurement
reference model (SPQM-RM)

page 16
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

Measurement concept of the SQuaRE model

page 17
© 2004 by Carnegie Mellon University
 Carnegie Mellon
Software Engineering Institute

Relationship between the SQuaRE SPQM-RM

and the ISO/IEC 15939 Information Model
Q u a lit y N e e d s / Q u a lit y E v a lu a tio n In fo r m a tio n
R e p o rt P ro d u c t

A s s e s s m e n t In te r p r e ta tio n
A n a ly s is
R a tin g
In d ic a to r
Q u a lity M e a s u r e s
A n a ly s is M o d e l

F u n c tio n D e r iv e d M e a s u r e
(F o r m u la )

M e a s u re m e n t
M e a s u re m e n t F u n c tio n
P r im itiv e s
B a s e M e a s u re
M e a s u re m e n t
M e th o d M e a s u re m e n t
M e th o d

A ttr ib u te s fo r Q u a lity
M e a s u re A ttrib u te s

S Q u a R E S P Q M -R M IS O /IE C 1 5 9 3 9

page 18
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

Outline
Background and Overview

Concepts and Models

Software Product Quality Measurement

Software Product Quality Requirements and Evaluation

page 19
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

Titles of the Measurement Standards:

2502n
Common:
Software Engineering -
Software product Quality Requirements and Evaluation
(SQuaRE) -
Division: Software Quality Measurement (For Explanation
Only)
25020 Measurement Reference Model and Guide
25021 Measurement Primitives
25022 Measurement of Internal Quality
25023 Measurement of External Quality
25024 Measurement of Quality In Use

page 20
© 2004 by Carnegie Mellon University
 Carnegie Mellon
Software Engineering Institute

Quality Measure and Measurement

Method (Concept)
Quality Measures: A variable, which shows satisfactory levels of a quality
characteristic, subcharacteristic or sub-sub-characteristic, to which a value is
assigned as the result of measurement
Quality Measure Set: A set of a Quality Measure, Measurement Primitives that are
used for deriving the quality measure, associated Scales and Measurement
Methods, a Formula to combine them to generate the value of Quality
Measure, and Guide to use them and analyze the results. for a Quality
Characteristic or Subcharacteristic. (A line of Quality Measure Table)
Quality Measurement Table: A set of Quality Measure Set for each Quality
Subcharacteristic

page 21
© 2004 by Carnegie Mellon University
 Carnegie Mellon
Software Engineering Institute

Application of SQuaRE
Activity 1 Activity 2 Activity 3 Activity 4 Activity 5 Activity 6 Activity 7 Activity 8

Phase Requirement analysis Architectural design Software detailed Software coding and Software integration System integration Software installation Software acceptance
(Software and (Software and design testing and software and system support
systems) systems) qualification testing qualification testing

25000 (SQuaRE) series model Required quality in Predicted quality in Predicted quality in Predicted quality in Predicted quality in Predicted quality in Predicted quality in Measured quality in
reference use, use, use, use, use, use, use, use,
Required internal Predicted external Predicted external Measured external Measured external Measured external Measured external Measured external
quality, quality, quality, quality, quality, quality, quality, quality,
Required external Measured internal Measured internal Predicted external Predicted external Measured internal Measured internal Measured internal
quality quality quality quality, quality, quality quality quality
Measured internal Measured internal
quality quality

Key deliverables of activity Quality in use Architecture design of Software detailed Software code, Software product, Integrated system, Installed system Delivered software
requirements Software / system design Test results Test results Test results product
(specified),
External quality
requirements
(specified),
Internal quality
requirements
(specified)

Applicable measures
Internal measures Internal measures Internal measures Internal measures Internal measures Internal measures Internal measures Quality in use
(External measures External measures External measures External measures External measures measures
may be applied to Internal measures
validate External measures
specifications)

page 22
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

User needs, characteristics & weights

Q u a lit y in u s e

C H A R A C T E R IS T IC W E IG H T

E ffe c tiv e n e s s H

P r o d u c tiv it y H

S a fe ty L

S a tis f a c ti o n M

E x te rn a l & in t e r n a l q u a lit y

C H A R A C T E R IS T IC S U B C H A R A C T E R IS T IC W E IG H T
( H ig h / M e d iu m / L o w )
F u n c t io n a lit y S u ita b ilit y H

A c c u ra c y H

In t e r o p e r a b ility L

S e c u r it y L

C o m p lia n c e M

R e lia b ilit y M a tu r ity L

F a u lt t o le r a n c e L

R e c o v e r a b ilit y H

C o m p lia n c e H

U s a b ilit y U n d e r s ta n d a b ilit y M

L e a r n a b ilit y L

O p e r a b ility H

A ttr a c tiv e n e s s M

C o m p lia n c e H

E f f ic ie n c y T im e b e h a v io u r H

R e s o u r c e u tiliz a tio n H

C o m p lia n c e H

M a in t a in a b ilit y A n a ly z a b ilit y H

C h a n g e a b ilit y M

S t a b ilit y L

T e s t a b ilit y M

C o m p lia n c e H

P o r t a b ilit y A d a p t a b ilit y H

In s ta lla b ilit y L

C o - e x is t e n c e H

R e p la c e a b ilit y M

C o m p lia n c e H

page 23
© 2004 by Carnegie Mellon University
 Carnegie Mellon
Software Engineering Institute
Quality measurement tables
E x te r n a l q u a lit y m e a s u r e m e n t c a t e g o r y
C H A R A C T E R IS T IC
F u n c t io n a lit y
R e li a b ilit y
U s a b il it y
E ffic ie n c y
M a in t a in a b i lit y
P o r t a b il it y
© 2004 by Carnegie Mellon University
Q u a lity in u s e m e a s u r e m e n t c a te g o r y
A S S E S S M E N T
R E Q U IR E D
C H A R A C T E R IS T IC M E A S U R E S A C T U A L
L E V E L
R E S U L T
E f fe c tiv e n e s s
P r o d u c tiv it y
S a fe ty
S a tis f a c tio n
In te r n a l q u a lit y m e a s u r e m e n t c a te g o r y
A S S E S S M E N T
R E Q U IR E D
A S S E S S M E N T C H A R A C T E R IS T IC S U B C H A R A C T E R IS T IC M E A S U R E S A C T U A L
R E Q U IR E D L E V E L
S U B C H A R A C T E R IS T IC M E A S U R E S A C T U A L R E S U L T
L E V E L
R E S U L T
F u n c t io n a lit y S u i ta b ilit y
A c c u ra c y
S u ita b ilit y
In t e r o p e r a b ility
A c c u ra c y
S e c u r it y
In t e r o p e r a b ility
C o m p lia n c e
S e c u r it y
R e lia b ilit y M a tu r ity
C o m p lia n c e
F a u lt t o le r a n c e
M a tu r ity
R e c o v e r a b ilit y ( d a t a , p r o c e s s ,
F a u lt to le r a n c e
C o m p lia n c e
R e c o v e r a b ilit y ( d a t a , p r o c e s s ,
U s a b ilit y U n d e r s ta n d a b ilit y
C o m p lia n c e L e a r n a b ilit y
U n d e r s ta n d a b ilit y O p e r a b ility
L e a r n a b ilit y A ttr a c tiv e n e s s
O p e r a b ility C o m p lia n c e
E f f ic ie n c y T im e b e h a v io u r
A ttr a c tiv e n e s s
C o m p lia n c e
R e s o u r c e u tilis a tio n
T im e b e h a v io u r
C o m p lia n c e
R e s o u r c e u tilis a tio n
M a in t a in a b ilit y A n a ly z a b ilit y
C o m p lia n c e C h a n g e a b ility
S t a b ilit y
A n a ly z a b ilit y T e s t a b ilit y
C h a n g e a b ilit y C o m p lia n c e
S t a b ilit y P o r t a b ilit y A d a p t a b ilit y
T e s t a b ilit y In s ta b ilit y
C o m p lia n c e C o - e x is t e n c e
R e p la c e a b ilit y
A d a p t a b ilit y
C o m p lia n c e
In s ta b ilit y
C o - e x is t e n c e
R e p la c e a b ilit y
C o m p lia n c e
page 24
 Carnegie Mellon
Software Engineering Institute

Measurement plan implications

S U B C H A R A C T E R I D E L IV E R A B L E S IN T E R N A L E X T E R N A L Q U A L IT Y IN U S E
S T IC T O B E M E A S U R E S T O B E M E A S U R E S T O B E M E A S U R E S T O B E
E V A L U A T E D A P P L IE D A P P L IE D A P P L IE D

1 . S u it a b ilit y 1 . 1 . 1 . ( N o t A p p lic a b le )

2 . 2 . 2 .

3 . 3 . 3 .

2 . S a t is f a c t io n 1 . ( N o t A p p lic a b le ) ( N o t A p p lic a b le ) 1 .

2 . 2 .

3 . 3 .

3 .

4 .

5 .

6 .

page 25
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

Measurement Primitive
A measure, either a base measure or a derived measure, that is
commonly used for deriving internal quality measures, external
quality measures and quality in use measures.
No definition (25000)
measure collected during Software Product Lifecycle from which Internal,
External and Quality in Use Measures are derived. (25020 and
25021)
Single value of measurement primitive generally does not indicate the
quality of the measured entity. NOTE The Quality is measured
afterwards by calculating the Quality Measures. (25020 and 25021)

page 26
© 2004 by Carnegie Mellon University
 Carnegie Mellon
Software Engineering Institute

Measurement Primitives in SPQM-RM

primitives concept
Measurement

page 27
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

Consideration views

Considerations
Measurement Measurement Measurement Measurement
Concept Scale Type Focus Method Type

Base Nominal Ordinal Internal Subjec

tive
External
Interval Ratio Objec
Derived Q in use tive
Absolute

Measurement primitive

page 28
© 2004 by Carnegie Mellon University
 Carnegie Mellon
Software Engineering Institute

Measurement Primitives Hyperlink Format

example

C o n s id e r a t io n V ie w s
M e a s u re m e n t c o n c e p t
B a se
D e r iv e d
M e a s u r e m e n t s c a le M e a s u r e m e n t P r im itiv e s Q u a lity M e a s u re s
N o m in a l : :
: N u m b e r o f f u n c tio n s P h y s ic a l a c c e s s ib ilit y
R a t io :
A b s o lu te :
: :
M e a s u re m e n t fo c u s T a s k t im e W a itin g t im e
In te rn a l : :
E x te rn a l : :
Q u a lit y in U s e T im e o r n u m b e r o f t a s k s T a s k e ffic ie n c y
M e a s u re m e n t m e th o d : :
O b je c t iv e
S u b je c tiv e

page 29
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

Outline
Background and Overview

Concepts and Models

Software Product Quality Measurement

Software Product Quality Requirements and Evaluation

page 30
© 2004 by Carnegie Mellon University
 Carnegie Mellon
Software Engineering Institute

Relationship between Needs and Requirements

Stakeholders’ Collected and Selected and Functional Functional

Needs in their Identified Specified Requirements Design &
Minds Stakeholders’ Needs & QIU Internal
Requirements Quality
Stated, (Business) Needs Requirements
Implied or
Unaware
Needs
Solicit & Identify

Select & Specify

Non Functional
External Quality Design &
Requirements Internal
Quality
Requirements

page 31
© 2004 by Carnegie Mellon University
 Carnegie Mellon
Software Engineering Institute

ISO/IEC 15288 System Life Cycle Processes to appear in 25030

ISO/IEC 25030 Stakeholder requirements

Quality Elicit and define quality requirements definition process
requirements
Formalise identified quality requirements Requirements analysis process

Identify & Architectural design process

formalise internal quality requirements
Implementation process

Integration process

Verification process

Transition process

Validation process

Operation process

Maintenance process

SQuaRE ISO/IEC 15288 (Technical Processes)

page 32
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

Evaluation process view according to ISO/IEC

14598-1

E s ta b lis h p u r p o s e o f e v a lu a tio n
E s ta b lis h
ev a lu a tio n I d e n tif y ty p e s o f p r o d u c t( s )
req u irem en ts
S p e c if y q u a lity m o d e l 9 1 2 6 -1 Q u a l i ty
C h a r a c te r is tic s

91 14 25 69 -6 2- 6E Ex tvear lnu aal t i o n

S e le c t m e t r ic s - MM e ot rrdi uc sl e s ri
19 1 2 6 8- 3- 6I nEt ve ar nl ua al t i o
S p ecify th e E s ta b lis h r a t in g le v e ls f o r m e tr ic s M
ev a lu a tio n M eo tdr ui clse s
9 1 2 6 - 4 Q u a lity in u s e
E s ta b lis h c r i te r ia f o r a s s e s s m e n t
M e tr ic s
1 4 5 9 8 - 6 E v a lu a tio n
M o d u les
D es ig n th e
P r o d u c e e v a lu a tio n p la n
ev a lu a tio n

M e a s u r e c h a r a c te r is tic s
E x ecu te th e
ev a lu a tio n C o m p a r e w it h c r i t e r i a

A s s e s s r e s u lts

page 33
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

Summary - 1
SQuaRE series addresses requirements and evaluation of
software product quality
• Internal, External, and Quality in Use Measures are the
link between requirements and evaluation

Software Product Quality Measurement Reference Model

is a specific instance of 15939 information model

Additional standards to create catalogues of measures for

quality attributes and measurement primitives

Need for validated measures to populate catalogues

page 34
© 2004 by Carnegie Mellon University
Carnegie Mellon
Software Engineering Institute

Summary - 2
Working to reconcile and harmonize SQuaRE series with
other standards

Concern over number of standards and fragmentation of

content

Reviewers sought

page 35
© 2004 by Carnegie Mellon University