You are on page 1of 83

ISO/IEC FCD 25040:2009(E)

ISO/IEC JTC1/SC7/WG6 Nxxxx


Date: 14-June-2009

ISO/IEC FCD 25040


TITLE:

Software engineering - Software product Quality Requirements Evaluation (SQuaRE) Evaluation reference model and guide
DATE: SOURCE: WORK ITEM: STATUS: 14-06-2009 JTC1/SC7/WG6 Project Version 3.0

and

DOCUMENT TYPE: FCD ACTION: Revision by Project editor before FDIS Ballot PROJECT EDITOR: Prof. Motoei AZUMA Department of Industrial Eng. and Management, Waseda University 3-4-1, Okubo, Shinjuku-ku, Tokyo 169-8555, Japan azumam@waseda.jp DOCUMENT EDITOR: Danilo SCALET CELEPAR Companhia de Informatica do Parana R Mateus Leme, 1142, 80530-010, Curitiba PR, Brazil danilo@pr.gov.br CO-EDITOR: CO-EDITOR: CO-EDITOR: CO-EDITOR: CO-EDITOR: Jrgen Begh, Denmark, jbh@terma.com Jean-Marc Desharnais, CA Mary Theofanos, USA Nigel BEVAN, UK, nigel@nigelbevan.com Kazuhiro Esaki, Japan, esaki.kazuhiro@ebara.com

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040:2009(E)

ISO/IEC JTC 1/SC 7 Nxxx Date: 2009-06-14

ISO/IEC FCD 25040


ISO/IEC JTC 1/SC 7/WG 6 Secretariat: SCC

Software engineering - Software product Quality Requirements and Evaluation (SQuaRE) Evaluation reference model and guide

Warning This document is not an ISO International Standard. It is distributed for review and comment. It is subject to change without notice and may not be referred to as an International Standard. Recipients of this document are invited to submit, with their comments, notification of any relevant patent rights of which they are aware and to provide supporting documentation.

Copyright notice
This ISO document is a working draft or committee draft and is copyright-protected by ISO. While the reproduction of working drafts or committee drafts in any form for use by participants in the ISO standards development process is permitted without prior permission from ISO, neither this document nor any extract from it may be reproduced, stored or transmitted in any form for any other purpose without prior written permission from ISO. Requests for permission to reproduce this document for the purpose of selling it should be addressed as shown below or to ISO's member body in the country of the requester: [Indicate the full address, telephone number, fax number, telex number, and electronic mail address, as appropriate, of the Copyright Manger of the ISO member body responsible for the secretariat of the TC or SC within the framework of which the working document has been prepared.] Reproduction for sales purposes may be subject to royalty payments or a licensing agreement. Violators may be prosecuted.

ii

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040:2009(E)

Contents

Page

Foreword ................................................................................................................................................................... vii Introduction................................................................................................................................................................ ix 1 2 3 4 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 4.10 4.11 5 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 6 6.1 6.2 6.3 6.3.1 6.3.2 6.3.3 6.3.4 6.4 6.4.1 6.4.2 6.4.3 6.5 6.5.1 6.6 6.6.1 6.6.2 6.6.3 6.7 6.7.1 6.7.2 Scope .............................................................................................................................................................. 1 Conformance ................................................................................................................................................. 1 Normative references.................................................................................................................................... 1 Terms and definitions ................................................................................................................................... 2 evaluation....................................................................................................................................................... 2 evaluation coverage ...................................................................................................................................... 2 evaluation level.............................................................................................................................................. 2 evaluation records......................................................................................................................................... 2 evaluation requester ..................................................................................................................................... 2 evaluation tool ............................................................................................................................................... 3 evaluation stringency.................................................................................................................................... 3 external measure of software quality .......................................................................................................... 3 independent evaluator .................................................................................................................................. 3 internal measure of software quality........................................................................................................... 3 software quality ............................................................................................................................................. 4 Software Product Quality Evaluation Reference Model ............................................................................ 4 Reference model - general ........................................................................................................................... 4 Reference model - evaluation processes.................................................................................................... 5 Inputs for the evaluation............................................................................................................................... 7 Outcomes of the evaluation ......................................................................................................................... 7 Constraints for the evaluation ..................................................................................................................... 7 Resources for the evaluation ....................................................................................................................... 8 Roles ............................................................................................................................................................... 8 Quality in the life cycle.................................................................................................................................. 9 Support for the evaluation.......................................................................................................................... 11 Software product quality evaluation process .......................................................................................... 11 General requirement ................................................................................................................................... 11 Documentation ............................................................................................................................................ 11 Establish the evaluation requirements ..................................................................................................... 14 Establish the purpose of the evaluation ................................................................................................... 14 Obtain the software product quality requirements.................................................................................. 15 Identify product parts to be included in the evaluation .......................................................................... 15 Define the stringency of the evaluation .................................................................................................... 16 Specify the evaluation................................................................................................................................. 16 Select quality measures (evaluation modules) ........................................................................................ 17 Define decision criteria for quality measures .......................................................................................... 17 Establish decision criteria for evaluation ................................................................................................. 18 Design the evaluation ................................................................................................................................. 18 Plan evaluation activities............................................................................................................................ 18 Execute the evaluation................................................................................................................................ 19 Make measurements ................................................................................................................................... 19 Apply decision criteria for measures ........................................................................................................ 20 Apply decision criteria for evaluation ....................................................................................................... 20 Conclude the evaluation ............................................................................................................................. 20 Review the evaluation result ...................................................................................................................... 20 Dispose evaluation data ............................................................................................................................. 20

ISO/IEC 2002 All rights reserved

iii

ISO/IEC FCD 25040:2009(E)

Annex A (normative) The Process for developers ...............................................................................................22 A.1 Activities and tasks .....................................................................................................................................22 A.1.1 General requirements..................................................................................................................................22 A.1.2 Documentation.............................................................................................................................................22 A.1.3 Establish evaluation requirements. ...........................................................................................................22 A.1.4 Specify the evaluation.................................................................................................................................23 A.1.5 Design the evaluation..................................................................................................................................25 A.1.6 Execute the evaluation................................................................................................................................25 A.1.7 Conclude the evaluation. ............................................................................................................................26 A.1.8 Quality evaluation review and feedback to the organization. .................................................................26 Annex B (normative) The Process for acquirers ..................................................................................................27 B.1 Activities and tasks .....................................................................................................................................27 B.1.1 General requirements..................................................................................................................................27 B.1.2 Documentation.............................................................................................................................................27 B.1.3 Establish evaluation requirements. This activity consists of the following tasks: ..............................27 B.1.4 Specify the evaluation.................................................................................................................................33 B.1.5 Design the evaluation..................................................................................................................................35 B.1.6 Execute the evaluation................................................................................................................................35 B.1.7 Conclude the evaluation. ............................................................................................................................36 Annex C (normative) The Process for independent evaluation..........................................................................37 C.1 Activities and tasks .....................................................................................................................................37 C.1.1 General requirements..................................................................................................................................37 C.1.2 Documentation.............................................................................................................................................39 C.1.3 Establish evaluation requirements. ...........................................................................................................39 C.1.4 Specify the evaluation.................................................................................................................................40 C.1.5 Design the evaluation..................................................................................................................................41 C.1.6 Execute the evaluation................................................................................................................................42 C.1.7 Conclude the evaluation. ............................................................................................................................43 Annex D (informative) Evaluation levels ...............................................................................................................45 D.1 Selection of evaluation levels ....................................................................................................................45 D.2 Selecting evaluation techniques from evaluation levels.........................................................................47 Annex E (informative) Evaluation methods ..........................................................................................................49 E.1 Review of user and technical product documentation (including on-line documentation)............................................................................................................................................49 E.2 Evaluation based on supplier courses and training ................................................................................49 E.3 Assessment of software engineering process .........................................................................................49 E.4 Review of operating history with the supplier..........................................................................................50 E.4.1 Operating history requirements.................................................................................................................50 E.4.2 Operating history review.............................................................................................................................50 E.5 Review of operating history with customers ............................................................................................51 E.6 Review of supplier capability, support, and quality system ...................................................................51 E.7 Prototyping and other evaluation methods ..............................................................................................52 E.7.1 Prototyping...................................................................................................................................................52 E.7.2 Other evaluation methods ..........................................................................................................................53 Annex F (informative) Example of Cost-Effectiveness Ranking of Evaluation Methods .................................54 Annex G (informative) Example of application of the evaluation process ........................................................55 Annex H (informative) Relationships between software product quality evaluation process reference model and software and system life cycle processes ..........................................................56 Annex I (informative) Evaluation report template ...............................................................................................57 I.1 Section 1 - Identifications ...........................................................................................................................57 I.2 Section 2 - Evaluation requirements .........................................................................................................58 I.3 Section 3 - Evaluation specification ..........................................................................................................58 I.4 Section 4 - Evaluation methods .................................................................................................................58 I.5 Section 5 - Evaluation Results ...................................................................................................................58

iv

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040:2009(E)

Annex J (Normative) Common terms and definitions ......................................................................................... 59 J.1 J.2 J.3 J.4 J.5 J.6 J.7 J.8 J.9 J.10 J.11 J.12 J.13 J.14 6.4 J.15 J.16 J.17 J.18 J.19 J.20 J.21 J.22 J.23 J.24 J.25 J.26 J.27 J.28 J.29 J.30 J.31 J.32 J.33 J.34 J.35 J.36 J.37 acquirer ........................................................................................................................................................ 59 analysis model............................................................................................................................................. 59 attribute ........................................................................................................................................................ 59 attribute for quality measure...................................................................................................................... 59 base measure............................................................................................................................................... 59 commercial-off-the-shelf software product .............................................................................................. 60 context of use .............................................................................................................................................. 60 custom software .......................................................................................................................................... 60 data ............................................................................................................................................................... 60 decision criteria ........................................................................................................................................... 60 derived measure .......................................................................................................................................... 60 developer...................................................................................................................................................... 60 division of standards .................................................................................................................................. 61 end user........................................................................................................................................................ 61 entity ............................................................................................................................................................. 61 evaluation method....................................................................................................................................... 61 evaluation module ....................................................................................................................................... 61 evaluator....................................................................................................................................................... 61 external software quality ............................................................................................................................ 61 failure ............................................................................................................................................................ 62 fault ............................................................................................................................................................... 62 functional requirement................................................................................................................................ 62 implied needs............................................................................................................................................... 62 indicator ....................................................................................................................................................... 62 information need ......................................................................................................................................... 62 information product .................................................................................................................................... 63 information system needs.......................................................................................................................... 63 intermediate software product................................................................................................................... 63 intermediate software product needs ....................................................................................................... 63 internal software quality ............................................................................................................................. 63 maintainer .................................................................................................................................................... 63 measure (noun)............................................................................................................................................ 63 measure (verb)............................................................................................................................................. 64 measurement ............................................................................................................................................... 64 measurement function ................................................................................................................................ 64 measurement method ................................................................................................................................. 64 measurement procedure ............................................................................................................................ 64 measurement process ................................................................................................................................ 64

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040:2009(E)

J.38 J.39 J.40 J.41 J.42 J.43 J.44 J.45 J.46 J.47 J.48 J.49 J.50 J.51 J.52 J.53 J.54 J.55 J.56 J.57 J.58 J.59 J.60 J.61 J.62 J.63

observation ..................................................................................................................................................65 operator ........................................................................................................................................................65 process .........................................................................................................................................................65 quality in use (measure)..............................................................................................................................65 Quality measure elements ..........................................................................................................................65 quality model................................................................................................................................................65 rating .............................................................................................................................................................65 rating level ....................................................................................................................................................66 requirements ................................................................................................................................................66 scale ..............................................................................................................................................................66 software product..........................................................................................................................................66 software product evaluation.......................................................................................................................66 software quality ...........................................................................................................................................66 software quality characteristic...................................................................................................................67 software quality evaluation.........................................................................................................................67 software quality in use ................................................................................................................................67 Software quality measure ...........................................................................................................................67 stakeholder...................................................................................................................................................67 supplier .........................................................................................................................................................67 system...........................................................................................................................................................68 target of process..........................................................................................................................................68 unit of measurement ...................................................................................................................................68 user ...............................................................................................................................................................68 validation ......................................................................................................................................................68 value..............................................................................................................................................................69 verification....................................................................................................................................................69

Bibliography ..............................................................................................................................................................70

vi

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040:2009(E)

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission) form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC participate in the development of International Standards through technical committees established by the respective organization to deal with particular fields of technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1. International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2. The main task of the joint technical committee is to prepare International Standards. Draft International Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as an International Standard requires approval by at least 75 % of the national bodies casting a vote. Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.

ISO/IEC 2002 All rights reserved

vii

ISO/IEC FCD 25040:2009(E)

ISO/IEC 25040 makes a part of SQuaRE series of standards and was prepared by Joint Technical Committee ISO/IEC JTC 1, information technology, Subcommittee SC 7, Software and System Engineering SQuaRE series of standards consists of the following divisions under the general title Software product Quality Requirements and Evaluation: ISO/IEC 2500n - Quality Management Division, ISO/IEC 2501n - Quality Model Division, ISO/IEC 2502n - Quality Measurement Division, ISO/IEC 2503n - Quality Requirements Division, and ISO/IEC 2504n - Quality Evaluation Division.

Annex A provides specific issues to be followed when the evaluation process is applied by developers. Annex B provides specific issues to be followed when the evaluation process is applied by acquirers. Annex C provides specific issues to be followed when the evaluation process is applied within a independent evaluation. Annex D provides an explanation on levels of evaluation, aspects to be considered when defining evaluation levels and suggestions on evaluation techniques to be applied according to the rank of evaluation level. Annex E provides examples of evaluation methods. Annex F provides a table showing relationships between some evaluation methods, possible costs rank and their effectiveness per software quality characteristics. Annex G provides an example of application of the evaluation process (to be included) Annex H provides relationships between software product quality evaluation process reference model and software and system life cycle processes. Annex I provides an example template for evaluation report. Annex J provides common SQuaRE definitions from ISO/IEC 25000.

viii

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040:2009(E)

Introduction
As the use of information technology grows, the number of critical computer systems also grows. Such systems include for example, security critical, life critical, economically critical and safety critical systems. The quality of software in these systems is particularly important because software faults maylead to serious consequences. Evaluation is the systematic determination of the extent to which an entity meets its specified criteria. The evaluation of software product quality is vital to both the acquisition and development of software. The relative importance of the various characteristics of software quality depends on the intended usage or objectives of the system of which the software is a part; software products need to be evaluated to decide whether relevant quality characteristics meet the requirements of the system. This document is part of SQuaRE series of standards and contains general requirements for software product quality evaluation as well as clarifies the associated general concepts. Specific issues related to the developers, acquirers and independent evaluation are provided in annexes. The general goal of creating the SQuaRE set of standards is to move to a logically organised, enriched and unified series covering two main processes: software quality requirements specification and software quality evaluation, supported by a software quality measurement process. The purpose of the SQuaRE set of standards is to assist those developing and acquiring software products with the specification and evaluation of quality requirements. It establishes criteria for the specification of software product quality requirements, their measurement, and evaluation. It includes a quality model for aligning customer definitions of quality with attributes of the development process. In addition, the series provides recommended measures of software product quality attributes that can be used by developers, acquirers, and evaluators. SQuaRE provides: Terms and definitions, Reference models, General guide, Individual division guides, and Standards for requirements specification, planning and management, measurement and evaluation purposes.

SQuaRE includes international standards on quality model and measures, as well as on quality requirements and evaluation. SQuaRE replaces the current ISO/IEC 9126 series and the 14598 series. This International Standard is intended to be used in conjunction with the other parts of the SQuaRE series of standards, and with ISO/IEC 14598 series and ISO/IEC 9126 series until superseded by the ISO/IEC 25000 series of standards.

ISO/IEC 2002 All rights reserved

ix

ISO/IEC FCD 25040:2009(E)

Quality Model Division 2501n Quality Requirements Division 2503n Quality Management Division 2500n Quality Measurement Division 2502n Extension Division Quality Evaluation Division 2504n

2503n

25050 - 25099

2504n

Figure 1 - Organisation of the SQuaRE series of standards Figure 1 illustrates the organisation of the SQuaRE series representing families of standards, further called Divisions. The Divisions within SQuaRE model are: ISO/IEC 2500n - Quality Management Division. The standards that form this division define all common models, terms and definitions referred further by all other standards from SQuaRE series. Referring paths (guidance through SQuaRE documents) and high level practical suggestions in applying proper standards to specific application cases offer help to all types of users. The division provides also requirements and guidance for a supporting function which is responsible for the management of software product requirements specification and evaluation. ISO/IEC 2501n - Quality Model Division. The standard that forms this division present detailed quality models for software, quality in use and data. Practical guidance on the use of the quality model is also provided. ISO/IEC 2502n - Quality Measurement Division. The standards that form this division include a software product quality measurement reference model, mathematical definitions of quality measures, and practical guidance for their application. Presented measures apply to internal software quality, external software quality and quality in use. Measurement primitives forming foundations for the latter measures are defined and presented, ISO/IEC 2503n - Quality Requirements Division. The standard that forms this division helps specifying quality requirements. These quality requirements can be used in the process of quality requirements elicitation for a software product to be developed or as inputs for an evaluation process. The requirements definition process is mapped to technical processes defined in ISO/IEC 15288 Information Technology - Life Cycle Management - System Life Cycle Processes, ISO/IEC 2504n - Quality Evaluation Division. The standards that form this division provide requirements, recommendations and guidelines for software product evaluation, whether performed by independent evaluators, acquirers or developers. The support for documenting a measure as an Evaluation Module is also presented.

ISO/IEC 25050 to ISO/IEC 25099 are reserved to be used for SQuaRE extension International Standards and/or Technical Reports.

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040:2009(E)

This International Standard is part of the 2504n Quality Evaluation Division that currently consists of the following International Standards: ISO/IEC 25040 Evaluation reference model and guide: contains general requirements for specification and evaluation of software quality and clarifies the general concepts. Provides a process description for evaluating quality of software product and states the requirements for the application of this process. The evaluation process is the basis for software product quality evaluation for different purposes and approaches. Therefore the process can be used for the evaluation of quality in use, external measure of software quality and internal measure of software quality as well as can be applied to evaluate the quality of pre-developed software or custom software during its development process. The software product quality evaluation can be conducted, for instance, by an acquirer, a developer organisation, or an independent evaluator. These three different approaches are presented as annexes that complement the main evaluation process providing issues to be followed according to each evaluation perspective. ISO/IEC 25041 - Evaluation modules: defines the structure and content of the documentation to be used to describe an Evaluation Module. These Evaluation Modules contain the specification of the quality model (i.e. characteristics, subcharacteristics and corresponding internal, external or quality in use measures), the associated data and information about the planned application of the model and the information about its actual application. Appropriate evaluation modules are selected for each evaluation. In some cases it may be necessary to develop new evaluation modules. Guidance for developing new evaluation modules is found in ISO/IEC 25041. This International Standard can also be used by organisations producing new evaluation modules; ISO/IEC 25045 - Evaluation module for recoverability: provides the specification to evaluate the subcharacteristic of recoverability defined under the characteristic of reliability of the quality model. It determines the external measures of software quality of resiliency and autonomic recovery index when the information system composed of one or more software products execution transactions is subjected to a series of disturbances. A disturbance could be an operational fault (e.g. an abrupt shutdown of an OS process that brings down a system) or an event (e.g. a significant increase of users to the system).

ISO/IEC 25040 - Evaluation reference model and guide is a revised version and replaces the current ISO/IEC 14598-1 - Software product evaluation General overview and ISO/IEC 14598-3, 4 and 5. Also, ISO/IEC 25041 - Evaluation modules replaces current ISO/IEC 14598-2.

ISO/IEC 2002 All rights reserved

xi

COMMITTEE DRAFT

ISO/IEC FCD 25040

Software engineering: Software product Quality Requirements and Evaluation (SQuaRE) Evaluation reference model and guide

Scope

This International Standard contains requirements and recommendations for the evaluation of software product quality and clarifies the general concepts. This document provides a process description for evaluating software product quality and states the requirements for the application of this process. The evaluation process can be used for different purposes and approaches. The process can be used for the evaluation of the quality of pre-developed software, commercial-off-the shelf software or custom software and can be used during or after the development process. This International Standard establishes the relationship of the evaluation reference model to the SQuaRE documents as well as shows how each SQuaRE document should be used during the activities of the evaluation process. This International Standard is intended for those responsible for software product evaluation and is appropriate for developers, acquirers and independent evaluators of software products. These three different approaches are presented as annexes that supplement the main evaluation process providing guidance to be followed according to each evaluation perspective. This International Standard is not intended for evaluation of other aspects of software products (such as functional requirements, process requirements, business requirements etc.).

Conformance

Evaluation of software product quality conforms to this International Standard if it follows the reference model of clause 6, which is common part for users who evaluate software, and any specific requirements in annex A, B or C, respectively to the developers, acquirers, or independent evaluation approach. For developers who evaluate their own software, the reference model of clause 6 and specific requirements in Annex A are applicable; For acquirers who evaluate acquiring software, the reference model of clause 6 and specific requirements in Annex B are applicable; For independent evaluation, the reference model of clause 6 and specific requirements in Annex C are applicable.

Normative references

The following normative documents contain provisions which, through reference in this text, constitute provisions of ISO/IEC 25040. For dated references, subsequent amendments to, or revisions of, any of these publications do not apply. However, parties to agreements based on ISO/IEC 25040 should investigate the possibility of applying the most recent editions of the normative documents indicated below. For undated references, the latest edition of the

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040

COMMITTEE DRAFT

normative documents referred to applies. Members of ISO and IEC maintain registers of currently valid International Standards. References informative to this standard are listed in Annex H. ISO/IEC 25000 Software engineering Software product Quality Requirements and Evaluation (SQuaRE) Guide to SQuaRE. ISO/IEC 25001 Software engineering Software product Quality Requirements and Evaluation (SQuaRE) Planning and management. ISO/IEC 25010 Software engineering Software product Quality Requirements and Evaluation (SQuaRE) Quality model. ISO/IEC 25020 Software engineering - Software product Quality Requirements and Evaluation (SQuaRE) Measurement reference model and guide. ISO/IEC 25030 Software engineering - Software product Quality Requirements and Evaluation (SQuaRE) Quality requirements. ISO/IEC 25041 Software engineering - Software product Quality Requirements and Evaluation (SQuaRE) Evaluation modules. Major revision of ISO/IEC 14598-6 that will replace this document when published.

Terms and definitions

For the purposes of this standard, the terms and definitions given in ISO/IEC 25000 (repeated in Annex J for convenience) and the following apply.

4.1 evaluation
systematic determination of the extent to which an entity meets its specified criteria
[ISO/IEC 12207:2008]

4.2

evaluation coverage

degree to which the evaluation covers the specified software product quality requirements

4.3 evaluation level


rigour to be applied during the evaluation that defines the depth or thoroughness of the evaluation in terns of evaluation techniques to be applied and evaluation results to be achieved

4.4 evaluation records


documented objective evidence of all activities performed and of all results achieved within the evaluation process

4.5 evaluation requester


person or organisation that requests an evaluation

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

4.6 evaluation tool


instrument that can be used during evaluation to collect data, to perform interpretation of data or to automate part of the evaluation.
NOTE: Examples of such tools are source code analysers to compute code metrics, CASE tools to produce formalised models, test environments to run the executable programs, checklists to collect inspection data or spreadsheets to produce syntheses of measures.

4.7 evaluation stringency


degree required for the software product quality characteristics and subcharacteristics to fulfil the expected use criticality of the software product

4.8 external measure of software quality


measure of the degree to which a software product enables the behaviour of a system to satisfy stated and implied needs when the system including the software is used under specified conditions
NOTE 1 Attributes of the behaviour can be verified and/or validated by executing the software product during testing and operation. EXAMPLE The number of failures found during testing is an external measure of software quality related to the number of faults present in the program. The two measures are not necessarily identical since testing may not find all faults, and a fault may give rise to apparently different failures in different circumstances. NOTE 2 Based on the ISO/IEC 25000:2005 definition of external software quality.

4.9 independent evaluator


individual or organization that performs an evaluation independently from developers and acquirers. NOTE: The individual or organization who performs as developers or acquirers for the target system to be evaluated may not become independent evaluator for the systems. The independent evaluator may be a organization. Independent evaluator may belong to the same organization as developers' organization as long as they are ind ependent from developers and acquirers.

4.10 internal measure of software quality


measure of the degree to which a set of static attributes of a software product satisfy stated and implied needs when the software product is used under specified conditions
NOTE 1 NOTE 2 Static attributes include those that relate to the software architecture, structure and its components. Static attributes can be verified by review, inspection and/or automated tools.

EXAMPLE Complexity measures and the number of faults found in a walk through are internal software quality measures made on the product itself. NOTE 3 Based on the ISO/IEC 25000:2005 definition of internal software quality.

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040

COMMITTEE DRAFT

4.11 software quality


degree to which the software product satisfies stated and implied needs when used under specified conditions
NOTE This definitions differs from the ISO 9000:2000 quality definition because the software quality definition refers to the satisfaction of stated and implied needs, while the ISO 9000 quality definition refers to the satisfaction of requirements.

[ISO/IEC 25000:2005 definition, rephrased as degree to which]

5
5.1

Software Product Quality Evaluation Reference Model


Reference model - general

The software product quality evaluation reference model describes the inputs and outcomes, constraints and resources for the software product quality evaluation process, as described in clause 6 (see Figure 2).

Constraints for the evaluation

Inputs for the evaluation

Evaluation process

Outcomes for the evaluation

Resources for the evaluation

Figure 2 Software product quality evaluation reference model The software product quality evaluation reference model applies to those responsible for software product evaluation. It is appropriate for organizations in their role as acquirers, developers, or evaluators. It is intended but not limited to, developers, acquirers and independent evaluators of software products. The software product quality evaluation reference model intends that the evaluation should be based on a software product quality requirement specification by using ISO/IEC 25030 before the evaluation and making clear the objectives and criteria of evaluations. ISO/IEC 25030 provides requirements and recommendations for software

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

product quality requirements specification. It applies other SQuaRE document such as ISO/IEC 25010 and ISO/IEC 2502n. Software product quality evaluation can be performed during or after the development process or acquisition process by the developer organization, the acquirer organization or an independent evaluator.

5.2

Reference model - evaluation processes

The software product quality evaluation process reference model describes the general processes and details the activities and tasks providing their purposes, inputs, outcomes and complementary information that can be used to guide a software product quality evaluation (Figure 3).

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040

COMMITTEE DRAFT

1. Establish purpose of the evaluation Establish evaluation requirements 2. Obtain the software product quality requirements 3. Identify product parts to be included in the evaluation 4. Define the stringency of the evaluation

Documentation Evaluation requirements Evaluation specification Evaluation design Evaluation result Evaluation conclusion

Specify the evaluation

1. Select measures (evaluation modules) 2. Define decision criteria for measures 3. Define decision criteria for evaluation

Design the evaluation

1. Plan evaluation activities

Execute the evaluation

1. Make measurements 2. Apply decision criteria for measures 3. Apply decision criteria for evaluaton

Conclude the evaluation

1. Review the evaluation results 2. Dispose evaluation data

Figure 3 Software Product Quality Evaluation Process Reference model Clause 6 details the activities and tasks providing their purposes, inputs, outcomes and complementary information that can be used to guide a software product quality evaluation. Notice that clause 6 intends to present the general

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

evaluation process and annex A, B and C provide specific requirements and guidance for each evaluation perspective, i.e. developer, acquirer and independent evaluator.
NOTE Annex A, B and C maintain the same structure of activities and tasks as in clause 6 in order to facilitate the traceability between the annex and the Software Product Quality Evaluation Process Reference Model and provide specific guidance and recommendations as necessary to each approach.

Requirements and recommendations for each activity and task are provided in clause 6.

5.3

Inputs for the evaluation

Inputs for the Software product quality evaluation process include the following: a) Software product quality evaluation requirements; b) Software product quality requirements specification (see ISO/IEC 25030); c) Software product and intermediate products to be evaluated.

5.4

Outcomes of the evaluation

As a result of the successful implementation of the software product quality evaluation process: a) A software product quality evaluation plan is established; b) The software product quality requirements are specified and prioritised; c) Quality measures are selected; d) Decision criteria defined for the quality measures; e) Decision criteria for the evaluation; f) The evaluation activities are planned;

g) The quality measures are obtained; h) The software product quality is assessed; i) j) The evaluation results are reviewed; A software product evaluation report is produced;

k) The evaluation data are disposed.

5.5

Constraints for the evaluation

Constraints for the software product quality evaluation process include the following: a) Software product quality evaluation needs constraints; b) Software product quality evaluation resources constraints; c) Software product quality evaluation schedule constraints;

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040

COMMITTEE DRAFT

d) Software product quality evaluation cost constraints; e) Software product quality evaluation environment constraints; f) Software product quality evaluation tools and methodology constraints;

g) Software product quality evaluation report constraints.

5.6

Resources for the evaluation

Resources for the software product quality evaluation process include the following: a) Applicable measurement tools and methodology, including evaluation modules; b) Applicable SQuaRE documents (ISO/IEC 25001, 25010, 2502n, 25041); c) Human resources for software product quality evaluation; d) Economical resource for software product quality evaluation; e) Information system for software product quality evaluation; f) Knowledge data base for software product quality evaluation.

5.7

Roles

Different roles including acquirers, developers, independent evaluator, suppliers, operators, and maintainers have different purposes. The acquirer: When acquiring a custom made software product, the acquirer establishes quality in use requirements and software quality requirements, specifies the requirements to the supplier, and evaluates potential purchases against these requirements before acquisition. When acquiring a product to be developed, the objective of specifying quality requirements is to ensure that the product meets the stated and implied needs of the user. When purchasing a software product, evaluation can be used to compare alternative products and to ensure that the selected product meets the quality requirements. The developer: When implementing a custom made software product, the developer should evaluate the intermediate software products or final product in order to assure the developed software quality. The developer can use the results of software product evaluation to ensure that products meet required quality criteria, which may be set by the acquirer, or by comparison with other products. The independent evaluators: When evaluating a target software product, the independent evaluator should evaluate the intermediate software products or final product in order to assure the software quality. The evaluation process for independent evaluation approach provides requirements and recommendations for the practical implementation of software product evaluation, when several parties need to understand, accept and trust evaluation results. It is used by evaluators carrying out an independent evaluation of a software product. This evaluation could be performed at the request of a developer, acquirer or some other party. The supplier: The supplier can use the results of software product evaluation to ensure that products meet required quality criteria, which may be set by the acquirer, or by comparison with other products. The operator: The individual or organisation, which operates a system of which the software product is a part, can use software quality evaluation to validate that quality requirements are met under variable

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

operating conditions, and to provide feedback on the need for any changes to those responsible for maintenance. The maintainer: The individual or organisation, which maintains a system of which the software product is a part, can use software evaluation to validate that quality requirements are still met, and requirements for maintainability and portability are achieved.

5.8

Quality in the life cycle

Figure 4 shows different approaches for specifying requirements and evaluating software quality. These approaches are related to the product to be evaluated: A software product in its context of use relates to quality in use; A software product as a part of a system in operation relates to external measures of software quality; Static artefacts of a software product relates to internal measures of software quality.

The objective is that when the software product is actually used by the user it meets the stated and implied needs. External measures of software quality can only be assessed for a system/subsystem of which the software product is a part. External measures are applied when executing the software. The values of external measures necessarily depend on more than the software, so the software has to be evaluated as part of a working system. Quality in use is the combined effect of the relevant quality characteristics for a particular user (who may be an end user, operator or maintainer). For software to have quality in use it has to meet the user needs to carry out particular tasks in particular hardware and software environments. Software, which performs satisfactorily in one environment, may show faults in another environment. External evaluation of quality characteristics therefore takes place under conditions that emulate as closely as possible the expected conditions of use. External measurements of characteristics are made when the code is complete, though as it may not be possible to emulate the exact conditions of use (e.g. network environment and user characteristics), external measures are often only indicators of the actual quality in use. If the external measures do not achieve the requirements for the external measures of software quality the results of the evaluation can be used as feedback to modify the software characteristics in order to improve the software quality, thus supporting an iterative improvement process.

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040

COMMITTEE DRAFT

Requirements

Needs

Product

Quality in use measures

Specifying

Quality in use Requirements Determine Validation

Quality in use

Evaluating

Indicates Software quality using external measures Indicates Software quality using internal measures Evaluating

External measures of software quality

Specifying

Software quality requirements Determine Verification & validation

Internal measures of software quality

Specifying

Software quality requirements Verification

Evaluating

Implementation

Figure 4 - Quality in the software life cycle For the purposes of development, requirements for the internal measures of software quality are defined which enable the quality of intermediate products to be verified. The static properties (e.g. the specification or source code) of the software can be measured by internal measures of software quality. Internal measures of software quality are of most interest during the development process. Internal measures of software quality can be used as indicators of external attributes of software quality. Test adequacy is an example of internal attribute of software quality that can be measured. Achievement of the required internal measures of software quality will contribute to meeting the requirements for the external measures of the software quality. Internal measures of software quality can thus be used as indicators to estimate external measures of software quality. For example, response time is an important measure required to evaluate the efficiency of the software, but response time cannot be measured during development. In order to evaluate the efficiency of the product during development, path length could be measured based on the intermediate products or specifications. This could be used as an indicator which provides rough estimates of response time under certain conditions. It is very important that internal measures of software quality attributes are related to requirements for external measures of software quality, so that the quality characteristics of software products under development (both intermediate and final item software products) can be assessed with respect to final system quality needs. Internal measures of software quality are of little value unless there is evidence that they are related to external measures of software quality. The specific attributes which are relevant to final quality will depend on the intended conditions of use - for an interactive product this will depend on the needs of the eventual end user and task. Other issues which will influence the needs for software product quality include whether the product is being purchased or developed, the stage of development, and the hardware, software and network environment in which the product will be used. External measures of software quality of a computer system can also be used as indirect measures of software quality in use. Thus the response time of a computer system can be used to measure the efficiency when using the software in a particular computing environment. For instance, in a banking teller system if the response time is

10

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

more than 3 or 4 seconds for a teller transaction, the system could be causing delays to customer service and lengthening customer wait times.

5.9

Support for the evaluation

These are activities for assisting evaluation by collecting information on software product quality evaluation methods and tools, developing and validating measures, and standardising evaluation process, and measures. ISO/IEC 25001 contains requirements and guidance for supporting processes for software product quality evaluation as well as for improving the organizational level evaluation process by means of, for instance, assessing each evaluation project and the own evaluation process. ISO/IEC 25041 defines the structure and contents of evaluation modules. Evaluation modules documents both evaluation technology and procedures for applying the technology.

6
6.1

Software product quality evaluation process


General requirement

The evaluator shall implement the activities and tasks described in clause 6 in accordance with applicable organization policies and procedures with respect to the software product quality evaluation process reference model (see Figure 3). Figure 3 is a simplified view of reality and does not show all iterations relating to specific activities or between activities. The evaluator shall ensure an infrastructure in the evaluators organization suitable for carrying out software product quality evaluation.
NOTE ISO/IEC 25001 provides guidance on planning and management organizational aspects related to software product quality evaluation.

The developer shall ensure an infrastructure that allows for data collection and process modifications based on data analysis. The evaluator shall ensure that personnel involved in the evaluation have the necessary skills and training.

6.2

Documentation

Each activity in the evaluation shall be documented. The documentation shall contain sufficient information required for the management of the evaluation. The documentation shall contain sufficient information for each activity for effective performance of subsequent activities of the evaluation. The documentation will at the end of the evaluation constitute the evaluation report. The evaluation plan is part of the evaluation report. The evaluation plan shall consider the evaluation purpose and the evaluation context within the organisation (see in ISO/IEC 25001 the role of an evaluation support group) and should include the following: The organisations involved in the evaluation, such as a independent evaluation organisation, software product developers and acquirers organizational units;

ISO/IEC 2002 All rights reserved

11

ISO/IEC FCD 25040

COMMITTEE DRAFT

The evaluation budget; Information products expected from the evaluation; Schedule for the evaluation milestones; Responsibilities for the parties involved in the evaluation; Evaluation methods and tools; Adopted standards; Evaluation activities. During the early evaluation some of the items of the evaluation plan can only be defined in a high level. Therefore, the evaluation plan shall be revised as the evaluation activities evolve, providing additional information that allows the plan to be adjusted or detailed.
NOTE 1 The high level evaluation plan is revised to the detailed level plan step by step in the successive evaluation activities and tasks.

An evaluation report format shall be prepared to ensure that sufficient information on evaluation results is provided. Such evaluation report format should be used from a standard such as the Common Industry Format (for instance ISO/IEC 25062), or it should be created by evaluators. The software product quality evaluation records shall include a detailed account of actions performed by the evaluator while executing the evaluation plan; these records are kept by the evaluator and used as basis for the creation of the evaluation report.
NOTE 2 The evaluation records are kept in order to allow re-processing of the evaluation results.

Depending on how the evaluation report is intended to be used, it should include the following items: 1. 2. 3. 4. 5. 6. 7. 8. 9. the software product quality evaluation requirements; the software product quality requirements; the software product quality evaluation plan; results from the measurements and analyses performed; intermediate results or interpretation decisions, when specified by the evaluation plan; any limitations, constraints, deficiencies, or exclusions in an evaluation activity, including their impact on the use, configuration, modification, or general maintenance of the software product over time; the evaluators and their qualifications; any differences between the product versions assessed and the corresponding evaluation inputs; i.e., documentation or courses; resolutions or workarounds in the event of a deficiency;

10. any other information necessary to be able to repeat or reproduce the evaluation;

12

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

11. result of the evaluation. As a result of the analysis of the evaluation activities the evaluation report should identify: 1. each deficiency, any relevant analysis, and how each deficiency was resolved. Resolution of deficiencies may include the fact that: one of the evaluation methods has provided assurance that the deficiency is not major; a satisfactory work-around can be found to alleviate the impact of the deficiency; e.g., modification to the product, disable or remove unneeded functionality, regenerate missing design requirements using reverse engineering; the original requirement is not mandatory and the deficiency can be accepted; the deficiency is acceptable provided that the use of the software product will be controlled by specific conditions or limitations; additional evaluation work is required to resolve the deficiency or gaps in the evaluation;

2.

any additional evaluations performed to resolve any identified deficiencies: to determine the scope or impact of a deficiency; to establish confidence that there is no deficiency; to verify that a workaround is technically feasible and/or suitable and acceptable; to verify the correct and acceptable performance of the software once a design change or changes have been made to correct deficiency.

3.

In a case where it is necessary to limit or control the use of the software product, whether the limitation: interferes with the software product meeting the mandatory requirements of the application; impacts on the application's design, budget, and schedule; requires additional evaluation work; introduces any possibility of failure in the application;

4.

any exclusions from scope of evaluation and/or restrictions on the results for each evaluation, such as: 'This evaluation does not include a detailed review of the functionality of the product.'; or 'This software product is deemed to be qualified to the required integrity level provided a full evaluation of the required functionality for the product is completed successfully..

5.

the integrated results of all the evaluation activities to allow an overall conclusion for the evaluation of the software product to be made.

NOTE 3 Extensive operating history may make up for a deficient software engineering process

ISO/IEC 2002 All rights reserved

13

ISO/IEC FCD 25040

COMMITTEE DRAFT

6.3

Establish the evaluation requirements

The following are required inputs for this activity: a) Software product quality evaluation needs; b) Software product quality requirements specification; c) Software product to be evaluated including intermediate products; d) Applicable measurement tools and methodology; e) Applicable SQuaRE documents (ISO/IEC 25001, 25010, 2502n, 25041 The following are outcomes of this activity: a) Specification of software product quality evaluation purposes;

b) Specification of software product quality evaluation requirements; c) Specification of software product quality evaluation plan.

This activity consists of the following tasks. 6.3.1 Establish the purpose of the evaluation

The purpose of the software product quality evaluation shall be documented as a basis for the further evaluation activities and tasks. Software product quality evaluation directly supports both the development and acquisition of software, which meets user and customer needs. The ultimate objective is to ensure that the product provides the required quality that it meets the stated and implied needs of the users (including operators, recipients of the results of the software, and maintainers of software). Software quality can be evaluated within a defined quality structure throughout the life cycle stages relating to software implementation process and acquisition process defined in ISO/IEC 12207 software life cycle processes and ISO/IEC 15288 system life cycle processes. The software product quality can be evaluated as an intermediate product or as a final product. The purpose of evaluation of intermediate software product quality may be to: decide on the acceptance of an intermediate software product from a subcontractor; access the ongoing feasibility of the development project; decide on the completion of a life cycle stage and when to send products to the next stage; predict or estimate final software product quality; collect information on intermediate software products in order to control and manage the process.

The purpose of evaluation of final software product quality may be to:

14

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

6.3.2

decide on the acceptance of the product; decide when to release the product; compare the product with competitive products; select a product from among alternative products; assess both positive and negative effects of a product when it is used; determine the cause of a failure in an investigation; decide when to enhance or replace the product. Obtain the software product quality requirements

The evaluator shall ensure that the stakeholders of the software product are identified.
NOTE 1 Information from the requester of the evaluation may be needed for identifying all stakeholders. NOTE 2 Stakeholders to be identified are a person, party or organisation and may be involved in the evaluation. Two kinds of stakeholders are to be identified. One is a stakeholder of software product, such as developer, acquirer, independent evaluator, user, operator, recipient of the results of the software, maintainer,, or supplier. Another is an evaluation requester who needs information about software quality, sponsors the evaluation and requires an evaluation report.

The evaluator shall ensure that the software product quality requirements are specified using a quality model.
NOTE 3 ISO/IEC 25010 defines a quality model in terms of quality characteristics and subcharacteristics. NOTE 4 ISO/IEC 25030 provides requirements and recommendations for specifying software product quality requirements. NOTE 5 If an existed software product quality requirements specification is available it may be reused, reviewed and refined.

The extent to which the quality evaluation covers the specified software quality requirements shall be defined, taking into account the software product quality requirements, in order to produce the software product evaluation requirements. This decision should be based on constraints such as evaluation budget, target date for the evaluation, purpose of the evaluation and use criticality of the software product.
NOTE 6 As the software product quality requirements specification may be improved during its development or acquisition process it may be necessary to refine the software product quality evaluation in order to make them compatible with the evaluation purpose.

6.3.3

Identify product parts to be included in the evaluation

The evaluator shall identify and document all product parts to be included in the evaluation. The type of intermediate or final software product to be evaluated (e.g. requirements specification, design diagrams and test documentation) depends on the stage in the life cycle and the purpose of the evaluation (see Figure 4). For instance, if the purpose of the evaluation is the selection of a product among alternative products, the products to be evaluated are mainly final software products or components. At the initial stages of the evaluation process it is not possible to identify a detailed list of products to be evaluated, since it also depends on the measures to be applied and on the artefacts to be produced along the software development life cycle. Therefore at that moment the components of the software product that are likely to be evaluated should be identified so that the initial list can be refined as the evaluation activities evolve.

ISO/IEC 2002 All rights reserved

15

ISO/IEC FCD 25040

COMMITTEE DRAFT

6.3.4

Define the stringency of the evaluation

The extent to which the quality evaluation covers the specified software quality requirements shall be defined, taking into account the software product quality requirements, in order to produce the software product evaluation requirements. This decision should be based on criteria such as evaluation budget, target date for the evaluation, purpose of the evaluation and use criticality of the software product.
NOTE 1 As the software product quality requirements specification may be improved during its development or acquisition process it may be necessary to refine the software product quality evaluation in order to make them compatible with the evaluation purpose.

The evaluation stringency shall be defined in order to provide confidence in the software product quality according to its intended use and purpose of the evaluation. The evaluation stringency should be related to a set of characteristics and subcharacteristics that establish expected evaluation levels which define the evaluation techniques to be applied and evaluation results to be achieved. ISO/IEC 15026 defines system and software integrity levels, see Annex D. The required integrity level of the software largely determines the rigor and formality of the evaluation process. The evaluation process should have the flexibility to accommodate the uniqueness of each application, to avoid unnecessary work or work that adds no value, and to provide a practical means of establishing the requisite confidence in the software.
NOTE 2 The following is an example of evaluation techniques to be applied to the functionality characteristic according to different evaluation levels requirements, from less demanding levels to more demanding levels: Functional or black box testing; Inspection of development documentation guided by checklists; Unit testing with test coverage criteria.

6.4

Specify the evaluation

The following are required inputs for this activity: a) Specification of software product quality evaluation requirements; b) Specification of the software product quality requirements; c) Applicable measurement tools and methodology; d) Constraints on the software product quality evaluation; e) Revised software product quality evaluation plan. The following are outcomes of this activity: a) Selected quality measures b) Specification of decision criteria for software product quality measures; c) Specification of decision criteria for software product quality assessment;

16

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

d) Revised software product quality evaluation plan. This activity consists of the following tasks. 6.4.1 Select quality measures (evaluation modules)

Software product quality evaluation requirements should be allocated to the software product components to which they are related in such a way that it is possible to define each appropriate measure and measurement method that can be used to evaluate the specified requirements. The evaluation methods shall be documented, taking into account the actions to be performed in order to achieve the evaluation results. When the evaluation method described is based on the use of a software tool, this tool shall be identified in the evaluation plan. Such identification shall include at least the name of the tool, its version identification and its origin (e.g. the supplier). The description of the evaluation methods shall be completed by the identification of product components on which the method is to be applied. When the evaluation specification is such that expert analysis of the measurements is required in order to interpret the results, the interpretation procedure shall be specified.
NOTE 1 ISO/IEC 25041 specifies how to package evaluation methods as an evaluation module, which also includes information on techniques, inputs to be evaluated, data to be measured and collected and supporting procedures and tools. NOTE 2 At this stage, the evaluation methods are related to elements in the evaluation specification, which are themselves related to evaluation requirements. Each of the evaluation methods is planned to be applied on the various product components submitted for evaluation. It can happen that several evaluation methods are to be applied to the same product component or consist of common parts. NOTE 3 When evaluation requirements refer to evaluation levels, informative annex E provides guidance on which evaluation technique to use as a function of the evaluation level and the quality characteristic considered. NOTE 4 The Technical ISO/IEC 25020 provides a Software Product Quality Measurement Reference Model, which may be applied to addresses the selection and construction of software quality measures.. NOTE 5 During the software quality requirements specification process, as defined in ISO/IEC 25030, it may be necessary to use quality measures as a reference to specify quality requirements, mainly those that relate to quantitative objectives.

Rigorous measurements are required to make reliable comparisons, either between products or with criterion values. Measurement procedures should measure the software quality characteristic (or subcharacteristic) they claim to be measuring with sufficient accuracy to allow criteria to be set and comparisons to be made. Data from checklists and expert opinion may not be reliable when comparing products with different attributes. Allowance should be made for possible measurement errors caused by measurement tools or human error.
NOTE 6 The Technical Reports ISO/IEC 25022, ISO/IEC 25023 and ISO/IEC 25024 give examples of measures that can be used as they are or adapted to specific needs. NOTE 7 The type of measurement required will depend on the purpose of evaluation. If the primary purpose is to understand and correct deficiencies, several measurements may be made on the software to monitor and control improvements. A wide range of measures can be useful for these purposes, including checklists and expert opinion. The primary requirement is that the measurements correctly identify the impact that any changes in the software have on quality.

6.4.2

Define decision criteria for quality measures

Decision criteria shall be established for the selected measures.


NOTE Decision criteria are numerical thresholds or targets used to determine the need for action or further investigation, or to describe the level of confidence in a given result. These will often be set with respect to quality requirements and corresponding evaluation criteria. Users may also use benchmarks, statistical control limits, historical data, customer requirements or other

ISO/IEC 2002 All rights reserved

17

ISO/IEC FCD 25040

COMMITTEE DRAFT

techniques to set decision criteria. For example, if estimated defect that exceeds the acceptable threshold, then perform additional defect detection and removal activities. This information is documented elsewhere; a reference to that location is adequate (see ISO/IEC 25020).

6.4.3

Establish decision criteria for evaluation

The evaluator should prepare a procedure for further summarization, with separate criteria for different quality characteristics, each of which may be in terms of individual subcharacteristics and quality measures, or a weighted combination of subcharacteristics and quality measures. The summarization results should be used as a basis for the software product quality assessment.
Note 1 The assessment criteria can be further used to support managerial decision, as the summarised quality can be compared with other aspects such as time and cost NOTE 2 To assess the quality of the product, the results of the evaluation of the different characteristics need to be summarised. NOTE 3 ISO/IEC 25020 shows the relationship between quality characteristics and subcharacteristics and quality measures as a software product quality measurement reference model.

6.5

Design the evaluation

The following are required inputs for this activity: a) Specification of software product quality evaluation requirements; b) Specification of software product quality requirements; c) Specification of selected quality measures (evaluation modules); d) Specification of decision criteria for software product quality measures; e) Specification of decision criteria for software product quality assessment; f) Software product quality evaluation plan.

The following are outcomes of this activity: a) Specification of detailed software product quality evaluation plan; b) Specification of software product quality evaluation methods. This activity consists of the following task.. 6.5.1 Plan evaluation activities

The identified software product quality evaluation activities shall be scheduled, taking into account the availability of resources such as personnel, software tools and computers.
NOTE 1 In scheduling the evaluation methods, it is important to recognize that a high degree of interdependence exists between the various evaluation methods, i.e. information obtained by one method may influence the focus of another method. As the nature of evaluation is iterative, issues may be revisited as information is gained. Therefore the evaluation plan will likely be altered as the evaluation is conducted. For example, it may be common for more detailed levels of evaluation to be considered either as unnecessary or as an additional requirement once the evaluation progresses.

18

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

NOTE 2 The evaluation of software products may be carried out in stages at different points in a development life cycle, or all at once, at one point in the life cycle. Different individuals or groups may be responsible for performing different parts of the evaluation. When the evaluation is done in stages, the steps of the evaluation activity are repeated in each stage until no further work is required.

The evaluation plan should avoid duplicating tasks within the evaluation and define decision points in the evaluation process which determine when and why the evaluation should be considered complete (i.e. acceptance or rejection criteria) and should be stopped This should be done in order to decrease the risk of errors and to reduce the planned evaluation effort, considering at least the following items: The evaluation budget; Evaluation methods and adapted standards; Evaluation tools; Evaluation activities, including the schedule and resources involved.

6.6

Execute the evaluation

The following are required inputs for this activity: a) Specification of detailed software product quality evaluation plan; b) Specification of software product quality evaluation requirements; c) Specification of software product quality requirements; d) Specification of selected quality measures; e) Specification of decision criteria for software product quality measures; f) Specification of decision criteria for software product quality assessment;

g) Specification of software product quality evaluation methods; h) Software product to be evaluated including intermediate products; i) Human recourses for evaluation.

The following are outcomes of this activity: a) Results of software product quality measures; b) Results of evaluation. This activity consists of the following tasks. 6.6.1 Make measurements

The selected software product quality measures shall be applied to the software product and components, according to the evaluation plan, resulting in values on the measurement scales.

ISO/IEC 2002 All rights reserved

19

ISO/IEC FCD 25040

COMMITTEE DRAFT

6.6.2

Apply decision criteria for measures

The decision criteria for the software product quality measures shall be applied to the measured values. The decision criteria for the software product quality assessment shall be applied to the measured values. 6.6.3 Apply decision criteria for evaluation

The set of decision criteria shall be summarised into subcharacteristics and characteristics, producing the assess results as a statement of the extent to which the software product meets quality requirements. The evaluation results should: 1. 2. 3. 4. 5. establish an appropriate degree of confidence that the software product is able to meet the evaluation requirements; identify any specific deficiencies with regard to the evaluation requirements and any additional evaluations needed to determine the scope of those deficiencies; identify any special limitations or conditions placed on the use of the software product; identify any weaknesses or omissions in the evaluation itself and any additional evaluation that is needed; identify any options for the use of the software product uncovered by the evaluation.

NOTE The apply decision criteria for evaluation task is a synthetic evaluation of software product quality and is not of software process assessment. The evaluation results can be used to support managerial decision, as the summarised quality is compared with other aspects such as time and cost. The managerial decision includes the acceptance or rejection, or on the release or no-release of the software product.

6.7

Conclude the evaluation

The following are required inputs for this activity: a) Revised software product quality requirements specification; b) Results of evaluation. The following are outcomes of this activity: a) Software product quality evaluation report. This activity consists of the following tasks. 6.7.1 Review the evaluation result

The evaluator and the requester shall carry out a joint review of the evaluation results. Comments to the evaluation report shall be disposed and included in the final version of the report. 6.7.2 Dispose evaluation data

When the evaluation is completed the data and evaluation items shall be disposed according to requirements of the requester.

20

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

If agreed by the requester data from the evaluation may be used by the evaluator to improve evaluation techniques.

ISO/IEC 2002 All rights reserved

21

ISO/IEC FCD 25040

COMMITTEE DRAFT

Annex A (normative) The Process for developers

A.1 Activities and tasks


The evaluation process for developers approach provides requirements and recommendations for the practical implementation of software product evaluation when the evaluation is conducted in parallel with the development. It is used by organisations that are planning to develop a new product or enhance an existing product and intending to perform product evaluation using members of its own technical staff. It focuses on the use of those indicators that can predict final product quality by measuring intermediate products developed during the life cycle. When applying the process for developers, the evaluator shall take into account the following specific issues.

A.1.1 General requirements


The developer shall develop the software following a disciplined development process that allows for planning and conducting software measurement and evaluation.
Note 1 Life cycle processes are described in ISO/IEC 12207. Development is described in subclause 5.3. Note 2 An overview of software product evaluation can be found in ISO/IEC 14598-1.

The developer shall coordinate evaluation activities with supporting processes and activities.
Note 3 Supporting processes are described in ISO/IEC 12207, including in particular the quality assurance process (subclause 6.3), the verification process (subclause 6.4), the validation process (subclause 6.5) and the audit process (subclause 6.7).

Many data analysis methods require data from previous projects developed under similar conditions and with comparable quality requirements. The developer should, therefore, apply a development model similar to one that has been used in previous projects in the developers organization. Also the same set of attributes should be applied in the projects to allow for data analysis.

A.1.2 Documentation
See clause 6.2.

A.1.3 Establish evaluation requirements.


A.1.3.1 Establish the purpose of evaluationsee clause 6.3.1

Software product quality requirements express the users needs for the software product under consideration, and are defined prior to the development (see ISO/IEC 25030). As a software product is decomposed into major components, the quality requirements derived from the overall product may differ for the different components, and may require different evaluation criteria. The results of software product quality evaluation can be used to obtain feedback on the extent to which different development processes, design methods or CASE tools can be used to meet quality requirements.

22

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

A.1.3.2

Obtain the software product quality requirementssee clause 6.3.2.

Prior to quality evaluation, quality requirements should be specified in terms of quality characteristics and subcharacteristics (see ISO/IEC 25010). At the initial stage of evaluation, these quality requirements should be studied and identified, for planning and implementing evaluation. The developer should establish both quality in use requirements based on user needs and requirements for external measures of software quality for each relevant quality characteristic. The completeness and correctness of the quality requirements specification should be evaluated to ensure that all the necessary requirements have been specified and unnecessary requirements excluded. The developer needs to evaluate the software product against these requirements before delivery. To achieve quality both stated and implied needs have to be met, so it is important to check that implied needs are specified in sufficient detail for all relevant quality characteristics. If possible, requirements should be assessed by procurers or purchasers, and by end users to assess the implied needs. User experience with prototypes frequently leads to a more accurate statement of requirements for quality in use. Software product quality evaluation should be used to predict and verify software product quality during development, by specifying requirements for internal measures of software quality for intermediate products in the development process. The external measures of software quality of the final product for specific intended uses can subsequently be evaluated against initial requirements. The goal is to identify problems in achieving the desired quality
as early as possible in the development process.

The developer should identify requirements for internal measures of software quality. When requirements for internal measures of software quality are used, the developer should identify these using a quality model which relates them to the requirements for external measures of software quality, and use the requirements for internal measures of software quality to verify quality of intermediate products during development. For the purposes of development, requirements for internal measures of software quality are defined which enable the quality of intermediate products to be verified. The internal measures of properties (e.g. the specification or source code) of the software can be measured by internal measures of software quality. Internal measures of software quality are of most interest during the development process. Internal measures of software quality can be used as indicators of external measures of quality attributes. Test adequacy is an example of quality attribute that can be measured by internal measures. Achievement of the required internal measures of software quality will contribute to meeting the requirements for external measures of software quality. Internal measures of software quality can thus be used as indicators to estimate external measures of software quality.

A.1.3.3 A.1.3.4

Identify products parts to be evaluatedsee clause 6.3.3 Define the stringency of the evaluationsee clause 6.3.4

A.1.4 Specify the evaluation.


A.1.4.1 Select measures (evaluation modules)see clause 6.4.1

This task is usually executed by iterations, mainly when evaluating quality of intermediate software products during development, since some measures can only be selected as the design of the software artifacts evolve. Internal measures of quality attributes can be used as quality indicators. In particular, internal measures of quality attributes are often used as indicators of external measures of quality attributes; but no general, direct relationship between quality indicators and external measures of quality attributes has been validated yet. However, it is commonly accepted that quality indicators provide useful guidance when used with care.

ISO/IEC 2002 All rights reserved

23

ISO/IEC FCD 25040

COMMITTEE DRAFT

Use of quality indicators allows the software developer to identify possible quality problems early in the development and to take corrective actions immediately. There is no known universal set of quality indicators that is suitable for every software development effort. There are differences in applications, development methods and tools, project organizations and cultural differences to mention some examples. Therefore, some indicators may be useful in one organization, but not work in another organization. For each requirement for external measures of software quality, one or more quality attributes are selected to represent that requirement during the development. Assigned target values to the internal measures of software attributes are used to control the quality during the development. The developer shall define a set of internal measures of software attributes that - covers every relevant intermediate product and activity, - is appropriate for the application domain and for the method to be used in the development, - cover identified product and development risks. Examples of development risks include unstable specifications, identified problems not being resolved, running behind schedule, etc. Appropriate trend measures should be included. When they are applied periodically, some measures are useful for identifying trends in the software development process. Examples of such trend measures are Number of completed modules, Number of resolved problems, Number of changed requirements, etc. The developer shall define a set of internal measures of software attributes that relate to all external measures of software attributes; i.e., to all quality requirements. These attributes are used as quality indicators. Relevant intermediate products should be analyzed and internal measurement data be collected for two purposes: evaluating the quality of the intermediate products to find indications of the fulfilment (or non fulfilment) of their quality requirements; getting an indication (prediction) of the quality of final product.

ISO/IEC 25022 can be used as guidance for selecting indicators. The developer shall describe the predictive model for the defined quality indicator; i.e., the relationship between the indicators and the external measures of software quality attributes. An indicator does not require a rigid one to one relationship with the quality attribute it seeks to measure. However, the link between the indicator(s) and the relevant quality attribute(s) should be clearly defined. For efficient management use, the number of indicators should be kept low. Priority should be given to indicators that can be supported by data already collected during existing processes, such as configuration management or integration testing. The developer shall define conditions under which the measurement is to be performed. This means identifying other attributes whose value influence the measurement and define the values of these attributes. By definition the value of internal measures of software attribute can be measured independently of other attributes.

24

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

A.1.4.2 A.1.4.3

Define decision criteria for measuressee clause 6.4.2 Establish criteria for evaluationsee clause 6.4.3

A.1.5 Design the evaluation.


A.1.5.1 Plan evaluation activitiessee clause 6.5.1

The developer shall define in which life cycle processes and activities the measurement and evaluation of the internal attributes will be implemented. The measurement and evaluation of the internal attributes will normally take place during the development process. The developer shall consider any influences on the software development activities. The set of measurements may imply a change in the development process, through its need for data acquisition. Hardware or software tools may have to be located, evaluated, purchased, adapted or developed to implement the measurements. The set of measurements may imply a change in the organizational structure used to produce the software system. The quality assurance / control organization or the entire development team may need training in the use of the measurements and data collection procedures. If the implementation of measurements has caused changes in the development process, the development team may need to be educated about the changes.

A.1.6 Execute the evaluation.


A.1.6.1 Make measurementssee clause 6.6.1

Quality monitoring and control takes place during the development. Actual values for the internal attributes are collected. In case of undesirable values, the cause is analysed, thereby allowing the developer to understand and react to problems. The developer shall collect actual measurement values for defined internal attributes according to the defined data collection actions. If the quality requirements are changed, the developer shall reconsider the specification of the evaluation and the design of the evaluation. A.1.6.2 A.1.6.3 Apply decision criteria for measuressee clause 6.6.2 Apply decision criteria for evaluationsee clause 6.6.3

During the software development it is usual to perform several assessments according to the milestones established in the evaluation plan. The results of the evaluation are important for supporting managerial decisions about next steps in the software development life cycle. For instance, do the requirements have to be changed or are more resources needed for the development process? The developer should use actual values of defined indicators to estimate final product quality. Experience from the development organizations previous projects with similar quality requirements should be taken into account. Quality prediction is dependent on validated indicators. A development organization will first need to collect indicator values and product measurement values for several projects to get a set of validated indicators. The developer should use actual values to monitor trends in order to identify development risks. Quality evaluation of the software product takes place when the development has been completed. Actual values for the external attributes are collected. If possible, components of the software may be measured before the development is completed.

ISO/IEC 2002 All rights reserved

25

ISO/IEC FCD 25040

COMMITTEE DRAFT

The developer should appropriately feedback the results of measures and evaluation to successive tasks or stages of development, maintenance of the software product or the next software development.

A.1.7 Conclude the evaluation.


A.1.7.1 A.1.7.2 Review evaluation resultsee clause 6.7.1 Dispose the evaluation resultssee clause 6.7.2

A.1.8 Quality evaluation review and feedback to the organization.


The developer shall make the data collected available to the organization for use in other development projects. The developer shall review the results of the evaluation and the validity of the evaluation process, indicators and metrics applied. Feedback from the review should be used in order to improve the evaluation process and evaluation modules. When it is necessary to improve the evaluation modules, the data collection for extra indicators should be included, in order to validate them for later use.
Note: Quality evaluation review and feedback is described in ISO/IEC 25001.

26

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

Annex B (normative) The Process for acquirers

B.1 Activities and tasks


The evaluation process for acquirers approach contains requirements, recommendations and guidelines for the systematic measurement and evaluation of software product quality during acquisition of Commercial-off-the-shelf software products, custom software products, or modifications to existing software products. It is used by organisations that are planning to acquire or reuse an existing or pre-developed software product. It can be applied for the purposes of deciding on the acceptance of the product or for selecting a product from among alternative products (a product may be self contained, a part of system, or it may be part of larger product); When applying the process for acquirers, the evaluator shall take into account the following specific issues.

B.1.1 General requirements B.1.2 Documentation B.1.3 Establish evaluation requirements. This activity consists of the following tasks:
B.1.3.1 Establish purpose of the evaluationsee clause 6.3.1

The evaluation process according to the acquirers perspective is, in general, part of an acquisition process, as defined in ISO/IEC 12207. In such case the purpose of the evaluation should establish: the acquisition process to be followed and how evaluation inputs requirements are to be communicated to the supplier; whether the software product will be used for a specific application, for a collection of specific applications, or for a generic range of applications; whether any evaluation has been done by second or third parties, or whether any evaluation activities are planned to be performed later.

NOTE See example of a combined evaluation and acquisition process in figures B.1 and B.2

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040

COMMITTEE DRAFT

Preselect products and suppliers based on product user feedback, review of product documentation and training courses, literature surveys, and product trial.

Prepare software and acquisition requirements and issue request proposal/tendering

Evaluate suppliers based on quality system, software engineering process, software maintenance process, and capability

Evaluate software products based on external evaluation results, product documentation, product operating experience, product prototyping, other product evaluation methods.

Select product and supplier and issue contract

Perform acceptance testing and accept/reject the product

Perform any additional evaluation

Integrate product into larger system or use standalone

Figure B.1 Example evaluation/acquisition process for off-the-shelf products

28

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

Issue request for information

Preselect supplier based on other customer feedback, quality system, software engineering/maintenance and capability.

Prepare software and acquisition requirements And issue request for proposal/tendering

Evaluate suppliers based on formal evaluation of quality system, software engineering process, and capability.

Select supplier

Specify requirements on supplier quality plan software engineering development process, verification and validation process, joint review and audit process, deliverables.

Issue contract along with any software product to be modified (if required) Manage and monitor supplier performance to supplier quality plan Perform acceptance testing and accept/reject delivery of Custom/modified software product

Perform any additional evaluation

Integrate product into larger system or use standalone

Figure B.2 Example evaluation/acquisition process for custom software or modifications to existing software The evaluation process may be combined with the acquisition process (defined in ISO/IEC 12207) summarized below, so that the evaluation results can contribute to the final acquisition objectives:

ISO/IEC 2002 All rights reserved

29

ISO/IEC FCD 25040

COMMITTEE DRAFT

Initiation - identification of software requirements for the product to be acquired, the acquisition plan, and the acceptance strategy and criteria; Request-for-proposal (-tender) preparation - specification and documentation of acquisition requirements; Contract preparation and update - supplier selection, contract preparation and negotiation, and contract change control; Supplier monitoring - evaluation activities performed during contract execution leading to acceptance and delivery of the software product; Acceptance and completion - activities performed during product acceptance and delivery of the final software product.

The acquirer needs to define both the evaluation process and the acquisition process to achieve the evaluation requirements during acquisition. In the context of larger system development, the acquisition and evaluation activities to be followed need to be integrated with other development and integration activities, and identified in a project measurement plan as specified in ISO/IEC 25001; i.e., specific acquisition implementation considerations for evaluation include the following considerations: a software requirements specification required for evaluation can form the basis for acquisition requirements required for the request-for-proposal (-tender); separate preliminary evaluation activities may be needed to pre-select software products and suppliers; supplier and product information requirements for evaluation need to be specified in the acquisition requirements or during contract preparation; evaluation activities can be executed as part of proposal evaluation, during monitoring of contract execution, as part of product development, as part of formal product acceptance, or after product delivery.

The evaluation process should consider: all project management constraints; i.e. availability of resources and expertise to perform the evaluation activities, schedule and budget allowance, possible dependencies or risks, key assumptions, or assumptions about the evaluation effort itself; the acquisition process and information required from suppliers during tendering; evaluations performed by other second (for example, acquirer who evaluate the software product provided by supplier) or third parties that could be used to reduce the evaluation effort for the product; whether the product will be reused in future applications and the documentation necessary to support any future evaluations of the product; whether the evaluation methods are still compatible with the requirements specification; The different views of evaluation according to the acquisition perspective as in figure B.3. This figure shows that the evaluation of a software product may take into account information from different views such as the product view, the process view and product in use view, resulting in the view of the evaluation results.

30

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

VIEWS OF EVALUATION INPUTS


User & Technical Documents
V I E W

PRODUCT VIEW Software Quality Cost Requirements Assessment

P R O C E S S

Quality System Engineering Process

Evaluation Process

Operating History Failure Information

P R O D V U I C E TW INU S E

Maintenance Process

Prototyping/Testing

Necessary Level of Confidence That Quality Requirements Met

Constraints Usage of Product

Additional Verification Activities Required

VIEW OF EVALUATION RESULTS

Figure B.3 - Software product evaluation process overview from acquirers perspective

B.1.3.2

Obtain the software product quality requirements see clause 6.3.2

The requirements specification forms the basis for the acquisition requirements used during the tendering step in the acquisition process and the basis against which subsequent product evaluation is performed. B.1.3.3 Identify product parts to be included in the evaluated see clause 6.3.3

Candidates for use and acquisition are software products, which can be integrated into a larger system as components or which can be used stand-alone. They are classified as: a) b) Commercial-off-the-shelf software products; Existing software products developed or acquired for other applications, or for a wide range of common applications;

ISO/IEC 2002 All rights reserved

31

ISO/IEC FCD 25040

COMMITTEE DRAFT

c)

Custom software products or modifications to existing software products.

In the case of software configuration items that are to be integrated into a larger system, software requirements need to be defined for each item. In other cases, the system and the software configuration items coincide, and may be considered equivalent. Hardware configuration items to be acquired may contain software such as an operating system resident in firmware (i.e. ROM, PROM). When the existing software forms an integral part of the hardware in this fashion it usually needs to be evaluated with the hardware configuration item. In addition the following shall be taken into consideration: a) whether the supplier or developer is willing and able to provide access to the required documentation, equipment, tools, software, courses and/or training, and the costs associated with this;

b) any conditions associated with the provision of access to any confidential or proprietary information; c) whether the supplier or third party is willing and able to provide access to individuals with the correct expertise to answer questions, and what are the costs associated with this, including travel costs;

d) the expertise that is required by the evaluator to conduct the evaluation based on the evaluation requirements, and any costs associated with obtaining this particular expertise; e) any pre-tests required to establish that a product is fit for full scale testing;

B.1.3.4

Define the stringency of the evaluation see clause 6.3.4

The scope of the evaluation process can be reduced through access to the results of evaluation activities performed by second or third parties as long as the results are trustworthy. Such evaluation activities can comprise preexisting certifications, product evaluations and/or process assessments. For example: software engineering processes for product development may be standardized to meet the requirements of ISO/IEC 12207, ISO/IEC 90003, or other sector-specific standards; the suppliers quality system under which the software is developed may be certified to ISO 9001 requirements by a third party; the software product may be evaluated by second or third parties to ISO/IEC 25040 or ISO/IEC 25051 conformance; the suppliers software process capability for acceptable product development may be assessed by third parties to ISO/IEC 15504 conformance; the software may be functionally evaluated as part of a larger system development phase; the software product may have been previously evaluated for another application with different integrity requirements; the level of coverage relative to the evaluation requirements that is necessary after review of any previous evaluations conducted by others; evaluations on the product may have been performed by other parties within the organization through informal or formal evaluation activities.

32

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

The additional costs and time required to obtain and interpret external evaluation results for the target application may affect the feasibility of this method. It may still be necessary to consult the evaluator or supplier in order to attain adequate confidence in the results of others. The evaluation process can be applied to a wide range of acquisition requirements, integrity requirements, and evaluator objectives; for example: acquirers of software packages may wish to evaluate a software package using only ISO/IEC 25051; acquirers of software products may use ISO/IEC 25040 and independent evaluation; a small or sole acquirer may require a less formal evaluation process with minimum documentation of the evaluation; for consumer software the evaluation process objective may simply be to select, test and acquire one product from among a number of similar products. The formal acquisition process is then reduced to outright purchase, and does not include contract preparation.

The acquisition process can also be tailored as for the evaluation process using the tailoring guidance given in ISO/IEC 12207 and the required integrity level for the specific software product to be acquired. Acquisition of complete software systems with high integrity requirements will usually invoke the full set of acquisition activities and tasks, along with corresponding supply process activities and tasks that are specified in ISO/IEC 12207. Generally, as the integrity level increases, so should the amount of rigor, and the number of activities and tasks associated with the acquisition and the evaluation process.

B.1.4 Specify the evaluation.


B.1.4.1 Select measures (evaluation modules)see clause 6.4.1

The selection of evaluation methods to be used includes review or assessment of one or more of: 1. 2. 3. 4. 5. 6. 7. 8. Software product user and technical documentation (including on-line documentation) (see Annex E.1); Software product evaluation based on supplier courses and training (see Annex E.2); Software engineering process, including intermediate software products (see Annex E.3); Product operating history with supplier (see Annex E.4); Product operating history with customers (see Annex E.5); Supplier capability, support, and quality system (see Annex E.6); Prototyping or other evaluation methods (see Annex E.7); Product deficiency lists and related information (usually found on web sites).

A combination of evaluation methods should be specified to allow selection of the product or to establish the product's fitness for use. Issues to be evaluated include: a) whether some of the considerations may be mutually conflicting (e.g. "Are costs of the selected methods within budget? may not be compatible with "Do the methods collectively address all of the evaluation requirements?"). In this case it is up to the evaluator to make the necessary tradeoffs based on the prioritisation of evaluation requirements;

ISO/IEC 2002 All rights reserved

33

ISO/IEC FCD 25040

COMMITTEE DRAFT

NOTE Annex F shows an example of ranking evaluation methods for software quality characteristics by cost and effectiveness.

b) whether the evaluation provides adequate coverage or scope in the combination of methods chosen considering: 1. 2. 3. 4. 5. 6. 7. 8. c) how to demonstrate that software meets its specifications; overlapping of coverage of methods to provide additional confidence; whether the set of activities, as a whole, provides an acceptable level of assurance that there is complete coverage for those software quality characteristics of concern; the degree to which the methods complement each other; the effectiveness and objectivity of each method in addressing a variety of characteristics; the variety of distinct approaches among the evaluation methods (e.g. some review, some analysis, and some testing based methods); taking credit for any evaluation activities on the application that will ultimately be conducted as part of the overall system development life cycle; crediting evaluations performed by others;

use of informal preliminary evaluation activities like reviews or surveys or peer/user anecdotal experience, trade journal product reviews, accessible product user documentation, or database repositories of product reviews, in order to narrow the selection of products considered functionally suitable for further evaluation.

Before crediting evaluations performed by others, the following should be considered: a) whether the evaluation addressed the evaluation requirements with a degree of rigor consistent with the integrity level of the application;

b) whether the evaluation report identified the version of the software product being assessed, the extent of the evaluation, the decision criteria used, and the conclusions reached; c) whether the evaluation report identified any deficiencies in the software product or the software engineering process, recommended any corrective action to address those deficiencies, and whether the corrective action was carried out;

d) whether the evaluator had appropriate profile, including: 1. 2. 3. 4. experience in performing evaluation and analysis; experience in software quality relative to internationally accepted standards; expertise in software engineering issues; total independence from the supplier.

34

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

B.1.4.2 B.1.4.3

Define decision criteria for measuressee clause 6.4..2 Define decision criteria for evaluation see clause 6.4.3

B.1.5 Design the evaluation.


B.1.5.1 Plan evaluation activitiessee clause 6.5.1

The evaluation plan should consider the following aspects that are mainly related to the acquirers perspective: f) any costs associated with providing the test environment (e.g. test hardware, support equipment and tools, specialized personnel) for performing the evaluation;

g) responsibility for the evaluation and the required schedule; h) any limitation or deficiency in the evaluation method in providing assurance of quality, and whether that limitation or deficiency is covered elsewhere in the plan; e.g. the evaluation method is unable to cover all subcharacteristics for a specific quality characteristic; i) j) any interdependencies between the various evaluation methods used; i.e., order dependencies (information obtained in one test may be useful in another), establishing an optimal sequence of evaluation methods; the resources required and the cost for the total evaluation and for each evaluation method;

k) the tie points between evaluation activities and acquisition activities (See example combined evaluation and acquisition process in Figures B1 and B2); l) the decision points in the evaluation process which determine when and why the evaluation should be considered complete (i.e. acceptance or rejection criteria) and should be stopped;

m) whether the rationale, justification, and assumptions behind any unusual or exceptional decisions made in defining the evaluation plan are documented;
NOTE Some of the previous issues may be applied to the general evaluation process, according to the requirements and conditions of each particular software product evaluation.

B.1.6 Execute the evaluation.


B.1.6.1 B.1.6.2 B.1.6.3 Make measurementssee clause 6.6.1 Apply decision criteria for measuressee clause 6.6.2 Apply decision criteria for evaluationsee clause 6.6.3

The conclusions should state whether the software product is adequate and appropriate for use in the intended application, taking into consideration the application integrity level and the actual evaluation requirements. If it is not possible to use the software product "as is", due to deficiencies discovered or due to lack of evaluation information, then it is necessary to make recommendations to perform further evaluation or to control or limit the use of the software product in its target application. The conclusions may be formalized using a "statement of requirements compliance" which would describe for each of the specific requirements the feature(s), function(s), or service(s) of the software product used to meet that requirement, and the evaluation method (s) that provide adequate confidence that the requirement has been met.

ISO/IEC 2002 All rights reserved

35

ISO/IEC FCD 25040

COMMITTEE DRAFT

Potential design strategies such as implementation of design diversity, redundancy in configuration, interface integrity checks and recovery techniques may compensate for deficiencies or potential failures of the software product. It is possible that the evaluation may result in a decision not to accept the software product for use, or a decision not to attempt to comply with the evaluation requirements, and in a recommendation to re-evaluate alternative approaches. The ultimate decision is to buy or not to buy. The decision to buy can then result in a contract to purchase the software product with possible additional evaluation in the form of product acceptance testing. The decision not to buy can result in possible alternatives which include modifying the software product, developing a custom software product, or changing the requirements.

B.1.7 Conclude the evaluation.


B.1.7.1 B.1.7.2 Review the evaluation resultssee clause 6.7.1 Dispose evaluation datasee clause 6.7.2

36

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

Annex C (normative) The Process for independent evaluation

C.1 Activities and tasks


The evaluation process for independent evaluation approach provides requirements and recommendations for the practical implementation of software product evaluation, when several parties need to understand, accept and trust evaluation results. It is used by evaluators carrying out an independent evaluation of a software product. This evaluation could be performed at the request of a developer, acquirer or some other party. This annex is intended for those who perform independent evaluation. Often they work for third party organisations.
NOTE In order to apply well designed methods and to provide more objective and reliable evaluation from an expert view with experience and knowledge, the independent evaluator is necessary and applied for software product quality evaluation. The most typical independent evaluator is an expert who works for a different company from acquirer and supplier and who evaluates software product objectively, based on request from any of user, acquirer, developer or supplier. The independent evaluator usually comes into situations or has roles to establish collaborative work with, to negotiate issues with and to agree with user, acquirer, developer or supplier on its evaluation activities such as access to the software product and executable environment.

This Annex C describes such specific issues for independent evaluator. When applying the process for evaluation, the evaluator shall take into account the following specific issues.

C.1.1 General requirements


In order to ensure repeatability, reproducibility, impartiality and objectivity of the evaluation results, the evaluator shall act in an organisational context that provides all necessary assurance to obtain sufficient quality for its activities. The responsibilities of the requester of the evaluation shall be: to establish necessary legal rights in the software product for the purpose of the evaluation; to provide information necessary for identification and description of the product; to state initial evaluation requirements and to negotiate with the evaluator to determine the actual evaluation requirements; these requirements for the evaluation should comply with relevant regulations and standards; to state confidentiality requirements concerning the information submitted to the evaluation; to act, whenever necessary, as an intermediary between the developer and the evaluator; to provide the evaluator, whenever necessary, with suitable access to computers and other equipment used for development and for operational use of the software product; to provide, whenever necessary, support to the evaluator, including training and access to suitable staff,;

ISO/IEC 2002 All rights reserved

37

ISO/IEC FCD 25040

COMMITTEE DRAFT

to ensure the timely supply, whenever necessary, of the software product, its description and components, including documentation and other material; to inform, whenever necessary, the evaluator of any factor that might invalidate the evaluation results.

Potential requesters of evaluations are, for example, software developers; software suppliers; software acquirers; software users; system integrators in their role of software acquirers.

The responsibilities of the evaluator shall be: to check that the requester has the sufficient legal rights in the software product for the evaluation to be performed; to do so, the evaluator may require an attestation from the requester; to keep the confidentiality as required, of all the information provided by the requester, including, for example, the product under evaluation, the evaluation records and the evaluation report; to provide qualified and trained staff to conduct the evaluation; to provide the evaluation tools and technology; to conduct the evaluation in accordance with the evaluation requirements and; to maintain records of any work performed during the evaluation which has an impact on the evaluation results; to ensure timely delivery of the evaluation report to the requester, to provide the visibility into the conduct of the evaluation to the extent requested by the requester.

Potential evaluators are, for example, independent testing laboratories; testing entities within software producing or distributing organizations; testing entities within software buying or using organizations; testing entities within system integration organisations; organisations making comparisons between products.

38

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

C.1.2 Documentation
Annex I provides a template for an evaluation report.

C.1.3 Establish evaluation requirements.


C.1.3.1 C.1.3.2 C.1.3.3 Establish purpose of the evaluationsee clause 6.3.1 Obtain the software product quality requirementssee clause 6.3.2 Identify product parts to be included in the evaluationsee clause 6.3.3

The evaluation requester shall provide a description of the product submitted for evaluation. The goal of this description is: to allow to define the scope of the evaluation, i.e. the identification of those software product components that are to be considered as part of the product and the identification of those software product components that are not to be considered as part of the product and which are only referred to for the ease of understanding the product;

NOTE 1 Such identification may be based on specifying which parts of the documents belong to the product, which function is implemented or not by the product. NOTE 2 Defining the scope of the evaluation is important when the software product submitted for evaluation is embedded in a system consisting of hardware, other software products, networks or organisations because the separation between such products is not always obvious.

to give to the evaluator the identification of product components submitted for evaluation, to understand their structure and to identify the information provided as well as how to access it.

This description shall contain the list of product components actually submitted for evaluation, a rationale about the structure of the product and a list of product related documents. The components listed may contain other smaller components which need not be listed. For each component and product related document in the lists, the following information shall be provided: description of the nature of the component; information about formalisms used within the component; information about the size of the component; relationship with other components; information about availability of the product component to the evaluator.

In any case, reference to appropriate software engineering standards should be made. The evaluator shall check that the product description conforms to the above mentioned requirements. The evaluator shall analyse the rationale provided as well as the description of the components in order to identify their relationship with the components identified in the evaluation requirements.
NOTE 3 In the evaluation requirements, components may be specified from a theoretical point of view, with regard to quality characteristics to be assessed. In the product description, actual components are listed. It may happen that some actual components of the product contain information that the evaluation requirements specifies as being in several components.

ISO/IEC 2002 All rights reserved

39

ISO/IEC FCD 25040

COMMITTEE DRAFT

NOTE 4 This information is needed in order to identify which evaluation can be performed. This will be used, together with the evaluation requirements, to build the evaluation specification. NOTE 5 The analysis of the product description may be improved by consultation with the developer of the product. This would provide an opportunity for the evaluator to establish whether an evaluation to the depth required will be possible, by performing a brief audit.

The requester should deliver the product components and product related documents to the evaluator. The evaluator shall register all the product components and product related documents. When the size and complexity of components justifies it, formal configuration management should be used. The registration information should be at least: component or document unique identifier; component name or document title; condition of document (including especially anomalies or physical conditions); version, configuration and date information as provided by the requester; date of receipt.

The confidentiality of all the product components and product related documents shall be protected by the evaluator unless otherwise agreed with the requester.
NOTE 6 The confidentiality requirements affect many aspects of evaluation work, including receipt, handling, storage and disposal of all the product related information.

C.1.3.4

Define the stringency of the evaluationsee clause 6.3.4

The evaluation requirements shall be approved as a result of a joint review between the requester and the evaluator.

C.1.4 Specify the evaluation.


The evaluation specification should not contain proprietary information of the evaluator.
NOTE The evaluation report, which contains the evaluation specification, is delivered to the evaluation requester who may disclose it to other parties. Therefore it would not be advisable for the evaluator to try to protect some proprietary information.

C.1.4.1

Select measures (evaluation modules)see clause 6.4.1

The evaluator shall check that the components listed in the product description provide all the necessary information to perform the evaluation according to the evaluation requirements. The evaluator shall also verify that the measurements and verifications specified are sufficient to meet the objectives of the evaluation as expressed in the evaluation requirements. The first check can lead to the identification of missing information in the components listed in the product description. This may be solved in one of the following ways:

40

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

a reference to a product component containing the missing information shall be added in the product description; this means that the requester will provide this component together with the others for the performing of the evaluation; the objectives of the evaluation shall be reduced, which means that the evaluation requirements are revised.

The second check aims at confirming that the measurements and verifications proposed in the evaluation specification are consistent with the technical state of the art. This may be done in one of the following ways: by identifying relevant measurement standards;

NOTE Such standards may be evaluation modules as prescribed in ISO/IEC 25041.

by providing a detailed justification, referencing appropriate authoritative literature in the field; this justification shall be included in the evaluation specification.

The evaluation specification shall comprise: the scope of the evaluation referring to the product components as identified in the product description; a cross-reference between the information needed to perform the evaluation and the product components and other related documents listed in the product description; a specification of measurements and verifications to be performed and the references to product components on which they are to be performed; a mapping between the specification of measurements and verifications and the evaluation requirements together with the reference to standards or the justification for each measurement or verification listed.

The evaluation specification shall be approved as a result of a joint review between the requester and the evaluator. C.1.4.2 C.1.4.3 Define decision criteria for measuressee clause 6.4.2 Define decision criteria for evaluationsee clause 6.4.3.

C.1.5 Design the evaluation.


C.1.5.1 Plan evaluation activitiessee clause 6.5.1

The evaluator shall take into account the availability of resources such as personnel, software tools and computers. The evaluator shall agree with the requester the delivery schedule for the product and its components. The medium and format for delivery of product components as well as the number of copies shall be specified. The requirements for meetings during the course of the evaluation shall be identified. When the requester is not the software product developer, the relations between the evaluator and the developer shall be identified. In particular, the support needed from the developer shall be specified. Such support may include, for example, training, informal discussions or office accommodation. Access to development and operational sites, when necessary, shall be specified together with the needed resources.

ISO/IEC 2002 All rights reserved

41

ISO/IEC FCD 25040

COMMITTEE DRAFT

The evaluation plan shall include the documentation of the evaluation methods and the schedule of the evaluator actions. The documentation of some evaluation method in the evaluation plan may consist of reference to private evaluator material. In that case, the evaluator should be able to justify the pertinence of the method with regard to the corresponding element of the evaluation specification and its own competence in applying the method. The evaluation plan shall be approved as a result of a joint review between the requester and the evaluator.

C.1.6 Execute the evaluation.


C.1.6.1 Take measuressee clause 6.6.1

Performing the evaluation actions usually consists of measuring the product and its components to obtain intermediate data and to interpret these data in order to produce results to be included in the evaluation report. Intermediate data may be of varied nature such as, for example, numbers, figures, diagrams, excerpts from components or formalised models produced for the evaluation. Confidentiality of intermediate data shall be protected in the same way as that of the original components and documents. Moreover, the evaluator shall make all the necessary effort to prevent any accidental or malevolent modification of these data. In particular, when the amount and the complexity of intermediate data is large, formal configuration management should be used to keep the consistency between intermediate evaluation results and the evaluated product. The evaluator shall include in the evaluation records any intermediate data on which any interpretation is based. The decisions made during the interpretation process shall also be included in the evaluation records as specified in the evaluation plan. The evaluation actions may require the use of software tools to collect raw data or to perform interpretation of intermediate data.
NOTE 1 Examples of such tools are source code analysers to compute code measures, CASE tools to produce formalised models, test environments to run the executable programs or spreadsheets to produce syntheses of measures.

When a tool is used to perform an evaluation action, reference to the tool shall be included in the evaluation report. The reference shall consist of the identification of the tool and of its supplier and the version of the tool. A more detailed reference to the tool used shall be included in the evaluation records. It shall include the detailed configuration of the tool and any pertinent information needed to be able to repeat the evaluation action in order to obtain the same intermediate result.
NOTE 2 This repeatability requirement refers to intermediate results which are not included in the evaluation report. NOTE 3 In some cases it might be pertinent to include a copy of the executable tool in the evaluation records.

The evaluator should make all necessary effort to assure that the tools used actually work as they are supposed to. The evaluator should maintain records of the actions undertaken to validate the tools which are used in the evaluation process.
NOTE 4 Such records can be based, for example, on the number of existing installation of the software or the amount of time during which the tool has been used.

The evaluator staff shall be trained to use the tools appropriately.

42

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

In some cases, evaluation actions cannot be performed within the evaluators premises. They may be performed, for example, at the developers site or at a site where the software product is in operation. When this occurs, the evaluator shall control all the evaluation actions performed. In particular the evaluator shall avoid any circumstances that would invalidate the evaluation results. The evaluator shall make all necessary effort to ensure that the confidentiality of evaluation results and of intermediate evaluation results be kept. When the evaluation plan requires that the executable program of the product be tested, the configuration under test and the environment for testing shall be precisely recorded. When an evaluation action requires that a document be inspected, the use of checklists is recommended. C.1.6.2 Apply decision criteria for measuressee clause 6.6.2

During the execution of the evaluation, intermediate evaluation results and final evaluation results are produced. In order to achieve maximal objectivity, evaluators staff different from the one that performed the action should check evaluation actions. C.1.6.3 Apply decision criteria for evaluationsee clause 6.6.3

All evaluation results shall be reviewed. The objective of the review depends on the nature of the evaluation action considered. The review shall be attended by at least one person not directly involved in the execution of the evaluation action concerned. The report of the review shall be included in the evaluation records.

C.1.7 Conclude the evaluation.


C.1.7.1 Review the evaluation resultssee clause 6.7.1

The draft evaluation report shall be delivered to the requester of the evaluation. A joint review between the requester and the evaluator should be organised. The requester should be given the opportunity to make comments on the evaluation report. If such comments are made, they should be included in a specific chapter of the evaluation report. Once the evaluation report has been formally delivered to the requester, the evaluator shall dispose of the data pertaining to the evaluation. This shall be done in the one of the following ways, depending on the type of data: the documents submitted to the evaluation shall be either returned to the requester or archived for a specified duration or destroyed in a secure way; the evaluation report and the evaluation records shall be archived for a specified duration; all other data shall be either archived for a specified duration or destroyed in a secure way.

When the specified archiving duration expires for some data, it shall be either archived again for a specified duration or destroyed in a secure way. Provided the requester explicitly agrees, intermediate evaluation results may be used by the evaluator in order to study evaluation techniques and software measures.

ISO/IEC 2002 All rights reserved

43

ISO/IEC FCD 25040

COMMITTEE DRAFT

C.1.7.2

Dispose evaluation datasee clause 6.7.2

Once the evaluation report has been formally delivered to the requester, the evaluator shall dispose of the data pertaining to the evaluation. This shall be done in the one of the following ways, depending on the type of data: the documents submitted to the evaluation shall be either returned to the requester or archived for a specified duration or destroyed in a secure way, the evaluation report and the evaluation records shall be archived for a specified duration, all other data shall be either archived for a specified duration or destroyed in a secure way.

When the specified archiving duration expires for some data, it shall be either archived again for a specified duration or destroyed in a secure way. Provided the requester explicitly agrees, intermediate evaluation results may be used by the evaluator in order to study evaluation techniques and software metrics.

44

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

Annex D (informative) Evaluation levels

As it seems difficult to obtain consensus on the generic specification of quality characteristics, including relation to subcharacteristics and to measures for the case to be evaluated, evaluation requirements may specify evaluation levels for the quality characteristics selected. Evaluation levels are to be related to software integrity levels as defined in ISO/IEC 15026 Information technology - System and software integrity levels. If a software integrity level has been assigned to a software product submitted for evaluation, this software integrity level may be used to select evaluation requirements. In particular, the degree of rigour associated to the software integrity level may be used as a guide to select evaluation techniques. Evaluation levels are related, on the one hand, to the importance attached by the requester to a given characteristic. The chosen level should be meaningful with regard to the assumed usage and environment of the software product (e.g. safety conditions, security constraints, economic risk, application constraints). On the other hand, an evaluation level defines the depth or thoroughness of the evaluation in terms of evaluation techniques to be applied and evaluation results to be achieved. As a consequence evaluation at different levels gives different level of confidence in the quality of the software product. The level can be chosen independently for each characteristic. This annex proposes four levels named A, B, C, and D. The levels constitute a hierarchy with A as the highest level and D as the lowest. At level A the most stringent evaluation techniques (taking into account reasonable amount of effort and time scale) are applied giving the highest confidence. Going down to level D gradually less stringent methods are used and consequently less effort is usually devoted to the evaluation. The evaluation level, for each software characteristic, may change for different components of a large product (for instance it is likely that critical components with high reliability requirements are kept separated from the other components of a system). The first section of this annex proposes guidance for selecting evaluation levels as function of the context of use of the product. The second one helps to select evaluation techniques.

D.1 Selection of evaluation levels


The evaluation levels may be selected independently for each of the relevant quality characteristics. When selecting the levels, several aspects should be considered. For example, important aspects are those related to safety, to economy, to security, to the environment and to the marketing of the product when appropriate. For a relevant quality characteristic, the risks and consequences involved by the non conformity of the product to requirements relating to this characteristic, as well as benefits from high quality, should be assessed for all the relevant aspects. For some of these aspects, the tables below provide the relationship between risks and levels to be selected. When several aspects need to be considered, the most stringent level should be selected. For the issues of economic risks and marketing benefits, the cost of the evaluation should be considered.

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040

COMMITTEE DRAFT

Safety aspects Evaluation level level D level C level B level A Consequences Small damage to property; no risk to people Damage to property; threat of injury to people Threat to human lives Many people killed

Economy aspects Evaluation level level D level C level B level A Consequences Negligible economic loss Significant economic loss (company affected) Large economic loss (company endangered) Financial disaster (company will not survive)

Security aspects Evaluation level level D level C level B level A Consequences No specific risk identified Protection against error risk Protection of critical data and services Protection of strategic data and services

46

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

Environment related aspects Evaluation level level D level C level B level A Consequences No environmental risk Local pollution Recoverable environmental damage Unrecoverable environmental damage

D.2 Selecting evaluation techniques from evaluation levels


In order to elaborate an evaluation specification to satisfy some evaluation requirements, it is necessary to specify measures. Measures are based on evaluation techniques that may be chosen according to quality characteristics and evaluation levels. In the following is proposed, for each quality characteristics, a list of evaluation techniques ranked from less demanding levels to more demanding levels. Functionality: functional or black box testing; inspection of development documentation guided by checklists; unit testing with test coverage criteria.

Reliability: verification of the use of specific programming language facilities; analysis of fault tolerance construct in the software design and code; reliability growth modeling.

Usability: user interface and documentation inspection; verification of the conformity to interface standards; performing usage experiments with real users.

Efficiency: execution time measurement; benchmark testing; analysis of the design to determine the algorithmic complexity.

Maintainability: inspection of development documentation guided by checklists;

ISO/IEC 2002 All rights reserved

47

ISO/IEC FCD 25040

COMMITTEE DRAFT

code measures and programming rules verification; analysis of traceability between elements of development documentation.

Portability: analysis of software installation procedures; programming rules verification; analysis of software design;

48

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

Annex E (informative) Evaluation methods


This Annex E provides examples of evaluation methods which may be used for software product quality evaluation.

E.1 Review of user and technical product documentation (including on-line documentation)
Product documentation may provide all the information necessary to make an evaluation of the functionality and usability requirements as well as other issues such as portability and maintainability. It may be possible to gain access to pertinent software product documentation without actually purchasing it, either by borrowing the documents or purchasing a documentation set. Although reviewing the software product documentation may not be as efficient as taking a course or training, it may prove to be the most economical, especially if the evaluator has the pertinent expertise.

E.2 Evaluation based on supplier courses and training


Product courses are offered for many software products, either through the supplier, or a third party. In the case of software products where no courses exist, it may be possible to arrange for special training from experienced users or developers of the software product. The advantage that product courses or training offer is allowing the evaluator to focus on specific areas and gain specific information on the functionality and usability of the software product in a short period of time. It may be possible to gain the same information through review of the software product documentation but that may prove more time consuming. The additional cost of a course or training needs to be weighed against efficiency in gaining information and the generality of the course material.

E.3 Assessment of software engineering process


The software engineering process assessment is a means of determining the quality of the software product by examining the interim products of the process; i.e. product quality plan, requirements specifications, architecture descriptions, detailed design descriptions, code listings, verification and validation records, code inspection and testing records, etc. To achieve this, it is necessary to define what constitutes an acceptable documentation baseline for software engineering processes which will provide adequate assurance in the quality of the resulting software product. An acceptable baseline can be defined by tailoring ISO/IEC 12207 requirements to the target integrity level in order to specify the required development and related support activities. This consists of determining: 1) 2) 3) required processes; required process output documentation; the requirements on process and process output documentation.

This assessment may be coupled with a supplier process capability level determination as defined in ISO/IEC 15504-2. ISO/IEC 12207 may be used to define process and expected outcomes to be applied according to the software product integrity level requirements.
NOTE Most of processes for software engineering process are described in Implementation processes, Software implementation processes and Software support processes of ISO/IEC 12207 Software life cycle processes.

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040

COMMITTEE DRAFT

For systems with high integrity requirements additional process and product requirements may be required by sector standards, e.g., IEC 880, IEC 1508, DOA-167A , MOD-55, etc. Then the supplier quality/development plan and associated methodology procedures can be used to assess supplier compliance to this target baseline. The level of compliance may then be determined by identifying the major deficiencies and assessing their potential impact on the quality of the software product. Additional evaluation or workarounds can address the impact of the deficiencies. It should be recognized that there are a variety of software engineering processes which are effective for particular organizations and different types of software products. The evaluation process should have the flexibility to accommodate a variety of reasonable software engineering processes and methods. It is recommended that the software engineering review be done in a staged manner. When it is deemed that the integrity level of software does not require a full evaluation of the software engineering process, the review may be halted after stage I or II.

E.4 Review of operating history with the supplier


A review of the operating history with the supplier can provide a very effective means of indicating the quality of the software product. This is achieved by reviewing the sales figures of the software product and details of the industries and the applications in which it is used. This review also addresses the history of revisions to the software, the way in which revisions are maintained, the way in which deficiency reports from customers are dealt with, and details of known deficiencies. The most convenient way to conduct the review is by interviewing the supplier's engineering, sales, and customer support staff and by examining any supporting records.

E.4.1 Operating history requirements


a) The sales figures should be at least six months old; i.e., the number of sales used in the evaluation will only include those sold over six months before the evaluation takes place. This criteria is based on the fact that it may take up to six months for the software product to be delivered, installed, commissioned, and placed in service; The software product should have gone through at least one major revision and there should be viable operating history data available on that revision. This is based on the assumption that the quality of the software product will depend on the amount of refinement it has gone through; There is a means for users of the software product to feed back deficiency reports to the supplier, and that there is evidence of this happening, and of resulting dispositions being implemented.

b)

c)

E.4.2 Operating history review


Review of product operating history should determine: a) Whether the software was produced by modifying another product and whether that product's operating history can be used. This would be contingent on the number of changes and the extent of the changes made to the software product; The number of unit-years of operation for the software product. This is calculated by the following steps: 1) Calculate the Sales Year = (initial sales [total sales in first year] + cumulative final total up to six months from present [assuming it typically takes 6 months delay before a unit is operational]) * (number of years the software product is in the market) / 2.

b)

50

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

2)

Calculate Unit-Years of operation = (Sales Year * duty cycle [percentage of the time the software is operational] * percentage of units actually in operation [this is pertinent, for example, to firmware in which a number of units of the host hardware may be kept as spares])

c) d) e) f) g) h) I) j) k) l)

If the operating experience provides evidence pertinent to the functionality required by the application. For example, is it used in similar applications by other customers? If the operating experience provides evidence that the breadth of the software product's functionality has been exercised. For example, has it been used in a wide variety of applications and industries? The number of unit-years of operation for each revision of the software and for each special option of the software product; The differences between the revisions, the extent of the changes, and whether the changes are isolated; Whether documented evidence exists to support the operating history data; How revisions of the software product and revisions of any related hardware are controlled and tracked; Whether it is possible to order a specific revision of the software product and the implications of ordering a revision which is not current; The supplier process for accepting and disposing of deficiency reports received from customers; Whether customers are kept abreast of reported deficiencies; Any outstanding deficiencies in the software and their impact.

E.5 Review of operating history with customers


A review of the operating history with actual customers who use or have used the software product in one of their applications allows the evaluator to get relatively unbiased answers to specific questions based on realistic operating conditions. Depending on how similar the customer applications are to the proposed application, the assurance gained in overall quality or specific functionality may be as useful as that obtained from prototyping or even extensive trial usage. The most convenient way to conduct the review is by interviewing the customers at their site of operations and possibly even viewing demonstrations or supporting records. The evaluator conducting an operating history review with customers should: a) b) c) d) Establish the degree of similarity of the application(s) to the proposed application; Attempt to view the software product in operation or obtain other supporting evidence; Question the customer(s) on the form and quality of support provided by the supplier; Determine the amount of error free operation.

E.6 Review of supplier capability, support, and quality system


Evaluation of the supplier's level of support capability and ability to maintain the software should address: a) financial stability, experience, and capabilities;

ISO/IEC 2002 All rights reserved

51

ISO/IEC FCD 25040

COMMITTEE DRAFT

b) c) d) e) f) g) h) i) j) k) l)

product development environmental support, including assembly tools, operating system used, and maintenance and use of other component/library objects; product interfaces to other products or product groupings, including interface standards; contingencies for third party involvement; sufficient resource commitment to product support; sufficient customer base to justify continued support; sufficient maintenance service for bug fixes, environmental support; adequate references from installed base; formalized and sufficient release and revision control procedures and evidence of practice; formalized regression testing and formal evaluation of design changes; documented and practiced problem reporting and resolution procedures; quality system in place;

m) standards used for hardware and software; n) o) plans for future development; i.e., strategy in place relative to current market position; product warranty.

E.7 Prototyping and other evaluation methods


E.7.1 Prototyping
Prototyping is a method of evaluation that can be used to resolve or fine tune requirements, to determine the technical feasibility of using the software product, or to eliminate unknowns or technical risks associated with specific functionality or usability requirements and their implementation. Prototyping may or may not utilize the full suite of functionality of the software product or address the entire set of requirements of the application. It should be noted that prototyping often requires the supplier to provide access to specialized equipment, personnel and documentation. Costs and schedule implications for these and other factors such as special environmental conditions or services should be considered in determining the feasibility of prototyping the software product. In addition to the general requirements for an evaluation, prototyping should: a) b) c) use examples which adequately address the requirements being assessed and provide realistic re-creation and simulation of key operating parameters; be adequately documented so that it may be repeated by a third party; make use of historical data pertinent to the proposed application where available.

52

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

E.7.2 Other evaluation methods


Evaluation techniques can be related to evaluation levels and software quality characteristics (see Annex D). The evaluation levels can correspond to integrity levels for the evaluation. Additional evaluation methods include: a) b) c) d) e) f) g) h) i) j) k) analysis of software architecture design (maintainability); fault tree analysis of software (reliability); statistical random usage based testing of software product (reliability); dynamic analysis of code to check syntax and semantics for correctness (reliability); hazards analysis of software design (reliability); review of software requirements specification (functionality); code inspection (functionality); black-box testing of software (functionality); benchmark testing (efficiency); analysis of requirements traceability (maintainability); simulated faults at the interfaces between components (reliability).

For complex software of high integrity, fault tree analysis or hazards analysis of software design can be used to isolate critical software modules for evaluation. This may eliminate the need to perform rigorous evaluation on software that has no impact on the integrity of the application.

ISO/IEC 2002 All rights reserved

53

COMMITTEE DRAFT

ISO/IEC FCD 25040

Annex F (informative) Example of Cost-Effectiveness Ranking of Evaluation Methods

This table presents a qualitative estimated ranking of the relative effectiveness and the gross relative cost of evaluation methods relative to specific product quality characteristics. This effectiveness rating assumes that the evaluation is conducted successfully and to the appropriate degree of rigor. This table can be used to select the target inputs that need to be evaluated in order to fully assess the adequacy of product quality characteristics against those specified in the software requirements specification. The evaluations required can be used to estimate total cost of product evaluation. Effectiveness* Ranking of Evaluation Methods per Evaluation Method
Final Product Documents, Courses and Training Intermediate Products Operating History Supplier Operating History Customer

Cost Ranking* Low


Functionality Reliability

Software Quality Characteristic

Usability

Efficiency

Maintainability

Portability

High

Low

High

Medium

Low

High

High Medium Medium

Low Medium High

Medium High Medium

Low Low High

Medium Low Medium

High Medium High

High Medium High

*NOTE Cost and effectiveness rankings shown are relative and dependent on the extent of the evaluation conducted.

ISO/IEC 2002 All rights reserved

54

COMMITTEE DRAFT

ISO/IEC FCD 25040

Annex G (informative) Example of application of the evaluation process

This annex should include a simplified example of the application of the evaluation process. At least it should present templates of artefacts that are generated during evaluation.

55

ISO/IEC 2002 All rights reserved

ISO/IEC FCD 25040

COMMITTEE DRAFT

Annex H (informative) Relationships between software product quality evaluation process reference model and software and system life cycle processes

Activities and tasks of evaluation process reference model can be performed associatively as a part of activities and tasks of Software Life Cycle Processes (ISO/IEC 12207) and/or System Life Cycle Processes (ISO/IEC 15288), when an evaluation is performed concurrently with software product development. Table F.1 Typical relationships between software product evaluation reference model and software and system life cycle processes (ISO/IEC 12207 and 15288) Activities and tasks of software Typical related processes in Examples of possible associative product evaluation reference ISO/IEC 12207 and/or 15288 work for product evaluation model in ISO/IEC 25040 - Purpose of evaluation can be 6.1.1 Acquisition Process 6.3 Establish evaluation aligned with project objectives. 6.2.2 Supply Process requirements - Evaluation plan can be developed 6.2.5 Quality Management 6.3.1 Establish the purpose of for negotiating agreements among Process evaluation acquirers and suppliers. 6.3.2 Obtain the software product (in ISO/IEC 12207 and 15288) - Evaluation plan can be coordinated 6.3.1 Project Planning Process quality requirements with or tailored from quality 6.3.3 Identify types of products to be 6.4.1 Stakeholder Requirements management process. Definition Process evaluated - Evaluation requirements, extent of 6.3.4 Define the stringency of the (in ISO/IEC 12207 and 15288) coverage and stringency of the 6.4.2 System Requirements evaluation evaluation can be obtained during Analysis Process requirements definition. 7.1.2 Software Requirements Analysis Process
(in ISO/IEC 12207)

6.4 Specify the evaluation


6.4.1 Select measures 6.4.2 Define decision criteria for measures 6.4.3 Define criteria for evaluation

6.3.7 Measurement Process


(in ISO/IEC 12207 and 15288)

6.5 Design the evaluation


6.5.1 Plan evaluation activities

6.4.5 Integration Process 6.4.6 Verification Process 6.4.8 Validation Process


(in ISO/IEC 15288)

6.6 Execute the evaluation 6.6.1 Make measurements


6.6.2 Apply decision criteria for measures 6.6.3 Apply decision criteria for evaluation

6.7 Conclude the evaluation


6.7.1 6.7.2 Review the evaluation result Create the evaluation report

7.1.6 Software Integration Process 7.2.4 Software Verification Process 7.2.5 Software Validation Process 7.2.6 Software Review Process
(in ISO/IEC 12207)

- Evaluation can be specified by selecting measures through measurement planning activity. - Plan of review, integration and testing for verification and validation can be used to design the evaluation of the software product quality. - Performance and results of review, integration and testing can be used to take measures, rate measured value and assess results of executing the evaluation. - Results of verification and/or validation can be used to prepare and create the evaluation report and/or it can be delivered to communicate the results to the measurement users.

56

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

Annex I (informative) Evaluation report template

This annex gives guidance on the structure and contents of the evaluation report. It summarises the reporting requirements stated in clause 6.3.6 of this standard. In addition, some ancillary information is required for inclusion in the report. The following sub-clauses describe the contents of the sections of an evaluation report.

I.1 Section 1 - Identifications


This section of the evaluation report contains identification information relative to the evaluation performed. Identification of the evaluator This sub-section contains the following information relative to the evaluator: name of the evaluators organisation, address of the evaluators organisation, location(s) where the evaluation has been carried out (if different from address above), name of the person responsible for the evaluation.

Identification of evaluation report This sub-section contains the identification of the report: unique identification of the report (e.g. serial number), number of pages in the report.

This information is copied on each page of the report. Each page is uniquely identified, for example by using a page number. Identification of requester and supplier This sub-section contains the following information relative to the requester of the evaluation and the supplier of the evaluated software product: name of requesters organisation, address of requesters organisation, name of software product supplier (if different from name above), address of software product supplier (if different from address above).

ISO/IEC 2002 All rights reserved

57

ISO/IEC FCD 25040

COMMITTEE DRAFT

I.2 Section 2 - Evaluation requirements


This section of the evaluation report contains the evaluation requirements as described in sub-clause 6.3.2 of this standard. In particular, it contains a general description of the product application domain, a general description of the product purpose,

the list of quality requirements and product information evaluated, possibly including reference to quality characteristics and evaluation levels.

I.3 Section 3 - Evaluation specification


This section of the evaluation report contains the evaluation specification as described in sub-clause 6.3.3 of this standard. In particular, it contains the scope of the evaluation, referring to the product description; when the product description is not a publicly available document it is annexed to the report; the cross-reference between the information requested in the evaluation requirements and the product components; the specification of measurements and verifications, the mapping between the specification of measurements and verifications and the evaluation requirements.

I.4 Section 4 - Evaluation methods


This section of the evaluation report shall contain the documentation of the evaluation methods used to perform the evaluation as specified in sub-clause 6.3.4.1 of this standard. When the evaluation method is documented elsewhere, it is permissible to include simply a reference to that documentation.
NOTE: Reference to the evaluation method will usually be used when it is a standard (evaluation module) or when it is proprietary.

For each evaluation method included here, the identification of product components on which the method has been applied is provided.

I.5 Section 5 - Evaluation Results


This section of the evaluation report contains the evaluation results as described in sub-clause 6.3.6.2 of this standard. In particular it contains the evaluation results themselves, intermediate results or interpretation decision, whenever necessary, reference to the tools used during the evaluation.

58

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

Annex J (Normative) Common terms and definitions

For the purposes of this document, the following terms and definitions given in ISO/IEC 25000 (repeated here for convenience) apply.
Note The definitions are common to all parts of SQuaRE series of International Standards.

J.1 acquirer
individual or organisation that acquires or procures a system, software product or software service from a supplier
Note Based on the definition in ISO/IEC 12207:1995.

J.2 analysis model


algorithm or calculation combining one or more base and/or derived measures with associated decision criteria

J.3 attribute
inherent property or characteristic of an entity that can be distinguished quantitatively or qualitatively by human or automated means
NOTE 1 based on ISO/IEC 15939:2002. NOTE 2 ISO 9000 distinguishes two types of attributes: a permanent characteristic existing inherently in something; and an assigned characteristic of a product, process or system (e.g. the price of a product, the owner of a product). The assigned characteristic is not an inherent quality characteristic of that product, process or system.

J.4 attribute for quality measure


Attribute that relates to software product itself, to the use of the software product or to its development process
NOTE Attributes for quality measure are used in order to obtain quality measure elements.

J.5 base measure


measure defined in terms of an attribute and the method for quantifying it
NOTE A base measure is functionally independent of other measures.

[ISO/IEC 15939: 2002, based on the definition in International Vocabulary of Basic and General Terms in Metrology, 1993].

ISO/IEC 2002 All rights reserved

59

ISO/IEC FCD 25040

COMMITTEE DRAFT

J.6 commercial-off-the-shelf software product


software product defined by a market-driven need, commercially available, and whose fitness for use has been demonstrated by a broad spectrum of commercial users

J.7 context of use


users, tasks, equipment (hardware, software and materials), and the physical and social environments in which a product is used [ISO 9241-11:1998]

J.8 custom software


software product developed for a specific application from a user requirements specification

J.9 data
collection of values assigned to base measures, derived measures and/or indicators [ISO/IEC 15939:2002]

J.10 decision criteria


thresholds, targets, or patterns used to determine the need for action or further investigation, or to describe the level of confidence in a given result. [ISO/IEC 15939:2002]

J.11 derived measure


measure that is defined as a function of two or more values of base measures [ISO/IEC 15939:2002, based on the definition in International Vocabulary of Basic and General Terms in Metrology, 1993].
NOTE A transformation of a base measure using a mathematical function can also be considered as a derived measure.

J.12 developer
individual or organisation that performs development activities (including requirements analysis, design, testing through acceptance) during the software life cycle process
Note Based on the definition in ISO/IEC 12207:1995

60

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

J.13 division of standards


division forms a family of standards serving complementary purposes

J.14 end user


Individual person who ultimately benefits from the outcomes of the system
NOTE The end user may be a regular operator of the software product or a casual user such as a member of the public.

6.4 entity
object that is to be characterised by measuring its attributes
EXAMPLE An object can be a process, product, project, or resource.

[ISO/IEC 15939:2002]

J.15 evaluation method


procedure describing actions to be performed by the evaluator in order to obtain results for the specified measurement applied to the specified product components or on the product as a whole

J.16 evaluation module


package of evaluation technology for measuring software quality characteristics, subcharacteristics or attributes
NOTE The package includes evaluation methods and techniques, input to be evaluated, data to be measured and collected and supporting procedures and tools.

J.17 evaluator
individual or organisation that performs an evaluation

J.18 external software quality


capability of a software product to enable the behaviour of a system to satisfy stated and implied needs when the system is used under specified conditions
NOTE Attributes of the behaviour can be verified and/or validated by executing the software product during testing and operation.

EXAMPLE The number of failures found during testing is an external software quality measure related to the number of faults present in the program. The two measures are not necessarily identical since testing may not find all faults, and a fault may give rise to apparently different failures in different circumstances.

ISO/IEC 2002 All rights reserved

61

ISO/IEC FCD 25040

COMMITTEE DRAFT

J.19 failure
termination of the ability of a product to perform a required function or its inability to perform within previously specified limits NOTE Based on the definition in IEEE 610.12-1990.

J.20 fault
incorrect step, process or data definition in a computer program [IEEE 610.12-1990]

J.21 functional requirement


requirement that specifies a function that a system or system component must be able to perform
[IEEE 610.12-1990]

NOTE The software quality characteristic functionality can be used to specify or evaluate the suitability, accuracy, interoperability, security and compliance of a function [ISO/IEC 25010].

J.22 implied needs


needs that may not have been stated but are actual needs
NOTE Some implied needs only become evident when the software product is used in particular conditions. EXAMPLE Implied needs include: needs not stated but implied by other stated needs and needs not stated because they are considered to be evident or obvious.

J.23 indicator
measure that provides an estimate or evaluation of specified attributes derived from a model with respect to defined information needs [ISO/IEC 15939:2002]
NOTE In ISO/IEC 14598 this definition was: "a measure that can be used to estimate or predict another measure".

J.24 information need


insight necessary to manage objectives, goals, risks, and problems [ISO/IEC 15939:2002]

62

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

J.25 information product


one or more indicators and their associated interpretations that address information need
EXAMPLE A comparison of a measured defect rate to planned defect rate along with an assessment of whether or not the difference indicates a problem.

[ISO/IEC 15939:2002]

J.26 information system needs


needs that can be specified as quality requirements by external measures and sometimes by internal measures

J.27 intermediate software product


product of the software development process that is used as input to another stage of the software development process
EXAMPLE Intermediate software products can include static and dynamic models, other documents and source code.

J.28 intermediate software product needs


needs that can be specified as quality requirements by internal measures

J.29 internal software quality


Capability of a set of static attributes of a software product to satisfy stated and implied needs when the software product is used under specified conditions.
NOTE 1 Static attributes include those that relate to the software architecture, structure and its components. NOTE 2 Static attributes can be verified by review, inspection and/or automated tools. EXAMPLE The number of lines of code, complexity measures and the number of faults found in a walk through are all internal software quality measures made on the product itself.

J.30 maintainer
individual or organisation that performs maintenance activities
Note Based on the definition in ISO/IEC 12207: 1995

J.31 measure (noun)


variable to which a value is assigned as the result of measurement

ISO/IEC 2002 All rights reserved

63

ISO/IEC FCD 25040

COMMITTEE DRAFT

NOTE The term measures is used to refer collectively to base measures, derived measures, and indicators.

[ISO/IEC 15939:2002]

J.32 measure (verb)


make a measurement [ISO/IEC 14598-1:1999]

J.33 measurement
set of operations having the object of determining a value of a measure [ISO/IEC 15939:2002, based on the definition in International Vocabulary of Basic and General Terms in Metrology, 1993] NOTE: Measurement can include assigning a qualitative category such as the language of a source program (ADA, C, COBOL, etc.).

J.34 measurement function


algorithm or calculation performed to combine two or more base measures [ISO/IEC 15939:2002]

J.35 measurement method


logical sequence of operations, described generically, used in quantifying an attribute with respect to a specified scale [ISO/IEC 15939:2002, based on the definition in International Vocabulary of Basic and General Terms in Metrology, 1993].

J.36 measurement procedure


set of operations, described specifically, used in the performance of a particular measurement according to a given method [ISO/IEC 15939:2002, based on the definition in International Vocabulary of Basic and General Terms in Metrology, 1993]

J.37 measurement process


process for establishing, planning, performing and evaluating software measurement within an overall project or organizational measurement structure

64

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

[ISO/IEC 15939:2002]

J.38 observation
instance of applying a measurement procedure to produce a value for a base measure [ISO/IEC 15939:2002]

J.39 operator
individual or organisation that operates the system
Note Based on the definition in ISO/IEC 12207:1995.

J.40 process
system of activities, which use resources to transform inputs into outputs [ISO 9000:2000]

J.41 quality in use (measure)


the extent to which a product used by specific users meets their needs to achieve specific goals with effectiveness, productivity, safety and satisfaction in specific contexts of use

J.42 Quality measure elements


Measure, which is either a base measure or a derived measure, that is used for constructing software quality measures
NOTE The software quality characteristic or subcharacteristic of the entity is derived afterwards by calculating a software quality measure.

J.43 quality model


defined set of characteristics, and of relationships between them, which provides a framework for specifying quality requirements and evaluating quality

J.44 rating
action of mapping the measured value to the appropriate rating level. Used to determine the rating level associated with the software product for a specific quality characteristic

ISO/IEC 2002 All rights reserved

65

ISO/IEC FCD 25040

COMMITTEE DRAFT

J.45 rating level


scale point on an ordinal scale, which is used to categorise a measurement scale
NOTE 1 The rating level enables software product to be classified (rated) in accordance with the stated or implied needs. NOTE 2 Appropriate rating levels may be associated with the different views of quality i.e. Users', Managers' or Developers'.

J.46 requirements
expression of a perceived need that something be accomplished or realized NOTE The requirements may be specified as part of a contract, or specified by the development organisation, as when a product is developed for unspecified users, such as consumer software, or the requirements may be more general, as when a user evaluates products for comparison and selection purpose.

J.47 scale
ordered set of values, continuous or discrete, or a set of categories to which the attribute is mapped [ISO/IEC 15939:2002, based on the definition in International Vocabulary of Basic and General Terms in Metrology, 1993]
EXAMPLE Types of scales are: a nominal scale which corresponds to a set of categories; an ordinal scale which corresponds to an ordered set of scale points; an interval scale which corresponds to an ordered scale with equidistant scale points; and a ratio scale which not only has equidistant scale point but also possesses an absolute zero. Measures using nominal or ordinal scales produce qualitative data, and measures using interval and ratio scales produce quantitative data.

J.48 software product


set of computer programs, procedures, and possibly associated documentation and data [ISO/IEC 12207:1995]
NOTE 1 Products include intermediate products, and products intended for users such as developers and maintainers. NOTE 2 In SQuaRE standards software quality has the same meaning as software product quality.

J.49 software product evaluation


technical operation that consists of producing an assessment of one or more characteristics of a software product according to a specified procedure

J.50 software quality


capability of software product to satisfy stated and implied needs when used under specified conditions

66

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

NOTE This definitions differs from the ISO 9000:2000 quality definition mainly because the software quality definition refers to the satisfaction of stated and implied needs, while the ISO 9000 quality definition refers to the satisfaction of requirements.

J.51 software quality characteristic


Category of software quality attributes that bears on software quality
NOTE Software quality characteristics may be refined into multiple levels of subcharacteristics and finally into software quality attributes.

J.52 software quality evaluation


systematic examination of the extent to which a software product is capable of satisfying stated and implied needs

J.53 software quality in use


capability of the software product to enable specific users to achieve specific goals with effectiveness, productivity, safety and satisfaction in specific contexts of use
NOTE Before the product is released, quality in use can be specified and measured in a test environment for the intended users, goals and contexts of use. Once in use, it can be measured for actual users, goals and contexts of use. The actual needs of users may not be the same as those anticipated in requirements, so actual quality in use may be different from quality in use measured earlier in a test environment.

J.54 Software quality measure


Measure of internal software quality, external software quality or software quality in use
NOTE Internal software quality, external software quality and software quality in use are described in the quality model in ISO/IEC 25010 [ISO/IEC 25010].

J.55 stakeholder
a party having a right, share or claim in a system or in its possession of characteristics that meet that partys needs and expectations [ISO/IEC 15288] NOTE Stakeholders include, but are not limited to, end users, end user organisations, supporters, developers, producers, trainers, maintainers, disposers, acquirers, supplier organisations and regulatory bodies

J.56 supplier
individual or organisation that enters into a contract with the acquirer for the supply of a system, software product or software service under the terms of the contract [ISO/IEC 12207:1995]

ISO/IEC 2002 All rights reserved

67

ISO/IEC FCD 25040

COMMITTEE DRAFT

J.57 system
a combination of interacting elements organised to achieve one or more stated purposes NOTE 1 A system may be considered as a product or as the services it provides. NOTE 2 In practice, the interpretation of its meaning is frequently clarified by the use of an associative noun, e.g. aircraft system. Alternatively the word system may be substituted simply by a context dependent synonym, e.g. aircraft, though this may then obscure a system principles perspective. [ISO/IEC 15288]

J.58 target of process


software product or task executed by software product to which measurement or evaluation process is applied

J.59 unit of measurement


particular quantity defined and adopted by convention, with which other quantities of the same kind are compared in order to express their magnitude relative to that quantity [ISO/IEC 15939:2002, based on the definition in International Vocabulary of Basic and General Terms in Metrology, 1993]

J.60 user
individual or organisation that uses the system to perform a specific function
NOTE Users may include operators, recipients of the results of the software, or developers or maintainers of software.

[ISO/IEC 15939:2002]

J.61 validation
confirmation, through the provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled
NOTE 1 "Validated" is used to designate the corresponding status.

[ISO 9000:2000]
NOTE 2 In design and development, validation concerns the process of examining a product to determine conformity with user needs. NOTE 3 Validation is normally performed on the final product under defined operating conditions. It may be necessary in earlier stages. NOTE 4 Multiple validations may be carried out if there are different intended uses.

68

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

J.62 value
number or category assigned to an attribute of an entity by making a measurement

J.63 verification
confirmation, through the provision of objective evidence, that specified requirements have been fulfilled
NOTE 1 "Verified" is used to designate the corresponding status.

[ISO 9000:2000] NOTE 2 In design and development, verification concerns the process of examining the result of a given activity to determine conformity with the stated requirement for that activity.

ISO/IEC 2002 All rights reserved

69

ISO/IEC FCD 25040

COMMITTEE DRAFT

Bibliography

ISO 9000:2000, Quality management systems Fundamentals and vocabulary ISO/IEC TR 9126-2:2003, Software engineering - Product quality - Part 2: External measures ISO/IEC TR 9126-3:2003, Software engineering - Product quality - Part 3: Internal measures ISO/IEC TR 9126-4:2004, Software engineering - Product quality - Part 4: Quality in Use ISO 9241-11:1998 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability ISO/IEC 12207:2008, Systems and software engineering Software life cycle processes ISO/IEC 13407:1999 Human-centred design processes for interactive systems ISO/IEC 14598-1:1999, Information technology Software product evaluation - Part 1: General overview ISO/IEC 14598-2:2000, Software engineering - Product evaluation - Part 2: Planning and management ISO/IEC 14598-3:2000, Software engineering - Product evaluation - Part 3: Process for developers ISO/IEC 14598-4:1999, Software engineering - Product evaluation - Part 4: Process for acquirers ISO/IEC 14598-5:1998, Information technology Software product evaluation - Part 5: Process for evaluators ISO/IEC 14598-6:2001, Software engineering - Product evaluation - Part 6: Documentation of evaluation modules ISO/IEC 15026 Information technology -- System and software integrity levels. ISO/IEC 15288, Systems and software engineering System Life Cycle Processes ISO/IEC 15504-2, Software Engineering - Process Assessment - Part 2: Performing an assessment ISO/IEC 15939:2007, System and software engineering Measurement process ISO/IEC TR 25021:2007 Software engineering - Software product Quality Requirements and Evaluation (SQuaRE) Quality measure elements ISO/IEC 25022 Software engineering - Software product Quality Requirements and Evaluation (SQuaRE) Measurement of internal quality (to be published) ISO/IEC 25023 Software engineering - Software product Quality Requirements and Evaluation (SQuaRE) Measurement of external quality (to be published) ISO/IEC 25024 Software engineering - Software product Quality Requirements and Evaluation (SQuaRE) Measurement of quality in use (to be published) ISO/IEC 25051:2006 - Software engineering: Software product Quality Requirements and Evaluation (SQuaRE) Requirements for quality of Commercial Off-The-Shelf (COTS) software product and instructions for testing.

70

ISO/IEC 2002 All rights reserved

COMMITTEE DRAFT

ISO/IEC FCD 25040

ISO/IEC 25062:2006 - Software engineering - Software product Quality Requirements and Evaluation (SQuaRE) -Common Industry Format (CIF) for usability test reports

ISO/IEC 2002 All rights reserved

71