Project IEC 62059-51

Software aspects of dependability

Prepared by
Győző Kmethy
Version 0.4 - 11th November 2002

2002.11.11 IEC 62059-51 outline v0.4 1

 Contents
• Current situation in metering
• Objectives of the 9-51 project
• Software quality and reliability
• Process quality and software quality - a review
– Process quality standards and tools
– New approaches to software development process
– Software quality assessment standards and tools
– Sources: IEC TC 56, ISO/IEC JTC 1 SC 7, IEEE etc.
• Legal metrology requirements
– Legal metrology documents: Welmec, PTB, SP, Nmi
– Case study: ESKOM project on electricity dispensers
• Outline of 9-51
2002.11.11 IEC 62059-51 outline v0.4 2
 Current situation in metering -1
• Current posture: type / acceptance testing to standards used to
check fitness for purpose of serial products
• IEC standards cover metrology, but not added functionality
(tariff- and load control, load profiles, etc.)
• Black-box testing is used, this may not be efficient if operation
principles are not known
• More functions, even metrology, implemented in SW
• Some legal metrology bodies and utilities started to impose
additional functional and software requirements
• EU directives: declaration of conformity may be based on QA
• MID essential requirements have implications on SW
• Legal metrology bodies derive SW essential and particular
requirements, concerned with uniform SW assessment
– protection, examination and conformity, documentation
2002.11.11 IEC 62059-51 outline v0.4 3
 Current situation in metering - 2
• Manufacturers are concerned with extra burden to
get type approval (documentation and time to market)
• Manufacturers are concerned with confidentiality
• Not all legal metrology bodies and utilities are
prepared for process and software assessment

• Lot of improvement in
– software development process and maturity,
– software life cycle support techniques and tools,
– software quality assessment tools.

2002.11.11 IEC 62059-51 outline v0.4 4

What is specific in electricity metering?
• Large quantities at low-low prices
• Few products to meet diverse needs of customers >>
– need customisation, parametrisation
• Generally more complex than other utility metering
– more functions, supply form network
• Unsupervised operation for long periods >>
– Can not afford field failures
• Embedded technology extensively used
• Type (HW and SW) under legal metrology control
• Hostile EMC and climatic environment, target to fraud
• Low life / environment risk, low to middle financial risk
2002.11.11 IEC 62059-51 outline v0.4 5
 Draft 9-51
• Audience: acquirers, suppliers, developers, legal metrology,
maintainers
• Objectives (to be refined)
– Provide a balanced user/manufacturer/legal metrology view on SW
aspects of dependability
– ensure that metering equipment containing SW meet well defined
quality and dependability requirements
– provide guidance to all stakeholders purchasing, specifying,
developing, approving and operating electricity metering equipment
on SW aspects of dependability
– support legal metrology approval on equal and comparable basis in
any country
– support cost effective process
• Possible consequence on TC 13: development of some
functional requirements may be necessary

2002.11.11 IEC 62059-51 outline v0.4 6

Not in the objectives of 9-51
• To specify
– what software development process to use
– what development methods and techniques to use
– what software process maturity level shall be complied with
• To overlap with legal metrology software requirement
documents
• To deal with functional safety - IEC 61508
– (should we?)

2002.11.11 IEC 62059-51 outline v0.4 7

 Hardware and software
Hardware Software (firmware)
• design method • design method
– well controlled process – often weak processes
– HW processes are not
applicable or not efficient
• testing method • testing method:
– well elaborated, – difficult to do
comprehensive comprehensive test
• failure mechanism • failure mechanism
– statistical – deterministic
– wear out – no wear out

Need careful balancing

2002.11.11 IEC 62059-51 outline v0.4 8

 Software quality and reliability
• Quality:
– The totality of features and
characteristics of a software
product that bear on its ability to
satisfy stated or implied needs
(ISO/IEC 9126: 2001)
• Constituents of quality
– Process quality
– Product quality
– Quality in use
– Usability:effectiveness, productivity,
safety, satisfaction
2002.11.11 IEC 62059-51 outline v0.4
• Reliability
– Probability that no fault in any
software element of the
system will be activated in a
given time interval in the
operation of the system under
given conditions (IEC
61713:2000)
– (instead of time, other natural
units may be used, e.g.
number of functions
activated)
9
Integrity and dependability
• Integrity: an inherent system
attribute for containing risk
• Dependability: product
quality characteristics which
affect system performance in
terms of reliability,
maintainability and
maintenance support

2002.11.11 IEC 62059-51 outline v0.4 10

SW process and quality evaluation
• Many SW process standards - towards consolidation
• Many SW process assessment standards / schemes
• New approach: “Agile” - iterative SW development
processes
• Many SW quality assessment methods / schemes
• WG 13 needs to review and select what is applicable
for electricity metering
• SW process and product assessment shall be based
on international standards where possible

2002.11.11 IEC 62059-51 outline v0.4 11

 Process vs. product approach
• Posture: a quality process is an
essential prerequisite for
producing quality software
• Objectives
– establish and document a
standardised and coherent
definition of SW development
activities
– provide a reference for
common terminology, definition
and vocabulary for SW
development
– establish clear expectations
between acquirer and supplier
• Posture: while mature process
is essential, the SW product
must also be measured
• Objectives
– define a SW quality model
through a coherent set of
quality characteristics
– provide reference for common
definition of SW product
measures
– establish a framework for
evaluation of SW products

“Agile” SW methods: focus on result, not process

2002.11.11 IEC 62059-51 outline v0.4 12
Process and product approach
Functionality

ility

Re
lia
b
rta

b
ilit
Po

y
ISO/IEC
9126

Ma

y
ilit
int

ab
ain

Us
ab
ilit
y
Efficiency

• Process approach is concerned with defining,

assessing and imporving te process
• Product approach is concerned with assessing the
end product

2002.11.11 IEC 62059-51 outline v0.4 13

Process and product standards
• Define processes / activities
necessary to produce quality
SW
• Manage and control process
• Standards
– ISO 9000-3:1991 > ISO
9001: 2001
– IEC 60300-3-6: 1996
– IEC 61713: 2000
– MIL STD 498
– ISO/IEC 12207: 1995/2001
– ISO/IEC 15504: 1998
• Process Assessment e.g.
– TickIT 5.0: 2000
– SEI CMM Level 1-5
2002.11.11 IEC 62059-51 outline v0.4
• Define general quality
models for SW products or
specific characteristics to be
considered while developing
or evaluating SW
• Some standards
– ISO/IEC 9126: 2001
– ISO/IEC 14598: 1999-2001
– ISO/IEC 14102: 1995
– ISO/IEC 15939: 2002
• SW assessment, e.g.
– SCOPE
– ITSEC
– IEEE Std. 830
14
Standardisation groups and
organisations
• MIL
• IEC TC56 Dependability
– WG 10: Software aspects
• ISO/IEC JTC /SC7: Information technology - software
engineering
– WG 9: Software integrity
• IEEE xxxx
• SEI: Software Engineering Institute
• SPI: Software Productivity Consortium
• And many more

2002.11.11 IEC 62059-51 outline v0.4 15

Status of IEC TC 56 work
• Concerned with dependability processes only
• IEC 60300-3-6 Ed. 2, 1997: Software aspects of dependability -
UNDER MAINTENANCE
• IEC 61713 Ed. 1, 2000: Guide to software dependability through
the software life cycle processes - PUBLISHED
• project IEC 61704: Guide to selection of software test methods
for reliability assessment - DELETED
• project IEC 61720: Guide to techniques and tools for achieving
confidence in software - DELETED
• TC 56 wants to initiate a liaison with TC 13 (Oslo meeting 2001)

2002.11.11 IEC 62059-51 outline v0.4 16

 Software development processes

2002.11.11 IEC 62059-51 outline v0.4 17

Compliance framework categories

• Standards and guidelines

• Process Improvement (PI) models and internal
appraisal methods
• Contractor Selection Vehicles
• Quality Awards
• Software engineering life cycle models
• System engineering models

2002.11.11 IEC 62059-51 outline v0.4 18

Evolution of life cycle standards (US)

2002.11.11 IEC 62059-51 outline v0.4 19

ISO 9000-3: 1991
• ISO 9000 (Ed 1.) was difficult to apply to software
• Contains guidelines for the application of ISO 9001 to
the development, supply and maintenance of SW
• Three main groups:
– General company and management requirements
– Project and maintenance phase requirements
– Supporting activities requirements
• Assessment by using the ISO 9001 process
• New ISO 9001 makes it obsolete
• Other developments: TickIT, SPICE

2002.11.11 IEC 62059-51 outline v0.4 20

ISO 9000-3 Software quality system
Framework processes Life-cycle processes Supporting processes
 Managing the  Reviewing contracts,  Configuration
organisation tenders and orders management
 Defining, maintaining  Defining requirements  Controlling documents
and improving the  Development planning and data
quality system  Quality planning  Controlling quality
 Conducting internal  Design and records
quality system audits implementation  Measurement
 Testing, verification and  Controlling rulées,
validation practices and
 Acceptance conventions
 Replication, delivery and  Controlling tools and
installation techniques
 Maintenance  Purchasing
 Controlling included
software products
 Training
All processes mapped to ISO 9001
2002.11.11 IEC 62059-51 outline v0.4 21
ISO 9001
• First published in 1987 - basis of quality assurance
• ISO 9001/9002/9003: basis for certification
• Latest release: December 2000
– ISO 9000: QMS - Fundamentals and vocabulary
– ISO 9001: QMS - Requirements
• based on a process model approach
• emphasis on achieving customer satisfaction
– ISO 9004: QMS - Guidelines for performance improvement
• emphasis on the needs of all interested parties
• Easier to apply for software

2002.11.11 IEC 62059-51 outline v0.4 22

Relationship between process standards
IEC 60300-1/ ISO 9000-4
Dependability programme management

IEC 60300-2
Dependability programme
elements and tasks

IEC 60300-3-6
Application guide on software aspects
of dependability

IEC 61713
Software dependability through
the software life cycle processes

ISO/IEC 12207
Software life cycle processes

2002.11.11 IEC 62059-51 outline v0.4 23

IEC 60300-3-6: 1996

• See separate summary

2002.11.11 IEC 62059-51 outline v0.4 24

IEC 61713: 2000 Software dependability
through the software life-cycle processes

• See separate summary

2002.11.11 IEC 62059-51 outline v0.4 25

 MIL STD-498: 19xx
• Software development and documentation
• Audience: acquirers, software product suppliers,
organisations operating and maintaining software
• Posture: must have a consistent framework, uniform
requirements and terminology, and established
activities and tasks for acquiring, developing, modifying
and documenting software
• Contents:
– Activities: tasks, documentation requirements, responsibilities
– Activities required to be tailored to each project
– “integral process”: configuration management, software product
evaluation, quality assurance, corrective action, joint technical
and management reviews
2002.11.11 IEC 62059-51 outline v0.4 26
MIL STD-498: Data item descriptions
• Each DID describes the purpose and contents of a
specific planning and engineering document
• Intended to be used as a checklist for information to
be defined and recorded in carrying out the related
activity, regardless of whether a deliverable is
required
• Appropriate set of DID-s must be selected for a
project
• Conversion to a commercial standard >> ISO/IEC
12207 (No more military standars after 1993!)

2002.11.11 IEC 62059-51 outline v0.4 27

ISO/IEC 12207: 1995/2002
Software lifecycle processes
• Audience:
– acquirers, suppliers, operators and maintainers of software
• Framework for SW life-cycle processes from concept through
retirement
• Recognises the roles of the acquirer and the supplier
• Intended for a two party, contract based environment
– (not applicable to COTS)
• Primary - Supporting - Organisational processes
• Does not imply any specific life cycle model
• Describes how to tailor the standard for a project
– Compliance means performance of selected processes, activities and
tasks
• No DID-s (Data Item Descriptions, i.e. docu requirements)
• ISO/IEC TR 15271: Guide to ISO/IEC 12207

2002.11.11 IEC 62059-51 outline v0.4 28

 ISO/IEC 12207 processes
Organisation
Management Infrastruct. Training Improvement

Primary
Acquisition Development Maintenance
Supply Operation

Supporting
Document. QA Validation
Config. mgmt Verification

Joint review Audit Problem res.

• Process: a set of interrelated activities, which transform inputs into outputs

• A process is divided into a set of activities, each activity is further divided into a set of tasks
• Each process has a defined purpose and outcome
2002.11.11 IEC 62059-51 outline v0.4 29
ISO/IEC 15288: System Life Cycle Processes
• To be published in Oct 2002
Guide expected in March 2003
• Based on IS 9001:2000 and ISO/IEC 12207:1995
• To be harmonised with updated ISO/IEC 12207,
ISO/IEC15504 and replacement of ISO 9000-3

2002.11.11 IEC 62059-51 outline v0.4 30

ISO/IEC 15288 processes
Acquisition and supply

Agreement process Enterprise environment mgmt.

Investment mgmt.
System Life Cycle Process mgmt.
Resource management
Enterprise process Quality mgmt.

Project planning, assessment, control

Decision making
Project process Risk mgmt.
Configuration mgmt.
Information mgmt.

Technical process Stakeholder requirements definition

Redquirements analysis
Architectural design
Implementation, Integration, Verification, Transition,
Systems analysis, Validation, Operation,
2002.11.11 Maintenance,
IEC 62059-51 outline v0.4Disposal 31
ENEL SQ/G/0002

• Guide for the preparation of software planning

documents
• Development plan
• Quality plan
• Configuration management
• Same items can be found in international
standards??

2002.11.11 IEC 62059-51 outline v0.4 32

ISO/IEC 14102: Guideline for evaluation
and selection of CASE tools
• CASE: Computer-Aided Software Engineering
• Audience: users/acquirers, suppliers, evaluatiors of
CASE tools
• Scope: establish processes and activities to evaluate
CASE tools and selecting the most appropriate ones
• Main steps:
– initiation process,
– structuring process, requirements against which to evaluate
– evaluation, to derive a profile of the tool
– selection process, to determine which is the most
appropriate

2002.11.11 IEC 62059-51 outline v0.4 33

 Software process assessment and
Improvement

2002.11.11 IEC 62059-51 outline v0.4 34

TickIT
• UK initiative to create a method for organisation, procedures
and rules for SW certification based on ISO 9001
• Objectives:
– interpret ISO 9001 for software
– ensure continuing conformity of certified suppliers
– develop capabilities for assessment
– no content relating to process improvement
• TickIT Guide 5.0: 2000
– Customer Guide
• issues relating to quality system certification from a customers’ view
• how customers can contribute to quality
– Supplier Guide: guidance to suppliers on the construction and certification
of quality systems
– Auditor Guide: guidance to auditors

2002.11.11 IEC 62059-51 outline v0.4 35

TickIT Customer Guide statements
• The Requirements Specification is the major interface between
customer and supplier. It provides both the baselines against
which validation and acceptance tests are carried out and [acts
as] the primary input to the supplier's design process. The
customer should define everything that the supplier needs to
know to produce a system, which is acceptable to the customer.

• Verification, validation and testing activities seek to ensure that

a product ....performs as expected, and is suitable for its
intended purpose.... The customer should [confirm that] quality
is being built into the product as well as ensuring that the right
product is built. ....The final product acceptance criteria [should
be] agreed by the customer and supplier as soon as possible.

2002.11.11 IEC 62059-51 outline v0.4 36

SPICE -
• Software Process Improvement and Capability
dEtermination
• Initiative to develop standard for software process
assessment - basis of ISO 15504: 1998 standard
• Posture:
– assessment leads to improvement
– capability determination used to determine risks to the
successful outcome of a contract, development or service
delivery

2002.11.11 IEC 62059-51 outline v0.4 37

ISO 15504:1998 - Software process assessment

• Audience: acquirers, suppliers, assessors

• Scope: process assessment, process improvement,
capability determination
• Covers acquisition, supply, development, operation,
maintenance, support

• Closely linked to ISO/IEC 12207

2002.11.11 IEC 62059-51 outline v0.4 38

SPICE - ISO/IEC 15504: 1998
Part 1 Part 9
Concepts and Introductory Guide Vocabulary

Part 7 Part 8 Part 6

Guide for use in Guide for use in Guide to
process improvement determining supplier qualification of
process capability assessors

Part 3 Part 4
Performing Guide to conducting
an assessment an assessment

Part 5 Part 2
An asssessment A reference model for Part 2 and 3 are
model and processes and normative
indicator guide process capability
2002.11.11 IEC 62059-51 outline v0.4 39
Bootstrap methodology for process assessment

• Follow up of SPICE - ISO/IEC 15504

• Process capability:
–
Level 0: Incomplete process
–
Level 1: Performed process
–
Level 2: Managed process
–
Level 3: Established process
–
Level 4: Predictable process
–
Level 5: Optimising process
• Available on licence

2002.11.11 IEC 62059-51 outline v0.4 40

ISO/IEC 15939:2001
• Title: Information technology - Software engineering -
Software measurement
• To be used by both acquirers and suppliers
• Defines a software measurement process applicable
to all software related engineering and management
disciplines
• Basis of PSM (Practical Software and Systems
Measurement) and CMMI
• To be further investigated

2002.11.11 IEC 62059-51 outline v0.4 41

 The SEI Software Capability Maturity
Model (SW-CMM)

Process
Optimising(5)
Optimising (5)
improvement
Process
measurement Manageed(4)
Manageed (4)

Process
Defined(3)
Defined (3)
definition
Basic
Management Repeatable(2)
Repeatable (2)
Control

Initial(1)
Initial (1)
Version 2 issued in 1997
Most companies are at level 1 or 2
2002.11.11 IEC 62059-51 outline v0.4 42
 SW-CMM characterisitics
1) Initial. The software process is characterized as ad hoc, and occasionally even
chaotic. Few processes are defined, and success depends on individual effort and
heroics.

2) Repeatable. Basic project management processes are established to track cost,

schedule, and functionality. The necessary process discipline is in place to
repeat earlier successes on projects with similar applications.

3) Defined. The software process for both management and engineering activities is
documented, standardized, and integrated into a standard software process for the
organization. All projects use an approved, tailored version of the organization's
standard software process for developing and maintaining software.

4) Managed. Detailed measures of the software process and product quality are
collected. Both the software process and products are quantitatively understood
and controlled.

5) Optimizing. Continuous process improvement is enabled by quantitative

feedback from the process and from piloting innovative ideas and technologies

2002.11.11 IEC 62059-51 outline v0.4 43

 SW-CMM Key Process Areas
DefectPrevention
Defect Prevention
Level 3 TechnologyChange
ChangeManagement
Management
Technology
Optimising ProcessChange
Process ChangeManagement
Management

Level 4 SoftwareQuality
Software QualityManagement
Management
Managed QuantitativeProcess
Quantitative ProcessManagement
Management

OrganisationProcess
Organisation ProcessFocus
Focus
OrganisationProcess
Organisation ProcessDefinition
Definition
IntegratedSoftware
Integrated SoftwareManagement
Management
Level 3 SoftwareProduct
ProductEngineering
Engineering
Software
Defined PeerReviews
Peer Reviews
TrainingProgramme
Training Programme
Inter-GroupCo-ordination
Inter-Group Co-ordination

SoftwareRequirements
Software RequirementsManagement
Management
SoftwareProject
Software ProjectPlanning
Planning
Level 2 SoftwareProject
Software ProjectTracking
Tracking
SoftwareQuality
Software QualityAssurance
Assurance
Repeatable SoftwareConfiguration
Software ConfigurationManagement
Management Necessary to evaluate and
SoftwareSubcontract
SubcontractManagement
Management
Software assess the processes
Level 1 NoKey
No KeyProcess
Process Areas
AreasatatLevel
Level11
Initial
2002.11.11 IEC 62059-51 outline v0.4 44
The SEI SA-CMM

• Software Acquisition Capability Maturity Model

• Emphasis on the needs of software acquirers
• 5 maturity levels

2002.11.11 IEC 62059-51 outline v0.4 45

AMI - quantitative approach to SPI
• Posture: combine analysis and benchmarking to
improve process (EU project)
• Assess
– define primary golas for measurement
• Analyse
– derive sub-goals and relevant metrics
– questionnaires
• Metricate
– implement measurement plan, process data
• Improve
– compare measurement data with goals
– reasses goals
2002.11.11 IEC 62059-51 outline v0.4 46
 Software life cycle models

2002.11.11 IEC 62059-51 outline v0.4 47

 Software lifecycle model
• Framework containing the processes, activities and tasks
involved in the development, operation and maintenance of a
software product spanning the life of the system

Risk
Requirements
Analysis

Design

Implementation

Integration
Test

Time

Waterfall model The V life cycle

2002.11.11 IEC 62059-51 outline v0.4 48

New approaches - Incremental and
iterative models

• eXtreme programming
• RUP
• Personal Software Process
• Team Software process
• Rapid prototyping
• UML

2002.11.11 IEC 62059-51 outline v0.4 49

Agile software development
• “We have come to value:
– Individuals and interactions over processes and tools
– Working software over comprehensive documentation
– Customer collaboration over contract negotiation
– Responding to change over following a plan.”

– That is, while there is value in the items on the right, we value the
items on the left more.”
from the Manifesto for Agile Software Development

• Move away from heavyweight to lightweight

2002.11.11 IEC 62059-51 outline v0.4 50

 12 practices of eXtreme Programming
• The planning game • Continuous integration
– “customer stories -cards” – several times a day, no “bing
– team: solutions bang”
– customer: scope and priorities • On-site customer
• Testing – continuous communication
– unit tests written before coding • Small releases
– customer writes acceptance test – small releases with business
• Pair programming value
– pairs write, know and review • 40-hour week
• Refactoring – overetime is symptom of problem
– incorporate the learning • Coding standards
• Simple design – simple, followed by all
– simples design that works-YAGNI • Metaphor
• Collective code ownership – like architecture, everybody
understands how it fits together
– any person can change the code -
and must fix if it breakes
2002.11.11 IEC 62059-51 outline v0.4 51
 RUP - the Rational Unified Process
• web enabled set of SW engineering processes
• RUP organizes
projects in terms of
disciplines and phases,
each consisting of one
or more iterations
• always customised to
project
• architecture centric
• risk and use case
driven
• matches the was SW
people like to work
2002.11.11 IEC 62059-51 outline v0.4 52
 Personal Software Process
• Methodology developed by SEI for small organisations
and projects - “light version of CMM”
• Posture: improving personal process discipline improves
team and project performance
• Adapts 12 of the 18 processess of CMM
• Personal maturity framework
– Baseline personal process: add basic measurements
– Personal planning process: add task/schedule planning
– Personal quality management: add review techniques
– Cycle personal process: iterative approach
• Complementary to organisational efforts (from CMM 2)

2002.11.11 IEC 62059-51 outline v0.4 53

The PSP framework

2002.11.11 IEC 62059-51 outline v0.4 54

 The Team Software Process
• Developed by SEI
• Objectives
• build self-directed teams
• to show managers how
to coach and motivate
temas
• To accelerate SW
process imporvement
• To provide improvement
guidance of high mturity
organisations

2002.11.11 IEC 62059-51 outline v0.4 55

 Software product evaluation

2002.11.11 IEC 62059-51 outline v0.4 56

Basics of software metrics
• Choose the right metric (just a few)
• Implement data collection
• Efficiency: relevance of results to costs
• Example
– Size
• line of code (LOC)
• number of function points (language and style independent)
– Time
• effort (hours)
• Schedule: measures completion of events (milestones, reviews, audits,
deliverables
– Defects
• By module
• By category

2002.11.11 IEC 62059-51 outline v0.4 57

 The six quality characteristics of software
Are the required functions
available in the software?
How easy is to
to transfer the How reliable is
software to Functionality the software ?
another

ity

Re
environment?
bil

lia
rta

bil
ity
Po

ISO/IEC
9126
Ma

y
int

ilit
How easy is to Is the software
ain

ab
modify the

Us
ab

easy to use?
ilit

software? Efficiency
y

How efficient is the software?

2002.11.11 IEC 62059-51 outline v0.4 58
ISO/IEC 9126-1: Software quality model
Suitability
Accuracy
Interoperability
Security
Functionality Maturity
Fault tolerance
Reliability Recoverability
Understandability
Learnability
Usability Operability
Attractiveness
Maintainability Analysability
Changeability
Stability
Efficiency Testability
Time behaviour
Resource utilisation
Portability
Adaptability
Installability
Co-existence
Which quality profiles for meters? Replaceability
2002.11.11 IEC 62059-51 outline v0.4 59
ISO/IEC 9126: Software quality characteristics and metrics
• Audience:
– software product suppliers / acquirers
– organisations performing software product evaluations
– software product users and maintainers
• Scope:
– provide a framework to specify and evaluate software product
quality objectively and quantitatively
• Part 2: External metrics
– metrics derived from the behaviour of the system during testing or
operation
• Part 3: Internal metrics
– metrics that can be applied during design and coding
• Part 4: Quality in use metrics
– metrics that can be applied to measure the users’ view of the
quality of the system
2002.11.11 IEC 62059-51 outline v0.4 60
SCOPE
• SCOPE: Software Assessment and CertificatiOn
Programme in Europe
• Based on ISO/IEC 9126
• SCOPEMark
– certification of quality of a SW product on a maturity scale
• SCOPE Maturity model
– a framework within which software product exist at differing
levels of maturity (as appropriate for the application)
• SCOPEmark:
– certification scheme to achieve confidence by users and to
improve quality by suppliers

2002.11.11 IEC 62059-51 outline v0.4 61

SCOPE evaluation vs. risk levels
Level D Level C Level B Level A
Small damage, Damage, few Threat to Many people
no risk to people disabled human lives killed
people
Functionality Functional Review (check Component Formal proof
testing (black lists) testing (glass
box) box)
Reliability Programming Fault tolerance Reliability Formal proof
language analysis growth model
facilities
Usability User interface Conformity to Laboratory User mental
inspection interface testing model
standards
Maintainability Inspection of Static analysis Analysis of Traceability
documents development evaluation
process
Efficiency Execution time Benchmark Algorithmic Performance
measurement testing complexity profiling
analysis
Portability Analysis of Conformity to Environment Program design
installation programming constraints evaluation
rules evaluation

2002.11.11
A possible profile for meters?
IEC 62059-51 outline v0.4 62
ISO/IEC 14598: Software products evaluation

• Part 1: General overview

• Part 2: Planning and management
• Part 3: Process for developers
• Part 4: Process for acquirers
• Part 5: Process for evaluators
• Part 6: Documentation of evaluation modules

• See separate presentation

2002.11.11 IEC 62059-51 outline v0.4 63

 ISO/IEC 15026: Systems and software integrity levels
• Audience: developers, acquirers, evaluators
• Abstract: The software integrity level, in the definition proposed by this
standard, denotes a range of values of a software property necessary to
maintain system risks within acceptable limits. In this context the risk can be
defined under different perspectives (e.g. safety, financial, security).
• The standard details a process starting with the system integrity level
determination, performed following a risk analysis approach. The same
integrity level is initially assigned to the software. Further analysis of the
system design may justify assigning a lower integrity level to the software.
• The standard also indicates some methods to increase the degree of
confidence in the software, such as techniques that minimize the introduction
of errors, verification and validation techniques to maximize the detection of
errors, evaluation of operating history.
• In addition this standard introduces the concept of an independent Integrity
Assurance Authority, responsible for issuing a certificate of compliance with
the integrity requirements.

2002.11.11 IEC 62059-51 outline v0.4 64

ISO/IEC 12219:1994 /NF Logiciel mark

• Software packages - Quality requirements and

testing
• Basis of NF logiciel mark (AFNOR - 1996)

2002.11.11 IEC 62059-51 outline v0.4 65

ITSEC
• Information Technology Security Evaluation Criteria
• Two aspects:
– confidence in the effectiveness of the scurity functionality
– confodence in the correctness of implementation
• Deal with:
– software product security specification
– software product security evaluation
• Developed product must meet security target

2002.11.11 IEC 62059-51 outline v0.4 66

Software re-engineering
Needs Existing
software • When a software product
has already been completely
Analysis developed without
application of any
Diagnosis development standard
and/or formal documentation
Definition of therapy and we need to re-structure
and re-document it.
Therapy
• (Some legal metrology
Intervention bodies’ preferred method)

Re-engineered
software

2002.11.11 IEC 62059-51 outline v0.4 67

 Legal metrology software
requirements

2002.11.11 IEC 62059-51 outline v0.4 68

Legal metrology on software

• MID essential requirements

• WELMEC Guide 7.1 and Guide 7.2
• PTB-A 50.7-1, -2, -3
• SP Method
• NMI

2002.11.11 IEC 62059-51 outline v0.4 69

WELMEC Guides

• Interpretation of MID essential requirements for

software
• Guide 7.1 (Issue 1, Oct 1999)
– Sotware Requirements on the Basis of the Measuring
Instruments Directive
• Guide 7.2 Issue 0.10 Aug 2001
– Software Requirements for Measuring Instruments Used for
Supply to the Customer by Mains
• See separate summary

2002.11.11 IEC 62059-51 outline v0.4 70

 WELMEC WG7 guide vs IEC 62059-51 “software”

• Metrological software • Software dependability / quality

requirements requirements
Input as requirements
• Focus on conformity • Focus on process & product
assessment assessment
Documentation is used for assessment

• Validation of metrolog. • Testing for reliability and quality

aspects
What... How to...

• Evaluation forms
• Checklists Possible overlap
2002.11.11 IEC 62059-51 outline v0.4 71
PTB-A 50.7
• Requirements on Electronic, Software-Controlled Measuring
Instruments and Add-On Units for Electricity, Gas, Water and
Heat
• Based on WELMEC Guide 7.1
• Covers complex meter function > complements MID
• Necessary for National Approval mark
• 50.7-1: Simple devices
• 50.7-2: Simple devices with data transmission over data
networks
• 50.7-3: Simple devices with software separation

• See separate summary

2002.11.11 IEC 62059-51 outline v0.4 72

SP Method 2081: 1998
• Watt-hour meters and additional units: Test of
software controlled functions
• Posture: faults can be discovered through
combination of black-box and glass-box testing
• Verification method: using check lists by function
• But: Re-engineering is justified only if the software
product has been developed without a development
standard and/or documentation

• See separate summary

2002.11.11 IEC 62059-51 outline v0.4 73

NMI

• Posture: black box testing is not possible in cases

where functionality is totally unknown
• Proposal: design for testabilty
• Split data capture and data processing functionalities
• Status of work to be further investigated

2002.11.11 IEC 62059-51 outline v0.4 74

 Work in South Africa on electricity dispensers

• Source documents:
– Report on the reliability of embedded software for use in
electricity dispensers (1993)
– Software reliability in high-intehrity systems (1995)
– The role of methodologies in imporving software integrity (1996)
– Improving software reliability through cross life-cycle testing
(1997)
– Procedure for producing software process assessment
documents (2000)

– See separate summary

2002.11.11 IEC 62059-51 outline v0.4 75

 Outline of project IEC 62059-51

2002.11.11 IEC 62059-51 outline v0.4 76

Underlying philosophy
• Quality: meet expressed customer needs across life cycle
• Application: low to medium risk, practically no failures tolerated
• Type 1 errors: responsibility of TC 13, customers
• Type 2, 3 errors: responsibility of manufacturer
• Avoid fault introduction: clear specs, good processes and tools
• Remove faults: good architecture, processes, and tools
• Investment in process QA shall be rewarded
• Declaration of conformity shall be based on self declaration
• Product certification should be third party
– for legal metrology certification, legal metrology body
• Certification testing shall be based on quantitative targets
• Certification and documentation reqs shall be tailored to risks

2002.11.11 IEC 62059-51 outline v0.4 77

Outline of IEC 62059-51
• Scope, objectives
• Review of software life cycle processes (12207)
• Stakeholder roles across the life cycle (60300-3-6, 61713)
• Process asssessment and improvement, process docs (15504)
• Review of life cycle models, methods and tools (literature)
• Review of SW product evaluation methods (9126, 14598)
• Quality profile for metering equipment (9126)
• Software evaluation modules (14598)
• Risk based and functional classification of metering equipment
(14598)
• Project documents (12207)
• Functional and evaluation requirements
• (Identification of functional specifications to be developed by WG
11/WG14 - based on COSEM where possible)
2002.11.11 IEC 62059-51 outline v0.4 78
 Selection of evaluation levels
Safety Ecomomy Fraud risk Security Environmental
Levels aspects aspects aspects aspects

Many people Financial ??? Protection of Unrecoverable

A killed disaster strategic data damage
and services
Threat to Large High risk Protection of Recoverable
B human lives economic loss critical data damage
and services
Damage to Significant Low risk Protection Local pollution
C property, few economic loss against error
people injured risk
Small damage Negligible No risk No specific No risk
D to property, no economic loss risk identified
risk to people

• Aspects are based on ISO/IEC 14598-6 (except fraud)

• Aspects shall be evaluated depending on application: Res, Comm, Ind, T&D
• Level shall be at least according to the higher level resulting from each aspect
2002.11.11 IEC 62059-51 outline v0.4 79
 Definition of levels (ISO/IEC 14598-6)
Level Technique When Description

Formal proof During all life No adequate techniques available

A cycles
Component During all life Test each component and integrated SW against
B testing cycles requirements.
White box Derive test data from program logic.
testing Perform test to required coverage (statement,
path, decision, condition, multiple condition)
Review After SW Visual inspection of SW components and
C Code development documentation
inspection Visual inspection of SW component. To be used
with checklists and static analysis of code
(control flow)
Functional Before Attempt to find discrepancies between the
D testing delivery of product and its specification (equivalence
Black-box product partitioning, boundary value, cause-effect
testing graphing..
Find circumstances when the SW does not
behave according to specifications)
2002.11.11 IEC 62059-51 outline v0.4 80
Meter functions and requirements

Metrology Requirement
Primary metrology: WG 11 (partly,
P, Q, S, cos fi algorithms missing)
Secondary metrology: -
U, I, phase angles, harmonics
Pulse inputs: physical, accuracy WG 11 (phy)
Processing
Integrated (energy) values COSEM
Demand values: current, max/min, cum COSEM
Load profiles COSEM, VDEW
Threshold monitoring COSEM

2002.11.11 IEC 62059-51 outline v0.4 81

Meter functions and requirements
Tarification/Scheduling Requirement
external control inputs -
ripple control WG 11
time switch/ scheduling WG 11, COSEM, VDEW
clock: accuracy, sync, setting WG 11, COSEM
Historical values COSEM
Man machine interface
Operation modes: calibration, -
parametrisation, normal, default
Display:
layout VDEW
data identification COSEM
resolution -
display sequences, scrolling VDEW
Keyboard VDEW
2002.11.11 IEC 62059-51 outline v0.4 82
Meter functions and requirements
Pulse/analogue outputs -
technology, electrical parameters
functions
Communication
protocols COSEM
priority of channels -
access control COSEM, Welmec, PTB
security/authentication COSEM, Welmec, PTB
Ancillary functions
Power supply
main power supply, power failure WG 11, COSEM
aux power supply WG 11, COSEM

2002.11.11 IEC 62059-51 outline v0.4 83

Meter functions and requirements

Ancillary functions cont’d

Status registers, logbook COSEM, VDEW
Protection:
- software WELMEC, PTB
- data WELMEC, PTB
- anti tamper WELMEC, PTB
- environment: operating range / WG 11 / -
outside of operating range
Internal diagnostics, memory checking, -
securing

2002.11.11 IEC 62059-51 outline v0.4 84

Proposal on defining functions

• Prepare comprehensive functional list, with existing

sources of requirements
• Check comprehensiveness of requirements
– need for specifying behavior outsode specified range?
• Remove eventual contradictions specifications
• Check which functions need to be standardised by
TC 13
• (Remember: COSEM is interface specification, not
functional specification)

2002.11.11 IEC 62059-51 outline v0.4 85