SMEs Quality Assurance Course 17-19 September 2001

SMEs Quality Assurance Course 17-19 September 2001
by
T. Sgobba
Quality Assurance
for
Space Projects
SMEs Quality Assurance Course 17-19 September 2001
These handouts are released for general
instruction purpose only. These handouts
are not meant to take
any precedence on applicable project
requirements, documents and definitions
SMEs Quality Assurance Course 17-19 September 2001
$1.4
1.2
1
.8
.6
.4
.2
0
‘96 ‘98 1980 ‘82 ‘84 ‘86 ‘88 ‘90 ‘92 ‘94 ‘00
Space insurance claims…lift-off
Billion
Source Space News
SMEs Quality Assurance Course 17-19 September 2001
2000 1960
Space systems failures causes
FAILURE SPACE: A Systems engineering look at 50 Space
System Failure-J.S.Newman NASA-Acta Astronautica Vol.48 No 5-12
FAILURE SPACE: A Systems engineering look at 50 Space
System Failure-J.S.Newman NASA-Acta Astronautica Vol.48 No 5-12
SMEs Quality Assurance Course 17-19 September 2001
Space systems failures causes (cont’d)

“…if there is any surprise in the synoptic results it is the prominence
of manufacturing, production and assembly…a strong cautionary note
to those seeking to marginally hold down production costs by cutting
quality assurance, inspection, audit…”*

* FAILURE SPACE: A Systems engineering look at 50 Space System Failure

J.S.Newman NASA-Acta Astronautica Vol.48 No 5-12, 2001
SMEs Quality Assurance Course 17-19 September 2001
Product Assurance disciplines
4
Quality Assurance
4
Reliability
4
Safety
4
Materials
4
EEE Components
4
Configuration Management
SMEs Quality Assurance Course 17-19 September 2001
Product Assurance definitions
4
Product Assurance:
A discipline devoted to the study, planning and implementation of
activities intended to assure that the design, controls, methods and
techniques in a project result in a satisfactory level of quality in a
product (ECSS-P-001A)
4
Mission Assurance:
Mission assurance provides confidence to the project and sponsors that
the mission will be successfully completed through its planned support
activities, standardised processes, and monitored project milestones.
Mission assurance brings a set of specialised disciplines to support the
project team environment (NASA)
SMEs Quality Assurance Course 17-19 September 2001
European Product Assurance Space Standards
4
OLD : ESA Procedures Standards and Specifications (PSS)
4
NEW: European Cooperation for Space Standardisation (ECSS)
PSS-01-10
Product Assurance
PSS-01-10
Product Assurance
PSS-01-20
Quality Assurance
PSS-01-20
Quality Assurance
PSS-01-30
Reliability Assurance
PSS-01-30
Reliability Assurance
PSS-01-40
System Safety
PSS-01-40
System Safety
PSS-01-60
EEE Components
PSS-01-60
EEE Components
PSS-01-70
Materials/Processes
PSS-01-70
Materials/Processes
PSS-01-21
S/W PA
PSS-01-21
S/W PA
PSS-01-11
Configuration Mgmt
PSS-01-11
Configuration Mgmt

ECSS-Q-80
S/W PA
ECSS-Q-80
S/W PA
ECSS-Q-70
Materials/ Processes
ECSS-Q-70
Materials/ Processes
ECSS-Q-60
EEE Components
ECSS-Q-60
EEE Components
ECSSS-Q-40
Safety
ECSSS-Q-40
Safety
ECSS-Q-30
Dependabilitry
ECSS-Q-30
Dependabilitry
ECSS-Q-20
Quality Assurance
ECSS-Q-20
Quality Assurance
ECSS-Q-00
Policy & Principles

ECSS-Q-00
Policy & Principles

SMEs Quality Assurance Course 17-19 September 2001
QA Course Outline
4
PART I : Concepts and Techniques

Introduction

Basic principles and values

Main elements/techniques

QA during major project phases
4
PART II: QA Standards

Standards evolution

ECSS-Q-20A
SMEs Quality Assurance Course 17-19 September 2001
QA Course Outline (cont’d)
4
PART III: Special Techniques

Statistical techniques
– Cost of Quality
– Alert Systems
4
PART IV: S/W Product Assurance
– Basic concepts

Space S/W standards

S/W Process Capability Assessment
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance
Part I : Concepts and Techniques
SMEs Quality Assurance Course 17-19 September 2001
Introduction
<<>>
SMEs Quality Assurance Course 17-19 September 2001
What is Quality?
4 In technical usage, quality can have two meanings:

(1) the characteristics of a product or service that bear on its
ability to satisfy stated or implied needs

(2) a product or service free of deficiencies.
4 It is also defined as:

Quality of design

Quality of conformance

Quality of performance

SMEs Quality Assurance Course 17-19 September 2001
Quality of Conformance

Traditional QA deals mainly with means and practices to achieve and
ensure achievement of:
QUALITY OF CONFORMANCE:
Quality of conformance relates to the fidelity with which the product
conforms to the design.
SMEs Quality Assurance Course 17-19 September 2001
Does Quality cost?

Quality of Conformance is profitable not costly

POOR QUALITY IS COSTLY!
Companies that measure their “poor quality”costs for the first time are
usually shocked at what they find

Reducing “poor quality ” costs has been compared

by J.M. Juran to mining gold (Gold in the Mine),

because it may translate directly into increased

profits and higher production capacity (Hidden Plant)
SMEs Quality Assurance Course 17-19 September 2001
What is a quality system?

Product quality depends on many variables. The processes, organisation, resources and procedures that manufacturers and suppliers use to control these variables to produce a product of
consistent quality which meets defined specifications is called a QUALITY SYSTEM*
* manufacturing environment case
P E R S O N N E L M A T E R I A L M E A N S D O C U M E N T A T I O N
Q U A L I T Y S Y S T E M
SMEs Quality Assurance Course 17-19 September 2001
The four QS controlled elements
4
PERSONNEL: to perform or supervise the production tasks
[Competence, training, identification, motivation, attributes]
4
MATERIAL: on which to apply production processes (component, sub-
assembly, etc.)[Type, condition, capability, quantity, identification]
4
MEANS: tools, equipment and facilities required to perform the task
[Type, condition, capability, identification, location, environment]
4
DOCUMENTATION: input/output (specifications, drawings, records, reports,
orders, etc.) [Content, control, condition, identification, distribution]
SMEs Quality Assurance Course 17-19 September 2001
Quality system as business strategy

There are companies, in particular SMEs, which enforce QA rules
only when required by the customer. Such companies miss the
strategic business value of the Quality System for:
'
competitiveness improvement
^
product improvement
·
organisational continuity/growth
´
personnel motivation
SMEs Quality Assurance Course 17-19 September 2001
Quality System as assurance technique
4
The ability not only to achieve Quality (meet customer requirements)
but to demonstrate that an effective system is in place and maintained
for that purpose
4
For the European Union (as for any organisation having regulatory
responsibilities for safety), the implementation of a Quality Assurance
system by a company is a pre-condition for valid certification
assessment of their products compliance to relevant norms
4
Safety and high reliability required by space systems has made QA
systems implementation mandatory during any project phase (design,
manufacturing and operations)
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance

- write what you do - perform verifications
- do as written
- prevent mistakes - prove verifications
- learn from mistakes
QUALITY ASSURANCE
METHOD
(to obtain)
OBJECTIVE
(to demonstrate)
Q U A L I T Y
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance (cont’d)

“ Quality assurance is the planned and systematic activities
implemented within the quality system and demonstrated as needed to
provide adequate confidence that an entity will fulfill requirements for
quality” (ISO-Vocabulary).
SMEs Quality Assurance Course 17-19 September 2001
Best
Practices
ISO 9000
Total Quality
Management
Evolution of quality system concepts
Major changes of focus:
- from external quality to internal quality
- from product/service improvement to process improvement
- from second-party certification to third-party certification
Artisan



Inspection
Quality
Control

Statistical
Quality
Control
Statistical
Process
Control
SMEs Quality Assurance Course 17-19 September 2001
From artisan to quality control
O
small workshop-proprietor knowing everything-customer in touch
-proprietor training workmen and checking the end product
(good product quality-optimum quality costs)
O
larger workshop-division of work-intermediate merchants-intermediate
supervisors-end product checked by supervisors
(poor external quality-high quality costs)
O
factories-larger quantities and use of machines-work “atomisation” and
quantity incentives- managers and intermediate supervisors-end product
checked by independent inspectors (good external quality-poor internal
quality-high quality costs)
SMEs Quality Assurance Course 17-19 September 2001
From artisan to quality control (cont’d)
O
evolving factories-long management chain-industrial engineering
functions-sales management-ubiquitous independent inspectors-
inspector supervisors-Inspection Department (good external quality-
good internal quality-high inspection costs)
O
modern factories-management driving quality-quality planning-
marketing functions-preventive quality approach-limited number of
inspectors-Quality Control Department (QC engineering etc.)
(good product quality-optimum quality costs)

“Quality control is operational techniques and activities that are used
to fulfill requirements for quality. It involves techniques that monitor
a process and eliminate causes of unsatisfactory performance at all
stages of the quality loop” (ISO-Vocabulary)
SMEs Quality Assurance Course 17-19 September 2001
Statistical quality control
O
Statistical Quality Control was initially the application of statistical
techniques and economical considerations to acceptance sampling
(pioneered by H.Dodge and H.Romig of Bell Telephone Labs)
O
The purpose was to avoid the cost of 100% lot inspection, while
lowering the customer’s risk to receive a defective product, and the
manufacturer’s risk to reject a good one
SMEs Quality Assurance Course 17-19 September 2001
Statistical process control
O
Statistical Process Control (SPC) is an analytical tool invented by
W.A.Shewhart of the Bell Telephone Labs to assess, monitor and
control manufacturing processes
O
SPC is applicable to areas in which a large number of parts are being
produced or a process is being repeated many times
Notes:
O
the term Statistical Quality Control includes nowadays statistical process control and statistical
acceptance sampling
O
SPC cannot be applied to software products because all copies of software are exactly identical
SMEs Quality Assurance Course 17-19 September 2001
Total quality management (TQM)
O
By the 1980s the idea was advancing that quality leadership could be
achieved only by applying the entire array of quality tools and
techniques to all functions and all levels of a company. This approach
was called Total Quality Management or TQM.(In Japan it is called
Company-wide Quality Control)
O
“Total quality management is the management approach of an
organization, centered on quality, based on the participation of all of its
members, and aiming at long-term success through customer
satisfaction and benefits to all members of the organization and to
society” (ISO-Vocabulary)
SMEs Quality Assurance Course 17-19 September 2001
TQM: process model for quality
4
A process is defined as a “set of interrelated or interacting activities
which transforms inputs into outputs”
C
U
S
T
O
M
E
R
R
e
q
u
i
r
e
m
e
n
t
s
C
U
S
T
O
M
E
R
S
a
t
i
s
f
a
c
t
i
o
n
Manage for
Quality
Provide
Resources
Measure
Analyze,
Improve
Develop Quality
Product or Sevice Input Output
Product
NASA-JSC JPD 5335.1C
SMEs Quality Assurance Course 17-19 September 2001
Total quality management (cont’d)
O
TQM definition is well expressed in practical terms by the “Malcom
Baldrige National Quality Award” criteria published by the U.S.
National Institute for Standards and Technology (NIST):
'
leadership
^
information/analysis
·
strategic quality planning
´
human resource development/management
I
management of process quality
I
quality and operational results
I
customer focus and satisfaction
SMEs Quality Assurance Course 17-19 September 2001
Best practices
O
Global competition encourages companies to seek out and emulate
best practices
O
Best practices refers to choosing working methods that has been
found to be the most effective and efficient
SMEs Quality Assurance Course 17-19 September 2001
What is ISO 9000?
O
ISO is the International Organisation for Standardisation, founded in
1946 to promote the development of international standards. ISO is
composed of member bodies from over 90 countries
O
In 1987, the ISO published the first issue of a series of five
international standards (ISO 9000, 9001, 9002, 9003 and 9004),
developed by ISO Technical Committee (TC) 176 on quality systems.
The ISO 9000 standards were intended to be advisory in nature and
for use in two-party contractual situations
O
Quality system registration (certification) involve the assessment and
periodic audit by a third-party (quality system registrar).
SMEs Quality Assurance Course 17-19 September 2001
Basic principles and values
<<>>
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance: basic principles and values
4
Integrity
4
Independence
4
Prevention/Improvement
4
Objective Evidence
4
Identification/Traceability
4
Segregation
SMEs Quality Assurance Course 17-19 September 2001
Integrity
4
Personal integrity:
– professional
– physical
4
Documentation integrity:
– technical
– physical
SMEs Quality Assurance Course 17-19 September 2001
Personal integrity: professional
4
Fraud and collusion related to quality are not uncommon.

Sometimes irregularity and frauds are not committed for personal
interest but to overcome what could seem a temporary contingency
harmful to the company.

Some basic QA practices (e.g. Certificate of Conformity signature,
quality records) are meant to discourage frauds and help their
investigation if the need arises.
SMEs Quality Assurance Course 17-19 September 2001
Personal integrity: professional (cont’d)
4
Exposure of the QA personnel itself to potentially dangerous
situations of collusion

“Within the daily range of activity there are many pressures to test the
integrity of the inspector…dangerous is the situation posed when
there is a potential rejection on a vendor. Here the person pleading
for a “break” is often an influential individual well supplied with
funds for entertainment or other forms of influence”

(J.M. Juran-Quality Control Handbook)
SMEs Quality Assurance Course 17-19 September 2001
Personal integrity: professional (cont’d)

Inspection/certification companies find sometimes appropriate to issue
a specific policy statement, as the following example:

“STATEMENT OF INTEGRITY

...For over a hundred years, we have been building our reputation as
the world leader in independent, reliable and high quality
verification, testing and certification services The cornerstone of this
reputation is integrity. Our integrity is our asset. It is our balance
sheet, it defines us to ourselves, our customers, our shareholders and
to the world…. Nothing, not client wishes, nor revenue growth, nor
profitability, nor instructions from above, is more important than
integrity.” (From the “Statement of Integrity” of SGS of Geneva )
SMEs Quality Assurance Course 17-19 September 2001
Personal integrity: professional (cont’d)
4
“Flinching”. This term describes the tendency of inspectors to falsify

the results of inspection of borderline product (excess of reading at

maximum of specified value and not beyond).
4
Incorrect use of QA stamps, as for example “lending” of personal
QA stamps
4
Willful errors or negligence. These are instances when it is to
personal advantage or convenience of the inspector to falsify the
results of an inspection
SMEs Quality Assurance Course 17-19 September 2001
Personal integrity: professional (cont’d)
4
The QA function must be vigilant, enforce the practical means and
exercise the authority to prevent the occurrence of questionable practices
”...The real remedy is by creating and maintaining an atmosphere of
respect for the facts as the ethical foundation of the QA function…”

(J.M. Juran - Quality Control Handbook)
SMEs Quality Assurance Course 17-19 September 2001
Personal integrity: physical
4
Some QA jobs rely on good physical conditions for their correct
accomplishment, in particular vision integrity.
Personnel performing nondestructive inspections (NDI) are subjected
to periodical eye exam as per norms (e.g., MIL-STD-410)
SMEs Quality Assurance Course 17-19 September 2001
Documentation integrity: technical & physical
4
Errors can occur due to use of:

unreleased draft documents

obsolete documents

defaced documents
SMEs Quality Assurance Course 17-19 September 2001
Documentation integrity: technical & physical
4
Documents must be released and modified through formal procedures
4
A “Documents Master List” must be continually updated/maintained and
made available to document users
4
Obsolete documents must be promptly removed from all point of use, or
clearly identified (e.g. red “OBSOLETE” ink stamp)
4
Document users must be forbidden to alter or deface released documents
(by adding personal notes, work instructions, etc.)
SMEs Quality Assurance Course 17-19 September 2001
Independence

“State or quality of being independent; freedom from the influence,
control, or determination of another or others”

(Webster’s Dictionary)

Placement of QA in the organisation should foster independence, but
independence must be first of all a professional attitude of QA
engineers and managers

Acceptance determination of products

shall not be influenced by cost and

schedule considerations!
SMEs Quality Assurance Course 17-19 September 2001
Identification/traceability
4 Materials
4 Personnel
4 Measurements
4 Design requirements/verifications
SMEs Quality Assurance Course 17-19 September 2001
Materials identification/traceability
4
For critical products the conformity certification is of little value without
traceability to the materials
4
It must be possible using the product identification (e.g., P/N or DWG/N +
S/N or Lot Nr ) to trace all the objective data which demonstrate conformity
to the design documentation
4
Traceability is achieved during manufacturing and storage by means of a
continuous chain of tagging, marking and identification
verifications

SMEs Quality Assurance Course 17-19 September 2001
Personnel traceability: stamps and signatures
4
Certain operations or decisions to be valid must be performed by
personnel specifically trained/certified and/or authorised:
/
inspectors
/
certified welders
/
project controllers
/
manufacturing engineers
/
MRB members
/
CCB members, etc
4
Traceability is ensured through issuance
and use of inspection and other stamps,
or signatures, which are formally controlled
SMEs Quality Assurance Course 17-19 September 2001
Measurements traceability
4
A measure is by definition the comparison to a reference standard
4
To ensure the accuracy of a measurement it must be possible to relate
it to national or international standards through an unbroken chain of
comparison
SMEs Quality Assurance Course 17-19 September 2001
Design requirements/verifications traceability
4
The paramount consideration in writing a design requirement is its
technical essence , verifiability and traceability
4
Design requirements of a product are linked hierarchically and
logically to system level requirements following the product-tree
structure. Three types of requirements exist:
– direct/requirement: a lower level requirement the satisfaction of which
totally satisfies the higher level requirement

partial/requirement: a lower level requirement that meets the system level
requirement but does not itself totally satisfy the higher level requirement

derived/requirement: a lower level requirement necessary at the lower
level to allow the subsystem or equipment to meet a higher level
requirement
SMEs Quality Assurance Course 17-19 September 2001
Design requirements/verifications traceability
(cont’d)
4
Design requirements identification (by progressive number) and
traceability is necessary to ensure design completeness/consistency

4
Design requirements to verifications (test, analysis, inspection, review
of design) traceability is essential to completely prove that customer
requirements are met
SMEs Quality Assurance Course 17-19 September 2001
Objective evidence
4
If it isn’t written down it does not exist
4
If it isn’t written down it didn’t happen
4
If it isn’t written down it is subjective
Objective evidence are data such as:
- laboratory analyses
- test reports
- shop-travelers
- NCR’s

Objective data must be archived for a pre-specified time
SMEs Quality Assurance Course 17-19 September 2001
Prevention

The continuous change of denominations for the quality systems and
the additions of new tools should not confuse about what was and still
is the prime concept of quality management: PREVENTION

v
Prevention of chronic troubles
v
Prevention of re-occurrence
v
Prevention activities and methods

“ Once the idea of such prevention is grasped, the label used to
describe it (quality control, quality improvement, prevention, etc) is
unimportant …” (Juran’s Quality Control Handbook)
SMEs Quality Assurance Course 17-19 September 2001
Prevention of chronic troubles
Sporadic spike
time
quality
improvement
C
o
s
t

o
f

p
o
o
r

q
u
a
l
i
t
y
new quality
control zone
old quality
control zone
{
SMEs Quality Assurance Course 17-19 September 2001
Steps of a quality improvement process

A quality improvement process includes the following steps:
4
assess current state and problems
4
define preferred state
4
identify barriers and root causes
4
develop improvement solutions
4
implement solutions
4
monitor results
SMEs Quality Assurance Course 17-19 September 2001
Segregation
4
Segregation is meant to prevent the inadvertent/unauthorised use of
materials, equipment and documents such as:
– incoming materials not yet inspected
– nonconforming materials
– materials undergoing re-lifing tests
– out-of-calibration measurement equipment
– scrapped materials (waiting disposal)
– obsolete design documents (waiting disposal)
4
Segregation areas will be secured and accessible only to authorised
personnel
SMEs Quality Assurance Course 17-19 September 2001
Main Elements/Techniques
<<>>
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance: main elements/techniques
4
Organisation
4
Manual, Procedures, Plans and Records
4
Inspections
4
Audits
4
Nonconformance Control
4
Training
4
Qualifications/Certifications
SMEs Quality Assurance Course 17-19 September 2001
Organisation

This is the story of four people:

EVERYBODY, SOMEBODY, ANYBODY and NOBODY
There was an important job to be done and EVERYBODY

was asked to do it.

EVERYBODY assumed that SOMEBODY would do it.
ANYBODY could have done it but NOBODY did it.
SOMEBODY got angry about that because it was EVERYBODY’s job.

EVERYBODY thought ANYBODY ought to do it, but NOBODY realised
EVERYBODY wouldn’t do it.
Finally, ANYBODY blamed EVERYBODY for not helping NOBODY and
SOMEBODY wisely concluded that NOBODY is the most helpful person in
the company.
SMEs Quality Assurance Course 17-19 September 2001
Organisation

Good organisation means:
4
clear responsibilities and authorities
4
well defined management system
4
adequate communication lines
4
quality function independence
SMEs Quality Assurance Course 17-19 September 2001
Organisation: clear responsibilities/authorities
4
defined company organisation structure
4
written job descriptions
4
no conflicting responsibilities
4
no overlap of duties
4
responsibilities commensurate to authority
4
adequate competence and abilities
4
adequate positions

SMEs Quality Assurance Course 17-19 September 2001
Organisation: defined management system
4
defined way of operation of each organisation unit
4
defined functional interfaces with other units
4
defined external interfaces
SMEs Quality Assurance Course 17-19 September 2001
Organisation: adequate communication lines
4
communication methods
4
information flow (internal external)
4
open attitude
SMEs Quality Assurance Course 17-19 September 2001
Quality function placement: example
P r o d u c t i o n M a n a g e r
I n s p e c t i o n Q C E n g i n e e r i n g Q u a l i t y A s s u r a n c e
Q u a l i t y M a n a g e r O t h e r M a n a g e r s
H i g h e r M a n a g e r
SMEs Quality Assurance Course 17-19 September 2001
Quality function placement: example
Q u a l i t y M a n a g e r
Q C / I n s p
P r o d u c t i o n M a n a g e r T e c h n i c a l M a n a g e r O t h e r M a n a g e r s
H i g h e r M a n a g e r
SMEs Quality Assurance Course 17-19 September 2001
Quality function placement: example
Q u a l i t y M a n a g e r
Q C / I n s p
P r o d u c t i o n M a n a g e r
P A / Q A
P r o j e c t M a n a g e r A
P A / Q A
P r o j e c t M a n a g e r B
H i g h e r M a n a g e r
SMEs Quality Assurance Course 17-19 September 2001
Quality function placement: example
I n s p
P r o d u c t i o n M a n a g e r
Q C / Q A
T e c h n i c a l M a n a g e r O t h e r M a n a g e r s
H i g h e r M a n a g e r
SMEs Quality Assurance Course 17-19 September 2001
ESA-Product Assurance and Safety function

.

.
ESA
Director General
TOS
Directorate
Directorates
Q
Department
Quality/
Depend/Safety Materials
Components
Departments
Project
Divisions
Project PA
Management
SMEs Quality Assurance Course 17-19 September 2001
NASA-Safety and Mission Assurance function
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance: main elements/techniques
4
Organisation
«
Manual, Procedures, Plans and Records
4
Inspections
4
Audits
4
Nonconformance Control
4
Training
4
Qualifications/Certifications
SMEs Quality Assurance Course 17-19 September 2001
Quality system documentation structure
Quality policies, standards, organisations,
responsibilities and documentation
Quality system key processes and activities
QM tailoring for a specific project
Detailed descriptions of specific tasks
Objective evidence of conformance
to a quality system requirements
SMEs Quality Assurance Course 17-19 September 2001
Quality manual

A Quality Manual is a compilation of company policies and requirements meant to
guide day-to-day practices, in particular to:
- make customer and employee aware of the company’s

quality philosophy and standards
- establish responsibilities of each organisational unit
- provide written guidance and instructions for quality
- serve as the basis from which audits are performed


SMEs Quality Assurance Course 17-19 September 2001
Quality manual (cont’d)

The Quality Manual is prefaced by a quality policy statement of the
Company’s chief executive
The company operating policy is described in several functional areas,
such as subcontracting, process control, procurement control, inspections,
tool and gage control, training, packaging/shipping etc. It will also
describe in general terms how decisions will be made regarding
conformance and product suitability

The Quality Manual is divided into sections based either on functional or
organisational areas
SMEs Quality Assurance Course 17-19 September 2001
Quality procedures: the six honest serving men

The beginning of Rudyard Kipling’s poem “The Elephant Child” goes
as follows:

“ I keep six honest serving men

(they taught me all I knew).

Their names are What and Why and When

and How and Where and Who”

These six words_ what, why, when, how, where and who_ are the six
points that every procedure should address to provide a clear and
focused content.
SMEs Quality Assurance Course 17-19 September 2001
Quality procedures: short and concise

Procedures should be as short and concise as possible (i.e., no more
than five pages). Inclusion of flow-diagrams is always encouraged to
summarise the sequence of steps.

SMEs Quality Assurance Course 17-19 September 2001
QA plans

In general, a plan is a method or strategy for achieving an objective.

The QA plan is the contractor’s management strategy for fulfilling the
specific quality requirements of a contract
ISO 8402:
“ Document setting out the specific quality practices, resources, and
sequence of activities relevant to a particular product, project or
contract”

- The quality plan usually references parts of the supplier’s quality
manual applicable to the specific case

- The quality plan may be called “quality program plan” or “quality
assurance plan” depending on the scope
SMEs Quality Assurance Course 17-19 September 2001
QA plans (cont’d)

To be meaningful, the QA Plan must build on a pre-established
company quality management system. The reason being that project
schedule is usually incompatible with the time required to start a
quality system from “zero”.
P r o j e c t " A "
C o n t r a c t
P r o j e c t " A "
Q A P l a n
Q u a l i t y M a n u a l
SMEs Quality Assurance Course 17-19 September 2001
QA plans (cont’d)

The basic QA plan content is as follows:
4
organisation: charts, functional roles and responsibilities
4
compliance matrix to customer requirement cross-
referencing to in-house applicable procedures
4
requirements made applicable to sub-contractors
4
project-wide procedures
4
approach to surveillance of subcontractors
4
chart identifying relationship between QA tasks and
project milestones
SMEs Quality Assurance Course 17-19 September 2001
Work instructions

In some cases, it is necessary to provide step-by-step instructions to
the work force for specific activities. It is beneficial to separate this
type of instruction from the more generalised procedures.
SMEs Quality Assurance Course 17-19 September 2001
Quality records
4
Quality records are documents which are maintained to provide
objective evidence of activities performed or results achieved
demonstrating:

conformance to specified requirements

effective operation of the quality system

A freshly minted test procedure is not a quality record, but it becomes
a quality record when filled in with the results of the test.
A minimum retention period must be defined and documented.
Periods of 5 to 7 years are commonly used.
SMEs Quality Assurance Course 17-19 September 2001
Quality records (cont’d)
4
Examples of Quality Records:
– Inspection and test records
– Shop-travelers (after completion)
– Nonconformance records
– Calibration records
– Audit plan
– Audit results/reports
– Training/certification records
– Identification and traceability records
– Special processes records
– Preventive actions records
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance: main elements/techniques
4
Organisation
4
Manual, Procedures, Plans and Records
«
Inspections
4
Audits
4
Nonconformance Control
4
Training
4
Qualifications/Certifications
SMEs Quality Assurance Course 17-19 September 2001
Inspection (hardware products)
What is an Inspection?

Quality Characteristics

Workmanship standards

Inspection planning

Information for Inspection

Inspection types/techniques

Mandatory Inspection Point (MIP)

Degree of inspection

Metrology and Calibration
SMEs Quality Assurance Course 17-19 September 2001
What is an Inspection?

A basic QA function is that of deciding through inspections
whether the product is conforming. Such inspections are generally
called “acceptance” inspections or QA inspections and consist of:

a) Interpretation of the applicable technical documents

b) Measurement/assessment of the product

c) Comparison of a) and b)
d) Judgement as to conformance (a/r criteria)
e) Disposition of the product
f) Recording of data
SMEs Quality Assurance Course 17-19 September 2001
Quality characteristics

A physical or chemical property, a dimension, a temperature, a
pressure, or any other requirement used to define the nature of a
product or a service is a quality characteristics
SMEs Quality Assurance Course 17-19 September 2001
Classification of quality characteristics
4
Design (Functional) Characteristics

ensure performance

ensure life/reliability

ensure safety

ensure interchangeability

ensure interfaces
4
Manufacturing (Non-Functional) Characteristics

related to method of manufacturing

to facilitate manufacturing

to provide information for toolmaking
SMEs Quality Assurance Course 17-19 September 2001
Classification of quality characteristics (cont’d)

The great number and variety of quality characteristics makes often
necessary to formally highlight their difference in importance so that
right care is taken by all concerned. In particular for inspection:
- need for independent inspection
- need to record inspection results
- responsibility for acceptance of defects

Example of classification codes for dimensional characteristics:
Critical
Major
Minor
Other Not marked
SMEs Quality Assurance Course 17-19 September 2001
Workmanship standards

“Something effected, made, or produced; the art or skill of a
workman; the quality imparted to a thing in the process of making”

(Webster’s Dictionary)

Workmanship standards establish the standard for the appearance of a
product (indicative of proper quality) and the basic techniques used in
producing it.

ESA and NASA have produced workmanship standards in particular
for electronics assembly techniques. A wider range of workmanship
standards has been established by national and international
standardisation organisations (ISO, DIN etc.).
SMEs Quality Assurance Course 17-19 September 2001
Workmanship standard (cont’d)
Handsoldering of high
reliability electrical
connections
ECSS-Q-70-08
[PSS-01-708]
NASA-STD-8739.3
[NHB5300.4(3A-2)]
NASA-STD-8739.4
[NHB5300.4(3G)]

NASA-STD-8739.4
[NHB5300.4(3G)]
NASA
---
---
PSS-01-738
PSS-01-726
ECSS-Q-70-30
[PSS-01-730]
ESA
Staking and conformal
coating of PCBs
Fiber optic
terminations, cable
assemblies/ install.
High reliability
soldering for SMT
and mixed technology
Crimping of high
reliability electrical
connections
Wire wrapping of
high reliability
electrical connections
NASA-STD-8739.2
[NHB5300.4(3M)]
NASA-STD-8739.5
NASA-STD-8739.1
[NHB5300.4(3J-1)]
SMEs Quality Assurance Course 17-19 September 2001
Workmanship standards : examples
SMEs Quality Assurance Course 17-19 September 2001
Workmanship standards : examples
SMEs Quality Assurance Course 17-19 September 2001
Inspection planning

The plan for conducting the overall inspection points on a project is
called Inspection Plan. Inspection points are usually identified
through:
4
contract requirements
4
specifications/standards
4
reliability and safety analyses (e.g., FMECA/CIL, PHA)
4
quality concerns (e.g., interfaces)
4
manufacturing constraints
4
operational experience

Usually inspection and manufacturing plans are fully integrated.
SMEs Quality Assurance Course 17-19 September 2001
Information for inspection
4
Shop-travelers
4
Drawings/engineering lists
4
Workmanship standards
4
Procurement specifications
4
Purchase orders/sample plan
4
Configuration Item Data List (CIDL)
4
Delivery documentation
SMEs Quality Assurance Course 17-19 September 2001
Inspection types/techniques
4
Incoming inspection
4
In-process inspection
4
Self-inspection
4
Final inspection
4
Source inspection
4
Non-destructive inspection
SMEs Quality Assurance Course 17-19 September 2001
Incoming inspection

Inspections performed to assure the conformity of received items to
purchase orders. Mandatory use of holding areas until acceptance.

Purchase Orders
- identify deliverable items and quality documents/records
- must be readily available at Incoming Inspection location
Incoming Inspection Instructions
- establish for group of items the verifications to be performed

Rejected materials
- usually processed for return to supplier
- sometimes processed via in-house MRB (impact on costs)
SMEs Quality Assurance Course 17-19 September 2001
Incoming inspection (cont’d)

Use of suppliers data

Normally comprehensive inspections/testing must be performed to
verify conformity of received items to relevant specifications. Such
activities may be considered redundant ( and therefore skipped) if
relevant data are requested to the vendor, and the vendor’s Quality
Assurance system has been audited/certified directly or by a reliable
third party (e.g., ISO).The bottom line on use of suppliers data is to be
cost effective without compromising the overall control of end-
product conformance

Shipping damages
Incoming items must be always examined for shipping damages
SMEs Quality Assurance Course 17-19 September 2001
Incoming inspection (cont’d)

Sometimes specific incoming inspections are anyhow performed to:
4
prevent potentially severe schedule and cost impacts on
the project due to late discovery
4
comply with applicable standards (e.g., procurement
of EEE)
4
prevent warranty time expiration for costly item not
of immediate use


SMEs Quality Assurance Course 17-19 September 2001
In-process inspection

In-process inspection serve to provide early detection of
nonconformances before items are completed or processed into
higher assemblies
SMEs Quality Assurance Course 17-19 September 2001
Self-inspection

Consist of allowing performance of in-process inspection by the same
operator who has performed the manufacturing operation. SI requires:
4
written quality procedure
4
trained/certified operators
4
written specific instructions and a/r criteria
4
quantitative record of results
4
operator traceability
4
sampling re-verifications by QA

Self-inspection of critical characteristics is allowed only for operations
which will be directly or indirectly verified by QA at a later time.
SMEs Quality Assurance Course 17-19 September 2001
Final inspection

Consist of inspecting the product and the manufacturing
documentation at the time of production completion to verify:
4
conformity to contract requirements
(including as-designed/ as built)
4
completion of all manufacturing activities
( including inspection and testing)
4
close-out of any previous NCR disposition
4
correct marking, including serialisation (if applicable)
4
complete/correct accompanying documents


SMEs Quality Assurance Course 17-19 September 2001
Source inspection
4
Inspection performed by the customer’s QA to confirm product’s
compliance to contractual requirements prior to shipment
4
Source inspection may also involve Engineering in a coordinated
fashion for issues related to technical performance
4
Acceptance at source: inspection and acceptance at supplier’s facility
prior to shipment to the extent that no further inspection is required
4
Source inspection is recommended for critical items when no
incoming inspection is feasible at customer’s premises (e.g., due to
lack of special equipment)
SMEs Quality Assurance Course 17-19 September 2001
Non-destructive inspection

These are techniques used to inspect products for internal defects

(porosity, impurities, inclusions, cracks, debonding etc.). These

techniques require considerable operator skill (certification) in
particular with for the interpretation of results


The most commonly used techniques are:
4 Liquid penetrants
4 Ultrasonic inspection
4 Radiography
SMEs Quality Assurance Course 17-19 September 2001
Non-destructive inspection: liquid penetrants

Technique based on property of certain liquids with very low surface
tension to penetrate cracks by capillary action. The penetrant can seep
into cracks as small as 0.1 micron. Two types used:
- fluorescent penetrant, visible under ultraviolet light
- visible penetrants, using dyes usually red in color
LP process steps:

1) Part cleaned and dried, 2) LP brushed or sprayed, 3) Excess
penetrant wiped off or washed away with water or solvents, 4)
developing agent applied LP will seep back to surface and spread to
the edges thus identifying location and size of defect
SMEs Quality Assurance Course 17-19 September 2001
Non-destructive inspection: ultrasonic inspection

Ultrasonic beam travels through the part to be inspected . An internal
defect interrupts the beam and reflects back a portion of the ultrasonic
energy.
Ultrasonic waves generated by transducer that operate on the
piezoelectrical principle using materials such quartz (usually 1-25
MHz)

Couplants ( water, oil) are used to transmit the ultrasonic waves from
the transducer to the test piece
SMEs Quality Assurance Course 17-19 September 2001
Non-destructive inspection: radiography

Method used to detect internal flaws, such as cracks and porosity
The principle involved is density differences. The metal surrounding a
defect will appear denser and hence show up as lighter than the flaws
on an x-ray film.

Radiation source an x-ray tube, visible permanent image made on

x-ray film.
SMEs Quality Assurance Course 17-19 September 2001
Mandatory Inspection Point (MIP)
4
Mandatory Inspection Point (MIP):
Activity performed by representatives of customer, prime contractor
or their delegates, to verify that the operations of fabrication and
control have been performed correctly
4
MIPs are performed at important stages of the fabrication process, in
particular at points where following operations would make
impossible the conformity assessment of previous operations
4
MIPs should be used not only as part of the product acceptance
process, but also to spot-check certain relevant elements of the
quality system
SMEs Quality Assurance Course 17-19 September 2001
MIP (cont’d)
4
A MIP may take the form of:

Inspection : direct performance of an inspection (by customer)


Witness : watch a specific operation


Results verification : review of records
SMEs Quality Assurance Course 17-19 September 2001
MIP (cont’d)
4
Severity of identified defects should also be assessed:

minor

impact on following fabrication operations

impact on item qualification/acceptance

impact on system integration

mission failure

safety
4
The identification of potential catastrophic defects undetectable until
their occurrence in flight would require immediate high profile actions
4
An inspection report should be issued for each MIP
SMEs Quality Assurance Course 17-19 September 2001
MIP (cont’d)

Average results from a major space contractor: (acceptance up to 90%
for experienced subcontractors; as low as 20% for inexperienced subs)
accepted
observat.
rework
NCR
reject
SMEs Quality Assurance Course 17-19 September 2001
Degree of Inspection
4
Redundant
4
100% Inspection
4
Statistical sampling/sampling
4
1st/Last piece
4
Audit
SMEs Quality Assurance Course 17-19 September 2001
Metrology and calibration

A vast variety of gages and instruments are used in industry for
measurement purpose (e.g. mechanical, electronic, optical,
pneumatic).
The QA function is concerned with appropriateness, use, maintenance
of such instruments (but not with their scientific development).
Calibration

The set of operations which establish, under specified conditions, the
relationship between values indicated by a measuring instrument and
the corresponding standard or known values derived from the
standard.
SMEs Quality Assurance Course 17-19 September 2001
Accuracy of measurement

Target analogy
4
Precise but not accurate
4
Accurate but not precise
4
Accurate and Precise
SMEs Quality Assurance Course 17-19 September 2001
Accuracy, precision and resolution
4
Accuracy
The degree of agreement of the measurement with the true value of that
parameter. The difference between measured and true value is defined
as error.
4
Precision
The ability of an instrument to repeat he same reading when making the
same measurement in the same manner and under identical conditions.
4
Resolution
The smallest change in input necessary to produce the smallest
detectable change in output of the instrument under test.
SMEs Quality Assurance Course 17-19 September 2001

Maintaining accuracy
From the moment an instrument is put into use it begins to deteriorate in
accuracy. The procedure for maintaining accuracy includes:
Number each instrument
 Establish a “card” record for each instrument
 Establish a checking interval
 Provide for adherence to the checking schedule
 Keep record of the findings of the check
 Analyse the results to take corrective actions (if needed)
 Analyse the trends to modify the intervals (longer/shorter)

SMEs Quality Assurance Course 17-19 September 2001
Checking interval

The initial checking interval is based on instrument manufacturer
recommendation and company experience.

The checking interval is based on:
Calendar

Amount of use

Adherence to the checking system makes or breaks the system
SMEs Quality Assurance Course 17-19 September 2001
Level of standards
4
Primary Standards:
These standards (gage blocks, standard cells, standard electrical
bridges) are certified (occasionally) by a national metrology
institute. They are used only to check secondary standards.
4
Secondary Standards:
These standards are checked periodically against the primary
standards. They are not used to measure the product but to
check the shop standards
4
Shop Standards:
These standards are used to check the product itself

SMEs Quality Assurance Course 17-19 September 2001
National and international (measurement) standards
4
Measurement Standard
A material measure, measuring instrument, reference material or
system intended to define, realise, conserve or reproduce a unit or one
or more known values of a quantity to serve as reference.
4
National Standard
A (measurement) standard recognised by an official national decision
to serve, in a country, as the basis for fixing the value of all other
standards of the quantity concerned.
4
International Standard
A (measurement) standard recognised by an international agreement to
serve internationally as the basis for fixing the value of all other
standards of the quantity concerned.
SMEs Quality Assurance Course 17-19 September 2001
Traceability to national/international standards

Traceability
The property of a result of a measurement whereby it can be related to
appropriate standards, generally national or international standards, through an
unbroken chain of comparison. (ANSI/NCSL Z540-1-1994)
The purpose of requiring traceability is to ensure that measurements are accurate
representations of the specific quantity subject to measurement, within the
uncertainty of the measurement.

Uncertainty
A parameter, associated with the result of a measurement, that characterises the
dispersion of the values that could reasonably be attributed to the quantity
subjected to measurement.
SMEs Quality Assurance Course 17-19 September 2001
Calibration records and certificates
4
A record system will be maintained including the list of instruments
under calibration control and records for each calibration performed
(containing, at least, sufficient information to permit the repetition of
the calibration, along with the identity of the performing technician).
4
When calibrations are performed by an outside (accredited) Lab a
Certificate of Calibration will be requested for each calibration. The
certificate will present, in a clear, unambiguous and objective manner,
all information necessary for the interpretation of the calibration results.
4
When instruments are sent for external calibration, an official record of
the shipment should be kept on file until the instrument is returned.
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance: main elements/techniques
4
Organisation
4
Manual, Procedures, Plans and Records
4
Inspections
«
Audits
4
Nonconformance Control
4
Training
4
Qualifications/Certifications
SMEs Quality Assurance Course 17-19 September 2001
What is an audit ?

An audit is an independent examination of objective evidence performed
by competent personnel, to determine whether or not the auditee
- is able to achieve its policies and objectives, and/or
- is capable to fulfil its contractual and legal obligations; and

- is effectively implementing a QA system
It is also the true and fair presentation of the results of such examination.
When conducting an audit, it is vital constantly to remember that one’s
aim is

FACT FINDING NOT FAULT FINDING

Obviously, there will be occasions when faults will be found, but that is a
fact!
SMEs Quality Assurance Course 17-19 September 2001
Audits
System audit
Process audit
Compliance audit
Product audit
SMEs Quality Assurance Course 17-19 September 2001
System audit

Documented activity performed to verify, by examination and
evaluation of objective evidence, that applicable elements of the
quality system are suitable and have been developed, documented, and
effectively implemented in accordance with specified requirements
A system audit probes whether or not there is an organisation and a
quality assurance system in existence and assesses their adequacy to
meet the requirements invoked by a contract.

It is advantageous to conduct a systems audit well in advance of the
actual starting of an activity. System audits are akin to readiness
reviews.
SMEs Quality Assurance Course 17-19 September 2001
Compliance audit

The compliance audit investigates whether or not the QA system is
implemented. The auditor will scrutinise the objective evidence that
should be generated by the implementation of the system. The auditor
checks that the auditee works to the letter of the QA procedures.

Compliance audit must not be confused with an “inspection”. The
latter is concerned with product acceptance and release for further
processing. The former concerns itself with verifying that the QA
system is implemented satisfactorily.
SMEs Quality Assurance Course 17-19 September 2001
Process audit

Analysis of elements of a process and appraisal of completeness,
correctness, or conditions
SMEs Quality Assurance Course 17-19 September 2001
Product audit

Quantitative assessment of conformance to required product
characteristics
SMEs Quality Assurance Course 17-19 September 2001
Internal audit
4
Internal audits are not performed in a single shot. At start of each year
an Internal Audit Plan is drawn up. Each area of the company will be
audited at least once annually
4
The objective is to ascertain if all personnel who have duties and
responsibilities in accordance to the Quality Manual (or PA Plan),
understand and satisfactorily perform their duties and responsibilities
4
The audit of the QA/QC organisation is considered accomplished by
customer’s audits, third party maintenance of certification checks (if
any), MIP’s
SMEs Quality Assurance Course 17-19 September 2001
External audit
4
Preparation
4
Performance
4 Conclusions
4
Report
4
Follow-up
SMEs Quality Assurance Course 17-19 September 2001
Audit preparation and performance
4
Pre-audit Questionnaire and visit
4
Audit team selection
4
Audit notification
4
Opening meeting
4
Audit performance
4
Debriefing meeting
4
Report and follow-up
SMEs Quality Assurance Course 17-19 September 2001
Pre-audit questionnaire and visit
4
To collect basic information about the company to be audited
4
To ensure no misunderstanding about audit scope and objectives
4
To obtain the QA Manual
(including compliance matrix w.r.t. audit baseline)
4
To Identify the reference documentation
4
To visit the facilities
SMEs Quality Assurance Course 17-19 September 2001
Audit team selection
4
It includes at least an audit team leader
4
If necessary one or more additional auditor because of areas of
expertise (e.g. S/W PA, RAMS, CM)
4
Project QA representative (if a project audit)
SMEs Quality Assurance Course 17-19 September 2001
Audit notification
4
A reasonable time before the audit (e.g. agreement on dates 2 months
before the audit, formal notification 1 month before audit)
4
Notification in writing and including:

scope/baseline of audit

schedule

name of lead auditor

draft agenda
SMEs Quality Assurance Course 17-19 September 2001
Opening meeting
4
Introduction of attendants:
– meeting the management
– meeting counterparts
4
Presentation of audit agenda (team leader):
– summarise audit scope and procedure
– finalise detail agenda (splinters if any)
4
Presentation of the company and logistics:
– assigned rooms
– documentation access
SMEs Quality Assurance Course 17-19 September 2001
Performance of audit
4
Investigate by means of audit checklist:
– use audit checklist is as guide
– not all questions are to be asked
– adapt the wording to your counterpart
– adapt the sequence of questions to
circumstances (locations) “to be re-inquired”
4
Look for OBJECTIVE EVIDENCE ( trace of facts, such as reports,
signatures, certificates):
– statements are not sufficient (it may be useful to note
them_”we are told that...”) and
– feelings are not sufficient
to conclude on conformance to requirements
SMEs Quality Assurance Course 17-19 September 2001
Performance of audit (cont’d)

If you identify a non conformance or deficiency
4
confirm it with your counterpart: questions to be answered by
“yes” or “no”, note explanations
4
determine if it is of a systematic nature in view of conclusions

REMINDER:

In the conduct of the audit keep in mind that the Customer/Supplier
relationship of the past that operated in an adversarial environment is
becoming ancient history
SMEs Quality Assurance Course 17-19 September 2001
Debriefing meeting
4
Debriefing meeting:

meeting with the management of audited organisation

each auditor presents his Observations Sheets (OS)

provide clarifications

collect signatures (if not yet done), as recognition that each
reported observation is as verbally indicated during the audit

audit team leader presents opinion on the audit results

request formal response to each observation
SMEs Quality Assurance Course 17-19 September 2001
Audit report
4
Audit Report:

purpose of the audit

requirements baseline

audit team members

audit agenda

personnel contacted

summary of results

Observations Sheets

recommendations on each OS
SMEs Quality Assurance Course 17-19 September 2001
Audit follow-up
4
Obtain written response for each OS
4
Evaluate response:

proposed corrective actions

proposed schedule
4
Agree and formalise corrective actions and schedule
4
Follow-up visits to verify close-out of corrective actions
4
Formal audit closure
SMEs Quality Assurance Course 17-19 September 2001
Audit terminology

Observation
An observation is made every time a total absence of arrangements to
meet an important requirement or arrangements which are clearly
inadequate is found. Observations are also made when lack of
implementation of Quality Manual or procedures is revealed.

The term “Non-Compliance” is also often used as synonym.
Recommendation
A Recommendation describes an acceptable solution to an
observation. A recommendation should be written when it considered
helpful to speed-up the achievement of agreement on a corrective
action.
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance: main elements/techniques
4
Organisation
4
Manual, Procedures and Plans
4
Inspections
4
Audits
«
Nonconformance Control
4
Training
4
Qualifications/Certifications
SMEs Quality Assurance Course 17-19 September 2001
Non-conformance procedure

Usual elements of a nonconformance reporting (NCR)* procedure:

- Forms definition for internal/external reporting

- Initiator responsibilities (is anybody authorised to issue a report or
only inspectors/QA?)
- Processing/meetings (formal meeting often only in special cases)

- Responsibility for investigation of causes, preventive actions

definition/concurrence and tracking
*often called SPR (S/W Problem Report) for S/W products
SMEs Quality Assurance Course 17-19 September 2001
Non-conformance procedure (cont’d)

- Material Review Board responsibilities and membership (by name)

- Serial or parallel NCR distribution distribution (e.g., next higher
level+customer)

- Interfaces with Configuration Control function (issuing of waivers)
- Disposition implementation. [Repair: to be treated as any

manufacturing operation (written instructions) except for very

simple cases]

- Segregation of non-conforming product
SMEs Quality Assurance Course 17-19 September 2001
Non-conformances: disposition
v
Return to supplier: Only applies to non-conforming procured items
v
Use-as-is: The non-conformance is acceptable at the status
v
Rework: Re-application of process will eliminate the non-conformity
v
Repair: It is basically a modification that will allow use of the item
v
Scrap: The item is not acceptable at the status or recoverable
Dispositions Use-as-is and Repair may require issuing a waiver.
SMEs Quality Assurance Course 17-19 September 2001
Nonconformance processing: example
QUALITY CONTROL
or other authorised unit
MANUFACTURING/TESTING
OR
RECEIVING INSPECTION
• PREPARES DISCREPANCY
NOTICE (DN)
• SEGREGATION
PRELIMINARY
REVIEW (PR)
• QA CONDUCTS PR
• DISPOSITION FOR SRP
OR MINOR REWORK
• PROMOTION TO MRB
QUALITY ASSURANCE
• NCR PREPARATION
• NOTIFIES OF FORTHCOMING
MRB
• ASSESSES CUMULATIVE
EFFECT OF PREVIOUS NCRs
MATERIAL REVIEW
BOARD (MRB)
• CUSTOMER
PARTICIPATION FOR
MAJOR NCR
• MRB DETERMINES
DISPOSITION
• CONDUCTS CCB
• DISPOSITIONS AND
APPROVES WAIVER
• CONTRACTUAL
AUTHORIZATION TO
PROCEED WITH WAIVER
(NOTE 5)
CUSTOMER
(ANY LEVEL)
CONFIGURATION
MANAGEMENT
• WAIVER NUMBERING
• SCHEDULES CHANGE
BOARD (if needed)
• PREPARES SUPPORTING
DATA AND DESCRIPTION /
JUSTIFICATION FOR WAIVER
QUALITY ASSURANCE QUALITY ASSURANCE
• VERIFIES CORRECT
IMPLEMENTATION OF
DISPOSITION (after
implementation)
WAIVER
YES NO
SMEs Quality Assurance Course 17-19 September 2001
Non-conformances: investigating root causes

Root cause is the real reason for a nonconformity. It is possible that
symptoms of the problem are identified as the cause, and it is the
symptoms that are treated rather than the cause.

A defect attributed to operator error may not be the real reason if the
operator has not been trained, had inadequate work instructions, or
was working with a poor design.
4
For each non conformity there is a root cause
4
Causes are preventable
4
Prevention is always cheaper

SMEs Quality Assurance Course 17-19 September 2001
Non-conformances: corrective actions

Corrective actions shall be aimed to prevent recurrence:

- Determine and document action to remove root causes

- Plan corrective actions (what, who and when)

- Identify interim actions

A Corrective Actions System should be in place to:

- summarise and collect nonconformances data (including costs)
- determine need/effectiveness of corrective actions
- initiate QIP (Quality Improvement Project) when appropriate
SMEs Quality Assurance Course 17-19 September 2001
The Material Review Board

Purpose of the MRB is to provide an organised review of non-conforming
product. The MRB is chaired by QA with members from Engineering and
other specialists teams (e.g. Materials, S/W)
Members of the MRB have collective responsibility and specific
responsibility. Specific responsibility relate to the field of specialisation.

Specific responsibility of the QA function in the MRB is to:
- investigate root causes to identify corrective/preventive actions

- ensure that remedial actions are properly implemented

- ensure that the process is correctly followed and records kept
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance: main elements/techniques
4
Organisation
4
Manual, Procedures, Plans and Records
4
Inspections
4
Audits
4
Nonconformance Control
«
Training
4
Qualifications/Certifications
SMEs Quality Assurance Course 17-19 September 2001
Training

The existence of documented plans, procedures and specifications will
not result in hardware quality if the skills of manufacturing personnel
are not sufficient to execute the instructions.

Training in any area is based upon:

- familiarisation with written process specification/procedures

- practice procedure under instructor’s observation/support

- hardware test/inspection providing feedback on success

For some processes, this is all that is required, and records will simply
note that the individual was given training on a certain date.
SMEs Quality Assurance Course 17-19 September 2001
Training (cont’d)

Advanced training including testing are required for critical processes
and special techniques such as:
4
Welding
4
Adhesive Bonding
4
Electronics Assembly
4
Conformal Coating
4
NDI
4
Metrology

Testing may be written, or an ability demonstration, or combination of
written and demonstration. Testing outcome must be quantified.
SMEs Quality Assurance Course 17-19 September 2001
ESA skills training centers
X
X
X
X
IdS
Paris (F)
SMT
X
Fiber Optic
Termination
Hytek
Aalborg
(DK)
IIS
Genova
(I)
Highbury College
Portsmouth
(UK)
Hand Soldering
Operator
ZVE
Oberpfaffenhofen
(D)
Hand Soldering
Inspector
Repair/Modif
PCBs
RF Cable Assy
Crimping
Wire Wrapping
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X X
IFE
Obrpfaffenhofen
(D
X
X
X
X
Instructor
X
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance: main elements/techniques
4
Organisation
4
Manual, Procedures, Plans and Records
4
Inspections
4
Audits
4
Nonconformance Control
4
Training
«
Qualifications/Certifications
SMEs Quality Assurance Course 17-19 September 2001
Qualification/certifications
4
Qualification

process qualification

product qualification
4
Certification

quality system certification

product conformity certification

personnel certification

stamp warranty
SMEs Quality Assurance Course 17-19 September 2001
Certificate of Conformity

“A document signed by an authorised

party affirming that the supplier

of a product or service has met the

requirements of the relevant

specifications, contract or regulation.”
SMEs Quality Assurance Course 17-19 September 2001
Personnel certification
4
Certification is very similar to testing but the process is more formal.
The testing process and standard for pass/fail must be documented
4
The certifying organisation may be:
– the contractor
– a space agency or a delegated organisation
– a nationally recognised organisation
4
Some skills must be practised on a regular basis in order to maintain
the currency of the certification

4
Some certification may require annual medical eye exam
SMEs Quality Assurance Course 17-19 September 2001
Personnel certification: records
4
Certification records will include at least:

location and date of training

certification process procedure

specification/procedure

grade/rating

instructor name

expiration date

periodic renewals

achieved job continuity
SMEs Quality Assurance Course 17-19 September 2001
Personnel certification: re-certification
4
Re-certification is required for one of the following reasons:

certification expiration

interruption of work

unsatisfactory performance

evolution of techniques
SMEs Quality Assurance Course 17-19 September 2001
Stamp warranty
4
Stamp warranty means that an inspector who signifies the status of an
inspection or test by stamping:
4
is qualified to determine that status
4
personally saw or performed the activity as documented
4
At the time of stamp release, the inspector will acknowledge in
writing his full understanding of the rules for use of stamps
SMEs Quality Assurance Course 17-19 September 2001
QA during Major Project Phases
<<>>
SMEs Quality Assurance Course 17-19 September 2001
Quality assurance during major phases
4
Design and Verification

4
Procurement
4
Manufacturing
4
Testing
4
Acceptance/ delivery
SMEs Quality Assurance Course 17-19 September 2001
Design and verification cycle: the V-cycle
Mission Requirements,
System Concept
System Specification
Sy Verification Plan
CI Specification,
CI Verification Plan
“Build-to” Documents
Inspection Plan
Fabricate, Assemble
and Code
Inspect to “build-to”
Documents
Assemble CI, perform
CI Verification
Integrate System,
perform System
Verification
Demonstrate and
Validate the System
D
e
c
o
m
p
o
s
i
t
i
o
n

a
n
d

D
e
f
i
n
i
t
i
o
n
S
e
q
u
e
n
c
e
I
n
t
e
g
r
a
t
i
o
n

a
n
d

V
e
r
i
f
i
c
a
t
i
o
n
S
e
q
u
e
n
c
e
SMEs Quality Assurance Course 17-19 September 2001
Design and verification
4
Procedures for engineering documents
4
Design/Development and Verification plans
4
QA support to internal design review for:

producibility

repeatibility

inspectability

testability
4
QA review of design changes for quality impact
SMEs Quality Assurance Course 17-19 September 2001
Design and verification: structured requirements
EXAMPLE
System Specification
External Interface Requirements
ESA
Requirements Document
Detailed
Design Document
Architectural
Design Document
SMEs Quality Assurance Course 17-19 September 2001
Types of design requirements
4
Functional Requirements
Tasks, actions or activities that must be accomplished by a system,
without specifying how it has to be done
4
Performance Requirements
The extent to which a function must be executed, generally covered in
terms of quantity, quality, coverage, timeliness or readiness
4
Design Constraints
Boundary conditions within which the designer must remain while
allocating performance requirements and/or synthesizing system
elements (safety, environmental, interface, technology etc.)
SMEs Quality Assurance Course 17-19 September 2001
Requirements for requirements
4
Design requirements must be:

complete

consistent

unambiguous

verifiable

traceable
The QA function must ensure that design requirements are properly
documented, and verifications consistently planned and thoroughly
executed. Ensure that QUALITY is designed into the product.
The QA function must ensure that design requirements are properly
documented, and verifications consistently planned and thoroughly
executed. Ensure that QUALITY is designed into the product.
SMEs Quality Assurance Course 17-19 September 2001
Quality assurance during major phases
4
Design and Verification
«
Procurement
4
Manufacturing
4
Testing
4
Acceptance/ delivery
SMEs Quality Assurance Course 17-19 September 2001
Procurement control
4
Selection of procurement sources
4
Control of purchase documents
4
Surveillance of suppliers
SMEs Quality Assurance Course 17-19 September 2001
Selection of procurement sources

The selection is usually based on the following criteria:
4
certification/approval for specific items
4
demonstrated capability (positive performance records)
4
pre-award audit
Maintain a list of subcontractors and suppliers including records
of supplies and performance data (e.g., % of defective items, NCRs)
Maintain a list of subcontractors and suppliers including records
of supplies and performance data (e.g., % of defective items, NCRs)
SMEs Quality Assurance Course 17-19 September 2001
Control of purchase documents

Procurement documents shall contain by statement or reference:
4
technical description of the item/service to be procured
4
QA requirements and deliverable/accompanying
documents (e.g., CoC, ADP, interface data records)
4
QA activities to be performed by the supplier
4
planned activities at source (if any)
Purchase Orders (PO) and Contracts shall be reviewed and formally
approved by the QA function prior to release
Purchase Orders (PO) and Contracts shall be reviewed and formally
approved by the QA function prior to release
SMEs Quality Assurance Course 17-19 September 2001
Surveillance of suppliers

QA surveillance of subcontractors and suppliers is performed through
periodic audits and inspections (MIPs, source inspection, incoming
inspection). In some special cases a resident inspector may be needed
The QA plan should always provide information and justification
on the planned surveillance activities of subcontractors and suppliers
The QA plan should always provide information and justification
on the planned surveillance activities of subcontractors and suppliers
SMEs Quality Assurance Course 17-19 September 2001
Quality assurance during major phases
4
Design and Verification
4
Procurement
«
Manufacturing
4
Testing
4
Acceptance/ delivery
SMEs Quality Assurance Course 17-19 September 2001
Manufacturing
4
Materials and parts control
4
Production equipment control
4
Cleanliness and contamination control
4
Control of manufacturing processes
4
Control of manufacturing documents
SMEs Quality Assurance Course 17-19 September 2001
Materials and parts control
4
Control must be exercised to ensure that materials and parts released
for manufacturing are:

correct /traceable

conforming

within their useful life (if life limited)
Ensure that shop-travelers always include materials and/or parts
identity verifications as first operation.
Ensure limited-life materials tracking and labeling with expiration dates
Ensure that shop-travelers always include materials and/or parts
identity verifications as first operation.
Ensure limited-life materials tracking and labeling with expiration dates
SMEs Quality Assurance Course 17-19 September 2001
Production equipment control
4
Control must be exercised to ensure that manufacturing tools are:

conform to their drawings

periodically checked (recurrent production)

properly stored to prevent deterioration/damage

identified

inventory kept
SMEs Quality Assurance Course 17-19 September 2001
Production equipment control (cont’d)
4
With the increasing use of computer-aided manufacturing and
inspection techniques, the relevant s/w need to be:

documented

tested and approved prior to use

configuration controlled

secured to prevent tampering and misuse
The QA function must have a primary and documented role in the
validation and control of production tools, including any COTS software
The QA function must have a primary and documented role in the
validation and control of production tools, including any COTS software
SMEs Quality Assurance Course 17-19 September 2001
Cleanliness and contamination control
4
Cleanliness levels
The required cleanliness levels of contamination-sensitive items will
be indicated on specifications, drawings and procedures
4
Contamination control plan
This is a plan identifying the control measures necessary to achieve
the required level of cleanliness during manufacturing, testing,
handling, packaging and storage of contamination-sensitive items
4
Cleanliness of facilities
“Clean facilities”conditions and operations need to be controlled to
ensure that their specified level of cleanliness is maintained.
SMEs Quality Assurance Course 17-19 September 2001
Cleanliness and contamination control (cont’d)
4
Clean rooms facilities instructions will cover:

entering/leaving (personnel and equipment)

cleaning methods and materials

preventive maintenance

control/recording of clean room parameters
(pressure, humidity, temperature)

performance of particle counts

action in case of alarm (stop work, shield items)
The QA function has a primary role in enforcing compliance to the
Contamination Control Plan and verification of clean-facilities
The QA function has a primary role in enforcing compliance to the
Contamination Control Plan and verification of clean-facilities
SMEs Quality Assurance Course 17-19 September 2001
Manufacturing processes
4
Process
A sequence of pre-defined steps intended to achieve a goal or
objective
4
Manufacturing process:

Special Process

Critical Process

Standard Process
SMEs Quality Assurance Course 17-19 September 2001
Special process

Special process is a process in which the outcome cannot be adequately
verified without destroying the manufactured part (e.g., heat treatment,
bonding, welding). Special process require:
4
qualification
4
control of process parameters
4
non-destructive inspection
4
destructive testing (e.g., chemical analysis, tensile test)
of representative samples/coupons
4
periodic control of process equipment/materials
4
training/certification of personnel
4
records maintenance/archiving
SMEs Quality Assurance Course 17-19 September 2001
Critical process

Critical Process is not necessarily the same as Special Process.
A critical process may or may not be a special process and vice-versa
A critical process is one:
v
which in case of failure can adversely affect performance
or destroy a major part or system function
v
the quality of which cannot be assessed solely by
examining the end product
v
which has caused problems previously
v
with which the contractor has no experience
SMEs Quality Assurance Course 17-19 September 2001
Critical process: PSS and ECSS definitions

.
PSS-01-700 ECSS-Q-70A DEFINITION
- A process that in the case of failure can adversely affect the
performance or destroy a major part or function of the space system
- A process with which the contractor in charge of it has had no
previous experience
-Where an ESA specification is available but is not being implemented
- The quality of which cannot be assessed solely by examining the
end product
- A process that has caused problems previously
YES
YES
-
YES
-
YES
-
YES
YES
YES
SMEs Quality Assurance Course 17-19 September 2001
Standard process
4
Standard process is one that is well documented, has a previous
history of use, is well understood and for which standard inspection
procedures exist
SMEs Quality Assurance Course 17-19 September 2001
Manufacturing process control
4
Manufacturing Process control

- Emphasis on process control depends on the criticality of the process

- Each process must be covered by a Process Specification, including

control methods, QA provisions and a/r criteria
- ESA requires delivery of a project Process List to verify that

processes are properly specified and controlled. Each process

will be classified as standard/established or critical
SMEs Quality Assurance Course 17-19 September 2001
Control of manufacturing documents
4
Manufacturing documents

identified/traceable to contract no./purchase order

include rev.status/date, release date

include item identification and configuration data

kept with product during manufacturing/inspection operations

include provisions for traceability of operators and inspectors
4
Manufacturing documents control

QA concurrence to ensure adequacy and right placement of
inspections (any later sequence change needs QA agreement)

Design Office approval for critical/special processes (any later
sequence change requires Design Office approval)
SMEs Quality Assurance Course 17-19 September 2001
Quality assurance during major phases
4
Design and Verification
4
Procurement
4
Manufacturing
«
Testing
4
Acceptance/ delivery
SMEs Quality Assurance Course 17-19 September 2001
Testing (qualification and acceptance)
4
Test facilities and equipment
4
Test reviews
4
Test surveillance
4
Test documentation:

test procedures

test reports
SMEs Quality Assurance Course 17-19 September 2001
Test facilities and equipment
4
Environmental tests at certified facilities
4
Test equipment verifiable without application to the test article
4
Test software validated/verified prior to use
SMEs Quality Assurance Course 17-19 September 2001
Test reviews
Test Procedure/
Test Readiness
TRR
authorisation
Test
Performance
Test
Report
PTR
yes
no
TRR: Test Readiness Review
PTR: Post-Test Review
SMEs Quality Assurance Course 17-19 September 2001
Test Reviews: TRR
4
Test Readiness Review

test article configuration (as-required/as-built)

any open NCR for impact

test procedure (availability/approval)

test facility readiness

test procedure (availability/approval)

test equipment calibration status

protection of personnel and test article
SMEs Quality Assurance Course 17-19 September 2001
Test Reviews: PTR
4
Post-test Review

completeness of required data

deviations/modifications for proper authorisation

NCRs during test for proper disposition

test results compliance with requirements

grant authorisation for further tests/processing
Ensure that QA is represented in the boards established for reviewing
test readiness and test results. Ensure QA surveillance during test.
Ensure that QA is represented in the boards established for reviewing
test readiness and test results. Ensure QA surveillance during test.
SMEs Quality Assurance Course 17-19 September 2001
Test surveillance
4
Test surveillance is performed by QA in one of the following forms:

continuous witnessing
(for tests requiring manual intervention)

observation
( periodic monitoring)

data review
( review recorded data)
4
QA personnel has generally the authority to stop the test for safety
reason or to prevent hardware damages
SMEs Quality Assurance Course 17-19 September 2001
Test documentation
4
Test procedure/report must include:
– test article identification
– test equipment and test set-up
– characteristics and design criteria to be tested
– data recording and pass/fail criteria
– sequence of operations (including inspections)
– measures to ensure safety
– environmental conditions
– test organisation (personnel/responsibilities)
– test results and conclusions (for test reports)
4
Test procedures must be reviewed and approved by QA
SMEs Quality Assurance Course 17-19 September 2001
Quality assurance during major phases
4
Design and Verification
4
Procurement
4
Manufacturing
4
Testing
«
Acceptance/delivery
SMEs Quality Assurance Course 17-19 September 2001
Acceptance/delivery
4
Acceptance Data Package

data to support acceptance

documents/data to support further integration
4
Acceptance review

formal acceptance review board, or

Delivery Review Board (DRB)
4
DRB responsibilities:

assess conformity to contractual requirements

verify nonconformances closed-out

accompanying documentation complete
SMEs Quality Assurance Course 17-19 September 2001
Acceptance/delivery
4
QA is a member of the DRB
4
QA will perform surveillance of packaging/labeling operations
4
QA will ensure that accompanying documentation is included
QA has a leading role in acceptance and delivery activities and
must be directly involved in their performance
QA has a leading role in acceptance and delivery activities and
must be directly involved in their performance
SMEs Quality Assurance Course 17-19 September 2001
Beware of the “enemy” called bureaucracy !

Many companies write a huge binders full of procedures to fulfill the
customer’s (or ISO standard’s) requirements regardless of whether the
procedures do the company any good or make any practical sense. In
such a case, engineers and personnel recognise that they are simply
facing a bureaucratic exercise and ridicule the QA System.
THE PURPOSE OF A QA SYSTEM IS TO CREATE PROSPERITY
NOT
ENGENDER BUREACRACY
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance
Part II : QA Standards
SMEs Quality Assurance Course 17-19 September 2001
Standards Evolution
<<>>
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance: standards evolution
4
ISO standards
4
Military standards
4
Space standards:
4
NASA standards
4
European standards
SMEs Quality Assurance Course 17-19 September 2001
Where does ISO 9000 come from?

1959 U.S. DoD establishes MIL-Q-9858 “Quality
Management Program”
1963 MIL-Q-9858 updated to RevA
1968 NATO adopts MIL-Q-9858A as AQAP-1

1970 U.K. MoD adopted AQAP-1 as DEF/STAN 05-8
1979 BSI (British Standards Institution) develops the first

commercial quality management standard, BS 5750
1987 ISO created ISO 9000 standards from BS 5750
SMEs Quality Assurance Course 17-19 September 2001
The ISO 9000(1994) series
4
The ISO 9000 (1994) series was composed of five standards:
ISO 9000 Guidelines for selection and use of 9001, 9002, 9003
ISO 9001 Standard for organisations involved in design,
production, and service
ISO 9002 Standard for organisations involved in production
ISO 9003 Standard for organisations involved in final inspection
and test
ISO 9004 Guidelines for quality management
SMEs Quality Assurance Course 17-19 September 2001
ISO 9001(1994) compared to “traditional” QA
ISO 9001
(1994)

- Management Review
- Contract Review
- Servicing
Quality Assurance
(Traditional)
Design Assurance
(Basic Principles)
SMEs Quality Assurance Course 17-19 September 2001
Cross reference table of ISO 9001, 9002 and 9003
Cross Reference of ISO9001, ISO 9002, and ISO 9003
ISO Clause and Title 9001 9002 9003
4.1 Management Responsibility
4.2 Quality System
4.3 Contract Review
4.4 Design Control NA NA
4.5 Document Control
4.6 Purchasing
4.7 Control of Customer-supplied Product
4.8 Product identification and Traceability
4.9 Process Control NA
4.10 Inspection and Testing
4.11 Inspection, Measuring and Test Equipment
4.12 Inspection and Test Status
4.13 Control and Non-conforming Product
4.14 Corrective Action
4.15 Handling, Storage, Package and Delivery
4.16 Quality Records
4.17 Internal Quality Audits
4.18 Training
4.19 Servicing NA
4.20 Statistical Techniques
Comprehensive Requirement
Less Comprehensive than 9001 and 9002
NA Element Not Present
SMEs Quality Assurance Course 17-19 September 2001
ISO 9001(1994): management review

Executive responsibility

Quality policy

Relevant

Organisation

Defined intervals

Management Suitability
Review & Effectiveness

Records
Management
Responsibility
SMEs Quality Assurance Course 17-19 September 2001
ISO 9001(1994): contract review
Procedure differences
incoming order/ capabilities
Requirements contract unwritten rqts
Review

outgoing proposal

process definition
Amendments int.
communication
ext.
Records

Contract
Review
SMEs Quality Assurance Course 17-19 September 2001
ISO 9001(1994): design control



Procedure



Design/Development Plan



Organisation



Design Input



Design Output



Design Review



Verification



Validation



Changes
Design
Control
SMEs Quality Assurance Course 17-19 September 2001
New format of ISO 9001(2000)
“Quality management systems - Requirements”
4
None of the ISO 9001(1994) requirements have been removed,
they have been reorganised in five main sections:

Quality Management System (QMS)
– Management Responsibility
– Resource Management
– Product Realisation
– Measurement, Analysis and
Improvement
4
ISO 9002 and 9003 cancelled. ISO 9001 will be used instead by
excluding certain requirements
SMEs Quality Assurance Course 17-19 September 2001
Basic terminology modifications
4 ISO 9000(1994)
4 ISO 9000(2000)
SUBCONTRACTOR
SUPPLIER
CUSTOMER
SUPPLIER
ORGANISATION
CUSTOMER
SMEs Quality Assurance Course 17-19 September 2001
ISO 9001(2000):
Clause 4 - Quality Management System (QMS)
4
Demonstrate continuous improvement in QMS effectiveness
4
Define processes for QMS application throughout the organisation
4
Documents demonstrate effective planning
SMEs Quality Assurance Course 17-19 September 2001
ISO 9001(2000):
Clause 5 - Management Responsibility
4
Customer requirements must be defined and fulfilled
4
Quality policy provides framework for objectives and is measurable
4
Effectiveness of QMS is communicated
SMEs Quality Assurance Course 17-19 September 2001
ISO 9001(2000):
Clause 6 - Resource Management
4
Establish competency needs
4
Determine training effectiveness
4
Infrastructure/working environment provided for
4
Determine resources to:

Implement/maintain QMS

Continually improve

Meet customer objectives
SMEs Quality Assurance Course 17-19 September 2001
ISO 9001(2000):
Clause 7 - Product Realisation
4
Quality objectives/product requirements set
4
Customer requirements for delivery and post delivery determined
4
Obligations related to product (e.g. statutory and regulatory
determined)
4
Product function and performance requirements defined
4
Control of any customer intellectual property
SMEs Quality Assurance Course 17-19 September 2001
ISO 9001(2000):
Clause 8 - Measurement, Analysis and Improvement
4
Monitor customer perceptions on fulfilment of requirements
4
Customer satisfaction data needed is defined and subsequently
analysed
4
Audits to verify compliance with planned arrangements, as well as
QMS requirements and ISO 9001 (2000) requirements
4
Audit frequency and methodology to take account of previous results
4
Process defined for waiver approval and defect reporting to relevant
authority
SMEs Quality Assurance Course 17-19 September 2001
Applying ISO 9001 to software

Software development, can only be successful if :
4
specific methods and controls are in place from the start
4
the development life cycle is made visible and manageable
4
customer and supplier come together at the defined stages
SMEs Quality Assurance Course 17-19 September 2001
Applying ISO 9001 to software (cont’d)
4
ISO 9000-3 “Guidelines for the application of ISO 9001 to the
Development, Supply, and Maintenance of Software” provides
guidance on:

software “life-cycle” activities and reviews

requirements specification

plans (quality, CM, development, verification, test, maintenance)

release and maintenance procedures/records

specific support activities (CM and DM, purchasing quality
records measurement, use of customer supplied s/w, training)
SMEs Quality Assurance Course 17-19 September 2001
Applying ISO 9001 to software (cont’d)
4
When looking for third party certification, it is important that the
identified auditor is software knowledgeable and that he looks for
good software development practices around key software processes.
4
TickIT is a British industry specific standard for auditing software
companies to the ISO 9001 standard using the ISO 9000-3 guidelines
SMEs Quality Assurance Course 17-19 September 2001
Problems with ISO 9000 (%)
12
12
16
18
18
19
24
25
27
31
0 20 40 60 80 100
Difficulty interpreting the standard
The vagueness of the standard
Time spent checking paperwork prior to audits
Lack of consistency between auditors
Lack of free advice
High cost of maintain the standard
Time required to complete implementation
High Cost of implementation
High volume of paperwork
Time required to write the manuals
SOURCE: MANCHESTER BUSINESS SCHOOL
SMEs Quality Assurance Course 17-19 September 2001
22
34
52
54
57
58
61
61
65
78
0 20 40 60 80 100
Cost reductions
Proof of ‘TQM’
Improve product quality
Make operations more efficient
ISO 9000 is a good ‘promotional tool’
Customer pressure
Improve service quality
Maintain/improve market share
Increase consistency of operations
Future customers likely demand for ISO 9000
Reasons for Seeking ISO 9000 (%)
SOURCE: MANCHESTER BUSINESS SCHOOL
SMEs Quality Assurance Course 17-19 September 2001
Benefits Resulting from ISO 9000 %
83
82
61
62
60
60
52
50
49
24
0 20 40 60 80 100
Improved management control
Improved customer satisfaction
A motivated workforce
Increased opportunity to win work
Increased productivity/ efficiency
Reduced wastage
More effective marketing
Reduced costs
Increased market share
Increased exports
SOURCE: MORI
SMEs Quality Assurance Course 17-19 September 2001
ISO 9000 COSTS
4
Internal resources
4
ISO 9000 consultants (if required)
4
Application fee
4
Optional pre-assessment
4
Assessment
4
Maintenance of system
4
Surveillance visits
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance: standards evolution
B
ISO quality standards
B
Military standards
B
Space standards:
B
NASA standards
B
European standards
SMEs Quality Assurance Course 17-19 September 2001
The fate of QA military standards
4
In the early nineties the U.S. DOD embarked on a new policy which
can allow contractors to use their existing quality system standard
(Performance-Based Contracting, Single Process Initiative)
4
In October of 1996, MIL-Q-9858A and MIL-I-45208A where
cancelled
4
In February 1993, NATO canceled the AQAP 1-15 documents and
adopted a new series of AQAP’s that incorporate the ISO 9000 series
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance: standards evolution
B
ISO quality standards
B
Military standards
B
Space standards:
B
NASA standards
B
European standards
SMEs Quality Assurance Course 17-19 September 2001
NASA-Mission Assurance

Mission Assurance is defined by NASA as a set of specialised disciplines to
support the project development
R i s k M a n a g e m e n t
C o n f i g u r a t i o n M a n a g e m e n t
S o f t w a r e A s s u r a n c e
S y s t e m s S a f e t y
M a t e r i a l s / P r o c e s s e s
E E E C o m p o n e n t s
E n v i r o n m e n t a l C o m p a t .
R e l i a b i l i t y / M a i n t a i n a b i l i t y
D E S I G N A S S U R A N C E Q U A L I T Y A S S U R A N C E
M I S S I O N A S S U R A N C E
SMEs Quality Assurance Course 17-19 September 2001
QA standards at NASA

As for U.S. DOD Military Specifications also NASA Specifications
are being phased out (NHB-5300.4/1-B/1-C), but certain NASA or
Military unique standards may be maintained, some indefinitely.

The "hierarchy of choice" is:
- National or International Voluntary and Consensus Standards

- Corporate Standards
- NASA or Military Standards and Practices

The new NASA system is called “Advanced Quality System (AQS)”
SMEs Quality Assurance Course 17-19 September 2001
NASA - Advanced Quality System (AQS)

NASA's goal is to eliminate unique processes/systems imposed on
contractors shared with DOD or other Federal agencies, unless
essential to ensure mission safety and reliability.
The new NASA business management model is based on three
elements:
'
baseline quality management system
^
advanced risk mitigating practices
·
program/project-unique risk mitigation measures
SMEs Quality Assurance Course 17-19 September 2001
'
NASA baseline QM system

In 1998 NASA issued the Policy Directive NPD 8730.3 “NASA
Quality Management System Policy (ISO 9000) and associated
Management Instruction NMI 1270.3 to:
4
implement an ISO 9000 based Quality System
4
mandate ISO certification of all NASA Centers
4
impose ISO standards as contractual requirements
(when appropriate and beneficial )
4
require suppliers third-party certification
(when appropriate and beneficial )
SMEs Quality Assurance Course 17-19 September 2001
^
NASA advanced practices

Instead of prescribing detailed contractual requirements, NASA
emphasises the identification and selection of those offerors who
propose effective advanced practices:
4Integrated Product/Process Development (IPPD)
4Identification and Control of Key Characteristics
4Design to Manufacturing Process Capability
4Design for Assembly/Manufacturing (DFA/M)
4“Robust” Design
4Geometric Dimensioning and Tolerancing (GD&T)
4Process Variability Reduction (PVR)

SMEs Quality Assurance Course 17-19 September 2001
^
NASA advanced practices (cont’d)
4Process Variability Reduction (PVR)
4Use of stable, capable processes for product acceptance
4Control of variation in the measurement system
4Root cause, closed loop corrective action
4Deployment of AQS elements to subcontractors
4Continuous Improvement (CI)

NASA will base future procurements by increasing the rigor of contractors
selection and placing more importance on past performance. In addition will place
quality incentives tied to Advanced Quality metrics (e.g., Cost of Quality)

More about AQS? Take a look at: More about AQS? Take a look at:
www.boeing.com./companyoffices/doingbiz/supplier/AQS_Guidelines.pdf www.boeing.com./companyoffices/doingbiz/supplier/AQS_Guidelines.pdf
SMEs Quality Assurance Course 17-19 September 2001
·
NASA unique measures

Many programs or projects will continue to have unique requirements
for risk mitigation in assembly, manufacturing or operational
environments.
SMEs Quality Assurance Course 17-19 September 2001
QA standards in european space activities
4
In 1965, CNES issued QA requirements for the satellite FR1
4
In 1975, ESRO published a Product Assurance Manual (QRA-31)
4
In 1981, ESA published a generic space QA standard, PSS-01-20
4
In 1988, CNES issued a coherent series of space project management
specification (SM) for the Ariane 5 development, including A-SM-50 for
quality assurance
P
S
S
SMEs Quality Assurance Course 17-19 September 2001
QA standards in european space activities (cont’d)
4
In 1990, the preparation of a common QA standard was started by ESA
and CNES for the HERMES project (later cancelled)
4
In 1991, ESA published a complete revision of PSS-01-20
4
In April 1996, the first European space QA standard was issued, ECSS-Q-
20, which took to a certain extent into account in its formulation the ISO
9000 series
4
At the end of 1999 the quality system at ESOC,
the ESA space operation center, has been
certified ISO 9001
SMEs Quality Assurance Course 17-19 September 2001
ECSS-Q-20A
<<>>
SMEs Quality Assurance Course 17-19 September 2001
Requirements flow-down and tailoring
ECSS
Standards
ECSS
Standards
SOW
(tailored ECSS)
SOW
(tailored ECSS)
Allocated
&
Derived
Rqts
Allocated
&
Derived
Rqts
Allocated
&
Derived
Rqts
Allocated
&
Derived
Rqts
Allocated
&
Derived
Rqts
Allocated
&
Derived
Rqts
ECSS
Organisation
ESA Project
Prime Contractor
SMEs Quality Assurance Course 17-19 September 2001
ECSS-Q-20A
ECSS-Q-20
ECSS-Q-20



QA Programme Management

QA General Requirements

QA Requirements for Design and Verification

QA Requirements for Manufacturing/Assembly/Integr.

Testing

QA Requirements for Acceptance and Delivery

Operations

Annex A- GSE
SMEs Quality Assurance Course 17-19 September 2001
ECSS-Q-20A 1/3
ECSS-Q-20 Summary/remarks
1. General
2. Quality Assurance Programme - Organisation
- Quality Assurance Programme Plan
- QA Status Reporting
- Personnel Training and Certification
- Quality Assurance Programme Audits
- QA role in Configuration Management: ensure
existence/implementation of CM requirements; QA as
a member of CCB; “as-built” vs “as-designed”
verifications
- Critical Items Control: items for which major
difficulties or uncertainties exist controlled to reduce
risk
3. Quality Assurance General Requirements - Documentation and data control
- Quality records
- Stamp Control
- Traceability
- Metrology and Calibration
- Non-Conformance control system
- Alert system: participation to ESA AS: generation of
alerts for items with multiple applications failing
within specified design/usage limits; assessment of
alerts from Customer
- Handling, Storage, Preservation
- Statistical Quality Control: SQC applications used for
acceptance to be approved by Customer
SMEs Quality Assurance Course 17-19 September 2001
ECSS-Q-20A 2/3
ECSS-Q-20 Summary/remarks
4. QA Requirements for Design and Verification - Design and verification planning
- Evaluation/qualification of new technologies and
critical processes
- Clear organisation interfaces between different
groups
- Design rules implemented to achieve producibility,
repeatibility, inspectability, testability, operability
- Standards and procedures for maintenance of
engineering drawings and specifications
- Requirements verification and qualification process
- Design changes
5. QA Requirements for Procurement - Selection of procurement sources
- Record and list of Procurement Sources
- Applicable requirements defined in procurement
documents
- Surveillance of procurement sources
- Receiving inspection
6. QA requirements for Manufacturing Assembly
and integration
- Planning of MAI activities
- Identification of MIPs
- Adequate instructions (e.g. shop-travellers)
- Identification of critical characteristics
- QA responsibilities (approval of documents etc.)
- Manufacturing Readiness Review
- Control of processes
- Critical processes
- Statistical Process Control
- Workmanship standards
- Materials and Parts Control: only conforming items
release; limited life items
- Equipment control
- Cleanliness and Contamination Control
- Inspection
SMEs Quality Assurance Course 17-19 September 2001
ECSS-Q-20A 3/3
ECSS-Q-20 Summary/remarks
7. Testing - Test facilities
- Test equipment
- Test documentation
- Test reports
- Test performance monitoring
- Test reviews
8. QA Requirements for Acceptance and Delivery- Acceptance Data Package
- Delivery Review Board: to authorise shipment and
certify conformity/completeness including
documentation
- Preparation for delivery
- Delivery: shipping control, items properly preserved
and packaged, accompanying documentation
included
9. Operations - Quality of mission products
- Validation and Qualification of Ground Support
Segments
- QA Plan for operations
- Training and Operator Certification
- Operations anomalies and Feedback corrective Loop;
reporting of anomalies and problems
- Deviations from procedures to be justified
Annex A: Ground Support Equipment - Development and Configuration Control: depend on
the criticality
- Production: as minimum ISO 9002 during MAIT
- Delivery/Maintenance
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance
Part III : Special techniques
SMEs Quality Assurance Course 17-19 September 2001
Statistical Techniques
<<>>
SMEs Quality Assurance Course 17-19 September 2001
Statistical techniques
4
Statistical process control
4
Statistical acceptance sampling
SMEs Quality Assurance Course 17-19 September 2001
Statistical Process Control (SPC)
4
Statistical Process Control is used to:

maintain a process within pre-defined limits
(evaluate process performance and correct)

determine the process capability
4
Statistical Process Control is based on the principle that:
A PROCESS IS IN A STATE OF STATISTICAL CONTROL WHEN
THE VARIATION OBSERVED IN ITS OUTPUT CAN BE
ATTRIBUTED ONLY TO RANDOM CHANCE
SMEs Quality Assurance Course 17-19 September 2001
Process performance and process capability
4
Process performance
Is the day-to-day behavior of a process including all random and assignable-cause effects
4
Process capability
Is an estimate of the best performance of which a process is thought to be capable, with all
effects of controllable, assignable-cause variation removed
4
PERFORMANCE - CAPABILITY = POTENTIAL IMPROVEMENT
SMEs Quality Assurance Course 17-19 September 2001
95%
99.7%
Natural process variability
4
Natural Process Variability
Process variation is due to several causes (e.g. temperature variations,
wearing, clearances, distortions, measurement error etc.).
When each cause is held to its practical minimum the resultant
process variability is called natural and follows a normal distribution
68%
μ
= 0
+2σ +3σ +1σ -3σ -2σ -1σ
Normal Distribution
SMEs Quality Assurance Course 17-19 September 2001
Detecting assignable causes
4
Assignable causes are variations above the minimum. They can be detected by
periodic sampling of the process and plotting data such to preserve the time-
history (i.e., in order of production sequence)
Production Sequence
x
x
x
x
x
x
x
x
x
x
x
mean
+3σ
-3σ
SMEs Quality Assurance Course 17-19 September 2001
Detecting assignable causes (cont’d)
4
Most probably an assignable cause exist when there are:

one or more points outside +/-3σ

2 or 3 successive points between 2-3σ (same side)

4 or 5 successive points beyond +/-1σ(same side)

8 successive points in either 1σ band

8 successive rising and falling trends
SMEs Quality Assurance Course 17-19 September 2001
Estimating process sigma
4
Variation within a sample (e.g. 5 consecutive measurements in a short
interval) reflect inherent process variability. The range “R” is the
excursion between maximum and minimum values within the sample
4
Variation between samples reflect variation due to inherent process
variability plus assignable causes (to the extent they may exist)
where σ’ (sigma-prime) is an “estimated” value of σ, R is the average
of R, and d2 is a correction factor depending on sample size
σ’ = R/d2
SMEs Quality Assurance Course 17-19 September 2001
Calculation of 3-sigma limits (subgroups)
4
For X chart:
4
For R chart :
LCL = X - A2 R
UCL = R D4
LCL = R D3
x
x
x
UCL = X + A2 R
x
SMEs Quality Assurance Course 17-19 September 2001
Calculation of 3-sigma limits (single values)
4
For X chart:
4
A2, I2, D3, D4 are correction factors depending on subgroup size

UCL = X + I2 R
x
x LCL = X - I2 R
x
SMEs Quality Assurance Course 17-19 September 2001
CORRECTION FACTORS
Subgroup
Size (n)
d2 A2 D3 D4 I2
2 1.128 1.880 0 3.267 2.88
3 1.693 1.023 0 2.574 -
4 2.059 0.729 0 2.282 -
5 2.326 0.577 0 2.114 -
6 2.534 0.483 0 2.004 -
7 2.704 0.419 0.076 1.924 -
8 2.847 0.373 0.136 1.864 -
9 2.970 0.337 0.184 1.816 -
10 3.078 0.308 0.223 1.777 -
SMEs Quality Assurance Course 17-19 September 2001
Example: Calculation of 3-sigma limits (subgroups)
E
x
a
m
p
l
e
C
h
a
r
a
c
t
e
r
i
s
t
i
c
s
:

0
.
6
0

+
/
-
0
.
0
0
3
SMEs Quality Assurance Course 17-19 September 2001
Process capability index


- C p above 1.33 Process adequate

- C p below 1.00 Process inadequate .

C p =
tolerance width
process capability
=
USL-LSL
6σ’
SMEs Quality Assurance Course 17-19 September 2001
Process control and capability
USL
UCL
Target value
LCL
LSL
Process out of control but within specification
USL
UCL
Target value
LCL
LSL
Process in control but out of specification
X
SMEs Quality Assurance Course 17-19 September 2001
Cpk
4
The capability index Cp does not take into account the centering of the process.
The ratio that reflects how the process is performing in terms of nominal center,
or target value is Cpk

where Z(min) is the smaller of :





Cpk = Z(min)/3
USL-X
σ’
Z(USL) =
X-LSL
σ’
Z(LSL) =
or
SMEs Quality Assurance Course 17-19 September 2001
“Six-sigma”
4
Six-sigma is a strategy developed by the Motorola Company to
achieve in their processes:
4
The main assumption is a “typical” centering error of +/- 1.5 sigma.
It means passing from 66,807 defects per million (for +/- 3 sigma)
to 3.4 defects per million (for +/- 6 sigma)
Cp = 2
SMEs Quality Assurance Course 17-19 September 2001
“Six-sigma” (cont’d)
4
The six-sigma implementation steps are:
'
Identify critical characteristics
(through marketing, engineering, R& D etc.)
Z
Identify the product elements that influence the
critical characteristics
·
Identify relevant process elements
J
Establish maximum tolerance for each product and
process element
·
Determine actual capability
4
The Motorola Six-Sigma quality initiative is also a motivational and
training initiative
SMEs Quality Assurance Course 17-19 September 2001
Statistical techniques
4
Statistical process control
«
Statistical acceptance sampling
SMEs Quality Assurance Course 17-19 September 2001
Statistical acceptance planning
4
Purpose of acceptance sampling is to avoid 100% inspection.
Accept/reject entire lot based on the results of sample inspection
4
Single sampling plan:
/
N = lot size
/
n = sample size
/
c = acceptance number
/
d = number of defective items in sample
/
p = % of defective items in lot

d c , lot accepted; otherwise rejected and controlled 100%
^
SMEs Quality Assurance Course 17-19 September 2001
Producer’s and consumer’s risk
4
AQL = (Acceptable Quality Level), the percentage of defective items

in the lot that represents a “good quality level”. It should lead
to acceptance most of the times (usually Pa=0.95)
4
LFTD = (Lot Tolerance Fraction-Defective Level), the percentage of

defective items in the lot that represents a “bad quality level”.
It should lead to rejection most of the times (usually Pa=0.10)

4
α = risk for the producer that lots at AQL are rejected probability
4
ß = risk for the consumer that lots at LTFD are accepted
SMEs Quality Assurance Course 17-19 September 2001
Operating Characteristic (OC) Curve
4
A good sampling plan is one that effectively identifies the
lots that can be accepted and those needing 100% inspection.
A tool to support the selection is the Operating Characteristic
(OC) Curve.
Pa
p
1-α
AQL
LTFD
ideal shape
n , c affect
curve shape
ß
SMEs Quality Assurance Course 17-19 September 2001
Double and multiple sampling plans
4
Double sampling
'
number of defective lower limit, accept
Z
number of defective upper limit, reject
·
number of defective between limits, take new sample
J
new accept/reject limits for the 2 samples
4
Multiple sampling
- steps 1, 2, 3, as above
J
continue sampling until accept or reject is achieved
^
^
SMEs Quality Assurance Course 17-19 September 2001
MIL-STD-105
4
Widely used for acceptance sampling by attribute since 1943

Rev A 1950, last Rev E 1989

Cancelled Feb. 1995

Replaced by ANSI/ASQC Z1.4-1993
4
Includes simple, double and multiple plans
4
Includes normal, reinforced and reduced plans to deal with lots which
are continuously received. Passing from normal to reinforced the
acceptance number is reduced but sample size remains the same.
Passing from normal to reduced, the sample size is reduced
SMEs Quality Assurance Course 17-19 September 2001
Cost of Quality
<<>>
SMEs Quality Assurance Course 17-19 September 2001
Cost of Quality
COQ = PREVENTION + APPRAISAL + FAILURE COSTS, or

COQ = CONFORMANCE COST + NONCONFORMANCE COST
Quality cost measurements provide guidance to the quality management program much as the cost
accounting system does for general management

SMEs Quality Assurance Course 17-19 September 2001
Cost of Quality: Influence of Quality Policy
Distribution : Inspected-in Quality
0%
10%
20%
30%
40%
50%
60%
Failure Prevention Appraisal
Distribution : Built-in Quality
0%
10%
20%
30%
40%
50%
60%
Failure Prevention Appraisal

Total COQ vary with emphasis placed on prevention and appraisal.
TOTAL COQ = 25-40% TOTAL COST TOTAL COQ = 5-10% TOTAL COST
SMEs Quality Assurance Course 17-19 September 2001
Cost of Quality: prevention costs

Prevention cost is the cost allocated to prevent the occurrence or
recurrence of nonconforming products:

- Quality planning
- Steps to ensure producible/robust design
- Metrology and calibration
- System audits
- Process capability evaluations
- Process qualifications
- Training
- Quality improvement projects
SMEs Quality Assurance Course 17-19 September 2001
Cost of Quality: appraisal costs

Appraisal cost is the cost of judging the acceptability of products and
to detect nonconformances. Appraisal costs are those associated with
measuring, evaluating, or audit:
- Design reviews

- Incoming and source inspections
- In-process and final inspection/test

- Product and process audits
SMEs Quality Assurance Course 17-19 September 2001
Cost of Quality: failure costs

Failure costs are costs resulting from nonconforming products. They
are divided into internal costs incurred prior to product delivery, and
external costs incurred after product delivery:

- Partial or full loss of the effort already expended

- Additional efforts to make the product acceptable

- Processing of customer reports

- Warranty claims

- Investigating causes

- Determining corrective actions
SMEs Quality Assurance Course 17-19 September 2001
Cost of Quality: failure costs-scrap
4
Cost of additional material to replace unusable material
4
Labor cost for manufacturing of replacement material
4
Labor cost for non-manufacturing personnel for actions coordination
4
Productivity loss due to rescheduling of manufacturing activities
4
Storage cost for extra inventory
SMEs Quality Assurance Course 17-19 September 2001
Cost of Quality: failure costs-rework/repair
4
Labor costs for manufacturing personnel performing rework/repair
4
Labor cost for non-manufacturing personnel for actions coordination
4
Productivity loss
4
Costs of materials involved in repairing
4
Costs of extra handling/transportation
4
Cost of extra rework/repair inspection
4
Administrative costs of waivers processing (for repairs)
SMEs Quality Assurance Course 17-19 September 2001
New model of optimum quality costs
NEW OLD
Source : “Quality”, D. Summers, Prentice Hall
SMEs Quality Assurance Course 17-19 September 2001
Hidden quality costs

There is a “multiplier effect” between failure costs and “true failure
costs”. Failure costs are the tip of an “iceberg”.
Source : “Quality”, D. Summers, Prentice Hall
SMEs Quality Assurance Course 17-19 September 2001
Alert systems
by
P. Secchi
<<>>
SMEs Quality Assurance Course 17-19 September 2001
Alert systems
4
What is an Alert?
4
Types of Alert systems
4
GIDEP
4
NASA Part Advisories
4
NASDA Alert System
4
CNES Groupe d’Alertes
4
ESA Alert System
SMEs Quality Assurance Course 17-19 September 2001
What is an alert?

Alert

Alert is a report used to provide a prompt warning on failures and
problems which may affect more than one user, or may re-occur in
other projects, if no preventive actions are taken
Alert report

The alert report will include a description of the observed problem,
its cause, the actions to be taken to correct it and to prevent
recurrence, comments from the manufacturer
SMEs Quality Assurance Course 17-19 September 2001
Types of Alert Systems
4
Government systems
4
Industry systems
astrium astrium
Parts Advisory
Official Business
U.S. Government
SMEs Quality Assurance Course 17-19 September 2001
GIDEP
Government-Industry Data Exchange Program
4
A co-operative activity between U.S. Government and industry
participants, managed and funded by the U.S. Government
4
Participating organisations:

US Government Agencies and Organisations (including NASA)

Industrial organisations producing parts, components and
equipment for the government
SMEs Quality Assurance Course 17-19 September 2001
GIDEP (cont’d)
GIDEP participants are provided electronic access to six major types data:
4
Engineering data
4
Failure experience data, including ALERTs, SAFE-ALERTs,
Problem Advisories and Agency Action Notices
4
Reliability and Maintainability data
4
Metrology data
4
Product information data
4
Urgent data request
SMEs Quality Assurance Course 17-19 September 2001
NASA Parts Advisories

Intentionally called “Advisory” to distinguish them from GIDEP Alerts and
TWX Alerts from other NASA Centers

Original intention was a rapid “heads-up” warning vehicle

Provides information and guidance but is not intended to require mandatory
action – emphasis on flexibility

To inform NASA of a significant risk, not to criticize a manufacturer

First NASA Part Advisory is dated 08/29/90
Parts Advisory
Official Business
U.S. Government
SMEs Quality Assurance Course 17-19 September 2001
NASDA Alert System
4
The NASDA Alert system has been being established since 1974.
4
Is operated by the Safety and Reliability Department (SRD) of
NASDA.
4
NASDA Alerts are called Reliability Technical Information(RTI).
4
An RTI describes:

observed failure/problem and its root cause

measures to be taken to correct it and to prevent its recurrence
4
Over 100 RTIs have been issued so far.
SMEs Quality Assurance Course 17-19 September 2001
CNES Groupe d’Alertes
CNES Groupe d’Alertes
4
Started in 1989
4
A Co-operation between CNES and French Space Industry
4
Group meets every three months
4
Report prepared and issued every three months
4
“Anomalies” classified into three levels
4
ESA receives CNES Alert Reports
4
CNES is a registered user of the ESA Alert System
SMEs Quality Assurance Course 17-19 September 2001
ESA Alert System

Operated by the ESTEQ TOS-QQ Division to:
4
Facilitate exchange of information on problems experienced in ESA
projects
4
Eliminate or minimise their impacts
4
Prevent recurrence on other projects
4
Enhance competitiveness of European space industry by avoiding
waste and mistakes
SMEs Quality Assurance Course 17-19 September 2001
ESA Alert System (cont’d)

ESA Alerts cover failure related to:
4
EEE parts
4
mechanical parts
4
pyrotechnic devices
4
materials
4
test equipment
4
software used by several users
4
equipment procured against a supplier’s specification
4
commercial, aviation & military (CAM/COTS) equipment
SMEs Quality Assurance Course 17-19 September 2001
ESA Alert System (cont’d)

ESA Alert System covers also problems with:
4
safety
4
manufacturing processes
4
handling procedures
4
standard test methods
4
standard operational procedures
4
software development & test methods and tools
4
continuity of production of an item
SMEs Quality Assurance Course 17-19 September 2001
Criteria to issue an ESA Alert

ESA Alerts are only issued when:
4
the observed problem may apply to more than one project or
organisation and
4
the problem was observed while the item was applied within its
specified limits and
4
a preliminary investigation has provided evidence of the root cause of
the problem and
4
the problem is confirmed not to be of a random nature
SMEs Quality Assurance Course 17-19 September 2001
Published ESA Alerts

ESA has published a total of 42 Alerts so far:
4
30 on EEE parts
4
5 on materials
4
3 on mechanical parts or mechanisms
4
1 on manufacturing processes
4
1 on test equipment
4
1 on flight equipment
4
2 on software product
SMEs Quality Assurance Course 17-19 September 2001
ESA Alerts processing

.
YES
Send PAI to
manufacturer
Fill PAI
Alert?
Designate
experts
Evaluate PAI
Close case
Evaluate
reply
Alert
confirmed?
Publish
Alert
No
N
O
YES
SMEs Quality Assurance Course 17-19 September 2001
ESA Alerts distribution
ESA Projects
ESA technical
departments
ESA contractors
SMEs Quality Assurance Course 17-19 September 2001
ESA Alert focal point
4
Co-ordinate Alert preliminary investigation
4
Organise & support proceedings of ESA Alert Committee
4
Publish Alerts on the web
4
Follow-up of actions to be implemented by manufacturer
4
Register new Users to the System
SMEs Quality Assurance Course 17-19 September 2001
Alert co-ordinators

Alert co-ordinators are the interfaces between company and ESA.
Large companies with different geographical locations have more than
one Alert co-ordinator

Alert Co-ordinator Alert Co-ordinator
ESA Alert Focal ESA Alert Focal
Point Point
PAIs PAIs
feedback/comments feedback/comments
Alerts Alerts
SMEs Quality Assurance Course 17-19 September 2001
Technical experts role
4
Review Preliminary Alert Information
4
Assess failure/problem against criteria for issuing an alert
4
Define recommended actions to:

to solve failure/ problem

to prevent recurrence
4
Assess:

Manufacturer’s response

Alert corrective actions

Feedback from users
4
Recommend whether to issue Alert or not
SMEs Quality Assurance Course 17-19 September 2001
ESA Alert System Web
Access restricted to registered participants only. It allows to:
4
browse ESA Alerts
4
search ESA Alerts
4
request Access
4
submit PAI on-line
4
submit feedback/comments on-line
Have a look at: www.estec.esa.nl/qq/alerts Have a look at: www.estec.esa.nl/qq/alerts
SMEs Quality Assurance Course 17-19 September 2001
Quality Assurance
Part IV : S/W Product Assurance
by
J. Carranza
SMEs Quality Assurance Course 17-19 September 2001
S/W PA basic concepts
<<>>
SMEs Quality Assurance Course 17-19 September 2001
QA versus SW QA
4
There is and should not be separation of the two
4
SW QA is an aspect or a branch of QA
4
Techniques used in SW QA are the same as those in QA
4
Emphasis on techniques and methods is different in SW because of
differences between essential properties of SW and HW
4
However there is nothing special about SW QA only differences
SMEs Quality Assurance Course 17-19 September 2001
Properties of SW
4
SW is highly volatile and can be changed completely at the touch of a
button
4
All faults in SW are systematic. Therefore:

statistical control, etc. cannot be applied

statistical measurements are applicable in a limited fashion only
4
Often there is no immediate direct relation between the apparent
importance of the fault and the importance of the failure that it may
produce
SMEs Quality Assurance Course 17-19 September 2001
Characteristics of SW engineering
4
Its implantation as an engineering discipline is weaker than in other
engineering disciplines due to:
– its relative “youth” as an engineering discipline compared to others
– the very quick pace at which technology, techniques and methods evolve
⇒need to adapt very quickly, lack of consolidation
4
It is difficult to separate new from old: everything from dinosaurs to primates
are alive and living together
4
It is difficult to separate good from bad:
– lack of historical information (experience) from
technologies/methods/tools
– high commercial pressure (rapidly evolving and aggressive tools market)
SMEs Quality Assurance Course 17-19 September 2001
Key issues in SW QA
4
Configuration management:

identification

baseline

change control

traceability
4
Verification:

Not only testing ⇒many more techniques and methods

At all stages of development from requirements to operation
4
Metrics
4
Criticality
4
Evolution of quality assurance concepts
SMEs Quality Assurance Course 17-19 September 2001
Configuration management for S/W
4
Configuration management in SW is the same as in other disciplines:

Configuration identification: how to identify items that are part of
the project

Configuration control: how to change the configuration

Status accounting: what is the state of the items
4
The following will describe some aspects of the above that are of
special importance in SW within the above fundamentals
SMEs Quality Assurance Course 17-19 September 2001
CM: identification
4
Volatility of SW allows very easy creation, deletion or modification of
SW items
4
Volatility makes it important to know which items are part of the
system and which are not
4
It is all too easy to become confused about what is part or not of the
SW development project or what is valid or not
4
SW systems are composed of many custom made components:
complexity and lack of standardisation
4
Configuration Item identification must be planned before starting to
create units
SMEs Quality Assurance Course 17-19 September 2001
CM: baseline
4
Although modern SW technology tries to reduce interdependencies
between components, SW systems are very tightly linked systems
4
Consistency is critical: change in A requires change in B otherwise the
system will not operate properly
4
Baselines are consistent and complete sets of all items that make up
the system and allow co-ordination across the team
4
Baselines are made accessible as libraries or repositories
SMEs Quality Assurance Course 17-19 September 2001
CM: baseline (cont’d)
Design,
specification verification
Specifications,
management
documentation
Software
items,
design items,
test items,
user
documentation
Baselines grow along lifecycle as the project advances, for example:
SMEs Quality Assurance Course 17-19 September 2001
CM: baseline evolution
4
Baselines are created more often for SW than for systems
4
Baselines must be planned and must have a clear purpose
4
Specially important baselines are:

inputs to formal reviews

outputs from formal reviews (often ignored)

test baselines

deliveries
SMEs Quality Assurance Course 17-19 September 2001
CM: change control
4
Complexity of SW makes it difficult to assess the impact of changes in
one part of the system on another
4
Individuals in a team many not be able to judge that impact
4
Change control is a mechanism to allow a baseline to change into
another
4
Key concepts:

authority to change (often not the engineer)

entry of a configuration item under configuration control

check points in change implementation

all of the above adequately sized for the project
SMEs Quality Assurance Course 17-19 September 2001
CM: change control (cont’d)
4
Change control is triggered by non-conformances (Software Problem
Reports) and change requests
4
SW changes much faster than HW ⇒ efficient change control
procedure, normally separate from HW procedure
4
In system’s development SW change control will need formal
interfaces to System’s Non-conformance handling and System’s
Change Requests
4
One key point: once a configuration item is under change control it is
not the “property” of the author any more
SMEs Quality Assurance Course 17-19 September 2001
CM: importance of traceability
4
Baselines are not formed only by SW components. They include also:

specification (requirements)

design

management documentation

verification items (verification records, testing tools, etc.)

user documentation, etc.
4
Traceability is a fundamental tool to ensure completeness and also
consistency of baselines
SMEs Quality Assurance Course 17-19 September 2001
CM: importance of traceability (cont’d)
4
Basic traceability:
Integration
test items
System test
items
Unit test items
Design
components
SW items Requirements
SMEs Quality Assurance Course 17-19 September 2001
Verification: definitions

Verification

Verification: activity to ensure that the outputs of a process conform to
its inputs (completeness, consistency, correctness)

Validation

Validation: activity to ensure that a product of services is adequate for
its intended use (use related)
SMEs Quality Assurance Course 17-19 September 2001
Verification: phases
4
Verification is not limited to testing
4
Verification takes place at each phase of the software lifecycle:

requirements specification: formal review, etc.

design: walkthrough, review, modelling, etc.

coding: code review, inspection, unit testing, etc.

operation: non-regression testing, etc.
4
Most of the above verification activities are supported by traceability
(if requirement x changes, how do I test the SW change?)
SMEs Quality Assurance Course 17-19 September 2001
Verification: scope
4
Verification covers more than just the SW
4
Items that should or may be verified are:

SW itself

implementation of processes

tools

infrastructure
SMEs Quality Assurance Course 17-19 September 2001
Verification: techniques
4
Different methods are used in SW
4
The main ones are:

Testing

Formal review

Inspection, walkthrough

Audit
4
Verification techniques may be intended for products (e.g. testing),
processes (maturity assessments) or both (formal reviews)
SMEs Quality Assurance Course 17-19 September 2001
Testing: levels
4
Testing is performed at different levels
4
Levels of testing have different targets and objectives:
Level Target
Unit Source code units
Integration Interfaces
System Software Specifications
Acceptance User or system specifications
4
The different levels may have different names or be organised in
slightly different ways
SMEs Quality Assurance Course 17-19 September 2001
Testing: types
4
Testing can be aimed to test implementation, with specific knowledge
of the internal logic ⇒white box testing
4
Testing can be aimed to test functionality at different levels, ignoring
the internal logic ⇒ black box testing
4
Black box testing can be of different types also:

nominal: test for normal conditions

back to back: same test on two or more versions of SW written
with same specifications

interface: written on the basis of Interface Control Documents

stress: test under extreme or abnormal conditions

statistical: test data selected according to distribution functions
SMEs Quality Assurance Course 17-19 September 2001
Testing: limits
4
Deciding when to stop testing (other than running out of time) is not
easy
4
Goals must be set beforehand to ensure sufficient and uniform testing
4
Goals for white box testing are typically based on code coverage
(statement, branch, path, etc.)
4
Goals for black box testing will depend on its objective (e.g. software
requirements, architectural design, etc.) and specific requirements (e.g.
test specific non-nominal conditions)
SMEs Quality Assurance Course 17-19 September 2001
Testing: verification
4
Testing itself is subject to verification
4
This will typically involve specific reviews of the test plan and test
designs, cases and procedures
4
It may involve additional verification: e.g. mutation testing
4
Coverage and status of the testing must be controlled formally:

Verification Control Document for requirements or design
components

Coverage reports for white box testing
SMEs Quality Assurance Course 17-19 September 2001
Testing: baselines change
4
Baselines are crucially important in SW testing
4
Any level or type of testing start from a properly controlled baseline ⇒
baseline identification (contents and versions)
4
Testing is performed in well defined cycles, each cycle using a single
baseline
4
Problems found during testing are processed through the change
control procedure
4
Implementation of changes leads to a new baseline which is input to
the next cycle of testing
4
Do not give in to the temptation of changing along the way
SMEs Quality Assurance Course 17-19 September 2001
Testing: baselines change (cont’d)
4
Changes during testing are localised ⇒testing should be adapted to the
change ⇒non-regression testing
4
Non-regression testing should be performed from the lowest level up
to the level at which the corresponding fault was found
4
Traceability is an invaluable tool to define non-regression testing (e.g.
which units do I test if a test for software requirement x failed)
4
Dependencies between components, etc. must be taken into account
4
Automate test where possible
SMEs Quality Assurance Course 17-19 September 2001
Verification: other techniques
4
Other verification techniques are very similar to general verification
techniques:

formal reviews: sometimes SW integrated in system reviews

audits

inspections: specific methods for SW, e.g. Fagan

internal or informal reviews: very important in SW, I cannot see
my own mistakes
SMEs Quality Assurance Course 17-19 September 2001
Metrics
4
Metrics provide quantitative information on products or processes
4
Metrics can be used for:

project purposes: controlling the development processes

organisation purposes: historical data
4
Project metrics must have goals and associated reaction mechanisms
to react to goals not being reached
4
An example:

metric: cyclomatic complexity

goal: < 10

reaction: re-design module
SMEs Quality Assurance Course 17-19 September 2001
Project Metrics
4
As important and metrics collection is reporting ⇒metrics reports
must be addressed to the people who must initiate the reaction
4
Metrics collection and reporting should be supported by tools as much
as possible
4
Metrics should be defined on the basis of the most relevant properties
of the specific SW being developed: Portability? Maintainability?
Efficiency? Reliability?
4
Quality models are the mechanism to define a consistent and useful
metrics program (e.g. ISO 9126, SPEC)
SMEs Quality Assurance Course 17-19 September 2001
Organisation metrics
4
Their objective goes beyond the project ⇒no immediate apparent
benefit for the project
4
They rely on an organisation at a higher level than the project ⇒
metrics from different project must be collected and analysed
4
They have to be highly independent of technology (e.g. design
complexity metrics from different models may not be comparable)
4
They are only useful in the long term
SMEs Quality Assurance Course 17-19 September 2001
Some metrics examples
4
Product metrics:

code size: number of lines of code

design size: number of design components

code complexity: cyclomatic complexity

design complexity: depth of inheritance tree
4
Process metrics:

actual effort against estimated effort

problems detected during system testing

rate of problems reported during operation
SMEs Quality Assurance Course 17-19 September 2001
Criticality: issues
4
A typical SW system will perform different functions some of which
may be more critical (safety, mission, etc.) than others
4
SW designs tend to be more and more complex every day
4
It is not obvious how relevant different SW components are to the
performance of the more critical functions
4
Availability or resources is limited
4
It is better to concentrate efforts on the most important functions ⇒
most important components
SMEs Quality Assurance Course 17-19 September 2001
Criticality: system and SW
4
Criticality of SW integrated in a system is always dependent on
criticality of the system functions
4
The criticality of the functions at system level is analysed and the
results of those analyses is fed into parallel analyses at SW level
4
There is feedback from SW to system level analyses including the
analysis of HW and SW interaction
4
Criticality of functions is translated into criticality of software
components: traceability again!!
4
Concentrate resources (e.g. verification) on most critical components
SMEs Quality Assurance Course 17-19 September 2001
Continuous
process
improvement
Quality system evolution: the S/W perspective
Artisan



Team
Defined
methods

Organization Quality
system
Best
practices

Standards
SMEs Quality Assurance Course 17-19 September 2001
The evolution: s/w “artisan”
4
Individuals working independently even within a team are still very
common
4
Often particularly able “artisans” will compensate resources or skills
missing in the organisation
4
“Artisans” carry with them essential skills an experience that is often
impossible to replace
SMEs Quality Assurance Course 17-19 September 2001
The evolution: s/w team
4
Most SW development projects require a team effort nowadays
4
Teams are often formed by “artisans”
4
Teams rely heavily on specific individuals, in particular the team
leader
SMEs Quality Assurance Course 17-19 September 2001
The evolution: defined methods
4
Nowadays established methods and tools exist for all activities
involved in SW development
4
For example object orientated methods, modelling tools, code
generation tools, maintenance tools, etc., etc., etc.
4
Proper use of those methods brings uniformity into SW development
and allows communication across the team
4
Build up of expertise in the use of methods ⇒ often it is more
important expertise in a method than how “good” or “new” it is: e.g.
many projects keep stumbling on their initial attempts at object
orientation
SMEs Quality Assurance Course 17-19 September 2001
The evolution: organisation
4
Organisations are based on roles, not specific individuals
4
This gives them a higher independence on staff turnover
4
Roles allow separation of skills which facilities training
4
Roles and individuals can be mapped flexibly.
This flexibility makes use of resources and skills more efficient
SMEs Quality Assurance Course 17-19 September 2001
The evolution: QA system
4
The QA system provides the framework that defines the roles,
processes and methods used in the organisation
4
In the case of SW development this is at least as important as in other
engineering disciplines
4
A generic QA system may not be enough for SW ⇒a certain level of
detailing and adaptation may be needed for SW
SMEs Quality Assurance Course 17-19 September 2001
The evolution: best practices and standards
4
Standards provide a common “vocabulary” for SW engineering (e.g.
ISO 12207)
4
They can be incorporated into the conception of the QA system
allowing to map the organisation’s processes and methods directly to
the standard allowing for automatic compliance
4
Problem: “The nice thing about standards is that there are so many of
them to choose from” ⇔ multiple customers/standards
4
Standards are often a channel to promote best practices across industry
SMEs Quality Assurance Course 17-19 September 2001
The evolution: best practices and standards (cont’d)
4
In the case of some standards it is possible to obtain a certification
4
The most common one is ISO 9001 under a number of SW or IT
specific schemes
4
TickIT is one example:

supported by UK and Swedish software industry

recognised by British government

provides assessment and certification services but also guidance
4
ISO 9000-3: guidance on applying ISO 9001
SMEs Quality Assurance Course 17-19 September 2001
The evolution: software process improvement
4
Best practices of today may not be so tomorrow
4
Methods, processes and their performance are always susceptible of
improvement
4
This improvement increases efficiency, maturity, reliability and
competitiveness
4
To improve a process or method it is necessary to be able to compare
current and past performance ⇒ quantitative control (e.g. organisation
metrics)
4
Reliable improvement must be based on historical data
SMEs Quality Assurance Course 17-19 September 2001
The evolution: software process improvement
(cont’d)
4
Software Process Improvement must be institutionalised and an
organisation-wide process in itself
4
SPI must be supported by adequate organisation and resources
4
SPI must be managed
4
SPI must be promoted by higher management
4
SPI has a cost and benefits
SMEs Quality Assurance Course 17-19 September 2001
Major changes of focus
4
From individual to team
4
From individual to role
4
From product to process but at the same time from quality assurance
to product assurance
4
However examples of all of the above steps of evolution co-exist
nowadays in the market (dinosaurs and primates) ⇒ difficulty in
differentiating them
SMEs Quality Assurance Course 17-19 September 2001
Space S/W standards
<<>>
SMEs Quality Assurance Course 17-19 September 2001
Old ESA SW standards
4
Previously ESA used the PSS series of standards
4
Two documents in the series were the most relevant to software:

ESA-PSS-05-0: ESA Software Engineering Standards, issue 2,
February 1991

ESA-PSS-01-21: Software Product Assurance Requirements for
ESA space systems, issue 2, April 1991
4
ESA-PSS-05-0 is complemented by a set of guides, ESA -PSS-05-x
4
They consider software only (no system)
SMEs Quality Assurance Course 17-19 September 2001
New ECSS for SW
4
The main documents are:

ECSS-E-40A, Space Engineering - Software, 13 April 1999

ECSS-Q-80B, Space Product Assurance - Software Product
Assurance, 19 April 1996
4
New draft versions are available and are the ones used at this time,
although pending formal publication:

ECSS-E-40B draft, Space Engineering - Software, 3 April 2000

ECSS-Q-80B draft, Space Product Assurance - Software Product
Assurance, 3 April 2000
4
The new versions resolve a number of gaps between the previous
versions and adapt to evolution of SW
SMEs Quality Assurance Course 17-19 September 2001
Alignment with international standards
4
The new versions of E-40 and Q-80 are based or aligned with a
number of key international SW standards:

ISO/IEC 12207:1995 Information Technology - Software lifecycle
processes

ISO/IEC 15504:1998 Information Technology - Software process
assessment

ISO 9126 Information Technology - Software product evaluation -
quality characteristics and guidelines for their use

IEEE 612.10 - 1990 IEEE Standard glossary of software
engineering terminology

IEEE 1062 - 1993 IEEE Standard recommended practices for
software acquisition
SMEs Quality Assurance Course 17-19 September 2001
Alignment with international standards (cont’d)
4
The alignment of E-40 and Q-80 with international standards makes
them:

internationally “readable” since they use recognised vocabulary
and models

simultaneous compliance with other standards easier

easier evolution in parallel with those standards
SMEs Quality Assurance Course 17-19 September 2001
SW ECSS: characteristics
4
The standards are based on the relation between SW and the system of
which it forms part
4
The standards are based on the concept of process as a set of activities
that contribute to the development lifecycle: e.g. configuration
management or design
4
The standards are based on a customer-supplier model
4
The model is recursive: the supplier may be the customer of next level
suppliers
4
The standards do not duplicate basic information which is general to
the system, references are made instead to other standards: e.g. project
management
SMEs Quality Assurance Course 17-19 September 2001
Documentation
4
Differently from ESA-PSS-05-0 documents are not dictated
4
ECSS refers to six collections of information:

Requirements baseline (E-40)

Technical specification (E-40/Q-80)

Design definition file (E-40)

Design justification file (E-40/Q-80)

Product assurance file (Q-80)

Maintenance file (Q-80)
SMEs Quality Assurance Course 17-19 September 2001
Documentation (cont’d)
4
These collections contain all the work products of the project like
plans, requirements document, design documents, technical
documentation, reports, etc.
4
The standards do not define tables of contents for documents either
SMEs Quality Assurance Course 17-19 September 2001
SW processes
SMEs Quality Assurance Course 17-19 September 2001
SW engineering
4
Life-cycles are not enforced and are mapped into:

system engineering

requirements engineering

design engineering

validation and acceptance
4
The above engineering processes may overlap to some extent
4
Major reviews mark the end of each of them: System Requirements
Review, Preliminary Design Review, Critical Design Review and
Acceptance Review
SMEs Quality Assurance Course 17-19 September 2001
ECSS-E-40 requirements
4
Requirements are established for the following processes:

system engineering processes related to SW

software requirements engineering

software design engineering

software validation and acceptance

software operations engineering

software maintenance

software verification and validation (supporting process)
SMEs Quality Assurance Course 17-19 September 2001
ECSS-E-40 requirements (cont’d)
4
Additional requirements exist for specific types of SW or
development:

space segment

ground segment

SW reuse

man-machine interfaces

critical SW
SMEs Quality Assurance Course 17-19 September 2001
ECSS-Q-80 requirements
4
Programmatic requirements on SW product assurance:
– organisation
– responsibilities
– planning
– management
4
Software process assurance:
– lifecycle
– software development processes
4
Software product assurance:
– quality objectives and metrication
– product quality requirements
SMEs Quality Assurance Course 17-19 September 2001
ECSS-E-40 example
Clause 5.4.3.4 The supplier shall develop and document a
top-level design for the interfaces external to
the software item and between the software
components of the software item.
EXPECTED OUTPUT:
a. Preliminary (top-level) external interfaces
design [ICD(TS); PDR]
b. Preliminary (top-level) internal interfaces
design [DDF; PDR]
SMEs Quality Assurance Course 17-19 September 2001
ECSS-Q-80 example
Clause 5.3.1.5 The supplier shall provide with his software
assurance plan a compliance matrix
documenting his compliance with the software
product assurance requirements applicable for
the project/contract.
EXPECTED OUTPUT:
Compliance matrix [PAF; SRR]
SMEs Quality Assurance Course 17-19 September 2001
S/W Process Capability Assessment
<<>>
SMEs Quality Assurance Course 17-19 September 2001
Software Process Assessment
4
Quality of software is directly related to the quality of the
development processes carried out to produce the software
4
Measuring the capability of the processes carried out by an
organisation to develop software provides key information on the
ability of that organisation to produce good quality software
consistently
4
SPiCE for Space (S4S) is a framework that allows measuring the
capability of software development processes
SMEs Quality Assurance Course 17-19 September 2001
What is S4S?
4
A method of evaluating space software development processes
4
ISO 15504 (SPiCE) conformant
4
Developed by ESA for the European space industry
4
Can be tailored to different classes of critical software
SMEs Quality Assurance Course 17-19 September 2001
ESA’s Goals
4
Encourage production of the best possible software products and
services
4
Develop customer-supplier relationships based on trust, not control
4
Promote and disseminate best practice concepts proven across the
software industry
4
Widen supplier base to companies traditionally outside of the
aerospace domain
SMEs Quality Assurance Course 17-19 September 2001
S4S purposes
S4S may be used in three defined modes:
4
Evaluate the maturity of current software processes within an
organisation (Capability Determination Mode)
4
Provide a solid basis for improving processes (Process Improvement
Mode)
4
Determine capability to comply with (European) space standards
(ECSS Compliance Mode)
SMEs Quality Assurance Course 17-19 September 2001
Inputs to S4S
4
ECSS Management, Engineering and Quality Standards for Space
Software
4
ISO 15504 (SPiCE): Software Process Assessment
ISO
15504
SMEs Quality Assurance Course 17-19 September 2001
What is ISO 15504?
4
International standard for assessing software processes
4
Developed in parallel with other software engineering standards (ex.
ISO 12207)
4
Scope

Comprehensive
»
Processes include acquisition, supply, development, operation,
maintenance and support

Modular
»
Can select which processes to assess
»
Each process is assessed on a scale of capability
SMEs Quality Assurance Course 17-19 September 2001
SPiCE : The Assessment Model
4
Two-dimensional model for processes
and process capability

Process Dimension
»
Process Categories
»
Processes (P1, …, Pn)

Capability Dimension
»
Capability Levels (CL1, …, CL5)
»
Process Capability Attributes
4
Each process receives a capability level
rating
CL5
CL4
CL3
CL2
CL1
CL0
CUS.1 CUS.2...ORG.6
SMEs Quality Assurance Course 17-19 September 2001
Level 1 Performed
PA.1.1 Process Performance
Level 1 Performed
PA.1.1 Process Performance
Level 2 Managed
PA.2.1 Performance Management
PA.2.2 Work Product Management
Level 2 Managed
PA.2.1 Performance Management
PA.2.2 Work Product Management
Level 3 Established
PA.3.1 Process Definition
PA.3.2 Process Resource
Level 3 Established
PA.3.1 Process Definition
PA.3.2 Process Resource
Level 4 Predictable
PA.4.1 Measurement
PA.4.2 Process Control
Level 4 Predictable
PA.4.1 Measurement
PA.4.2 Process Control
Level 5 Optimizing
PA.5.1 Process Change
PA.5.2 Continuous Improvement
Level 5 Optimizing
PA.5.1 Process Change
PA.5.2 Continuous Improvement
Incomplete
Performance and results are
incomplete, chaotic processes
Performed
processes are intuitively performed, input
and
output work products are available
Predictable
Metrics make process performance
and results controllable
Optimising
Quantitative measures used for continuous
improvement process
Established
Predefined processes are tailored
for
specific use, resources are
managed..
Level 0 Incomplete
Level 0 Incomplete
Managed
Process and work products are
managed, responsibilities
identified.
SMEs Quality Assurance Course 17-19 September 2001
S4S: The Process Dimension
Customer-Supplier
CUS.1 Acquisition
CUS.1.1 Acquisition Preparation
CUS.1.2 Supplier Selection
CUS.1.3 Supplier Monitoring
CUS.1.4 Customer Acceptance
CUS.2 Supply
CUS.2.1 Supply Preparation
CUS.2.2 Delivery
CUS.3 Requirements Elicitation
CUS.4 Operation
CUS.4.1 Operational Use
CUS.4.2 Customer Support
CUS.5 Contract Maintenance
Customer-Supplier
CUS.1 Acquisition
CUS.1.1 Acquisition Preparation
CUS.1.2 Supplier Selection
CUS.1.3 Supplier Monitoring
CUS.1.4 Customer Acceptance
CUS.2 Supply
CUS.2.1 Supply Preparation
CUS.2.2 Delivery
CUS.3 Requirements Elicitation
CUS.4 Operation
CUS.4.1 Operational Use
CUS.4.2 Customer Support
CUS.5 Contract Maintenance
Support
SUP.1 Documentation
SUP.2 Configuration
Management
SUP.3 Quality Assurance
SUP.4 Verification
SUP.5 Validation
SUP.6 Joint Reviews
SUP.7 Audit
SUP.8 Problem Resolution
SUP.9 Safety and Dependability
SUP.10 Independent Software
Verification and Validation
Support
SUP.1 Documentation
SUP.2 Configuration
Management
SUP.3 Quality Assurance
SUP.4 Verification
SUP.5 Validation
SUP.6 Joint Reviews
SUP.7 Audit
SUP.8 Problem Resolution
SUP.9 Safety and Dependability
SUP.10 Independent Software
Verification and Validation
Engineering
ENG.1 Development
ENG.1.1 System Requirements Analysis
& Design
ENG.1.2 Software Requirements Analysis
ENG.1.3 Software Design
ENG.1.4 Software Construction
ENG.1.5 Software Integration
ENG.1.6 Software Testing
ENG.1.7 System Integration & Testing
ENG.2 System & Software
Maintenance
Engineering
ENG.1 Development
ENG.1.1 System Requirements Analysis
& Design
ENG.1.2 Software Requirements Analysis
ENG.1.3 Software Design
ENG.1.4 Software Construction
ENG.1.5 Software Integration
ENG.1.6 Software Testing
ENG.1.7 System Integration & Testing
ENG.2 System & Software
Maintenance
Management
MAN.1 Management
MAN.2 Project Manangement
MAN.3 Quality Management
MAN.4 Risk Management
MAN.5 Information Management
Management
MAN.1 Management
MAN.2 Project Manangement
MAN.3 Quality Management
MAN.4 Risk Management
MAN.5 Information Management
Organisation
ORG.1 Organisational Alignment
ORG.2 Improvement
ORG.2.1 Process Establishment
ORG.2.2 Process Assessment
ORG.2.3 Process Improvement
ORG.3 Human Resource Management
ORG.4 Infrastructure
ORG.5 Measurement
ORG.6 Reuse
Organisation
ORG.1 Organisational Alignment
ORG.2 Improvement
ORG.2.1 Process Establishment
ORG.2.2 Process Assessment
ORG.2.3 Process Improvement
ORG.3 Human Resource Management
ORG.4 Infrastructure
ORG.5 Measurement
ORG.6 Reuse
SMEs Quality Assurance Course 17-19 September 2001
Example Assessment Results
SMEs Quality Assurance Course 17-19 September 2001
Differences between S4S and ISO 15504
In comparison with SPiCE, S4S has:
4
4 New Processes
– CUS.5 Contract Maintenance
– MAN.5 Information Management
– SUP.9 Safety and Dependability
– SUP.10 Independent Software Verification and Validation
4
2 New Component Processes
CUS 2.0 Supply ⇒CUS.2.1 Supply Preparation + CUS.2.2 Delivery
4
Many lower level elements
(41 New Base Practices, 73 New Notes, 22 New Work Products)
SMEs Quality Assurance Course 17-19 September 2001
Performing a S4S Assessment
Assessment Steps
4
Initiation

define purpose, scope, context of assessment
4
Planning
4
Briefing
4
Data Acquisition

through interviews and document examination
4
Data Validation
4
Process Rating
4
Reporting
A
A
B
B
C
C
SMEs Quality Assurance Course 17-19 September 2001
Typical Capability Levels
ENG.1.2 Software
Requirements
ENG.1.3 Software
Design
SUP.3 Quality
Assurance
MAN.2 Project
Management
MAN.4 Risk
Management
SUP.2 Configuration
Management
Managed
Managed
Performed
Managed
Managed
Managed
SMEs Quality Assurance Course 17-19 September 2001
ISO 15504 Improvement Cycle
Examine
organisation's
needs
Initiate
process
improvement
Perform
process
assessment
S4S S4S
Analyse
results
and
derive
action
plan
Implement
improvement
Confirm the
improvement
Sustain
improvement
gain
Monitor
performance
Re-Assessment
Request
Analysed
assessment
results
Implemented
improvements
Approved
action plan
Validated improvement
results
Institutionalised
improvements
Improvement
initiation
Preliminary
improvement
programme plan Assessment
results
SMEs Quality Assurance Course 17-19 September 2001
Motivations for Process Improvement
Why do you want to improve your processes?
4
We need to improve the quality of our products
4
We want to be better than our competitors
4
We need more efficiency
4

4
My customer has asked me to do so!
SMEs Quality Assurance Course 17-19 September 2001
S4S Risk Architecture
4
S4S-R (S4S-Risk) is an extension of S4S with a new Risk Dimension
integrated in the method
4
S4S-R addresses risks arising from inappropriate process management
4
S4S-R is orientated to the risks involved in processes (from the S4S
model)
SMEs Quality Assurance Course 17-19 September 2001
Main issues
4
Process orientated risks originate from the actual capability of a
process not reaching the required capability
4
Process orientated risks are due to inadequate or insufficient
management of the process performance
4
More than one process may contribute to one risk
SMEs Quality Assurance Course 17-19 September 2001
Elements of S4S-R
4
In S4S-R the gap between the actual and required capability is
analysed in detail (all process attributes)
4
Risks are evaluated and classified according to their importance
(acceptable, unacceptable, intermediate)
4
S4S-R allows to identify actual management practices that need to be
addressed in order to mitigate risks
4
S4S-R differs from process improvement in that it does not try to
reach a target capability but to correct those deficiencies that allow
mitigation of the unacceptable process orientated risks
SMEs Quality Assurance Course 17-19 September 2001
Good luck!

These handouts are released for general instruction purpose only. These handouts are not meant to take any precedence on applicable project requirements, documents and definitions

SMEs Quality Assurance Course

17-19 September 2001

Space insurance claims…lift-off
Billion
$1.4 1.2 1 .8 .6 .4 .2 0 1980 ‘82 ‘84 ‘86 ‘88 ‘90 ‘92 ‘94 ‘96 ‘98 ‘00

Source Space News

SMEs Quality Assurance Course

17-19 September 2001

Space systems failures causes

FAILURE SPACE: A Systems engineering look at 50 Space FAILURE SPACE: A Systems engineering look at 50 Space System Failure-J.S.Newman NASA-Acta Astronautica Vol.48 No 5-12 System Failure-J.S.Newman NASA-Acta Astronautica Vol.48 No 5-12

2000
SMEs Quality Assurance Course

1960
17-19 September 2001

Space systems failures causes (cont’d)

“…if there is any surprise in the synoptic results it is the prominence of manufacturing, production and assembly…a strong cautionary note to those seeking to marginally hold down production costs by cutting quality assurance, inspection, audit…”*

* FAILURE SPACE: A Systems engineering look at 50 Space System Failure J.S.Newman NASA-Acta Astronautica Vol.48 No 5-12, 2001

SMEs Quality Assurance Course

17-19 September 2001

Product Assurance disciplines
x

Quality Assurance Reliability Safety Materials EEE Components Configuration Management
17-19 September 2001

x

x

x

x

x

SMEs Quality Assurance Course

Product Assurance definitions
x

Product Assurance: A discipline devoted to the study, planning and implementation of activities intended to assure that the design, controls, methods and techniques in a project result in a satisfactory level of quality in a product (ECSS-P-001A)

x

Mission Assurance: Mission assurance provides confidence to the project and sponsors that the mission will be successfully completed through its planned support activities, standardised processes, and monitored project milestones. Mission assurance brings a set of specialised disciplines to support the project team environment (NASA)
SMEs Quality Assurance Course 17-19 September 2001

European Product Assurance Space Standards x OLD : ESA Procedures Standards and Specifications (PSS) PSS-01-10 PSS-01-10 Product Assurance Product Assurance PSS-01-20 PSS-01-20 Quality Assurance Quality Assurance PSS-01-30 PSS-01-30 Reliability Assurance Reliability Assurance PSS-01-40 PSS-01-40 System Safety System Safety PSS-01-60 PSS-01-60 EEE Components EEE Components PSS-01-70 PSS-01-70 Materials/Processes Materials/Processes PSS-01-21 PSS-01-21 S/W PA S/W PA PSS-01-11 PSS-01-11 Configuration Mgmt Configuration Mgmt x NEW: European Cooperation for Space Standardisation (ECSS) ECSS-Q-00 ECSS-Q-00 Policy & Principles Policy & Principles ECSS-Q-20 ECSS-Q-20 Quality Assurance Quality Assurance ECSS-Q-30 ECSS-Q-30 Dependabilitry Dependabilitry ECSSS-Q-40 ECSSS-Q-40 Safety Safety ECSS-Q-60 ECSS-Q-60 EEE Components EEE Components ECSS-Q-70 ECSS-Q-70 Materials/ Processes Materials/ Processes ECSS-Q-80 ECSS-Q-80 S/W PA S/W PA SMEs Quality Assurance Course 17-19 September 2001 .

QA Course Outline x PART I : Concepts and Techniques – Introduction – Basic principles and values – Main elements/techniques – QA during major project phases PART II: QA Standards – Standards evolution – ECSS-Q-20A x SMEs Quality Assurance Course 17-19 September 2001 .

QA Course Outline (cont’d) x PART III: Special Techniques – Statistical techniques – Cost of Quality – Alert Systems x PART IV: S/W Product Assurance – Basic concepts – Space S/W standards – S/W Process Capability Assessment SMEs Quality Assurance Course 17-19 September 2001 .

Quality Assurance Part I : Concepts and Techniques SMEs Quality Assurance Course 17-19 September 2001 .

Introduction <<>> SMEs Quality Assurance Course 17-19 September 2001 .

It is also defined as: – Quality of design – Quality of conformance – Quality of performance x SMEs Quality Assurance Course 17-19 September 2001 . quality can have two meanings: – (1) the characteristics of a product or service that bear on its ability to satisfy stated or implied needs – (2) a product or service free of deficiencies.What is Quality? x In technical usage.

Quality of Conformance Traditional QA deals mainly with means and practices to achieve and ensure achievement of: QUALITY OF CONFORMANCE: Quality of conformance relates to the fidelity with which the product conforms to the design. SMEs Quality Assurance Course 17-19 September 2001 .

because it may translate directly into increased profits and higher production capacity (Hidden Plant) SMEs Quality Assurance Course 17-19 September 2001 . Juran to mining gold (Gold in the Mine).M.Does Quality cost? Quality of Conformance is profitable not costly POOR QUALITY IS COSTLY! Companies that measure their “poor quality”costs for the first time are usually shocked at what they find Reducing “poor quality ” costs has been compared by J.

What is a quality system? Product quality depends on many variables. resources and procedures that manufacturers and suppliers use to control these variables to produce a product of consistent quality which meets defined specifications is called a QUALITY SYSTEM* * manufacturing environment case Q U A L I T Y S Y S T E P E R S MO AN T N E E M R L EI A A DL N O S C U M E SMEs Quality Assurance Course 17-19 September 2001 . organisation. The processes.

identification. subassembly. identification] MEANS: tools. equipment and facilities required to perform the task [Type.The four QS controlled elements x PERSONNEL: to perform or supervise the production tasks [Competence. capability. orders. control. training. condition. quantity. drawings. records. location. condition. condition. etc. environment] DOCUMENTATION: input/output (specifications. distribution] x x x SMEs Quality Assurance Course 17-19 September 2001 . identification.)[Type. motivation. reports. identification. attributes] MATERIAL: on which to apply production processes (component. capability. etc.) [Content.

which enforce QA rules only when required by the customer.Quality system as business strategy There are companies. in particular SMEs. Such companies miss the strategic business value of the Quality System for: Œ  Ž  competitiveness improvement product improvement organisational continuity/growth personnel motivation SMEs Quality Assurance Course 17-19 September 2001 .

Quality System as assurance technique x The ability not only to achieve Quality (meet customer requirements) but to demonstrate that an effective system is in place and maintained for that purpose For the European Union (as for any organisation having regulatory responsibilities for safety). the implementation of a Quality Assurance system by a company is a pre-condition for valid certification assessment of their products compliance to relevant norms Safety and high reliability required by space systems has made QA systems implementation mandatory during any project phase (design. manufacturing and operations) x x SMEs Quality Assurance Course 17-19 September 2001 .

prove verifications SMEs Quality Assurance Course 17-19 September 2001 .write what you do .do as written .prevent mistakes .Quality Assurance QUALITY ASSURANCE METHOD (to obtain) OBJECTIVE (to demonstrate) .learn from mistakes QUALITY .perform verifications .

Quality Assurance (cont’d) “ Quality assurance is the planned and systematic activities implemented within the quality system and demonstrated as needed to provide adequate confidence that an entity will fulfill requirements for quality” (ISO-Vocabulary). SMEs Quality Assurance Course 17-19 September 2001 .

from external quality to internal quality .from product/service improvement to process improvement .from second-party certification to third-party certification SMEs Quality Assurance Course 17-19 September 2001 .Evolution of quality system concepts Artisan Inspection Quality Control Statistical Quality Control Statistical Process Control Best Practices Total Quality Management ISO 9000 Major changes of focus: .

managers and intermediate supervisors-end product checked by independent inspectors (good external quality-poor internal quality-high quality costs) t t SMEs Quality Assurance Course 17-19 September 2001 .From artisan to quality control t small workshop-proprietor knowing everything-customer in touch -proprietor training workmen and checking the end product (good product quality-optimum quality costs) larger workshop-division of work-intermediate merchants-intermediate supervisors-end product checked by supervisors (poor external quality-high quality costs) factories-larger quantities and use of machines-work “atomisation” and quantity incentives.

) (good product quality-optimum quality costs) “Quality control is operational techniques and activities that are used to fulfill requirements for quality. It involves techniques that monitor a process and eliminate causes of unsatisfactory performance at all stages of the quality loop” (ISO-Vocabulary) t SMEs Quality Assurance Course 17-19 September 2001 .From artisan to quality control (cont’d) t evolving factories-long management chain-industrial engineering functions-sales management-ubiquitous independent inspectorsinspector supervisors-Inspection Department (good external qualitygood internal quality-high inspection costs) modern factories-management driving quality-quality planningmarketing functions-preventive quality approach-limited number of inspectors-Quality Control Department (QC engineering etc.

Dodge and H. while lowering the customer’s risk to receive a defective product.Statistical quality control u Statistical Quality Control was initially the application of statistical techniques and economical considerations to acceptance sampling (pioneered by H.Romig of Bell Telephone Labs) The purpose was to avoid the cost of 100% lot inspection. and the manufacturer’s risk to reject a good one u SMEs Quality Assurance Course 17-19 September 2001 .

Shewhart of the Bell Telephone Labs to assess. monitor and control manufacturing processes SPC is applicable to areas in which a large number of parts are being produced or a process is being repeated many times t t t Notes: the term Statistical Quality Control includes nowadays statistical process control and statistical acceptance sampling SPC cannot be applied to software products because all copies of software are exactly identical SMEs Quality Assurance Course 17-19 September 2001 .Statistical process control t Statistical Process Control (SPC) is an analytical tool invented by W.A.

centered on quality. This approach was called Total Quality Management or TQM. and aiming at long-term success through customer satisfaction and benefits to all members of the organization and to society” (ISO-Vocabulary) t SMEs Quality Assurance Course 17-19 September 2001 . based on the participation of all of its members.Total quality management (TQM) t By the 1980s the idea was advancing that quality leadership could be achieved only by applying the entire array of quality tools and techniques to all functions and all levels of a company.(In Japan it is called Company-wide Quality Control) “Total quality management is the management approach of an organization.

TQM: process model for quality x A process is defined as a “set of interrelated or interacting activities which transforms inputs into outputs” C U S T O M E R M anage for R e q u i r e m e n t s Q uality Provide Resources M ea sure A nalyze. Im prove Develop Q uality Input Product or Sevice O utput S a t i s f a c t i o n Product C U S T O M E R NASA-JSC JPD 5335.1C SMEs Quality Assurance Course 17-19 September 2001 .

S.Total quality management (cont’d) t TQM definition is well expressed in practical terms by the “Malcom Baldrige National Quality Award” criteria published by the U. National Institute for Standards and Technology (NIST): Œ  Ž   ‘ ’ leadership information/analysis strategic quality planning human resource development/management management of process quality quality and operational results customer focus and satisfaction SMEs Quality Assurance Course 17-19 September 2001 .

Best practices u Global competition encourages companies to seek out and emulate best practices Best practices refers to choosing working methods that has been found to be the most effective and efficient u SMEs Quality Assurance Course 17-19 September 2001 .

What is ISO 9000? t ISO is the International Organisation for Standardisation. the ISO published the first issue of a series of five international standards (ISO 9000. 9001. 9003 and 9004). founded in 1946 to promote the development of international standards. developed by ISO Technical Committee (TC) 176 on quality systems. The ISO 9000 standards were intended to be advisory in nature and for use in two-party contractual situations Quality system registration (certification) involve the assessment and periodic audit by a third-party (quality system registrar). 9002. ISO is composed of member bodies from over 90 countries In 1987. t t SMEs Quality Assurance Course 17-19 September 2001 .

Basic principles and values <<>> SMEs Quality Assurance Course 17-19 September 2001 .

Quality Assurance: basic principles and values x Integrity Independence Prevention/Improvement Objective Evidence Identification/Traceability Segregation x x x x x SMEs Quality Assurance Course 17-19 September 2001 .

Integrity x Personal integrity: – professional – physical x Documentation integrity: – technical – physical SMEs Quality Assurance Course 17-19 September 2001 .

Sometimes irregularity and frauds are not committed for personal interest but to overcome what could seem a temporary contingency harmful to the company.Personal integrity: professional x Fraud and collusion related to quality are not uncommon. Some basic QA practices (e. SMEs Quality Assurance Course 17-19 September 2001 .g. quality records) are meant to discourage frauds and help their investigation if the need arises. Certificate of Conformity signature.

Juran-Quality Control Handbook) SMEs Quality Assurance Course 17-19 September 2001 .M.Personal integrity: professional (cont’d) x Exposure of the QA personnel itself to potentially dangerous situations of collusion “Within the daily range of activity there are many pressures to test the integrity of the inspector…dangerous is the situation posed when there is a potential rejection on a vendor. Here the person pleading for a “break” is often an influential individual well supplied with funds for entertainment or other forms of influence” (J.

For over a hundred years. Our integrity is our asset. It is our balance sheet. Nothing. is more important than integrity.Personal integrity: professional (cont’d) Inspection/certification companies find sometimes appropriate to issue a specific policy statement. nor revenue growth.” (From the “Statement of Integrity” of SGS of Geneva ) SMEs Quality Assurance Course 17-19 September 2001 . nor instructions from above. it defines us to ourselves.. as the following example: “STATEMENT OF INTEGRITY . testing and certification services The cornerstone of this reputation is integrity. nor profitability. our customers. not client wishes. reliable and high quality verification. our shareholders and to the world….. we have been building our reputation as the world leader in independent.

Incorrect use of QA stamps. These are instances when it is to personal advantage or convenience of the inspector to falsify the results of an inspection x x SMEs Quality Assurance Course 17-19 September 2001 . This term describes the tendency of inspectors to falsify the results of inspection of borderline product (excess of reading at maximum of specified value and not beyond). as for example “lending” of personal QA stamps Willful errors or negligence.Personal integrity: professional (cont’d) x “Flinching”.

..M. Juran .Quality Control Handbook) SMEs Quality Assurance Course 17-19 September 2001 .The real remedy is by creating and maintaining an atmosphere of respect for the facts as the ethical foundation of the QA function…” (J. enforce the practical means and exercise the authority to prevent the occurrence of questionable practices ”.Personal integrity: professional (cont’d) x The QA function must be vigilant.

g. Personnel performing nondestructive inspections (NDI) are subjected to periodical eye exam as per norms (e.. in particular vision integrity. MIL-STD-410) SMEs Quality Assurance Course 17-19 September 2001 .Personal integrity: physical x Some QA jobs rely on good physical conditions for their correct accomplishment.

Documentation integrity: technical & physical x Errors can occur due to use of: – unreleased draft documents – obsolete documents – defaced documents SMEs Quality Assurance Course 17-19 September 2001 .

etc.) x x x SMEs Quality Assurance Course 17-19 September 2001 .g. red “OBSOLETE” ink stamp) Document users must be forbidden to alter or deface released documents (by adding personal notes. or clearly identified (e.Documentation integrity: technical & physical x Documents must be released and modified through formal procedures A “Documents Master List” must be continually updated/maintained and made available to document users Obsolete documents must be promptly removed from all point of use. work instructions.

freedom from the influence. but independence must be first of all a professional attitude of QA engineers and managers Acceptance determination of products shall not be influenced by cost and schedule considerations! SMEs Quality Assurance Course 17-19 September 2001 . or determination of another or others” (Webster’s Dictionary) Placement of QA in the organisation should foster independence. control.Independence “State or quality of being independent.

Identification/traceability x Materials Personnel Measurements Design requirements/verifications x x x SMEs Quality Assurance Course 17-19 September 2001 .

Materials identification/traceability x For critical products the conformity certification is of little value without traceability to the materials It must be possible using the product identification (e. marking and identification verifications x x SMEs Quality Assurance Course 17-19 September 2001 .. P/N or DWG/N + S/N or Lot Nr ) to trace all the objective data which demonstrate conformity to the design documentation Traceability is achieved during manufacturing and storage by means of a continuous chain of tagging.g.

Personnel traceability: stamps and signatures x Certain operations or decisions to be valid must be performed by personnel specifically trained/certified and/or authorised:  inspectors  certified welders  project controllers  manufacturing engineers  MRB members  CCB members. which are formally controlled x SMEs Quality Assurance Course 17-19 September 2001 . etc Traceability is ensured through issuance and use of inspection and other stamps. or signatures.

Measurements traceability x A measure is by definition the comparison to a reference standard To ensure the accuracy of a measurement it must be possible to relate it to national or international standards through an unbroken chain of comparison x SMEs Quality Assurance Course 17-19 September 2001 .

Three types of requirements exist: – direct/requirement: a lower level requirement the satisfaction of which totally satisfies the higher level requirement – partial/requirement: a lower level requirement that meets the system level requirement but does not itself totally satisfy the higher level requirement – derived/requirement: a lower level requirement necessary at the lower level to allow the subsystem or equipment to meet a higher level requirement SMEs Quality Assurance Course 17-19 September 2001 x . verifiability and traceability Design requirements of a product are linked hierarchically and logically to system level requirements following the product-tree structure.Design requirements/verifications traceability x The paramount consideration in writing a design requirement is its technical essence .

inspection.Design requirements/verifications traceability (cont’d) x Design requirements identification (by progressive number) and traceability is necessary to ensure design completeness/consistency Design requirements to verifications (test. analysis. review of design) traceability is essential to completely prove that customer requirements are met x SMEs Quality Assurance Course 17-19 September 2001 .

laboratory analyses .shop-travelers .Objective evidence x If it isn’t written down it does not exist If it isn’t written down it didn’t happen If it isn’t written down it is subjective Objective evidence are data such as: .NCR’s Objective data must be archived for a pre-specified time x x SMEs Quality Assurance Course 17-19 September 2001 .test reports .

Prevention The continuous change of denominations for the quality systems and the additions of new tools should not confuse about what was and still is the prime concept of quality management: PREVENTION Prevention of chronic troubles Prevention of re-occurrence Prevention activities and methods “ Once the idea of such prevention is grasped. quality improvement. the label used to describe it (quality control. etc) is unimportant …” (Juran’s Quality Control Handbook) SMEs Quality Assurance Course 17-19 September 2001 . prevention.

Prevention of chronic troubles Sporadic spike old quality control zone Cost of poor quality quality improvement { new quality control zone time SMEs Quality Assurance Course 17-19 September 2001 .

Steps of a quality improvement process A quality improvement process includes the following steps: x x x x x x assess current state and problems define preferred state identify barriers and root causes develop improvement solutions implement solutions monitor results SMEs Quality Assurance Course 17-19 September 2001 .

equipment and documents such as: – – – – – – incoming materials not yet inspected nonconforming materials materials undergoing re-lifing tests out-of-calibration measurement equipment scrapped materials (waiting disposal) obsolete design documents (waiting disposal) x Segregation areas will be secured and accessible only to authorised personnel 17-19 September 2001 SMEs Quality Assurance Course .Segregation x Segregation is meant to prevent the inadvertent/unauthorised use of materials.

Main Elements/Techniques <<>> SMEs Quality Assurance Course 17-19 September 2001 .

Plans and Records Inspections Audits Nonconformance Control Training Qualifications/Certifications x x x x x x SMEs Quality Assurance Course 17-19 September 2001 .Quality Assurance: main elements/techniques x Organisation Manual. Procedures.

ANYBODY could have done it but NOBODY did it. SOMEBODY.Organisation This is the story of four people: EVERYBODY. Finally. SMEs Quality Assurance Course 17-19 September 2001 . ANYBODY and NOBODY There was an important job to be done and EVERYBODY was asked to do it. EVERYBODY thought ANYBODY ought to do it. ANYBODY blamed EVERYBODY for not helping NOBODY and SOMEBODY wisely concluded that NOBODY is the most helpful person in the company. EVERYBODY assumed that SOMEBODY would do it. but NOBODY realised EVERYBODY wouldn’t do it. SOMEBODY got angry about that because it was EVERYBODY’s job.

Organisation Good organisation means: x clear responsibilities and authorities well defined management system adequate communication lines quality function independence x x x SMEs Quality Assurance Course 17-19 September 2001 .

Organisation: clear responsibilities/authorities x x x x x x x defined company organisation structure written job descriptions no conflicting responsibilities no overlap of duties responsibilities commensurate to authority adequate competence and abilities adequate positions SMEs Quality Assurance Course 17-19 September 2001 .

Organisation: defined management system x x x defined way of operation of each organisation unit defined functional interfaces with other units defined external interfaces SMEs Quality Assurance Course 17-19 September 2001 .

Organisation: adequate communication lines x x x communication methods information flow (internal external) open attitude SMEs Quality Assurance Course 17-19 September 2001 .

Quality function placement: example H ig h e r M a n a g e r P r o d u c t io n M Q a na a i t g y e Mr u l a nO a t h g e r M a n a g I n s p e c t i o Qn C E n g i n Q e u ra i n i t g y e l A s s u r a SMEs Quality Assurance Course 17-19 September 2001 .

Quality function placement: example H Q u a l i t y i g M h a e n r a M g a e n r a g e r P r o d u c t i o n T M e c a h n n a i cg a e l r M O a t h n e a r g Me ra n a g e r s Q C / I n s p SMEs Quality Assurance Course 17-19 September 2001 .

Quality function placement: example H Q u a l i t y i g M h a e n r a M g a e n r a g e r P r o d u c t i o n P Mr o a j e n c a t g M e r a P n r ao g j e e c r t A M a n a g e r B Q C / I n s p P A / Q A P A / Q A SMEs Quality Assurance Course 17-19 September 2001 .

Quality function placement: example H ig h e r M a n a g e r P r o d u c t io n T M e ac nh an gi c e a r l M O a tn h a e g r e M r a n a g I n s p Q C / Q A SMEs Quality Assurance Course 17-19 September 2001 .

.ESA-Product Assurance and Safety function . ESA Director General TOS Directorate Directorates Departments Project Divisions Q Department Quality/ Depend/Safety Materials Components Project PA Management SMEs Quality Assurance Course 17-19 September 2001 .

NASA-Safety and Mission Assurance function SMEs Quality Assurance Course 17-19 September 2001 .

Quality Assurance: main elements/techniques x Organisation Manual. Procedures. Plans and Records Inspections Audits Nonconformance Control Training Qualifications/Certifications  x x x x x SMEs Quality Assurance Course 17-19 September 2001 .

standards.Quality system documentation structure Quality policies. responsibilities and documentation Quality system key processes and activities QM tailoring for a specific project Detailed descriptions of specific tasks Objective evidence of conformance to a quality system requirements SMEs Quality Assurance Course 17-19 September 2001 . organisations.

serve as the basis from which audits are performed SMEs Quality Assurance Course 17-19 September 2001 . in particular to: .Quality manual A Quality Manual is a compilation of company policies and requirements meant to guide day-to-day practices.establish responsibilities of each organisational unit .provide written guidance and instructions for quality .make customer and employee aware of the company’s quality philosophy and standards .

procurement control.Quality manual (cont’d) The Quality Manual is prefaced by a quality policy statement of the Company’s chief executive The company operating policy is described in several functional areas. process control. such as subcontracting. training. It will also describe in general terms how decisions will be made regarding conformance and product suitability The Quality Manual is divided into sections based either on functional or organisational areas SMEs Quality Assurance Course 17-19 September 2001 . packaging/shipping etc. inspections. tool and gage control.

where and who_ are the six points that every procedure should address to provide a clear and focused content.Quality procedures: the six honest serving men The beginning of Rudyard Kipling’s poem “The Elephant Child” goes as follows: “ I keep six honest serving men (they taught me all I knew). why. when. Their names are What and Why and When and How and Where and Who” These six words_ what. how. SMEs Quality Assurance Course 17-19 September 2001 .

Quality procedures: short and concise Procedures should be as short and concise as possible (i. no more than five pages)..e. SMEs Quality Assurance Course 17-19 September 2001 . Inclusion of flow-diagrams is always encouraged to summarise the sequence of steps.

project or contract” . resources. and sequence of activities relevant to a particular product.The quality plan may be called “quality program plan” or “quality assurance plan” depending on the scope SMEs Quality Assurance Course 17-19 September 2001 . a plan is a method or strategy for achieving an objective. The QA plan is the contractor’s management strategy for fulfilling the specific quality requirements of a contract ISO 8402: “ Document setting out the specific quality practices.The quality plan usually references parts of the supplier’s quality manual applicable to the specific case .QA plans In general.

the QA Plan must build on a pre-established company quality management system. SMEs Quality Assurance Course 17-19 September 2001 . The reason being that project schedule is usually incompatible with the time required to start a quality system from “zero”.QA plans (cont’d) P r o j e c t Q" A u " a C o n t r a c t l i t y M a n u a l P r o Q A j e P c t l a " A n " To be meaningful.

QA plans (cont’d) The basic QA plan content is as follows: x x x x x x organisation: charts. functional roles and responsibilities compliance matrix to customer requirement crossreferencing to in-house applicable procedures requirements made applicable to sub-contractors project-wide procedures approach to surveillance of subcontractors chart identifying relationship between QA tasks and project milestones SMEs Quality Assurance Course 17-19 September 2001 .

It is beneficial to separate this type of instruction from the more generalised procedures. SMEs Quality Assurance Course 17-19 September 2001 . it is necessary to provide step-by-step instructions to the work force for specific activities.Work instructions In some cases.

but it becomes a quality record when filled in with the results of the test. A minimum retention period must be defined and documented. SMEs Quality Assurance Course 17-19 September 2001 .Quality records x Quality records are documents which are maintained to provide objective evidence of activities performed or results achieved demonstrating: – conformance to specified requirements – effective operation of the quality system A freshly minted test procedure is not a quality record. Periods of 5 to 7 years are commonly used.

Quality records (cont’d) x Examples of Quality Records: – – – – – – – – – – Inspection and test records Shop-travelers (after completion) Nonconformance records Calibration records Audit plan Audit results/reports Training/certification records Identification and traceability records Special processes records Preventive actions records SMEs Quality Assurance Course 17-19 September 2001 .

Plans and Records Inspections Audits Nonconformance Control Training Qualifications/Certifications x  x x x x SMEs Quality Assurance Course 17-19 September 2001 .Quality Assurance: main elements/techniques x Organisation Manual. Procedures.

Inspection (hardware products)          What is an Inspection? Quality Characteristics Workmanship standards Inspection planning Information for Inspection Inspection types/techniques Mandatory Inspection Point (MIP) Degree of inspection Metrology and Calibration SMEs Quality Assurance Course 17-19 September 2001 .

Such inspections are generally called “acceptance” inspections or QA inspections and consist of: a) Interpretation of the applicable technical documents b) Measurement/assessment of the product c) Comparison of a) and b) d) Judgement as to conformance (a/r criteria) e) Disposition of the product f) Recording of data SMEs Quality Assurance Course 17-19 September 2001 .What is an Inspection? A basic QA function is that of deciding through inspections whether the product is conforming.

Quality characteristics A physical or chemical property. a temperature. or any other requirement used to define the nature of a product or a service is a quality characteristics SMEs Quality Assurance Course 17-19 September 2001 . a pressure. a dimension.

Classification of quality characteristics x Design (Functional) Characteristics – ensure performance – ensure life/reliability – ensure safety – ensure interchangeability – ensure interfaces Manufacturing (Non-Functional) Characteristics – related to method of manufacturing – to facilitate manufacturing – to provide information for toolmaking x SMEs Quality Assurance Course 17-19 September 2001 .

responsibility for acceptance of defects Example of classification codes for dimensional characteristics: Critical Major Minor Other Not marked SMEs Quality Assurance Course 17-19 September 2001 .Classification of quality characteristics (cont’d) The great number and variety of quality characteristics makes often necessary to formally highlight their difference in importance so that right care is taken by all concerned. In particular for inspection: .need for independent inspection .need to record inspection results .

Workmanship standards “Something effected. or produced. SMEs Quality Assurance Course 17-19 September 2001 . DIN etc. the quality imparted to a thing in the process of making” (Webster’s Dictionary) Workmanship standards establish the standard for the appearance of a product (indicative of proper quality) and the basic techniques used in producing it.). the art or skill of a workman. made. A wider range of workmanship standards has been established by national and international standardisation organisations (ISO. ESA and NASA have produced workmanship standards in particular for electronics assembly techniques.

4(3A-2)] 17-19 September 2001 PSS-01-738 ----- ECSS-Q-70-08 [PSS-01-708] .4(3M)] NASA-STD-8739.4 [NHB5300.5 NASA-STD-8739.2 [NHB5300.4(3J-1)] NASA-STD-8739. cable assemblies/ install. Staking and conformal coating of PCBs Handsoldering of high reliability electrical connections SMEs Quality Assurance Course ECSS-Q-70-30 [PSS-01-730] PSS-01-726 NASA NASA-STD-8739.1 [NHB5300.4(3G)] NASA-STD-8739.Workmanship standard (cont’d) ESA Wire wrapping of high reliability electrical connections Crimping of high reliability electrical connections High reliability soldering for SMT and mixed technology Fiber optic terminations.4(3G)] NASA-STD-8739.3 [NHB5300.4 [NHB5300.

Workmanship standards : examples SMEs Quality Assurance Course 17-19 September 2001 .

Workmanship standards : examples SMEs Quality Assurance Course 17-19 September 2001 .

.g.Inspection planning The plan for conducting the overall inspection points on a project is called Inspection Plan. FMECA/CIL.. Inspection points are usually identified through: x contract requirements x specifications/standards x reliability and safety analyses (e. PHA) x quality concerns (e.g. interfaces) x manufacturing constraints x operational experience Usually inspection and manufacturing plans are fully integrated. SMEs Quality Assurance Course 17-19 September 2001 .

Information for inspection x Shop-travelers Drawings/engineering lists Workmanship standards Procurement specifications Purchase orders/sample plan Configuration Item Data List (CIDL) Delivery documentation 17-19 September 2001 x x x x x x SMEs Quality Assurance Course .

Inspection types/techniques x Incoming inspection In-process inspection Self-inspection Final inspection Source inspection Non-destructive inspection x x x x x SMEs Quality Assurance Course 17-19 September 2001 .

Mandatory use of holding areas until acceptance.Incoming inspection Inspections performed to assure the conformity of received items to purchase orders.identify deliverable items and quality documents/records . Purchase Orders .must be readily available at Incoming Inspection location Incoming Inspection Instructions .sometimes processed via in-house MRB (impact on costs) SMEs Quality Assurance Course 17-19 September 2001 .establish for group of items the verifications to be performed Rejected materials .usually processed for return to supplier .

The bottom line on use of suppliers data is to be cost effective without compromising the overall control of endproduct conformance Shipping damages Incoming items must be always examined for shipping damages SMEs Quality Assurance Course 17-19 September 2001 .Incoming inspection (cont’d) Use of suppliers data Normally comprehensive inspections/testing must be performed to verify conformity of received items to relevant specifications. and the vendor’s Quality Assurance system has been audited/certified directly or by a reliable third party (e.g. ISO).. Such activities may be considered redundant ( and therefore skipped) if relevant data are requested to the vendor.

g.. procurement of EEE) prevent warranty time expiration for costly item not of immediate use x x SMEs Quality Assurance Course 17-19 September 2001 .Incoming inspection (cont’d) Sometimes specific incoming inspections are anyhow performed to: x prevent potentially severe schedule and cost impacts on the project due to late discovery comply with applicable standards (e.

In-process inspection In-process inspection serve to provide early detection of nonconformances before items are completed or processed into higher assemblies SMEs Quality Assurance Course 17-19 September 2001 .

Self-inspection Consist of allowing performance of in-process inspection by the same operator who has performed the manufacturing operation. SI requires: x x x x x x written quality procedure trained/certified operators written specific instructions and a/r criteria quantitative record of results operator traceability sampling re-verifications by QA Self-inspection of critical characteristics is allowed only for operations which will be directly or indirectly verified by QA at a later time. SMEs Quality Assurance Course 17-19 September 2001 .

Final inspection Consist of inspecting the product and the manufacturing documentation at the time of production completion to verify: x x x x x conformity to contract requirements (including as-designed/ as built) completion of all manufacturing activities ( including inspection and testing) close-out of any previous NCR disposition correct marking. including serialisation (if applicable) complete/correct accompanying documents SMEs Quality Assurance Course 17-19 September 2001 .

Source inspection x Inspection performed by the customer’s QA to confirm product’s compliance to contractual requirements prior to shipment Source inspection may also involve Engineering in a coordinated fashion for issues related to technical performance Acceptance at source: inspection and acceptance at supplier’s facility prior to shipment to the extent that no further inspection is required Source inspection is recommended for critical items when no incoming inspection is feasible at customer’s premises (e. due to lack of special equipment) x x x SMEs Quality Assurance Course 17-19 September 2001 .g..

These techniques require considerable operator skill (certification) in particular with for the interpretation of results The most commonly used techniques are: x Liquid penetrants x Ultrasonic inspection x Radiography SMEs Quality Assurance Course 17-19 September 2001 . impurities.).Non-destructive inspection These are techniques used to inspect products for internal defects (porosity. cracks. inclusions. debonding etc.

1 micron. The penetrant can seep into cracks as small as 0. 4) developing agent applied LP will seep back to surface and spread to the edges thus identifying location and size of defect SMEs Quality Assurance Course 17-19 September 2001 . visible under ultraviolet light . 2) LP brushed or sprayed.fluorescent penetrant.visible penetrants.Non-destructive inspection: liquid penetrants Technique based on property of certain liquids with very low surface tension to penetrate cracks by capillary action. 3) Excess penetrant wiped off or washed away with water or solvents. Two types used: . using dyes usually red in color LP process steps: 1) Part cleaned and dried.

Non-destructive inspection: ultrasonic inspection Ultrasonic beam travels through the part to be inspected . An internal defect interrupts the beam and reflects back a portion of the ultrasonic energy. oil) are used to transmit the ultrasonic waves from the transducer to the test piece SMEs Quality Assurance Course 17-19 September 2001 . Ultrasonic waves generated by transducer that operate on the piezoelectrical principle using materials such quartz (usually 1-25 MHz) Couplants ( water.

Non-destructive inspection: radiography Method used to detect internal flaws. such as cracks and porosity The principle involved is density differences. visible permanent image made on x-ray film. The metal surrounding a defect will appear denser and hence show up as lighter than the flaws on an x-ray film. SMEs Quality Assurance Course 17-19 September 2001 . Radiation source an x-ray tube.

in particular at points where following operations would make impossible the conformity assessment of previous operations MIPs should be used not only as part of the product acceptance process.Mandatory Inspection Point (MIP) x Mandatory Inspection Point (MIP): Activity performed by representatives of customer. to verify that the operations of fabrication and control have been performed correctly MIPs are performed at important stages of the fabrication process. prime contractor or their delegates. but also to spot-check certain relevant elements of the quality system x x SMEs Quality Assurance Course 17-19 September 2001 .

MIP (cont’d) x A MIP may take the form of: – Inspection : direct performance of an inspection (by customer) – Witness : watch a specific operation – Results verification : review of records SMEs Quality Assurance Course 17-19 September 2001 .

MIP (cont’d) x Severity of identified defects should also be assessed: – minor – impact on following fabrication operations – impact on item qualification/acceptance – impact on system integration – mission failure – safety The identification of potential catastrophic defects undetectable until their occurrence in flight would require immediate high profile actions An inspection report should be issued for each MIP 17-19 September 2001 x x SMEs Quality Assurance Course .

as low as 20% for inexperienced subs) accepted observat.MIP (cont’d) Average results from a major space contractor: (acceptance up to 90% for experienced subcontractors. rework NCR reject SMEs Quality Assurance Course 17-19 September 2001 .

Degree of Inspection x Redundant 100% Inspection Statistical sampling/sampling 1st/Last piece Audit x x x x SMEs Quality Assurance Course 17-19 September 2001 .

Metrology and calibration
A vast variety of gages and instruments are used in industry for measurement purpose (e.g. mechanical, electronic, optical, pneumatic). The QA function is concerned with appropriateness, use, maintenance of such instruments (but not with their scientific development). Calibration The set of operations which establish, under specified conditions, the relationship between values indicated by a measuring instrument and the corresponding standard or known values derived from the standard.

SMEs Quality Assurance Course

17-19 September 2001

Accuracy of measurement
Target analogy

x

Precise but not accurate

x

Accurate but not precise

x

Accurate and Precise

SMEs Quality Assurance Course

17-19 September 2001

Accuracy, precision and resolution
x

Accuracy The degree of agreement of the measurement with the true value of that parameter. The difference between measured and true value is defined as error. Precision The ability of an instrument to repeat he same reading when making the same measurement in the same manner and under identical conditions. Resolution The smallest change in input necessary to produce the smallest detectable change in output of the instrument under test.

x

x

SMEs Quality Assurance Course

17-19 September 2001

Maintaining accuracy
From the moment an instrument is put into use it begins to deteriorate in accuracy. The procedure for maintaining accuracy includes:  Number each instrument  Establish a “card” record for each instrument  Establish a checking interval  Provide for adherence to the checking schedule  Keep record of the findings of the check  Analyse the results to take corrective actions (if needed)  Analyse the trends to modify the intervals (longer/shorter)

SMEs Quality Assurance Course

17-19 September 2001

Checking interval
The initial checking interval is based on instrument manufacturer recommendation and company experience. The checking interval is based on:  Calendar  Amount of use Adherence to the checking system makes or breaks the system

SMEs Quality Assurance Course

17-19 September 2001

Level of standards
x

Primary Standards: These standards (gage blocks, standard cells, standard electrical bridges) are certified (occasionally) by a national metrology institute. They are used only to check secondary standards. Secondary Standards: These standards are checked periodically against the primary standards. They are not used to measure the product but to check the shop standards Shop Standards: These standards are used to check the product itself

x

x

SMEs Quality Assurance Course

17-19 September 2001

National and international (measurement) standards
x

Measurement Standard A material measure, measuring instrument, reference material or system intended to define, realise, conserve or reproduce a unit or one or more known values of a quantity to serve as reference. National Standard A (measurement) standard recognised by an official national decision to serve, in a country, as the basis for fixing the value of all other standards of the quantity concerned. International Standard A (measurement) standard recognised by an international agreement to serve internationally as the basis for fixing the value of all other standards of the quantity concerned.
17-19 September 2001

x

x

SMEs Quality Assurance Course

(ANSI/NCSL Z540-1-1994) The purpose of requiring traceability is to ensure that measurements are accurate representations of the specific quantity subject to measurement. through an unbroken chain of comparison. associated with the result of a measurement. SMEs Quality Assurance Course 17-19 September 2001 . within the uncertainty of the measurement. that characterises the dispersion of the values that could reasonably be attributed to the quantity subjected to measurement.Traceability to national/international standards Traceability The property of a result of a measurement whereby it can be related to appropriate standards. generally national or international standards. Uncertainty A parameter.

at least. When instruments are sent for external calibration. an official record of the shipment should be kept on file until the instrument is returned. unambiguous and objective manner. x x SMEs Quality Assurance Course 17-19 September 2001 . When calibrations are performed by an outside (accredited) Lab a Certificate of Calibration will be requested for each calibration. all information necessary for the interpretation of the calibration results.Calibration records and certificates x A record system will be maintained including the list of instruments under calibration control and records for each calibration performed (containing. along with the identity of the performing technician). sufficient information to permit the repetition of the calibration. in a clear. The certificate will present.

Quality Assurance: main elements/techniques x Organisation Manual. Procedures. Plans and Records Inspections Audits Nonconformance Control Training Qualifications/Certifications x x  x x x SMEs Quality Assurance Course 17-19 September 2001 .

and . When conducting an audit. it is vital constantly to remember that one’s aim is FACT FINDING NOT FAULT FINDING Obviously.is capable to fulfil its contractual and legal obligations. there will be occasions when faults will be found. but that is a fact! SMEs Quality Assurance Course 17-19 September 2001 . and/or .is effectively implementing a QA system It is also the true and fair presentation of the results of such examination.is able to achieve its policies and objectives. to determine whether or not the auditee .What is an audit ? An audit is an independent examination of objective evidence performed by competent personnel.

Audits System audit Compliance audit Process audit Product audit SMEs Quality Assurance Course 17-19 September 2001 .

System audit Documented activity performed to verify. SMEs Quality Assurance Course 17-19 September 2001 . by examination and evaluation of objective evidence. documented. It is advantageous to conduct a systems audit well in advance of the actual starting of an activity. System audits are akin to readiness reviews. that applicable elements of the quality system are suitable and have been developed. and effectively implemented in accordance with specified requirements A system audit probes whether or not there is an organisation and a quality assurance system in existence and assesses their adequacy to meet the requirements invoked by a contract.

The former concerns itself with verifying that the QA system is implemented satisfactorily. The auditor will scrutinise the objective evidence that should be generated by the implementation of the system. SMEs Quality Assurance Course 17-19 September 2001 . Compliance audit must not be confused with an “inspection”. The latter is concerned with product acceptance and release for further processing.Compliance audit The compliance audit investigates whether or not the QA system is implemented. The auditor checks that the auditee works to the letter of the QA procedures.

or conditions SMEs Quality Assurance Course 17-19 September 2001 . correctness.Process audit Analysis of elements of a process and appraisal of completeness.

Product audit Quantitative assessment of conformance to required product characteristics SMEs Quality Assurance Course 17-19 September 2001 .

Each area of the company will be audited at least once annually The objective is to ascertain if all personnel who have duties and responsibilities in accordance to the Quality Manual (or PA Plan). MIP’s x x SMEs Quality Assurance Course 17-19 September 2001 .Internal audit x Internal audits are not performed in a single shot. At start of each year an Internal Audit Plan is drawn up. third party maintenance of certification checks (if any). understand and satisfactorily perform their duties and responsibilities The audit of the QA/QC organisation is considered accomplished by customer’s audits.

External audit x Preparation Performance Conclusions Report Follow-up x x x x SMEs Quality Assurance Course 17-19 September 2001 .

Audit preparation and performance x Pre-audit Questionnaire and visit Audit team selection Audit notification Opening meeting Audit performance Debriefing meeting Report and follow-up 17-19 September 2001 x x x x x x SMEs Quality Assurance Course .

r.Pre-audit questionnaire and visit x To collect basic information about the company to be audited To ensure no misunderstanding about audit scope and objectives To obtain the QA Manual (including compliance matrix w.t. audit baseline) To Identify the reference documentation To visit the facilities x x x x SMEs Quality Assurance Course 17-19 September 2001 .

Audit team selection x It includes at least an audit team leader If necessary one or more additional auditor because of areas of expertise (e. RAMS. S/W PA.g. CM) Project QA representative (if a project audit) x x SMEs Quality Assurance Course 17-19 September 2001 .

Audit notification x A reasonable time before the audit (e.g. agreement on dates 2 months before the audit. formal notification 1 month before audit) Notification in writing and including: – scope/baseline of audit – schedule – name of lead auditor – draft agenda x SMEs Quality Assurance Course 17-19 September 2001 .

Opening meeting x Introduction of attendants: – meeting the management – meeting counterparts Presentation of audit agenda (team leader): – summarise audit scope and procedure – finalise detail agenda (splinters if any) Presentation of the company and logistics: – assigned rooms – documentation access x x SMEs Quality Assurance Course 17-19 September 2001 .

signatures... such as reports. certificates): – statements are not sufficient (it may be useful to note them_”we are told that.”) and – feelings are not sufficient to conclude on conformance to requirements SMEs Quality Assurance Course 17-19 September 2001 .Performance of audit x x Investigate by means of audit checklist: – use audit checklist is as guide – not all questions are to be asked – adapt the wording to your counterpart – adapt the sequence of questions to circumstances (locations) “to be re-inquired” Look for OBJECTIVE EVIDENCE ( trace of facts.

note explanations x determine if it is of a systematic nature in view of conclusions REMINDER: In the conduct of the audit keep in mind that the Customer/Supplier relationship of the past that operated in an adversarial environment is becoming ancient history SMEs Quality Assurance Course 17-19 September 2001 .Performance of audit (cont’d) If you identify a non conformance or deficiency x confirm it with your counterpart: questions to be answered by “yes” or “no”.

as recognition that each reported observation is as verbally indicated during the audit – audit team leader presents opinion on the audit results – request formal response to each observation SMEs Quality Assurance Course 17-19 September 2001 .Debriefing meeting x Debriefing meeting: – meeting with the management of audited organisation – each auditor presents his Observations Sheets (OS) – provide clarifications – collect signatures (if not yet done).

Audit report x Audit Report: – – – – – – – – purpose of the audit requirements baseline audit team members audit agenda personnel contacted summary of results Observations Sheets recommendations on each OS SMEs Quality Assurance Course 17-19 September 2001 .

Audit follow-up x Obtain written response for each OS Evaluate response: – proposed corrective actions – proposed schedule Agree and formalise corrective actions and schedule Follow-up visits to verify close-out of corrective actions Formal audit closure x x x x SMEs Quality Assurance Course 17-19 September 2001 .

SMEs Quality Assurance Course 17-19 September 2001 . Recommendation A Recommendation describes an acceptable solution to an observation.Audit terminology Observation An observation is made every time a total absence of arrangements to meet an important requirement or arrangements which are clearly inadequate is found. The term “Non-Compliance” is also often used as synonym. Observations are also made when lack of implementation of Quality Manual or procedures is revealed. A recommendation should be written when it considered helpful to speed-up the achievement of agreement on a corrective action.

Procedures and Plans Inspections Audits Nonconformance Control Training Qualifications/Certifications x x x  x x SMEs Quality Assurance Course 17-19 September 2001 .Quality Assurance: main elements/techniques x Organisation Manual.

Forms definition for internal/external reporting .Processing/meetings (formal meeting often only in special cases) .Non-conformance procedure Usual elements of a nonconformance reporting (NCR)* procedure: .Initiator responsibilities (is anybody authorised to issue a report or only inspectors/QA?) .Responsibility for investigation of causes. preventive actions definition/concurrence and tracking *often called SPR (S/W Problem Report) for S/W products SMEs Quality Assurance Course 17-19 September 2001 .

next higher level+customer) .Interfaces with Configuration Control function (issuing of waivers) .Segregation of non-conforming product SMEs Quality Assurance Course 17-19 September 2001 ..Material Review Board responsibilities and membership (by name) .Non-conformance procedure (cont’d) . [Repair: to be treated as any manufacturing operation (written instructions) except for very simple cases] .g.Disposition implementation.Serial or parallel NCR distribution distribution (e.

SMEs Quality Assurance Course 17-19 September 2001 .Non-conformances: disposition Return to supplier: Only applies to non-conforming procured items Use-as-is: The non-conformance is acceptable at the status Rework: Re-application of process will eliminate the non-conformity Repair: It is basically a modification that will allow use of the item Scrap: The item is not acceptable at the status or recoverable Dispositions Use-as-is and Repair may require issuing a waiver.

Nonconformance processing: example QUALITY CONTROL or other authorised unit MANUFACTURING/TESTING OR RECEIVING INSPECTION PRELIMINARY REVIEW (PR) QUALITY ASSURANCE MATERIAL REVIEW BOARD (MRB) • PREPARES DISCREPANCY NOTICE (DN) • SEGREGATION • QA CONDUCTS PR • DISPOSITION FOR SRP OR MINOR REWORK • PROMOTION TO MRB • NCR PREPARATION • NOTIFIES OF FORTHCOMING MRB • ASSESSES CUMULATIVE EFFECT OF PREVIOUS NCRs • CUSTOMER PARTICIPATION FOR MAJOR NCR • MRB DETERMINES DISPOSITION NO YES WAIVER QUALITY ASSURANCE CUSTOMER (ANY LEVEL) CONFIGURATION MANAGEMENT QUALITY ASSURANCE • VERIFIES CORRECT IMPLEMENTATION OF DISPOSITION (after implementation) • CONDUCTS CCB • DISPOSITIONS AND APPROVES WAIVER • CONTRACTUAL AUTHORIZATION TO PROCEED WITH WAIVER (NOTE 5) • WAIVER NUMBERING • SCHEDULES CHANGE BOARD (if needed) • PREPARES SUPPORTING DATA AND DESCRIPTION / JUSTIFICATION FOR WAIVER SMEs Quality Assurance Course 17-19 September 2001 .

and it is the symptoms that are treated rather than the cause. A defect attributed to operator error may not be the real reason if the operator has not been trained.Non-conformances: investigating root causes Root cause is the real reason for a nonconformity. It is possible that symptoms of the problem are identified as the cause. x For each non conformity there is a root cause x Causes are preventable x Prevention is always cheaper SMEs Quality Assurance Course 17-19 September 2001 . or was working with a poor design. had inadequate work instructions.

Identify interim actions A Corrective Actions System should be in place to: .initiate QIP (Quality Improvement Project) when appropriate SMEs Quality Assurance Course 17-19 September 2001 .Non-conformances: corrective actions Corrective actions shall be aimed to prevent recurrence: . who and when) .determine need/effectiveness of corrective actions .Determine and document action to remove root causes .Plan corrective actions (what.summarise and collect nonconformances data (including costs) .

g. S/W) Members of the MRB have collective responsibility and specific responsibility.investigate root causes to identify corrective/preventive actions .ensure that the process is correctly followed and records kept SMEs Quality Assurance Course 17-19 September 2001 . Specific responsibility of the QA function in the MRB is to: . The MRB is chaired by QA with members from Engineering and other specialists teams (e.The Material Review Board Purpose of the MRB is to provide an organised review of non-conforming product. Materials.ensure that remedial actions are properly implemented . Specific responsibility relate to the field of specialisation.

Procedures.Quality Assurance: main elements/techniques x Organisation Manual. Plans and Records Inspections Audits Nonconformance Control Training Qualifications/Certifications x x x x  x SMEs Quality Assurance Course 17-19 September 2001 .

practice procedure under instructor’s observation/support . this is all that is required. procedures and specifications will not result in hardware quality if the skills of manufacturing personnel are not sufficient to execute the instructions. and records will simply note that the individual was given training on a certain date.Training The existence of documented plans. SMEs Quality Assurance Course 17-19 September 2001 . Training in any area is based upon: .familiarisation with written process specification/procedures .hardware test/inspection providing feedback on success For some processes.

Testing outcome must be quantified. SMEs Quality Assurance Course 17-19 September 2001 .Training (cont’d) Advanced training including testing are required for critical processes and special techniques such as: x x x x x x Welding Adhesive Bonding Electronics Assembly Conformal Coating NDI Metrology Testing may be written. or an ability demonstration. or combination of written and demonstration.

ESA skills training centers IdS Paris (F) Hand Soldering Operator Hand Soldering Inspector Repair/Modif PCBs RF Cable Assy Crimping Wire Wrapping SMT Highbury College ZVE Portsmouth Oberpfaffenhofen (UK) (D) IIS Genova (I) Hytek Aalborg (DK) IFE Obrpfaffenhofen (D X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X Instructor Fiber Optic Termination SMEs Quality Assurance Course 17-19 September 2001 .

Quality Assurance: main elements/techniques x Organisation Manual. Plans and Records Inspections Audits Nonconformance Control Training Qualifications/Certifications 17-19 September 2001 x x x x x  SMEs Quality Assurance Course . Procedures.

Qualification/certifications x Qualification – process qualification – product qualification x Certification – – – – quality system certification product conformity certification personnel certification stamp warranty SMEs Quality Assurance Course 17-19 September 2001 .

Certificate of Conformity “A document signed by an authorised party affirming that the supplier of a product or service has met the requirements of the relevant specifications.” SMEs Quality Assurance Course 17-19 September 2001 . contract or regulation.

The testing process and standard for pass/fail must be documented The certifying organisation may be: – the contractor – a space agency or a delegated organisation – a nationally recognised organisation Some skills must be practised on a regular basis in order to maintain the currency of the certification Some certification may require annual medical eye exam x x x SMEs Quality Assurance Course 17-19 September 2001 .Personnel certification x Certification is very similar to testing but the process is more formal.

Personnel certification: records x Certification records will include at least: – – – – – – – – location and date of training certification process procedure specification/procedure grade/rating instructor name expiration date periodic renewals achieved job continuity SMEs Quality Assurance Course 17-19 September 2001 .

Personnel certification: re-certification x Re-certification is required for one of the following reasons: – – – – certification expiration interruption of work unsatisfactory performance evolution of techniques SMEs Quality Assurance Course 17-19 September 2001 .

Stamp warranty x Stamp warranty means that an inspector who signifies the status of an inspection or test by stamping: x is qualified to determine that status personally saw or performed the activity as documented x x At the time of stamp release. the inspector will acknowledge in writing his full understanding of the rules for use of stamps SMEs Quality Assurance Course 17-19 September 2001 .

QA during Major Project Phases <<>> SMEs Quality Assurance Course 17-19 September 2001 .

Quality assurance during major phases x Design and Verification Procurement Manufacturing Testing Acceptance/ delivery x x x x SMEs Quality Assurance Course 17-19 September 2001 .

perform CI Verification Inspect to “build-to” Documents i ri f Ve d ce an n n ue tio Seq ra teg In ti ca on s it i Se on q u an en d D ce e f i “Build-to” Documents Inspection Plan ni tio n Fabricate. Assemble and Code SMEs Quality Assurance Course 17-19 September 2001 . System Concept System Specification Sy Verification Plan CI Specification. perform System Verification Assemble CI. CI Verification Plan De c om po Demonstrate and Validate the System Integrate System.Design and verification cycle: the V-cycle Mission Requirements.

Design and verification x Procedures for engineering documents Design/Development and Verification plans QA support to internal design review for: – producibility – repeatibility – inspectability – testability QA review of design changes for quality impact x x x SMEs Quality Assurance Course 17-19 September 2001 .

Design and verification: structured requirements EXAMPLE ESA Requirements Document External Interface Requirements System Specification Architectural Design Document Detailed Design Document SMEs Quality Assurance Course 17-19 September 2001 .

quality. coverage. without specifying how it has to be done Performance Requirements The extent to which a function must be executed. interface. technology etc.Types of design requirements x Functional Requirements Tasks. timeliness or readiness Design Constraints Boundary conditions within which the designer must remain while allocating performance requirements and/or synthesizing system elements (safety. generally covered in terms of quantity. environmental.) x x SMEs Quality Assurance Course 17-19 September 2001 . actions or activities that must be accomplished by a system.

Ensure that QUALITY is designed into the product. and verifications consistently planned and thoroughly executed. Ensure that QUALITY is designed into the product. SMEs Quality Assurance Course 17-19 September 2001 .Requirements for requirements x Design requirements must be: – – – – – complete consistent unambiguous verifiable traceable The QA function must ensure that design requirements are properly The QA function must ensure that design requirements are properly documented. executed. and verifications consistently planned and thoroughly documented.

Quality assurance during major phases x Design and Verification Procurement Manufacturing Testing Acceptance/ delivery  x x x SMEs Quality Assurance Course 17-19 September 2001 .

Procurement control x Selection of procurement sources Control of purchase documents Surveillance of suppliers x x SMEs Quality Assurance Course 17-19 September 2001 .

g. NCRs) SMEs Quality Assurance Course 17-19 September 2001 . NCRs) of supplies and performance data (e. % of defective items.g. % of defective items...Selection of procurement sources The selection is usually based on the following criteria: x certification/approval for specific items demonstrated capability (positive performance records) pre-award audit x x Maintain a list of subcontractors and suppliers including records Maintain a list of subcontractors and suppliers including records of supplies and performance data (e.

interface data records) QA activities to be performed by the supplier planned activities at source (if any) Purchase Orders (PO) and Contracts shall be reviewed and formally Purchase Orders (PO) and Contracts shall be reviewed and formally approved by the QA function prior to release approved by the QA function prior to release SMEs Quality Assurance Course 17-19 September 2001 .Control of purchase documents Procurement documents shall contain by statement or reference: x x x x technical description of the item/service to be procured QA requirements and deliverable/accompanying documents (e. CoC.g. ADP..

source inspection. In some special cases a resident inspector may be needed The QA plan should always provide information and justification The QA plan should always provide information and justification on the planned surveillance activities of subcontractors and suppliers on the planned surveillance activities of subcontractors and suppliers SMEs Quality Assurance Course 17-19 September 2001 .Surveillance of suppliers QA surveillance of subcontractors and suppliers is performed through periodic audits and inspections (MIPs. incoming inspection).

Quality assurance during major phases x Design and Verification Procurement Manufacturing Testing Acceptance/ delivery x  x x SMEs Quality Assurance Course 17-19 September 2001 .

Manufacturing x Materials and parts control Production equipment control Cleanliness and contamination control Control of manufacturing processes Control of manufacturing documents x x x x SMEs Quality Assurance Course 17-19 September 2001 .

identity verifications as first operation.Materials and parts control x Control must be exercised to ensure that materials and parts released for manufacturing are: – correct /traceable – conforming – within their useful life (if life limited) Ensure that shop-travelers always include materials and/or parts Ensure that shop-travelers always include materials and/or parts identity verifications as first operation. Ensure limited-life materials tracking and labeling with expiration dates Ensure limited-life materials tracking and labeling with expiration dates SMEs Quality Assurance Course 17-19 September 2001 .

Production equipment control x Control must be exercised to ensure that manufacturing tools are: – – – – – conform to their drawings periodically checked (recurrent production) properly stored to prevent deterioration/damage identified inventory kept SMEs Quality Assurance Course 17-19 September 2001 .

including any COTS software validation and control of production tools. including any COTS software SMEs Quality Assurance Course 17-19 September 2001 .Production equipment control (cont’d) x With the increasing use of computer-aided manufacturing and inspection techniques. the relevant s/w need to be: – – – – documented tested and approved prior to use configuration controlled secured to prevent tampering and misuse The QA function must have aaprimary and documented role in the The QA function must have primary and documented role in the validation and control of production tools.

testing. drawings and procedures Contamination control plan This is a plan identifying the control measures necessary to achieve the required level of cleanliness during manufacturing.Cleanliness and contamination control x Cleanliness levels The required cleanliness levels of contamination-sensitive items will be indicated on specifications. packaging and storage of contamination-sensitive items Cleanliness of facilities “Clean facilities”conditions and operations need to be controlled to ensure that their specified level of cleanliness is maintained. handling. x x SMEs Quality Assurance Course 17-19 September 2001 .

Cleanliness and contamination control (cont’d) x Clean rooms facilities instructions will cover: – entering/leaving (personnel and equipment) – cleaning methods and materials – preventive maintenance – control/recording of clean room parameters (pressure. shield items) The QA function has aaprimary role in enforcing compliance to the The QA function has primary role in enforcing compliance to the Contamination Control Plan and verification of clean-facilities Contamination Control Plan and verification of clean-facilities SMEs Quality Assurance Course 17-19 September 2001 . humidity. temperature) – performance of particle counts – action in case of alarm (stop work.

Manufacturing processes x Process A sequence of pre-defined steps intended to achieve a goal or objective Manufacturing process: – Special Process – Critical Process – Standard Process x SMEs Quality Assurance Course 17-19 September 2001 .

.g.g. bonding. tensile test) of representative samples/coupons periodic control of process equipment/materials training/certification of personnel records maintenance/archiving SMEs Quality Assurance Course 17-19 September 2001 .Special process Special process is a process in which the outcome cannot be adequately verified without destroying the manufactured part (e.. welding). heat treatment. chemical analysis. Special process require: x x x x x x x qualification control of process parameters non-destructive inspection destructive testing (e.

A critical process may or may not be a special process and vice-versa A critical process is one: which in case of failure can adversely affect performance or destroy a major part or system function the quality of which cannot be assessed solely by examining the end product which has caused problems previously with which the contractor has no experience SMEs Quality Assurance Course 17-19 September 2001 .Critical process Critical Process is not necessarily the same as Special Process.

The quality of which cannot be assessed solely by examining the end product .A process that has caused problems previously -Where an ESA specification is available but is not being implemented . that in the caseDEFINITIONadversely affect the .Critical process: PSS and ECSS definitions .A process with which the contractor in charge of it has had no previous experience PSS-01-700 YES YES YES ECSS-Q-70A YES YES YES YES SMEs Quality Assurance Course 17-19 September 2001 .A process of failure can performance or destroy a major part or function of the space system .

has a previous history of use.Standard process x Standard process is one that is well documented. is well understood and for which standard inspection procedures exist SMEs Quality Assurance Course 17-19 September 2001 .

Emphasis on process control depends on the criticality of the process . including control methods.Manufacturing process control x Manufacturing Process control . QA provisions and a/r criteria .Each process must be covered by a Process Specification. Each process will be classified as standard/established or critical SMEs Quality Assurance Course 17-19 September 2001 .ESA requires delivery of a project Process List to verify that processes are properly specified and controlled.

/purchase order – include rev. release date – include item identification and configuration data – kept with product during manufacturing/inspection operations – include provisions for traceability of operators and inspectors Manufacturing documents control – QA concurrence to ensure adequacy and right placement of inspections (any later sequence change needs QA agreement) – Design Office approval for critical/special processes (any later sequence change requires Design Office approval) 17-19 September 2001 x SMEs Quality Assurance Course .status/date.Control of manufacturing documents x Manufacturing documents – identified/traceable to contract no.

Quality assurance during major phases x Design and Verification Procurement Manufacturing Testing Acceptance/ delivery x x  x SMEs Quality Assurance Course 17-19 September 2001 .

Testing (qualification and acceptance) x Test facilities and equipment Test reviews Test surveillance Test documentation: – test procedures – test reports x x x SMEs Quality Assurance Course 17-19 September 2001 .

Test facilities and equipment x Environmental tests at certified facilities Test equipment verifiable without application to the test article Test software validated/verified prior to use x x SMEs Quality Assurance Course 17-19 September 2001 .

Test reviews Test Procedure/ Test Readiness TRR: Test Readiness Review PTR: Post-Test Review TRR no yes authorisation Test Performance Test Report PTR SMEs Quality Assurance Course 17-19 September 2001 .

Test Reviews: TRR x Test Readiness Review – – – – – – – test article configuration (as-required/as-built) any open NCR for impact test procedure (availability/approval) test facility readiness test procedure (availability/approval) test equipment calibration status protection of personnel and test article SMEs Quality Assurance Course 17-19 September 2001 .

Test Reviews: PTR x Post-test Review – – – – – completeness of required data deviations/modifications for proper authorisation NCRs during test for proper disposition test results compliance with requirements grant authorisation for further tests/processing Ensure that QA is represented in the boards established for reviewing Ensure that QA is represented in the boards established for reviewing test readiness and test results. test readiness and test results. SMEs Quality Assurance Course 17-19 September 2001 . Ensure QA surveillance during test. Ensure QA surveillance during test.

Test surveillance x Test surveillance is performed by QA in one of the following forms: – continuous witnessing (for tests requiring manual intervention) – observation ( periodic monitoring) – data review ( review recorded data) x QA personnel has generally the authority to stop the test for safety reason or to prevent hardware damages 17-19 September 2001 SMEs Quality Assurance Course .

Test documentation x Test procedure/report must include: – test article identification – test equipment and test set-up – characteristics and design criteria to be tested – data recording and pass/fail criteria – sequence of operations (including inspections) – measures to ensure safety – environmental conditions – test organisation (personnel/responsibilities) – test results and conclusions (for test reports) Test procedures must be reviewed and approved by QA 17-19 September 2001 x SMEs Quality Assurance Course .

Quality assurance during major phases x Design and Verification Procurement Manufacturing Testing Acceptance/delivery x x x  SMEs Quality Assurance Course 17-19 September 2001 .

or – Delivery Review Board (DRB) DRB responsibilities: – assess conformity to contractual requirements – verify nonconformances closed-out – accompanying documentation complete 17-19 September 2001 x x SMEs Quality Assurance Course .Acceptance/delivery x Acceptance Data Package – data to support acceptance – documents/data to support further integration Acceptance review – formal acceptance review board.

Acceptance/delivery x QA is a member of the DRB QA will perform surveillance of packaging/labeling operations QA will ensure that accompanying documentation is included x x QA has a leading role in acceptance and delivery activities and QA has a leading role in acceptance and delivery activities and must be directly involved in their performance must be directly involved in their performance SMEs Quality Assurance Course 17-19 September 2001 .

THE PURPOSE OF A QA SYSTEM IS TO CREATE PROSPERITY NOT ENGENDER BUREACRACY SMEs Quality Assurance Course 17-19 September 2001 . In such a case. engineers and personnel recognise that they are simply facing a bureaucratic exercise and ridicule the QA System.Beware of the “enemy” called bureaucracy ! Many companies write a huge binders full of procedures to fulfill the customer’s (or ISO standard’s) requirements regardless of whether the procedures do the company any good or make any practical sense.

Quality Assurance Part II : QA Standards SMEs Quality Assurance Course 17-19 September 2001 .

Standards Evolution <<>> SMEs Quality Assurance Course 17-19 September 2001 .

Quality Assurance: standards evolution x ISO standards Military standards Space standards: x NASA standards x European standards x x SMEs Quality Assurance Course 17-19 September 2001 .

K.S. DoD establishes MIL-Q-9858 “Quality Management Program” MIL-Q-9858 updated to RevA NATO adopts MIL-Q-9858A as AQAP-1 U.Where does ISO 9000 come from? 1959 1963 1968 1970 1979 U. BS 5750 ISO created ISO 9000 standards from BS 5750 17-19 September 2001 1987 SMEs Quality Assurance Course . MoD adopted AQAP-1 as DEF/STAN 05-8 BSI (British Standards Institution) develops the first commercial quality management standard.

9002. and service Standard for organisations involved in production Standard for organisations involved in final inspection and test Guidelines for quality management SMEs Quality Assurance Course 17-19 September 2001 . production. 9003 Standard for organisations involved in design.The ISO 9000(1994) series x The ISO 9000 (1994) series was composed of five standards: ISO 9000 ISO 9001 ISO 9002 ISO 9003 ISO 9004 Guidelines for selection and use of 9001.

ISO 9001(1994) compared to “traditional” QA ISO 9001 (1994) .Management Review .Servicing Quality Assurance (Traditional) Design Assurance (Basic Principles) SMEs Quality Assurance Course 17-19 September 2001 .Contract Review .

3 Contract Review 4.7 Control of Customer-supplied Product 4.10 Inspection and Testing 4. Measuring and Test Equipment 4.1 Management Responsibility 4.17 Internal Quality Audits 4.8 Product identification and Traceability 4.14 Corrective Action 4.20 Statistical Techniques Comprehensive Requirement Less Comprehensive than 9001 and 9002 NA Element Not Present 9003 NA NA NA SMEs Quality Assurance Course 17-19 September 2001 .18 Training 4. Package and Delivery 4.9 Process Control 4.15 Handling. ISO 9002.16 Quality Records 4.11 Inspection.Cross reference table of ISO 9001.19 Servicing 4. 9002 and 9003 Cross Reference of ISO9001.5 Document Control 4. and ISO 9003 ISO Clause and Title 9001 9002 4.4 Design Control NA 4.12 Inspection and Test Status 4.6 Purchasing 4. Storage.2 Quality System 4.13 Control and Non-conforming Product 4.

ISO 9001(1994): management review Executive responsibility Quality policy Relevant Management Responsibility Organisation Defined intervals Management Review Suitability & Effectiveness Records SMEs Quality Assurance Course 17-19 September 2001 .

Records SMEs Quality Assurance Course 17-19 September 2001 . communication ext.ISO 9001(1994): contract review Procedure Requirements Review incoming order/ contract differences capabilities unwritten rqts Contract Review Amendments outgoing proposal process definition int.

ISO 9001(1994): design control Procedure Design/Development Plan Organisation Design Input Design Output Design Review Verification Validation Changes Design Control SMEs Quality Assurance Course 17-19 September 2001 .

they have been reorganised in five main sections: – – – – – Quality Management System (QMS) Management Responsibility Resource Management Product Realisation Measurement.New format of ISO 9001(2000) “Quality management systems . ISO 9001 will be used instead by excluding certain requirements SMEs Quality Assurance Course 17-19 September 2001 .Requirements” x None of the ISO 9001(1994) requirements have been removed. Analysis and Improvement x ISO 9002 and 9003 cancelled.

Basic terminology modifications x ISO 9000(1994) SUBCONTRACTOR SUPPLIER CUSTOMER x ISO 9000(2000) SUPPLIER ORGANISATION CUSTOMER SMEs Quality Assurance Course 17-19 September 2001 .

ISO 9001(2000): Clause 4 .Quality Management System (QMS) x Demonstrate continuous improvement in QMS effectiveness Define processes for QMS application throughout the organisation Documents demonstrate effective planning x x SMEs Quality Assurance Course 17-19 September 2001 .

Management Responsibility x Customer requirements must be defined and fulfilled Quality policy provides framework for objectives and is measurable Effectiveness of QMS is communicated x x SMEs Quality Assurance Course 17-19 September 2001 .ISO 9001(2000): Clause 5 .

Resource Management x Establish competency needs Determine training effectiveness Infrastructure/working environment provided for Determine resources to: – Implement/maintain QMS – Continually improve – Meet customer objectives x x x SMEs Quality Assurance Course 17-19 September 2001 .ISO 9001(2000): Clause 6 .

g.Product Realisation x Quality objectives/product requirements set Customer requirements for delivery and post delivery determined Obligations related to product (e.ISO 9001(2000): Clause 7 . statutory and regulatory determined) Product function and performance requirements defined Control of any customer intellectual property x x x x SMEs Quality Assurance Course 17-19 September 2001 .

Analysis and Improvement x Monitor customer perceptions on fulfilment of requirements Customer satisfaction data needed is defined and subsequently analysed Audits to verify compliance with planned arrangements.Measurement. as well as QMS requirements and ISO 9001 (2000) requirements Audit frequency and methodology to take account of previous results Process defined for waiver approval and defect reporting to relevant authority SMEs Quality Assurance Course 17-19 September 2001 x x x x .ISO 9001(2000): Clause 8 .

can only be successful if : x specific methods and controls are in place from the start x the development life cycle is made visible and manageable x customer and supplier come together at the defined stages SMEs Quality Assurance Course 17-19 September 2001 .Applying ISO 9001 to software Software development.

maintenance) release and maintenance procedures/records specific support activities (CM and DM. test. use of customer supplied s/w. Supply. training) SMEs Quality Assurance Course 17-19 September 2001 .Applying ISO 9001 to software (cont’d) x ISO 9000-3 “Guidelines for the application of ISO 9001 to the Development. purchasing quality records measurement. development. CM. and Maintenance of Software” provides guidance on: – – – – – software “life-cycle” activities and reviews requirements specification plans (quality. verification.

TickIT is a British industry specific standard for auditing software companies to the ISO 9001 standard using the ISO 9000-3 guidelines x SMEs Quality Assurance Course 17-19 September 2001 .Applying ISO 9001 to software (cont’d) x When looking for third party certification. it is important that the identified auditor is software knowledgeable and that he looks for good software development practices around key software processes.

Problems with ISO 9000 (%) Time required to write the manuals High volume of paperwork High Cost of implementation Time required to complete implementation High cost of maintain the standard Lack of free advice Lack of consistency between auditors Time spent checking paperwork prior to audits The vagueness of the standard Difficulty interpreting the standard 31 27 25 24 19 18 18 16 12 12 SOURCE: MANCHESTER BUSINESS SCHOOL SMEs Quality Assurance Course 0 20 40 17-19 September 2001 60 80 100 .

Reasons for Seeking ISO 9000 (%) Future customers likely demand for ISO 9000 Increase consistency of operations Maintain/improve market share Improve service quality Customer pressure ISO 9000 is a good ‘promotional tool’ Make operations more efficient Improve product quality Proof of ‘TQM’ Cost reductions 78 65 61 61 58 57 54 52 34 22 SOURCE: MANCHESTER BUSINESS SCHOOL SMEs Quality Assurance Course 0 20 40 17-19 September 2001 60 80 100 .

Benefits Resulting from ISO 9000 % Increased exports Increased market share Reduced costs More effective marketing Reduced wastage Increased productivity/ efficiency Increased opportunity to win work A motivated workforce Improved customer satisfaction Improved management control 24 49 50 52 60 60 62 61 82 83 SOURCE: MORI SMEs Quality Assurance Course 0 20 40 17-19 September 2001 60 80 100 .

ISO 9000 COSTS x x x x x x x Internal resources ISO 9000 consultants (if required) Application fee Optional pre-assessment Assessment Maintenance of system Surveillance visits SMEs Quality Assurance Course 17-19 September 2001 .

Quality Assurance: standards evolution q ISO quality standards Military standards Space standards: q NASA standards q European standards q q SMEs Quality Assurance Course 17-19 September 2001 .

The fate of QA military standards x In the early nineties the U. DOD embarked on a new policy which can allow contractors to use their existing quality system standard (Performance-Based Contracting. Single Process Initiative) In October of 1996.S. MIL-Q-9858A and MIL-I-45208A where cancelled In February 1993. NATO canceled the AQAP 1-15 documents and adopted a new series of AQAP’s that incorporate the ISO 9000 series x x SMEs Quality Assurance Course 17-19 September 2001 .

Quality Assurance: standards evolution q ISO quality standards Military standards Space standards: q NASA standards q European standards q q SMEs Quality Assurance Course 17-19 September 2001 .

E E E C o m p o n e n t s M a t e r ia ls /P r o c e s s e s S y s t e m s S a f e t y S o f t w a r e A s s u r a n c e C o n f ig u r a t io n M a n a g e m e n t SMEs Quality Assurance Course 17-19 September 2001 .NASA-Mission Assurance Mission Assurance is defined by NASA as a set of specialised disciplines to support the project development M R is k M IS S IO N e n t A S S U R A N C E a n a g e m D E S IG N A S S U R A N C E Q U A L IT Y A S S U R A N C E R e lia b ilit y /M a in t a in a b ilit y E n v ir o n m e n t a l C o m p a t .

National or International Voluntary and Consensus Standards .Corporate Standards .S. DOD Military Specifications also NASA Specifications are being phased out (NHB-5300.QA standards at NASA As for U.4/1-B/1-C). some indefinitely. The "hierarchy of choice" is: .NASA or Military Standards and Practices The new NASA system is called “Advanced Quality System (AQS)” SMEs Quality Assurance Course 17-19 September 2001 . but certain NASA or Military unique standards may be maintained.

NASA . unless essential to ensure mission safety and reliability.Advanced Quality System (AQS) NASA's goal is to eliminate unique processes/systems imposed on contractors shared with DOD or other Federal agencies. The new NASA business management model is based on three elements: Œ baseline quality management system  advanced risk mitigating practices Ž program/project-unique risk mitigation measures SMEs Quality Assurance Course 17-19 September 2001 .

ΠNASA baseline QM system
In 1998 NASA issued the Policy Directive NPD 8730.3 “NASA Quality Management System Policy (ISO 9000) and associated Management Instruction NMI 1270.3 to: x implement an ISO 9000 based Quality System x mandate ISO certification of all NASA Centers x impose ISO standards as contractual requirements (when appropriate and beneficial ) x require suppliers third-party certification (when appropriate and beneficial )

SMEs Quality Assurance Course

17-19 September 2001

 NASA advanced practices
Instead of prescribing detailed contractual requirements, NASA emphasises the identification and selection of those offerors who propose effective advanced practices: x Integrated Product/Process Development (IPPD) x Identification and Control of Key Characteristics x Design to Manufacturing Process Capability x Design for Assembly/Manufacturing (DFA/M) x “Robust” Design x Geometric Dimensioning and Tolerancing (GD&T) x Process Variability Reduction (PVR)

SMEs Quality Assurance Course

17-19 September 2001

 NASA advanced practices (cont’d)
xProcess Variability Reduction (PVR) xUse of stable, capable processes for product acceptance xControl of variation in the measurement system xRoot cause, closed loop corrective action xDeployment of AQS elements to subcontractors xContinuous Improvement (CI) NASA will base future procurements by increasing the rigor of contractors selection and placing more importance on past performance. In addition will place quality incentives tied to Advanced Quality metrics (e.g., Cost of Quality)
More about AQS? Take a look at: www.boeing.com./companyoffices/doingbiz/supplier/AQS_Guidelines.pdf

SMEs Quality Assurance Course

17-19 September 2001

Ž NASA unique measures
Many programs or projects will continue to have unique requirements for risk mitigation in assembly, manufacturing or operational environments.

SMEs Quality Assurance Course

17-19 September 2001

QA standards in european space activities
x

In 1965, CNES issued QA requirements for the satellite FR1 In 1975, ESRO published a Product Assurance Manual (QRA-31) In 1981, ESA published a generic space QA standard, PSS-01-20 In 1988, CNES issued a coherent series of space project management specification (SM) for the Ariane 5 development, including A-SM-50 for quality assurance

x

x

x

PSS

SMEs Quality Assurance Course

17-19 September 2001

QA standards in european space activities (cont’d)
x

In 1990, the preparation of a common QA standard was started by ESA and CNES for the HERMES project (later cancelled) In 1991, ESA published a complete revision of PSS-01-20 In April 1996, the first European space QA standard was issued, ECSS-Q20, which took to a certain extent into account in its formulation the ISO 9000 series At the end of 1999 the quality system at ESOC, the ESA space operation center, has been certified ISO 9001

x

x

x

SMEs Quality Assurance Course

17-19 September 2001

ECSS-Q-20A

<<>>

SMEs Quality Assurance Course

17-19 September 2001

Requirements flow-down and tailoring ECSS ECSS Standards Standards ECSS Organisation SOW SOW (tailored ECSS (tailored ECSS)) ESA Project Allocated Allocated & & Derived Derived Rqts Rqts Allocated Allocated & & Derived Derived Rqts Rqts Allocated Allocated & & Derived Derived Rqts Rqts Prime Contractor SMEs Quality Assurance Course 17-19 September 2001 .

GSE SMEs Quality Assurance Course 17-19 September 2001 .ECSS-Q-20A QA Programme Management QA General Requirements QA Requirements for Design and Verification ECSS-Q-20 ECSS-Q-20 QA Requirements for Manufacturing/Assembly/Integr. Testing QA Requirements for Acceptance and Delivery Operations Annex A.

assessment of alerts from Customer Handling. “as-built” “as-designed” vs verifications Critical Items Control: items for which major difficulties or uncertainties exist controlled to reduce risk Documentation and data control Quality records Stamp Control Traceability Metrology and Calibration Non-Conformance control system Alert system: participation to ESA AS: generation of alerts for items with multiple applications failing within specified design/usage limits. Preservation Statistical Quality Control: SQC applications used for acceptance to be approved by Customer 3. Quality Assurance Programme - 1/3 Summary/remarks Organisation Quality Assurance Programme Plan QA Status Reporting Personnel Training and Certification Quality Assurance Programme Audits QA role in Configuration Management: ensure existence/implementation of CM requirements. QA as a member of CCB. Storage. Quality Assurance General Requirements - - SMEs Quality Assurance Course 17-19 September 2001 . General 2.ECSS-Q-20A ECSS-Q-20 1.

QA Requirements for Design and Verification - 2/3 5. QA requirements for Manufacturing Assembly and integration Summary/remarks Design and verification planning Evaluation/qualification of new technologies and critical processes Clear organisation interfaces between different groups Design rules implemented to achieve producibility. shop-travellers) Identification of critical characteristics QA responsibilities (approval of documents etc. testability.) Manufacturing Readiness Review Control of processes Critical processes Statistical Process Control Workmanship standards Materials and Parts Control: only conforming items release. repeatibility.ECSS-Q-20A ECSS-Q-20 4. QA Requirements for Procurement 6. operability Standards and procedures for maintenance of engineering drawings and specifications Requirements verification and qualification process Design changes Selection of procurement sources Record and list of Procurement Sources Applicable requirements defined in procurement documents Surveillance of procurement sources Receiving inspection Planning of MAI activities Identification of MIPs Adequate instructions (e. inspectability. limited life items Equipment control Cleanliness and Contamination Control Inspection SMEs Quality Assurance Course 17-19 September 2001 .g.

items properly preserved and packaged. reporting of anomalies and problems Deviations from procedures to be justified Development and Configuration Control: depend on the criticality Production: as minimum ISO 9002 during MAIT Delivery/Maintenance SMEs Quality Assurance Course 17-19 September 2001 .ECSS-Q-20A ECSS-Q-20 7. Operations Annex A: Ground Support Equipment - 3/3 Summary/remarks Test facilities Test equipment Test documentation Test reports Test performance monitoring Test reviews Acceptance Data Package Delivery Review Board: to authorise shipment and certify conformity/completeness including documentation Preparation for delivery Delivery: shipping control. QA Requirements for Acceptance and Delivery 9. accompanying documentation included Quality of mission products Validation and Qualification of Ground Support Segments QA Plan for operations Training and Operator Certification Operations anomalies and Feedback corrective Loop. Testing 8.

Quality Assurance Part III : Special techniques SMEs Quality Assurance Course 17-19 September 2001 .

Statistical Techniques <<>> SMEs Quality Assurance Course 17-19 September 2001 .

Statistical techniques x Statistical process control Statistical acceptance sampling x SMEs Quality Assurance Course 17-19 September 2001 .

Statistical Process Control (SPC) x Statistical Process Control is used to: – maintain a process within pre-defined limits (evaluate process performance and correct) – determine the process capability x Statistical Process Control is based on the principle that: A PROCESS IS IN A STATE OF STATISTICAL CONTROL WHEN THE VARIATION OBSERVED IN ITS OUTPUT CAN BE ATTRIBUTED ONLY TO RANDOM CHANCE SMEs Quality Assurance Course 17-19 September 2001 .

Process performance and process capability x Process performance Is the day-to-day behavior of a process including all random and assignable-cause effects Process capability Is an estimate of the best performance of which a process is thought to be capable.CAPABILITY = POTENTIAL IMPROVEMENT x x SMEs Quality Assurance Course 17-19 September 2001 . with all effects of controllable. assignable-cause variation removed PERFORMANCE .

g. temperature variations. wearing. When each cause is held to its practical minimum the resultant process variability is called natural and follows a normal distribution Normal Distribution 68% 95% 99.). clearances. Natural process variability x Natural Process Variability Process variation is due to several causes (e. distortions.7% -3σ -2σ -1σ μ=0 +1σ +2σ +3σ 17-19 September 2001 SMEs Quality Assurance Course . measurement error etc.

Detecting assignable causes x Assignable causes are variations above the minimum. in order of production sequence) +3σ x x mean x -3σ x x x x x x x x Production Sequence SMEs Quality Assurance Course 17-19 September 2001 ..e. They can be detected by periodic sampling of the process and plotting data such to preserve the timehistory (i.

Detecting assignable causes (cont’d) x Most probably an assignable cause exist when there are: – – – – – one or more points outside +/-3σ 2 or 3 successive points between 2-3σ(same side) 4 or 5 successive points beyond +/-1σ(same side) 8 successive points in either 1σ band 8 successive rising and falling trends SMEs Quality Assurance Course 17-19 September 2001 .

R is the average of R. and d2 is a correction factor depending on sample size SMEs Quality Assurance Course 17-19 September 2001 . The range “R” is the excursion between maximum and minimum values within the sample Variation between samples reflect variation due to inherent process variability plus assignable causes (to the extent they may exist) x σ’ = R/d2 where σ’ (sigma-prime) is an “estimated” value of σ. 5 consecutive measurements in a short interval) reflect inherent process variability.g.Estimating process sigma x Variation within a sample (e.

Calculation of 3-sigma limits (subgroups) x For X chart: UCL = X + A2 x R LCL = X .A2 x R x For R chart : UCL = R x D4 LCL = R SMEs Quality Assurance Course x D3 17-19 September 2001 .

I2 xxR x A2. I2.Calculation of 3-sigma limits (single values) x For X chart: UCL = X + I2 x R LCL = X . D3. D4 are correction factors depending on subgroup size SMEs Quality Assurance Course 17-19 September 2001 .

326 2.777 D4 2.534 2.483 0.267 2.078 1.574 2.693 2.880 1.816 1.023 0.373 0.184 0.076 0.308 A2 0 0 0 0 0 0.223 D3 3.729 0.CORRECTION FACTORS Subgroup Size (n) 2 3 4 5 6 7 8 9 10 d2 1.282 2.88 I2 SMEs Quality Assurance Course 17-19 September 2001 .337 0.004 1.704 2.924 1.128 1.114 2.577 0.059 2.847 2.136 0.970 3.864 1.419 0.

60 +/-0.003 SMEs Quality Assurance Course 17-19 September 2001 .Example: Calculation of 3-sigma limits (subgroups) Example Characteristics: 0.

Process capability index Cp = tolerance width process capability = USL-LSL 6σ’ -C p above 1. .C p below 1.33 Process adequate Process inadequate .00 SMEs Quality Assurance Course 17-19 September 2001 .

Process control and capability USL UCL Target value LCL LSL Process out of control but within specification USL Target value UCL LSL X LCL Process in control but out of specification SMEs Quality Assurance Course 17-19 September 2001 .

Cpk x The capability index Cp does not take into account the centering of the process. The ratio that reflects how the process is performing in terms of nominal center. or target value is Cpk Cpk = Z(min)/3 where Z(min) is the smaller of : Z(USL) = USL-X σ’ SMEs Quality Assurance Course or Z(LSL) = X-LSL σ’ 17-19 September 2001 .

3 sigma) to 3.“Six-sigma” x Six-sigma is a strategy developed by the Motorola Company to achieve in their processes: Cp = 2 x The main assumption is a “typical” centering error of +/.5 sigma.6 sigma) SMEs Quality Assurance Course 17-19 September 2001 .807 defects per million (for +/.1.4 defects per million (for +/. It means passing from 66.

engineering.) y Identify the product elements that influence the critical characteristics z Identify relevant process elements { Establish maximum tolerance for each product and process element | Determine actual capability x The Motorola Six-Sigma quality initiative is also a motivational and training initiative 17-19 September 2001 SMEs Quality Assurance Course .“Six-sigma” (cont’d) x The six-sigma implementation steps are: x Identify critical characteristics (through marketing. R& D etc.

Statistical techniques x Statistical process control Statistical acceptance sampling  SMEs Quality Assurance Course 17-19 September 2001 .

otherwise rejected and controlled 100% x SMEs Quality Assurance Course ^ 17-19 September 2001 . lot accepted. Accept/reject entire lot based on the results of sample inspection Single sampling plan:  N = lot size  n = sample size  c = acceptance number  d = number of defective items in sample  p = % of defective items in lot d c .Statistical acceptance planning x Purpose of acceptance sampling is to avoid 100% inspection.

It should lead to rejection most of the times (usually Pa=0.95) LFTD = (Lot Tolerance Fraction-Defective Level). the percentage of defective items in the lot that represents a “bad quality level”.10) α = risk for the producer that lots at AQL are rejected probability x x x ß = risk for the consumer that lots at LTFD are accepted SMEs Quality Assurance Course 17-19 September 2001 .Producer’s and consumer’s risk x AQL = (Acceptable Quality Level). the percentage of defective items in the lot that represents a “good quality level”. It should lead to acceptance most of the times (usually Pa=0.

Operating Characteristic (OC) Curve x A good sampling plan is one that effectively identifies the lots that can be accepted and those needing 100% inspection. Pa ideal shape 1-α n . c affect curve shape ß p AQL LTFD 17-19 September 2001 SMEs Quality Assurance Course . A tool to support the selection is the Operating Characteristic (OC) Curve.

accept number of defective upper limit. reject number of defective between limits. 2.Double and multiple sampling plans x Double sampling x y z { number of defective lower limit. as above { continue sampling until accept or reject is achieved SMEs Quality Assurance Course 17-19 September 2001 . take new sample new accept/reject limits for the 2 samples ^ ^ x Multiple sampling .steps 1. 3.

Passing from normal to reinforced the acceptance number is reduced but sample size remains the same.4-1993 Includes simple. the sample size is reduced x x SMEs Quality Assurance Course 17-19 September 2001 . Passing from normal to reduced. double and multiple plans Includes normal.MIL-STD-105 x Widely used for acceptance sampling by attribute since 1943 – Rev A 1950. last Rev E 1989 – Cancelled Feb. reinforced and reduced plans to deal with lots which are continuously received. 1995 – Replaced by ANSI/ASQC Z1.

Cost of Quality <<>> SMEs Quality Assurance Course 17-19 September 2001 .

Cost of Quality COQ = PREVENTION + APPRAISAL + FAILURE COSTS. or COQ = CONFORMANCE COST + NONCONFORMANCE COST Quality cost measurements provide guidance to the quality management program much as the cost accounting system does for general management SMEs Quality Assurance Course 17-19 September 2001 .

Cost of Quality: Influence of Quality Policy Total COQ vary with emphasis placed on prevention and appraisal. Distribution : Inspected-in Quality 60% 50% 40% 30% 20% 10% 0% Failure Prevention Appraisal 60% 50% 40% 30% 20% 10% 0% Distribution : Built-in Quality Failure Prevention Appraisal TOTAL COQ = 25-40% TOTAL COST TOTAL COQ = 5-10% TOTAL COST SMEs Quality Assurance Course 17-19 September 2001 .

Steps to ensure producible/robust design .Training .Process qualifications .Process capability evaluations .Quality planning .Cost of Quality: prevention costs Prevention cost is the cost allocated to prevent the occurrence or recurrence of nonconforming products: .Quality improvement projects SMEs Quality Assurance Course 17-19 September 2001 .Metrology and calibration .System audits .

Product and process audits SMEs Quality Assurance Course 17-19 September 2001 . Appraisal costs are those associated with measuring. or audit: .In-process and final inspection/test . evaluating.Design reviews .Cost of Quality: appraisal costs Appraisal cost is the cost of judging the acceptability of products and to detect nonconformances.Incoming and source inspections .

Investigating causes .Partial or full loss of the effort already expended . They are divided into internal costs incurred prior to product delivery.Cost of Quality: failure costs Failure costs are costs resulting from nonconforming products. and external costs incurred after product delivery: .Processing of customer reports .Determining corrective actions SMEs Quality Assurance Course 17-19 September 2001 .Warranty claims .Additional efforts to make the product acceptable .

Cost of Quality: failure costs-scrap x Cost of additional material to replace unusable material Labor cost for manufacturing of replacement material Labor cost for non-manufacturing personnel for actions coordination Productivity loss due to rescheduling of manufacturing activities Storage cost for extra inventory x x x x SMEs Quality Assurance Course 17-19 September 2001 .

Cost of Quality: failure costs-rework/repair x Labor costs for manufacturing personnel performing rework/repair Labor cost for non-manufacturing personnel for actions coordination Productivity loss Costs of materials involved in repairing Costs of extra handling/transportation Cost of extra rework/repair inspection Administrative costs of waivers processing (for repairs) 17-19 September 2001 x x x x x x SMEs Quality Assurance Course .

Summers.New model of optimum quality costs NEW OLD Source : “Quality”. Prentice Hall SMEs Quality Assurance Course 17-19 September 2001 . D.

D. Source : “Quality”. Failure costs are the tip of an “iceberg”. Summers.Hidden quality costs There is a “multiplier effect” between failure costs and “true failure costs”. Prentice Hall SMEs Quality Assurance Course 17-19 September 2001 .

Secchi <<>> SMEs Quality Assurance Course 17-19 September 2001 .Alert systems by P.

Alert systems x What is an Alert? Types of Alert systems GIDEP NASA Part Advisories NASDA Alert System CNES Groupe d’Alertes ESA Alert System 17-19 September 2001 x x x x x x SMEs Quality Assurance Course .

if no preventive actions are taken Alert report The alert report will include a description of the observed problem. its cause. the actions to be taken to correct it and to prevent recurrence. or may re-occur in other projects.What is an alert? Alert Alert is a report used to provide a prompt warning on failures and problems which may affect more than one user. comments from the manufacturer SMEs Quality Assurance Course 17-19 September 2001 .

Types of Alert Systems x Government systems Parts Advisory Official Business U.S. Government x Industry systems astrium SMEs Quality Assurance Course 17-19 September 2001 .

S. Government and industry participants.GIDEP Government-Industry Data Exchange Program x A co-operative activity between U. Government Participating organisations: – US Government Agencies and Organisations (including NASA) – Industrial organisations producing parts. components and equipment for the government x SMEs Quality Assurance Course 17-19 September 2001 . managed and funded by the U.S.

including ALERTs. Problem Advisories and Agency Action Notices Reliability and Maintainability data Metrology data Product information data Urgent data request SMEs Quality Assurance Course 17-19 September 2001 . SAFE-ALERTs.GIDEP (cont’d) GIDEP participants are provided electronic access to six major types data: x x x x x x Engineering data Failure experience data.

not to criticize a manufacturer First NASA Part Advisory is dated 08/29/90     Parts Advisory Official Business U.NASA Parts Advisories  Intentionally called “Advisory” to distinguish them from GIDEP Alerts and TWX Alerts from other NASA Centers Original intention was a rapid “heads-up” warning vehicle Provides information and guidance but is not intended to require mandatory action – emphasis on flexibility To inform NASA of a significant risk. Government SMEs Quality Assurance Course 17-19 September 2001 .S.

NASDA Alerts are called Reliability Technical Information(RTI). An RTI describes: – observed failure/problem and its root cause – measures to be taken to correct it and to prevent its recurrence Over 100 RTIs have been issued so far.NASDA Alert System x x x x x The NASDA Alert system has been being established since 1974. Is operated by the Safety and Reliability Department (SRD) of NASDA. SMEs Quality Assurance Course 17-19 September 2001 .

CNES Groupe d’Alertes x x x x x x x Started in 1989 A Co-operation between CNES and French Space Industry Group meets every three months Report prepared and issued every three months “Anomalies” classified into three levels ESA receives CNES Alert Reports CNES is a registered user of the ESA Alert System SMEs Quality Assurance Course 17-19 September 2001 .

ESA Alert System Operated by the ESTEQ TOS-QQ Division to: x Facilitate exchange of information on problems experienced in ESA projects Eliminate or minimise their impacts Prevent recurrence on other projects Enhance competitiveness of European space industry by avoiding waste and mistakes x x x SMEs Quality Assurance Course 17-19 September 2001 .

aviation & military (CAM/COTS) equipment SMEs Quality Assurance Course 17-19 September 2001 .ESA Alert System (cont’d) ESA Alerts cover failure related to: x x x x x x x x EEE parts mechanical parts pyrotechnic devices materials test equipment software used by several users equipment procured against a supplier’s specification commercial.

ESA Alert System (cont’d) ESA Alert System covers also problems with: x x x x x x x safety manufacturing processes handling procedures standard test methods standard operational procedures software development & test methods and tools continuity of production of an item SMEs Quality Assurance Course 17-19 September 2001 .

Criteria to issue an ESA Alert ESA Alerts are only issued when: x the observed problem may apply to more than one project or organisation and the problem was observed while the item was applied within its specified limits and a preliminary investigation has provided evidence of the root cause of the problem and the problem is confirmed not to be of a random nature x x x SMEs Quality Assurance Course 17-19 September 2001 .

Published ESA Alerts ESA has published a total of 42 Alerts so far: x x x x x x x 30 on EEE parts 5 on materials 3 on mechanical parts or mechanisms 1 on manufacturing processes 1 on test equipment 1 on flight equipment 2 on software product SMEs Quality Assurance Course 17-19 September 2001 .

ESA Alerts processing Fill PAI . Designate experts Evaluate PAI Alert? YES No Close case Send PAI to manufacturer Evaluate reply N O YES Publish Alert confirmed? Alert SMEs Quality Assurance Course 17-19 September 2001 .

ESA Alerts distribution ESA technical departments ESA contractors ESA Projects SMEs Quality Assurance Course 17-19 September 2001 .

ESA Alert focal point x x x x x Co-ordinate Alert preliminary investigation Organise & support proceedings of ESA Alert Committee Publish Alerts on the web Follow-up of actions to be implemented by manufacturer Register new Users to the System SMEs Quality Assurance Course 17-19 September 2001 .

Alert co-ordinators Alert co-ordinators are the interfaces between company and ESA. Large companies with different geographical locations have more than one Alert co-ordinator PAIs feedback/comments Alert Co-ordinator Alerts ESA Alert Focal Point SMEs Quality Assurance Course 17-19 September 2001 .

Technical experts role x x x x Review Preliminary Alert Information Assess failure/problem against criteria for issuing an alert Define recommended actions to: – to solve failure/ problem – to prevent recurrence Assess: – Manufacturer’s response – Alert corrective actions – Feedback from users Recommend whether to issue Alert or not SMEs Quality Assurance Course 17-19 September 2001 x .

It allows to: x browse ESA Alerts x search ESA Alerts x request Access x submit PAI on-line x submit feedback/comments on-line Have a look at: www.nl/qq/alerts SMEs Quality Assurance Course 17-19 September 2001 .esa.estec.ESA Alert System Web Access restricted to registered participants only.

Carranza SMEs Quality Assurance Course 17-19 September 2001 .Quality Assurance Part IV : S/W Product Assurance by J.

S/W PA basic concepts <<>> SMEs Quality Assurance Course 17-19 September 2001 .

QA versus SW QA x x x x x There is and should not be separation of the two SW QA is an aspect or a branch of QA Techniques used in SW QA are the same as those in QA Emphasis on techniques and methods is different in SW because of differences between essential properties of SW and HW However there is nothing special about SW QA only differences SMEs Quality Assurance Course 17-19 September 2001 .

cannot be applied – statistical measurements are applicable in a limited fashion only Often there is no immediate direct relation between the apparent importance of the fault and the importance of the failure that it may produce SMEs Quality Assurance Course 17-19 September 2001 .Properties of SW x x x SW is highly volatile and can be changed completely at the touch of a button All faults in SW are systematic. etc. Therefore: – statistical control.

techniques and methods evolve ⇒ need to adapt very quickly.Characteristics of SW engineering x x x Its implantation as an engineering discipline is weaker than in other engineering disciplines due to: – its relative “youth” as an engineering discipline compared to others – the very quick pace at which technology. lack of consolidation It is difficult to separate new from old: everything from dinosaurs to primates are alive and living together It is difficult to separate good from bad: – lack of historical information (experience) from technologies/methods/tools – high commercial pressure (rapidly evolving and aggressive tools market) SMEs Quality Assurance Course 17-19 September 2001 .

Key issues in SW QA x x x x x Configuration management: – identification – baseline – change control – traceability Verification: – Not only testing ⇒ many more techniques and methods – At all stages of development from requirements to operation Metrics Criticality Evolution of quality assurance concepts SMEs Quality Assurance Course 17-19 September 2001 .

Configuration management for S/W x Configuration management in SW is the same as in other disciplines: – Configuration identification: how to identify items that are part of the project – Configuration control: how to change the configuration – Status accounting: what is the state of the items The following will describe some aspects of the above that are of special importance in SW within the above fundamentals x SMEs Quality Assurance Course 17-19 September 2001 .

CM: identification
x

x

x

x

x

Volatility of SW allows very easy creation, deletion or modification of SW items Volatility makes it important to know which items are part of the system and which are not It is all too easy to become confused about what is part or not of the SW development project or what is valid or not SW systems are composed of many custom made components: complexity and lack of standardisation Configuration Item identification must be planned before starting to create units

SMEs Quality Assurance Course

17-19 September 2001

CM: baseline
x

x

x

x

Although modern SW technology tries to reduce interdependencies between components, SW systems are very tightly linked systems Consistency is critical: change in A requires change in B otherwise the system will not operate properly Baselines are consistent and complete sets of all items that make up the system and allow co-ordination across the team Baselines are made accessible as libraries or repositories

SMEs Quality Assurance Course

17-19 September 2001

CM: baseline (cont’d)
Baselines grow along lifecycle as the project advances, for example:

Specifications, management documentation Design, specification verification

Software items, design items, test items, user documentation

SMEs Quality Assurance Course

17-19 September 2001

CM: baseline evolution

x x x

Baselines are created more often for SW than for systems Baselines must be planned and must have a clear purpose Specially important baselines are: – inputs to formal reviews – outputs from formal reviews (often ignored) – test baselines – deliveries

SMEs Quality Assurance Course

17-19 September 2001

CM: change control

x

x x

x

Complexity of SW makes it difficult to assess the impact of changes in one part of the system on another Individuals in a team many not be able to judge that impact Change control is a mechanism to allow a baseline to change into another Key concepts: – authority to change (often not the engineer) – entry of a configuration item under configuration control – check points in change implementation – all of the above adequately sized for the project

SMEs Quality Assurance Course

17-19 September 2001

CM: change control (cont’d)
x

x

x

x

Change control is triggered by non-conformances (Software Problem Reports) and change requests SW changes much faster than HW ⇒ efficient change control procedure, normally separate from HW procedure In system’s development SW change control will need formal interfaces to System’s Non-conformance handling and System’s Change Requests One key point: once a configuration item is under change control it is not the “property” of the author any more

SMEs Quality Assurance Course

17-19 September 2001

CM: importance of traceability

x

x

Baselines are not formed only by SW components. They include also: – specification (requirements) – design – management documentation – verification items (verification records, testing tools, etc.) – user documentation, etc. Traceability is a fundamental tool to ensure completeness and also consistency of baselines

SMEs Quality Assurance Course

17-19 September 2001

CM: importance of traceability (cont’d) x Basic traceability: Requirements Design components SW items System test items SMEs Quality Assurance Course Integration test items Unit test items 17-19 September 2001 .

consistency. correctness) Validation Validation: activity to ensure that a product of services is adequate for its intended use (use related) SMEs Quality Assurance Course 17-19 September 2001 .Verification: definitions Verification Verification: activity to ensure that the outputs of a process conform to its inputs (completeness.

etc. – coding: code review. unit testing. how do I test the SW change?) SMEs Quality Assurance Course 17-19 September 2001 . – design: walkthrough. etc. modelling. – operation: non-regression testing. inspection. etc. review. Most of the above verification activities are supported by traceability (if requirement x changes. etc.Verification: phases x x x Verification is not limited to testing Verification takes place at each phase of the software lifecycle: – requirements specification: formal review.

Verification: scope x x Verification covers more than just the SW Items that should or may be verified are: – SW itself – implementation of processes – tools – infrastructure SMEs Quality Assurance Course 17-19 September 2001 .

testing). walkthrough – Audit Verification techniques may be intended for products (e.Verification: techniques x x x Different methods are used in SW The main ones are: – Testing – Formal review – Inspection.g. processes (maturity assessments) or both (formal reviews) SMEs Quality Assurance Course 17-19 September 2001 .

Testing: levels x x Testing is performed at different levels Levels of testing have different targets and objectives: Level Unit Integration System Acceptance Target Source code units Interfaces Software Specifications User or system specifications x The different levels may have different names or be organised in slightly different ways SMEs Quality Assurance Course 17-19 September 2001 .

ignoring the internal logic ⇒ black box testing Black box testing can be of different types also: – nominal: test for normal conditions – back to back: same test on two or more versions of SW written with same specifications – interface: written on the basis of Interface Control Documents – stress: test under extreme or abnormal conditions – statistical: test data selected according to distribution functions SMEs Quality Assurance Course 17-19 September 2001 . with specific knowledge of the internal logic ⇒ white box testing Testing can be aimed to test functionality at different levels.Testing: types x x x Testing can be aimed to test implementation.

g. etc. branch. architectural design. etc.g.) Goals for black box testing will depend on its objective (e. software requirements. path.) and specific requirements (e. test specific non-nominal conditions) SMEs Quality Assurance Course 17-19 September 2001 .Testing: limits x x x x Deciding when to stop testing (other than running out of time) is not easy Goals must be set beforehand to ensure sufficient and uniform testing Goals for white box testing are typically based on code coverage (statement.

Testing: verification x x x x Testing itself is subject to verification This will typically involve specific reviews of the test plan and test designs. cases and procedures It may involve additional verification: e.g. mutation testing Coverage and status of the testing must be controlled formally: – Verification Control Document for requirements or design components – Coverage reports for white box testing SMEs Quality Assurance Course 17-19 September 2001 .

Testing: baselines change x x x x x x Baselines are crucially important in SW testing Any level or type of testing start from a properly controlled baseline ⇒ baseline identification (contents and versions) Testing is performed in well defined cycles. each cycle using a single baseline Problems found during testing are processed through the change control procedure Implementation of changes leads to a new baseline which is input to the next cycle of testing Do not give in to the temptation of changing along the way SMEs Quality Assurance Course 17-19 September 2001 .

Testing: baselines change (cont’d) x x x x x Changes during testing are localised ⇒ testing should be adapted to the change ⇒ non-regression testing Non-regression testing should be performed from the lowest level up to the level at which the corresponding fault was found Traceability is an invaluable tool to define non-regression testing (e. must be taken into account Automate test where possible SMEs Quality Assurance Course 17-19 September 2001 .g. etc. which units do I test if a test for software requirement x failed) Dependencies between components.

Verification: other techniques x Other verification techniques are very similar to general verification techniques: – formal reviews: sometimes SW integrated in system reviews – audits – inspections: specific methods for SW.g. Fagan – internal or informal reviews: very important in SW. I cannot see my own mistakes SMEs Quality Assurance Course 17-19 September 2001 . e.

Metrics x x x x Metrics provide quantitative information on products or processes Metrics can be used for: – project purposes: controlling the development processes – organisation purposes: historical data Project metrics must have goals and associated reaction mechanisms to react to goals not being reached An example: – metric: cyclomatic complexity – goal: < 10 – reaction: re-design module SMEs Quality Assurance Course 17-19 September 2001 .

g. SPEC) SMEs Quality Assurance Course 17-19 September 2001 .Project Metrics x x x x As important and metrics collection is reporting ⇒ metrics reports must be addressed to the people who must initiate the reaction Metrics collection and reporting should be supported by tools as much as possible Metrics should be defined on the basis of the most relevant properties of the specific SW being developed: Portability? Maintainability? Efficiency? Reliability? Quality models are the mechanism to define a consistent and useful metrics program (e. ISO 9126.

design complexity metrics from different models may not be comparable) They are only useful in the long term SMEs Quality Assurance Course 17-19 September 2001 .Organisation metrics x x x x Their objective goes beyond the project ⇒ no immediate apparent benefit for the project They rely on an organisation at a higher level than the project ⇒ metrics from different project must be collected and analysed They have to be highly independent of technology (e.g.

Some metrics examples x x Product metrics: – code size: number of lines of code – design size: number of design components – code complexity: cyclomatic complexity – design complexity: depth of inheritance tree Process metrics: – actual effort against estimated effort – problems detected during system testing – rate of problems reported during operation SMEs Quality Assurance Course 17-19 September 2001 .

Criticality: issues x x x x x A typical SW system will perform different functions some of which may be more critical (safety. etc.) than others SW designs tend to be more and more complex every day It is not obvious how relevant different SW components are to the performance of the more critical functions Availability or resources is limited It is better to concentrate efforts on the most important functions ⇒ most important components SMEs Quality Assurance Course 17-19 September 2001 . mission.

verification) on most critical components SMEs Quality Assurance Course 17-19 September 2001 .Criticality: system and SW x x x x x Criticality of SW integrated in a system is always dependent on criticality of the system functions The criticality of the functions at system level is analysed and the results of those analyses is fed into parallel analyses at SW level There is feedback from SW to system level analyses including the analysis of HW and SW interaction Criticality of functions is translated into criticality of software components: traceability again!! Concentrate resources (e.g.

Quality system evolution: the S/W perspective Artisan Team Defined methods Organization Quality system Best practices Standards Continuous process improvement SMEs Quality Assurance Course 17-19 September 2001 .

The evolution: s/w “artisan” x x x Individuals working independently even within a team are still very common Often particularly able “artisans” will compensate resources or skills missing in the organisation “Artisans” carry with them essential skills an experience that is often impossible to replace SMEs Quality Assurance Course 17-19 September 2001 .

The evolution: s/w team x x x Most SW development projects require a team effort nowadays Teams are often formed by “artisans” Teams rely heavily on specific individuals. in particular the team leader SMEs Quality Assurance Course 17-19 September 2001 .

etc. etc. Proper use of those methods brings uniformity into SW development and allows communication across the team Build up of expertise in the use of methods ⇒ often it is more important expertise in a method than how “good” or “new” it is: e. maintenance tools.The evolution: defined methods x x x x Nowadays established methods and tools exist for all activities involved in SW development For example object orientated methods.. etc.g. modelling tools.. many projects keep stumbling on their initial attempts at object orientation SMEs Quality Assurance Course 17-19 September 2001 . code generation tools.

This flexibility makes use of resources and skills more efficient SMEs Quality Assurance Course 17-19 September 2001 .The evolution: organisation x x x x Organisations are based on roles. not specific individuals This gives them a higher independence on staff turnover Roles allow separation of skills which facilities training Roles and individuals can be mapped flexibly.

The evolution: QA system

x

x

x

The QA system provides the framework that defines the roles, processes and methods used in the organisation In the case of SW development this is at least as important as in other engineering disciplines A generic QA system may not be enough for SW ⇒ a certain level of detailing and adaptation may be needed for SW

SMEs Quality Assurance Course

17-19 September 2001

The evolution: best practices and standards

x

x

x

x

Standards provide a common “vocabulary” for SW engineering (e.g. ISO 12207) They can be incorporated into the conception of the QA system allowing to map the organisation’s processes and methods directly to the standard allowing for automatic compliance Problem: “The nice thing about standards is that there are so many of them to choose from” ⇔ multiple customers/standards Standards are often a channel to promote best practices across industry

SMEs Quality Assurance Course

17-19 September 2001

The evolution: best practices and standards (cont’d)

x x

x

x

In the case of some standards it is possible to obtain a certification The most common one is ISO 9001 under a number of SW or IT specific schemes TickIT is one example: – supported by UK and Swedish software industry – recognised by British government – provides assessment and certification services but also guidance ISO 9000-3: guidance on applying ISO 9001

SMEs Quality Assurance Course

17-19 September 2001

The evolution: software process improvement
x x

x

x

x

Best practices of today may not be so tomorrow Methods, processes and their performance are always susceptible of improvement This improvement increases efficiency, maturity, reliability and competitiveness To improve a process or method it is necessary to be able to compare current and past performance ⇒ quantitative control (e.g. organisation metrics) Reliable improvement must be based on historical data

SMEs Quality Assurance Course

17-19 September 2001

The evolution: software process improvement (cont’d)
x

x x x x

Software Process Improvement must be institutionalised and an organisation-wide process in itself SPI must be supported by adequate organisation and resources SPI must be managed SPI must be promoted by higher management SPI has a cost and benefits

SMEs Quality Assurance Course

17-19 September 2001

Major changes of focus

x x x

x

From individual to team From individual to role From product to process but at the same time from quality assurance to product assurance However examples of all of the above steps of evolution co-exist nowadays in the market (dinosaurs and primates) ⇒ difficulty in differentiating them

SMEs Quality Assurance Course

17-19 September 2001

Space S/W standards

<<>>

SMEs Quality Assurance Course

17-19 September 2001

Old ESA SW standards x x x x Previously ESA used the PSS series of standards Two documents in the series were the most relevant to software: – ESA-PSS-05-0: ESA Software Engineering Standards. April 1991 ESA-PSS-05-0 is complemented by a set of guides. February 1991 – ESA-PSS-01-21: Software Product Assurance Requirements for ESA space systems. issue 2. ESA -PSS-05-x They consider software only (no system) SMEs Quality Assurance Course 17-19 September 2001 . issue 2.

13 April 1999 – ECSS-Q-80B. Space Product Assurance .Software.Software Product Assurance. 3 April 2000 – ECSS-Q-80B draft. 19 April 1996 New draft versions are available and are the ones used at this time.Software Product Assurance. Space Product Assurance . 3 April 2000 The new versions resolve a number of gaps between the previous versions and adapt to evolution of SW SMEs Quality Assurance Course 17-19 September 2001 . Space Engineering . Space Engineering . although pending formal publication: – ECSS-E-40B draft.New ECSS for SW x x x The main documents are: – ECSS-E-40A.Software.

Software process assessment – ISO 9126 Information Technology .1993 IEEE Standard recommended practices for software acquisition SMEs Quality Assurance Course 17-19 September 2001 .Alignment with international standards x The new versions of E-40 and Q-80 are based or aligned with a number of key international SW standards: – ISO/IEC 12207:1995 Information Technology .1990 IEEE Standard glossary of software engineering terminology – IEEE 1062 .Software lifecycle processes – ISO/IEC 15504:1998 Information Technology .10 .Software product evaluation quality characteristics and guidelines for their use – IEEE 612.

Alignment with international standards (cont’d) x The alignment of E-40 and Q-80 with international standards makes them: – internationally “readable” since they use recognised vocabulary and models – simultaneous compliance with other standards easier – easier evolution in parallel with those standards SMEs Quality Assurance Course 17-19 September 2001 .

references are made instead to other standards: e. project management SMEs Quality Assurance Course 17-19 September 2001 .SW ECSS: characteristics x x x x x The standards are based on the relation between SW and the system of which it forms part The standards are based on the concept of process as a set of activities that contribute to the development lifecycle: e.g.g. configuration management or design The standards are based on a customer-supplier model The model is recursive: the supplier may be the customer of next level suppliers The standards do not duplicate basic information which is general to the system.

Documentation x x Differently from ESA-PSS-05-0 documents are not dictated ECSS refers to six collections of information: – Requirements baseline (E-40) – Technical specification (E-40/Q-80) – Design definition file (E-40) – Design justification file (E-40/Q-80) – Product assurance file (Q-80) – Maintenance file (Q-80) SMEs Quality Assurance Course 17-19 September 2001 .

Documentation (cont’d) x x These collections contain all the work products of the project like plans. etc. technical documentation. requirements document. design documents. The standards do not define tables of contents for documents either SMEs Quality Assurance Course 17-19 September 2001 . reports.

SW processes SMEs Quality Assurance Course 17-19 September 2001 .

Critical Design Review and Acceptance Review SMEs Quality Assurance Course 17-19 September 2001 . Preliminary Design Review.SW engineering x x x Life-cycles are not enforced and are mapped into: – system engineering – requirements engineering – design engineering – validation and acceptance The above engineering processes may overlap to some extent Major reviews mark the end of each of them: System Requirements Review.

ECSS-E-40 requirements x Requirements are established for the following processes: – system engineering processes related to SW – software requirements engineering – software design engineering – software validation and acceptance – software operations engineering – software maintenance – software verification and validation (supporting process) SMEs Quality Assurance Course 17-19 September 2001 .

ECSS-E-40 requirements (cont’d) x Additional requirements exist for specific types of SW or development: – space segment – ground segment – SW reuse – man-machine interfaces – critical SW SMEs Quality Assurance Course 17-19 September 2001 .

ECSS-Q-80 requirements x x x Programmatic requirements on SW product assurance: – organisation – responsibilities – planning – management Software process assurance: – lifecycle – software development processes Software product assurance: – quality objectives and metrication – product quality requirements SMEs Quality Assurance Course 17-19 September 2001 .

PDR] SMEs Quality Assurance Course 17-19 September 2001 . EXPECTED OUTPUT: a.ECSS-E-40 example Clause 5. Preliminary (top-level) internal interfaces design [DDF. PDR] b.4. Preliminary (top-level) external interfaces design [ICD(TS).3.4 The supplier shall develop and document a top-level design for the interfaces external to the software item and between the software components of the software item.

5 The supplier shall provide with his software assurance plan a compliance matrix documenting his compliance with the software product assurance requirements applicable for the project/contract.1.3. SRR] SMEs Quality Assurance Course 17-19 September 2001 .ECSS-Q-80 example Clause 5. EXPECTED OUTPUT: Compliance matrix [PAF.

S/W Process Capability Assessment <<>> SMEs Quality Assurance Course 17-19 September 2001 .

Software Process Assessment x x x Quality of software is directly related to the quality of the development processes carried out to produce the software Measuring the capability of the processes carried out by an organisation to develop software provides key information on the ability of that organisation to produce good quality software consistently SPiCE for Space (S4S) is a framework that allows measuring the capability of software development processes SMEs Quality Assurance Course 17-19 September 2001 .

What is S4S? x A method of evaluating space software development processes x ISO 15504 (SPiCE) conformant x Developed by ESA for the European space industry x Can be tailored to different classes of critical software SMEs Quality Assurance Course 17-19 September 2001 .

ESA’s Goals x Encourage production of the best possible software products and services Develop customer-supplier relationships based on trust. not control Promote and disseminate best practice concepts proven across the software industry Widen supplier base to companies traditionally outside of the aerospace domain x x x SMEs Quality Assurance Course 17-19 September 2001 .

S4S purposes S4S may be used in three defined modes: x Evaluate the maturity of current software processes within an organisation (Capability Determination Mode) x Provide a solid basis for improving processes (Process Improvement Mode) x Determine capability to comply with (European) space standards (ECSS Compliance Mode) SMEs Quality Assurance Course 17-19 September 2001 .

Engineering and Quality Standards for Space Software ISO 15504 (SPiCE): Software Process Assessment x ISO 15504 SMEs Quality Assurance Course 17-19 September 2001 .Inputs to S4S x ECSS Management.

development. supply. ISO 12207) Scope – Comprehensive » Processes include acquisition. operation.What is ISO 15504? x x x International standard for assessing software processes Developed in parallel with other software engineering standards (ex. maintenance and support – Modular » Can select which processes to assess » Each process is assessed on a scale of capability SMEs Quality Assurance Course 17-19 September 2001 .

2. CL5) » Each process receives a capability level rating x CL5 CL4 CL3 CL2 CL1 CL0 CUS.1 CUS.. ….6 SMEs Quality Assurance Course 17-19 September 2001 . ….ORG.SPiCE : The Assessment Model x Two-dimensional model for processes and process capability – Process Dimension » Process Categories » Processes (P1.. Pn) – Capability Dimension » Capability Levels (CL1.

5. responsibilities identified.1.4.3.1 PA. Level 0 Level 0 Incomplete Incomplete Performed processes are intuitively performed.2.1 PA.1 PA.4.2 Optimizing Optimizing Process Change Process Change Continuous Improvement Continuous Improvement Level 4 Level 4 PA.1 PA.2 Predictable Predictable Measurement Measurement Process Control Process Control Level 3 Level 3 PA.2 Level 5 Level 5 PA.5. input and output work products are available Incomplete Performance and results are incomplete.2 Established Established Process Definition Process Definition Process Resource Process Resource Level 2 Level 2 Managed Managed Performance Management Performance Management Work Product Management Work Product Management Level 1 Level 1 PA.1 PA.1 PA.1 PA.2.2 PA.3.1 PA.2 PA.5.. chaotic processes 17-19 September 2001 SMEs Quality Assurance Course .1 PA.5.3.2 PA.4. resources are managed.4. PA.3.2.2.1 Performed Performed Process Performance Process Performance Managed Process and work products are managed.1.2 PA.Optimising Quantitative measures used for continuous improvement process Predictable Metrics make process performance and results controllable Established Predefined processes are tailored for specific use.

2.1 CUS.5 Measurement ORG.9 SUP.1 SUP.10 Documentation Documentation Configuration Configuration Management Management Quality Assurance Quality Assurance Verification Verification Validation Validation Joint Reviews Joint Reviews Audit Audit Problem Resolution Problem Resolution Safety and Dependability Safety and Dependability Independent Software Independent Software Verification and Validation Verification and Validation Engineering Engineering ENG.1.1 CUS.2 MAN.3 ENG.1 MAN.2.4 CUS.3 ENG.6 ENG.1 MAN.1 CUS.1 ENG.2 CUS.1.4 MAN.1.2 SUP.2 CUS.4 ENG.1.7 ENG.1 SUP.2 Improvement ORG.3 SUP.2 CUS.3 CUS.4 SUP.1.3 CUS.2 CUS.3 MAN.1.1 ENG.1.7 SUP.S4S: The Process Dimension Customer-Supplier Customer-Supplier CUS.4 CUS.1 CUS.1.3 Process Improvement ORG.2 MAN.4.1.1 CUS.6 SUP.1.1.1 Organisational Alignment ORG.2 CUS.8 SUP.2.1.6 SUP.2 ENG.5 MAN.1.4.3 Human Resource Management ORG.2 Development Development System Requirements Analysis System Requirements Analysis & Design & Design Software Requirements Analysis Software Requirements Analysis Software Design Software Design Software Construction Software Construction Software Integration Software Integration Software Testing Software Testing System Integration & Testing System Integration & Testing System & Software System & Software Maintenance Maintenance Management Management MAN.7 SUP.5 Management Management Project Manangement Project Manangement Quality Management Quality Management Risk Management Risk Management Information Management Information Management SMEs Quality Assurance Course 17-19 September 2001 .2.5 SUP.1 ENG.9 SUP.6 Reuse ORG.4 MAN.2 Process Assessment ORG.2 Improvement ORG.4 CUS.3 SUP.4.3 MAN.1 CUS.1.2 CUS.5 CUS.1.1 ENG.4.4 Infrastructure ORG.5 ENG.2 SUP.6 ENG.3 CUS.1 Organisational Alignment ORG.10 SUP.1 CUS.2 Process Assessment ORG.2.1.3 Human Resource Management ORG.2.1.1 Process Establishment ORG.5 ENG.4 SUP.2 CUS.4 ENG.1 CUS.7 ENG.3 Process Improvement ORG.1.5 Measurement ORG.4 Infrastructure ORG.4 CUS.1.5 SUP.2.1.2 ENG.5 Acquisition Acquisition Acquisition Preparation Acquisition Preparation Supplier Selection Supplier Selection Supplier Monitoring Supplier Monitoring Customer Acceptance Customer Acceptance Supply Supply Supply Preparation Supply Preparation Delivery Delivery Requirements Elicitation Requirements Elicitation Operation Operation Operational Use Operational Use Customer Support Customer Support Contract Maintenance Contract Maintenance Organisation Organisation ORG.1.2 CUS.1 Process Establishment ORG.6 Reuse Support Support SUP.1.2.2.8 SUP.3 CUS.2 ENG.2.

Example Assessment Results SMEs Quality Assurance Course 17-19 September 2001 .

1 Supply Preparation + CUS. S4S has: x 4 New Processes – CUS.5 Contract Maintenance – MAN.9 Safety and Dependability – SUP.0 Supply ⇒ CUS.2. 22 New Work Products) SMEs Quality Assurance Course 17-19 September 2001 .2 Delivery x x Many lower level elements (41 New Base Practices.2.10 Independent Software Verification and Validation 2 New Component Processes CUS 2.5 Information Management – SUP. 73 New Notes.Differences between S4S and ISO 15504 In comparison with SPiCE.

context of assessment x Planning x Briefing x Data Acquisition – through interviews and document examination x Data Validation x Process Rating x B C Reporting SMEs Quality Assurance Course 17-19 September 2001 . scope.Performing a S4S Assessment Assessment Steps A x Initiation – define purpose.

2 Software Requirements ENG.3 Software Design SUP.Typical Capability Levels ENG.2 Configuration Management SUP.4 Risk Management SMEs Quality Assurance Course Managed Managed Performed Managed Managed Managed 17-19 September 2001 .2 Project Management MAN.1.3 Quality Assurance MAN.1.

ISO 15504 Improvement Cycle Examine organisation's needs Institutionalised improvements Monitor performance Improvement initiation Sustain improvement gain Validated improvement results Confirm the improvement Implemented improvements Re-Assessment Request Analysed assessment results Initiate process improvement Implement improvement Preliminary improvement programme plan Perform process assessment S4S Assessment results Analyse results and derive action plan Approved action plan SMEs Quality Assurance Course 17-19 September 2001 .

Motivations for Process Improvement Why do you want to improve your processes? x x x x x We need to improve the quality of our products We want to be better than our competitors We need more efficiency … My customer has asked me to do so! SMEs Quality Assurance Course 17-19 September 2001 .

S4S Risk Architecture x x x S4S-R (S4S-Risk) is an extension of S4S with a new Risk Dimension integrated in the method S4S-R addresses risks arising from inappropriate process management S4S-R is orientated to the risks involved in processes (from the S4S model) SMEs Quality Assurance Course 17-19 September 2001 .

Main issues x x x Process orientated risks originate from the actual capability of a process not reaching the required capability Process orientated risks are due to inadequate or insufficient management of the process performance More than one process may contribute to one risk SMEs Quality Assurance Course 17-19 September 2001 .

intermediate) S4S-R allows to identify actual management practices that need to be addressed in order to mitigate risks S4S-R differs from process improvement in that it does not try to reach a target capability but to correct those deficiencies that allow mitigation of the unacceptable process orientated risks SMEs Quality Assurance Course 17-19 September 2001 .Elements of S4S-R x x x x In S4S-R the gap between the actual and required capability is analysed in detail (all process attributes) Risks are evaluated and classified according to their importance (acceptable. unacceptable.

Good luck! SMEs Quality Assurance Course 17-19 September 2001 .

Sign up to vote on this title
UsefulNot useful