IMPRESSION

0103F07_KPI Manual_Part I

IN FO R MA TION S OC IETI ES TEC HN O LO GY ( IS T) P RO GR A MM E

IMPRESSION
Initiative for Software Requirements Management Best Practice in Embedded Systems and E-Services Organisations

DELIVERABLE EXTERNAL WORKPACKAGE: WP2 Implement systematic performance monitoring

D3-KPI Manual Part I - General Methodology

Authors: C. Sorovou, D.Politou, A.Kalligeris, S.Topouzidou (ATC)

Submission Date: 5/10/2001

Partners: Athens Technology Center (GR), The City University (UK), University of Lancaster (UK). Members: Automate (GR), Integrated Aerospace Sciences Corporation (GR), Oxford Company (GR), International Carpet Center (GR), KOLIOS (GR), Euroelectronics (GR), Initiative Media (GR), PRISMA (GR), DISIOS (GR). IST-2000-28760 Page 1 of 14

IMPRESSION

0103F07_KPI Manual_Part I

CONTENTS
1. INTRODUCTION..............................................................................................................................................................3 1.1. DEFINITIONS, A CRONYMS AND A BBREVIATIONS ....................................................................................................3 1.2. LIST OF RELATED DOCUMENTS..................................................................................................................................3 2. METHODOLOGY FOR DEFINING, DEVELOPING, USING AND MAINTENING KPIS .....................3 2.1. GENERAL PRINCIPLES..................................................................................................................................................3 2.2. DEVELOPMENT AND USE OF KPIS ..............................................................................................................................3 2.3. M ETHODOLOGY AT A GLANCE ....................................................................................................................................3 3. APPENDIX A. INDICATIVE PROJECT TEAM S TRUCTURE........................................................................3

IST-2000-28760

Page 2 of 14

IMPRESSION

0103F07_KPI Manual_Part I

1. INTRODUCTION
The purpose of the Key Performance Indicators (KPIs) Manual developed in the context of the IMPRESSION project is to assist the IMPRESSION members to identify, develop and measure KPIs, in order to acquire the benefits of best practice requirements management in software development for embedded systems & systems for e-business and e-services. The manual is designed to be used by the IMPRESSION members as a practical guide to the process of introducing and using KPIs, following a methodology for their implementation. This document describes the general methodology to be followed for defining, developing, measuring, evaluating and reporting KPIs. The Application of the IMPRESSION KPI Manual methodology for e ach user organisation can be found in the respective Part II documents.

1.1. Definitions, Acronyms and Abbreviations

KPI: Key Performance Indicator CSF: Critical Success Factor

1.2. List of Related Documents

IST-2000-28760 IMPRESSION Project Annex I Description of Work D2- Yardsticks for each user D3-KPI Manual Part II - Applying the KPI Manual Ausindustries, Key Performance Indicators Manual, Pitman Publishing, Melbourne, 1996.

IST-2000-28760

Page 3 of 14

IMPRESSION

0103F07_KPI Manual_Part I

2. METHODOLOGY FOR DEFINING, DEVELOPING, USING AND MAINTENING KPIs

This section provides an overview of the total process for developing and using KPIs. It describes the general principles regarding the KPIs as well as the eight steps to be followed in the process of development and use of them.

2.1. General Principles

Key Performance Indicators are the measures that monitor the performance of 'key result areas of business activities, which are absolutely critical to the success and growth of the business. The development and use of KPIs should form the basis for the analysis of an organisations current performance, its future requirements and the improving strategies required for ongoing success. Some general principles regarding the KPIs should be taken under consideration during their application: KPIs should not be an end in themselves, but be considered as an aid to management. They are a start to a proper informed debate that should lead to a plan for improvement. They should be seen within their local context and have more a meaning as a comparison over time than as a comparison between organisations. A set of performance indicators should be balanced. For example, measures of efficiency should be set against measures of effectiveness and measures of cost against quality and user perception. After being proposed and applied, KPIs should be reviewed and updated. The review determines the management utility of each indicator and the feasibility of getting source data for continuing use. The targeted performance description, which is described in measurable terms through the KPIs, must be deployed to the organisational level that has the authority and knowledge to take the necessary action.

Every initial proposal for KPIs is expected to be imperfect. However, it is important for the organisations to understand and apply the appropriate KPIs so that they can develop some experience in using them, from which they can derive real expertise.

IST-2000-28760

Page 4 of 14

IMPRESSION

0103F07_KPI Manual_Part I

2.2. Development and use of KPIs

There is a nine steps approach in the procedure of development and use of KPIs, which is divided into four phases: Initiate, Build, Implement and Review. PHASE I - INITIATE STEP 1: Create the project team It is proposed that a cross- functional team will be set up to implement the KPIs measurement. The team must be consisted of high- level employees, coming from different sections that have a clear view of the organisations goals and priorities. The team must also be familiar with the performance measures or be given a short training course, otherwise. The number of the team members depends on the size of the organisation and may vary between 4 to 6 employees. An indicative structure of the Project team is presented in Appendix A. STEP 2: Align KPIs development with key areas and improvement strategies in requirements engineering A strong correspondence must be established between the key areas, the Business Goals to be achieved and the improvement strategies to be followed, as KPIs will be developed to support these strategies. Therefore, the fist step is the definition of the key areas and the respective Business Goals, which are critical for the organisation, so that they can be aligned with the KPIs we intend to develop. Such Business Goals may be: Improve RE Process Maturity Improve Managerial Satisfaction Improve User Satisfaction Improve Efficiency of the System Implementation Process Improve Maintainability of System Improve Productivity of S/W development

The KPIs should focus on simple ideas that represent key goals of high relevance to the organisation in terms of the project assessed to ensure that the organisation comprehends the link between them and the business strategy. STEP 3: Explain the purpose of the use and development of KPIs The aim of measuring an organisations performance is to assess how well it is implementing strategies and action plans. To improve an orga nisations performance you must measure it first. It is therefore necessary to develop KPIs, which is the set of indicators that will monitor the performance of 'key result areas of business activities. KPIs are a significant tool to assist improvement towards organisational objectives aiming to the

IST-2000-28760

Page 5 of 14

IMPRESSION

0103F07_KPI Manual_Part I

overall prosperity. KPIs also create an essential feedback and learning mechanism to support management decisions. The purpose of the use and development of KPIs must be disseminated in the organisation, from the lower to the higher- level employees in order to make them understand their own role in helping to achieve the organisations goals and the measurement and improvement process. STEP 4: Establish an agreed process for KPIs development and use. After having created the team that will be responsible for the KPIs development, defined the key goals, which have to be measured and therefore aligned with the KPIs, and communicated and disseminated the purpose of KPIs implementation, an agreed development and use process must be established. The process should include: Definition of the team members duties For each KPI that will be selected by the project team, a set of Input, Calculate, Output and Report Procedures should be reported during the implementation phase as follows: Input involves all procedures for collecting data and the parameters used for evaluating KPIs. Calculate involves all mechanism and calculation required for evaluating KPI, from given Input data. Output involves all processing and presentation of results e.g. graphical forms, data etc. Report involves all reporting Requirements for monitoring RE procedure improvement. It may consist of a composite report where progress is shown through a number of Output formats

PHASE II - BUILD STEP 5: Identify critical success factors for the organisation. Critical success factors are the core competencies or capabilities that must be accomplished for the organisation to achieve its vision. The set of critical success factors related to the process of requirements engineering and the Business Goals already defined, may include the following:

IST-2000-28760

Page 6 of 14

IMPRESSION Business Goals

0103F07_KPI Manual_Part I

Critical Success Factors Use of RE Standards Improve Traceability of requirements throughout the System Lifecycle Reduce Uncertainty throughout the System Lifecycle Reduce Users (internal external) Complaints Get things right in the first place Reduce time to market Implementation within the time table Keeping within budget

Improve RE Process Maturity

Improve Managerial Satisfaction

Improve User Satisfaction

Improve Efficiency of the System Implementation Process

Improve Maintainability of System

Improve extentability Reduce maintenance cost

Improve Productivity of S/W development

Reduce code production related to rework Reduce Effort of Rework

STEP 6: Teams define yardsticks and select KPIs. Following the definition of Business Goals and critical success factors, a set of yardsticks should be defined. The yardsticks will specify the practical result that each organisation and the project should achieve according to the specified Business Goals. They will also serve in monitoring and evaluating the process towards the achievement of these goals. A set of Yardsticks for the Business Goals and the Critical Success Factors defined, may be:

IST-2000-28760

Page 7 of 14

IMPRESSION Critical Success Factors

0103F07_KPI Manual_Part I

Yardsticks The increased number of RE Standards

Use of RE Standards

The Existence of Requirements Document Improve Traceability of requirements throughout the System Lifecycle The increased number of unique requirements in Requirements Document A high percentage of unique requirements

Reduce Uncertainty throughout the System Lifecycle Reduce Users (internal external) Complaints Get things right in the first place

A low percentage of reworked requirements

The decreased number of modifications coming from RE errors A high percentage of requirements satisfied in the initial design The minimization of time between system conception and delivery The minimization of extra time spent for the implementation The minimization of extra time spent for the RE process

Reduce time to market

Implementation within the time table

A low percentage of deviation from total budget Keeping within budget A low percentage of deviation from the RE budget The increased number of additional requirements satisfied / number of additional requirements requested

Improve extentability

IST-2000-28760

Page 8 of 14

IMPRESSION Reduce maintenance cost

0103F07_KPI Manual_Part I

The lowering in cost of maintenance per 1000 lines of code The decreased number of code lines / person day spent on rework The minimization of the number of person days needed for rework due to requirements capturing inefficiency

Reduce code production related to rework

Reduce Effort of Rework

KPIs should have a set of characteristics that will allow them to be effective, namely they should be: Restricted to a few strategically important objectives. Based on strategic goals, making general goals more specific. Consistent, giving a balanced view, relating to financial performance, satisfaction, efficiency and improvement. Uncomplicated and understood by those being measured. Easy to collect data and calculate. Dynamic (reviewed at least annually as a part of business planning, to ensure that they reflect business priorities). Agreed, not imposed by management. Communicated.

KPIs should not: Conflict with other measures without prioritising. Produce misleading information. Be or viewed as trivial.

All the above characteristics should be taken under consideratio n for the process of the KPIs development. Suggested KPIs related to the respective Yardsticks, Critical Success Factors and Business Goals are:

IST-2000-28760

Page 9 of 14

IMPRESSION

0103F07_KPI Manual_Part I

Yardsticks

Key Performance Indicators

The increased number of RE Standards

Number of RE Standards used

The Existence of Requirements Document The increased number of unique requirements in Requirements Document A high percentage of unique requirements

Existence of Requirements Document

Number of Unique Requirements

% Unique Requirements completed

A low percentage of reworked requirements

% Requirements to be reworked

The decreased number of modifications coming from RE errors A high percentage of requirements satisfied in the initial design The minimization of time between system conception and delivery The minimization of extra time spent for the implementation The minimization of extra time spent for the RE process A low percentage of deviation from total budget A low percentage of deviation from the RE budget

Number of modifications coming from RE errors / month % Requirements satisfied in the initial design Time between system conception and delivery Number of extra months spent for the implementation Number of extra months spent for the RE process

% Deviation from total budget

% Deviation from the RE budget

IST-2000-28760

Page 10 of 14

IMPRESSION

0103F07_KPI Manual_Part I

The increased number of additional requirements satisfied / number of additional requirements requested

Number of additional requirements satisfied / number of additional requirements requested

The lowering in cost of maintenance per 1000 lines of code

Cost of maintenance per 1000 lines of code

The decreased number of code lines / person day spent on rework

Lines of code/person day spent on rework

The minimization of the number of person days needed for rework due to requirements capturing inefficiency PHASE III - IMPLEMENT

Number of person days needed for rework due to requirements capturing inefficiency

STEP 7: Develop display, reporting and review frameworks at all levels. The KPIs should be displayed and reported frequently. The KPIs reporting should show management the trend. Thus, graphics going back are considered a good practice. Such a daily or weekly display will not take very long once the process is embedded. The measurements collected should be analysed and form the basis for the final report. STEP 8: Facilitate the use of KPIs to assist performance improvement Education and training will facilitate the use of KPIs inside the organisation. Employees should be motivated and empowered by giving them controlled authority and responsibility for improvement, in relation to the processes they are responsible for. Managers should be encouraged to delegate responsibility and workers to identify their own measures and look for solutions.

PHASE IV - REVIEW STEP 9: Refine and modify KPIs to retain their relevance. The KPIs should be reviewed every time the critical success factors are reviewed. They also KPIs should be checked in terms of their relevance and be adapted to the eventually changing environment, processes and conditions. IST-2000-28760 Page 11 of 14

IMPRESSION

0103F07_KPI Manual_Part I

Executives must be actively involved in reviewing the measures and leading the implementation effort. Continuous communication regarding the status and results of the measurement system is important in terms of cont inually improving the system.

2.3. Methodology at a glance

The overall methodology is represented in the figure below:

STEP 1: Create the project team

STEP 2: Align KPIs development with key areas and improvement strategies in requirements engineering

STEP 3 : Explain the purpose of the use and development of KPIs

STEP 4 : Establish an agreed process for KPIs development and use

Initiate

STEP 6 : Teams define yardsticks and select KPIs

STEP 5 : Identify critical success factors for the organisation

Build

STEP 7 : Develop display, reporting and review frameworks at all levels

STEP 8 : Facilitate the use of KPIs to assist performance improvement

Implement

STEP 9 : Refine and modify KPIs to retain their relevance

Review

IST-2000-28760

Page 12 of 14

IMPRESSION

0103F07_KPI Manual_Part I

3. Appendix A. Indicative Project Team Structure

The project team will be consisted of high- level employees coming from different departments of the organisation, namely: The Co-ordinator: High ranking official who will undertake the coordination of the team. He has an overall view of the project and he will communicate with the team involved in the project for the definition of KPIs. He will also be responsible for the measurements. General Manager: He has an overall view of the organisation and he will set the business goals and discuss the priorities and the critical success factors with the rest members of the team, in order to proceed with the definition of the KPIs. The Production / Technical Manager or high-ranking representative: He coordinates the resources and activities required for the production (in the case of industrial production companies) and the project implementation process (in the case of IT companies). He will be responsible for the data obtained, related the production & the IT implementation process. The IT Manager or high-ranking representative: He will provide all the necessary knowledge about the technical part that has to be measured. He will also communicate with the software engineers and the developers during the process of the definition, the implementation and the dissemination of KPIs. The Marketing & Sales Manager or high-ranking representative: He is responsible for the organisations promotion, the competition and the improvement of the organisations market stake. He will ensure that the decisions made are according to the marketing strategies. The Financial Manager or high-ranking representative : He will define and monitor the critical financial figures. Quality Manager or high-ranking representative: He is responsible for the quality assurance of the organisation. He ensures that the procedures followed during the KPIs definition & implementation procedure are according to the organisations predefined quality assurance procedures. All team members will participate in the process of definition, implementation, reporting and reviewing of the KPIs. The indicative team structure is represented in the figure below:

IST-2000-28760

Page 13 of 14

IMPRESSION

0103F07_KPI Manual_Part I

Customer

Implement

Co-ordinator

Define

General Manager Production Manager Report IT Manager Marketing & Sales Manager Financial Manager Review

Quality Manager

IST-2000-28760

Page 14 of 14